v1.0.0 — durable migrations, analytics, semantic retrieval (usearch HNSW), Kimetsu Remote

[RodCor]

Ships Kimetsu v1.0.0: the brain becomes durable, measurable, semantically sharp, shareable over the network — and proven to scale to ~1M memories.

Highlights

• Durable schema migrations — brain.db migrates forward automatically on open (versioned, forward-only, one transaction per migration, backup sidecar before any version-advancing migration).
• Proof-of-value analytics — kimetsu brain insights (+ MCP tool): hit/skip rate, citation rate, proposal acceptance, usefulness trend, corpus health, token economy.
• Semantic retrieval (usearch HNSW) — meaning-based recall on the embeddings build, O(log N) per query; embedding-MMR + absolute relevance floors. Lean (FTS-only) build unchanged.
• Warm embedder daemon — a single per-user daemon holds the ONNX models in memory and serves semantic retrieval to the UserPromptSubmit hook over a local socket / named pipe; ≤300ms budget with hard FTS fallback (the prompt is never blocked); pre-warmed at harness SessionStart.
• Cross-encoder reranking — final-stage rerank of the candidate pool (fastembed curated models or any HF ONNX repo via hf-hub). Wired into the local daemon and the Remote server.
• Proactive & cost-shrinking recall — "known pitfalls" before the first attempt, task classification, per-run dedupe ledger, on-demand capsule expansion (adaptive sublinear budget).
• Kimetsu Remote (beta) — share one brain per repo over HTTP MCP; bearer auth; optional org-brain + server-side ingest; TLS via proxy or in-process; ANN pre-warm at startup; operator-level --reranker stage.

Retrieval quality — measured, not vibes

New measurement tooling drove every ranking default in this release:

• kimetsu brain eval — recall@2/4 + MRR across fts / semantic / semantic+rerank modes against a committed fixture.
• kimetsu brain bench — full embedder × reranker grid over a 100-memory / 210-case dataset built from real exported memories (confusable clusters, oblique phrasings, in-domain no-answer, open multi-answer cases); per-case expected-vs-obtained, latency, and per-combo RAM (process-isolated). --remote drives the same dataset through a real kimetsu-remote server over HTTP (sequential + concurrent) and measures server RSS + throughput.
• Benchmark-chosen defaults: local = jina-v2-base-code + ms-marco-tinybert-l-2-v2 (MRR 0.914 @ ~132ms retrieval+rerank — the fastest reranked combo, within ~2% of the grid best); remote = same embedder + --reranker jina-reranker-v1-tiny-en (no latency budget there). FTS-only baseline: MRR ~0.81.
• Relevance floors that actually fire: an IDF-weighted lexical coverage floor (kills "matches only the project name" noise, with query-side stemming so inflected words still match), and a model-aware AUTO cosine floor (0.35 on bge-family, off elsewhere — the remote benchmark caught the bge-calibrated floor deleting relevant jina-v2 results, MRR 0.90 → 0.77).

Agent-loop fixes (found via real-world usage)

• Stop hook emits valid hook JSON — bare-text banners tripped Claude Code's "invalid stop hook JSON output"; the harvest cue now actually reaches the model via decision: "block".
• First-prompt hang fixed — the lazily-spawned daemon inherited the hook's stdout handle on Windows and could stall the harness until its hook timeout; the daemon now binds before loading models and the hook clears HANDLE_FLAG_INHERIT before spawning.
• Write tools work out of the box locally — kimetsu.mcp_write_tools (default on) replaces the env-only gate that contradicted the brain's own "record a lesson" workflow every session. Remote stays env-only, default-deny (cloned-repo config is untrusted).

Performance & scale (this release's deep-dive)

The vector path went from a brute-force O(N) scan to a real ANN index, validated at 1,000,000 memories:

• usearch HNSW replaces sqlite-vec (dropped) for both retrieval and conflict-detection-on-write — both now O(log N).
• Batched embedding for bulk ingest; parallel index build across cores; incremental maintenance + reconcile-on-hit (never stale); sidecar persistence (a restarted server loads instead of rebuilding) + startup pre-warm on Remote.
• f16 index quantization by default (KIMETSU_ANN_QUANTIZATION=i8|f32 to tune) — the index is candidate-generation only, with an exact f32 cosine rerank from SQLite, so quantizing is near-lossless (recall@10 0.994 f16 / 0.966 i8).
• Measured at 1M (bge-small, ext4): ~1.8 s p99 semantic retrieval; ~3.1 GB RAM (f16) / ~2.8 GB (i8) — fits a 3–4 GB container. Remote no longer collapses at scale (0 errors, isolation enforced).
• Remote at 100 memories (HTTP, concurrency 4): ~0.906-0.909 MRR, ~420-700ms/request, ~4-5 rps depending on embedder.

Design + plan: docs/superpowers/specs/2026-06-06-tier3-hnsw-usearch-design.md, docs/superpowers/plans/2026-06-06-tier3-hnsw-usearch.md.

Verification

Every change gated on cargo fmt, clippy -D warnings (lean and embeddings flavors), the workspace failure-audit (empty), and cargo tree -p kimetsu-cli -i usearch empty (the native ANN dep never reaches the lean CLI). 802 workspace tests pass (lean build; the daemon/reranker/ANN suites additionally run under --features embeddings). Retrieval defaults validated on three datasets (synthetic fixture, 18-real-memory corpus, 100-memory confusable-cluster corpus) plus live latency smokes on the real brain.

🤖 Generated with Claude Code