Memory Retrieval Architecture v2 (Epic) #21
Description
Context
We are redesigning the memory retrieval architecture to match the strongest known patterns from qmd and claude-mem, while keeping ELF invariants (evidence binding, deterministic writes, and rebuildable indexing).
Goal
Deliver a best-in-class retrieval pipeline focused on recall, precision, ranking stability, and controllable context growth.
Scope (tracked by sub-issues)
- Implement query expansion with multi-query hybrid retrieval and RRF fusion #12 Implement query expansion with multi-query hybrid retrieval and RRF fusion
- Add chunked embeddings with overlap and chunk stitching #13 Add chunked embeddings with overlap and chunk stitching
- Add context metadata and context-aware boosting in retrieval #14 Add context metadata and context-aware boosting in retrieval
- Normalize rerank scores and apply position-aware blending #15 Normalize rerank scores and apply position-aware blending
- Add optional note-level diversity control (MMR / redundancy filter) #28 Add optional note-level diversity control (MMR / redundancy filter)
- Add optional deterministic ranking signals (lexical overlap, hit reinforcement, decay) #29 Add optional deterministic ranking signals (lexical overlap, hit reinforcement, decay)
- Introduce progressive disclosure search endpoints (index, timeline, details) #16 Introduce progressive disclosure search endpoints (index, timeline, details)
- Add structured memory fields with field-level embeddings #17 Add structured memory fields with field-level embeddings
- Graph-lite entities and relations as structured memory fields (no graph DB) #27 Graph-lite entities and relations as structured memory fields (no graph DB)
- Add LLM cache for query expansion and reranking #18 Add LLM cache for query expansion and reranking
- Add search explainability outputs #19 Add search explainability outputs
- Build a retrieval evaluation harness and regression gates #20 Build a retrieval evaluation harness and regression gates
Backlog (explicitly out of scope for this epic)
- Add a read-only web viewer for sessions and traces #26 Add a read-only web viewer for sessions and traces (deferred)
Acceptance criteria
- All in-scope sub-issues are delivered.
- Each change that affects ranking is evaluated with the retrieval harness (same dataset, before/after comparison).
References