feat(search): hybrid retrieval + FTS/vector indexes + grounded cited answer (folds #934)#938
Open
seonghobae wants to merge 2 commits into
Open
feat(search): hybrid retrieval + FTS/vector indexes + grounded cited answer (folds #934)#938seonghobae wants to merge 2 commits into
seonghobae wants to merge 2 commits into
Conversation
…ts the scalability plan) /api/search previously computed ts_rank_cd minus cosine_distance over every owner-scoped row on every query — a full sequential scan that no index could serve (verified by EXPLAIN: the planner ignored GIN/HNSW even when present) and an unnormalized fusion of incomparable scales. Reshape (docs/plans/2026-07-05-search-scalability-hybrid-retrieval.md): - Candidate generation is now up to four index-eligible arms: lexical email/ attachment arms gated by @@ (GIN-served, ts_rank_cd ordered, LIMIT) and vector email/attachment arms as pure cosine-distance ORDER BY + LIMIT (HNSW-served). The embedding-failure fallback keeps working (lexical arms only). - Fusion moves to the app as reciprocal-rank fusion (RRF, k=60), replacing the score subtraction. Metadata + reply counts are fetched in a second bounded query for the fused candidate ids only, so joins never disqualify the ANN index. - Owner (user/organization) scoping preserved on every arm and the metadata query. Migration 0010 (postgresql-guarded, CONCURRENTLY in an autocommit block): GIN expression indexes on email/attachment text and hnsw (ivfflat below pgvector 0.5) vector_cosine_ops indexes, with the DDL shared between the migration and tests via db/search_indexes.py. Verification: - tests/test_search_pg_indexes.py (postgres marker) ran against a real pgvector 16 instance: EXPLAIN shows the lexical arm using ix_email_records_body_fts and the vector arm using ix_email_records_embedding_vec. - tests/test_search.py rewritten for the new shape (16 passed): endpoint behavior, owner scoping, fallback, session split, RRF fusion, arm shapes. - Regressions green: alembic/bootstrap (37), emails_api/db_session (66), ruff clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017RkKdtHRLG4wSLh6PVsp8J
Retrieval existed but nothing generated from it: search returned rows, never answers. This adds /api/search/answer — question answering over the user's own emails with the property that an answer cannot reference material retrieval did not surface. - services/rag_service.py: the model sees only retrieved, owner-scoped email content passed as JSON data (prompt-injection hardening, mirrors api/llm.py), must return cited_email_ids, and citations are validated against the retrieved set (unretrieved ids dropped + logged). Empty retrieval makes no LLM call at all. Context bounded to 5 emails x 1500 chars. - /api/search/answer reuses the hybrid retrieval arms + RRF fusion from the search reshape (owner scoping preserved), fetches bounded metadata for the fused top ids, and returns answer + citations (subject/sender/snippet) + provenance. Tests (6): no-context -> no LLM call; unretrieved citation ids dropped; context bounded to MAX_CONTEXT_EMAILS; long content truncated; endpoint happy path returns answer+citations from retrieved-only context; empty query short-circuits. Search suite still green (16), ruff clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017RkKdtHRLG4wSLh6PVsp8J
seonghobae
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What (the moat demo: cited answers over your own mailbox)
Retrieval existed but nothing generated from it.
/api/search/answeranswers a question from the user's own emails with a hard property: the answer cannot reference material retrieval did not surface.services/rag_service.py: model sees only retrieved, owner-scoped content as JSON data (injection-hardened, mirrorsapi/llm.py); must returncited_email_ids; citations validated against the retrieved set (unretrieved ids dropped + logged); empty retrieval → no LLM call at all; context bounded (5 emails × 1500 chars).answer + citations(subject/sender/snippet) + provenance.Verification (local)
6 new tests (no-context short-circuit · unretrieved-citation drop · context bounds · truncation · endpoint happy path with retrieved-only context · empty-query) + search suite 16 green + ruff clean.
Stack
#934 (hybrid retrieval) → this. Completes the moat chain: substrate (#927) → grounded extraction (#931) → scalable retrieval (#934) → cited generation.
🤖 Generated with Claude Code