feat(search): hybrid retrieval + FTS/vector indexes — removes the search scaling cliff#934
Closed
seonghobae wants to merge 1 commit into
Closed
feat(search): hybrid retrieval + FTS/vector indexes — removes the search scaling cliff#934seonghobae wants to merge 1 commit into
seonghobae wants to merge 1 commit into
Conversation
…ts the scalability plan) /api/search previously computed ts_rank_cd minus cosine_distance over every owner-scoped row on every query — a full sequential scan that no index could serve (verified by EXPLAIN: the planner ignored GIN/HNSW even when present) and an unnormalized fusion of incomparable scales. Reshape (docs/plans/2026-07-05-search-scalability-hybrid-retrieval.md): - Candidate generation is now up to four index-eligible arms: lexical email/ attachment arms gated by @@ (GIN-served, ts_rank_cd ordered, LIMIT) and vector email/attachment arms as pure cosine-distance ORDER BY + LIMIT (HNSW-served). The embedding-failure fallback keeps working (lexical arms only). - Fusion moves to the app as reciprocal-rank fusion (RRF, k=60), replacing the score subtraction. Metadata + reply counts are fetched in a second bounded query for the fused candidate ids only, so joins never disqualify the ANN index. - Owner (user/organization) scoping preserved on every arm and the metadata query. Migration 0010 (postgresql-guarded, CONCURRENTLY in an autocommit block): GIN expression indexes on email/attachment text and hnsw (ivfflat below pgvector 0.5) vector_cosine_ops indexes, with the DDL shared between the migration and tests via db/search_indexes.py. Verification: - tests/test_search_pg_indexes.py (postgres marker) ran against a real pgvector 16 instance: EXPLAIN shows the lexical arm using ix_email_records_body_fts and the vector arm using ix_email_records_embedding_vec. - tests/test_search.py rewritten for the new shape (16 passed): endpoint behavior, owner scoping, fallback, session split, RRF fusion, arm shapes. - Regressions green: alembic/bootstrap (37), emails_api/db_session (66), ruff clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017RkKdtHRLG4wSLh6PVsp8J
Contributor
Author
|
Folded into #938, which is now retargeted to |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What (implements #922's plan)
/api/searchseq-scanned every owner-scoped row per query — proven index-blind (EXPLAIN ignored GIN+HNSW even when present) because of query shape: no@@predicate and FTS+vector fused in one ORDER BY of incomparable scales.Reshape
@@-gated → GIN,ts_rank_cdordered, LIMIT) + vector email/attachment (pure<=>ORDER BY + LIMIT → HNSW). Embedding-failure fallback preserved (lexical-only).ts_rank_cd − cosine_distancesubtraction.Migration 0010
GIN expression indexes (email body, attachment content) +
hnsw vector_cosine_ops(ivfflat < pgvector 0.5),CONCURRENTLYin an autocommit block, postgres-guarded. DDL is a single source (db/search_indexes.py) shared by migration and tests.Verification
tests/test_search_pg_indexes.pyran against pgvector:pg16 — EXPLAIN shows the lexical arm onix_email_records_body_ftsand the vector arm onix_email_records_embedding_vec. ✅tests/test_search.pyrewritten for the new shape — 16 passed (endpoint, owner scoping, fallback, session split, RRF, arm shapes).Behavior notes for review
Result ranking and membership change by design (that's the fix): lexical candidates now require an actual FTS match; semantic-only hits still surface via the vector arms. Scores are RRF values (rank-based), not raw score arithmetic.
🤖 Generated with Claude Code