Before adopting as a primary index: BM25 lexical leg + scale-proof benchmarks (+ good-to-haves)

[octalpixel]

Evaluating samesake as the primary hybrid index for a production workload. A fair amount is already in good shape, and calling it out matters because it shaped what I think the remaining gaps are:

• per-result explain with per-channel value / weight / contribution (packages/server/src/core/explain.ts) — excellent for trusting why something ranked where it did;
• efSearch per-query HNSW recall/latency dial (10–1000) + pgvector 0.8 iterative scans (see packages/server/CHANGELOG.md);
• per-channel weights, RRF fusion, and BYO rerank adapters (Cohere/Voyage).

Given all that, two things still hold me back from adopting it as the primary index — plus a few good-to-haves that would seal the decision.

1. Lexical relevance rides ts_rank_cd, not BM25

The FTS channel orders by ts_rank_cd over websearch_to_tsquery — packages/server/src/core/search.ts:

ORDER BY ts_rank_cd(fts, <andTsq>) DESC, ts_rank_cd(fts, <orTsq>) DESC [, ts_rank_cd(fts_phon, <phonTsq>) DESC]

ts_rank_cd is cover-density ranking — it is not BM25. It has no term-frequency saturation (BM25 k1) and no document-length normalization (BM25 b). On real corpora that means:

• long / keyword-dense documents over-rank (TF keeps scaling roughly linearly);
• short, on-topic documents lose to long, diffuse ones.

For a framework whose headline is hybrid search, the lexical leg is the part adopters most expect to behave like a Lucene/Elasticsearch BM25 out of the box, and today it measurably won't on length-varied corpora. This is the single biggest hesitation before making it the primary index.

Ask

• Offer a true BM25 lexical path — e.g. the ParadeDB pg_search BM25 index, or a documented BM25 scoring function over the existing tsvector — selectable per channel.
• Add a lexical-only nDCG@10 fixture comparing ts_rank_cd vs BM25 on a length-varied corpus, gated in the eval harness.
• Document the tradeoff (and the recommended fallback) for environments where a BM25 index isn't available.

2. No relevance / latency numbers beyond ~5k documents

BENCHMARKS.md is thorough, but every figure is at 4,555 → 5,052 docs / 20–28 sources ("median latency 0.6s warm @5,052 docs"). There is nothing at 50k / 100k / 500k / 1M. Postgres FTS + pgvector behave very differently across two orders of magnitude — HNSW recall/ef tradeoffs, planner flips, index-build time and memory, autovacuum pressure — so confidence at production scale is currently a projection, not a measurement. For an index, "how does ranking quality and tail latency hold as the corpus grows 20–200×?" is the load-bearing adoption question.

Ask

• Add a scale sweep to the eval harness: synthetic corpora at 10k / 100k / 1M, same golden queries + judge.
• Report per-scale nDCG@10 (does relevance hold?) and p50 / p95 / p99 latency + ingest / index-build time (does the tail hold?).
• Publish the curve in BENCHMARKS.md so adopters can read confidence at their own target size.

Good-to-haves (would seal the "adopt" decision)

• A scale/adoption playbook that turns §2's data into guidance: for a given corpus size, the recommended efSearch / HNSW m / ef_construction and the expected recall + latency band. The dials already exist (efSearch); what's missing is "what should I set them to at 100k, and what do I get?"
• Ingest throughput numbers alongside the scale sweep (docs/sec through enrich → embed → index) so adopters can size a backfill.
• Keep the ts_rank_cd-vs-BM25 ablation in CI once §1 lands, so the lexical leg can't silently regress.

Acceptance

• BM25 lexical path selectable per channel (§1)
• Lexical-only nDCG@10 fixture: ts_rank_cd vs BM25 on a length-varied corpus, gated (§1)
• Eval scale sweep at 10k / 100k / 1M with nDCG@10 + p50/p95/p99 latency + index-build time (§2)
• Scale curve published in BENCHMARKS.md (§2)
• Scale/adoption playbook: corpus size → efSearch/HNSW params → expected recall/latency (good-to-have)
• Ingest throughput numbers in the sweep (good-to-have)

Blocked by: none — both gaps are independent and can land separately.