fix(search-algorithms): Soundex and offset bugs; add BM25 inverted index

[frankstupak]

What this does

Uplifts src/api/search-algorithms with correctness fixes to the phonetic and match-offset logic, a new inverted index that makes ranked search dramatically faster, and a memory-lean edit-distance — plus the tests and benchmarks to back all of it. Public API is unchanged; every ranked result is numerically identical to before.

Scoreboard: +36 tests (98 total pass), tsc clean, eslint clean, and BM25 goes ~29x faster on repeated queries.

---

Correctness bugs fixed

Soundex was wrong on the classic reference cases

The frozen implementation skipped vowels/H/W but never applied the separator rules, and never de-duplicated the first letter's own code. It failed the canonical Soundex reference table:

Name	Frozen ref	Correct	Rule missed
Tymczak	T520	T522	a vowel between two same-coded consonants re-codes the second
Pfister	P123	P236	first two letters share a code → collapse to P
Honeyman	H500	H555	vowel separators keep both Ns

Now 9/9 against the reference table (Robert R163, Rubin R150, Ashcraft A261, etc.), and homophones like Smith/Smyth still collapse together as phonetic search relies on. H and W stay transparent (Ashcraft A261, not A226); vowels act as separators.

Repeated words reported the wrong position

fuzzySearch, phoneticSearch, and wildcardSearch split fields with split(/\s+/) and recovered positions with fieldValue.indexOf(word). That has two bugs:

1. Punctuation stayed glued to words ("TypeScript," never equals the query "TypeScript"), quietly hurting recall.
2. indexOf always returns the first occurrence, so every repeat of a word reported the same (wrong) offset — which corrupts highlighting.

Replaced with a single offset-tracking Unicode tokenizer. A field containing "word … word … word" now yields three distinct, correct offsets instead of three copies of the first.

N-gram search couldn't find short queries in real documents

ngramSearch compared the whole field against the query with bigram Jaccard. When the query is much shorter than a multi-word field, their padded bigram sets barely overlap relative to their union, so the score collapses toward zero and never clears the threshold — e.g. "JavaScript" vs a 3-word title scored 0.36 and a one-word title match was effectively unreachable for realistic documents. Now it also compares per word and keeps the best match (whole-field score retained as a floor), so short queries actually match, and the highlighted span is the specific word rather than the entire field.

---

Performance

Inverted index for BM25 / TF-IDF (~29x / ~49x on repeated queries)

RankingAlgorithms.calculateBM25 / .calculateTFIDF re-tokenize every document and rebuild the document-frequency table on every query — O(N·L) regardless of how selective the query is. New InvertedIndex does that work once at build time, then answers each query by walking only the query terms' postings. SearchEngine builds it lazily and invalidates it on addItems / updateItems / removeItems.

Scores are a drop-in — verified identical to the stateless functions (max abs diff 0.0 over 1750 doc×query checks, same top-5 ordering; a repeated query term is counted the same way).

Benchmark (1000 docs, 20–60 words each, 200 queries):

	frozen ref	indexed	speedup
BM25	15.356 ms/query	0.520 ms/query	29.5x (25.3x incl. one-time build)
TF-IDF	19.215 ms/query	0.390 ms/query	49.3x

End-to-end SearchEngine.search({algorithm:"bm25"}) with the index cached: ~1.0 ms/query.

Levenshtein: full matrix → rolling buffer

levenshteinDistance allocated the full (m+1)×(n+1) matrix. Damerau transposition only needs the row two above the current one, so it's now a rolling 3-row buffer — identical results (verified against an independent full-matrix reference, including transpositions and custom edit costs). On a 2003×2003 pair the allocation drops from ~32.1 MB → ~48 KB with the same output.

---

Build fix

server.ts imports and registers @fastify/swagger, but it was never declared as a dependency — the subproject failed tsc on the missing module out of the box. Added it to dependencies (matching the version already used by api-scenarios).

---

Tests

src/uplift.test.ts adds 36 tests: the Soundex reference table, Levenshtein rolling-buffer parity vs an independent full-matrix reference (incl. transposition + custom costs), InvertedIndex BM25/TF-IDF equivalence + incremental addDocuments, and SearchEngine offset/tokenization/n-gram/index-invalidation coverage. All 62 original tests still pass — 98 total. tsc and eslint are clean.

— Lumen Industries