nova-bf: Qdrant word-tokenizer match_text semantics, single-pass + threaded filter

[Seth Ockerman (OckermanSethGVSU)]

What

Switches match_text/match_text_from_query from the \b-regex substring approximation to real Qdrant word-tokenizer semantics (split on non-alphanumeric, lowercase each token, AND of query tokens vs the row's token set), and rebuilds the evaluator around a single tokenization pass per field (thread-parallel, byte-bounded batches) instead of a column scan per distinct query word. Builds on this branch's earlier commits (large_string cast, B+D single-pass hybrid), replacing the bit-identical hybrid with the full semantics switch.

Semantics change (deliberate): ground-truth bits move on hyphen/punctuation/underscore/unicode edges — high-fat now matches "high fat diet", C++ tokenizes to c, com_content contains token content. In exchange, nova-bf now computes exactly what a Qdrant word/lowercase: true full-text index computes. Old GT files with text filters should be regenerated.

Design

• One tokenizer definition (nova_bf/tokenize.py): query strings and corpus columns run the same Arrow kernels (split_pattern_regex on [^\p{L}\p{N}]+, then utf8_lower — split-then-lower, Qdrant's order). No cross-language equivalence to maintain.
• filters._token_row_masks: per field, one pass — split/lower each batch, index_in against the query vocabulary, one boolean scatter per batch into a shared (n_tokens, n_rows) grid from a thread pool (disjoint row ranges → race-free; Arrow kernels release the GIL). Cost is ~independent of query-vocabulary size.
• evaluate() unions token needs across all of a filter's text conditions per field: the production ms_marco shape (must on text + 9 should slots on url) tokenizes each column once, not per condition.
• Batches are byte-bounded (32MB text each): bounds transient memory on huge-document corpora and stays far under split_pattern_regex's 2^31 per-batch list-offset ceiling (which the large_string cast alone did not fix).

Review (8-angle, adversarially verified)

The review caught a real correctness bug in the first draft: query-side tokenization via Python str.lower/str.isalnum vs corpus-side Arrow — the two disagree on thousands of codepoints (Unicode-version skew; full-vs-simple case mapping), reproduced end-to-end (İstanbul, Greek final-sigma words silently matched nothing). Fixed by routing query strings through the same Arrow kernels; a dedicated test pins the divergent codepoints. Also from review: the residual list-offset overflow, transient-memory bounding, per-field pass sharing, and the direct-scatter design (no argsort/searchsorted).

Known residual divergence vs Qdrant: Arrow utf8_lower = simple case mapping; Qdrant (Rust) = full mapping. Differs on ~50 codepoints (word-final Σ, İ). Documented in tokenize.py.

Validation

• 266 tests pass, incl. 90 fuzz cases asserting the Arrow pipeline against an independently formulated pure-Python reference, and pinned tests for the semantics change and the divergent codepoints.
• Live Qdrant parity (1.18.2, word tokenizer + lowercase index): 300/300 queries EXACT — same match sets — on 20k real MS MARCO v1.1 passages × 300 real tiered queries (production filter shape). The old approximation had ~8/300 documented divergences (underscore / non-ASCII adjacency); they're gone.

Speedup (real data, 200k MS MARCO passages × 5,000 tiered queries, 3,505 distinct must-words, 24-core box, best of 3)

version	time	speedup
master (per-word regex + literal prefilter + word cache)	125.5s	1x
branch B+D hybrid (73628c2)	7.3s	17x
this PR (tokenized, single-pass/field, threaded)	3.0s	41x

Match-set size differs from master by only ~0.03% of true cells (the semantics change), and the parity run shows the new bits are the Qdrant-correct ones.

Follow-ups (out of scope)

• Cross-filter (per-file) tokenization sharing in compute.py's concurrent CPU-fallback dispatch — same-field columns are still tokenized once per filter.
• Making batch bytes / worker cap params-tunable like io_workers.

---

Round 2 (a753fb6): fused query-major combine

Profiling showed 83% of the remaining filter time was memory traffic: each per-query text condition expanded to its own (n_queries, rows) boolean array (10 arrays + combines ≈ 20GB traffic per 200k-row file). evaluate() now groups queries by their combo of token sets across the filter's text conditions, computes each distinct combo's combined (rows,) result once from the shared per-token masks, and writes each query's finished row exactly once. Bit-identical — 50-seed A/B fuzz vs a condition-major reference, an independent 400-seed differential fuzz vs the previous commit, full-scale bit-equality on 200k×5000, and live-Qdrant parity re-verified 300/300 exact.

version	time (200k×5k, best of 3)	cumulative speedup
master	125.5s	1×
tokenized single-pass (56d554c)	2.95s	43×
+ fused combine (a753fb6)	0.87s	144×

"Move filter eval off the scoring path" turned out to already be implemented — evaluation runs inside the io_workers reader threads (verified end-to-end: wall-clock < read+filter+GPU components; with the old filter code the same pipeline scales 92s→24.5s going from io_workers 1→6), so no pipeline change was made.

Review fixes folded in: loud ValueError on mismatched query-column lengths (fused path would otherwise truncate silently), refcounted eviction for the per-combo phrase-mask cache, and reference-path docstrings on the now-production-dead per-condition builder. 316 tests pass.

🤖 Generated with Claude Code