perf(autocomplete): trie prefix index; fix cache poisoning and XSS escape - #6
Open
frankstupak wants to merge 1 commit into
Open
perf(autocomplete): trie prefix index; fix cache poisoning and XSS escape#6frankstupak wants to merge 1 commit into
frankstupak wants to merge 1 commit into
Conversation
…che poisoning, remove server-side debounce, escape XSS in highlights - NEW PrefixIndex trie: O(|prefix|+k) prefix lookups. 0.0058 ms/query vs 112.6 ms for a correctly-configured Fuse.js prefix scan at 100k items (~19,500x); vs 361.3 ms for the code path as shipped (~62,600x). bench/prefix-bench.ts, npm run bench:prefix. - FIX prefix strategy: '^query' was passed to Fuse without useExtendedSearch, so the caret was fuzzy-matched as a literal char. - FIX cache key omitted category/tags: cached results for one filter were served verbatim to different filters (cross-filter cache poisoning). Key now also normalizes query case and fuzzy-default semantics. - FIX server-side lodash.debounce on the request path: per lodash docs, debounced calls return the LAST invocation's result (undefined before the first), so concurrent users could receive each other's responses or undefined. Replaced with single-flight request coalescing. - FIX filters applied after limit: category/tag filters could return 0 results despite matches existing past the pre-filter cutoff. - FIX threshold 0 ignored (|| default) in sanitize + applyFilters. - FIX stored XSS: highlighted title/description now HTML-escaped before <mark> insertion; overlapping match ranges merged. - FIX exact strategy now does exact phrase matching (was identical to fuzzy). - FIX popular-query avgExecutionTime: true running mean, not (avg+x)/2 drift. - Background index-rebuild timer unref'd; @fastify/swagger dep added (tsc was failing); unused lodash/lodash.debounce deps removed. - +29 tests (65 total): trie unit suite + regression tests for every fix.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Third delivery from Lumen Industries. The autocomplete subproject had a broken prefix path, a poisoned cache, a response-leaking request pipeline, and an XSS hole. All fixed, all tested, headline number is five digits.
What was wrong
performSearchpassed^queryto Fuse.js, but the^prefix operator only exists underuseExtendedSearch: true(Fuse.js docs) — which was never set. Every 1–2 char query fuzzy-matched a literal caret.categoryandtags. A cached response for?q=x&category=bookswas served verbatim to?q=x&category=movies. The existing "consistent cache keys" test never caught it because it assertedtoBeDefined()— andnullis defined.lodash.debounceon the request path. Per the lodash contract, debounced calls return the result of the last invocation — andundefinedbefore any invocation runs. WithdebounceMs > 0, concurrent users could receiveundefinedor each other's search responses. Debounce belongs on the client keystroke, never on a shared server path.highlightedTitle/highlightedDescriptioninterpolated raw item data around<mark>tags. Any item titled<img src=x onerror=…>executed in the demo frontend.threshold: 0silently ignored (|| defaultin two places), "exact" strategy was identical to fuzzy, and the popular-query "average" was(avg+x)/2— an exponential drift where the first sample's weight decays to1/2^(N-1).What's better
PrefixIndex— character trie indexing full title, title words, and tags.O(|prefix| + k)lookups, exact-token matches ranked first, Unicode-safe, deterministic ordering. Wired into the prefix strategy.threshold: 0honored, HTML-escaped highlights with overlapping-range merge, true running mean, rebuild timerunref()d.@fastify/swaggeradded (tscwas failing on a missing dep), unusedlodash/lodash.debounceremoved.Numbers
npm run bench:prefix— 100,000 items, 200 query rounds, limit 10 (Node 22, Xeon):^q, no extended search)^q,useExtendedSearch)Trie build for 100k items: 614 ms one-time (200,030 tokens).
Tests
not.toBeNull()assertions, a concurrent-request response-ownership test, and an XSS escape test.tsc --noEmitclean (it wasn't before). ESLint: 0 errors, 0 warnings on all touched files.debounceMsis still accepted, documented as client-side-only.— Lumen Industries