Enrichment-accuracy eval + search-eval/red-team hardening + fixes (2.6.0) - #86
Merged
Conversation
…ent + fashion.evalAttributes Closes the loop named as samesake's make-or-break risk: search relevance was measured (evaluateSearch), enrichment correctness was not. A mis-extracted color or missed neckline only shows up in search as blurred noise; this measures the root cause — the enrichment twin of evaluateSearch. Framework (baked in, no stubs, generic): - packages/server/src/core/evaluate-enrich.ts — pure scoreEnrichment() (per-attribute TP/FP/FN -> precision/recall/F1, micro+macro, coverage, per-product diffs) + makeEvaluateEnrichService reading enriched rows; wired onto matcher.evaluateEnrichment and exported from @samesake/server. - packages/sdk/src/templates/fashion.ts — fashion.evalAttributes() default attribute spec, alongside fashion.fields/spaces/nlq. Harness + data (example, mirrors eval.ts + golden-queries): - examples/fashion-search/eval-enrichment.ts — runner (--fixture offline / live / --bootstrap), consuming fashion.evalAttributes() + matcher.evaluateEnrichment (nothing hand-rolled). - evals/golden-enrichment-fashion-lk.json — 50 real LK products labeled from titles (enum-validated). - evals/fixtures/enrichment-demo-store-predicted.json — captured pipeline output for offline scoring. Baseline on the 50-product demo_store LK corpus (live == offline): micro F1 97.8%, macro 98.2% (category 94, gender 100, colors 99, pattern 100, is_apparel 98). The run surfaced a real bug: a shoe-care brush classified as an apparel accessory and NOT gated (status=ready) — leaking into accessory search. Verify: 11 new unit tests green; root tsc --noEmit clean; sdk+server build (ESM+CJS+DTS) clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…category)
Reproducible search-relevance eval on the real fashionparity corpus (5,512 products),
scored framework-direct via matcher.evaluateSearch (LLM-judge = gemini-3.1-flash-lite),
with a pre-fix baseline and a post-fix run to compare.
Eval:
- examples/fashion-search/eval-search.ts — --phase=baseline|postfix; applies the config in-process
(production path; NLQ schema/instructions are functions that can't come from the DB), groups the
framework's per-query output into buckets, persists topIds, writes evals/runs/<ts>-search-<phase>.
- evals/search-queries-typo.json — 12 misspelled queries added to the 50 golden queries.
Data-driven finding: typo tolerance was NOT the gap (gemini-embedding-2 already handles typos —
typo bucket 1.92 > 1.61 overall). The baseline exposed the real gap: use-case queries returned
30% NO RESULTS ("office wear for women", "smart casual outfit for men", "resort wear for a holiday").
Root cause + fix (packages/sdk/src/templates/fashion.ts): NLQ mapped vague queries to
category:"other" — the non-apparel bucket — as a HARD filter, matching zero apparel. Removed "other"
from the NLQ category enum and instructed null-for-vague; the model can no longer emit the poison
value. Root fix, no engine coercion.
Pre -> Post (k=5): use-case grade 1.23 -> 1.77 (+0.54), use-case no-results 30% -> 0%,
overall no-results 5% -> 0%, overall grade 1.611 -> 1.679. Small style/local deltas (<=0.16) are
within single-LLM-judge noise (unchanged hit counts) — not over-claimed; follow-up: persist a
shared judge cache for deterministic pre/post.
Also: annotated the stale gemini-3-flash-preview reference in BENCHMARKS.md — the live pipeline is
gemini-embedding-2 + gemini-3.1-flash-lite (no flash 3).
Verify: root tsc --noEmit clean; enrichment scorer tests still green (11/11).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ion, contradiction, polysemy
A devil's-advocate suite built to FAIL the engine, run against live fashionparity.
- evals/adversarial-queries.json — 50 queries, 7 buckets, expect: relevant|empty|graceful + price bounds.
- examples/fashion-search/eval-adversarial.ts — per-query search in try/catch (crash = finding),
deterministic price-violation checks, framework LLM judge for relevance/false-positive; ranked
FAILURES artifact.
Result: 50 queries -> 16 findings, 0 crashes, 0 injection breaches.
- HELD: SQL/prompt/XSS/template injection all treated as literal (parameterized, no leak);
contradictions graceful; polysemy resolved (tank/clutch/pumps/mules/boxers); degenerate handled.
- FINDINGS: (1) no OOD rejection — 7/8 off-domain queries ("gaming laptop", "iphone", "office chair")
return nearest-neighbour junk because relevanceFloor is unset; (2) price-phrase parsing gaps
("for 0 rupees", negative) + price=0 data-quality rows; (3) weak Sinhala/Tamil multilingual retrieval;
(4) eval-harness: candidateSummary omits price so numeric grades are partly judge artifacts.
Each finding has a guarded fix path in the notes (none are blind constants). No code fix in this
commit — this is the measurement.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…); gate #2/#3 rejected by eval Live re-enrich harness (eval-enrichment.ts --reenrich): re-enriches the 50 gold products through the current pipeline (text-only; demo images are expired) and scores, so enrich-prompt changes are actually exercised. Shipped #4: fashionExtractSchema colour rule collapses compound single-shade names to one base ("navy blue"→[navy], not [navy,blue]). Clean win: colors F1 0.99→1.00, category unchanged 0.94, micro 0.978→0.981, no collateral. NOT shipped #2 (non-apparel gate) / #3 (kids-garment): every classify-prompt variant NET-REGRESSED and the re-enrich gate caught each — #3 broke activewear(→bottoms)+kidswear(→tops) (cat 0.94→0.90); #2 made watches classify non-apparel, narrower variant scrambled watch categories (cat 0.82). The classify stage is too sensitive to global-instruction edits for these ambiguous edge cases; the correct mechanism is the few-shot correction loop (review.ts), left as follow-up. Details in notes. Also fixed --tag=post parsing in the runner. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…sees price, deterministic cache) P3 (measurement honesty — done first so P2 verification isn't judge-noise): - #9 judge sees price: candidateSummary (judge.ts) + hitText (calibrate-search.ts) now include price, so the LLM judge can verify numeric constraints. golden price bucket 1.8->2.64; red-team's 3 numerical "WEAK" false-alarms (num-01/03/04) vanish — they were judge blind-spots. red-team 16->13. - #10 persistent judge cache: grades persisted via stage-cache keyed by (judge-version, query, doc) → a doc seen in both pre & post reuses its grade → deterministic pre/post. Proven: p2base->p2post left keyword/broad/style/typo buckets bit-identical. P2 (retrieval/NLQ robustness): - #5 NLQ price robustness: strip $/Rs, "5k"->5000, ignore <=0 / inverted bounds, "for N" not a price. Red-team: negative/zero-bound queries now ignore the nonsensical bound instead of surfacing price=0 junk. - #7 price hygiene gate: fashionIndexing quarantines price<=0 (reason invalid-price); unit-tested. - #6 styles-soft: REVERTED — neutral on golden style, small local regression, no measured benefit (same discipline as P1 #2/#3). SDK default is already soft; example left as-is. Also fixes a pre-existing stale NLQ-schema test (missing styles/price_budget_hint fields). Deferred: OOD rejection (P0 relevanceFloor calibration), multilingual, lexical/BM25 (#11 -> research #17). Verify: tsc --noEmit clean; 26 tests pass (fashion-template incl new #7 gate test + evaluate-enrich). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…11 decision, roadmap (task #17) Firecrawl-sourced research (2024–2026) synthesized to samesake's actual position (PG-native, ~5.5k docs today, catalog-scale ceiling, weak ts_rank leg, Neon/Fly deploy). Findings: - Scale honesty: a single tuned Postgres + pgvector HNSW comfortably serves ~1–2M products/project; PG's hard wall (HNSW index > RAM) is ~10M×1536-dim (~80–120 GB) — 10–1000× beyond our "1". We do NOT need pgvectorscale/VectorChord/Citus/sharding at catalog scale. - #11 (BM25 lexical): ts_rank_cd is genuinely weak (no IDF/length-norm/TF-saturation/efficient top-N). Real BM25 = ParadeDB pg_search (mature, Tantivy) or VectorChord vchord_bm25 (early). BOTH need shared_preload_libraries → not on stock RDS/Aurora/Cloud SQL, and pg_search was DROPPED from new Neon projects (Mar 2026), both AGPL-3.0. => a deployment decision, not a code edit. Recommend pg_search on controlled PG (Fly/EC2/Docker) in P-next; else setweight + pg_trgm mitigations. - Adopt now (any managed PG, free): halfvec (2x smaller, <1% recall loss), iterative scans (pgvector 0.8, fixes filtered-vector under-return), ef_search tuning, keyset pagination, setweight FTS. - Escape hatch: external engine only past ~few-million SKUs/tenant with sub-second faceted UX. Deliverable per task #17: scale-ceiling estimate + phased roadmap (P-now / P-next / P-later) + full cited sources. Folds in the deferred #11. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… recommendation
- 5.5k was test data; launch ~100k+ (still trivial for PG: ~0.6GB vectors/~1.2GB index, ~100x below wall).
- Remove ParadeDB lean (it was content-marketing footprint, not evidence). New stance: measure whether
the lexical leg is even the bottleneck (semantic leg already absorbs typos); ship setweight+pg_trgm
mitigations first; only if lexical is confirmed limiting, bake-off {ts_rank, pg_search, vchord_bm25}
on our own corpus. VectorChord suite may be a more coherent single bet for a vector-first framework.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…end-state blueprint The '1' documented as a layered architecture (L0 infra → L1 enrichment moat → L2 retrieval → L3 toolkit → L4 eval spine), with the guiding policy (enrichment is the product, own-your-data PG-native, eval-gated, BYO models, definite-direction/lean-increments), end-to-end index+query flows, an honest built-vs-to-build map, the scale path (from #17), and the capability-discipline 'do NOT build' list. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ment, eval (29 sources) Firecrawl digest across 3 tracks (search architecture, LLM product-data enrichment, hybrid+rerank+ LLM-judge eval) from Instacart/Faire/Etsy/DoorDash/Zalando/Lyst/Mercari/Constructor/Shopify/Wayfair/ Amazon + Weaviate/Vespa/Cohere/Elastic + IR papers. Validates/missing/new, mapped to our architecture. Key: enrichment-first is the strongest-validated bet (DoorDash +31% data vs +6% encoder); Instacart runs prod search on Postgres and is consolidating into pgvector; Mannheim proves LLM self-correction fails while few-shot-from-similar wins (validates this session's P1 few-shot-loop lesson). Top gaps: internal-catalog RAG enrichment (+34% recall), query-entropy adaptive recall, whole-page rerank, bucketed/bias-aware judge calibration, multimodal retrieval. Risk: LLM judge flatters LLM-written data. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…re diagrams) Read Zepto's search cluster directly — flagship 'Building Search for a 10-Minute World' (3 arch diagrams viewed as images: query-resolution chain, full architecture, ranking evolution), semantic-search deep-dive, and product-enrichment(read-assembly) post. Key: enrichment feeds the embedding (name+type+brand+LLM attributes; 'retrieval quality sets the ceiling') — validates our enrichment-first bet. Steal-this: head/tail query-understanding tiers (pre-compute NLQ for head, live for tail), synthetic tail data + query generation, catalog-grounded LLM query correction (RAG) for multilingual (our deferred gap), explore stream, ops rules layer. Do-not-copy (quick-commerce-specific, premature for fashion): hyperlocal city-pvid per-hub index, MoE cohort ranking, no-caching. Terminology nuance noted: Zepto 'product enrichment' = read-assembly, not attribute extraction. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…esign (task #16) Implementation-ready RFC for the L3 frontend SDK. Design only (no package scaffolded — respects "no impl yet" + YAGNI). - 3-layer, transport-agnostic architecture: L1 typed core client (HTTP or in-process matcher, zero deps) → L2 framework-agnostic headless search-state controller (debounce/abort/dedupe/URL codec) → L3 React headless hooks + <SearchProvider> + optional unstyled components. - Prior-art steal/avoid: take InstantSearch's connectors/headless + batched-request + URL-sync; avoid its widget/CSS bloat + proprietary DSL. Native client is core; InstantSearch adapter deferred. - Surfaces samesake differentiators no generic SDK has: intent/similar modes, NLQ hard-filter CHIPS + relaxation transparency (the headline DX), image/"more like this", spaces weight sliders, enriched facets, /explain overlay. - Grounded 1:1 against the verified packages/server API (SearchOpts/SearchResult/facets/search-explain) — no fictional fields. SSR/RSC via in-process client + hydrate. Two zero-dep-core packages. Phased build plan P1–P5, opinionated defaults, non-goals. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…uides/reference - Sidebar: surface the orphaned "Measure enrichment accuracy" guide (eval-enrichment). - what-is-samesake: new "Measuring quality" section — the three eval loops (enrichment accuracy via evaluateEnrichment, relevance via evaluateSearch with deterministic judge, adversarial red-team). - enrich-pipeline: "Did the enrichment actually work?" — measure with evaluateEnrichment + evalAttributes. - tuning-search: close the enrich loop (measure per-attribute F1); rewrite §8 as three loops + note the judge now sees price and persists grades per (query,doc) for deterministic pre/post + bucketed reporting + OOD relevanceFloor. - eval-enrichment: document the shipped --reenrich pre/post mode + caution on global-prompt regressions (prefer the few-shot correction loop). - relevance-judge: candidateSummary now includes price (so the judge can verify numeric constraints). Verify: astro build clean (30 pages, no broken links). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Published to npm. 2.6.0 ships: enrichment-accuracy eval (matcher.evaluateEnrichment + scoreEnrichment + fashion.evalAttributes), NLQ category-'other' fix, colour over-emission fix, NLQ price robustness, price<=0 index gate, and eval-harness honesty (judge sees price, deterministic persistent grade cache). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…/colour/price fixes Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ships the enrichment-accuracy eval loop (the make-or-break measurement samesake was missing), hardens the search eval + adds an adversarial red-team, fixes several enrichment/NLQ defects the evals surfaced, and lands supporting research + docs. Released to npm as 2.6.0 (core/server/cli/mcp).
Framework changes (published in 2.6.0)
matcher.evaluateEnrichment+ purescoreEnrichment(per-attribute P/R/F1, micro+macro, coverage, diffs);fashion.evalAttributes()default spec. The root-cause loop beneath relevance. (packages/server/src/core/evaluate-enrich.ts,+ 11 unit tests)evaluateSearchpersists grades per (judge-version, query, doc) → deterministic pre/post.category:"other"no-results fix (use-case no-results 30%→0%), NLQ price robustness,price≤0index gate (unit-tested).Eval suites (new)
examples/fashion-search/eval-search.ts(golden relevance, LLM-judge) +search-queries-typo.jsonexamples/fashion-search/eval-adversarial.ts(OOD / numerical / injection / contradiction / polysemy red-team) — 50 queries → surfaced no-OOD-rejection + multilingual gaps; 0 crashes, 0 injection breacheseval-enrichment.ts --reenrichlive pre/post harnessResearch + design (docs/)
docs/architecture/full-scale-fashion-search.md— build-for-one end-state blueprintdocs/research/— Postgres high-scale search (BM25/Replace shape-based config discovery with SDK type guards #11 decision), industry search practices (29 sources), Zepto search notesrfcs/rfc-samesake-client-sdk.md—@samesake/client+@samesake/reactfrontend SDK designDocs site
Updated guides/reference for the eval loops; deployed to Cloudflare Pages (samesake-docs).
Verification
tsc --noEmitclean · server tests green (incl. 11 new enrichment scorer tests + price-gate test) · sdk+server built (ESM+CJS+DTS) · all 4 packages published + registry-confirmed at 2.6.0 · docsastro buildclean (30 pages).Note: 2.6.0 was already published to npm from this branch (local release flow); this PR brings
mainin line with what shipped.🤖 Generated with Claude Code
https://claude.ai/code/session_01M3xKQXdta7zrijMd4ZXcky