Enrichment-accuracy eval + search-eval/red-team hardening + fixes (2.6.0)

BENCHMARKS.md

@@ -72,10 +72,38 @@ Spaces capability ships but stays **off by default**. Flat-weighted segmented ve
 
 Verdict and engineering findings: [`docs/spaces-gate.md`](./docs/spaces-gate.md).
 
+## Enrichment accuracy — the root-cause gate (separate from search relevance)
+
+Search relevance is downstream of enrichment: a mis-extracted color or missed neckline corrupts
+ranking, but a relevance metric only sees blurred noise. `matcher.evaluateEnrichment(...)` scores the
+pipeline's extracted attributes against a **human-labeled gold set** (`evals/golden-enrichment-fashion-lk.json`,
+50 real LK products from the `demo_store` corpus, labeled independently from titles) with per-attribute
+precision/recall/F1. Reproduce: `cd examples/fashion-search && bun eval-enrichment.ts --fixture`
+(offline, no DB/LLM) or `bun --env-file=../../.env eval-enrichment.ts` (live) — both give identical numbers.
+
+| attribute | precision | recall | F1 |
+|---|---|---|---|
+| category | 94.0% | 94.0% | 94.0% |
+| gender | 100% | 100% | 100% |
+| colors | 98.1% | 100% | 99.0% |
+| pattern | 100% | 100% | 100% |
+| is_apparel_product | 98.0% | 98.0% | 98.0% |
+| **overall (micro)** | **97.6%** | **98.1%** | **97.8%** |
+| **macro F1** | | | **98.2%** |
+
+Scoring: each value is a set token; TP = pred∩gold, FP = hallucinated, FN = missed ("NULL is worse
+than wrong"). v1 gold covers the controlled, gate/filter-critical attributes only (free-text
+`product_type` and image-derived `occasions/styles/fit/material` are out of scope until labeled from
+images — see [implementation notes](./search-enrichment-accuracy-implementation-notes.md)).
+
+The disagreement list is the payoff: this run flagged a shoe-care brush (`6842`) mis-classified as an
+apparel accessory and **not** gated (leaking into accessory search) — a real bug the search eval
+could not have localized.
+
 ## Methodology
 
 - **Golden set**: 50 queries covering keyword, attribute, use-case, price, negation, style, local, and broad intent types.
-- **Judge**: ESCI LLM grading (0–3 relevance scale), `gemini-3-flash-preview`, results cached per (query, result-set hash).
+- **Judge**: ESCI LLM grading (0–3 relevance scale), results cached per (query, result-set hash). The parity/post-wave tables above were produced by a historical spike run that used `gemini-3-flash-preview`; the **framework's current judge + generate model is `gemini-3.1-flash-lite`** (see `examples/fashion-search/gemini.ts`) — there is no "flash 3" in the live pipeline. New eval runs (e.g. `eval-search.ts`) stamp the model used into their artifact.
 - **Corpus**: LK fashion e-commerce — Shopify/Woo connectors, enrichment pipeline (classify + extract), pgvector 1536d embeddings.
 - **Metrics**: mean grade@10 (primary), P@5 (precision at relevance ≥2), nDCG@10, price-violation rate, zero-result rate, median latency.
 

CHANGELOG.md

@@ -2,6 +2,20 @@
 
 All notable changes to samesake. Format roughly follows [Keep a Changelog](https://keepachangelog.com/).
 
+## [2.6.0]
+
+### Added
+
+- **Enrichment-accuracy eval** (`@samesake/server`) — `matcher.evaluateEnrichment(project, collection, { gold, attributes })` + the pure `scoreEnrichment(gold, predicted, attributes)` score the enrich pipeline's extracted attributes against a human-labeled gold set with **per-attribute precision / recall / F1** (micro + macro, coverage, per-product diffs). The root-cause loop beneath search relevance — the enrichment twin of `evaluateSearch`. `@samesake/core` ships `fashion.evalAttributes()` (+ `EnrichEvalAttr`) as the default fashion attribute spec.
+- **Eval-harness honesty** (`@samesake/server`) — the relevance judge now **sees each candidate's price** (`candidateSummary`/`hitText`) so it can verify numeric constraints ("under N"); and `evaluateSearch` **persists judge grades** per `(judge-version, query, doc)` via the stage cache, so pre/post comparisons reflect a real retrieval change, not judge re-roll noise.
+- **Price-hygiene index gate** (`@samesake/core`) — `fashionIndexing` quarantines rows with `price ≤ 0` (`reason: "invalid-price"`).
+
+### Fixed
+
+- **NLQ `category:"other"` no-results** (`@samesake/core`) — vague use-case queries ("office wear for women", "resort wear") were mapped to the non-apparel `other` category as a **hard filter** → zero results. `"other"` is removed from the NLQ category enum; vague queries now leave `category` null and let `semantic_query` carry intent. (use-case no-results 30% → 0%.)
+- **Colour over-emission** (`@samesake/core`) — the extract rule now collapses compound single-shade names to one base ("navy blue" → `["navy"]`, not `["navy","blue"]`; "off white" → `["white"]`).
+- **NLQ price robustness** (`@samesake/core`) — strip `$`/`Rs`/`rupees`, `"5k"` → 5000, and ignore non-positive / inverted (`min > max`) bounds instead of surfacing junk.
+
 ## [2.5.0]
 
 ### Added

README.md

@@ -123,6 +123,8 @@ Six fashion/e-commerce primitives are baked into the core, on the principle of g
 
 Self-tuning: `matcher.evaluateSearch(...)` scores graded relevance@k / nDCG@k (caller labels or the configured LLM as judge), and `matcher.calibrateSearch(...)` sweeps a mode/weight grid and returns the recommended default — so "no config" can mean samesake calibrates itself.
 
+Enrichment accuracy (the root cause under relevance): `matcher.evaluateEnrichment(...)` scores the pipeline's extracted attributes against a human-labeled gold set — per-attribute precision/recall/F1 — so a mis-extracted color or a missed neckline is caught at the source, not blamed on ranking. Search relevance is only as good as the attributes enrichment pulls; measure both. See the [enrichment-accuracy guide](./apps/docs/src/content/docs/guides/eval-enrichment.mdx) and `examples/fashion-search/eval-enrichment.ts`.
+
 ### Fashion enrichment template (best defaults)
 
 Attribute-aware search needs structured attributes (a "Crimson" title should be retrievable under "red dress"). `@samesake/core` ships a fashion enrichment template so you get that without hand-writing a taxonomy + schemas:
@@ -201,7 +203,7 @@ Runnable demo (stub embed, weight flip): [`bun examples/hello-spaces/run.ts`](./
 | NLQ → hard filters + semantic residual | Structured parse gates (brand, size, internal code) |
 | Multi-stage enrichment pipeline + stage cache | Confirm / decline → alias active learning |
 | Connectors (Shopify, Woo, JSONL) + document push | `/explain` per-channel score breakdown |
-| Eval harness (golden queries + ESCI judge) | F1 threshold calibration per scope |
+| Eval harness: search relevance (golden queries + ESCI judge) **and** enrichment accuracy (per-attribute P/R/F1) | F1 threshold calibration per scope |
 | Query-time channel weights | `/match-batch` for bulk workloads |
 
 Search and match share embeddings, Postgres caches, and per-project runtime DDL.

apps/docs/astro.config.mjs

@@ -66,6 +66,7 @@ export default defineConfig({
             { label: "Fashion app with Porulle + Next.js", slug: "guides/porulle-fashion-app" },
             { label: "Pipeline lifecycle", slug: "guides/pipeline-lifecycle" },
             { label: "Tuning search relevance", slug: "guides/tuning-search" },
+            { label: "Measure enrichment accuracy", slug: "guides/eval-enrichment" },
             { label: "Eval from search snapshots", slug: "guides/eval-from-snapshots" },
             { label: "Eval gate — tune floor and exponents", slug: "guides/eval-gate" },
           ],

apps/docs/src/content/docs/guides/enrich-pipeline.mdx

@@ -79,8 +79,18 @@ await matcher.apply("market", { entities: [], collections: [products] });
 Pulling from Shopify, WooCommerce, or Medusa? The [integration guides](/integrations/shopify/) give you the connector that turns a store feed into the `rows` you pass to `pushDocuments`. The durable wrapper is the same no matter where the rows come from.
 </Aside>
 
+## Did the enrichment actually work?
+
+Enrichment quality is the make-or-break for search — so measure it, don't assume it. After a run,
+`matcher.evaluateEnrichment(project, collection, { gold, attributes: fashion.evalAttributes() })`
+scores the pipeline's extracted attributes against a human-labeled gold set with **per-attribute
+precision / recall / F1** (and flags where the model hallucinated or missed a value). Gate your
+enrich-prompt, taxonomy, or confidence-floor changes on it. See
+[Measure enrichment accuracy](/guides/eval-enrichment/).
+
 ## Where to go next
 
 - **Status machine for every row** — [Pipeline lifecycle](/guides/pipeline-lifecycle/)
+- **Measure enrichment accuracy** — [per-attribute P/R/F1 vs a gold set](/guides/eval-enrichment/)
 - **Marketplace loop in plain language** — [Search for a fashion marketplace](/guides/marketplace-search/)
 - **Single-store version** — [From a store idea to search](/guides/idea-to-search/)

apps/docs/src/content/docs/guides/eval-enrichment.mdx

@@ -0,0 +1,100 @@
+---
+title: Measure enrichment accuracy
+description: Score the enrich pipeline's extracted attributes against a human-labeled gold set — per-attribute precision/recall/F1 — so you gate enrich/taxonomy changes on measured accuracy, not vibes.
+---
+
+import { Steps, Aside } from '@astrojs/starlight/components';
+
+Search relevance is only as good as the attributes the enrich pipeline extracts: a mis-labeled color
+or a missed neckline silently corrupts ranking. `matcher.evaluateSearch(...)` measures the
+downstream symptom; `matcher.evaluateEnrichment(...)` measures the **root cause** — did classify +
+extract pull the *right* structured attributes?
+
+<Aside type="tip">This is the enrichment twin of the [search eval](/guides/eval-from-snapshots/). Run both: enrichment accuracy first (is the data right?), then search relevance (is the ranking right?).</Aside>
+
+## How it scores
+
+Every attribute value is treated as a set token. For each product × attribute:
+
+- `TP` = values in both gold and prediction, `FP` = predicted but not in gold (hallucination),
+  `FN` = in gold but not predicted (a miss — *"NULL is worse than wrong"*).
+- Aggregated per attribute → precision / recall / F1, plus micro (pooled) and macro (mean per attribute).
+
+A gold label that is **absent** means "unlabeled" (skipped). A label of `[]` or `"unknown"` means
+"explicitly no value" and *is* scored — so a hallucinated value counts against you.
+
+## Run it
+
+<Steps>
+
+1. **Label a gold set** — real products, attributes labeled independently of the pipeline. The
+   fashion example ships one: `evals/golden-enrichment-fashion-lk.json` (50 LK products, labeled from
+   titles). Bootstrap a blank template for a new corpus:
+
+   ```bash
+   cd examples/fashion-search
+   bun eval-enrichment.ts --bootstrap        # → evals/golden-enrichment.template.json
+   ```
+
+2. **Score offline** (no DB, no LLM — CI-safe) against captured pipeline output:
+
+   ```bash
+   bun eval-enrichment.ts --fixture
+   ```
+
+3. **Score live** against your seeded corpus:
+
+   ```bash
+   bun --env-file=../../.env eval-enrichment.ts
+   ```
+
+4. **Testing an enrich-prompt / taxonomy change?** The seeded corpus is baked, so re-enrich the gold
+   products live through the *current* pipeline, then score — this is what actually exercises a prompt
+   change. Run once before and once after your change to get a clean pre/post:
+
+   ```bash
+   bun --env-file=../../.env eval-enrichment.ts --reenrich --tag=pre    # before the change
+   # …edit the enrich prompt/schema, rebuild…
+   bun --env-file=../../.env eval-enrichment.ts --reenrich --tag=post   # after
+   ```
+
+</Steps>
+
+<Aside type="caution">
+Global enrich-prompt edits are risky — a change that fixes one product often breaks others (e.g.
+tightening the non-apparel rule can misclassify watches). The re-enrich pre/post gate catches this. For
+narrow edge cases, prefer the few-shot **correction loop** (`review.ts` → corrections fed back as
+exemplars) over rewriting the global prompt.
+</Aside>
+
+Both write `evals/runs/<ts>-enrichment-*.{json,md}` and print a per-attribute scorecard:
+
+```
+| attribute          | precision | recall |  F1  |
+| category           |   94.0%   | 94.0%  | 94.0%|
+| gender             |  100.0%   |100.0%  |100.0%|
+| colors             |   98.1%   |100.0%  | 99.0%|
+| is_apparel_product |   98.0%   | 98.0%  | 98.0%|
+| overall (micro)    |   97.6%   | 98.1%  | 97.8%|
+```
+
+The disagreement list is the payoff — it names each product where the pipeline and gold differ
+(missed vs hallucinated values), so a bad classification (e.g. a non-apparel item that wasn't gated)
+is visible and regressable.
+
+## In code
+
+```ts
+const result = await matcher.evaluateEnrichment("shop", "products", {
+  gold: [{ id: "1", labels: { category: "dresses", colors: ["red"], is_apparel_product: true } }],
+  attributes: [
+    { name: "category", kind: "single" },
+    { name: "colors", kind: "multi" },
+    { name: "is_apparel_product", kind: "single", empty: [] },
+  ],
+});
+// result.attributes[].{precision,recall,f1}, result.overall.microF1, result.diffs
+```
+
+Gate your enrich-prompt, taxonomy, or `FASHION_CONFIDENCE_FLOOR` changes on `result.overall.microF1`
+(or a per-attribute floor) the same way ranking changes are gated on retrieval nDCG.

apps/docs/src/content/docs/guides/tuning-search.mdx

@@ -67,6 +67,11 @@ vs `responseJsonSchema` — stays in your `generate` function.
 Run with `matcher.enrich(project, collection)`. Colours went from mostly-empty to accurate
 ("RED PUFF SLEEVE MAXI DRESS" → `solid red`). See [the full pipeline](/guides/porulle-fashion-app/#enrich-attributes-from-the-image-samesake-enrich-pipeline).
 
+**Then measure the enrichment itself** — `matcher.evaluateEnrichment(...)` scores per-attribute
+precision / recall / F1 against a gold set, so you know whether the fix actually landed (and catch
+regressions where a prompt tweak fixes one product but breaks others). It's the root-cause loop
+beneath search relevance — see [Measure enrichment accuracy](/guides/eval-enrichment/).
+
 ## 4. Compose what you embed — the `indexing` DSL
 
 An embedding only knows what's in the text it was built from. Declare **surface builders** on the collection — they run at enrich time, persist to the row, and the indexer reads them (no separate compose step, no string template on the embeddings block):
@@ -146,13 +151,26 @@ See [Reranking](/reference/reranking/) and [Relevance judge](/reference/relevanc
 **Empirical tuning:** `FASHION_CONFIDENCE_FLOOR` and `relevanceExponent` are placeholders until
 you run the offline eval gate — see [Eval gate — tune floor and exponents](/guides/eval-gate/).
 
-## 8. Measure, and respect corpus size
-
-Tune against a fixed query set, not vibes — see [Eval from search snapshots](/guides/eval-from-snapshots/)
-for relevance@k + constraint compliance, and [Eval gate](/guides/eval-gate/) for the golden-set
-harness (`runEval`) that gates ranking and floor changes. And be honest about scale: at ~30 products colour is a weak
-discriminator no matter what, because the embedding is dominated by category. Real relevance wins
-need both clean attributes **and** a catalog big enough to disambiguate.
+## 8. Measure with three loops, and respect corpus size
+
+Tune against fixed sets, not vibes. samesake has three complementary eval loops:
+
+- **Enrichment accuracy** (root cause) — `matcher.evaluateEnrichment(...)`, per-attribute P/R/F1 vs a
+  gold set. Fix this first; search can't beat the data it ranks. See [Measure enrichment accuracy](/guides/eval-enrichment/).
+- **Search relevance** — `matcher.evaluateSearch(...)` / `runEval` over a fixed query set:
+  relevance@k, nDCG, constraint compliance. See [Eval from search snapshots](/guides/eval-from-snapshots/)
+  and [Eval gate](/guides/eval-gate/). The LLM judge **sees each candidate's price** (so it can verify
+  "under N" constraints) and **persists grades per (query, doc)** — so a pre/post comparison reflects a
+  real *retrieval* change, not judge re-roll noise. Report **by query bucket** (keyword / attribute /
+  use-case / price / negation / style / local); an overall nDCG win can hide a tail regression.
+- **Adversarial red-team** — deliberately-breaking queries (out-of-distribution, numerical/malformed,
+  injection, contradiction, degenerate, polysemy) to confirm the engine fails *gracefully*: no crashes,
+  no leaked secrets, and no confident junk for off-domain queries (tune `search.relevanceFloor` so
+  "gaming laptop" returns nothing, not five random dresses).
+
+Be honest about scale: at ~30 products colour is a weak discriminator no matter what, because the
+embedding is dominated by category. Real relevance wins need clean attributes **and** a catalog big
+enough to disambiguate.
 
 ## Running enrichment in the background
 

apps/docs/src/content/docs/reference/relevance-judge.mdx

@@ -65,7 +65,7 @@ Candidate products:
 Return a grade 0|1|2 per candidate with facet sub-grades and a short reason. Keep the original candidate order.
 ```
 
-Candidate text comes from `candidateSummary` when built from product data (title, brand, category, colors, occasions, etc.) or from the `text` field passed in (rerank path uses `rerankCandidateText`).
+Candidate text comes from `candidateSummary` when built from product data (title, brand, **price**, category, colors, occasions, styles, material, pattern, fit, description) or from the `text` field passed in (rerank path uses `rerankCandidateText`). Price is included so the judge can verify numeric constraints like "under 5000" — without it, numeric queries get under-graded (a judge that can't see the price can't confirm the bound).
 
 ## Structured output schema
 

apps/docs/src/content/docs/start/what-is-samesake.mdx

@@ -43,6 +43,14 @@ Three retrieval signals combine through **reciprocal-rank fusion (RRF)**:
 
 Hard filters (`price ≤ X`, `available = true`, `colors ∋ red`) compile to SQL predicates that gate the result set before ranking.
 
+## Measuring quality
+
+Search quality is only as good as the attributes enrichment extracts, so samesake ships three eval loops — the reason relevance is measured, not asserted:
+
+- **Enrichment accuracy** — `matcher.evaluateEnrichment(...)` scores the pipeline's extracted attributes against a human-labeled gold set (per-attribute precision / recall / F1). This is the *root cause*: a mis-extracted colour or a missed neckline is caught here, not blamed on ranking. See [Measure enrichment accuracy](/guides/eval-enrichment/).
+- **Search relevance** — `matcher.evaluateSearch(...)` grades results with an LLM-as-judge (or your labels); `matcher.calibrateSearch(...)` sweeps configs. Judge grades are cached per (query, doc) so pre/post comparisons are deterministic. See [Eval from search snapshots](/guides/eval-from-snapshots/).
+- **Adversarial red-team** — out-of-distribution, numerical, injection, contradiction, and polysemy queries to prove the engine fails *gracefully* (no crashes, no junk for off-domain queries).
+
 ## Three ways to call it
 
 `createMatcher(config)` returns one object you can call three ways: