feat!: P0 iron-out — generic deletes, de-fashioned core, honest ESCI judge, one env contract

.changeset/tier-zero-defaults.md

@@ -0,0 +1,76 @@
+---
+"@samesake/core": major
+"@samesake/server": major
+"@samesake/cli": major
+"@samesake/mcp": major
+---
+
+Tier-0 retrieval defaults baked in, and the zero-config indexing path is fixed.
+
+**Breaking (recreate + reindex collections; requires pgvector ≥ 0.7, ≥ 0.8 recommended):**
+
+- Collection `embedding` and `space_vec` columns are now `halfvec` (fp16): ~2× smaller storage
+  and index, ~2× faster HNSW build, <1% recall loss, and embedding dims up to 4000 (was 2000).
+  Existing tables keep `vector` columns and will fail search after upgrading — re-apply the
+  project on a fresh schema (or drop/re-add the vector columns) and reindex. Entity-resolution
+  tables are unchanged.
+- The `fts` generated column is now weighted: `setweight(fts_src_a, 'A') || setweight(fts_src, 'B')`.
+  New column `fts_src_a` carries title-class text. Declare it via `f.text({ searchable: true,
+  ftsWeight: "A" })` or an indexing fts surface with `weight: "A"`. `CollectionTextFieldDef.weight`
+  (dead) is removed.
+- `DEFAULT_PRODUCT_PARSE_INSTRUCTIONS` (deprecated since 0.7.x) is removed; use
+  `DEFAULT_PRODUCT_PARSE_BODY`.
+
+**Breaking (de-fashioned core — rename, no behavior change unless noted):**
+
+- `matcher.fashionSearch` / `POST …/fashion-search` → `matcher.shopSearch` / `POST …/shop-search`;
+  `matcher.syncFashionCatalogEvent` / `POST …/fashion-sync` → `matcher.syncCatalogEvent` /
+  `POST …/catalog-sync`. Types: `FashionSearchRequest/Response/Explanation/ImageInput` →
+  `ShopSearch*`, `FashionPersonalizationContext` → `ShopperContext`, `FashionCatalogSyncEvent` →
+  `CatalogSyncEvent`; `FashionRankingPolicy` is deleted (use `RankingPolicy`).
+- **Behavior change:** `shopSearch`'s `recoverNoResults` relaxes nothing unless the collection
+  declares `search.relaxableFilters` (new on `CollectionSearchDef`). The fashion template supplies
+  its list via `fashion.searchDefaults()` / `fashionSearchDefaults()` — spread it into your
+  `search` def to keep the old fashion relax behavior.
+- `fashionRerank` → `llmRerank` (score mapping is now ESCI `grade/3`).
+- Catalog-sync deletes route through `removeDocuments`; `DELETE …/documents` (body `{ ids }`) and
+  the CLI `samesake remove --ids=…` expose document deletion on every surface.
+
+**Breaking (judge honesty — evals re-grade from scratch):**
+
+- The LLM relevance judge is now 4-class ESCI (Exact=3 / Substitute=2 / Complement=1 /
+  Irrelevant=0; Substitute is a soft positive — default `relevanceFloor` is 2). `JudgedHit.grade`
+  is `0|1|2|3` with an `esci` label; facet sub-grades (`FacetGrades`) are removed.
+  `FASHION_JUDGE_SYSTEM` → `ESCI_JUDGE_SYSTEM`.
+- Judge versions are content-pinned: `makeLlmJudge` versions resolve to `<tag>@<sha256(rubric)[:8]>`,
+  so prompt edits auto-invalidate cached grades (file cache and the persisted search-judge cache).
+- **Same-family enrich+judge is rejected.** `runEval` and `evaluateSearch`/`calibrateSearch` throw
+  when a collection has an enrich pipeline and the judge model is missing or from the same model
+  family (self-preference bias). Declare a cross-family judge: in-process via
+  `evaluateSearch(…, { judge: { model, generate? } })` / `makeLlmJudge(gen, { model })`, over HTTP
+  via the new `judgeModel` body field on `…/search/evaluate` and `…/search/calibrate`.
+- Golden-query `constraints` now use the search filter vocabulary
+  (`{ "price": { "$lte": 5000 }, "colors": { "$exclude": ["black"] } }`) checked against whatever
+  fields the collection schema declares — the price/color/gender/category hardcoding is gone.
+
+**Breaking (one env contract):**
+
+- Canonical env vars everywhere: `SAMESAKE_DATABASE_URL` and `SAMESAKE_API_KEY`. The bare
+  `DATABASE_URL` / `API_KEY` fallbacks and the `apps/matcher` mapping shim are deleted — no
+  aliases. Provider keys keep their provider-canonical names (`GEMINI_API_KEY`,
+  `OPENAI_API_KEY`); `GOOGLE_GENERATIVE_AI_API_KEY` is no longer read.
+
+**Fixed:**
+
+- Collections without an enrich pipeline indexed nothing since the S1c indexing migration
+  (every doc skipped as "empty embedding document") — the README/quickstart `collection → push →
+  index → search` path was broken. `indexing` is optional again: without it, the engine composes
+  surfaces at index time from each embedding's restored `source` template and `searchable` fields;
+  with `indexing` but no enrich pipeline, the declared surfaces are built inline at index time.
+
+**Added:**
+
+- pgvector 0.8 iterative index scans (`hnsw.iterative_scan = relaxed_order`) are enabled
+  automatically on vector legs, fixing filtered-ANN under-return ("hard filters stay hard").
+- `efSearch` search option (HTTP + in-process, 10–1000): per-query HNSW recall/latency dial.
+- Apply now fails fast with a clear error when pgvector < 0.7.

.env.example

@@ -3,6 +3,6 @@ SAMESAKE_PORT=3030
 SAMESAKE_API_KEY=replace-me-with-a-real-key
 
 # At least one model API key must be set; pick whatever your config uses.
-GOOGLE_GENERATIVE_AI_API_KEY=your-gemini-api-key-here
+GEMINI_API_KEY=your-gemini-api-key-here
 # VOYAGE_API_KEY=
 # OPENAI_API_KEY=

.gitignore

@@ -22,6 +22,14 @@ examples/real-world-sample/_raw.json
 # Skill telemetry — sentinel files, scratchpads, delegate transcripts.
 .handoff/
 
+# Local agent tooling state (never source).
+.agents/
+.codex/
+.metadata_cache/
+.wrangler/
+
 # Generated eval/benchmark outputs and Porulle local media (runtime, not source).
 .samesake/
 apps/playground/.data/
+.plandesk/token
+.plandesk/server.json

README.md

@@ -48,8 +48,8 @@ const products = collection("products", {
 });
 
 const matcher = createMatcher({
-  databaseUrl: process.env.DATABASE_URL!,
-  apiKey: process.env.API_KEY!,
+  databaseUrl: process.env.SAMESAKE_DATABASE_URL!,
+  apiKey: process.env.SAMESAKE_API_KEY!,
   embed: async ({ text, dim }) => /* your embed fn */,
 });
 
@@ -221,7 +221,7 @@ Search and match share embeddings, Postgres caches, and per-project runtime DDL.
 
 ```bash
 bun install
-cp .env.example .env   # DATABASE_URL + API keys
+cp .env.example .env   # SAMESAKE_DATABASE_URL + API keys
 
 # Search (no LLM)
 bun examples/hello-search/run.ts
@@ -290,7 +290,7 @@ Cross-script matching, product parse gates, and the 19-assertion smoke test live
 |-------|--------|
 | Runtime | [Bun](https://bun.sh) 1.3+ |
 | HTTP | [Hono](https://hono.dev/) — universal `fetch` handler |
-| Database | Postgres 15+ with [pgvector](https://github.com/pgvector/pgvector) + `pg_trgm` + `unaccent` + `fuzzystrmatch` |
+| Database | Postgres 15+ with [pgvector](https://github.com/pgvector/pgvector) ≥ 0.7 (0.8 recommended — enables iterative index scans) + `pg_trgm` + `unaccent` + `fuzzystrmatch` |
 | Driver | [postgres-js](https://github.com/porsager/postgres) via Drizzle (raw SQL; schema generated per project at runtime) |
 | Validation | [Zod](https://zod.dev) |
 | AI | BYO — consumer supplies `embed` and optional `generate` / `parse` |
@@ -330,7 +330,7 @@ internal job runner. To run them durably, the caller wraps the calls in a platfo
 
 ## Status & naming
 
-NPM packages: **`@samesake/core`** (SDK), **`@samesake/server`**, **`@samesake/cli`** at **1.0.0**. The current public name is **Samesake**. The HTTP app still lives at `apps/matcher/`.
+NPM packages: **`@samesake/core`** (SDK), **`@samesake/server`**, **`@samesake/cli`**, **`@samesake/mcp`** at **2.6.0**. The current public name is **Samesake**. The HTTP app still lives at `apps/matcher/`.
 
 Search and match share embeddings, Postgres caches, and per-project runtime DDL.
 

apps/bom-quotation/README.md

@@ -31,7 +31,7 @@ BOM (PDF / XLSX)
 
 ## Run
 
-Requires `DATABASE_URL` (Postgres + pgvector) and `GEMINI_API_KEY` — already in the
+Requires `SAMESAKE_DATABASE_URL` (Postgres + pgvector) and `GEMINI_API_KEY` — already in the
 repo-root `.env`.
 
 ```bash

apps/bom-quotation/server/src/app.ts

@@ -16,9 +16,9 @@ import { priceLine } from "./pipeline/price.ts";
 import type { CustomerRef, MatchedLine, Quotation } from "../../shared/types.ts";
 
 loadEnv();
-const url = process.env.DATABASE_URL;
+const url = process.env.SAMESAKE_DATABASE_URL;
 if (!url || !process.env.GEMINI_API_KEY) {
-  console.error("DATABASE_URL and GEMINI_API_KEY are required");
+  console.error("SAMESAKE_DATABASE_URL and GEMINI_API_KEY are required");
   process.exit(1);
 }
 

apps/bom-quotation/server/src/config.ts

@@ -5,9 +5,9 @@ import type { Company, CatalogPart } from "../../shared/types.ts";
 const ROOT = join(import.meta.dir, "../..");
 const REPO_ROOT = join(ROOT, "../..");
 
-/** Load DATABASE_URL + GEMINI_API_KEY from the repo-root .env if not already set. */
+/** Load SAMESAKE_DATABASE_URL + GEMINI_API_KEY from the repo-root .env if not already set. */
 export function loadEnv(): void {
-  if (process.env.DATABASE_URL && process.env.GEMINI_API_KEY) return;
+  if (process.env.SAMESAKE_DATABASE_URL && process.env.GEMINI_API_KEY) return;
   try {
     const env = readFileSync(join(REPO_ROOT, ".env"), "utf8");
     for (const line of env.split("\n")) {
@@ -17,7 +17,7 @@ export function loadEnv(): void {
       if (eq === -1) continue;
       const k = t.slice(0, eq).trim();
       const v = t.slice(eq + 1).trim();
-      if (k === "DATABASE_URL" || k === "GEMINI_API_KEY") process.env[k] ??= v;
+      if (k === "SAMESAKE_DATABASE_URL" || k === "GEMINI_API_KEY") process.env[k] ??= v;
     }
   } catch {
     /* no .env — rely on the ambient environment */

apps/bom-quotation/server/src/run.ts

@@ -10,9 +10,9 @@ import { renderQuotationPdf } from "./pipeline/quote.ts";
 import type { CustomerRef } from "../../shared/types.ts";
 
 loadEnv();
-const url = process.env.DATABASE_URL;
+const url = process.env.SAMESAKE_DATABASE_URL;
 if (!url || !process.env.GEMINI_API_KEY) {
-  console.error("DATABASE_URL and GEMINI_API_KEY are required");
+  console.error("SAMESAKE_DATABASE_URL and GEMINI_API_KEY are required");
   process.exit(1);
 }
 

apps/bom-quotation/server/src/setup.ts

@@ -6,9 +6,9 @@ import { loadEnv, company, PROJECT } from "./config.ts";
 import { makeMatcher, setupCatalog } from "./catalog.ts";
 
 loadEnv();
-const url = process.env.DATABASE_URL;
+const url = process.env.SAMESAKE_DATABASE_URL;
 if (!url) {
-  console.error("DATABASE_URL required (set it in the repo-root .env)");
+  console.error("SAMESAKE_DATABASE_URL required (set it in the repo-root .env)");
   process.exit(1);
 }
 if (!process.env.GEMINI_API_KEY) {

apps/bom-quotation/web/README.md

@@ -4,7 +4,7 @@ TanStack Start frontend for the BOM → quotation pipeline.
 
 ## Prerequisites
 
-- API env vars in the repo-root `.env`: `DATABASE_URL`, `GEMINI_API_KEY`
+- API env vars in the repo-root `.env`: `SAMESAKE_DATABASE_URL`, `GEMINI_API_KEY`
 - Catalog bootstrapped: `bun run setup` (from `apps/bom-quotation`)
 
 ## Run

apps/docs/.gitignore

@@ -19,3 +19,4 @@ pnpm-debug.log*
 
 # macOS-specific files
 .DS_Store
+.vercel

apps/docs/src/content/docs/guides/conversational-search.mdx

@@ -54,17 +54,24 @@ Currency symbols and commas are stripped (`"Rs 5,000"` → `5000`). Crucially, t
 Hard filters always win over semantic similarity. A gorgeous 5,200 dress that's a perfect semantic match for *"summer wedding"* is still dropped by `price ≤ 5000`. That's the point — the shopper drew a wall.
 </Aside>
 
-## You get this for free with the fashion preset
+## You get this for free with the fashion template
 
-`fashionSearchPreset` (and `fashion.nlq`) wire the parser, the schema, and the filterable fields for you:
+The `fashion` template (`fashion.nlq`) wires the parser, the schema, and the filterable fields for you:
 
 ```ts
-import { fashionSearchPreset } from "@samesake/core";
-
-const products = fashionSearchPreset({
-  textModel: "gemini-embedding-2",
-  textDim: 1536,
-  enrichmentModel: "gemini-3.1-flash-lite",
+import { collection, Channels, fashion } from "@samesake/core";
+
+const products = collection("products", {
+  fields: fashion.fields(),
+  indexing: fashion.indexing(),
+  embeddings: { doc: { model: "gemini-embedding-2", dim: 1536 } },
+  spaces: fashion.spaces(),
+  enrich: fashion.enrichPipeline(),
+  search: {
+    channels: [Channels.fts({ fields: ["title"] }), Channels.cosine({ embedding: "doc" }), Channels.spaces({})],
+    combiner: "rrf",
+    nlq: { instructions: fashion.nlq.instructions, schema: fashion.nlq.schema() },
+  },
 });
 ```
 

apps/docs/src/content/docs/guides/enrich-pipeline.mdx

@@ -50,16 +50,16 @@ Each target guide below shows the same spine:
 Keep the three main stages as **separate durable steps**. That's how you get per-stage retries, per-stage timing in your observability tool, and the guarantee that a flaky embed call doesn't force you to re-fetch your whole catalog.
 
 ```ts title="matcher.ts — shared setup"
-import { createMatcher, fashionRerank } from "@samesake/server";
+import { createMatcher, llmRerank } from "@samesake/server";
 import { products } from "./catalog.ts";
 import { geminiEmbed, geminiGenerate } from "./gemini.ts";
 
 export const matcher = createMatcher({
-  databaseUrl: process.env.DATABASE_URL!,
-  apiKey: process.env.API_KEY!,
+  databaseUrl: process.env.SAMESAKE_DATABASE_URL!,
+  apiKey: process.env.SAMESAKE_API_KEY!,
   embed: geminiEmbed,                       // gemini-embedding-2
   generate: geminiGenerate,                 // gemini-3.1-flash-lite
-  rerank: fashionRerank(geminiGenerate),
+  rerank: llmRerank(geminiGenerate),
 });
 
 await matcher.apply("market", { entities: [], collections: [products] });

apps/docs/src/content/docs/guides/eval-gate.mdx

@@ -17,16 +17,17 @@ Do not change these without running the harness. Fabricating "calibrated" number
 [`examples/fashion-search/eval-judge.ts`](https://github.com/asyncdotengineering/samesake/blob/main/examples/fashion-search/eval-judge.ts) is the runnable CI gate. It:
 
 1. Loads the frozen golden set `evals/golden-queries-fashion-lk.json` (50 queries with objective `constraints`).
-2. Calls `matcher.runEval(...)` with a graded LLM judge (`makeLlmJudge`).
+2. Calls `matcher.runEval(...)` with a graded ESCI LLM judge (`makeLlmJudge`) — a **different model family** than the enrich pipeline (`runEval` rejects a same-family or undeclared judge on enriched collections).
 3. Aggregates Hit@K, nDCG@K, MRR, null-rate, and constraint-violation rate.
 4. Sets `pass` from `thresholds` — exits **0** when all metrics meet thresholds, **nonzero** when any regress.
 
 ```ts
 const res = await matcher.runEval(PROJECT, COLLECTION, {
   queries: golden.queries,
-  judge,
+  judge, // e.g. makeLlmJudge(openaiGenerate, { model: "gpt-4.1-mini" }) vs a Gemini pipeline
   k: 10,
-  relevanceFloor: 1,
+  relevanceFloor: 2, // ESCI: Substitute or better counts as relevant
+
   thresholds: { ndcgAtK: 0.6, nullRate: 0.1, constraintViolationRate: 0 },
 });
 process.exit(res.pass ? 0 : 1);
@@ -36,7 +37,7 @@ process.exit(res.pass ? 0 : 1);
 
 ### Dry-run without a key
 
-When `GEMINI_API_KEY` is unset, the script **dry-runs cleanly** (exit 0): it prints how many queries would run and does not call the judge. Use this in CI type-check paths; live gating requires the key.
+When `GEMINI_API_KEY` or `OPENAI_API_KEY` is unset, the script **dry-runs cleanly** (exit 0): it prints how many queries would run and does not call the judge. Use this in CI type-check paths; live gating requires both keys (Gemini pipeline + cross-family OpenAI judge).
 
 ```bash
 # Dry-run (no key) — verifies the script loads; exit 0
@@ -50,7 +51,7 @@ GEMINI_API_KEY=... bun examples/fashion-search/eval-judge.ts
 
 <Steps>
 
-1. **Calibrate the judge (once per prompt change).** Run `matcher.calibrateSearch` / `calibrate.ts` against a labeled fixture so judge-vs-human F1 meets your trust bar. Bump the judge `version` string when the rubric changes — stale cache entries must not mix versions. See [Relevance judge](/reference/relevance-judge/) for the judge API, prompt, and schema.
+1. **Calibrate the judge (once per prompt change).** Run `matcher.calibrateSearch` / `calibrate.ts` against a labeled fixture so judge-vs-human F1 meets your trust bar. Rubric edits invalidate cached grades automatically (the judge version embeds a content hash of the prompt — `esci-v1@<hash>`); pass a new `version` tag only for model or methodology changes. See [Relevance judge](/reference/relevance-judge/) for the judge API, prompt, and schema.
 
 2. **Baseline the current placeholders.** With `GEMINI_API_KEY` set, run the gate and save the artifact:
 
@@ -78,7 +79,7 @@ GEMINI_API_KEY=... bun examples/fashion-search/eval-judge.ts
 | Confidence floor (G2) | `FASHION_CONFIDENCE_FLOOR` / `fashion.indexing().gate` | Re-enrich + re-index (gate changes quarantine) |
 | Relevance exponent (G7) | `search.rankingPolicy.relevanceExponent` | Re-run eval only |
 | Axis weights (G7) | `rankingPolicy.weights.*` | Re-run eval only |
-| Rerank blend | `matcher` `rerank: fashionRerank(generate)` | Re-run eval only |
+| Rerank blend | `matcher` `rerank: llmRerank(generate)` | Re-run eval only |
 
 Core ranking uses **multiplicative fusion**: normalized relevance raised to `relevanceExponent`, multiplied by availability / business / personalization axes. Hard axes (e.g. availability) apply multiplicatively; soft axes add after the product. A `minRelevanceFloor` drops hits below a relevance cutoff before fusion.
 

apps/docs/src/content/docs/guides/faceted-search.mdx

@@ -86,15 +86,15 @@ For a catalog that changes often, schedule that trio (or wire it to webhooks) so
 Pass the facet field names on `search`. samesake runs the same query and filters you would for results, then aggregates counts for each requested facet **on that narrowed set**:
 
 ```ts title="search.ts"
-import { createMatcher, fashionRerank } from "@samesake/server";
+import { createMatcher, llmRerank } from "@samesake/server";
 import { geminiEmbed, geminiGenerate } from "./gemini.ts";
 
 const matcher = createMatcher({
-  databaseUrl: process.env.DATABASE_URL!,
-  apiKey: process.env.API_KEY!,
+  databaseUrl: process.env.SAMESAKE_DATABASE_URL!,
+  apiKey: process.env.SAMESAKE_API_KEY!,
   embed: geminiEmbed,
   generate: geminiGenerate,
-  rerank: fashionRerank(geminiGenerate),
+  rerank: llmRerank(geminiGenerate),
 });
 
 const result = await matcher.search("shop", "products", {
@@ -195,7 +195,7 @@ Two latency knobs worth knowing:
 - **`rerank: false`** on hot paths skips the second-stage LLM judge when you want the snappiest first-stage RRF order.
 - **`weights`** per query lean harder on keywords vs meaning vs visual without reindexing.
 
-For production quality on fashion, keep `fashionRerank(geminiGenerate)` wired globally and only opt out on typeahead if profiling says you need to.
+For production quality on fashion, keep `llmRerank(geminiGenerate)` wired globally and only opt out on typeahead if profiling says you need to.
 
 ## Sensible defaults recap
 
@@ -204,7 +204,7 @@ For production quality on fashion, keep `fashionRerank(geminiGenerate)` wired gl
 | Field shapes + facet flags | `fashion.fields()` |
 | Enrich + gate | `fashion.enrichPipeline()` + `fashion.indexing()` |
 | Embeddings | `gemini-embedding-2`, dim 1536 |
-| Generate / judge / rerank | `gemini-3.1-flash-lite` via `fashionRerank(geminiGenerate)` |
+| Generate / judge / rerank | `gemini-3.1-flash-lite` via `llmRerank(geminiGenerate)` |
 | Visual signal | `fashion.spaces({ visual: true })` |
 
 Copy the provider adapters from [Providers](/reference/providers/), `apply` once, then the push → enrich → index loop above.

apps/docs/src/content/docs/guides/idea-to-search.mdx

@@ -70,16 +70,16 @@ That's a *lot* of good behaviour for not a lot of typing — because the preset
 The `matcher` holds your database connection and your two model functions. We'll also switch on a second-stage **re-ranker** — a quick "is this *actually* relevant?" pass over the top results — because for fashion it's worth it.
 
 ```ts title="search.ts"
-import { createMatcher, fashionRerank } from "@samesake/server";
+import { createMatcher, llmRerank } from "@samesake/server";
 import { products } from "./catalog.ts";
 import { geminiEmbed, geminiGenerate } from "./gemini.ts"; // your two model fns — see Providers
 
 const matcher = createMatcher({
-  databaseUrl: process.env.DATABASE_URL!,
-  apiKey: process.env.API_KEY!,
+  databaseUrl: process.env.SAMESAKE_DATABASE_URL!,
+  apiKey: process.env.SAMESAKE_API_KEY!,
   embed: geminiEmbed,                       // turns text/images into vectors (gemini-embedding-2)
   generate: geminiGenerate,                 // reads products + judges relevance (gemini-3.1-flash-lite)
-  rerank: fashionRerank(geminiGenerate),    // the relevance double-check, on by default once wired
+  rerank: llmRerank(geminiGenerate),    // the relevance double-check, on by default once wired
 });
 
 await matcher.apply("shop", { entities: [], collections: [products] });