docs(evals): scoring rubric for /recce-verify v1

.gitignore

@@ -43,6 +43,10 @@ temp/
 *.tmp
 .worktrees/
 
+# DRC-3402 eval fixtures — regenerable artifacts (see evals/agent-blind-spots/fixtures/README.md)
+evals/agent-blind-spots/.tmp/
+evals/agent-blind-spots/fixtures/*/artifacts/
+
 # Coverage / editor logs
 .coverage
 .nvimlog

evals/agent-blind-spots/README.md

@@ -0,0 +1,59 @@
+# Eval — Agent Blind Spots / `/recce-verify` v1
+
+Qualitative case-study eval for the `/recce-verify` skill v1.
+
+## What this is
+
+Six PR fixtures from `DataRecce/jaffle_shop_golden` exercising distinct verification classes (semantic, row-grain, refactor, type, schema-expansion, multi-model). Each fixture produces one structured case study viewed through three lenses (binary catch, primary evidence tier, counterfactual delta against a frozen agent-only baseline) — these are observation axes, not metrics; **do not aggregate them**. Output is an action-prioritized gap report (target ≤5 entries, with an overflow path) that gates v1 backend additions.
+
+## Why qualitative, not quantitative
+
+N=6 PRs is too small for statistics. The eval is a **named-case narrative**, not a leaderboard. Presenting it as "X% accuracy" or "Y% improvement" overstates rigor and breaks credibility with Super / 205DataLab. The rubric is built around case studies; resist the temptation to aggregate.
+
+## Layout
+
+```
+evals/agent-blind-spots/
+├── README.md            ← this file
+├── RUBRIC.md            ← scoring rules; read before adding or scoring fixtures
+├── build_fixtures.sh    ← rebuilds the gitignored artifacts/ per fixture (DRC-3402)
+├── fixtures/            ← one directory per PR fixture
+│   ├── README.md               ← fixture-set caveats + build instructions
+│   └── <pr-id-slug>/
+│       ├── README.md           ← what the PR does + expected verdicts
+│       ├── tier-0-baseline.md  ← frozen agent-only verdict (template in templates/)
+│       ├── commits.txt         ← base + head SHAs read by build_fixtures.sh
+│       ├── diff.patch          ← small source-models diff base..head (committed)
+│       └── artifacts/          ← gitignored; produced by build_fixtures.sh
+├── templates/
+│   ├── tier-0-baseline.md      ← per-fixture frozen baseline template
+│   └── gap-report.md           ← gap-report template (target ≤5 entries)
+└── runs/
+    └── <YYYY-MM-DD>/
+        ├── gap-report.md           ← filled gap report for the run
+        └── <pr-id>-scoring.md      ← per-fixture scoring per RUBRIC.md
+```
+
+Before any eval run, build the gitignored artifacts:
+
+```bash
+cd evals/agent-blind-spots && ./build_fixtures.sh
+```
+
+See [`fixtures/README.md`](./fixtures/README.md) for the full per-fixture caveats (PR #16 merge head, PR #20 intermediate trap, PR #46 stress test, empty-DuckDB catalog stats, PR #14 older base) and pinned versions.
+
+## How to run
+
+1. Pick a fixture in `fixtures/`.
+2. If `tier-0-baseline.md` is missing, run the agent in Tier-0 mode (no Recce) and capture the verdict per the template. The baseline file is **frozen at commit**: the commander commits it (per the workspace's normal change-control flow) before authorizing the with-Recce run, and the baseline must not be edited afterwards even if later evidence suggests revision.
+3. Run the agent with `/recce-verify` available.
+4. Score the run in `runs/<YYYY-MM-DD>/<pr-id>-scoring.md` using `RUBRIC.md`.
+5. Once all six fixtures are scored, fill `runs/<YYYY-MM-DD>/gap-report.md` (action-prioritized; target ≤5 entries, exceed-with-rationale allowed).
+
+## References
+
+- Project: [Agent-blind spots: /recce-verify v1](https://linear.app/recce/project/agent-blind-spots-recce-verify-v1-d2bb2d77bff8)
+- Rubric source: [DRC-3403](https://linear.app/recce/issue/DRC-3403)
+- Fixture set: [DRC-3402](https://linear.app/recce/issue/DRC-3402)
+- Skill: [DRC-3404](https://linear.app/recce/issue/DRC-3404)
+- Notion: [Single-env Capability Breakdown](https://www.notion.so/35a79451d357807ba2befc365cb74217)

evals/agent-blind-spots/RUBRIC.md

@@ -0,0 +1,118 @@
+# Scoring Rubric — Agent Blind Spots / `/recce-verify` v1
+
+## Framing
+
+Qualitative case studies, **not** quantitative eval. N=6 PR fixtures is too small for statistics. The rubric below produces structured case studies; reading them as "X% accuracy" or "Y% improvement" is wrong and breaks credibility with Super / 205DataLab. Always present results as named-case narratives, not aggregates.
+
+## Per-fixture case-study lenses (3, not metrics)
+
+Each lens produces a structured observation, not a score. The three together form one case study per fixture. **Do not aggregate across fixtures.**
+
+### 1. Binary catch
+
+Did the agent reach the correct verdict — catch the intentional bug, or correctly flag a behavior-preserving refactor as safe?
+
+| Value | Meaning |
+|-------|---------|
+| `catch` | Agent correctly identified the situation and recommended the right action (block or approve). |
+| `miss` | Agent shipped the change as safe when it wasn't, or rejected a safe change. |
+| `partial` | Agent flagged a real concern but for the wrong reason, or missed half of a multi-issue PR. |
+
+### 2. Primary evidence tier (+ capability subset used)
+
+Which capability tier produced the **decisive** piece of evidence the agent cited in its verdict? Tiers are nested — a higher tier always has access to lower-tier inputs.
+
+If the agent cited evidence from multiple tiers, record the higher tier as decisive (tier ordering `0 < 1 < 2` applies across tiers) and list secondary citations under Notes. Within Tier 1, the subsets `1a / 1b / 1c` are **not** ordered — they are orthogonal capability surfaces (and `1a` is computed via `1b`, so they're entangled). If the agent cited evidence from multiple Tier-1 subsets, mark the subset whose evidence appeared in the verdict sentence as **primary**, the rest as **supporting**; primary/supporting replaces "highest" within Tier 1.
+
+| Tier | Agent capability set |
+|------|----------------------|
+| 0 | dbt manifest + compiled SQL + git diff only — no Recce. |
+| 1 | Tier 0 + Recce against a **single dev environment**. v1 target. Record *which subset* the agent actually used: **1a** column-level lineage (CLL), **1b** AST / SQL semantic analysis, **1c** structured queries against the current dev env (row counts, distributions, nulls). "Tier 1" alone is ambiguous — always record the subset(s). |
+| 2 | **Beyond v1 — base environment needed.** Not part of `/recce-verify` v1's offering. Record only when a fixture's verdict provably requires a base/prod comparison (data diff, row-grain delta, lineage delta vs prod). A fixture reaching Tier 2 is a **v2 signal** for the gap report, not a v1 capability claim. |
+
+### 3. Counterfactual delta against frozen baseline
+
+Binary catch (lens 1) is recorded **for both** the Tier-0 baseline run and the with-Recce run. The case study's headline finding is the **delta** between those two binary-catch values, not the absolute with-Recce value.
+
+Order the three catch values as `catch > partial > miss` (closer to ground truth → less close). Every baseline → with-Recce pair falls into one of three buckets:
+
+| Delta bucket | Examples | What it means |
+|--------------|----------|---------------|
+| **Improvement** | `miss → catch`, `miss → partial`, `partial → catch` | Recce shifted the verdict toward ground truth. Positive signal — describe *what* evidence drove the shift. |
+| **Same** | `miss → miss`, `partial → partial`, `catch → catch` | No shift in verdict. Still useful — for `catch → catch`, record what Recce showed (validates redundancy or reveals Recce wasn't needed); for `miss → miss` or `partial → partial`, the agent ignored or didn't surface decisive evidence — feed back into the skill prompt. |
+| **Regression** | `catch → partial`, `catch → miss`, `partial → miss` | Recce misled the agent. Rare but important — investigate in Notes; this is a v1-release-blocker signal. |
+
+The baseline file is **frozen at commit**: once it lands on the branch, the with-Recce run can begin, and the baseline must not be edited even if later evidence suggests it should be revised. Without this control, results conflate model variance with Recce signal. Baseline format → see `templates/tier-0-baseline.md`.
+
+**Same-model contract.** The with-Recce run MUST use the same agent + model as the frozen Tier-0 baseline (the `Agent` and `Model` fields in `templates/tier-0-baseline.md`). If a model upgrade lands mid-eval, either re-capture the baseline (and re-commit it) or record the mismatch in the per-fixture artifact's Notes section and treat that fixture's delta as confounded. Without this constraint, the lens-3 delta conflates Recce signal with model drift.
+
+## Tier-0 agent runtime contract
+
+To make Tier-0 baselines reproducible across runs, the agent receives **the same raw material Recce ingests, minus Recce's structured surfacing**. This isolates "what Recce contributes" from "what the agent could have figured out from artifacts alone." A weaker Tier-0 setup (e.g., diff-only, no artifacts) would understate the agent's solo capability and overstate Recce's signal.
+
+**Inputs per fixture** (populated by `build_fixtures.sh`):
+
+- `fixtures/<id>/diff.patch` — source-model diff between base and head
+- `fixtures/<id>/artifacts/manifest-before.json`, `manifest-after.json` — dbt manifests pre/post
+- `fixtures/<id>/artifacts/compiled-before/`, `compiled-after/` — compiled SQL trees pre/post
+- `fixtures/<id>/artifacts/catalog-before.json`, `catalog-after.json` — schema-only (row/col stats are zero in this fixture set; documented in the fixtures README)
+- Read access to the dbt project source at the head SHA — materialized **per fixture** by `build_fixtures.sh` at `evals/agent-blind-spots/.tmp/sources/<id>/`. The shared clone at `.tmp/jaffle_shop_golden/` is build-script scratch; do NOT read from it because the build loop leaves it at the last fixture's SHA.
+
+**Generic tools allowed at Tier 0:** file read, grep / ripgrep, `jq`, `git log` / `git diff` / `git show` against the per-fixture source tree at `.tmp/sources/<id>/`. Anything the agent could plausibly run on a developer's laptop without Recce installed *and without regenerating any frozen Tier-0 input*.
+
+**Explicitly NOT allowed at Tier 0:** Recce CLI, Recce MCP, any `/recce-*` skill (including `/recce-verify`), warehouse access, `dbt run`, `dbt test`, `dbt parse`, `dbt compile`, `dbt docs generate`, live SQL execution, or any other command that regenerates the manifest/compiled/catalog artifacts. The artifacts under `fixtures/<id>/artifacts/` are the **frozen Tier-0 inputs**; regenerating them lets the agent reach beyond the captured snapshot (e.g., picking up later upstream-package changes) and breaks reproducibility across runs. Also not allowed: comparison against a base/prod environment beyond what is already in the artifacts above.
+
+**Prompt shape** — eval runners write the actual prompt and **MUST record it verbatim** in the Tier-0 baseline's "Prompt given to agent" section, including any agent-specific framing. To keep baselines comparable across runs:
+
+- The prompt MUST describe the inputs above without paraphrasing what each contains (re-describing the inputs primes the agent in ways that vary between runners).
+- The prompt MUST ask for catch / miss / partial and approve / request-changes / abstain in those terms.
+- The prompt MUST NOT add steering language toward humility ("be cautious about flagging issues", "only flag when you're confident") OR aggression ("find as many issues as possible", "be thorough"). Use neutral framing; "review this PR" is enough.
+- Agent-specific scaffolding (file-access mode, tool whitelisting, system prompt) is allowed but must be recorded in the baseline's Notes section so the delta is interpretable.
+
+A reference shape that satisfies the constraints:
+
+> "Review this dbt PR. The inputs listed in the Tier-0 runtime contract are available. Decide catch / miss / partial per the rubric, recommend approve / request-changes / abstain, and write verdict + verbatim reasoning into `tier-0-baseline.md`."
+
+## Per-fixture artifact
+
+Each fixture's scoring lives in `runs/<YYYY-MM-DD>/<pr-id>-scoring.md` with this structure:
+
+```markdown
+# Fixture <pr-id> — <one-line title>
+
+- Tier-0 baseline (frozen): <link to fixtures/<pr-id>/tier-0-baseline.md>
+- Run date: <YYYY-MM-DD>
+- Agent / model: <name>
+- Primary evidence tier + subset: <0 | 1a | 1b | 1c | 2>  (list secondary citations under Notes if any)
+- Binary catch (this run): <catch | miss | partial>
+- Binary catch (Tier-0 baseline): <catch | miss | partial>
+- Delta: <baseline → this run, e.g. `miss → catch`>
+
+## Evidence Recce surfaced
+
+Verbatim from tool output (or paraphrased with a link). One bullet per piece of evidence.
+
+## Conclusion the agent reached
+
+What the agent actually said / recommended. Quote.
+
+## Delta vs Tier-0 baseline
+
+What changed between Tier-0 baseline and this run. Why. If no delta, say so explicitly.
+
+## Notes
+
+Failure modes, hallucinations, suspicious reasoning, anything worth feeding back into the skill prompt.
+```
+
+## Gap report (output of the eval run)
+
+The eval output is **prioritized for action**, **targets ≤5 entries**, and lives at `runs/<YYYY-MM-DD>/gap-report.md`. Format → see `templates/gap-report.md`.
+
+The broader capability backlog (e.g., the 33-item Notion list) is reference material; the gap report is the deliberate cut of "what to do next" derived from the six case studies. The ≤5 target is a discipline against the backlog bleeding back in — if a receipts-style review leaves ≥6 genuinely independent blockers, exceed the target and add a one-line note explaining why one couldn't be subsumed or deferred. "Prioritized" here means *priority of action*, not ordinal performance.
+
+## What this rubric is NOT
+
+- **Not a leaderboard.** Don't compute "agent A vs agent B" scores.
+- **Not a regression suite.** Don't run it on every commit; re-run when the skill prompt or backend changes meaningfully.
+- **Not statistical evidence.** N=6 means stories, not averages. If a stakeholder asks for "the number", refer them to this section.