docs(evals): scoring rubric for /recce-verify v1

[even-wei]

Summary

• Add evals/agent-blind-spots/ — qualitative case-study eval contract for the /recce-verify v1 skill, gating v1 backend additions on a ≤5-entry gap report.
• RUBRIC.md defines three case-study lenses (binary catch, primary evidence tier, counterfactual delta vs frozen baseline) and resists aggregation: explicitly not a leaderboard, not a regression suite, not statistical evidence.
• Templates for per-fixture Tier-0 baseline (frozen at commit) and gap report (target ≤5 with overflow-with-rationale path); README sets up the run procedure and the Super / 205DataLab framing.

Why this shape

N=6 is too small for statistics. The eval is built around named-case narratives so it survives presentation to Super / 205DataLab without the credibility trap of implied false rigor. Two review passes shaped this: Codex via /codex:rescue (11 findings — leaderboard-slip language, tier ambiguity, delta-vs-absolute conflict, gap-report cap) and a self-review via /recce-dev:claude-code-review (3 ISSUEs + 3 NOTEs — delta taxonomy non-exhaustive, frozen-baseline wording inconsistent, commit-instruction collision with workspace rule).

Tier 1 capability subsets

/recce-verify v1 targets Tier 1 (single dev environment). The rubric splits this into recordable subsets so attribution is unambiguous: 1a column-level lineage, 1b AST / SQL semantic analysis, 1c structured queries against the current dev env. Tier 2 (base environment) is explicitly out of v1; a fixture reaching it is a v2 signal for the gap report, not a v1 capability claim.

Out of scope

• Fixtures themselves — built separately in DRC-3402 (parallel issue).
• Skill implementation — DRC-3404.
• Eval run — DRC-3405; uses this rubric as input.

Test plan

• Codex /codex:rescue adversarial review — findings addressed in b672c9c
• /recce-dev:claude-code-review self-review on branch — findings addressed in 7206e10
• First real eval run consumes this rubric (gated on DRC-3402 fixtures + DRC-3404 skill)

Resolves DRC-3403.

[even-wei]

Update (HEAD: c0cb2a5) — Verdict: GO

Re-reviewed after commit c0cb2a5 ("docs(evals): tighten Tier-0 contract and gap-report overflow") and verified the cross-PR fix at PR #29 commit 767ca54. Both prior ISSUEs are resolved; 2 of 5 NOTEs are resolved; the 3 remaining NOTEs are non-blocking observations that can be addressed in a follow-up.

Prior findings — resolution

#	Finding	Status
ISSUE 1	Template-vs-rubric drift on Tier-0 inputs	Resolved — templates/tier-0-baseline.md:16 now references the full Tier-0 runtime contract and enumerates all six artifact files + the per-fixture head-SHA source tree at .tmp/sources/<fixture-id>/. Template no longer undercounts.
ISSUE 2	build_fixtures.sh working-tree contract breakage	Resolved — RUBRIC.md:55 retargets read access to .tmp/sources/<id>/ (per-fixture worktree) and explicitly warns against reading from the shared .tmp/jaffle_shop_golden/ clone. PR #29's build_fixtures.sh at 767ca54 calls git worktree add --detach "${source_dir}" "${head_sha}" after each fixture is built, materializing six distinct head SHAs. Verified the build script end-to-end against the spec.
NOTE 1	dbt parse/compile/docs generate ambiguous	Resolved — RUBRIC.md:59 adds these three commands to the explicit-NOT-allowed list with the right rationale ("regenerating … lets the agent reach beyond the captured snapshot"). Template echoes the constraint at line 16.
NOTE 2	Gap-report cap gameable via fix-label splitting	Resolved — templates/gap-report.md:5–7 makes "Cheapest fix" the explicit dedup key, gives a worked example (six symptoms → one entry under "no base-environment comparison"), and the overflow line instructs reviewers to skim the Cheapest-fix column for duplicates.
NOTE 3	"Regression = v1-release-blocker" too strong given N=6	Still open — RUBRIC.md:41 unchanged. Non-blocking; can soften in a follow-up.
NOTE 4	Lens 2 singular-vs-plural inconsistency	Still open — RUBRIC.md:23 ("the decisive piece"), :28 ("subset(s)"), and :82 (<0 | 1a | 1b | 1c | 2> pipe-or) remain inconsistent. Non-blocking.
NOTE 5	1a / 1b / 1c not cleanly distinguishable	Still open — no worked example added. Non-blocking.

New findings introduced by c0cb2a5

None at ISSUE or BLOCKER severity. Minor observations:

• NOTE A — Path style inconsistency (cosmetic). RUBRIC.md:55 writes the per-fixture source path as evals/agent-blind-spots/.tmp/sources/<id>/ (repo-root-relative), while RUBRIC.md:57 and templates/tier-0-baseline.md:16 write .tmp/sources/<id>/ (relative to the eval directory). Both resolve to the same place given the build script's SCRIPT_DIR, but the two forms next to each other invite reader confusion. Pick one and use it consistently.
• NOTE B — <id> vs <slug> terminology drift. RUBRIC and template use <id> / <fixture-id>; build_fixtures.sh and the on-disk directories use ${slug} (pr1-fix-clv, etc.). The PR docs(evals): build six PR fixture set for /recce-verify v1 eval #29 fixtures/README.md presumably reconciles these, but a one-line glossary line in RUBRIC.md ("<id> = fixture slug, e.g. pr1-fix-clv") would close the loop. Non-blocking.
• NOTE C — Intermediate-commit artifacts unmentioned. build_fixtures.sh produces an extra artifacts/intermediate-commit-<sha>/ directory for pr44-promotion-flags, and that fixture also ships a diff-from-base-to-intermediate.patch. RUBRIC.md:49–55 enumerates the standard six artifacts but doesn't acknowledge this PR-specific extra. Either add a one-liner ("Some fixtures ship additional intermediate snapshots — see the fixture's README.md") or accept this as documented at the fixture level. Non-blocking.

Verification

• git diff ca837ed..c0cb2a5 --stat confirms the fix commit touches only the three intended files (RUBRIC.md +11/-6, gap-report.md +3/-1, tier-0-baseline.md +1/-1).
• Cross-PR claim verified: fetched build_fixtures.sh from PR docs(evals): build six PR fixture set for /recce-verify v1 eval #29 767ca540 directly; confirmed git worktree add --detach "${SOURCES_DIR}/${slug}" "${head_sha}" lines exist and run inside build_fixture() after each pair of build_at_sha calls.
• No stale references to the old "checkout left by the build script" promise remain in either file.

Recommendation

Ready to merge. The two ISSUEs that blocked the prior review are cleanly resolved with concrete, traceable changes. Remaining NOTEs (3 carried over, 3 new) are quality observations that don't gate v1 — fold them into a follow-up doc pass if desired.

---

Original review (HEAD reviewed: ca837ed) — NO-GO, kept for audit history

Code Review: PR #28 — docs(evals): scoring rubric for /recce-verify v1

Files reviewed: 4 (README.md, RUBRIC.md, templates/tier-0-baseline.md, templates/gap-report.md)
Categories: docs-only
Passes run: A (correctness/coherence), C (cross-doc consistency), F (diff-specific drift across 4 commits)
HEAD reviewed: ca837ed
Stance: Independent adversarial pass — no sibling reviews consulted.

Validation Results

Pass A — Coherence of the three lenses — PASS (with one NOTE)

The three lenses (binary catch, primary evidence tier, counterfactual delta) compose without contradiction. A with-Recce run whose Primary evidence tier scores 0 maps cleanly to a Same delta, which is internally consistent — though a reader could briefly conflate "this run had Tier-0 access only" with "this with-Recce run cited Tier-0 evidence as decisive." See NOTE 4.

Pass C — Cross-doc consistency (README ↔ RUBRIC ↔ templates ↔ Tier-0 contract) — FAIL

The Tier-0 runtime contract added in ca837ed is not reflected in templates/tier-0-baseline.md. See ISSUE 1.

Pass F — Diff-specific cross-reference with PR #29 (build_fixtures.sh) — FAIL

Rubric line 55 promises "read access to the dbt project source at the head SHA (the .tmp/jaffle_shop_golden/ checkout left by the build script)." The implementation in PR #29 leaves the checkout at the LAST fixture built (pr46-net-clv-segments), not the current fixture. See ISSUE 2.

Verification Results

Docs-only change; no lint/test suite applies. CI guard bundle-freshness.yml is unaffected.

Verdict: NO-GO

Issues

1. Template-vs-rubric drift on Tier-0 inputs — evals/agent-blind-spots/templates/tier-0-baseline.md:16 says inputs are "dbt manifest, compiled SQL pre/post, git diff" (singular manifest, no catalog, no source checkout). RUBRIC.md:49–55 specifies six artifact files (manifest-before.json, manifest-after.json, compiled-before/, compiled-after/, catalog-before.json, catalog-after.json) PLUS read access to .tmp/jaffle_shop_golden/. An evaluator filling out the template alone will run Tier-0 with a smaller input set than the rubric specifies, understating the agent's solo capability and overstating Recce's delta — the exact failure mode commit ca837ed was added to prevent. Fix: either expand the template's "Inputs available to agent" line to enumerate all six artifacts + source checkout (preferred — keeps the template self-contained), or have the template explicitly reference RUBRIC.md § Tier-0 agent runtime contract as the authoritative input list.

2. Working-tree contract not met by build_fixtures.sh — evals/agent-blind-spots/RUBRIC.md:55 promises "read access to the dbt project source at the head SHA (the .tmp/jaffle_shop_golden/ checkout left by the build script)." But build_fixtures.sh in PR docs(evals): build six PR fixture set for /recce-verify v1 eval #29 runs build_at_sha six (sometimes seven) times in a loop, each call doing git reset --hard && git clean -fdx && git checkout --detach <sha>. After the script finishes, the checkout is at pr46-net-clv-segments's head SHA — not the current fixture's. An evaluator running, say, pr1-fix-clv who reads source from .tmp/jaffle_shop_golden/ will be reading PR #46's models, not PR refactor(plugin): rename recce-dev to recce, add recce-dev for internal tooling #13's. Two options:

   • (a) Drop the working-tree dependency from the contract. Source at head SHA is already encoded in compiled-after/ + diff.patch; the agent doesn't need a live checkout.
   • (b) Add a "before-fixture" step to the rubric: git -C .tmp/jaffle_shop_golden checkout <head-sha-of-this-fixture> so the working tree matches the fixture under review. Document that the checkout is shared mutable state across fixture runs.

Notes

1. dbt parse/dbt compile are ambiguous at Tier 0 — RUBRIC.md:57–59 lists dbt run, dbt test, "live SQL execution" as forbidden, but not dbt parse, dbt compile, or dbt docs generate. These don't necessarily hit the warehouse (with --empty-catalog), but they regenerate manifest.json against the source repo's Snowflake-only profile (no credentials → may fail; may also succeed differently than the frozen manifest-after.json because seeds/ and dbt_packages/ aren't staged as build_fixtures.sh does). The phrase "Anything the agent could plausibly run on a developer's laptop without Recce installed" leaves an agent reasonably permitted to run them. Either explicitly forbid dbt parse|compile|docs generate at Tier 0 (preferred — preserves the "frozen artifacts" premise), or explicitly permit them with a one-line caveat about reproducibility.

2. Gap-report cap is gameable via fix-label splitting — templates/gap-report.md:5–6 collapses entries that "share a fix," but a motivated runner can split one gap into two by choosing different "Cheapest fix" labels (e.g., "skill-side prompt change" for entry A, "MCP tool addition" for entry B) even when one fix would close both. No reviewer gate exists. Suggest tightening: "Two entries that share a fix collapse to one — even if you could rephrase the fix label. If the same sentence change in the skill prompt would close two entries, they are one entry."

3. "Regression = v1-release-blocker" is too strong given N=6 framing — RUBRIC.md:41 says any regression bucket case is "a v1-release-blocker signal." Given the rubric's own "N=6 means stories, not averages" stance, a single regression could be model variance rather than a real Recce-misleads-agent failure mode. Suggest softening to "warrants a release-readiness discussion before shipping; do not dismiss without investigating root cause."

4. Lens 2 singular-vs-plural inconsistency — RUBRIC.md:23 says record "the decisive piece of evidence" (singular). Line 28 says "always record the subset(s)" (plural). Line 77's example field shows <0 | 1a | 1b | 1c | 2> (pipe-or, singular). For a fixture like pr1-fix-clv an agent can legitimately cite 1a + 1b + 1c (CLL shows the dependency, AST shows the new WHERE, query confirms row drop). Add an explicit tie-breaker within Tier 1 — e.g., "If multiple Tier-1 subsets contribute, record the one that produced the citation closest to the verdict statement as primary, list the rest under Notes."

5. 1a / 1b / 1c are not cleanly distinguishable — CLL (1a) is computed via AST analysis (1b). Row-grain effects are detected by AST and confirmed by 1c. The rubric handles overlap via "highest decisive tier," but 0 < 1 < 2 doesn't help within Tier 1. The note plural-subset recording mitigates this, but the lens table would be sharper if it gave a concrete example of "this is a 1a citation; this is a 1b citation; this is a 1c citation" for the same PR (e.g., pr1-fix-clv).

What I could not verify

• I could not run the eval end-to-end. The cross-reference to build_fixtures.sh is based on reading the script in PR docs(evals): build six PR fixture set for /recce-verify v1 eval #29; the working-tree behavior could differ if a future commit on PR docs(evals): build six PR fixture set for /recce-verify v1 eval #29 changes the checkout discipline.
• I did not check whether the Linear issue DRC-3403 description requires anything not in this PR.

What I looked for and did not find

• Quantitative-framing slips ("X% accuracy", "Y% improvement"). RUBRIC.md and README.md both explicitly reject these and there are no accidental slips.
• A "Recce vs no-Recce" leaderboard framing. Absent.
• Per-tier scoring aggregation (e.g., "Tier 1 mean catch rate"). Absent.
• A "this run vs prior run" delta that conflates model variance with Recce signal. The frozen-baseline rule (RUBRIC.md:43) explicitly prevents this.
• Tier 2 promoted as a v1 capability claim. RUBRIC.md:29 correctly flags Tier 2 as v2 signal only.
• Internal contradictions in delta-bucket examples. miss → catch, miss → partial, partial → catch all correctly land in Improvement; catch → partial, catch → miss, partial → miss all correctly land in Regression. Ordering catch > partial > miss is consistent across the document.

---

Independent review pass — no sibling reviews consulted. Posted as COMMENT (workspace pattern: don't REQUEST CHANGES on own PR).

[wcchang1115]

Three points from cross-reading RUBRIC.md against the templates and the paired PR #29. All three are docs-level, no scope creep. Order is roughly severity × cost-to-fix-now.

1. Pin same-model contract between baseline and with-Recce run

templates/tier-0-baseline.md records Agent / Model for the frozen baseline, and RUBRIC.md's per-fixture artifact section records Agent / model for the with-Recce run — but nothing requires them to match.

If the baseline is captured on Opus 4.6 and the with-Recce run lands weeks later on 4.7, the lens-3 delta confounds Recce signal with model drift. Since DRC-3405 isn't gated on a calendar, this is a real risk.

Suggested edit — one line, somewhere in lens 3 or under the Tier-0 runtime contract:

The with-Recce run MUST use the same agent + model as the frozen baseline. If a model upgrade lands mid-eval, either re-capture the baseline (and re-commit it) or record the mismatch under Notes and treat that fixture's delta as confounded.

2. Tier-0 runtime contract leaks "future" commits via shared object DB

Confirmed against PR #29's build_fixtures.sh: .tmp/sources/<slug>/ is a git worktree (the build script does git worktree prune and explicitly sweeps orphan worktree registrations), and the parent clone at .tmp/jaffle_shop_golden/ fetches all PR heads via +refs/pull/*/head:refs/remotes/origin/pr/*. Since worktrees share the parent .git object database, an agent inside .tmp/sources/pr1-fix-clv/ can run git log --all, git show <other-sha>, or git rev-parse origin/pr/<n> and immediately see (a) later commits on the same branch that contain the actual fix, and (b) every other fixture's head SHA.

That defeats the Tier-0 frozen-input contract.

Two paths:

• Rubric-side: add to "explicitly NOT allowed" — git log --all, git rev-list --all, git show / git rev-parse against non-ancestor SHAs, git fetch. Brittle (relies on agent compliance with a ban list it can trivially violate).
• Build-side (preferred): have build_fixtures.sh produce a fresh git clone --depth 1 per fixture instead of a worktree. The leak then becomes structurally impossible — there is no other object to reach.

The second is more robust because the contract is enforced by the filesystem, not by the agent's good behavior.

3. "Highest decisive tier" is ambiguous within Tier 1

RUBRIC.md lens 2: "record the highest decisive tier and list secondary citations under Notes." Tiers 0 < 1 < 2 have an order, but 1a / 1b / 1c are not ordered — they're orthogonal capability surfaces (and 1a is computed via 1b, so they're entangled too). Two runners scoring the same fixture could legitimately disagree on which subset is "highest."

I see this is already noted as deferred (NOTE 5 in 7206e10, and called out as a D2 risk). Reasonable to defer, but the fix is genuinely one line and removes the ambiguity before any scoring run hits it:

Suggested edit to lens 2:

If the agent cited evidence from multiple Tier-1 subsets, mark the subset whose evidence appeared in the verdict sentence as primary, the rest as supporting. Tier ordering (0 < 1 < 2) applies across tiers; within Tier 1, primary/supporting replaces "highest."

If you'd rather defer the wording until the first scoring run produces real evidence of the ambiguity, that's also defensible — just wanted to surface that the cheap fix exists now.

---

Not blocking the PR. #1 is the only one I'd consider a pre-merge ask, and even that is "land in this PR or a follow-up before DRC-3405 starts."

[even-wei]

Thanks for the cross-read — folding two of three in this PR; the third is being addressed structurally in PR #29.

#1 — Same-model contract: folded in 0fb378a.

Added a "Same-model contract" paragraph to the lens-3 frozen-baseline section (next to the existing model-variance note, since that's the constraint it extends):

The with-Recce run MUST use the same agent + model as the frozen Tier-0 baseline (the Agent and Model fields in templates/tier-0-baseline.md). If a model upgrade lands mid-eval, either re-capture the baseline (and re-commit it) or record the mismatch in the per-fixture artifact's Notes section and treat that fixture's delta as confounded. Without this constraint, the lens-3 delta conflates Recce signal with model drift.

This wires to the Agent / Model fields already in templates/tier-0-baseline.md, and the "record under Notes / treat as confounded" path lets DRC-3405 continue mid-stream without invalidating earlier fixtures.

#2 — Tier-0 cross-fixture leakage via shared object DB: being folded in PR #29 structurally, not here.

Agree the build-side fix is the right one — a ban list the agent can git log --all past is not a contract. PR #29's build_fixtures.sh is switching from git worktree add to git clone --depth 1 per fixture, so each .tmp/sources/<id>/ ends up with its own object database containing only the head SHA's history. After that lands, git log --all / git rev-parse origin/pr/<n> / git show <other-sha> are structurally impossible from inside a fixture, regardless of what the rubric says. Not adding the ban list to RUBRIC.md in this PR — the filesystem will enforce it, and a brittle ban list would just rot.

#3 — Tier-1 primary/supporting wording: folded in 0fb378a.

Reworded lens 2. Old phrasing "record the highest decisive tier" collapsed into one rule something that has two regimes; split that out:

If the agent cited evidence from multiple tiers, record the higher tier as decisive (tier ordering 0 < 1 < 2 applies across tiers) and list secondary citations under Notes. Within Tier 1, the subsets 1a / 1b / 1c are not ordered — they are orthogonal capability surfaces (and 1a is computed via 1b, so they're entangled). If the agent cited evidence from multiple Tier-1 subsets, mark the subset whose evidence appeared in the verdict sentence as primary, the rest as supporting; primary/supporting replaces "highest" within Tier 1.

Kept the cross-tier ordering rule explicit so runners don't second-guess the 0 < 1 < 2 direction.

[even-wei]

Re: point #2 — Tier-0 runtime contract leak via shared object DB.

Folded into PR #29 as commit ae28f37. Replying here since this thread is where the discussion lives; will keep #29's PR description focused on the original DRC-3402 scope.

What changed — replaced the per-fixture git worktree add off the shared .tmp/jaffle_shop_golden/ cache with a fresh standalone repo per fixture:

git -C "${source_dir}" init
git -C "${source_dir}" fetch --quiet --depth 1 --no-tags \
    "${JSG_DIR}" "+${full_head}:refs/fixture/head"
git -C "${source_dir}" checkout --quiet --detach refs/fixture/head

No git remote add, no refspec other than the one head SHA — the resulting .git carries one commit, zero remotes, and no other refs. Cache repo gets uploadpack.allowAnySHA1InWant=true once so the SHA-targeted fetch is allowed.

Built-in sanity check — the script now exits non-zero if git rev-list --all HEAD --count inside .tmp/sources/<slug>/ returns anything other than 1. Re-runs cleanly on all six fixtures; adversarial probe of pr1-fix-clv confirms the leak surface is sealed:

Probe	Before (worktree)	After (standalone clone)
git log --all	shows full branch + all PR heads	shows only 2abf877 PR1
git show 297eb54 (sibling PR head)	succeeds	fatal: ambiguous argument '297eb54'
git rev-parse origin/pr/20	resolves	fatal: ambiguous argument 'origin/pr/20'
git show 23b96ca (PR #20 intermediate)	succeeds	fatal: ambiguous argument '23b96ca'
git fetch	pulls from origin	no-op (no remote configured)

Cold-start build: 2:48 (was ~2:40). Idempotent: rebuild run 2:39. Auth pre-flight preserved. Orphan-worktree pruning removed (no worktrees anymore).

PR #20 intermediate-commit replay (23b96ca) still works — that path only ever wrote artifacts to <artifacts>/intermediate-commit-23b96ca/ via the shared cache (build-script scratch, not agent-readable); it never populated .tmp/sources/<slug>/.

Re: points #1 and #3 — being folded separately (#1 into the RUBRIC same-model contract by another teammate on PR #28; #3 will follow once we see how it lands).

[wcchang1115]

Fixture / test-workflow integration check (HEAD: d60295d) — Verdict: GO

Targeted re-read at your request: does build_fixtures.sh produce what RUBRIC.md + the per-fixture templates expect? Yes. Verified the contract end-to-end by running the build locally and probing the produced artifacts. One pre-existing NOTE worth folding into the rubric before DRC-3405 starts; not blocking.

Build produces the contract RUBRIC.md describes

Ran ./build_fixtures.sh cold-start on this branch (d60295d). All six fixtures OK. For each fixture under evals/agent-blind-spots/fixtures/<slug>/:

Contract item (RUBRIC.md:53–59)	Produced?	How
diff.patch	✓	committed
artifacts/manifest-{before,after}.json	✓	build_at_sha, lines 168, scrubbed
artifacts/compiled-{before,after}/	✓	build_at_sha, lines 171–175
artifacts/catalog-{before,after}.json	✓	build_at_sha, line 177
evals/agent-blind-spots/.tmp/sources/<slug>/ head-SHA tree	✓	build_fixture per-fixture git init + +<sha>:refs/fixture/head, line 234–248

Spot-checks:

• metadata.user_id / metadata.invocation_id in manifest-after.json → "redacted" (scrubber works).
• compiled-after/jaffle_shop/models/... tree shape matches expectation (model SQL + schema.yml-derived test SQL).
• pr44-promotion-flags/artifacts/intermediate-commit-23b96ca/ present, as required by that fixture's two-scenarios design.
• All six per-fixture tier-0-baseline.md Inputs lines correctly resolve to ../../.tmp/sources/<actual-slug>/ after the build runs.

Tier-0 leak surface is structurally sealed

The hardest part of the contract (RUBRIC.md:59 — "do NOT read from .tmp/jaffle_shop_golden/ because the build loop leaves it at the last fixture's SHA") is enforced by the build, not by the agent's good behavior. The fresh-clone approach in build_fixtures.sh:240–248 (git init + single-SHA git fetch --depth 1 --no-tags, no git remote add) plus the runtime assertion at build_fixtures.sh:253–258 (git rev-list --all HEAD --count == 1) caught every fixture cleanly. Adversarial probe of pr1-fix-clv's .tmp/sources/pr1-fix-clv/:

Probe	Result
git log --all --oneline	2abf877 PR1 (1 commit) ✓
git show 297eb54 (pr46 head)	fatal: ambiguous argument '297eb54' ✓
git show 23b96ca (pr44 intermediate)	fatal: ambiguous argument '23b96ca' ✓
git rev-parse origin/pr/20	fatal: ambiguous argument ✓
git remote -v	empty ✓

Worth calling out as a positive — the rev-list --all HEAD --count invariant in build_fixtures.sh:254 makes the contract structurally testable at build time. This is the kind of runtime assertion that makes the workflow trustworthy.

NOTE — Contract integrity gap on agent file-access

This is the one rough edge worth surfacing now rather than discovering during DRC-3405.

The Tier-0 runtime contract (RUBRIC.md:53–63) enumerates what inputs the agent gets, but it does not constrain what the agent can also see in the working tree. An eval runner who invokes Claude Code (or any agent with default workspace read access) from the repo root will give the agent read access to:

• evals/agent-blind-spots/fixtures/<slug>/README.md — which contains an explicit ## Expected agent verdict — Tier 0 section ("Likely catch quality: partial", etc.). A diligent agent that greps for context will find these and bias toward them.
• evals/agent-blind-spots/RUBRIC.md — which describes the rubric the agent is being scored against.
• Sibling fixtures' files — for cross-comparison.

RUBRIC.md:70 acknowledges that file access varies ("Agent-specific scaffolding (file-access mode, tool whitelisting, system prompt) is allowed but must be recorded in the baseline's Notes section") — but it doesn't require the runner to sandbox the agent away from spoilers. A naive runner produces silently-contaminated baselines that look identical to clean ones.

Suggested addition to the Tier-0 runtime contract, near line 60 ("Generic tools allowed at Tier 0"):

Agent view restriction (eval-runner contract). The eval runner MUST restrict the agent's file-read access to: fixtures/<slug>/diff.patch, fixtures/<slug>/artifacts/, and .tmp/sources/<slug>/. The agent MUST NOT be able to read fixtures/<slug>/README.md (contains expected-verdict spoilers), RUBRIC.md, other fixtures' files, or anywhere else in this repository. How this is enforced is runner-specific (chroot, tool-whitelist, separate working directory, sandbox policy); record the mechanism in the baseline's Notes section.

The fix is a paragraph in RUBRIC.md, not a code change. Folding it before DRC-3405 starts costs less than retroactively quarantining a baseline run.

Verification summary

Check	Result
Build runs cleanly cold-start	✓ (~3 min)
All six fixtures OK	✓
Contract paths match build output	✓
Tier-0 leak check (rev-list --all HEAD --count == 1)	✓ for all six
Adversarial probe against sibling SHAs from inside .tmp/sources/pr1-fix-clv/	✓ all sealed
Manifest scrubbing applied	✓
Per-fixture template Inputs paths resolve correctly	✓ all six

Recommendation

Ready to merge. The fixture build and rubric/templates are contract-compatible and the Tier-0 leak surface is structurally sealed. The agent-view restriction NOTE is a follow-up that should land before the first DRC-3405 run — either in this PR or a quick follow-up to RUBRIC.md.

[wcchang1115]

Fixture build + rubric/templates contract verified end-to-end on d60295d — see follow-up comment. One non-blocking NOTE on agent-view restriction worth folding into RUBRIC before DRC-3405.