chore(evals): smoke baseline preview — 4 cells on pr1-fix-clv

[even-wei]

Stacked on #36. Validation preview of the spike driver landed in #37 (now squash-merged as 1d3c95d on feature/drc-3584-sandbox-profiles). Not DRC-3405 — 3405 wants 6 fixtures, DRC-3585 manual baselines, and a hand-curated gap report. This is just the baseline run that proves the pipeline works end-to-end.

What's in here

runs/2026-05-29/spike-driver/
├── verdicts.csv     ← 4 cells × 2 judge passes (11 columns)
└── summary.md       ← rendered matrix + judge self-consistency

No transcripts (gitignored), no cells.json (it has absolute paths from this run — regenerable).

Headline results

Cell	Catch	Evidence	Delta	Status
claude · tier-0	catch	0	same	ok
claude · tier-1	catch	1c	improvement	ok
codex · tier-0	miss	0	same	rc=2 (driver bug)
codex · tier-1	miss	0	same	rc=2 (driver bug)

Claude × Tier-0 → Tier-1: evidence shifted from 0 to 1c, delta to improvement. That's exactly the lens-3 counterfactual signal the project is trying to measure — the Recce-equipped agent reached for structured 1c-tier evidence (current-env queries) that Tier-0 couldn't produce. Real signal on this fixture.

Judge self-consistency (DRC-3586 acceptance #2 — proxy)

Axis	Agreement	Vs 80% bar
catch	100% (4/4)	✓ above
tier	100% (4/4)	✓ above
delta	75% (3/4)	✗ just below

The single delta flip (claude × tier-0: same → improvement across two judge passes) is expected without a DRC-3585 manual baseline. The driver currently passes baseline_catch="unknown" to the judge; without an anchor, the judge has to guess what delta means. Plug --baseline-dir in once 3585 produces baselines and re-judge — delta should stabilize.

--baseline-dir was added in the same PR exactly for this case.

Four findings to action separately

1. 🚨 Tier-0 LEAK — recce.yml not stripped from pr1-fix-clv source tree

The agent at Tier-0 cited recce.yml's preset names verbatim:

"recce.yml's row_count_diff (state:modified,config.materialized:table) shows 0 rows changed."
"Same columns, same types → recce.yml's schema_diff shows nothing."

This is a Tier-0 sandbox leak that contaminates the lens-3 counterfactual. Investigation:

• build_fixtures.sh line 292 includes recce.yml in stripped_paths[]. The strip logic is correct.
• The fixture's .tmp/sources/pr1-fix-clv/recce.yml IS present at filesystem level.
• Likely cause: .tmp/sources/ was built before recce.yml was added to the strip list, and not rebuilt since.

Mitigation: re-run ( cd evals/agent-blind-spots && rm -rf .tmp/ && ./build_fixtures.sh ) before any DRC-3585 or DRC-3405 run. This is a hard requirement — without it, the eval results are contaminated.

Action: surface as follow-up against DRC-3430.

2. 🐛 Codex driver — incompatible with codex-cli 0.133.0

$ codex exec --sandbox=read-only --ask-for-approval=never --config <path> "..."
error: unexpected argument '--ask-for-approval' found

codex-cli 0.133.0 removed both --ask-for-approval=never and --config <FILE>. Both codex cells failed with rc=2 before the agent ever ran; the judge then graded the empty stderr-only transcripts as miss.

Fix shape (deferred to its own ticket — not appropriate to include in a "preview" PR):

• Use CODEX_HOME=<dir-containing-config.toml> instead of --config <file>
• Add -c approval_policy="never" for unattended approval
• Validate runner-configs/codex/tier-{0,1}/config.toml schema against codex 0.133.0 (the [sandbox] table syntax may also be deprecated)

Action: filing as new follow-up ticket linked from DRC-3586.

3. Judge stability needs DRC-3585's baseline

As above. 75% on the delta axis is in spec for "no baseline yet". Plug --baseline-dir in when 3585 lands.

4. Stale fixture worktrees are a foot-gun

If anyone re-runs the eval and .tmp/sources/ exists from a prior build, the strip list may be out of date. Adding a hard prerequisite in the spike-driver README: "Always rm -rf .tmp/ before a fresh eval run". Small follow-up commit.

Linear

• Not a Linear-ticket-closing PR. Preview only.
• Surfaces follow-up actions against DRC-3430 and DRC-3586.
• The actual DRC-3405 deliverable comes after DRC-3585 (manual baselines) is hand-graded.

🤖 Generated with Claude Code

[even-wei]

Code Review: PR #39

SHA 60e2bac · Verdict GO

This is a chore PR shipping two data artifacts (verdicts.csv, summary.md) from a smoke eval run. The PR body itself documents four findings the preview surfaced (Tier-0 recce.yml leak, codex-cli 0.133.0 driver breakage, 75% judge delta self-consistency pending DRC-3585 baselines, stale .tmp/sources/ foot-gun); those are the value of this preview and are not re-raised here. Review focused on artifact correctness, internal consistency, and PR-body-vs-artifact agreement.

Math in summary.md checks out:

• Catch agreement: 4/4 = 100% (matches CSV pass1 vs pass2: catch=catch, catch=catch, miss=miss, miss=miss).
• Tier agreement: 4/4 = 100% (0=0, 1c=1c, 0=0, 0=0).
• Delta agreement: 3/4 = 75% (the single flip is claude × tier-0: same vs improvement — matches the PR body's explanation about missing DRC-3585 baseline).

Cell matrix in summary.md reflects pass1 verdicts from verdicts.csv correctly for all four cells.

Notes

1. evals/agent-blind-spots/runs/2026-05-29/spike-driver/verdicts.csv:1 — PR body says verdicts.csv has "10 columns"; header is actually 11: fixture,agent,tier,catch,evidence_tier,delta,catch_2,tier_2,delta_2,returncode,error.
   Evidence: head -1 ... | awk -F',' '{print NF}' → 11. Descriptor drift in the PR description, not in the artifact. Worth correcting in the PR body for future readers; the artifact itself is fine.
   Pass C.

2. evals/agent-blind-spots/runs/2026-05-29/spike-driver/summary.md:15-16 — codex rows show Status: ok in summary.md, but verdicts.csv has returncode=2 for both codex cells (the driver bug the PR body calls out as finding Feedback by Kent Huang #2). A reader looking at only summary.md will miss that the codex cells were graded miss/0/same because the agent never ran, not because the agent failed to catch.
   Evidence: verdicts.csv lines 4-5 end in ,2, (returncode column); summary.md row 6 ("Cells errored: 0") treats them as non-errored. The CSV is the source of truth; the rendered summary under-reports. Consider future-renderer changes to set Status: rc=N when returncode != 0, or include the returncode column in the rendered matrix.
   Pass C.

Neither note blocks merge — this preview's job is to surface findings via the artifacts and the PR body, and it does that. The two observations above are quality nits for the next eval-driver render pass.

---

Reviewed by /claude-code-review · Passes A, C, F · 2 files / +29 lines

[even-wei]

GO (self-authored — cannot --approve). Two NOTEs in the issue comment, neither blocks: #39 (comment)

[wcchang1115]

Code Review: PR #39 (supplement)

SHA 60e2bac · Verdict GO

Builds on @even-wei's review above (2026-06-01) — I concur with both notes there (codex rc=2 vs summary.md "errored: 0 / ok"; the "10 columns" → 11 PR-body nit, now fixed in the description). One finding to add, plus a nit:

1. summary.md:18-23 — the judge self-consistency block (the DRC-3586 acceptance Feedback by Kent Huang #2 proxy) is computed over all 4 cells, including the 2 empty rc=2 codex transcripts, which trivially agree and pad the denominator. delta is 75% (3/4) over all cells but 50% (1/2) over the 2 real (rc=0) transcripts — claude × tier-0 flips same → improvement between passes (verdicts.csv:2). The matrix also renders that same flipping cell as a settled same (summary.md:13), i.e. the less-favorable arm of the one coin-flip. So the "≥80% per axis" gate is evaluated against a denominator padded with degenerate cells. The PR body honestly flags delta as the weak axis, so this is a sharpening, not a contradiction — but the headline reads stronger than the judgeable evidence supports. When DRC-3585 baselines land and codex runs for real, recompute self-consistency over rc=0 cells only. Pass C/E.

2. Nit: verdicts.csv is CRLF, summary.md is LF — mixed line endings in the same committed artifact directory, with no .gitattributes normalization.

GO — preview/smoke artifact, the raw verdicts are correct, and the one substantive gap (the rendered summary under-reporting the codex crash) is an upstream summary-generator issue already owned by the author's follow-up #2 against DRC-3586.

---

Reviewed by /claude-code-review · supplement to prior review · 2 files / +29 lines

[wcchang1115]

Claude Code Review: GO. Concur with @even-wei's prior review; data is sound. One supplement note added (judge self-consistency is 50% over the 2 real rc=0 transcripts vs 75% over all cells).