From ff6fae030c407dfdaf9423ee22045b09bce29bff Mon Sep 17 00:00:00 2001 From: even-wei Date: Fri, 29 May 2026 19:16:13 +0800 Subject: [PATCH] =?UTF-8?q?chore(evals):=20smoke=20baseline=20=E2=80=94=20?= =?UTF-8?q?4=20cells=20on=20pr1-fix-clv=20(preview=20of=20DRC-3405)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Validation preview from the Karpathy spike driver landed in #37. Runs: 1 fixture (pr1-fix-clv) x 2 agents (claude, codex) x 2 tiers (0, 1) = 4 cells, with --judge-stability (two judge passes per transcript). NOT a DRC-3405 deliverable. 3405 wants: - 6 fixtures (not 1) - Manual baselines from DRC-3585 to anchor lens-3 delta - Hand-curated gap report This commit ships verdicts.csv + summary.md only. cells.json and transcripts/ stay local (cells.json has absolute paths; transcripts/ is gitignored). ## Headline results | Cell | Catch | Evidence | Delta | Status | |---|---|---|---|---| | claude · tier-0 | catch | 0 | same | ok | | claude · tier-1 | catch | 1c | improvement | ok | | codex · tier-0 | miss | 0 | same | rc=2 (driver bug) | | codex · tier-1 | miss | 0 | same | rc=2 (driver bug) | Claude moved from Tier-0 (evidence=0, no delta) to Tier-1 (evidence=1c, improvement). The lens-3 counterfactual is producing real signal on this fixture -- Recce-equipped agent reached for structured 1c-tier evidence (current-env queries) that Tier-0 couldn't. ## Judge self-consistency - catch: 100% (4/4 cells) -> above 80% bar - tier: 100% (4/4 cells) -> above 80% bar - delta: 75% (3/4 cells) -> just below 80% bar The delta flip on claude x tier-0 (same -> improvement across the two judge passes) is expected behaviour without a DRC-3585 manual baseline: the driver passes "baseline_catch=unknown" and the judge has to guess what delta means. Once 3585 lands, plug --baseline-dir in and re-judge. ## Findings to action separately 1. Tier-0 LEAK: recce.yml is present in pr1-fix-clv's .tmp/sources/ tree. The agent cited recce.yml preset names (row_count_diff, schema_diff) in its Tier-0 reasoning. build_fixtures.sh strip list includes recce.yml (line 292) but the .tmp/sources/ directory is stale -- it was built before the strip was added. Mitigation: re-run build_fixtures.sh. (Follow-up against DRC-3430.) 2. Codex driver: codex-cli 0.133.0 removed both --ask-for-approval=never and --config . Driver invocation in run_codex() fails with rc=2 before the agent ever runs. Both codex cells show miss/0/same because the judge had nothing useful to score. Driver needs an update: CODEX_HOME-based config loading + `-c approval_policy="never"` override + a sanity check that the codex schema in our config.toml matches the installed codex version. (New ticket needed.) 3. Stale fixture worktrees: more broadly, if anyone re-runs the eval they should `rm -rf .tmp/` first to guarantee a fresh strip. Will document in the spike-driver README in a follow-up. Signed-off-by: even-wei --- .../runs/2026-05-29/spike-driver/summary.md | 24 +++++++++++++++++++ .../runs/2026-05-29/spike-driver/verdicts.csv | 5 ++++ 2 files changed, 29 insertions(+) create mode 100644 evals/agent-blind-spots/runs/2026-05-29/spike-driver/summary.md create mode 100644 evals/agent-blind-spots/runs/2026-05-29/spike-driver/verdicts.csv diff --git a/evals/agent-blind-spots/runs/2026-05-29/spike-driver/summary.md b/evals/agent-blind-spots/runs/2026-05-29/spike-driver/summary.md new file mode 100644 index 0000000..9c2bc9e --- /dev/null +++ b/evals/agent-blind-spots/runs/2026-05-29/spike-driver/summary.md @@ -0,0 +1,24 @@ +# Spike driver run — summary + +- Run date: 2026-05-29 +- Cells attempted: 4 +- Cells with transcript: 4 +- Cells judged: 4 +- Cells errored: 0 + +## Cell matrix + +| Fixture | Agent | Tier | Catch | Evidence | Delta | Status | +|---|---|---|---|---|---|---| +| pr1-fix-clv | claude | 0 | catch | 0 | same | ok | +| pr1-fix-clv | claude | 1 | catch | 1c | improvement | ok | +| pr1-fix-clv | codex | 0 | miss | 0 | same | ok | +| pr1-fix-clv | codex | 1 | miss | 0 | same | ok | + +## Judge self-consistency (two passes on the same transcript) + +- **catch**: 100% (4/4 double-judged cells) +- **tier**: 100% (4/4 double-judged cells) +- **delta**: 75% (3/4 double-judged cells) + +**Stability bar:** ≥80% per axis. Below that, the judge can't replace human grading at N=6. diff --git a/evals/agent-blind-spots/runs/2026-05-29/spike-driver/verdicts.csv b/evals/agent-blind-spots/runs/2026-05-29/spike-driver/verdicts.csv new file mode 100644 index 0000000..d1187b1 --- /dev/null +++ b/evals/agent-blind-spots/runs/2026-05-29/spike-driver/verdicts.csv @@ -0,0 +1,5 @@ +fixture,agent,tier,catch,evidence_tier,delta,catch_2,tier_2,delta_2,returncode,error +pr1-fix-clv,claude,0,catch,0,same,catch,0,improvement,0, +pr1-fix-clv,claude,1,catch,1c,improvement,catch,1c,improvement,0, +pr1-fix-clv,codex,0,miss,0,same,miss,0,same,2, +pr1-fix-clv,codex,1,miss,0,same,miss,0,same,2,