feat(evals): Karpathy spike driver for /recce-verify v1 eval (DRC-3586) - #37
Merged
even-wei merged 4 commits intoMay 29, 2026
Merged
Conversation
added 3 commits
May 29, 2026 17:34
Single-file Python driver at evals/agent-blind-spots/spike-driver/. Dispatches
up to 6 fixtures x 2 agents x 2 tiers = 24 cells, captures each agent's
transcript, runs a Claude-as-judge pass per transcript to produce a three-axis
verdict (catch / tier / delta), and writes a CSV + Markdown summary under
runs/<date>/spike-driver/.
Stability checks (both optional, both produce judge-quality signal):
- --judge-stability: double-judges each transcript, reports per-axis
self-consistency (catch / tier / delta). Floor 80%.
- --baseline-dir <path>: compares judge verdicts against DRC-3585 manual
baseline once it lands (judge-vs-human catch agreement on Tier-0 cells).
Graceful degradation:
- Skips an agent's cells if its CLI is not on PATH; codex commonly absent
in lighter dev envs (recipe in runner-configs/codex/tier-{0,1}/README.md).
- --no-run mode re-judges existing transcripts without re-running agents.
Sandbox profile integration (from DRC-3584):
- Claude Code cells stamp tier-N claude-overlay/ into the fixture worktree
before invoking `claude --print`, with a neutered CLAUDE_CONFIG_DIR per
cell and scrubbed warehouse env.
- Codex cells use `codex exec --sandbox=<read-only|workspace-write>
--ask-for-approval=never --config <tier-N/config.toml>` plus PATH scrub.
Non-goals for this spike (deferred):
- Durable harness with resume / parallel dispatch -> DRC-3587 Inspect AI port.
- Gap-report generator across 6 fixtures -> DRC-3405.
- Auto-iteration on prompt / skill changes (closed-loop optimization overfits
at N=6 -- explicit non-goal of the spike).
Stacks on PR #36 (DRC-3584 + DRC-3430). When #36 merges to main, this PR
should be rebased onto main; GitHub auto-retargets the base branch.
Signed-off-by: even-wei <evenwei@infuseai.io>
The agent's cwd is .tmp/sources/<id>/ (the per-fixture standalone repo),
but the frozen Tier-0 inputs (diff.patch, artifacts/{manifest, compiled,
catalog}*) live at fixtures/<id>/, outside cwd. Without staging:
- Claude Code reaches them via absolute paths (Read tool isn't cwd-anchored),
but the prompt doesn't tell the agent where to look.
- Codex Tier-0 read-only sandbox blocks reads outside cwd entirely, so the
agent literally cannot reach the artifacts.
Driver now symlinks fixtures/<id>/{diff.patch, artifacts} into a
_eval_inputs/ subdir under cwd before each cell. Prompt updated to point
at _eval_inputs/. Works under both agent sandboxes.
Verified locally:
>>> stage_inputs(SOURCES_DIR / 'pr1-fix-clv', 'pr1-fix-clv')
>>> sorted(p.name for p in (SOURCES_DIR / 'pr1-fix-clv' / '_eval_inputs').iterdir())
['artifacts', 'diff.patch']
Signed-off-by: even-wei <evenwei@infuseai.io>
ENFORCEMENT.md recipe step 3 sets CLAUDE_CONFIG_DIR to a fresh mktemp dir
to neuter user-level ~/.claude/settings.json. That step also strips the
Claude Code auth state, so a child `claude --print` invoked by the spike
driver fails with "Not logged in" and no transcript ever lands.
For unattended runs, the load-bearing enforcement is the project-level
.claude/settings.json overlay (stamped per cell) plus the PreToolUse hook
(deny-tier-N.py); a user-level `permissions.allow` cannot bypass an
exit-2 hook. Skip the CLAUDE_CONFIG_DIR override by default.
For paranoid mode (e.g. when running the eval on a machine with risky
~/.claude/ contents), set RECCE_EVAL_STRICT_CONFIG=1 to enable the
override. The operator is responsible for preseeding auth under the
per-cell _claude_cfg dir.
Verified end-to-end smoke run after this fix:
uv run driver.py --smoke --agents claude --tiers 0
[run] pr1-fix-clv · claude · tier-0
[judge] pr1-fix-clv · claude · tier-0
Agent VERDICT: catch · request-changes
Judge verdict: {catch, tier-0, same} (delta=same expected with no T1 paired)
Signed-off-by: even-wei <evenwei@infuseai.io>
Author
Smoke validation — first end-to-end run produces a verdict ✓Ran the driver from this session against the worktree's prebuilt fixtures (symlinked from the sibling DRC-3584 worktree's Agent transcript ending: Judge verdict: {"catch": "catch", "tier": "0", "delta": "same"}Driver returned 0 across the cell. Three driver gaps surfaced and were fixed inline:
What this proves vs. what's still open
Recommended next runs (operator)# Codex smoke under Tier-0 sandbox (verifies acceptance #3)
uv run evals/agent-blind-spots/spike-driver/driver.py --smoke --agents codex --tiers 0
# Full smoke matrix (4 cells: 1 fixture × 2 agents × 2 tiers)
uv run evals/agent-blind-spots/spike-driver/driver.py --smoke --judge-stability
# Full 24-cell run after DRC-3585's baseline lands
uv run evals/agent-blind-spots/spike-driver/driver.py \
--baseline-dir evals/agent-blind-spots/fixtures/The agent's first transcript is at |
Author
Code Review: PR #37 (iter-2)SHA Reviewed only commit Verification of iter-1 fixes
Quality checks
Notes
This iter-2 confirms iter-1 fixes resolved the original 3 ISSUEs. Ready to merge once the base PR #36 lands. |
even-wei
commented
May 29, 2026
even-wei
left a comment
even-wei
left a comment
Author
There was a problem hiding this comment.
NO-GO at 95a1bee — 3 issues, 4 notes. Full review: #37 (comment)
1. Drop dead VERDICT_TAIL_RE regex. Defined in driver.py:99 but never
used; agent verdict parsing is fully handled by judge_cell()
downstream. Removed alongside the regex line.
2. Add surgical .gitignore entries for spike-driver runtime outputs:
evals/agent-blind-spots/runs/*/spike-driver/transcripts/
evals/agent-blind-spots/runs/*/spike-driver/_claude_cfg/
These are the genuinely-transient subdirs. verdicts.csv,
summary.md, cells.json under runs/<date>/spike-driver/ are
intentionally NOT ignored — they may be committed manually as
run baselines for cross-iteration tracking.
3. stage_inputs() now raises FileNotFoundError when no Tier-0
inputs (neither diff.patch nor artifacts/) could be staged under
the fixture's source dir. run_cell() catches and records into
cell.error so the cell shows up as failed in the matrix instead
of silently feeding the agent an empty _eval_inputs/ and letting
it review with no inputs.
Verified:
python3 -m py_compile spike-driver/driver.py -> ok
stage_inputs(.../pr1-fix-clv, 'pr1-fix-clv') -> happy path
stage_inputs(/tmp/dne, 'does-not-exist') -> raises FileNotFoundError
NOTE-level findings 4-7 from the review are accepted as-is per the
spike framing (judge inherits parent env intentionally for auth;
same-day run dirs are an operator concern; broken-symlink in
overlay copy is rare and surfaced via run_cell's general except;
no tests is consistent with the Karpathy-spike scope and the
DRC-3587 follow-up which adds Inspect AI test harness).
Signed-off-by: even-wei <evenwei@infuseai.io>
Author
pr-cycle iteration 1 — review addressedThanks for the clean pass. All three ISSUEs fixed in ISSUEs — fixed
NOTEs — accepted as-is
Quality checks
Closing out iteration 1. Ready for next pass or merge. |
This was referenced May 29, 2026
even-wei
pushed a commit
that referenced
this pull request
Jun 3, 2026
…3584, DRC-3430) (#36) * feat(evals): Tier-0/Tier-1 sandbox profiles + Recce-aware strip (DRC-3584, DRC-3430) Make the /recce-verify v1 eval's Tier-0 / Tier-1 split enforceable rather than aspirational, so the lens-3 counterfactual delta (Tier-0 verdict → Tier-1 verdict) actually isolates Recce's contribution. DRC-3584 — sandbox profile templates under runner-configs/: - Claude Code: per-tier .claude/{settings.json, hooks/deny-tier-N.sh}. permissions.deny + a PreToolUse hook (belt-and-suspenders per open Claude Code issue #6699). Tier 0 denies Recce CLI, Recce MCP namespaces (mcp__recce__*, mcp__plugin_recce_*), /recce-* skills, dbt subcommands that regenerate frozen Tier-0 inputs, and direct SQL clients. Tier 1 allows Recce CLI + MCP but keeps dbt regen and raw SQL clients denied. - Codex: per-tier README + config.toml. Process sandbox flag (--sandbox=read-only / workspace-write) + an empty (Tier 0) or templated (Tier 1) mcp_servers table + a PATH scrub recipe to drop recce/dbt binaries. - ENFORCEMENT.md: per-agent recipes, the agent-view-restriction folded from PR #28 follow-up, what's mechanically enforced vs. contract-only, and the recording-in-baseline requirement. - RUBRIC.md: adds the agent-view-restriction paragraph pointing at ENFORCEMENT.md. - templates/tier-0-baseline.md: adds the required "Sandbox profile used" block in Notes; missing block disqualifies the baseline. - runs/2026-05-28/sandbox-verification.md: synthetic hook unit tests (Tier-0 6 cells, Tier-1 4 cells) all pass; one end-to-end claude --print × Tier-0 confirms the PreToolUse hook fires on a real agent attempt at `recce check`. CC × Tier-1, Codex × Tier-0, Codex × Tier-1 cells deferred to the operator with recipes. DRC-3430 (bundled because the DRC-3584 verify step is contaminated without it) — build_fixtures.sh: - Strip .github/prompts/, .github/workflows/recce-*.yml, .github/workflows/claude.yml, recce.yml from each per-fixture source tree right after the head-SHA checkout. - Belt-and-suspenders post-strip grep for mcp__recce__|recce.yml|RECCE_API_TOKEN with --exclude-dir=.git. - Initial run surfaced claude.yml (a "Claude Code + Recce MCP" reviewer workflow that wasn't on the original strip list); now stripped. All six fixtures rebuild cleanly with empty post-strip grep. - fixtures/README.md documents the strip list and rationale. gitignore exception added so the runner-configs/**/.claude/ templates can be committed under the existing repo-wide .claude/ ignore. Signed-off-by: even-wei <evenwei@infuseai.io> * fix(evals): address PR #36 review — Python hooks + expanded strip (DRC-3584, DRC-3430) Review found 6 BLOCKERs + 4 ISSUEs + 3 NOTEs against the v1 sandbox profiles. Every blocker was an exit-0 from the case-glob hook on an input the rubric said must be exit-2. This commit collapses them. Hook rewrite — Python with shlex tokenization (B1+B2+B3+B4+B5+I10): - New `deny-tier-{0,1}.py` (replacing `.sh`). Splits commands on shell metacharacters (`;`, `&`, `|`, `()`, newline) plus `$()` and backtick substitutions, shlex-parses each segment, basenames the executable (so `/usr/local/bin/recce` is caught), and recurses into `sh -c` / `bash -lc` / `zsh -c` arguments. Skill match is now case-insensitive. MCP namespace regex broadened to `^mcp__(plugin_)?recce(_|-|$)` so `mcp__recce_dev__*` and future variants are covered. - Tier-0 flipped to a positive allowlist (git/grep/jq/file-read + POSIX text utilities). The legitimate Tier-0 surface is tiny per RUBRIC.md; allowlist is safer than chasing bypass shapes. - Tier-1 stays a denylist but the matching is principled. dbt's banned subcommands are detected anywhere after `dbt`, so `dbt --debug parse` and `dbt --profiles-dir /tmp parse` are caught. - `runs/2026-05-28/sandbox-verification.md` grew a Bypass attempts table with every reviewer shape; all rows pass exit-2 against v2. ENFORCEMENT.md fix (B6): - Corrected the false claim that the hook blocks `Bash(cat ../../<host-path>)`. The hook does not gate Read/Grep/Glob and `cat` is in the Tier-0 allowlist. Spoiler-path protection comes from cwd separation alone — the recipe's `cd "${FIXTURE_DIR}"` step is therefore not optional. Document says so now. - Reframed the `permissions.deny` + hook relationship: the hook is the load-bearing layer, not a backup. `permissions.deny` is documentation that mirrors the documented surface, not the full bypass coverage. Gitignore restructure (I8): - Renamed `runner-configs/claude-code/tier-{0,1}/.claude/` → `claude-overlay/`. Per gitignore docs, you can't re-include a file under an excluded directory, so the v1 broad un-ignore would have let arbitrary files under `.claude/` slip through. Renaming sidesteps the `.claude/` ignore rule entirely; the eval runner now copies `claude-overlay/` → `.claude/` (one extra arg to `cp -r`). Gitignore exception removed. Strip + leak-grep expansion (I9+N11): - Strip list grew with `.devcontainer/`, `.github/mcp_config.json`, `.github/workflows/{recce_*.yml, recce_*.yaml, dbt_base.yml, dbt-build-pr.yml, dbt-build-base.yml}` — surfaced by the broadened leak grep on the first re-build. - Leak grep broadened from `mcp__recce__|recce\.yml|RECCE_API_TOKEN` to case-insensitive `mcp__recce|recce\.yml|RECCE_API_TOKEN|recce` with `profiles.yml` whitelisted (`role: RECCE` is the Snowflake role name, not Recce-the-tool priming, and dbt needs the file). - `printf ' %s\n' ${leak_hits}` (unquoted) replaced with `sed 's/^/ /' <<< "${leak_hits}"` — robust against whitespace in filenames. Other doc fixes (N12, N13): - Tier-1 hook header notes Recce MCP is intentionally not gated. - `runner-configs/README.md` calls out that `permissions.deny` and the hook now intentionally cover different surfaces. - Maintenance section updated for the v2 layout. Build verified end-to-end after every change. The Bypass attempts table in `runs/2026-05-28/sandbox-verification.md` covers all 16 hook unit tests plus 1 end-to-end claude --print cell — every row green. Signed-off-by: even-wei <evenwei@infuseai.io> * fix(evals): close cycle-review bypasses (Tier-1 dbt flag-with-value + exec wrappers) The pr-cycle reviewer found 2 new BLOCKERs in v2 that the prior review missed. Both are static hook-logic bugs; v3 fixes them with two small changes plus a regex tweak. BLOCKER 1 — Tier-1 dbt-flag-with-value bypass (`deny-tier-1.py`): `has_denied_dbt_subcommand` skipped `-`-prefixed tokens but never consumed the value following a value-bearing flag, so the value became the "first positional" and the real subcommand was never inspected. Eight verified shapes: dbt --target dev parse → was exit 0, now exit 2 dbt --profiles-dir /tmp parse → was exit 0, now exit 2 dbt --project-dir /x run → was exit 0, now exit 2 dbt --vars "x: 1" parse → was exit 0, now exit 2 dbt --log-format json parse → was exit 0, now exit 2 dbt --log-level debug parse → was exit 0, now exit 2 dbt --printer-width 80 parse → was exit 0, now exit 2 dbt --profile myprof parse → was exit 0, now exit 2 Fix: scan EVERY token after `dbt` for a banned subcommand, not just the "first positional". False positives require the agent to pass a literal banned-subcommand name as a flag value (e.g. `--target parse`), which is perverse and would deserve denial anyway since Tier-1 warehouse access is supposed to be mediated through Recce MCP. Also expanded DBT_DENIED_SUBCOMMANDS with `run-operation`, `debug`, `source` — all hit the warehouse and were missing from the prior list. BLOCKER 2 — exec-wrapper bypass (both hooks): `xargs recce` / `find -exec recce` / `time recce` / `nohup recce` spawned a denied binary as a child process; the hook only inspected the wrapper's head token and let the call through. Tier-0's positive allowlist let `xargs` and `find` pass; Tier-1's denylist saw a non-DENIED_BINS head and stopped looking. Fix: new EXEC_WRAPPERS set covers xargs, time, nice, nohup, setsid, parallel, exec, timeout, watch, ionice, chrt, stdbuf plus `find` (which needs special handling because its wrapped command lives after `-exec`/`-execdir`). For each wrapper at the head of a segment, walk the args for the first executable-shaped token and recurse-check from there. Heuristic `looks_like_executable` skips numeric positionals (timeout duration, chrt priority), flags, and placeholders (`{}`, `[]`). Regex fix (split_segments): `(?<!\\)[;&|()\n]+` — negative lookbehind on backslash so escaped separators (`find -exec ... \;`) stay attached to the same segment. Without this, `find -exec dbt parse \;` would split at the `;` and shlex would error on the trailing backslash; the malformed-payload path would silently allow. Verification: All v3 cells exit-2 in red-team testing. Bypass attempts table in `runs/2026-05-28/sandbox-verification.md` grew from 16 to 38 rows covering every reviewer-named bypass (B1-B6, I7, cycle B1, cycle B2) plus counter-claim regressions (Recce/MCP/dbt-list allowed at T1, benign xargs/find at T0). Build re-verified across all 6 fixtures — the broader leak grep + expanded strip still pass clean. Closes the cycle review's BLOCKER 1 and BLOCKER 2. Addresses ISSUE 3 (per-tier disambiguation now in the table) and ISSUE 4 (collapsed by BLOCKER 1's fix). Signed-off-by: even-wei <evenwei@infuseai.io> * fix(evals): close iter-2 bypasses (eval / \$() smuggling / dbt clone+retry) The pr-cycle iter-2 review surfaced three more bypass classes at Tier-1 — all real, none caught by v3. v4 closes them. 1. `eval` shell-builtin smuggling `eval dbt run` / `eval "dbt run"` were exit 0 because `eval` was not in SHELL_WRAPPERS. Fix: added `eval` to Tier-0's SHELL_WRAPPERS (banned outright since the positive allowlist has no legit reason for a shell) and added a dedicated `eval ARGS...` branch in Tier-1's check_tokens that joins the args and recurses into the resulting command — same threat surface as `sh -c`, different syntactic shape. 2. `$()` / backtick substitution at command-head position `$(echo dbt) run`, `` `echo dbt` run ``, `$(printf %s dbt) parse` evaluate to `dbt run` / `dbt parse` at bash runtime; the substitution provides the head binary, which the static check_tokens path can't see because the substitution's content has been stripped from the outer segment. Fix: new `check_substitution_at_head` in Tier-1 pre-walks each segment for `$(...)` / backtick at head and, if the substitution payload contains a denied binary name, denies the outer command. For dbt, it also combines payload tail + outer args to decide whether a denied subcommand is present (so `$(echo dbt)` alone is still allowed — bare dbt is discovery-only). Segmentation for this pre-check uses `(?<!\\)[;&|\n]+` (no parens) so `$(...)` stays intact. The main split_segments still includes parens because it pre-extracts substitutions before splitting. 3. Missing dbt subcommands `clone` and `retry` `dbt clone` (dbt-core ≥1.6) materialises cloned models in the warehouse. `dbt retry` re-executes the prior failed command, so it inherits any deny semantics of that prior command. Both added to DBT_DENIED_SUBCOMMANDS. False-positive sanity (all exit 0): git log --grep=\$(echo psql) — $() at non-head position echo \$(date) — benign substitution \$(which python) script.py — non-denied binary \$(echo grep) -rn foo . — non-denied binary \$(echo dbt) — dbt alone (allowed) echo \$(echo dbt) is good — dbt in non-head $() All v3 / v2 regressions still pass. Bypass attempts table in `runs/2026-05-28/sandbox-verification.md` grew with the iter-2 cells plus the new false-positive counter-claims. Build re-verified across all 6 fixtures. Signed-off-by: even-wei <evenwei@infuseai.io> * refactor(evals): hooks → bashlex AST parser (DRC-3584 v5, closes iter-3 bypasses) The pr-cycle iter-3 review surfaced 9 more bypass classes that the shlex+regex hooks couldn't model — all real, all rooted in Bash semantics shlex doesn't represent. Iter-3 BLOCKERs: 1. Nested $() — `$(sh -c "$(echo dbt) run")`, `$($(echo dbt) run)`, `` `$(echo dbt) run` `` (single-level regex misses nesting) 2. `eval $(...)` — substitution stripped before eval branch ran 3. `xargs $(echo dbt)` — wrapper-at-head + substitution-as-arg; find_wrapped_command saw no executable-shaped token 4. `dbt $(echo run)` — substitution supplied the subcommand; has_denied_dbt_subcommand only scanned outer literal tokens 5. Bash ANSI-C $'dbt' / parameter expansion ${a:-dbt} — shlex passed through as opaque basenames that didn't equal 'dbt' 6. `coproc dbt run` — Bash keyword not in any wrapper set 7. `command dbt run` / `builtin eval dbt run` — Bash command modifiers not handled 8. `! dbt run` — Bash logical-not prefix not matched 9. `xargs -I {} sh -c "{} parse" dbt` — xargs substitutes dbt into {} at runtime; hook saw only the literal {} Rather than patch each class with another regex, this commit **replaces the regex/shlex approach with `bashlex` (a real Bash AST parser)**. Substitutions are walked as a tree; ANSI-C / parameter expansion are extracted from their AST nodes; Bash keywords (!, command, builtin) are handled as transparent prefixes; `coproc` is denied outright (bashlex can't represent it). New dependency: `bashlex` (pure Python, MIT, well-maintained). Both hooks fail closed if bashlex isn't installed — better to deny a legit call than silently allow a bypass. ENFORCEMENT.md + runner- configs README document the install step (`python3 -m pip install bashlex`). Architecture: Two passes per command (in walk()): 1. WALK INTO every $()/backtick substitution as if it were a top-level command. Catches `$(sh -c "dbt run")` because the inner sh -c "dbt run" is itself denied. 2. RESOLVE the head word of each command to candidate output strings (literal text, ANSI-C decoded via raw-position lookup, parameter defaults, substitution payload words). At Tier 0, ALL candidates must be in the allowlist — `$(echo dbt) run` resolves to ['echo','dbt'] and 'dbt' isn't allowlisted, so deny. At Tier 1, any candidate matching a denied binary (with denied subcommand visible anywhere in args, including substitution payloads) denies. Exec wrappers (xargs/find -exec/time/nohup/etc.) now scan EVERY wrapped-command word's candidates for a denied basename — exec wrappers supply args from stdin/find-output, so "bare dbt is allowed" reasoning doesn't apply under a wrapper. Shell wrappers (sh -c, bash -lc, ...) reparse the LITERAL word text of the -c arg (not its substitution candidates) so the inner command structure stays intact. Catches nested $(sh -c "$(echo dbt) run"). Verification: 51 cells across both tiers. Every iter-1/iter-2/iter-3 BLOCKER closed. False-positive sanity for benign $() at non-head, $(echo grep), $(which python), echo $(date), command grep, find without -exec, xargs grep, etc. — all allow correctly. Build re-verified across all 6 fixtures. Closes the iter-3 9 BLOCKER classes. The reviewer's recommended follow-up (pytest-parameterized over the bypass matrix) is filed as an out-of-PR improvement. Signed-off-by: even-wei <evenwei@infuseai.io> * fix(evals): close iter-4 BLOCKERs (process sub / CompoundNode / brace / xargs-chain / \$0) Iter-4 cycle review found 7 BLOCKERs + 5 ISSUEs against v5's bashlex hooks — all rooted in AST-walker gaps inside the bashlex code (not in bashlex itself). v6 closes them. BLOCKERs: 1+7. **Process substitution `<(...)` / `>(...)`** — `_walk_substitutions` only handled `commandsubstitution` kind. Now also handles `processsubstitution` (same `.command` attribute). Catches `diff <(dbt run) /etc/hosts` (head-position) AND `echo x 2> >(dbt parse)` (process sub inside a RedirectNode — `walk()` now also iterates `kind == 'redirect'` parts and walks their `.output` word's substitutions). 2. **CompoundNode.list** — bashlex's subshell `(...)` and group `{ ...; }` store children in `.list`, not `.parts`. The walker's descent path now tries `.list` first then `.parts`, so `(dbt parse)` and `{ dbt parse; }` are no longer silently allowed. 3+4. **Brace expansion** — bashlex returns `{dbt,bash}` and `{parse,run}` as literal `word` text with no `parts`. New `_expand_brace_literal()` regex decodes `prefix{a,b,c}suffix` into candidate list (`['prefix.a.suffix', 'prefix.b.suffix', ...]`). Used by `resolve_word` for both head and args. Multi-candidate head fix: `{dbt,bash} run` resolves to `{'dbt','bash'}`. Before v6, the shell-wrapper branch fired on `bash` and returned without checking the dbt arm. Now an eager-deny up front checks dbt+args and direct DENIED_BINS BEFORE the shell-wrapper / eval / exec-wrapper branches — any candidate matching a denied shape denies regardless of what other candidates are present. 5. **xargs → sh -c chain** — `echo dbt | xargs -I {} sh -c "{} parse"` exec-wrapper found `sh` as wrapped, but `sh` isn't in DENIED_BINS so the wrapper-scan exited without escalating. v6 adds: if the wrapped binary IS a shell wrapper, find the `-c` arg; if the xargs `-I` placeholder appears inside the shell body, conservatively deny (placeholder substitution from stdin is dynamic — can't be statically resolved, and the pattern has very narrow legit use). Otherwise re-parse the literal -c text. 6. **`$0` head** — bashlex parses `$0` as ParameterNode(value='0'). `resolve_word`'s parameter branch only handled `:-`/`:=` defaults. Now also handles `0`, `BASH`, `SHELL`, `BASH_SOURCE` — these resolve at runtime to a shell name; we surface them as the candidate `"sh"` so the shell-wrapper branch fires conservatively. ISSUEs: 8. **coproc regex inside string literals** — pre-parse `\bcoproc\b` match was firing on `echo coproc`, `echo 'no coproc here'`, etc. Removed the pre-parse check; rely on bashlex's NotImplementedError raised when the actual `coproc` keyword is present. Error message is inspected for `'coproc'` to differentiate from other unsupported constructs. 9. **NotImplementedError fail-closed contradicted docstring** — bashlex raises NotImplementedError on legit `$((arith))`, `select`, etc. v5 denied all such cases. v6 only denies if the error message mentions `coproc`; other unsupported constructs fall through (the rest of the sandbox catches denied binaries). 10. **`_looks_like_executable`** — added `[[`, `]]`, `;;`, `&&`, `||` to the skip set. 11. **ANSI-C decoder** — switched from `re.sub(r"\\(.)", r"\1", inner)` to `codecs.decode(inner, 'unicode_escape')` for proper bash escape semantics. Falls back to the naive regex on decode error. 12. **Reparse fallback** — already scans every token, no change needed. Verification: 49 cells across both tiers + iter-1/2/3/4 regressions + false-positive sanity. All green. Build re-verified across all 6 fixtures. Closes the iter-4 cycle's 7 BLOCKERs and 5 ISSUEs. Signed-off-by: even-wei <evenwei@infuseai.io> * docs(evals): add non-adversarial threat-model section to ENFORCEMENT.md Document the v1 enforcement contract explicitly: hooks are calibrated for a non-adversarial code agent (RUBRIC.md's stated subject), not a red-teamer actively evading the sandbox. Catalogue the seven classes of static-Bash-analysis-unreachable bypass (bashlex NIE, heredoc-body, pipeline string flow, stdin-supplied subcommands, trap evaluation, interpreter shell-out, variable-flow loops/assignments) as accepted out-of-scope for v1, with rationale and the follow-up trigger (non-adversarial agent reaching one in practice). Reframes the iter-3/4/5 cycle-reviewer BLOCKER inflation as adversarial review against a non-adversarial artifact; closes that loop on the documentation layer rather than the code layer. Signed-off-by: even-wei <evenwei@infuseai.io> * feat(recce): opt-in PostHog telemetry scaffolding for /recce-verify (DRC-3597) (#38) Foundation for the L3 funnel signal proposed in the 2026-05-29 project rethink. Tells the project whether real-world agent users reach for /recce-verify, complete it, and convert downstream -- the production complement to the L1 offline eval (DRC-3405) and L2 in-driver trace metrics (DRC-3586). What ships: - plugins/recce/hooks/scripts/telemetry.sh Event emitter. Fires recce_verify.* events to PostHog via curl fire-and-forget. Off by default; opt in via RECCE_TELEMETRY_OPT_IN=1 or ~/.recce/config.yml `telemetry_opt_in: true`. Anonymous stable installation ID at ~/.recce/installation-id (UUID4). Failure modes are silent and non-blocking by construction. - plugins/recce/hooks/scripts/test-telemetry.sh Audit script that exercises every opt-in / opt-out short-circuit without firing network traffic. Currently 4/4 pass on default-off, opt-in + no key, opt-in + DISABLED bypass, and missing-event-name edge case. - plugins/recce/hooks/scripts/README-telemetry.md Documents the event schema (skill_invoked, tier_degraded, tool_call, verdict_emitted, session_completed), opt-in mechanism, per-event wiring recommendations, plugin-maintainer responsibilities (PostHog key), and audit script usage. What's NOT in this PR (deferred to follow-up): - Auto-wiring in hooks.json (which PostToolUse / Stop hooks fire which events; needs design decision on session-scoping) - Inline emit calls in plugins/recce/skills/recce-verify/SKILL.md - The actual PostHog project key (plugin maintainer fills in at release) - Recce Cloud signup join key (coordinate with Andy when Cloud picks up the parameter) This is independent of the eval chain (DRC-3585 / 3586 / 3587 / 3405) and can land on its own without disturbing the in-flight eval work. Signed-off-by: even-wei <evenwei@infuseai.io> * feat(evals): Karpathy spike driver for /recce-verify v1 eval (DRC-3586) (#37) * feat(evals): Karpathy spike driver for /recce-verify v1 eval (DRC-3586) Single-file Python driver at evals/agent-blind-spots/spike-driver/. Dispatches up to 6 fixtures x 2 agents x 2 tiers = 24 cells, captures each agent's transcript, runs a Claude-as-judge pass per transcript to produce a three-axis verdict (catch / tier / delta), and writes a CSV + Markdown summary under runs/<date>/spike-driver/. Stability checks (both optional, both produce judge-quality signal): - --judge-stability: double-judges each transcript, reports per-axis self-consistency (catch / tier / delta). Floor 80%. - --baseline-dir <path>: compares judge verdicts against DRC-3585 manual baseline once it lands (judge-vs-human catch agreement on Tier-0 cells). Graceful degradation: - Skips an agent's cells if its CLI is not on PATH; codex commonly absent in lighter dev envs (recipe in runner-configs/codex/tier-{0,1}/README.md). - --no-run mode re-judges existing transcripts without re-running agents. Sandbox profile integration (from DRC-3584): - Claude Code cells stamp tier-N claude-overlay/ into the fixture worktree before invoking `claude --print`, with a neutered CLAUDE_CONFIG_DIR per cell and scrubbed warehouse env. - Codex cells use `codex exec --sandbox=<read-only|workspace-write> --ask-for-approval=never --config <tier-N/config.toml>` plus PATH scrub. Non-goals for this spike (deferred): - Durable harness with resume / parallel dispatch -> DRC-3587 Inspect AI port. - Gap-report generator across 6 fixtures -> DRC-3405. - Auto-iteration on prompt / skill changes (closed-loop optimization overfits at N=6 -- explicit non-goal of the spike). Stacks on PR #36 (DRC-3584 + DRC-3430). When #36 merges to main, this PR should be rebased onto main; GitHub auto-retargets the base branch. Signed-off-by: even-wei <evenwei@infuseai.io> * fix(evals): stage frozen Tier-0 inputs into cwd before agent run The agent's cwd is .tmp/sources/<id>/ (the per-fixture standalone repo), but the frozen Tier-0 inputs (diff.patch, artifacts/{manifest, compiled, catalog}*) live at fixtures/<id>/, outside cwd. Without staging: - Claude Code reaches them via absolute paths (Read tool isn't cwd-anchored), but the prompt doesn't tell the agent where to look. - Codex Tier-0 read-only sandbox blocks reads outside cwd entirely, so the agent literally cannot reach the artifacts. Driver now symlinks fixtures/<id>/{diff.patch, artifacts} into a _eval_inputs/ subdir under cwd before each cell. Prompt updated to point at _eval_inputs/. Works under both agent sandboxes. Verified locally: >>> stage_inputs(SOURCES_DIR / 'pr1-fix-clv', 'pr1-fix-clv') >>> sorted(p.name for p in (SOURCES_DIR / 'pr1-fix-clv' / '_eval_inputs').iterdir()) ['artifacts', 'diff.patch'] Signed-off-by: even-wei <evenwei@infuseai.io> * fix(evals): inherit CLAUDE_CONFIG_DIR for auth in unattended runs ENFORCEMENT.md recipe step 3 sets CLAUDE_CONFIG_DIR to a fresh mktemp dir to neuter user-level ~/.claude/settings.json. That step also strips the Claude Code auth state, so a child `claude --print` invoked by the spike driver fails with "Not logged in" and no transcript ever lands. For unattended runs, the load-bearing enforcement is the project-level .claude/settings.json overlay (stamped per cell) plus the PreToolUse hook (deny-tier-N.py); a user-level `permissions.allow` cannot bypass an exit-2 hook. Skip the CLAUDE_CONFIG_DIR override by default. For paranoid mode (e.g. when running the eval on a machine with risky ~/.claude/ contents), set RECCE_EVAL_STRICT_CONFIG=1 to enable the override. The operator is responsible for preseeding auth under the per-cell _claude_cfg dir. Verified end-to-end smoke run after this fix: uv run driver.py --smoke --agents claude --tiers 0 [run] pr1-fix-clv · claude · tier-0 [judge] pr1-fix-clv · claude · tier-0 Agent VERDICT: catch · request-changes Judge verdict: {catch, tier-0, same} (delta=same expected with no T1 paired) Signed-off-by: even-wei <evenwei@infuseai.io> * fix(evals): address pr-cycle review of #37 — 3 ISSUEs 1. Drop dead VERDICT_TAIL_RE regex. Defined in driver.py:99 but never used; agent verdict parsing is fully handled by judge_cell() downstream. Removed alongside the regex line. 2. Add surgical .gitignore entries for spike-driver runtime outputs: evals/agent-blind-spots/runs/*/spike-driver/transcripts/ evals/agent-blind-spots/runs/*/spike-driver/_claude_cfg/ These are the genuinely-transient subdirs. verdicts.csv, summary.md, cells.json under runs/<date>/spike-driver/ are intentionally NOT ignored — they may be committed manually as run baselines for cross-iteration tracking. 3. stage_inputs() now raises FileNotFoundError when no Tier-0 inputs (neither diff.patch nor artifacts/) could be staged under the fixture's source dir. run_cell() catches and records into cell.error so the cell shows up as failed in the matrix instead of silently feeding the agent an empty _eval_inputs/ and letting it review with no inputs. Verified: python3 -m py_compile spike-driver/driver.py -> ok stage_inputs(.../pr1-fix-clv, 'pr1-fix-clv') -> happy path stage_inputs(/tmp/dne, 'does-not-exist') -> raises FileNotFoundError NOTE-level findings 4-7 from the review are accepted as-is per the spike framing (judge inherits parent env intentionally for auth; same-day run dirs are an operator concern; broken-symlink in overlay copy is rare and surfaced via run_cell's general except; no tests is consistent with the Karpathy-spike scope and the DRC-3587 follow-up which adds Inspect AI test harness). Signed-off-by: even-wei <evenwei@infuseai.io> --------- Signed-off-by: even-wei <evenwei@infuseai.io> * fix(evals): close iter-6 BLOCKER (git history leak) + ISSUE (Tier-1 dbt policy) Two findings from the pr-cycle iter-6 review of #36, both re-confirming wcchang1115 (Andy) review comments that remained open. ## BLOCKER: per-fixture git history leaks stripped content build_fixtures.sh fetched the upstream head SHA into refs/fixture/head and `rm -rf`'d Recce-aware paths from the working tree. The original commit (and its tree) was untouched, so a Tier-0 agent allowlisted to run `git` could recover everything via `git show HEAD:recce.yml`, `git cat-file -p HEAD^{tree}`, `git log -p`, etc. The working-tree strip is bypassable by anyone reading git objects. Fix: after strip + leak-grep, rewrite the per-fixture repo's history into a single fresh commit whose tree IS the stripped working tree. Delete .git, `git init`, `git add -A`, `git commit`. No ancestor references the original head's tree; no other ref / reflog entry keeps it reachable. Verified end-to-end on a copy of pr1-fix-clv's source: Before: git show HEAD:recce.yml -> dumps recce.yml content (leak) git ls-tree -r HEAD | grep recce.yml -> recce.yml is listed After rewrite: git cat-file -e HEAD:recce.yml -> exit 128 (unreachable) git cat-file -e HEAD:.github/prompts/system-prompt.md -> unreachable git cat-file -e HEAD:.github/workflows/claude.yml -> unreachable git cat-file -e HEAD:.github/workflows/recce_ci.yml -> unreachable git cat-file -e HEAD:.devcontainer/post-create.sh -> unreachable rev-list --all HEAD --count -> 1 (single commit invariant preserved) Post-rewrite path-leak regex tightened to anchor at path components (`(^|/)recce\.yml$` etc.) so the generic .devcontainer.json VS Code config (no Recce content) isn't a false positive. ## ISSUE: Tier-1 dbt subcommand policy divergence Codex Tier-1 PATH-scrubs the `dbt` binary entirely (no dbt at all); Claude Code Tier-1 hook denylisted only {run,test,parse,compile,...} and allowed `dbt list/show/ls/deps/--help/--version`. The lens-3 cross-runner delta would partially measure "Claude Code can call read-only dbt subcommands but Codex can't" instead of the Recce-equipped-vs-not-equipped signal. Fix: at Tier-1, deny ALL dbt invocations regardless of subcommand. The recce-verify SKILL.md uses `git diff --name-only` + Recce MCP for model discovery, not `dbt list` — so denying read-only dbt subcommands costs nothing in capability. The DENIED_BINS check at line 340 still has the `- {"dbt"}` exclusion so the dbt branch gives its own specific message; the message is updated to reflect the broader policy ("not reachable at Tier-1"). Synthetic hook tests (Tier-1): dbt list -> BLOCKED ✓ dbt show -> BLOCKED ✓ dbt --version -> BLOCKED ✓ dbt run -> BLOCKED ✓ (already was) dbt parse -> BLOCKED ✓ (already was) git status -> allowed ✓ recce list -> allowed ✓ Signed-off-by: even-wei <evenwei@infuseai.io> * chore(evals): address iter-7 NOTEs — remove dead Tier-1 dbt code + doc drift Both findings from pr-cycle iter-7 review of #36 (explicit non-blockers, author already acknowledged the doc-drift one inline). NOTE 1: dead code in deny-tier-1.py after the iter-6 ISSUE fix. The unconditional `if "dbt" in head_names: deny(...)` at line ~340 makes the entire subcommand-discrimination path unreachable. Removed: - DBT_DENIED_SUBCOMMANDS frozenset (was lines 64-68) - _dbt_args_have_denied() function (was lines 519-527) - The `# --- dbt ---` block in walk() (was lines 458-465) deny-tier-0.py still has its own DBT_DENIED_SUBCOMMANDS — Tier-0 denies the same set + bare `dbt`. The two hook files are kept separate intentionally; cross-file dedup would require restructuring the per-tier overlay layout. Comment at line 65-68 documents this. NOTE 2: doc drift at fixtures/README.md:32 — the prior paragraph mentioned only the simple `grep -E '...'` belt-and-suspenders and made no reference to the history rewrite. Updated to describe: - Both leak-grep layers (tight identifier + loose `[Rr]ecce`) - profiles.yml whitelist rationale - The git history rewrite step + post-rewrite path-leak check - Why the rewrite matters (Tier-0-allowlisted `git` recoverability) Verified: py_compile ok, no functional orphan refs (only one comment mentions the deleted DBT_DENIED_SUBCOMMANDS name as cross-reference to tier-0). 10/10 synthetic hook tests still pass: - dbt list/show/run/parse/--version/deps -> all BLOCKED - git status, recce list -> allowed - sh -c "dbt parse" -> BLOCKED (shell wrapper) - {dbt,bash} run -> BLOCKED (brace expansion) Signed-off-by: even-wei <evenwei@infuseai.io> * fix(evals): close iter-8 BLOCKERs + ISSUEs on spike driver + overlays Address @wcchang1115's re-review at ff46b98 (PR #36). BLOCKER 1 — overlay-as-spoiler at the agent's cwd. The Tier-0 enforcement overlay copied to `<fixture>/.claude/` contains Recce-shaped strings (`mcp__recce__*`, `recce`, `RECCE_API_TOKEN`); `cat`/`ls`/`grep` over `.claude/` would let a Tier-0 agent read its own enforcement file and defeat the "agent cannot reach Recce-shaped signals" contract. * Tier-0 + Tier-1 `settings.json`: deny `Read/Glob/Grep(.claude/**)`. (Project-level deny rules apply even with `--dangerously-skip-permissions`; that flag bypasses the interactive prompt, not the rule.) * Tier-0 hook (`deny-tier-0.py`): new Pass 0 over command words and redirect targets. Any argument matching `(^|[/=])\.claude(/|$)` exits 2 with a "references the Tier-0 enforcement overlay" message. Covers `cat .claude/...`, `cat ./.claude/...`, `cat < .claude/...`, `ls .claude/`, `find .claude`, `grep -r recce .claude/`, and `cat $(echo .claude/...)` — i.e., the non-adversarial shapes. Interpreter-shell-out (awk/python `-c`) is intentionally out of scope per ENFORCEMENT.md § "Threat model — non-adversarial". BLOCKER 2 — Codex Tier-0 PATH-regex scrub was a no-op for normal installs. `grep -v -E '/recce(/|$)|/dbt(/|$)|\.recce'` strips dirs whose path contains literal `/recce/` or `/dbt/` — NOT dirs containing the binary named `recce`/`dbt`. So `/opt/homebrew/bin`, `.venv/bin`, `~/.local/bin` all survived and Codex Tier-0 could still run `recce list` / `dbt list`. * New `runner-configs/codex/tier-0/stub-bin/{recce,dbt}` — exit-127 stubs prepended to PATH so the real binaries are masked regardless of which bin dir they live in. * `driver.py::codex_tier0_path()` builds the PATH (stub-bin first, then the legacy regex scrub as belt-and-suspenders). * `driver.py::assert_codex_tier0_masked()` runs `command -v recce|dbt` under the final env before the agent launches; if either resolves outside `stub-bin/`, the cell errors out instead of contaminating the Tier-0 baseline. Verified: regression detected when stub-bin is skipped. * `codex/tier-0/README.md` updated with the new recipe + pre-flight assertion shell snippet. ISSUE 1 — Tier-0 read/write asymmetry. Codex Tier-0 is OS read-only; Claude Code Tier-0 had no Write/Edit deny + matcher didn't fire on those tool kinds + `--dangerously-skip-permissions` was set. * Tier-0 `settings.json`: deny `Write`, `Edit`, `MultiEdit`, `NotebookEdit`. Deny rules are enforced separately from `--dangerously-skip-permissions`. ISSUE 2 — cells don't reset state between runs. `fixture_dir = SOURCES_DIR / cell.fixture` was reused across all four cells per fixture with no `git reset`, so a write from cell N persisted into cell N+1. * New `driver.py::reset_fixture_dir()` runs `git reset --hard HEAD` + `git clean -fdx` at the top of `run_cell()`, before `stage_inputs()` (whose `_eval_inputs/` symlinks would otherwise be cleaned away) and before the overlay copy. Each cell now starts from the same committed base. ISSUE 3 — no model pinning (ENFORCEMENT.md:157). * `driver.py`: new `--model` flag (default `claude-opus-4-5`), `--no-model` opt-out. The pin is passed to `claude --model` and `codex --model` for every agent invocation and to the judge call. Cell metadata (`cells.json`, transcript headers) records the pinned model so cross-cell comparability is auditable. Deferred (Andy's Notes, non-blocking for this PR): - N1: tier-1 dbt deny-list / docs drift. - N2: `_eval_inputs/` symlinks could be written through at Tier-1. - N3: `codex/tier-1/config.toml` `command = ""` unvalidated. Verification: - `py_compile` clean on deny-tier-{0,1}.py + driver.py. - JSON valid on both `settings.json` files. - Synthetic Bash-hook tests: 12/12 `.claude/` access shapes denied; 8/8 allowlisted commands pass. - `codex_tier0_path()` + `assert_codex_tier0_masked()` pre-flight works under the new stub-bin recipe and catches regressions if stub-bin is dropped. - `driver.py --help` shows `--model` / `--no-model`. Signed-off-by: even-wei <evenwei@infuseai.io> * fix(evals): close iter-8 BLOCKER (Tier-0 overlay leak via cwd) structurally Andy's iter-8 review (PR #36, on 80d5ea3) flagged that the Pass 0 overlay-leak guard in deny-tier-0.py only matches the literal `.claude` substring, so a non-adversarial Tier-0 agent dumps the overlay's Recce vocabulary via ordinary recursive reads the Bash AST hook can't see ahead of glob expansion: cat .*/settings.json (glob expands AFTER the hook reads literal) grep -r recce . (recurse descends into .claude/) find . -type f -exec cat {} \; cat $(find . -type f) Token-matching is whack-a-mole; the root cause is structural — the overlay (mcp__recce__*, Bash(recce *), RECCE_API_TOKEN, recce literals in the hook source) lives in the agent's cwd as <fixture>/.claude/. Andy's recommended fix (verified on CC v2.1.160): load settings via `claude --settings <abs-path>` from outside cwd, with the hook command pointing to an absolute path also outside cwd. The agent's cwd then becomes a pristine dbt project with no Recce-shaped files — leak is structurally impossible. Driver change (spike-driver/driver.py): * Add render_claude_settings() — reads the template settings.json, substitutes ${RUNNER_HOOK_PATH} with the absolute path to the tier-N hook script (under runner-configs/), writes the rendered file to run_dir/_settings/<fixture>_t<tier>.json. * run_claude() drops shutil.copytree(overlay_src, fixture/.claude), calls render_claude_settings(), and passes --settings <rendered-abs-path> to `claude --print`. * Codex path (run_codex, assert_codex_tier0_masked) unchanged — Codex uses --config <toml> + PATH stub-bin masking, separate enforcement shape. Template change (tier-{0,1}/claude-overlay/settings.json): * `hooks.PreToolUse[].hooks[].command` now uses ${RUNNER_HOOK_PATH} placeholder; the runner substitutes at render time. * Hook matcher unchanged: Bash|Skill|mcp__(plugin_)?recce(_|-).* * permissions.deny rules unchanged (belt-and-suspenders, even though the overlay is no longer in cwd). Doc updates: * ENFORCEMENT.md Claude Code recipe: bash example now renders the template to a temp path and passes `claude --settings <abs-path>` instead of `cp -r claude-overlay/ .claude/`. Same shape for Tier-1. * runner-configs/README.md: directory map and Quick start updated; added the "why not copy" explainer pointing at the leak. Verification: * py_compile passes for driver.py, deny-tier-{0,1}.py. * JSON parse passes for tier-{0,1} settings templates. * Filesystem check: after reset_fixture_dir() + stage_inputs(), find <fixture> -maxdepth 2 -name "*.claude*" returns empty. Fixture cwd contains dbt project files + _eval_inputs/ symlink only. * Rendered settings live at run_dir/_settings/, outside fixture cwd. * Rendered hook command field carries an absolute path that resolves to an existing deny-tier-N.py with no ${RUNNER_HOOK_PATH} leakage. * Andy's 4 bypass shapes (cat .*/settings.json, grep -r Bash(recce ., find . -type f -exec grep -l deny-tier {} \;, cat $(find . -name deny-tier-0.py)) all return empty / no-such-file — overlay simply isn't there. * Standalone hook invocation (Bash dbt parse → exit 2; Bash ls → exit 0) still works when the hook script is executed from outside cwd. * Codex Tier-0 PATH masking (assert_codex_tier0_masked) is untouched and still aborts when recce/dbt resolve outside stub-bin. Out of scope (per the iter-8 review): * Codex --ignore-rules / --dangerously-bypass-hook-trust (codex-only, separate code path). * B1 (git history leak) and I2 (Tier-1 dbt policy) — already closed in 6f20289. * 7 adversarial-bypass classes documented in ENFORCEMENT.md § "Threat model — non-adversarial code agent". * The 3 NOTEs Andy explicitly deferred. Refs: DRC-3584 Signed-off-by: even-wei <evenwei@infuseai.io> --------- Signed-off-by: even-wei <evenwei@infuseai.io>
even-wei
pushed a commit
that referenced
this pull request
Jun 3, 2026
…-3405) Validation preview from the Karpathy spike driver landed in #37. Runs: 1 fixture (pr1-fix-clv) x 2 agents (claude, codex) x 2 tiers (0, 1) = 4 cells, with --judge-stability (two judge passes per transcript). NOT a DRC-3405 deliverable. 3405 wants: - 6 fixtures (not 1) - Manual baselines from DRC-3585 to anchor lens-3 delta - Hand-curated gap report This commit ships verdicts.csv + summary.md only. cells.json and transcripts/ stay local (cells.json has absolute paths; transcripts/ is gitignored). ## Headline results | Cell | Catch | Evidence | Delta | Status | |---|---|---|---|---| | claude · tier-0 | catch | 0 | same | ok | | claude · tier-1 | catch | 1c | improvement | ok | | codex · tier-0 | miss | 0 | same | rc=2 (driver bug) | | codex · tier-1 | miss | 0 | same | rc=2 (driver bug) | Claude moved from Tier-0 (evidence=0, no delta) to Tier-1 (evidence=1c, improvement). The lens-3 counterfactual is producing real signal on this fixture -- Recce-equipped agent reached for structured 1c-tier evidence (current-env queries) that Tier-0 couldn't. ## Judge self-consistency - catch: 100% (4/4 cells) -> above 80% bar - tier: 100% (4/4 cells) -> above 80% bar - delta: 75% (3/4 cells) -> just below 80% bar The delta flip on claude x tier-0 (same -> improvement across the two judge passes) is expected behaviour without a DRC-3585 manual baseline: the driver passes "baseline_catch=unknown" and the judge has to guess what delta means. Once 3585 lands, plug --baseline-dir in and re-judge. ## Findings to action separately 1. Tier-0 LEAK: recce.yml is present in pr1-fix-clv's .tmp/sources/ tree. The agent cited recce.yml preset names (row_count_diff, schema_diff) in its Tier-0 reasoning. build_fixtures.sh strip list includes recce.yml (line 292) but the .tmp/sources/ directory is stale -- it was built before the strip was added. Mitigation: re-run build_fixtures.sh. (Follow-up against DRC-3430.) 2. Codex driver: codex-cli 0.133.0 removed both --ask-for-approval=never and --config <file>. Driver invocation in run_codex() fails with rc=2 before the agent ever runs. Both codex cells show miss/0/same because the judge had nothing useful to score. Driver needs an update: CODEX_HOME-based config loading + `-c approval_policy="never"` override + a sanity check that the codex schema in our config.toml matches the installed codex version. (New ticket needed.) 3. Stale fixture worktrees: more broadly, if anyone re-runs the eval they should `rm -rf .tmp/` first to guarantee a fresh strip. Will document in the spike-driver README in a follow-up. Signed-off-by: even-wei <evenwei@infuseai.io>
even-wei
pushed a commit
that referenced
this pull request
Jun 3, 2026
…-3405) (#39) Validation preview from the Karpathy spike driver landed in #37. Runs: 1 fixture (pr1-fix-clv) x 2 agents (claude, codex) x 2 tiers (0, 1) = 4 cells, with --judge-stability (two judge passes per transcript). NOT a DRC-3405 deliverable. 3405 wants: - 6 fixtures (not 1) - Manual baselines from DRC-3585 to anchor lens-3 delta - Hand-curated gap report This commit ships verdicts.csv + summary.md only. cells.json and transcripts/ stay local (cells.json has absolute paths; transcripts/ is gitignored). ## Headline results | Cell | Catch | Evidence | Delta | Status | |---|---|---|---|---| | claude · tier-0 | catch | 0 | same | ok | | claude · tier-1 | catch | 1c | improvement | ok | | codex · tier-0 | miss | 0 | same | rc=2 (driver bug) | | codex · tier-1 | miss | 0 | same | rc=2 (driver bug) | Claude moved from Tier-0 (evidence=0, no delta) to Tier-1 (evidence=1c, improvement). The lens-3 counterfactual is producing real signal on this fixture -- Recce-equipped agent reached for structured 1c-tier evidence (current-env queries) that Tier-0 couldn't. ## Judge self-consistency - catch: 100% (4/4 cells) -> above 80% bar - tier: 100% (4/4 cells) -> above 80% bar - delta: 75% (3/4 cells) -> just below 80% bar The delta flip on claude x tier-0 (same -> improvement across the two judge passes) is expected behaviour without a DRC-3585 manual baseline: the driver passes "baseline_catch=unknown" and the judge has to guess what delta means. Once 3585 lands, plug --baseline-dir in and re-judge. ## Findings to action separately 1. Tier-0 LEAK: recce.yml is present in pr1-fix-clv's .tmp/sources/ tree. The agent cited recce.yml preset names (row_count_diff, schema_diff) in its Tier-0 reasoning. build_fixtures.sh strip list includes recce.yml (line 292) but the .tmp/sources/ directory is stale -- it was built before the strip was added. Mitigation: re-run build_fixtures.sh. (Follow-up against DRC-3430.) 2. Codex driver: codex-cli 0.133.0 removed both --ask-for-approval=never and --config <file>. Driver invocation in run_codex() fails with rc=2 before the agent ever runs. Both codex cells show miss/0/same because the judge had nothing useful to score. Driver needs an update: CODEX_HOME-based config loading + `-c approval_policy="never"` override + a sanity check that the codex schema in our config.toml matches the installed codex version. (New ticket needed.) 3. Stale fixture worktrees: more broadly, if anyone re-runs the eval they should `rm -rf .tmp/` first to guarantee a fresh strip. Will document in the spike-driver README in a follow-up. Signed-off-by: even-wei <evenwei@infuseai.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes DRC-3586. Single-file Python driver at
evals/agent-blind-spots/spike-driver/driver.py(~290 lines, stdlib only) that:claude --printandcodex exec.runs/<date>/spike-driver/transcripts/.verdicts.csv,summary.md, and re-judgeablecells.jsonper run.This is the spike, not the durable harness. If the spike's stability + sandbox checks pass, the next ticket (DRC-3587) ports to Inspect AI.
Two stability modes (both optional)
--judge-stability--baseline-dir <path>Without either, the driver still produces the full 24-cell verdict matrix — useful for inspecting agent behaviour even before the judge is trusted.
Graceful degradation
claudeorcodexisn't on PATH, the relevant cells skip with a recorded error rather than crashing. Lets you run claude-only or codex-only without flags..tmp/sources/<slug>/isn't built record an error pointing atbuild_fixtures.sh. No silent half-runs.--no-run: re-judges existing transcripts in--run-dirwithout re-running agents. Cheap iteration on judge-prompt changes.Codex-under-sandbox check (DRC-3586 acceptance #3)
After a run, grep
transcripts/*_codex_*.txtfor:mcp__recce__*references in Tier-0 → leak (MCP table is empty inrunner-configs/codex/tier-0/config.toml).recceshell calls in Tier-0 → exit was non-zero (PATH scrub).../README.mdor../../RUBRIC.md→ leak (cwd separation).dbtshell calls in either tier → exit was non-zero (PATH scrub).spike-driver/README.mddocuments the grep recipe.What's in here
No changes to
runner-configs/,ENFORCEMENT.md,RUBRIC.md,build_fixtures.sh, orfixtures/. The spike consumes what #36 produced; it does not modify the contract.Verification
python3 -m py_compile evals/agent-blind-spots/spike-driver/driver.py→ ok.python3 driver.py --help→ renders full usage block.--smoke) requiresclaudeon PATH + a built fixture worktree; recipe documented inspike-driver/README.md. Not executed in this background job — first smoke run is part of operator acceptance for this PR.Non-goals (explicit, per ticket)
RUBRIC.mdframing).Linear chain
Note: per Linear, DRC-3586 is blocked-by DRC-3585. Shipping the driver code first is a pragmatic move — it can be reviewed and merged independently, and the
--baseline-dirflag plugs in once 3585's manual baseline lands. Self-consistency mode (--judge-stability) is useful immediately.🤖 Generated with Claude Code