feat(evals): Tier-0/Tier-1 sandbox profiles + Recce-aware strip (DRC-3584, DRC-3430) - #36
Conversation
…3584, DRC-3430)
Make the /recce-verify v1 eval's Tier-0 / Tier-1 split enforceable
rather than aspirational, so the lens-3 counterfactual delta (Tier-0
verdict → Tier-1 verdict) actually isolates Recce's contribution.
DRC-3584 — sandbox profile templates under runner-configs/:
- Claude Code: per-tier .claude/{settings.json, hooks/deny-tier-N.sh}.
permissions.deny + a PreToolUse hook (belt-and-suspenders per open
Claude Code issue #6699). Tier 0 denies Recce CLI, Recce MCP namespaces
(mcp__recce__*, mcp__plugin_recce_*), /recce-* skills, dbt subcommands
that regenerate frozen Tier-0 inputs, and direct SQL clients. Tier 1
allows Recce CLI + MCP but keeps dbt regen and raw SQL clients denied.
- Codex: per-tier README + config.toml. Process sandbox flag
(--sandbox=read-only / workspace-write) + an empty (Tier 0) or
templated (Tier 1) mcp_servers table + a PATH scrub recipe to drop
recce/dbt binaries.
- ENFORCEMENT.md: per-agent recipes, the agent-view-restriction folded
from PR #28 follow-up, what's mechanically enforced vs. contract-only,
and the recording-in-baseline requirement.
- RUBRIC.md: adds the agent-view-restriction paragraph pointing at
ENFORCEMENT.md.
- templates/tier-0-baseline.md: adds the required "Sandbox profile
used" block in Notes; missing block disqualifies the baseline.
- runs/2026-05-28/sandbox-verification.md: synthetic hook unit tests
(Tier-0 6 cells, Tier-1 4 cells) all pass; one end-to-end claude
--print × Tier-0 confirms the PreToolUse hook fires on a real agent
attempt at `recce check`. CC × Tier-1, Codex × Tier-0, Codex × Tier-1
cells deferred to the operator with recipes.
DRC-3430 (bundled because the DRC-3584 verify step is contaminated
without it) — build_fixtures.sh:
- Strip .github/prompts/, .github/workflows/recce-*.yml,
.github/workflows/claude.yml, recce.yml from each per-fixture source
tree right after the head-SHA checkout.
- Belt-and-suspenders post-strip grep for
mcp__recce__|recce.yml|RECCE_API_TOKEN with --exclude-dir=.git.
- Initial run surfaced claude.yml (a "Claude Code + Recce MCP" reviewer
workflow that wasn't on the original strip list); now stripped. All
six fixtures rebuild cleanly with empty post-strip grep.
- fixtures/README.md documents the strip list and rationale.
gitignore exception added so the runner-configs/**/.claude/ templates
can be committed under the existing repo-wide .claude/ ignore.
Signed-off-by: even-wei <evenwei@infuseai.io>
Code Review: PR #36SHA The PR's stated north star is enforceability: the lens-3 counterfactual delta only isolates Recce's contribution if a Tier-0 agent provably cannot reach Recce-shaped signals. The Blockers
Issues
Notes
Recommended fix shapeFor the six blockers, two changes are likely enough:
The verification matrix in The PR's documentation/RUBRIC discipline is unusually careful and the build_fixtures.sh strip + leak-grep is sensible defense-in-depth — those parts hold up. The verdict is NO-GO purely because the hook does not enforce what the rubric requires. |
even-wei
left a comment
even-wei
left a comment
There was a problem hiding this comment.
NO-GO verdict (would be --request-changes if not self-authored). 6 BLOCKERs + 4 ISSUEs. The PreToolUse hook — load-bearing because permissions.deny is conceded to be unreliable per anthropics/claude-code#6699 — does not actually backstop the documented attack surface. true;recce, sh -c, (recce), absolute paths, dbt --debug parse, and capitalized Skill names all exit 0 from the hook. Full evidence per finding: #36 (comment)
…C-3584, DRC-3430)
Review found 6 BLOCKERs + 4 ISSUEs + 3 NOTEs against the v1 sandbox
profiles. Every blocker was an exit-0 from the case-glob hook on an
input the rubric said must be exit-2. This commit collapses them.
Hook rewrite — Python with shlex tokenization (B1+B2+B3+B4+B5+I10):
- New `deny-tier-{0,1}.py` (replacing `.sh`). Splits commands on shell
metacharacters (`;`, `&`, `|`, `()`, newline) plus `$()` and backtick
substitutions, shlex-parses each segment, basenames the executable
(so `/usr/local/bin/recce` is caught), and recurses into `sh -c` /
`bash -lc` / `zsh -c` arguments. Skill match is now case-insensitive.
MCP namespace regex broadened to `^mcp__(plugin_)?recce(_|-|$)` so
`mcp__recce_dev__*` and future variants are covered.
- Tier-0 flipped to a positive allowlist (git/grep/jq/file-read +
POSIX text utilities). The legitimate Tier-0 surface is tiny per
RUBRIC.md; allowlist is safer than chasing bypass shapes.
- Tier-1 stays a denylist but the matching is principled. dbt's
banned subcommands are detected anywhere after `dbt`, so
`dbt --debug parse` and `dbt --profiles-dir /tmp parse` are caught.
- `runs/2026-05-28/sandbox-verification.md` grew a Bypass attempts
table with every reviewer shape; all rows pass exit-2 against v2.
ENFORCEMENT.md fix (B6):
- Corrected the false claim that the hook blocks
`Bash(cat ../../<host-path>)`. The hook does not gate Read/Grep/Glob
and `cat` is in the Tier-0 allowlist. Spoiler-path protection comes
from cwd separation alone — the recipe's `cd "${FIXTURE_DIR}"` step
is therefore not optional. Document says so now.
- Reframed the `permissions.deny` + hook relationship: the hook is the
load-bearing layer, not a backup. `permissions.deny` is
documentation that mirrors the documented surface, not the full
bypass coverage.
Gitignore restructure (I8):
- Renamed `runner-configs/claude-code/tier-{0,1}/.claude/` →
`claude-overlay/`. Per gitignore docs, you can't re-include a file
under an excluded directory, so the v1 broad un-ignore would have
let arbitrary files under `.claude/` slip through. Renaming
sidesteps the `.claude/` ignore rule entirely; the eval runner now
copies `claude-overlay/` → `.claude/` (one extra arg to `cp -r`).
Gitignore exception removed.
Strip + leak-grep expansion (I9+N11):
- Strip list grew with `.devcontainer/`, `.github/mcp_config.json`,
`.github/workflows/{recce_*.yml, recce_*.yaml, dbt_base.yml,
dbt-build-pr.yml, dbt-build-base.yml}` — surfaced by the broadened
leak grep on the first re-build.
- Leak grep broadened from `mcp__recce__|recce\.yml|RECCE_API_TOKEN`
to case-insensitive `mcp__recce|recce\.yml|RECCE_API_TOKEN|recce`
with `profiles.yml` whitelisted (`role: RECCE` is the Snowflake
role name, not Recce-the-tool priming, and dbt needs the file).
- `printf ' %s\n' ${leak_hits}` (unquoted) replaced with
`sed 's/^/ /' <<< "${leak_hits}"` — robust against whitespace in
filenames.
Other doc fixes (N12, N13):
- Tier-1 hook header notes Recce MCP is intentionally not gated.
- `runner-configs/README.md` calls out that `permissions.deny` and
the hook now intentionally cover different surfaces.
- Maintenance section updated for the v2 layout.
Build verified end-to-end after every change. The Bypass attempts
table in `runs/2026-05-28/sandbox-verification.md` covers all 16
hook unit tests plus 1 end-to-end claude --print cell — every row
green.
Signed-off-by: even-wei <evenwei@infuseai.io>
v2 — review responseThanks for the adversarial pass; every blocker was real. New SHA Blockers — all fixed1. Shell-separator bypass — 2. Absolute-path bypass — 3. 4. dbt global-flag bypass — The hook now walks past 5. Skill case-sensitivity — 6. ENFORCEMENT.md:71 false claim — Rewrote the paragraph to credit cwd separation accurately. The hook does NOT gate Issues — all fixed7. Verification matrix only covers happy path — 8. Gitignore un-ignore too permissive — The narrow-leaf un-ignore I tried first doesn't work in gitignore (parent directory excluded → leaf un-ignore is ignored, per docs). Switched approach: renamed the template directory from 9. Leak-grep undercovers — Broadened from 10. MCP namespace incomplete — Regex now Notes — all fixed11. Unquoted 12. Tier-1 hook header silence on Recce MCP — Header docstring now spells it out: "Recce MCP is intentionally not gated here — the rubric explicitly allows it." 13. Build re-verifiedPost-strip grep returns empty across all 6 fixtures. |
Code Review: PR #36 — Tier-0/Tier-1 sandbox profiles + Recce-aware stripSHA Two new BLOCKERs in the v2 hooks that survive the prior reviewer's B-numbered findings — both are bypass shapes the test harness in Blockers
Issues
Notes
Hook tests run during this review (reproducible):
Deferred verification cells (CC × Tier-1, Codex × Tier-{0,1}) are appropriately scoped as operator-deferred in the PR body and documented in |
even-wei
left a comment
even-wei
left a comment
There was a problem hiding this comment.
NO-GO — see full review. Two BLOCKERs: Tier-1 dbt-flag-with-value bypass (8 shapes); Tier-0/1 xargs/find -exec/time wrapper bypass.
even-wei
left a comment
even-wei
left a comment
There was a problem hiding this comment.
NO-GO — see full review. Two BLOCKERs: Tier-1 dbt-flag-with-value bypass (8 shapes); Tier-0/1 xargs/find -exec/time wrapper bypass.
… exec wrappers)
The pr-cycle reviewer found 2 new BLOCKERs in v2 that the prior review
missed. Both are static hook-logic bugs; v3 fixes them with two small
changes plus a regex tweak.
BLOCKER 1 — Tier-1 dbt-flag-with-value bypass (`deny-tier-1.py`):
`has_denied_dbt_subcommand` skipped `-`-prefixed tokens but never
consumed the value following a value-bearing flag, so the value
became the "first positional" and the real subcommand was never
inspected. Eight verified shapes:
dbt --target dev parse → was exit 0, now exit 2
dbt --profiles-dir /tmp parse → was exit 0, now exit 2
dbt --project-dir /x run → was exit 0, now exit 2
dbt --vars "x: 1" parse → was exit 0, now exit 2
dbt --log-format json parse → was exit 0, now exit 2
dbt --log-level debug parse → was exit 0, now exit 2
dbt --printer-width 80 parse → was exit 0, now exit 2
dbt --profile myprof parse → was exit 0, now exit 2
Fix: scan EVERY token after `dbt` for a banned subcommand, not just
the "first positional". False positives require the agent to pass a
literal banned-subcommand name as a flag value (e.g. `--target parse`),
which is perverse and would deserve denial anyway since Tier-1
warehouse access is supposed to be mediated through Recce MCP.
Also expanded DBT_DENIED_SUBCOMMANDS with `run-operation`, `debug`,
`source` — all hit the warehouse and were missing from the prior list.
BLOCKER 2 — exec-wrapper bypass (both hooks):
`xargs recce` / `find -exec recce` / `time recce` / `nohup recce`
spawned a denied binary as a child process; the hook only inspected
the wrapper's head token and let the call through. Tier-0's positive
allowlist let `xargs` and `find` pass; Tier-1's denylist saw a
non-DENIED_BINS head and stopped looking.
Fix: new EXEC_WRAPPERS set covers
xargs, time, nice, nohup, setsid, parallel, exec, timeout, watch,
ionice, chrt, stdbuf
plus `find` (which needs special handling because its wrapped command
lives after `-exec`/`-execdir`). For each wrapper at the head of a
segment, walk the args for the first executable-shaped token and
recurse-check from there. Heuristic `looks_like_executable` skips
numeric positionals (timeout duration, chrt priority), flags, and
placeholders (`{}`, `[]`).
Regex fix (split_segments):
`(?<!\\)[;&|()\n]+` — negative lookbehind on backslash so escaped
separators (`find -exec ... \;`) stay attached to the same segment.
Without this, `find -exec dbt parse \;` would split at the `;` and
shlex would error on the trailing backslash; the malformed-payload
path would silently allow.
Verification:
All v3 cells exit-2 in red-team testing. Bypass attempts table in
`runs/2026-05-28/sandbox-verification.md` grew from 16 to 38 rows
covering every reviewer-named bypass (B1-B6, I7, cycle B1, cycle B2)
plus counter-claim regressions (Recce/MCP/dbt-list allowed at T1,
benign xargs/find at T0). Build re-verified across all 6 fixtures —
the broader leak grep + expanded strip still pass clean.
Closes the cycle review's BLOCKER 1 and BLOCKER 2. Addresses ISSUE 3
(per-tier disambiguation now in the table) and ISSUE 4 (collapsed by
BLOCKER 1's fix).
Signed-off-by: even-wei <evenwei@infuseai.io>
v3 — cycle-review responseThanks for the second adversarial pass. Both new BLOCKERs were real; v3 fixes them. New SHA BLOCKERs — both closedCycle B1 (Tier-1 dbt flag-with-value bypass) — False-positive surface: an agent would have to pass a literal banned-subcommand name as a flag value (e.g. Also expanded Cycle B2 (exec-wrapper bypass, both tiers) — new EXEC_WRAPPERS = {"xargs", "time", "nice", "nohup", "setsid",
"parallel", "exec", "timeout", "watch",
"ionice", "chrt", "stdbuf"}…plus Regression check: benign wrappers still allowed. Regex tweak alongside: ISSUEsI3 (per-tier disambiguation in the table) — the Bypass attempts table in I4 ( NOTEsN5 (matcher regex barely permissive for N6, N7, N8 — all confirmed positive (leak grep, ENFORCEMENT.md:74 fix, fail-open-on-malformed-JSON intent). No action. Verification surface
Ready for the next cycle pass. |
…retry) The pr-cycle iter-2 review surfaced three more bypass classes at Tier-1 — all real, none caught by v3. v4 closes them. 1. `eval` shell-builtin smuggling `eval dbt run` / `eval "dbt run"` were exit 0 because `eval` was not in SHELL_WRAPPERS. Fix: added `eval` to Tier-0's SHELL_WRAPPERS (banned outright since the positive allowlist has no legit reason for a shell) and added a dedicated `eval ARGS...` branch in Tier-1's check_tokens that joins the args and recurses into the resulting command — same threat surface as `sh -c`, different syntactic shape. 2. `$()` / backtick substitution at command-head position `$(echo dbt) run`, `` `echo dbt` run ``, `$(printf %s dbt) parse` evaluate to `dbt run` / `dbt parse` at bash runtime; the substitution provides the head binary, which the static check_tokens path can't see because the substitution's content has been stripped from the outer segment. Fix: new `check_substitution_at_head` in Tier-1 pre-walks each segment for `$(...)` / backtick at head and, if the substitution payload contains a denied binary name, denies the outer command. For dbt, it also combines payload tail + outer args to decide whether a denied subcommand is present (so `$(echo dbt)` alone is still allowed — bare dbt is discovery-only). Segmentation for this pre-check uses `(?<!\\)[;&|\n]+` (no parens) so `$(...)` stays intact. The main split_segments still includes parens because it pre-extracts substitutions before splitting. 3. Missing dbt subcommands `clone` and `retry` `dbt clone` (dbt-core ≥1.6) materialises cloned models in the warehouse. `dbt retry` re-executes the prior failed command, so it inherits any deny semantics of that prior command. Both added to DBT_DENIED_SUBCOMMANDS. False-positive sanity (all exit 0): git log --grep=\$(echo psql) — $() at non-head position echo \$(date) — benign substitution \$(which python) script.py — non-denied binary \$(echo grep) -rn foo . — non-denied binary \$(echo dbt) — dbt alone (allowed) echo \$(echo dbt) is good — dbt in non-head $() All v3 / v2 regressions still pass. Bypass attempts table in `runs/2026-05-28/sandbox-verification.md` grew with the iter-2 cells plus the new false-positive counter-claims. Build re-verified across all 6 fixtures. Signed-off-by: even-wei <evenwei@infuseai.io>
v4 — cycle iter-2 responseThree more bypass classes closed. New SHA Iter-2 findings — all closed at Tier-1 (Tier-0's positive allowlist already caught them)
Root cause: Fix:
Root cause: at bash runtime the substitution provides the head binary. The static hook can't follow string flow — Fix: new Missing dbt subcommands
False-positive sanity (all exit 0)The smuggling check is conservative on purpose: it only fires when (a) the substitution is at command-head position AND (b) its payload contains a denied basename AND (c) for dbt specifically, a denied subcommand appears somewhere in the combined payload+args. RegressionAll v3 / v2 cells still green. The Bypass attempts table in Ready for iter-3 if the cycle wants another pass. |
Code Review: PR #36SHA BlockersAll Blockers are real Bash-runtime bypasses against the Tier-1 deny hook ( 1. Nested
|
even-wei
left a comment
even-wei
left a comment
There was a problem hiding this comment.
NO-GO at 76d7640 (iter-3 adversarial). 9 BLOCKER classes / 15+ confirmed bypass shapes against deny-tier-1.py. Root causes: single-level coproc, command, builtin, !), and xargs -I {} substitution. Full evidence + runtime confirmation in #36 (comment) (cannot --request-changes on own PR; using --comment + treating as NO-GO).
…-3 bypasses)
The pr-cycle iter-3 review surfaced 9 more bypass classes that the
shlex+regex hooks couldn't model — all real, all rooted in Bash
semantics shlex doesn't represent.
Iter-3 BLOCKERs:
1. Nested $() — `$(sh -c "$(echo dbt) run")`, `$($(echo dbt) run)`,
`` `$(echo dbt) run` `` (single-level regex misses nesting)
2. `eval $(...)` — substitution stripped before eval branch ran
3. `xargs $(echo dbt)` — wrapper-at-head + substitution-as-arg;
find_wrapped_command saw no executable-shaped token
4. `dbt $(echo run)` — substitution supplied the subcommand;
has_denied_dbt_subcommand only scanned outer literal tokens
5. Bash ANSI-C $'dbt' / parameter expansion ${a:-dbt} — shlex
passed through as opaque basenames that didn't equal 'dbt'
6. `coproc dbt run` — Bash keyword not in any wrapper set
7. `command dbt run` / `builtin eval dbt run` — Bash command
modifiers not handled
8. `! dbt run` — Bash logical-not prefix not matched
9. `xargs -I {} sh -c "{} parse" dbt` — xargs substitutes dbt into
{} at runtime; hook saw only the literal {}
Rather than patch each class with another regex, this commit
**replaces the regex/shlex approach with `bashlex` (a real Bash AST
parser)**. Substitutions are walked as a tree; ANSI-C / parameter
expansion are extracted from their AST nodes; Bash keywords (!,
command, builtin) are handled as transparent prefixes; `coproc` is
denied outright (bashlex can't represent it).
New dependency: `bashlex` (pure Python, MIT, well-maintained). Both
hooks fail closed if bashlex isn't installed — better to deny a
legit call than silently allow a bypass. ENFORCEMENT.md + runner-
configs README document the install step (`python3 -m pip install
bashlex`).
Architecture:
Two passes per command (in walk()):
1. WALK INTO every $()/backtick substitution as if it were a
top-level command. Catches `$(sh -c "dbt run")` because the
inner sh -c "dbt run" is itself denied.
2. RESOLVE the head word of each command to candidate output
strings (literal text, ANSI-C decoded via raw-position lookup,
parameter defaults, substitution payload words). At Tier 0,
ALL candidates must be in the allowlist — `$(echo dbt) run`
resolves to ['echo','dbt'] and 'dbt' isn't allowlisted, so
deny. At Tier 1, any candidate matching a denied binary
(with denied subcommand visible anywhere in args, including
substitution payloads) denies.
Exec wrappers (xargs/find -exec/time/nohup/etc.) now scan EVERY
wrapped-command word's candidates for a denied basename — exec
wrappers supply args from stdin/find-output, so "bare dbt is
allowed" reasoning doesn't apply under a wrapper.
Shell wrappers (sh -c, bash -lc, ...) reparse the LITERAL word
text of the -c arg (not its substitution candidates) so the
inner command structure stays intact. Catches nested $(sh -c
"$(echo dbt) run").
Verification: 51 cells across both tiers. Every iter-1/iter-2/iter-3
BLOCKER closed. False-positive sanity for benign $() at non-head,
$(echo grep), $(which python), echo $(date), command grep, find
without -exec, xargs grep, etc. — all allow correctly.
Build re-verified across all 6 fixtures.
Closes the iter-3 9 BLOCKER classes. The reviewer's recommended
follow-up (pytest-parameterized over the bypass matrix) is filed as
an out-of-PR improvement.
Signed-off-by: even-wei <evenwei@infuseai.io>
v5 — iter-3 response (bashlex refactor)The 9 iter-3 BLOCKERs were all rooted in Bash semantics that regex+shlex couldn't model. Rather than patch each class with another regex, v5 replaces the regex/shlex approach with Iter-3 BLOCKERs — all closedArchitectureTwo passes per command in
Exec wrappers ( Shell wrappers ( New dependency
python3 -m pip install bashlexFalse-positive sanityVerification surface51 cells across both tiers (13 iter-3 BLOCKERs + iter-1/iter-2 BLOCKERs as regressions + 18 false-positive sanity). All green. Build re-verified across all 6 fixtures. Out-of-PR follow-upThe reviewer's recommendation to "convert the bypass table into a pytest file in CI" is solid and would catch regressions automatically. Filing as a separate follow-up — out of scope for this PR which is already a substantial refactor. Ready for iter-4 if the cycle wants another pass. |
Code Review: PR #36 — bashlex refactorSHA The bashlex refactor closes the iter-3 BLOCKERs it was scoped to close, but the AST walk is structurally incomplete: several bashlex node kinds the walker doesn't traverse hide live Blockers
Issues
Notes
Verification commands run
RecommendationThe bypass pattern is the same as iter-3: bashlex's AST has node kinds the walker doesn't handle (CompoundNode, ProcesssubstitutionNode, brace-literal WordNode). The fix is mechanical:
51/51 author cells will still pass with these fixes because they don't touch the substitution-shaped cells. The structurally-clean refactor's value is realized only once every reachable AST node kind is in the walker. |
even-wei
left a comment
even-wei
left a comment
There was a problem hiding this comment.
iter-4 review: NO-GO at 5b92f50. 7 BLOCKERs (CompoundNode walker gap, process-substitution, brace expansion at head and arg, xargs→sh -c chain, $0 wrapper, Tier-0 process-sub). 5 ISSUEs (coproc regex over-matches, NotImplementedError fails closed on legit constructs, ANSI-C decoder mismodels, exec-wrapper token filter, reparse-fallback first-token only). Full review: #36 (comment)
… / xargs-chain / \$0)
Iter-4 cycle review found 7 BLOCKERs + 5 ISSUEs against v5's bashlex
hooks — all rooted in AST-walker gaps inside the bashlex code (not
in bashlex itself). v6 closes them.
BLOCKERs:
1+7. **Process substitution `<(...)` / `>(...)`** — `_walk_substitutions`
only handled `commandsubstitution` kind. Now also handles
`processsubstitution` (same `.command` attribute). Catches
`diff <(dbt run) /etc/hosts` (head-position) AND
`echo x 2> >(dbt parse)` (process sub inside a RedirectNode —
`walk()` now also iterates `kind == 'redirect'` parts and walks
their `.output` word's substitutions).
2. **CompoundNode.list** — bashlex's subshell `(...)` and group
`{ ...; }` store children in `.list`, not `.parts`. The walker's
descent path now tries `.list` first then `.parts`, so
`(dbt parse)` and `{ dbt parse; }` are no longer silently
allowed.
3+4. **Brace expansion** — bashlex returns `{dbt,bash}` and
`{parse,run}` as literal `word` text with no `parts`. New
`_expand_brace_literal()` regex decodes `prefix{a,b,c}suffix`
into candidate list (`['prefix.a.suffix', 'prefix.b.suffix',
...]`). Used by `resolve_word` for both head and args.
Multi-candidate head fix: `{dbt,bash} run` resolves to
`{'dbt','bash'}`. Before v6, the shell-wrapper branch fired on
`bash` and returned without checking the dbt arm. Now an
eager-deny up front checks dbt+args and direct DENIED_BINS
BEFORE the shell-wrapper / eval / exec-wrapper branches —
any candidate matching a denied shape denies regardless of
what other candidates are present.
5. **xargs → sh -c chain** — `echo dbt | xargs -I {} sh -c "{} parse"`
exec-wrapper found `sh` as wrapped, but `sh` isn't in DENIED_BINS
so the wrapper-scan exited without escalating. v6 adds: if the
wrapped binary IS a shell wrapper, find the `-c` arg; if the
xargs `-I` placeholder appears inside the shell body,
conservatively deny (placeholder substitution from stdin is
dynamic — can't be statically resolved, and the pattern has very
narrow legit use). Otherwise re-parse the literal -c text.
6. **`$0` head** — bashlex parses `$0` as ParameterNode(value='0').
`resolve_word`'s parameter branch only handled `:-`/`:=` defaults.
Now also handles `0`, `BASH`, `SHELL`, `BASH_SOURCE` — these
resolve at runtime to a shell name; we surface them as the
candidate `"sh"` so the shell-wrapper branch fires conservatively.
ISSUEs:
8. **coproc regex inside string literals** — pre-parse `\bcoproc\b`
match was firing on `echo coproc`, `echo 'no coproc here'`, etc.
Removed the pre-parse check; rely on bashlex's NotImplementedError
raised when the actual `coproc` keyword is present. Error message
is inspected for `'coproc'` to differentiate from other
unsupported constructs.
9. **NotImplementedError fail-closed contradicted docstring** — bashlex
raises NotImplementedError on legit `$((arith))`, `select`, etc.
v5 denied all such cases. v6 only denies if the error message
mentions `coproc`; other unsupported constructs fall through
(the rest of the sandbox catches denied binaries).
10. **`_looks_like_executable`** — added `[[`, `]]`, `;;`, `&&`, `||`
to the skip set.
11. **ANSI-C decoder** — switched from `re.sub(r"\\(.)", r"\1", inner)`
to `codecs.decode(inner, 'unicode_escape')` for proper bash
escape semantics. Falls back to the naive regex on decode error.
12. **Reparse fallback** — already scans every token, no change
needed.
Verification: 49 cells across both tiers + iter-1/2/3/4 regressions
+ false-positive sanity. All green. Build re-verified across all 6
fixtures.
Closes the iter-4 cycle's 7 BLOCKERs and 5 ISSUEs.
Signed-off-by: even-wei <evenwei@infuseai.io>
v6 — iter-4 response7 BLOCKERs + 5 ISSUEs closed. New SHA BLOCKERsArchitecture refinementsMulti-candidate head handling. When a head resolves to multiple candidates ( Redirect targets walked. Brace expansion. New Shell name parameters. xargs placeholder smuggling. New ISSUEs
Regression49 cells across both tiers. Every iter-1/2/3/4 BLOCKER closed. False-positive sanity for benign Build re-verified clean across all 6 fixtures. Ready for iter-5 if the cycle wants another pass. |
Code Review: PR #36 — bashlex v6SHA The iter-4 BLOCKERs the v6 diff was scoped to close (process sub, CompoundNode.list, brace, xargs→sh-c chain, Blockers1.
|
Document the v1 enforcement contract explicitly: hooks are calibrated for a non-adversarial code agent (RUBRIC.md's stated subject), not a red-teamer actively evading the sandbox. Catalogue the seven classes of static-Bash-analysis-unreachable bypass (bashlex NIE, heredoc-body, pipeline string flow, stdin-supplied subcommands, trap evaluation, interpreter shell-out, variable-flow loops/assignments) as accepted out-of-scope for v1, with rationale and the follow-up trigger (non-adversarial agent reaching one in practice). Reframes the iter-3/4/5 cycle-reviewer BLOCKER inflation as adversarial review against a non-adversarial artifact; closes that loop on the documentation layer rather than the code layer. Signed-off-by: even-wei <evenwei@infuseai.io>
Closing comment — cycle-reviewer NO-GO reframed as out-of-scopeThe cycle reviewer (iter-3/4/5, NO-GO at 12 BLOCKERs) correctly identifies adversarial-evasion shapes — heredocs, pipelines, interpreter shell-out, stdin-supplied subcommands, But Resolution shipped in
Stacked work: DRC-3586 (Karpathy spike — judge stability + Codex-in-sandbox loop) will land in PR #37 stacked on this branch, so this PR can merge without waiting for the spike to complete. Ready for human review/merge. |
Code Review: PR #36 — additional findings (delta on the v1–v6 rounds)SHA This review adds only what the existing v1–v6 rounds on this PR have not already covered. The bulk of the NO-GO case — the Blocker (new)B1. The DRC-3430 strip is recoverable with allowlisted
The build-time leak-grep ( No published measurement is corrupted yet (the 6×2×2 run is deferred, and the one e2e smoke cell only exercised the deny-hook, not the read path), but every fixture already built by the current script carries the leak in its Fix: after stripping, rebuild Pass F. Issue (new framing)I2. Claude Code and Codex don't have equivalent Tier-0 enforcement — which contaminates the cross-runner axis of the eval. This is distinct from the already-raised "use PATH scrub to fix the hook's incompleteness". The point here is comparison validity, not the hook's completeness:
So a Claude Code Tier-0 agent can still reach Note this is not only a mechanism gap — the policies already diverge: Codex Tier-1 drops Fix: apply the same binary-blocking mechanism to both runners (PATH scrub, or better, make the denied binaries genuinely absent at Tier-0 so absolute paths and Pass I. Limits
🤖 Reviewed with Claude Code |
…DRC-3597) (#38) Foundation for the L3 funnel signal proposed in the 2026-05-29 project rethink. Tells the project whether real-world agent users reach for /recce-verify, complete it, and convert downstream -- the production complement to the L1 offline eval (DRC-3405) and L2 in-driver trace metrics (DRC-3586). What ships: - plugins/recce/hooks/scripts/telemetry.sh Event emitter. Fires recce_verify.* events to PostHog via curl fire-and-forget. Off by default; opt in via RECCE_TELEMETRY_OPT_IN=1 or ~/.recce/config.yml `telemetry_opt_in: true`. Anonymous stable installation ID at ~/.recce/installation-id (UUID4). Failure modes are silent and non-blocking by construction. - plugins/recce/hooks/scripts/test-telemetry.sh Audit script that exercises every opt-in / opt-out short-circuit without firing network traffic. Currently 4/4 pass on default-off, opt-in + no key, opt-in + DISABLED bypass, and missing-event-name edge case. - plugins/recce/hooks/scripts/README-telemetry.md Documents the event schema (skill_invoked, tier_degraded, tool_call, verdict_emitted, session_completed), opt-in mechanism, per-event wiring recommendations, plugin-maintainer responsibilities (PostHog key), and audit script usage. What's NOT in this PR (deferred to follow-up): - Auto-wiring in hooks.json (which PostToolUse / Stop hooks fire which events; needs design decision on session-scoping) - Inline emit calls in plugins/recce/skills/recce-verify/SKILL.md - The actual PostHog project key (plugin maintainer fills in at release) - Recce Cloud signup join key (coordinate with Andy when Cloud picks up the parameter) This is independent of the eval chain (DRC-3585 / 3586 / 3587 / 3405) and can land on its own without disturbing the in-flight eval work. Signed-off-by: even-wei <evenwei@infuseai.io>
…6) (#37) * feat(evals): Karpathy spike driver for /recce-verify v1 eval (DRC-3586) Single-file Python driver at evals/agent-blind-spots/spike-driver/. Dispatches up to 6 fixtures x 2 agents x 2 tiers = 24 cells, captures each agent's transcript, runs a Claude-as-judge pass per transcript to produce a three-axis verdict (catch / tier / delta), and writes a CSV + Markdown summary under runs/<date>/spike-driver/. Stability checks (both optional, both produce judge-quality signal): - --judge-stability: double-judges each transcript, reports per-axis self-consistency (catch / tier / delta). Floor 80%. - --baseline-dir <path>: compares judge verdicts against DRC-3585 manual baseline once it lands (judge-vs-human catch agreement on Tier-0 cells). Graceful degradation: - Skips an agent's cells if its CLI is not on PATH; codex commonly absent in lighter dev envs (recipe in runner-configs/codex/tier-{0,1}/README.md). - --no-run mode re-judges existing transcripts without re-running agents. Sandbox profile integration (from DRC-3584): - Claude Code cells stamp tier-N claude-overlay/ into the fixture worktree before invoking `claude --print`, with a neutered CLAUDE_CONFIG_DIR per cell and scrubbed warehouse env. - Codex cells use `codex exec --sandbox=<read-only|workspace-write> --ask-for-approval=never --config <tier-N/config.toml>` plus PATH scrub. Non-goals for this spike (deferred): - Durable harness with resume / parallel dispatch -> DRC-3587 Inspect AI port. - Gap-report generator across 6 fixtures -> DRC-3405. - Auto-iteration on prompt / skill changes (closed-loop optimization overfits at N=6 -- explicit non-goal of the spike). Stacks on PR #36 (DRC-3584 + DRC-3430). When #36 merges to main, this PR should be rebased onto main; GitHub auto-retargets the base branch. Signed-off-by: even-wei <evenwei@infuseai.io> * fix(evals): stage frozen Tier-0 inputs into cwd before agent run The agent's cwd is .tmp/sources/<id>/ (the per-fixture standalone repo), but the frozen Tier-0 inputs (diff.patch, artifacts/{manifest, compiled, catalog}*) live at fixtures/<id>/, outside cwd. Without staging: - Claude Code reaches them via absolute paths (Read tool isn't cwd-anchored), but the prompt doesn't tell the agent where to look. - Codex Tier-0 read-only sandbox blocks reads outside cwd entirely, so the agent literally cannot reach the artifacts. Driver now symlinks fixtures/<id>/{diff.patch, artifacts} into a _eval_inputs/ subdir under cwd before each cell. Prompt updated to point at _eval_inputs/. Works under both agent sandboxes. Verified locally: >>> stage_inputs(SOURCES_DIR / 'pr1-fix-clv', 'pr1-fix-clv') >>> sorted(p.name for p in (SOURCES_DIR / 'pr1-fix-clv' / '_eval_inputs').iterdir()) ['artifacts', 'diff.patch'] Signed-off-by: even-wei <evenwei@infuseai.io> * fix(evals): inherit CLAUDE_CONFIG_DIR for auth in unattended runs ENFORCEMENT.md recipe step 3 sets CLAUDE_CONFIG_DIR to a fresh mktemp dir to neuter user-level ~/.claude/settings.json. That step also strips the Claude Code auth state, so a child `claude --print` invoked by the spike driver fails with "Not logged in" and no transcript ever lands. For unattended runs, the load-bearing enforcement is the project-level .claude/settings.json overlay (stamped per cell) plus the PreToolUse hook (deny-tier-N.py); a user-level `permissions.allow` cannot bypass an exit-2 hook. Skip the CLAUDE_CONFIG_DIR override by default. For paranoid mode (e.g. when running the eval on a machine with risky ~/.claude/ contents), set RECCE_EVAL_STRICT_CONFIG=1 to enable the override. The operator is responsible for preseeding auth under the per-cell _claude_cfg dir. Verified end-to-end smoke run after this fix: uv run driver.py --smoke --agents claude --tiers 0 [run] pr1-fix-clv · claude · tier-0 [judge] pr1-fix-clv · claude · tier-0 Agent VERDICT: catch · request-changes Judge verdict: {catch, tier-0, same} (delta=same expected with no T1 paired) Signed-off-by: even-wei <evenwei@infuseai.io> * fix(evals): address pr-cycle review of #37 — 3 ISSUEs 1. Drop dead VERDICT_TAIL_RE regex. Defined in driver.py:99 but never used; agent verdict parsing is fully handled by judge_cell() downstream. Removed alongside the regex line. 2. Add surgical .gitignore entries for spike-driver runtime outputs: evals/agent-blind-spots/runs/*/spike-driver/transcripts/ evals/agent-blind-spots/runs/*/spike-driver/_claude_cfg/ These are the genuinely-transient subdirs. verdicts.csv, summary.md, cells.json under runs/<date>/spike-driver/ are intentionally NOT ignored — they may be committed manually as run baselines for cross-iteration tracking. 3. stage_inputs() now raises FileNotFoundError when no Tier-0 inputs (neither diff.patch nor artifacts/) could be staged under the fixture's source dir. run_cell() catches and records into cell.error so the cell shows up as failed in the matrix instead of silently feeding the agent an empty _eval_inputs/ and letting it review with no inputs. Verified: python3 -m py_compile spike-driver/driver.py -> ok stage_inputs(.../pr1-fix-clv, 'pr1-fix-clv') -> happy path stage_inputs(/tmp/dne, 'does-not-exist') -> raises FileNotFoundError NOTE-level findings 4-7 from the review are accepted as-is per the spike framing (judge inherits parent env intentionally for auth; same-day run dirs are an operator concern; broken-symlink in overlay copy is rare and surfaced via run_cell's general except; no tests is consistent with the Karpathy-spike scope and the DRC-3587 follow-up which adds Inspect AI test harness). Signed-off-by: even-wei <evenwei@infuseai.io> --------- Signed-off-by: even-wei <evenwei@infuseai.io>
Code Review: PR #36 — iter-6 (post-threat-model commit)SHA Scope per the PR's own re-framing (commit The threat-model section is the right reframing. The hooks' calibration matches a non-adversarial agent and the seven out-of-scope classes are correctly excluded. Two findings remain net-new at this HEAD — both raised by Blockers
Issues
Notes
Limits
🤖 Reviewed with Claude Code |
even-wei
left a comment
even-wei
left a comment
There was a problem hiding this comment.
NO-GO at 1d3c95d (iter-6, delta on threat-model commit). 1 BLOCKER + 1 ISSUE — both re-confirm wcchang1115's 09:28Z findings (B1: git-history leak in DRC-3430 strip; I2: cross-runner Tier-1 dbt policy divergence). Out-of-scope adversarial-evasion classes correctly excluded per ENFORCEMENT.md § "Threat model — non-adversarial code agent". See full review.
…bt policy) Two findings from the pr-cycle iter-6 review of #36, both re-confirming wcchang1115 (Andy) review comments that remained open. ## BLOCKER: per-fixture git history leaks stripped content build_fixtures.sh fetched the upstream head SHA into refs/fixture/head and `rm -rf`'d Recce-aware paths from the working tree. The original commit (and its tree) was untouched, so a Tier-0 agent allowlisted to run `git` could recover everything via `git show HEAD:recce.yml`, `git cat-file -p HEAD^{tree}`, `git log -p`, etc. The working-tree strip is bypassable by anyone reading git objects. Fix: after strip + leak-grep, rewrite the per-fixture repo's history into a single fresh commit whose tree IS the stripped working tree. Delete .git, `git init`, `git add -A`, `git commit`. No ancestor references the original head's tree; no other ref / reflog entry keeps it reachable. Verified end-to-end on a copy of pr1-fix-clv's source: Before: git show HEAD:recce.yml -> dumps recce.yml content (leak) git ls-tree -r HEAD | grep recce.yml -> recce.yml is listed After rewrite: git cat-file -e HEAD:recce.yml -> exit 128 (unreachable) git cat-file -e HEAD:.github/prompts/system-prompt.md -> unreachable git cat-file -e HEAD:.github/workflows/claude.yml -> unreachable git cat-file -e HEAD:.github/workflows/recce_ci.yml -> unreachable git cat-file -e HEAD:.devcontainer/post-create.sh -> unreachable rev-list --all HEAD --count -> 1 (single commit invariant preserved) Post-rewrite path-leak regex tightened to anchor at path components (`(^|/)recce\.yml$` etc.) so the generic .devcontainer.json VS Code config (no Recce content) isn't a false positive. ## ISSUE: Tier-1 dbt subcommand policy divergence Codex Tier-1 PATH-scrubs the `dbt` binary entirely (no dbt at all); Claude Code Tier-1 hook denylisted only {run,test,parse,compile,...} and allowed `dbt list/show/ls/deps/--help/--version`. The lens-3 cross-runner delta would partially measure "Claude Code can call read-only dbt subcommands but Codex can't" instead of the Recce-equipped-vs-not-equipped signal. Fix: at Tier-1, deny ALL dbt invocations regardless of subcommand. The recce-verify SKILL.md uses `git diff --name-only` + Recce MCP for model discovery, not `dbt list` — so denying read-only dbt subcommands costs nothing in capability. The DENIED_BINS check at line 340 still has the `- {"dbt"}` exclusion so the dbt branch gives its own specific message; the message is updated to reflect the broader policy ("not reachable at Tier-1"). Synthetic hook tests (Tier-1): dbt list -> BLOCKED ✓ dbt show -> BLOCKED ✓ dbt --version -> BLOCKED ✓ dbt run -> BLOCKED ✓ (already was) dbt parse -> BLOCKED ✓ (already was) git status -> allowed ✓ recce list -> allowed ✓ Signed-off-by: even-wei <evenwei@infuseai.io>
iter-6 findings addressed in
|
Code Review: PR #36 — iter-7 (post-iter-6 BLOCKER + ISSUE fix)SHA The iter-6 BLOCKER (git history leak via Tier-0-allowlisted BlockersNone. IssuesNone. Notes
Verification
ConvergenceBLOCKER count: iter-6 Recommendation: merge when ready. Optionally fold the two NOTEs into the same follow-up doc-only commit the author already planned. |
even-wei
left a comment
even-wei
left a comment
There was a problem hiding this comment.
iter-7 GO — convergence achieved. See #36 (comment)
…c drift Both findings from pr-cycle iter-7 review of #36 (explicit non-blockers, author already acknowledged the doc-drift one inline). NOTE 1: dead code in deny-tier-1.py after the iter-6 ISSUE fix. The unconditional `if "dbt" in head_names: deny(...)` at line ~340 makes the entire subcommand-discrimination path unreachable. Removed: - DBT_DENIED_SUBCOMMANDS frozenset (was lines 64-68) - _dbt_args_have_denied() function (was lines 519-527) - The `# --- dbt ---` block in walk() (was lines 458-465) deny-tier-0.py still has its own DBT_DENIED_SUBCOMMANDS — Tier-0 denies the same set + bare `dbt`. The two hook files are kept separate intentionally; cross-file dedup would require restructuring the per-tier overlay layout. Comment at line 65-68 documents this. NOTE 2: doc drift at fixtures/README.md:32 — the prior paragraph mentioned only the simple `grep -E '...'` belt-and-suspenders and made no reference to the history rewrite. Updated to describe: - Both leak-grep layers (tight identifier + loose `[Rr]ecce`) - profiles.yml whitelist rationale - The git history rewrite step + post-rewrite path-leak check - Why the rewrite matters (Tier-0-allowlisted `git` recoverability) Verified: py_compile ok, no functional orphan refs (only one comment mentions the deleted DBT_DENIED_SUBCOMMANDS name as cross-reference to tier-0). 10/10 synthetic hook tests still pass: - dbt list/show/run/parse/--version/deps -> all BLOCKED - git status, recce list -> allowed - sh -c "dbt parse" -> BLOCKED (shell wrapper) - {dbt,bash} run -> BLOCKED (brace expansion) Signed-off-by: even-wei <evenwei@infuseai.io>
Code Review: PR #36 — re-reviewSHA Re-review scope: the prior rounds (yours + pr-cycle, iters 1–7) were entirely about the bashlex deny-hook; those Bash-laundering bypasses are now legitimately scoped out as non-adversarial non-goals. Your B1 (git-history leak) and I2 (Tier-1 dbt policy) are confirmed genuinely fixed. The findings below are new, sit on the squashed-in spike driver ( Direct answer to the two review questions
Blockers
Issues
Notes
Already tracked
Confirmed fixed
Limits
🤖 Reviewed with Claude Code |
wcchang1115
left a comment
wcchang1115
left a comment
There was a problem hiding this comment.
Claude Code Review: NO-GO — 2 blockers, 3 issues. See review comment (#36 (comment)).
Address @wcchang1115's re-review at ff46b98 (PR #36). BLOCKER 1 — overlay-as-spoiler at the agent's cwd. The Tier-0 enforcement overlay copied to `<fixture>/.claude/` contains Recce-shaped strings (`mcp__recce__*`, `recce`, `RECCE_API_TOKEN`); `cat`/`ls`/`grep` over `.claude/` would let a Tier-0 agent read its own enforcement file and defeat the "agent cannot reach Recce-shaped signals" contract. * Tier-0 + Tier-1 `settings.json`: deny `Read/Glob/Grep(.claude/**)`. (Project-level deny rules apply even with `--dangerously-skip-permissions`; that flag bypasses the interactive prompt, not the rule.) * Tier-0 hook (`deny-tier-0.py`): new Pass 0 over command words and redirect targets. Any argument matching `(^|[/=])\.claude(/|$)` exits 2 with a "references the Tier-0 enforcement overlay" message. Covers `cat .claude/...`, `cat ./.claude/...`, `cat < .claude/...`, `ls .claude/`, `find .claude`, `grep -r recce .claude/`, and `cat $(echo .claude/...)` — i.e., the non-adversarial shapes. Interpreter-shell-out (awk/python `-c`) is intentionally out of scope per ENFORCEMENT.md § "Threat model — non-adversarial". BLOCKER 2 — Codex Tier-0 PATH-regex scrub was a no-op for normal installs. `grep -v -E '/recce(/|$)|/dbt(/|$)|\.recce'` strips dirs whose path contains literal `/recce/` or `/dbt/` — NOT dirs containing the binary named `recce`/`dbt`. So `/opt/homebrew/bin`, `.venv/bin`, `~/.local/bin` all survived and Codex Tier-0 could still run `recce list` / `dbt list`. * New `runner-configs/codex/tier-0/stub-bin/{recce,dbt}` — exit-127 stubs prepended to PATH so the real binaries are masked regardless of which bin dir they live in. * `driver.py::codex_tier0_path()` builds the PATH (stub-bin first, then the legacy regex scrub as belt-and-suspenders). * `driver.py::assert_codex_tier0_masked()` runs `command -v recce|dbt` under the final env before the agent launches; if either resolves outside `stub-bin/`, the cell errors out instead of contaminating the Tier-0 baseline. Verified: regression detected when stub-bin is skipped. * `codex/tier-0/README.md` updated with the new recipe + pre-flight assertion shell snippet. ISSUE 1 — Tier-0 read/write asymmetry. Codex Tier-0 is OS read-only; Claude Code Tier-0 had no Write/Edit deny + matcher didn't fire on those tool kinds + `--dangerously-skip-permissions` was set. * Tier-0 `settings.json`: deny `Write`, `Edit`, `MultiEdit`, `NotebookEdit`. Deny rules are enforced separately from `--dangerously-skip-permissions`. ISSUE 2 — cells don't reset state between runs. `fixture_dir = SOURCES_DIR / cell.fixture` was reused across all four cells per fixture with no `git reset`, so a write from cell N persisted into cell N+1. * New `driver.py::reset_fixture_dir()` runs `git reset --hard HEAD` + `git clean -fdx` at the top of `run_cell()`, before `stage_inputs()` (whose `_eval_inputs/` symlinks would otherwise be cleaned away) and before the overlay copy. Each cell now starts from the same committed base. ISSUE 3 — no model pinning (ENFORCEMENT.md:157). * `driver.py`: new `--model` flag (default `claude-opus-4-5`), `--no-model` opt-out. The pin is passed to `claude --model` and `codex --model` for every agent invocation and to the judge call. Cell metadata (`cells.json`, transcript headers) records the pinned model so cross-cell comparability is auditable. Deferred (Andy's Notes, non-blocking for this PR): - N1: tier-1 dbt deny-list / docs drift. - N2: `_eval_inputs/` symlinks could be written through at Tier-1. - N3: `codex/tier-1/config.toml` `command = ""` unvalidated. Verification: - `py_compile` clean on deny-tier-{0,1}.py + driver.py. - JSON valid on both `settings.json` files. - Synthetic Bash-hook tests: 12/12 `.claude/` access shapes denied; 8/8 allowlisted commands pass. - `codex_tier0_path()` + `assert_codex_tier0_masked()` pre-flight works under the new stub-bin recipe and catches regressions if stub-bin is dropped. - `driver.py --help` shows `--model` / `--no-model`. Signed-off-by: even-wei <evenwei@infuseai.io>
|
Thanks for the re-review @wcchang1115 — all 2 blockers + 3 issues addressed in Summary
Notes (deferred — surfaced but not fixed this round)
Verification before push
Ready for re-review. 🤖 Replied with Claude Code |
|
/re-review |
Code Review: PR #36 — iter-8SHA Re-review scope: only the delta Verified fixes (5/5 confirmed resolved at file level)
Notes (non-blocking; flagged by Andy as deferrals, surfaced FYI)The three Notes from Andy's Net-new findings from
|
even-wei
left a comment
even-wei
left a comment
There was a problem hiding this comment.
Claude Code Review iter-8: GO. All 5 of Andy's ff46b98 findings (B1, B2, I1, I2, I3) verified resolved at file level; zero net-new findings introduced by 80d5ea3. See review comment (#36 (comment)).
Code Review: PR #36 — Incremental (since
|
wcchang1115
left a comment
wcchang1115
left a comment
There was a problem hiding this comment.
Claude Code incremental review (SHA 80d5ea3): NO-GO. BLOCKER 2 + ISSUE 1/2 confirmed closed; one blocker remains — Tier-0 overlay leak is narrowed but reachable via recursive Bash reads (grep -r / find -exec cat / glob). Recommended structural fix (move overlay out of cwd via --settings, verified viable on v2.1.160) is in the review comment.
…urally Andy's iter-8 review (PR #36, on 80d5ea3) flagged that the Pass 0 overlay-leak guard in deny-tier-0.py only matches the literal `.claude` substring, so a non-adversarial Tier-0 agent dumps the overlay's Recce vocabulary via ordinary recursive reads the Bash AST hook can't see ahead of glob expansion: cat .*/settings.json (glob expands AFTER the hook reads literal) grep -r recce . (recurse descends into .claude/) find . -type f -exec cat {} \; cat $(find . -type f) Token-matching is whack-a-mole; the root cause is structural — the overlay (mcp__recce__*, Bash(recce *), RECCE_API_TOKEN, recce literals in the hook source) lives in the agent's cwd as <fixture>/.claude/. Andy's recommended fix (verified on CC v2.1.160): load settings via `claude --settings <abs-path>` from outside cwd, with the hook command pointing to an absolute path also outside cwd. The agent's cwd then becomes a pristine dbt project with no Recce-shaped files — leak is structurally impossible. Driver change (spike-driver/driver.py): * Add render_claude_settings() — reads the template settings.json, substitutes ${RUNNER_HOOK_PATH} with the absolute path to the tier-N hook script (under runner-configs/), writes the rendered file to run_dir/_settings/<fixture>_t<tier>.json. * run_claude() drops shutil.copytree(overlay_src, fixture/.claude), calls render_claude_settings(), and passes --settings <rendered-abs-path> to `claude --print`. * Codex path (run_codex, assert_codex_tier0_masked) unchanged — Codex uses --config <toml> + PATH stub-bin masking, separate enforcement shape. Template change (tier-{0,1}/claude-overlay/settings.json): * `hooks.PreToolUse[].hooks[].command` now uses ${RUNNER_HOOK_PATH} placeholder; the runner substitutes at render time. * Hook matcher unchanged: Bash|Skill|mcp__(plugin_)?recce(_|-).* * permissions.deny rules unchanged (belt-and-suspenders, even though the overlay is no longer in cwd). Doc updates: * ENFORCEMENT.md Claude Code recipe: bash example now renders the template to a temp path and passes `claude --settings <abs-path>` instead of `cp -r claude-overlay/ .claude/`. Same shape for Tier-1. * runner-configs/README.md: directory map and Quick start updated; added the "why not copy" explainer pointing at the leak. Verification: * py_compile passes for driver.py, deny-tier-{0,1}.py. * JSON parse passes for tier-{0,1} settings templates. * Filesystem check: after reset_fixture_dir() + stage_inputs(), find <fixture> -maxdepth 2 -name "*.claude*" returns empty. Fixture cwd contains dbt project files + _eval_inputs/ symlink only. * Rendered settings live at run_dir/_settings/, outside fixture cwd. * Rendered hook command field carries an absolute path that resolves to an existing deny-tier-N.py with no ${RUNNER_HOOK_PATH} leakage. * Andy's 4 bypass shapes (cat .*/settings.json, grep -r Bash(recce ., find . -type f -exec grep -l deny-tier {} \;, cat $(find . -name deny-tier-0.py)) all return empty / no-such-file — overlay simply isn't there. * Standalone hook invocation (Bash dbt parse → exit 2; Bash ls → exit 0) still works when the hook script is executed from outside cwd. * Codex Tier-0 PATH masking (assert_codex_tier0_masked) is untouched and still aborts when recce/dbt resolve outside stub-bin. Out of scope (per the iter-8 review): * Codex --ignore-rules / --dangerously-bypass-hook-trust (codex-only, separate code path). * B1 (git history leak) and I2 (Tier-1 dbt policy) — already closed in 6f20289. * 7 adversarial-bypass classes documented in ENFORCEMENT.md § "Threat model — non-adversarial code agent". * The 3 NOTEs Andy explicitly deferred. Refs: DRC-3584 Signed-off-by: even-wei <evenwei@infuseai.io>
Fix: structural, exactly as recommended — load Claude Code settings via Result: the agent's cwd is a pristine dbt project — no SHA pushed: Files changed
Verification (all green)
Before / after — fixture cwd shapeSettings.json + hook now live at the rendered path ( Out of scope (per the iter-8 review)
Ready for re-review. |
Code Review: PR #36SHA Scope: the single commit Prior Blocker 1 — CLOSED (structural + verified)
BlockersNone. IssuesNone. Notes
Carried over (unchanged by this commit — tracked, not regressions)
Limits
🤖 Reviewed with Claude Code |
wcchang1115
left a comment
wcchang1115
left a comment
There was a problem hiding this comment.
Claude Code Review (incremental, 8b5a504): GO. Prior Blocker 1 (Tier-0 overlay leak via cwd) structurally closed and verified on CC v2.1.160. No enforcement regression; 2 non-blocking NOTEs. See review comment.
2 participants
What this PR does
Make the `/recce-verify v1` eval's Tier-0 / Tier-1 split enforceable rather than aspirational. The lens-3 counterfactual delta (Tier-0 verdict → Tier-1 verdict) only isolates Recce's contribution if a Tier-0 agent provably cannot reach Recce-shaped signals. Without enforcement, the delta is contaminated by "agent context differences" rather than "Recce vs no-Recce".
Four Linear tickets land together (the latter two squashed in from their own PRs):
Andy's findings — fix locations + verification
B1 — git history leak
End-to-end verified on a copy of pr1-fix-clv source:
```
After fix:
git cat-file -e HEAD:recce.yml -> exit 128 ✓
git cat-file -e HEAD:.github/prompts/system-prompt.md -> exit 128 ✓
git cat-file -e HEAD:.github/workflows/claude.yml -> exit 128 ✓
git cat-file -e HEAD:.github/workflows/recce_ci.yml -> exit 128 ✓
git cat-file -e HEAD:.devcontainer/post-create.sh -> exit 128 ✓
git rev-list --all HEAD --count -> 1 (single-commit invariant preserved)
```
Post-rewrite path-leak regex is anchored at path components (`(^|/)recce.yml$`) so `.devcontainer.json` (generic VS Code dbt config, zero Recce content — verified across all 6 fixtures) is NOT a false positive.
I2 — Tier-1 dbt subcommand divergence
Synthetic hook tests (10/10 pass):
```
'dbt list' -> BLOCKED ✓
'dbt show' -> BLOCKED ✓
'dbt run' -> BLOCKED ✓
'dbt parse' -> BLOCKED ✓
'dbt --version' -> BLOCKED ✓
'dbt deps' -> BLOCKED ✓
'git status' -> allowed ✓
'recce list' -> allowed ✓
'sh -c "dbt parse"' -> BLOCKED ✓ (shell-wrapper bypass)
'{dbt,bash} run' -> BLOCKED ✓ (brace-expansion bypass)
```
Review history
The iter-1→5 spiral inverted under the threat-model reframing: real findings (B1, I2) surfaced and converged in one fix-and-verify cycle.
What's in the diff
Sandbox profile templates (DRC-3584)
Strip + git history rewrite (DRC-3430)
Karpathy spike driver (DRC-3586, from squash-merged #37)
Opt-in PostHog telemetry (DRC-3597, from squash-merged #38)
Updated artifacts
End-to-end verification
The 4-cell smoke baseline lives in PR #39 (stacked on this branch), which also documents 4 follow-up findings worth knowing before the full DRC-3405 run.
Threat model — explicit scope
Hooks enforce what a non-adversarial code agent reaches:
Seven adversarial-bypass classes are explicit non-goals for v1, documented at `ENFORCEMENT.md` § "Threat model — non-adversarial code agent":
For Codex all seven are closed by the OS-level process sandbox (`--sandbox=read-only` / `workspace-write`) + PATH scrub. Claude Code accepts them in v1 because Claude Code does not expose an OS sandbox primitive.
Deferred (operator follow-ups, NOT merge blockers)
Linear
Closes DRC-3584, DRC-3430, DRC-3586, DRC-3597 (the latter two via squash-merge of #37 + #38 into this branch).
Project chain after merge:
```
DRC-3430 + DRC-3584 + DRC-3586 + DRC-3597 ← all close on merge of this PR
↓ unblocks
DRC-3585 Lock rubric via 6×2×2 manual run ← Todo (1-2 day human hand-grade)
↓ unblocks
DRC-3587 Port to Inspect AI durable harness ← Backlog (gated on spike outcome)
↓ unblocks
DRC-3405 Run eval + produce ranked gap report ← Todo
```
🤖 Generated with Claude Code