From 0aa6b918ed7d25a903488cd7525f6dee5ea2929d Mon Sep 17 00:00:00 2001 From: even-wei Date: Thu, 28 May 2026 14:37:38 +0800 Subject: [PATCH 01/13] feat(evals): Tier-0/Tier-1 sandbox profiles + Recce-aware strip (DRC-3584, DRC-3430) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Make the /recce-verify v1 eval's Tier-0 / Tier-1 split enforceable rather than aspirational, so the lens-3 counterfactual delta (Tier-0 verdict → Tier-1 verdict) actually isolates Recce's contribution. DRC-3584 — sandbox profile templates under runner-configs/: - Claude Code: per-tier .claude/{settings.json, hooks/deny-tier-N.sh}. permissions.deny + a PreToolUse hook (belt-and-suspenders per open Claude Code issue #6699). Tier 0 denies Recce CLI, Recce MCP namespaces (mcp__recce__*, mcp__plugin_recce_*), /recce-* skills, dbt subcommands that regenerate frozen Tier-0 inputs, and direct SQL clients. Tier 1 allows Recce CLI + MCP but keeps dbt regen and raw SQL clients denied. - Codex: per-tier README + config.toml. Process sandbox flag (--sandbox=read-only / workspace-write) + an empty (Tier 0) or templated (Tier 1) mcp_servers table + a PATH scrub recipe to drop recce/dbt binaries. - ENFORCEMENT.md: per-agent recipes, the agent-view-restriction folded from PR #28 follow-up, what's mechanically enforced vs. contract-only, and the recording-in-baseline requirement. - RUBRIC.md: adds the agent-view-restriction paragraph pointing at ENFORCEMENT.md. - templates/tier-0-baseline.md: adds the required "Sandbox profile used" block in Notes; missing block disqualifies the baseline. - runs/2026-05-28/sandbox-verification.md: synthetic hook unit tests (Tier-0 6 cells, Tier-1 4 cells) all pass; one end-to-end claude --print × Tier-0 confirms the PreToolUse hook fires on a real agent attempt at `recce check`. CC × Tier-1, Codex × Tier-0, Codex × Tier-1 cells deferred to the operator with recipes. DRC-3430 (bundled because the DRC-3584 verify step is contaminated without it) — build_fixtures.sh: - Strip .github/prompts/, .github/workflows/recce-*.yml, .github/workflows/claude.yml, recce.yml from each per-fixture source tree right after the head-SHA checkout. - Belt-and-suspenders post-strip grep for mcp__recce__|recce.yml|RECCE_API_TOKEN with --exclude-dir=.git. - Initial run surfaced claude.yml (a "Claude Code + Recce MCP" reviewer workflow that wasn't on the original strip list); now stripped. All six fixtures rebuild cleanly with empty post-strip grep. - fixtures/README.md documents the strip list and rationale. gitignore exception added so the runner-configs/**/.claude/ templates can be committed under the existing repo-wide .claude/ ignore. Signed-off-by: even-wei --- .gitignore | 7 ++ evals/agent-blind-spots/ENFORCEMENT.md | 118 ++++++++++++++++++ evals/agent-blind-spots/RUBRIC.md | 2 + evals/agent-blind-spots/build_fixtures.sh | 50 ++++++++ evals/agent-blind-spots/fixtures/README.md | 13 ++ .../runner-configs/README.md | 55 ++++++++ .../tier-0/.claude/hooks/deny-tier-0.sh | 71 +++++++++++ .../claude-code/tier-0/.claude/settings.json | 33 +++++ .../tier-1/.claude/hooks/deny-tier-1.sh | 56 +++++++++ .../claude-code/tier-1/.claude/settings.json | 30 +++++ .../runner-configs/codex/tier-0/README.md | 54 ++++++++ .../runner-configs/codex/tier-0/config.toml | 14 +++ .../runner-configs/codex/tier-1/README.md | 61 +++++++++ .../runner-configs/codex/tier-1/config.toml | 35 ++++++ .../runs/2026-05-28/sandbox-verification.md | 102 +++++++++++++++ .../templates/tier-0-baseline.md | 11 ++ 16 files changed, 712 insertions(+) create mode 100644 evals/agent-blind-spots/ENFORCEMENT.md create mode 100644 evals/agent-blind-spots/runner-configs/README.md create mode 100755 evals/agent-blind-spots/runner-configs/claude-code/tier-0/.claude/hooks/deny-tier-0.sh create mode 100644 evals/agent-blind-spots/runner-configs/claude-code/tier-0/.claude/settings.json create mode 100755 evals/agent-blind-spots/runner-configs/claude-code/tier-1/.claude/hooks/deny-tier-1.sh create mode 100644 evals/agent-blind-spots/runner-configs/claude-code/tier-1/.claude/settings.json create mode 100644 evals/agent-blind-spots/runner-configs/codex/tier-0/README.md create mode 100644 evals/agent-blind-spots/runner-configs/codex/tier-0/config.toml create mode 100644 evals/agent-blind-spots/runner-configs/codex/tier-1/README.md create mode 100644 evals/agent-blind-spots/runner-configs/codex/tier-1/config.toml create mode 100644 evals/agent-blind-spots/runs/2026-05-28/sandbox-verification.md diff --git a/.gitignore b/.gitignore index 7f5fa70..06ffee9 100644 --- a/.gitignore +++ b/.gitignore @@ -54,6 +54,13 @@ evals/agent-blind-spots/fixtures/*/artifacts/ # Claude Code local state (eval runs, transcripts, etc.). # Note: `.claude-plugin/` is a different path and remains tracked. .claude/ +# Exception: Tier-0 / Tier-1 sandbox profile templates for the +# /recce-verify v1 eval (DRC-3584). These are committed templates the +# eval runner copies into per-fixture working trees; they happen to +# live under a `.claude/` subpath because they mirror the layout the +# consumer expects after the copy. +!evals/agent-blind-spots/runner-configs/**/.claude/ +!evals/agent-blind-spots/runner-configs/**/.claude/** # Local recce config (project-specific, not for the marketplace repo) recce.yml diff --git a/evals/agent-blind-spots/ENFORCEMENT.md b/evals/agent-blind-spots/ENFORCEMENT.md new file mode 100644 index 0000000..efaefd2 --- /dev/null +++ b/evals/agent-blind-spots/ENFORCEMENT.md @@ -0,0 +1,118 @@ +# Enforcement — Tier-0 / Tier-1 sandbox profiles + +Operational counterpart of `RUBRIC.md`'s Tier-0 runtime contract. The rubric defines **what** the agent must (and must not) see; this document defines **how** the runner makes that real and **how it gets recorded** in each baseline. + +This is load-bearing: the lens-3 counterfactual delta (Tier-0 verdict → Tier-1 verdict) only isolates Recce's contribution if a Tier-0 agent provably cannot reach Recce-shaped signals. If Tier-0 can sneak a peek — Recce MCP tools registered, `recce.yml` in the source tree, warehouse credentials in env, live SQL accessible, spoiler READMEs reachable — every downstream conclusion is contaminated. + +The sandbox profile templates live under [`runner-configs/`](runner-configs/). See [`runner-configs/README.md`](runner-configs/README.md) for the per-agent / per-tier file map; this document is the recipe that turns those templates into a recorded baseline. + +## Recipe — Claude Code + +### Tier 0 + +```bash +SLUG="pr1-fix-clv" # the fixture you're scoring +WT_ROOT="$(git rev-parse --show-toplevel)" # eval host repo root +FIXTURE_DIR="${WT_ROOT}/evals/agent-blind-spots/.tmp/sources/${SLUG}" +TIER_DIR="${WT_ROOT}/evals/agent-blind-spots/runner-configs/claude-code/tier-0" + +# 1. Build fixtures if you haven't. +( cd "${WT_ROOT}/evals/agent-blind-spots" && ./build_fixtures.sh ) + +# 2. Stamp the per-fixture working tree with the Tier-0 sandbox config. +cp -r "${TIER_DIR}/.claude" "${FIXTURE_DIR}/" + +# 3. Neuter user-level Claude Code settings so a stray ~/.claude/settings.json +# can't widen the sandbox. (Skip this step at your own risk.) +export CLAUDE_CONFIG_DIR="$(mktemp -d)" + +# 4. Clear warehouse and Recce credentials in the shell that spawns Claude Code. +unset RECCE_API_TOKEN DBT_PROFILES_DIR +unset SNOWFLAKE_USER SNOWFLAKE_PASSWORD SNOWFLAKE_ACCOUNT +unset POSTGRES_PASSWORD BIGQUERY_PROJECT +# (extend to match your environment; goal is no live credential in env) + +# 5. Run the agent with cwd at the per-fixture working tree. +cd "${FIXTURE_DIR}" +claude "" +``` + +The Tier-0 `.claude/settings.json` declares `permissions.deny` for Recce MCP namespaces and the dbt-regen / SQL-client Bash patterns, and registers a `PreToolUse` hook (`deny-tier-0.sh`) as belt-and-suspenders behind that — see `runner-configs/README.md` for the rationale. + +### Tier 1 + +Same shape as Tier 0, with three differences: + +1. `TIER_DIR` points to `runner-configs/claude-code/tier-1/`. +2. Leave the **dev-environment** warehouse credentials in the shell (or load them from a secrets file). Do **not** also export base/prod credentials — Tier 1 is single-env. +3. Recce MCP must be reachable to Claude Code (typically already true if `/recce-verify` works locally). The Tier-1 settings allow Recce MCP tools and Recce CLI; only dbt-regen and direct SQL clients stay denied. + +## Recipe — Codex + +See: + +* [`runner-configs/codex/tier-0/README.md`](runner-configs/codex/tier-0/README.md) +* [`runner-configs/codex/tier-1/README.md`](runner-configs/codex/tier-1/README.md) + +Both layer process sandboxing (`--sandbox=read-only` / `workspace-write`), MCP allowlisting (`config.toml`'s `mcp_servers` table), and a `PATH` scrub. Codex does not load `.claude/settings.json`, so the enforcement shape is different from Claude Code's even though the contract it satisfies is the same. + +## Agent view restriction (folded from PR #28 follow-up) + +The Tier-0 runtime contract enumerates the *inputs the agent gets*. It is not enough to enumerate what's allowed — the runner must also keep spoilers out of the agent's reach. A naive `claude` / `codex` invocation from the eval host repo root gives the agent file-read access to: + +- `evals/agent-blind-spots/fixtures//README.md` — contains `## Expected agent verdict — Tier 0` sections that *literally tell the agent the answer* +- `evals/agent-blind-spots/RUBRIC.md` — the rubric the agent is being scored against +- Sibling fixtures' files — useful for cross-comparison the agent should not have + +The runner therefore **must** launch the agent with cwd set to the per-fixture working tree at `.tmp/sources//` (a freshly-initialised standalone git repo from `build_fixtures.sh` — see step 2 in the script). The per-fixture worktree is structurally separated from the eval host repo: it has one commit, zero remotes, and no path back to `evals/`. With cwd at that worktree, the spoiler paths are outside the agent's filesystem reach for the duration of the run. + +This complements the cwd separation: + +* Claude Code: the project-level `.claude/settings.json` lives inside the per-fixture worktree, so it travels with the agent's cwd. The PreToolUse hook also blocks any `Bash(cat ../../)` attempt. +* Codex: the process sandbox is anchored on cwd; absolute paths outside cwd require approval at Tier-0 read-only mode and writes are uniformly denied. + +## Recording in the baseline + +Every Tier-0 and Tier-1 baseline records which sandbox profile fired, so deltas across iterations are reproducible. The Notes section of `templates/tier-0-baseline.md` has dedicated fields: + +```markdown +## Notes + +... + +### Sandbox profile used + +- Agent runtime: +- Tier: <0 | 1> +- Profile source: > +- Enforcement mechanism summary: +- Deviation from the recipe: +``` + +Without a recorded mechanism the baseline is unreproducible; treat a missing block as a baseline-of-record disqualifier. + +## Verifying enforcement (one-fixture smoke test) + +Per DRC-3584 acceptance criterion #2, run one fixture × {Claude Code, Codex} × {Tier-0, Tier-1} by hand and inspect the agent trace to confirm enforcement fires when the agent attempts a banned operation. The minimum trace check per cell: + +| Cell | What to grep the trace for | Pass condition | +|---|---|---| +| Claude Code · Tier 0 | `mcp__recce__`, `recce ` shell call, `dbt parse`, `dbt compile`, `Skill recce-verify` | All blocked (PreToolUse hook stderr line `Tier-0 sandbox blocks: ...`, exit 2) | +| Claude Code · Tier 1 | `mcp__recce__` succeeded | At least one Recce MCP tool call returned data | +| Claude Code · Tier 1 | `dbt parse`, `dbt compile`, raw `duckdb`/`psql`/`snowsql`/`bq` | All blocked | +| Codex · Tier 0 | `recce ` shell call, `dbt parse`, `mcp__recce__` reference | All blocked (`command not found` from PATH scrub; MCP table empty so tool name unresolved) | +| Codex · Tier 1 | `mcp__recce__` succeeded | At least one Recce MCP tool call returned data | +| Codex · Tier 1 | `dbt parse`, raw SQL clients | All blocked | +| Any cell | Read of `../README.md`, `../../RUBRIC.md`, sibling fixture file | Blocked (cwd-out-of-workspace; sandbox denies) | + +Record the result in a `runs//sandbox-verification.md` per cell so future contributors can repro. + +## What this does NOT enforce + +Mechanical enforcement only catches what the runner can introspect. Three categories are still contract-based: + +1. **Same-model contract** (`RUBRIC.md`). Nothing here pins the model — the runner must pin it. +2. **Tier-1 base/prod credential discipline**. The Tier-1 recipe assumes the shell that spawns the agent has *only* single-env credentials. There is no programmatic check that base/prod credentials are absent; the runner must verify before launch. +3. **Prompt-shape neutrality** (`RUBRIC.md` Tier-0 prompt-shape contract). The sandbox doesn't read the prompt. A prompt that steers the agent toward humility or aggression violates the rubric but does not trip a hook. + +Treat these three as runner discipline that's recorded in the baseline's Notes section. diff --git a/evals/agent-blind-spots/RUBRIC.md b/evals/agent-blind-spots/RUBRIC.md index 3bb7bc6..3f3e6f9 100644 --- a/evals/agent-blind-spots/RUBRIC.md +++ b/evals/agent-blind-spots/RUBRIC.md @@ -62,6 +62,8 @@ To make Tier-0 baselines reproducible across runs, the agent receives **the same **Explicitly NOT allowed at Tier 0:** Recce CLI, Recce MCP, any `/recce-*` skill (including `/recce-verify`), warehouse access, `dbt run`, `dbt test`, `dbt parse`, `dbt compile`, `dbt docs generate`, live SQL execution, or any other command that regenerates the manifest/compiled/catalog artifacts. The artifacts under `fixtures//artifacts/` are the **frozen Tier-0 inputs**; regenerating them lets the agent reach beyond the captured snapshot (e.g., picking up later upstream-package changes) and breaks reproducibility across runs. Also not allowed: comparison against a base/prod environment beyond what is already in the artifacts above. +**Agent view restriction (eval-runner contract).** The runner MUST restrict the agent's file-read access to `.tmp/sources//` (the per-fixture standalone repo from `build_fixtures.sh`). The agent MUST NOT be able to read `fixtures//README.md` (contains expected-verdict spoilers), `RUBRIC.md`, other fixtures' files, or anywhere else in the eval host repository. Enforcement is runner-specific (`.claude/settings.json` PreToolUse hook for Claude Code, `--sandbox` + PATH scrub for Codex) — see [`ENFORCEMENT.md`](ENFORCEMENT.md) for the recipes and the sandbox profile templates in [`runner-configs/`](runner-configs/). The mechanism used for each baseline MUST be recorded in `templates/tier-0-baseline.md`'s Notes section — without that record, the baseline is unreproducible and disqualified. + **Prompt shape** — eval runners write the actual prompt and **MUST record it verbatim** in the Tier-0 baseline's "Prompt given to agent" section, including any agent-specific framing. To keep baselines comparable across runs: - The prompt MUST describe the inputs above without paraphrasing what each contains (re-describing the inputs primes the agent in ways that vary between runners). diff --git a/evals/agent-blind-spots/build_fixtures.sh b/evals/agent-blind-spots/build_fixtures.sh index e44be19..15a7373 100755 --- a/evals/agent-blind-spots/build_fixtures.sh +++ b/evals/agent-blind-spots/build_fixtures.sh @@ -257,6 +257,56 @@ build_fixture() { return 1 fi + # DRC-3430: strip Recce-aware automation that would leak the + # with-Recce answer to a Tier-0 agent reading the source tree. + # `jaffle_shop_golden` is a Recce-dogfood repo, so the per-fixture + # checkout ships: + # * .github/prompts/*.md — system prompt + worked-example tables + # for Recce-aware PR review (lists mcp__recce__ tool names, + # prescribed call sequences, numeric anchors for the affected + # column) + # * .github/workflows/recce-*.yml — Recce CI workflows + # * .github/workflows/claude.yml — "Claude Code + Recce MCP" + # reviewer workflow (primes the agent with the same playbook) + # * recce.yml — preset Recce checks + # A random dbt project in the wild would not have these; their + # presence here is fixture-source-specific, not a contract bug. + # Strip them so a Tier-0 baseline measures what the agent deduces + # without Recce-style structured surfacing. + local stripped_paths=( + ".github/prompts" + ".github/workflows/claude.yml" + "recce.yml" + ) + for stripped in "${stripped_paths[@]}"; do + rm -rf "${source_dir:?}/${stripped}" + done + # Glob removal: any .github/workflows/recce-*.{yml,yaml} — guard + # against an empty match because `nullglob` isn't set globally. + if compgen -G "${source_dir}/.github/workflows/recce-*.yml" > /dev/null; then + rm -f "${source_dir}"/.github/workflows/recce-*.yml + fi + if compgen -G "${source_dir}/.github/workflows/recce-*.yaml" > /dev/null; then + rm -f "${source_dir}"/.github/workflows/recce-*.yaml + fi + + # DRC-3430 belt-and-suspenders: grep for known Recce-shaped strings + # in the stripped working tree. Catches future regressions where a + # new Recce-aware file lands at a path the strip list doesn't + # cover. Scope the grep to the working tree (`--exclude-dir=.git`) + # so packed objects under .git/ don't false-positive — the agent + # cannot read packed git internals without first invoking a git + # command, and even then sees decoded content, not raw byte + # strings. + local leak_hits + leak_hits="$(grep -rlE --exclude-dir=.git 'mcp__recce__|recce\.yml|RECCE_API_TOKEN' "${source_dir}" 2>/dev/null || true)" + if [[ -n "${leak_hits}" ]]; then + echo "FAIL ${slug} (Tier-0 strip leak — Recce-shaped strings still present in:)" >&2 + printf ' %s\n' ${leak_hits} >&2 + echo " Extend the strip list in build_fixtures.sh and re-run." >&2 + return 1 + fi + # PR #20 intermediate snapshot — keyed on the well-known SHA in commits.txt. if [[ "${slug}" == "pr44-promotion-flags" ]]; then local intermediate_sha="23b96ca" diff --git a/evals/agent-blind-spots/fixtures/README.md b/evals/agent-blind-spots/fixtures/README.md index 701be36..07bb9c5 100644 --- a/evals/agent-blind-spots/fixtures/README.md +++ b/evals/agent-blind-spots/fixtures/README.md @@ -15,6 +15,19 @@ The script clones `DataRecce/jaffle_shop_golden` into `.tmp/jaffle_shop_golden/` Re-running is idempotent — existing `artifacts/` directories are removed and rebuilt. +## Per-fixture source tree — what gets stripped + +`build_fixtures.sh` materialises each fixture's head-SHA checkout at `.tmp/sources//` (a freshly-initialised standalone git repo with one commit, zero remotes — see the in-script comment for the leak-proofing rationale). Immediately after checkout, the build script **strips Recce-aware automation** from that working tree so a Tier-0 agent reading the source cannot find the with-Recce answer pre-baked into the repo (see [DRC-3430](https://linear.app/recce/issue/DRC-3430)): + +- `.github/prompts/` — Recce-aware GitHub Action system prompt + worked-example output tables (names `mcp__recce__` tools, prescribes call sequences, anchors numeric values for the column the fixture's PR affects) +- `.github/workflows/recce-*.yml` and `recce-*.yaml` — Recce CI workflow definitions +- `.github/workflows/claude.yml` — "Claude Code + Recce MCP" reviewer workflow (primes the agent with the same playbook) +- `recce.yml` — preset Recce check definitions + +The strip is followed by a `grep -E 'mcp__recce__|recce\.yml|RECCE_API_TOKEN'` belt-and-suspenders sweep over the source tree; the build fails fast (`FAIL `) if any new Recce-shaped file slips in at an unanticipated path. When that happens, extend the strip list in `build_fixtures.sh` and re-run. + +This is fixture-source-specific, not a contract bug: `DataRecce/jaffle_shop_golden` is a Recce-dogfood repo and the stripped paths are the Recce team's own automation. A random dbt project in the wild would not have them. See [`../ENFORCEMENT.md`](../ENFORCEMENT.md) for how the sandbox profiles relate to this strip. + ## Index | Fixture | Source PR | Class | Notes | diff --git a/evals/agent-blind-spots/runner-configs/README.md b/evals/agent-blind-spots/runner-configs/README.md new file mode 100644 index 0000000..ec2dac8 --- /dev/null +++ b/evals/agent-blind-spots/runner-configs/README.md @@ -0,0 +1,55 @@ +# Runner configs — Tier-0 / Tier-1 sandbox profiles + +Self-contained sandbox profile templates for the eval runners (Claude Code and Codex). They make the Tier-0 / Tier-1 distinction enforceable rather than aspirational: the lens-3 counterfactual delta in `RUBRIC.md` (Tier-0 verdict → Tier-1 verdict) is only meaningful if a Tier-0 agent provably cannot reach Recce-shaped signals. + +See [`../ENFORCEMENT.md`](../ENFORCEMENT.md) for the end-to-end runner recipe and the per-baseline recording requirement. + +``` +runner-configs/ +├── README.md ← (this file) +├── claude-code/ +│ ├── tier-0/ +│ │ └── .claude/ +│ │ ├── settings.json ← permissions.deny + PreToolUse hook +│ │ └── hooks/deny-tier-0.sh ← exit-2 block on Recce + dbt regen + SQL clients +│ └── tier-1/ +│ └── .claude/ +│ ├── settings.json ← narrower deny (Recce allowed; dbt regen + SQL clients still denied) +│ └── hooks/deny-tier-1.sh +└── codex/ + ├── tier-0/ + │ ├── README.md ← invocation recipe (PATH scrub + sandbox flag + cwd) + │ └── config.toml ← empty mcp_servers + └── tier-1/ + ├── README.md + └── config.toml ← template mcp_servers.recce entry to fill in +``` + +## Quick start + +**Claude Code, Tier 0:** + +```bash +SLUG="pr1-fix-clv" +WT_ROOT="$(git rev-parse --show-toplevel)" +FIXTURE_DIR="${WT_ROOT}/evals/agent-blind-spots/.tmp/sources/${SLUG}" +TIER_DIR="${WT_ROOT}/evals/agent-blind-spots/runner-configs/claude-code/tier-0" + +cp -r "${TIER_DIR}/.claude" "${FIXTURE_DIR}/" + +# Optional but recommended — neuter user-level settings for this run: +export CLAUDE_CONFIG_DIR="$(mktemp -d)" + +cd "${FIXTURE_DIR}" +claude "" +``` + +**Codex, Tier 0:** see [`codex/tier-0/README.md`](codex/tier-0/README.md). + +## Why both `permissions.deny` and a `PreToolUse` hook + +Claude Code's `permissions.deny` is the documented mechanism, but open issue [anthropics/claude-code#6699](https://github.com/anthropics/claude-code/issues/6699) shows it can be bypassed in some shapes. The PreToolUse hook is belt-and-suspenders: a deny-by-default exit-2 script. If `permissions.deny` regresses, the hook still blocks; if the hook regresses, `permissions.deny` still blocks. + +## Maintenance + +When you add a new Recce CLI verb, new MCP tool namespace, or a new dbt subcommand that mutates state, update **both** the relevant tier's `settings.json` deny list **and** the hook script's case statement. The hook is the source of truth — `settings.json` mirrors it for the documented path. diff --git a/evals/agent-blind-spots/runner-configs/claude-code/tier-0/.claude/hooks/deny-tier-0.sh b/evals/agent-blind-spots/runner-configs/claude-code/tier-0/.claude/hooks/deny-tier-0.sh new file mode 100755 index 0000000..11e0b08 --- /dev/null +++ b/evals/agent-blind-spots/runner-configs/claude-code/tier-0/.claude/hooks/deny-tier-0.sh @@ -0,0 +1,71 @@ +#!/usr/bin/env bash +# +# Tier-0 PreToolUse hook for the /recce-verify v1 eval. +# Belt-and-suspenders alongside `permissions.deny` in ../settings.json +# (see DRC-3584 issue body: hooks are more reliable than deny rules per +# open Claude Code issue #6699). +# +# Reads the tool-use payload on stdin (JSON) and exits 2 with a stderr +# message to block when the call would let a Tier-0 agent reach for +# Recce-shaped signals or regenerate frozen Tier-0 inputs. +# +# Tier 0 denies: +# * Recce MCP tools (any mcp__recce__* or mcp__plugin_recce_*) +# * Recce CLI (`recce ...` in Bash) +# * `/recce-*` skills (invoked through the Skill tool) +# * dbt subcommands that regenerate manifest/compiled/catalog or hit +# a warehouse (`dbt run|test|parse|compile|docs`) +# * Direct SQL clients (`duckdb`, `psql`, `snowsql`, `bq`) + +set -euo pipefail + +payload="$(cat)" + +tool_name="$(printf '%s' "${payload}" | jq -r '.tool_name // ""')" +command="$(printf '%s' "${payload}" | jq -r '.tool_input.command // ""')" +skill="$(printf '%s' "${payload}" | jq -r '.tool_input.skill // ""')" + +deny() { + printf 'Tier-0 sandbox blocks: %s\n' "$1" >&2 + exit 2 +} + +# Recce MCP — any namespace variant +case "${tool_name}" in + mcp__recce__*|mcp__plugin_recce_*) + deny "Recce MCP tool '${tool_name}' (Tier-0 disallows Recce)" + ;; +esac + +# Skill invocation — /recce-* or recce:* (plugin-namespaced) +case "${skill}" in + recce-*|recce:*) + deny "Recce skill '${skill}' (Tier-0 disallows /recce-* skills)" + ;; +esac + +# Bash command inspection +if [ "${tool_name}" = "Bash" ] && [ -n "${command}" ]; then + case "${command}" in + recce|recce[[:space:]]*|*[[:space:]]recce|*[[:space:]]recce[[:space:]]*) + deny "Recce CLI invocation (matched in: ${command})" + ;; + *dbt[[:space:]]run*|*dbt[[:space:]]test*|*dbt[[:space:]]parse*|*dbt[[:space:]]compile*|*dbt[[:space:]]docs*) + deny "dbt subcommand regenerates frozen Tier-0 artifacts or hits a warehouse (matched in: ${command})" + ;; + duckdb|duckdb[[:space:]]*|*[[:space:]]duckdb|*[[:space:]]duckdb[[:space:]]*) + deny "Direct SQL client 'duckdb' (matched in: ${command})" + ;; + psql|psql[[:space:]]*|*[[:space:]]psql|*[[:space:]]psql[[:space:]]*) + deny "Direct SQL client 'psql' (matched in: ${command})" + ;; + snowsql|snowsql[[:space:]]*|*[[:space:]]snowsql|*[[:space:]]snowsql[[:space:]]*) + deny "Direct SQL client 'snowsql' (matched in: ${command})" + ;; + bq|bq[[:space:]]*|*[[:space:]]bq|*[[:space:]]bq[[:space:]]*) + deny "Direct SQL client 'bq' (matched in: ${command})" + ;; + esac +fi + +exit 0 diff --git a/evals/agent-blind-spots/runner-configs/claude-code/tier-0/.claude/settings.json b/evals/agent-blind-spots/runner-configs/claude-code/tier-0/.claude/settings.json new file mode 100644 index 0000000..6ac527d --- /dev/null +++ b/evals/agent-blind-spots/runner-configs/claude-code/tier-0/.claude/settings.json @@ -0,0 +1,33 @@ +{ + "$comment": "Tier-0 sandbox profile for the /recce-verify v1 eval. Drop this .claude/ directory into the per-fixture working tree (.tmp/sources//) before invoking Claude Code. See ../../../ENFORCEMENT.md for the runner recipe.", + "permissions": { + "deny": [ + "mcp__recce__*", + "mcp__plugin_recce_*", + "Bash(recce *)", + "Bash(dbt run*)", + "Bash(dbt test*)", + "Bash(dbt parse*)", + "Bash(dbt compile*)", + "Bash(dbt docs*)", + "Bash(duckdb *)", + "Bash(psql *)", + "Bash(snowsql *)", + "Bash(bq *)" + ] + }, + "hooks": { + "PreToolUse": [ + { + "matcher": "Bash|Skill|mcp__recce__.*|mcp__plugin_recce_.*", + "hooks": [ + { + "type": "command", + "command": "bash \"$CLAUDE_PROJECT_DIR/.claude/hooks/deny-tier-0.sh\"", + "timeout": 5 + } + ] + } + ] + } +} diff --git a/evals/agent-blind-spots/runner-configs/claude-code/tier-1/.claude/hooks/deny-tier-1.sh b/evals/agent-blind-spots/runner-configs/claude-code/tier-1/.claude/hooks/deny-tier-1.sh new file mode 100755 index 0000000..3df94dd --- /dev/null +++ b/evals/agent-blind-spots/runner-configs/claude-code/tier-1/.claude/hooks/deny-tier-1.sh @@ -0,0 +1,56 @@ +#!/usr/bin/env bash +# +# Tier-1 PreToolUse hook for the /recce-verify v1 eval. Tier 1 is +# Tier 0 plus Recce CLI, Recce MCP, and single-env warehouse credentials +# (read-only on the dev environment). Base/prod environment access stays +# denied — that's Tier 2 territory, out of v1 scope. +# +# Reads the tool-use payload on stdin (JSON) and exits 2 with a stderr +# message to block when the call would either regenerate frozen Tier-0 +# inputs or reach for a base/prod environment. +# +# Tier 1 denies (relative to Tier 0): +# * dbt subcommands that regenerate frozen artifacts or hit a warehouse +# directly (`dbt run|test|parse|compile|docs`). Recce reads the +# frozen artifacts; the agent never needs to regenerate them. +# * Direct SQL clients (`duckdb`, `psql`, `snowsql`, `bq`) — at Tier 1 +# warehouse access is mediated through Recce MCP tools (e.g., +# `mcp__recce__query`), never raw shell. +# +# Tier 1 explicitly ALLOWS (which Tier 0 denied): +# * Recce MCP tools and `/recce-*` skills +# * `recce` CLI + +set -euo pipefail + +payload="$(cat)" + +tool_name="$(printf '%s' "${payload}" | jq -r '.tool_name // ""')" +command="$(printf '%s' "${payload}" | jq -r '.tool_input.command // ""')" + +deny() { + printf 'Tier-1 sandbox blocks: %s\n' "$1" >&2 + exit 2 +} + +if [ "${tool_name}" = "Bash" ] && [ -n "${command}" ]; then + case "${command}" in + *dbt[[:space:]]run*|*dbt[[:space:]]test*|*dbt[[:space:]]parse*|*dbt[[:space:]]compile*|*dbt[[:space:]]docs*) + deny "dbt subcommand regenerates frozen artifacts or hits a warehouse (matched in: ${command})" + ;; + duckdb|duckdb[[:space:]]*|*[[:space:]]duckdb|*[[:space:]]duckdb[[:space:]]*) + deny "Direct SQL client 'duckdb' (use Recce MCP query instead; matched in: ${command})" + ;; + psql|psql[[:space:]]*|*[[:space:]]psql|*[[:space:]]psql[[:space:]]*) + deny "Direct SQL client 'psql' (use Recce MCP query instead; matched in: ${command})" + ;; + snowsql|snowsql[[:space:]]*|*[[:space:]]snowsql|*[[:space:]]snowsql[[:space:]]*) + deny "Direct SQL client 'snowsql' (use Recce MCP query instead; matched in: ${command})" + ;; + bq|bq[[:space:]]*|*[[:space:]]bq|*[[:space:]]bq[[:space:]]*) + deny "Direct SQL client 'bq' (use Recce MCP query instead; matched in: ${command})" + ;; + esac +fi + +exit 0 diff --git a/evals/agent-blind-spots/runner-configs/claude-code/tier-1/.claude/settings.json b/evals/agent-blind-spots/runner-configs/claude-code/tier-1/.claude/settings.json new file mode 100644 index 0000000..d4d436e --- /dev/null +++ b/evals/agent-blind-spots/runner-configs/claude-code/tier-1/.claude/settings.json @@ -0,0 +1,30 @@ +{ + "$comment": "Tier-1 sandbox profile for the /recce-verify v1 eval. Tier 1 is Tier 0 plus Recce CLI, Recce MCP, and single-env warehouse credentials (read-only on the dev environment). Drop this .claude/ directory into the per-fixture working tree (.tmp/sources//) before invoking Claude Code. See ../../../ENFORCEMENT.md for the runner recipe.", + "permissions": { + "deny": [ + "Bash(dbt run*)", + "Bash(dbt test*)", + "Bash(dbt parse*)", + "Bash(dbt compile*)", + "Bash(dbt docs*)", + "Bash(duckdb *)", + "Bash(psql *)", + "Bash(snowsql *)", + "Bash(bq *)" + ] + }, + "hooks": { + "PreToolUse": [ + { + "matcher": "Bash", + "hooks": [ + { + "type": "command", + "command": "bash \"$CLAUDE_PROJECT_DIR/.claude/hooks/deny-tier-1.sh\"", + "timeout": 5 + } + ] + } + ] + } +} diff --git a/evals/agent-blind-spots/runner-configs/codex/tier-0/README.md b/evals/agent-blind-spots/runner-configs/codex/tier-0/README.md new file mode 100644 index 0000000..5e1b403 --- /dev/null +++ b/evals/agent-blind-spots/runner-configs/codex/tier-0/README.md @@ -0,0 +1,54 @@ +# Codex — Tier-0 sandbox profile + +Codex (OpenAI CLI) does not load `.claude/settings.json`. Enforcement at Tier 0 is a combination of: + +1. **Process sandbox** via `--sandbox=read-only` — blocks any write the agent attempts outside the workspace, which already blocks `dbt run`/`test`/`parse`/`compile`/`docs generate` (all write `target/`) and any `recce` invocation that wants to mutate state under `~/.recce/`. +2. **MCP allowlist** — invoke Codex with a `config.toml` that registers zero MCP servers (see `config.toml` in this directory). The Recce MCP server is not reachable to the agent. +3. **PATH scrub** — run Codex with a sanitised `PATH` so Recce-related binaries are not on it. This catches read-only Recce CLI calls (e.g., `recce list`) that the process sandbox would otherwise allow. +4. **Per-fixture working directory** — Codex runs with cwd at `.tmp/sources//` (per-fixture standalone repo from `build_fixtures.sh`). Sibling fixtures, `evals/agent-blind-spots/RUBRIC.md`, and `fixtures//README.md` are outside the workspace and not in `cwd`, so file reads can't reach them. Make sure you do **not** launch Codex from the eval repo root. + +## Invocation recipe + +```bash +SLUG="pr1-fix-clv" +WT_ROOT="$(git rev-parse --show-toplevel)" +FIXTURE_DIR="${WT_ROOT}/evals/agent-blind-spots/.tmp/sources/${SLUG}" + +# Strip Recce + dbt binaries from PATH so even read-only CLI calls fail. +SAFE_PATH="$(echo "$PATH" \ + | tr ':' '\n' \ + | grep -v -E '/recce(/|$)|/dbt(/|$)|\.recce' \ + | paste -sd: -)" + +cd "${FIXTURE_DIR}" + +PATH="${SAFE_PATH}" \ +RECCE_API_TOKEN="" \ +DBT_PROFILES_DIR="" \ +codex exec \ + --sandbox=read-only \ + --ask-for-approval=never \ + --config "${WT_ROOT}/evals/agent-blind-spots/runner-configs/codex/tier-0/config.toml" \ + "" +``` + +`RECCE_API_TOKEN=""` and `DBT_PROFILES_DIR=""` are explicit even though they look redundant — leaving real values in the parent shell silently grants the sandbox more reach than intended. Make the empty-string assignments part of the invocation. + +## What's enforced vs. what's contract + +| Concern | Enforced by | Notes | +|---|---|---| +| Cannot regenerate frozen artifacts (`manifest`, `compiled`, `catalog`) | `--sandbox=read-only` (no `target/` writes) + PATH scrub (no `dbt` binary) | Belt-and-suspenders | +| Cannot reach Recce MCP | `config.toml` registers zero MCP servers | The `mcp_servers` table is intentionally empty in `config.toml` | +| Cannot invoke `recce` CLI | PATH scrub | Process sandbox alone doesn't help for read-only Recce calls | +| Cannot reach a warehouse | No warehouse credentials in the eval shell + read-only sandbox + PATH scrub of SQL clients | Recipe should never `export SNOWFLAKE_PASSWORD=...` for Tier 0 | +| Cannot read spoiler files (`fixtures//README.md`, `RUBRIC.md`, sibling fixtures) | cwd at `.tmp/sources//` (a separate git repo from the eval host) | Out-of-workspace paths are not in the read sandbox | + +## Verification check + +After the run, inspect the agent transcript for: + +- Any `mcp__recce__*` tool reference → leak (should not appear; MCP server isn't registered) +- Any `recce` shell call → check exit was non-zero (PATH scrub) +- Any read of `../README.md` or `../../RUBRIC.md` → leak (cwd should prevent this) +- Any `dbt` shell call → check exit was non-zero diff --git a/evals/agent-blind-spots/runner-configs/codex/tier-0/config.toml b/evals/agent-blind-spots/runner-configs/codex/tier-0/config.toml new file mode 100644 index 0000000..49ff275 --- /dev/null +++ b/evals/agent-blind-spots/runner-configs/codex/tier-0/config.toml @@ -0,0 +1,14 @@ +# Codex Tier-0 sandbox profile config for the /recce-verify v1 eval. +# +# The `mcp_servers` table is intentionally empty: a Tier-0 agent must +# not have any path to a Recce MCP tool. Do not add entries here. +# +# Pair this config with `--sandbox=read-only`, a PATH scrub that drops +# `recce` and `dbt`, and cwd set to `.tmp/sources//`. See +# README.md alongside this file for the full invocation recipe. + +[sandbox] +mode = "read-only" + +[mcp_servers] +# (intentionally empty) diff --git a/evals/agent-blind-spots/runner-configs/codex/tier-1/README.md b/evals/agent-blind-spots/runner-configs/codex/tier-1/README.md new file mode 100644 index 0000000..7c9deab --- /dev/null +++ b/evals/agent-blind-spots/runner-configs/codex/tier-1/README.md @@ -0,0 +1,61 @@ +# Codex — Tier-1 sandbox profile + +Tier 1 = Tier 0 plus Recce CLI, Recce MCP, and single-env warehouse credentials (read-only on the dev environment). Base/prod environment access stays denied — Tier 2 territory, out of v1 scope. + +## What changes from Tier 0 + +| Layer | Tier 0 | Tier 1 | +|---|---|---| +| Process sandbox | `--sandbox=read-only` | `--sandbox=workspace-write` (Recce writes to `~/.recce/`, log files, etc.) | +| MCP allowlist | empty `[mcp_servers]` | `[mcp_servers.recce]` registers the local Recce MCP server | +| PATH | scrub `recce` and `dbt` | scrub `dbt` only; keep `recce` reachable | +| Warehouse env | all credentials blanked | single-env credentials present (e.g., `SNOWFLAKE_*` for the dev account); base/prod credentials must be **absent** | +| `dbt` binary | not on PATH | not on PATH (Recce reads frozen artifacts; agent never needs to regenerate them) | + +## Invocation recipe + +```bash +SLUG="pr1-fix-clv" +WT_ROOT="$(git rev-parse --show-toplevel)" +FIXTURE_DIR="${WT_ROOT}/evals/agent-blind-spots/.tmp/sources/${SLUG}" + +# Strip dbt only — recce stays on PATH for Tier 1. +SAFE_PATH="$(echo "$PATH" \ + | tr ':' '\n' \ + | grep -v -E '/dbt(/|$)' \ + | paste -sd: -)" + +# Load dev-environment warehouse credentials here. Do NOT also export +# base/prod credentials in this shell — Tier 1 is single-env. +# export SNOWFLAKE_USER=... +# export SNOWFLAKE_PASSWORD=... +# export SNOWFLAKE_ACCOUNT=... +# (or equivalent for your warehouse) + +cd "${FIXTURE_DIR}" + +PATH="${SAFE_PATH}" \ +codex exec \ + --sandbox=workspace-write \ + --ask-for-approval=never \ + --config "${WT_ROOT}/evals/agent-blind-spots/runner-configs/codex/tier-1/config.toml" \ + "" +``` + +## What's enforced vs. what's contract + +| Concern | Enforced by | Notes | +|---|---|---| +| Cannot regenerate frozen artifacts | PATH scrub (no `dbt` binary) | Workspace-write would otherwise allow `target/` writes | +| Cannot reach a base/prod environment | No base/prod credentials exported in the eval shell | Contract-based; the runner must verify the shell env before launching Codex | +| Cannot read spoiler files | cwd at `.tmp/sources//` | Same as Tier 0 | +| Recce MCP available | `config.toml` registers the local Recce MCP server | The exact `command`/`args` for the server are environment-specific; fill in the template before running | + +## Verification check + +After the run, inspect the agent transcript for: + +- `mcp__recce__*` tool calls completed successfully → expected at Tier 1 +- Any reference to a base/prod profile or to `target-base/` → leak; the agent constructed a Tier-2 path that should not be available +- Any `dbt` shell call → check exit was non-zero (PATH scrub) +- Any read of `../README.md` or `../../RUBRIC.md` → leak (cwd should prevent this) diff --git a/evals/agent-blind-spots/runner-configs/codex/tier-1/config.toml b/evals/agent-blind-spots/runner-configs/codex/tier-1/config.toml new file mode 100644 index 0000000..ea8bb54 --- /dev/null +++ b/evals/agent-blind-spots/runner-configs/codex/tier-1/config.toml @@ -0,0 +1,35 @@ +# Codex Tier-1 sandbox profile config for the /recce-verify v1 eval. +# +# Tier 1 = Tier 0 plus Recce CLI, Recce MCP, and single-env warehouse +# credentials. Base/prod environment access stays denied — that's Tier +# 2 territory, out of v1 scope. +# +# The `[mcp_servers.recce]` entry below is a template: the exact +# `command` and `args` depend on how Recce is installed in the eval +# runner's environment. Fill it in before launching Codex. +# +# Pair this config with `--sandbox=workspace-write`, a PATH scrub that +# drops `dbt` (Recce reads the frozen artifacts; the agent never needs +# to regenerate them), and cwd set to `.tmp/sources//`. See +# README.md alongside this file for the full invocation recipe. + +[sandbox] +mode = "workspace-write" + +[mcp_servers.recce] +# TEMPLATE — fill in the launcher for your local Recce MCP server before +# running the eval. The two common shapes are: +# +# command = "recce" +# args = ["mcp", "--transport", "stdio"] +# +# or, if pinning a specific version under a virtualenv: +# +# command = "/path/to/.venv/bin/python" +# args = ["-m", "recce.mcp", "--transport", "stdio"] +# +# Leaving `command` empty is intentional — a half-filled MCP entry is +# more dangerous than no entry at all because the eval would silently +# proceed without Recce reachable. +command = "" +args = [] diff --git a/evals/agent-blind-spots/runs/2026-05-28/sandbox-verification.md b/evals/agent-blind-spots/runs/2026-05-28/sandbox-verification.md new file mode 100644 index 0000000..9ed730f --- /dev/null +++ b/evals/agent-blind-spots/runs/2026-05-28/sandbox-verification.md @@ -0,0 +1,102 @@ +# Sandbox verification — 2026-05-28 + +Per [DRC-3584](https://linear.app/recce/issue/DRC-3584) acceptance criterion #2: one fixture × {Claude Code, Codex} × {Tier-0, Tier-1} verified by hand, with agent traces inspected to confirm enforcement actually fires. + +**Fixture:** `pr1-fix-clv`. **Worktree:** `.claude/worktrees/drc-3584-sandbox-profiles` (commit pending). + +## Synthetic hook tests (5 cells × 2 tiers) + +Hooks fed JSON payloads on stdin; exit code + stderr captured. + +### Tier 0 hook (`deny-tier-0.sh`) + +| Input | Expected | Observed | +|---|---|---| +| `Bash` + `recce check` | exit 2, "Recce CLI invocation" | ✅ exit 2, "Tier-0 sandbox blocks: Recce CLI invocation (matched in: recce check)" | +| `mcp__recce__row_count_diff` | exit 2, "Recce MCP tool" | ✅ exit 2, "Tier-0 sandbox blocks: Recce MCP tool 'mcp__recce__row_count_diff' (Tier-0 disallows Recce)" | +| `Skill` + `recce-verify` | exit 2, "Recce skill" | ✅ exit 2, "Tier-0 sandbox blocks: Recce skill 'recce-verify' (Tier-0 disallows /recce-* skills)" | +| `Bash` + `dbt parse --target dev` | exit 2, "dbt subcommand" | ✅ exit 2, "Tier-0 sandbox blocks: dbt subcommand regenerates frozen Tier-0 artifacts or hits a warehouse (matched in: dbt parse --target dev)" | +| `Bash` + `git diff HEAD` (benign) | exit 0 | ✅ exit 0 | +| `Read` + `/some/file` (benign) | exit 0 | ✅ exit 0 | + +### Tier 1 hook (`deny-tier-1.sh`) + +| Input | Expected | Observed | +|---|---|---| +| `Bash` + `recce check` (now allowed) | exit 0 | ✅ exit 0 | +| `mcp__recce__row_count_diff` (now allowed) | exit 0 | ✅ exit 0 | +| `Bash` + `dbt parse` (still blocked) | exit 2 | ✅ exit 2, "Tier-1 sandbox blocks: dbt subcommand regenerates frozen artifacts or hits a warehouse (matched in: dbt parse)" | +| `Bash` + `psql -h host -c "select 1"` (still blocked) | exit 2 | ✅ exit 2, "Tier-1 sandbox blocks: Direct SQL client 'psql' (use Recce MCP query instead; matched in: psql -h host -c \"select 1\")" | + +## End-to-end cell: Claude Code × Tier 0 + +Real `claude --print` invocation with cwd at the per-fixture working tree. + +**Setup:** + +```bash +SLUG=pr1-fix-clv +WT=/Users/evenwei/InfuseAI/recce-claude-plugin/.claude/worktrees/drc-3584-sandbox-profiles +FIXTURE=$WT/evals/agent-blind-spots/.tmp/sources/$SLUG +TIER=$WT/evals/agent-blind-spots/runner-configs/claude-code/tier-0 + +cp -r $TIER/.claude $FIXTURE/ + +cd $FIXTURE +claude --print --permission-mode acceptEdits \ + "Run the bash command 'recce check' and report what happened, including any error message." +``` + +**Agent trace (excerpt):** + +``` +The command was blocked by a sandbox hook. + +**What happened:** +- Command attempted: `recce check` +- Result: Blocked before execution + +**Error message:** +PreToolUse:Bash hook error: [bash "$CLAUDE_PROJECT_DIR/.claude/hooks/deny-tier-0.sh"]: +Tier-0 sandbox blocks: Recce CLI invocation (matched in: recce check) +``` + +**Result:** ✅ PASS — the PreToolUse hook fired exactly as designed; the agent recognised the block and surfaced it in its own output. + +## Cells deferred to operator (acceptance #2 leftover) + +The remaining three cells need an environment with Recce MCP reachable and the `codex` CLI configured against the same fixture working tree. Run them once before declaring DRC-3584 done: + +| Cell | What to check | Pass condition | +|---|---|---| +| Claude Code × Tier 1 | `mcp__recce__*` call from the agent succeeds; raw `psql`/`dbt parse` blocked | At least one Recce MCP tool returns data; the dbt/SQL attempts surface the Tier-1 deny message | +| Codex × Tier 0 | `recce check` in Bash → "command not found" (PATH scrub); `mcp__recce__*` unresolved (empty MCP allowlist) | Both attempts fail; transcript shows the failure surface | +| Codex × Tier 1 | `mcp__recce__*` from the agent succeeds; `dbt parse` → "command not found" | Recce MCP works; dbt is unreachable | + +Recipes for each cell: see [`../../ENFORCEMENT.md`](../../ENFORCEMENT.md) (Claude Code) and [`../../runner-configs/codex/tier-{0,1}/README.md`](../../runner-configs/codex/) (Codex). + +## DRC-3430 strip — verified concurrently + +Build_fixtures.sh re-run end-to-end (all 6 fixtures): + +``` +OK pr1-fix-clv +OK pr2-refactor-cte-to-models +OK pr3-amount-double-to-decimal +OK pr42-is-closed-filter +OK pr44-promotion-flags +OK pr46-net-clv-segments +``` + +Post-build leak audit: + +``` +$ find .tmp/sources -path '*/.git' -prune -o \ + \( -name 'recce.yml' -o -name 'claude.yml' -o -path '*/.github/prompts*' -o -name 'recce-*.yml' \) -print +(empty) + +$ grep -rlE --exclude-dir=.git 'mcp__recce__|RECCE_API_TOKEN|recce\.yml' .tmp/sources +(empty) +``` + +Initial run surfaced `claude.yml` (a "Claude Code + Recce MCP" reviewer workflow that wasn't on the original strip list) plus a false positive on `.git/index`; fixed inline in `build_fixtures.sh` before this run. diff --git a/evals/agent-blind-spots/templates/tier-0-baseline.md b/evals/agent-blind-spots/templates/tier-0-baseline.md index 3f28272..9c25fa1 100644 --- a/evals/agent-blind-spots/templates/tier-0-baseline.md +++ b/evals/agent-blind-spots/templates/tier-0-baseline.md @@ -39,3 +39,14 @@ Verbatim. The baseline reasoning is the thing Recce's evidence will or won't shi ## Notes Anything weird about the run worth flagging: hallucinations, refusal, looping, contradictory statements, references to information the agent didn't actually have, etc. + +### Sandbox profile used + +Required for every baseline (Tier 0 and Tier 1). Without this block the baseline is unreproducible and disqualified — see `ENFORCEMENT.md`. + +- Agent runtime: `` +- Tier: `<0 | 1>` +- Profile source: `>` +- Enforcement mechanism summary: `` +- Trace evidence enforcement fired: `` +- Deviation from the recipe: `` From e91ec19e2c6fda93328bb9e07f238c1a8b2d37a9 Mon Sep 17 00:00:00 2001 From: even-wei Date: Thu, 28 May 2026 17:12:39 +0800 Subject: [PATCH 02/13] =?UTF-8?q?fix(evals):=20address=20PR=20#36=20review?= =?UTF-8?q?=20=E2=80=94=20Python=20hooks=20+=20expanded=20strip=20(DRC-358?= =?UTF-8?q?4,=20DRC-3430)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review found 6 BLOCKERs + 4 ISSUEs + 3 NOTEs against the v1 sandbox profiles. Every blocker was an exit-0 from the case-glob hook on an input the rubric said must be exit-2. This commit collapses them. Hook rewrite — Python with shlex tokenization (B1+B2+B3+B4+B5+I10): - New `deny-tier-{0,1}.py` (replacing `.sh`). Splits commands on shell metacharacters (`;`, `&`, `|`, `()`, newline) plus `$()` and backtick substitutions, shlex-parses each segment, basenames the executable (so `/usr/local/bin/recce` is caught), and recurses into `sh -c` / `bash -lc` / `zsh -c` arguments. Skill match is now case-insensitive. MCP namespace regex broadened to `^mcp__(plugin_)?recce(_|-|$)` so `mcp__recce_dev__*` and future variants are covered. - Tier-0 flipped to a positive allowlist (git/grep/jq/file-read + POSIX text utilities). The legitimate Tier-0 surface is tiny per RUBRIC.md; allowlist is safer than chasing bypass shapes. - Tier-1 stays a denylist but the matching is principled. dbt's banned subcommands are detected anywhere after `dbt`, so `dbt --debug parse` and `dbt --profiles-dir /tmp parse` are caught. - `runs/2026-05-28/sandbox-verification.md` grew a Bypass attempts table with every reviewer shape; all rows pass exit-2 against v2. ENFORCEMENT.md fix (B6): - Corrected the false claim that the hook blocks `Bash(cat ../../)`. The hook does not gate Read/Grep/Glob and `cat` is in the Tier-0 allowlist. Spoiler-path protection comes from cwd separation alone — the recipe's `cd "${FIXTURE_DIR}"` step is therefore not optional. Document says so now. - Reframed the `permissions.deny` + hook relationship: the hook is the load-bearing layer, not a backup. `permissions.deny` is documentation that mirrors the documented surface, not the full bypass coverage. Gitignore restructure (I8): - Renamed `runner-configs/claude-code/tier-{0,1}/.claude/` → `claude-overlay/`. Per gitignore docs, you can't re-include a file under an excluded directory, so the v1 broad un-ignore would have let arbitrary files under `.claude/` slip through. Renaming sidesteps the `.claude/` ignore rule entirely; the eval runner now copies `claude-overlay/` → `.claude/` (one extra arg to `cp -r`). Gitignore exception removed. Strip + leak-grep expansion (I9+N11): - Strip list grew with `.devcontainer/`, `.github/mcp_config.json`, `.github/workflows/{recce_*.yml, recce_*.yaml, dbt_base.yml, dbt-build-pr.yml, dbt-build-base.yml}` — surfaced by the broadened leak grep on the first re-build. - Leak grep broadened from `mcp__recce__|recce\.yml|RECCE_API_TOKEN` to case-insensitive `mcp__recce|recce\.yml|RECCE_API_TOKEN|recce` with `profiles.yml` whitelisted (`role: RECCE` is the Snowflake role name, not Recce-the-tool priming, and dbt needs the file). - `printf ' %s\n' ${leak_hits}` (unquoted) replaced with `sed 's/^/ /' <<< "${leak_hits}"` — robust against whitespace in filenames. Other doc fixes (N12, N13): - Tier-1 hook header notes Recce MCP is intentionally not gated. - `runner-configs/README.md` calls out that `permissions.deny` and the hook now intentionally cover different surfaces. - Maintenance section updated for the v2 layout. Build verified end-to-end after every change. The Bypass attempts table in `runs/2026-05-28/sandbox-verification.md` covers all 16 hook unit tests plus 1 end-to-end claude --print cell — every row green. Signed-off-by: even-wei --- .gitignore | 7 - evals/agent-blind-spots/ENFORCEMENT.md | 9 +- evals/agent-blind-spots/build_fixtures.sh | 89 ++++++--- evals/agent-blind-spots/fixtures/README.md | 7 +- .../runner-configs/README.md | 35 +++- .../tier-0/.claude/hooks/deny-tier-0.sh | 71 ------- .../claude-overlay/hooks/deny-tier-0.py | 176 ++++++++++++++++++ .../{.claude => claude-overlay}/settings.json | 7 +- .../tier-1/.claude/hooks/deny-tier-1.sh | 56 ------ .../claude-overlay/hooks/deny-tier-1.py | 163 ++++++++++++++++ .../{.claude => claude-overlay}/settings.json | 5 +- .../runs/2026-05-28/sandbox-verification.md | 113 +++++++---- 12 files changed, 526 insertions(+), 212 deletions(-) delete mode 100755 evals/agent-blind-spots/runner-configs/claude-code/tier-0/.claude/hooks/deny-tier-0.sh create mode 100755 evals/agent-blind-spots/runner-configs/claude-code/tier-0/claude-overlay/hooks/deny-tier-0.py rename evals/agent-blind-spots/runner-configs/claude-code/tier-0/{.claude => claude-overlay}/settings.json (63%) delete mode 100755 evals/agent-blind-spots/runner-configs/claude-code/tier-1/.claude/hooks/deny-tier-1.sh create mode 100755 evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/hooks/deny-tier-1.py rename evals/agent-blind-spots/runner-configs/claude-code/tier-1/{.claude => claude-overlay}/settings.json (82%) diff --git a/.gitignore b/.gitignore index 06ffee9..7f5fa70 100644 --- a/.gitignore +++ b/.gitignore @@ -54,13 +54,6 @@ evals/agent-blind-spots/fixtures/*/artifacts/ # Claude Code local state (eval runs, transcripts, etc.). # Note: `.claude-plugin/` is a different path and remains tracked. .claude/ -# Exception: Tier-0 / Tier-1 sandbox profile templates for the -# /recce-verify v1 eval (DRC-3584). These are committed templates the -# eval runner copies into per-fixture working trees; they happen to -# live under a `.claude/` subpath because they mirror the layout the -# consumer expects after the copy. -!evals/agent-blind-spots/runner-configs/**/.claude/ -!evals/agent-blind-spots/runner-configs/**/.claude/** # Local recce config (project-specific, not for the marketplace repo) recce.yml diff --git a/evals/agent-blind-spots/ENFORCEMENT.md b/evals/agent-blind-spots/ENFORCEMENT.md index efaefd2..f177169 100644 --- a/evals/agent-blind-spots/ENFORCEMENT.md +++ b/evals/agent-blind-spots/ENFORCEMENT.md @@ -20,7 +20,10 @@ TIER_DIR="${WT_ROOT}/evals/agent-blind-spots/runner-configs/claude-code/tier-0" ( cd "${WT_ROOT}/evals/agent-blind-spots" && ./build_fixtures.sh ) # 2. Stamp the per-fixture working tree with the Tier-0 sandbox config. -cp -r "${TIER_DIR}/.claude" "${FIXTURE_DIR}/" +# `claude-overlay/` becomes `.claude/` inside the fixture — the source +# directory is named differently so it isn't swallowed by the repo's +# `.claude/` gitignore rule. +cp -r "${TIER_DIR}/claude-overlay" "${FIXTURE_DIR}/.claude" # 3. Neuter user-level Claude Code settings so a stray ~/.claude/settings.json # can't widen the sandbox. (Skip this step at your own risk.) @@ -37,7 +40,7 @@ cd "${FIXTURE_DIR}" claude "" ``` -The Tier-0 `.claude/settings.json` declares `permissions.deny` for Recce MCP namespaces and the dbt-regen / SQL-client Bash patterns, and registers a `PreToolUse` hook (`deny-tier-0.sh`) as belt-and-suspenders behind that — see `runner-configs/README.md` for the rationale. +The Tier-0 `.claude/settings.json` declares `permissions.deny` rules for the documented Recce MCP namespaces and dbt/SQL-client Bash patterns, and registers a `PreToolUse` hook (`deny-tier-0.py`). The hook is the **load-bearing layer** — `permissions.deny` is conceded to be unreliable (Claude Code [issue #6699](https://github.com/anthropics/claude-code/issues/6699)) and uses shell-glob matching whose surface differs from the hook's tokenizer-based check. Treat `permissions.deny` as documentation; trust the hook. See `runner-configs/README.md` for the divergence details. ### Tier 1 @@ -68,7 +71,7 @@ The runner therefore **must** launch the agent with cwd set to the per-fixture w This complements the cwd separation: -* Claude Code: the project-level `.claude/settings.json` lives inside the per-fixture worktree, so it travels with the agent's cwd. The PreToolUse hook also blocks any `Bash(cat ../../)` attempt. +* Claude Code: the project-level `.claude/settings.json` lives inside the per-fixture worktree, so it travels with the agent's cwd. **Spoiler-path protection comes from cwd alone** — the PreToolUse hook does not gate `Read`/`Grep`/`Glob` and the Tier-0 Bash allowlist includes `cat`. If the runner mistakenly launches `claude` from the eval host repo root, the agent can read `RUBRIC.md` and the per-fixture spoiler README. The recipe above (step 5: `cd "${FIXTURE_DIR}"`) is therefore not optional. * Codex: the process sandbox is anchored on cwd; absolute paths outside cwd require approval at Tier-0 read-only mode and writes are uniformly denied. ## Recording in the baseline diff --git a/evals/agent-blind-spots/build_fixtures.sh b/evals/agent-blind-spots/build_fixtures.sh index 15a7373..138a5df 100755 --- a/evals/agent-blind-spots/build_fixtures.sh +++ b/evals/agent-blind-spots/build_fixtures.sh @@ -265,44 +265,87 @@ build_fixture() { # for Recce-aware PR review (lists mcp__recce__ tool names, # prescribed call sequences, numeric anchors for the affected # column) - # * .github/workflows/recce-*.yml — Recce CI workflows + # * .github/workflows/recce-*.yml, recce_*.yml — Recce CI workflows # * .github/workflows/claude.yml — "Claude Code + Recce MCP" # reviewer workflow (primes the agent with the same playbook) + # * .github/workflows/dbt-build-{pr,base}.yml, dbt_base.yml — + # dbt CI that uses `DataRecce/recce-cloud-cicd-action` (the + # names are dbt-shaped but the steps wire Recce in) + # * .github/mcp_config.json — explicit Recce MCP registration + # * .devcontainer/ — Recce-specific dev container + post-start + # script that boots Recce # * recce.yml — preset Recce checks - # A random dbt project in the wild would not have these; their - # presence here is fixture-source-specific, not a contract bug. - # Strip them so a Tier-0 baseline measures what the agent deduces - # without Recce-style structured surfacing. + # + # A random dbt project in the wild would not have any of these; + # their presence here is fixture-source-specific, not a contract + # bug. Strip them so a Tier-0 baseline measures what the agent + # deduces without Recce-style structured surfacing. local stripped_paths=( ".github/prompts" + ".github/mcp_config.json" ".github/workflows/claude.yml" + ".github/workflows/recce_ci.yml" + ".github/workflows/dbt-build-pr.yml" + ".github/workflows/dbt-build-base.yml" + ".github/workflows/dbt_base.yml" + ".devcontainer" "recce.yml" ) for stripped in "${stripped_paths[@]}"; do rm -rf "${source_dir:?}/${stripped}" done - # Glob removal: any .github/workflows/recce-*.{yml,yaml} — guard - # against an empty match because `nullglob` isn't set globally. - if compgen -G "${source_dir}/.github/workflows/recce-*.yml" > /dev/null; then - rm -f "${source_dir}"/.github/workflows/recce-*.yml - fi - if compgen -G "${source_dir}/.github/workflows/recce-*.yaml" > /dev/null; then - rm -f "${source_dir}"/.github/workflows/recce-*.yaml - fi + # Glob removal: any .github/workflows/recce-*.{yml,yaml} or + # recce_*.{yml,yaml}. Guard against empty matches because + # `nullglob` isn't set globally. + for glob in \ + "${source_dir}/.github/workflows/recce-*.yml" \ + "${source_dir}/.github/workflows/recce-*.yaml" \ + "${source_dir}/.github/workflows/recce_*.yml" \ + "${source_dir}/.github/workflows/recce_*.yaml" + do + if compgen -G "${glob}" > /dev/null; then + rm -f ${glob} + fi + done - # DRC-3430 belt-and-suspenders: grep for known Recce-shaped strings - # in the stripped working tree. Catches future regressions where a - # new Recce-aware file lands at a path the strip list doesn't - # cover. Scope the grep to the working tree (`--exclude-dir=.git`) - # so packed objects under .git/ don't false-positive — the agent - # cannot read packed git internals without first invoking a git - # command, and even then sees decoded content, not raw byte - # strings. + # DRC-3430 belt-and-suspenders: grep for Recce-shaped strings in the + # stripped working tree. Catches future regressions where a new + # Recce-aware file lands at a path the strip list doesn't cover. + # + # Two layers: + # + # (a) Tight identifier match — MCP namespace, recce.yml filename, + # known env vars. False-positive-free, but only catches the + # exact shapes we know about. + # (b) Loose `[Rr]ecce` substring — catches natural-language + # priming like a CI workflow that says + # `prompt: "Use the recce CLI to review this PR"`. False + # positives are possible on a fresh source tree (a model + # column called `recce_score`, say); when one surfaces we + # either expand the strip list or whitelist the path here. + # + # Both grep over the working tree only (`--exclude-dir=.git`) so + # packed objects don't false-positive — the agent cannot read those + # without first invoking a git command, and even then sees decoded + # content, not raw byte strings. + # `profiles.yml` is whitelisted: the upstream Snowflake role is + # literally named "RECCE", which is unfortunate naming, not Recce + # priming. A role name doesn't tell the agent how to use Recce- + # the-tool. Whitelisting one filename is preferable to tightening + # the regex back to identifier-only matches (which would let + # natural-language priming like `prompt: "use the recce CLI"` + # through). local leak_hits - leak_hits="$(grep -rlE --exclude-dir=.git 'mcp__recce__|recce\.yml|RECCE_API_TOKEN' "${source_dir}" 2>/dev/null || true)" + leak_hits="$(grep -rlEi --exclude-dir=.git --exclude=profiles.yml \ + 'mcp__recce|recce\.yml|RECCE_API_TOKEN|recce' \ + "${source_dir}" 2>/dev/null || true)" if [[ -n "${leak_hits}" ]]; then echo "FAIL ${slug} (Tier-0 strip leak — Recce-shaped strings still present in:)" >&2 - printf ' %s\n' ${leak_hits} >&2 + # Indent each path on its own line; sed is robust against + # whitespace in filenames where `printf ' %s\n' ${unquoted}` + # would word-split. dbt projects don't use spaces in paths + # today, but the cost of belt-and-suspenders is one sed call. + sed 's/^/ /' <<< "${leak_hits}" >&2 echo " Extend the strip list in build_fixtures.sh and re-run." >&2 return 1 fi diff --git a/evals/agent-blind-spots/fixtures/README.md b/evals/agent-blind-spots/fixtures/README.md index 07bb9c5..cf52a3a 100644 --- a/evals/agent-blind-spots/fixtures/README.md +++ b/evals/agent-blind-spots/fixtures/README.md @@ -20,10 +20,15 @@ Re-running is idempotent — existing `artifacts/` directories are removed and r `build_fixtures.sh` materialises each fixture's head-SHA checkout at `.tmp/sources//` (a freshly-initialised standalone git repo with one commit, zero remotes — see the in-script comment for the leak-proofing rationale). Immediately after checkout, the build script **strips Recce-aware automation** from that working tree so a Tier-0 agent reading the source cannot find the with-Recce answer pre-baked into the repo (see [DRC-3430](https://linear.app/recce/issue/DRC-3430)): - `.github/prompts/` — Recce-aware GitHub Action system prompt + worked-example output tables (names `mcp__recce__` tools, prescribes call sequences, anchors numeric values for the column the fixture's PR affects) -- `.github/workflows/recce-*.yml` and `recce-*.yaml` — Recce CI workflow definitions +- `.github/workflows/recce-*.{yml,yaml}` and `recce_*.{yml,yaml}` — Recce CI workflow definitions - `.github/workflows/claude.yml` — "Claude Code + Recce MCP" reviewer workflow (primes the agent with the same playbook) +- `.github/workflows/dbt_base.yml`, `dbt-build-pr.yml`, `dbt-build-base.yml` — dbt CI workflows that wire `DataRecce/recce-cloud-cicd-action` in (the names look generic but the steps prime Recce) +- `.github/mcp_config.json` — explicit Recce MCP server registration +- `.devcontainer/` — Recce-specific dev container plus a post-start script that boots Recce - `recce.yml` — preset Recce check definitions +`profiles.yml` is **not** stripped despite carrying the literal string `RECCE` (the upstream Snowflake role name) — it's required for dbt to parse, and a role name is not Recce-the-tool priming. The post-build leak grep whitelists it via `--exclude=profiles.yml`. + The strip is followed by a `grep -E 'mcp__recce__|recce\.yml|RECCE_API_TOKEN'` belt-and-suspenders sweep over the source tree; the build fails fast (`FAIL `) if any new Recce-shaped file slips in at an unanticipated path. When that happens, extend the strip list in `build_fixtures.sh` and re-run. This is fixture-source-specific, not a contract bug: `DataRecce/jaffle_shop_golden` is a Recce-dogfood repo and the stripped paths are the Recce team's own automation. A random dbt project in the wild would not have them. See [`../ENFORCEMENT.md`](../ENFORCEMENT.md) for how the sandbox profiles relate to this strip. diff --git a/evals/agent-blind-spots/runner-configs/README.md b/evals/agent-blind-spots/runner-configs/README.md index ec2dac8..7a78aa6 100644 --- a/evals/agent-blind-spots/runner-configs/README.md +++ b/evals/agent-blind-spots/runner-configs/README.md @@ -9,13 +9,13 @@ runner-configs/ ├── README.md ← (this file) ├── claude-code/ │ ├── tier-0/ -│ │ └── .claude/ -│ │ ├── settings.json ← permissions.deny + PreToolUse hook -│ │ └── hooks/deny-tier-0.sh ← exit-2 block on Recce + dbt regen + SQL clients +│ │ └── claude-overlay/ ← copied to /.claude/ by the runner +│ │ ├── settings.json ← permissions.deny (documentation) + PreToolUse hook +│ │ └── hooks/deny-tier-0.py ← exit-2 block, positive Bash allowlist │ └── tier-1/ -│ └── .claude/ -│ ├── settings.json ← narrower deny (Recce allowed; dbt regen + SQL clients still denied) -│ └── hooks/deny-tier-1.sh +│ └── claude-overlay/ +│ ├── settings.json ← narrower deny (Recce allowed; dbt regen + SQL clients denied) +│ └── hooks/deny-tier-1.py ← exit-2 block, tokenised denylist └── codex/ ├── tier-0/ │ ├── README.md ← invocation recipe (PATH scrub + sandbox flag + cwd) @@ -25,6 +25,8 @@ runner-configs/ └── config.toml ← template mcp_servers.recce entry to fill in ``` +The template directory is named `claude-overlay/` rather than `.claude/` so it isn't swallowed by the repo's `.claude/` gitignore rule. The eval runner renames it to `.claude/` when it copies the overlay into the per-fixture working tree (see Quick start below). + ## Quick start **Claude Code, Tier 0:** @@ -35,7 +37,7 @@ WT_ROOT="$(git rev-parse --show-toplevel)" FIXTURE_DIR="${WT_ROOT}/evals/agent-blind-spots/.tmp/sources/${SLUG}" TIER_DIR="${WT_ROOT}/evals/agent-blind-spots/runner-configs/claude-code/tier-0" -cp -r "${TIER_DIR}/.claude" "${FIXTURE_DIR}/" +cp -r "${TIER_DIR}/claude-overlay" "${FIXTURE_DIR}/.claude" # Optional but recommended — neuter user-level settings for this run: export CLAUDE_CONFIG_DIR="$(mktemp -d)" @@ -48,8 +50,23 @@ claude "" ## Why both `permissions.deny` and a `PreToolUse` hook -Claude Code's `permissions.deny` is the documented mechanism, but open issue [anthropics/claude-code#6699](https://github.com/anthropics/claude-code/issues/6699) shows it can be bypassed in some shapes. The PreToolUse hook is belt-and-suspenders: a deny-by-default exit-2 script. If `permissions.deny` regresses, the hook still blocks; if the hook regresses, `permissions.deny` still blocks. +Claude Code's `permissions.deny` uses shell-glob matching that is conceded to be unreliable: open issue [anthropics/claude-code#6699](https://github.com/anthropics/claude-code/issues/6699) shows bypasses, and the glob shape can't express the things the rubric requires (path-stripped basenames, `sh -c ""` recursion, dbt subcommands with global flags interposed, case-insensitive skill matching). + +The PreToolUse hook is therefore the **load-bearing layer**, not a backup. It uses `shlex` tokenisation, basenames each executable, recurses into shell wrappers, lowercases skill names, and matches MCP namespaces with a regex. + +**`permissions.deny` and the hook do not mirror each other**, and that's deliberate: + +* `permissions.deny` covers only what shell-glob can express crisply — the user-facing documented attack surface. +* The hook covers the full surface, including the bypass shapes the glob can't address. + +Treat `permissions.deny` as documentation for human readers. Trust the hook for enforcement. ## Maintenance -When you add a new Recce CLI verb, new MCP tool namespace, or a new dbt subcommand that mutates state, update **both** the relevant tier's `settings.json` deny list **and** the hook script's case statement. The hook is the source of truth — `settings.json` mirrors it for the documented path. +When you add a new Recce CLI verb, new MCP tool namespace, or a new dbt subcommand that mutates state: + +1. Update the hook script (`deny-tier-{0,1}.py`) — it's the source of truth. +2. Update the documented `permissions.deny` patterns to match the new shape. +3. Add the new shape to the Bypass attempts table in `runs//sandbox-verification.md` so a future regression is caught. + +The hook's MCP namespace regex is `^mcp__(plugin_)?recce(_|-|$)` — adding a `mcp__recce___*` namespace is automatically covered. New Bash basenames (e.g. a `recce` CLI rename) require a Python-level change. diff --git a/evals/agent-blind-spots/runner-configs/claude-code/tier-0/.claude/hooks/deny-tier-0.sh b/evals/agent-blind-spots/runner-configs/claude-code/tier-0/.claude/hooks/deny-tier-0.sh deleted file mode 100755 index 11e0b08..0000000 --- a/evals/agent-blind-spots/runner-configs/claude-code/tier-0/.claude/hooks/deny-tier-0.sh +++ /dev/null @@ -1,71 +0,0 @@ -#!/usr/bin/env bash -# -# Tier-0 PreToolUse hook for the /recce-verify v1 eval. -# Belt-and-suspenders alongside `permissions.deny` in ../settings.json -# (see DRC-3584 issue body: hooks are more reliable than deny rules per -# open Claude Code issue #6699). -# -# Reads the tool-use payload on stdin (JSON) and exits 2 with a stderr -# message to block when the call would let a Tier-0 agent reach for -# Recce-shaped signals or regenerate frozen Tier-0 inputs. -# -# Tier 0 denies: -# * Recce MCP tools (any mcp__recce__* or mcp__plugin_recce_*) -# * Recce CLI (`recce ...` in Bash) -# * `/recce-*` skills (invoked through the Skill tool) -# * dbt subcommands that regenerate manifest/compiled/catalog or hit -# a warehouse (`dbt run|test|parse|compile|docs`) -# * Direct SQL clients (`duckdb`, `psql`, `snowsql`, `bq`) - -set -euo pipefail - -payload="$(cat)" - -tool_name="$(printf '%s' "${payload}" | jq -r '.tool_name // ""')" -command="$(printf '%s' "${payload}" | jq -r '.tool_input.command // ""')" -skill="$(printf '%s' "${payload}" | jq -r '.tool_input.skill // ""')" - -deny() { - printf 'Tier-0 sandbox blocks: %s\n' "$1" >&2 - exit 2 -} - -# Recce MCP — any namespace variant -case "${tool_name}" in - mcp__recce__*|mcp__plugin_recce_*) - deny "Recce MCP tool '${tool_name}' (Tier-0 disallows Recce)" - ;; -esac - -# Skill invocation — /recce-* or recce:* (plugin-namespaced) -case "${skill}" in - recce-*|recce:*) - deny "Recce skill '${skill}' (Tier-0 disallows /recce-* skills)" - ;; -esac - -# Bash command inspection -if [ "${tool_name}" = "Bash" ] && [ -n "${command}" ]; then - case "${command}" in - recce|recce[[:space:]]*|*[[:space:]]recce|*[[:space:]]recce[[:space:]]*) - deny "Recce CLI invocation (matched in: ${command})" - ;; - *dbt[[:space:]]run*|*dbt[[:space:]]test*|*dbt[[:space:]]parse*|*dbt[[:space:]]compile*|*dbt[[:space:]]docs*) - deny "dbt subcommand regenerates frozen Tier-0 artifacts or hits a warehouse (matched in: ${command})" - ;; - duckdb|duckdb[[:space:]]*|*[[:space:]]duckdb|*[[:space:]]duckdb[[:space:]]*) - deny "Direct SQL client 'duckdb' (matched in: ${command})" - ;; - psql|psql[[:space:]]*|*[[:space:]]psql|*[[:space:]]psql[[:space:]]*) - deny "Direct SQL client 'psql' (matched in: ${command})" - ;; - snowsql|snowsql[[:space:]]*|*[[:space:]]snowsql|*[[:space:]]snowsql[[:space:]]*) - deny "Direct SQL client 'snowsql' (matched in: ${command})" - ;; - bq|bq[[:space:]]*|*[[:space:]]bq|*[[:space:]]bq[[:space:]]*) - deny "Direct SQL client 'bq' (matched in: ${command})" - ;; - esac -fi - -exit 0 diff --git a/evals/agent-blind-spots/runner-configs/claude-code/tier-0/claude-overlay/hooks/deny-tier-0.py b/evals/agent-blind-spots/runner-configs/claude-code/tier-0/claude-overlay/hooks/deny-tier-0.py new file mode 100755 index 0000000..f58322a --- /dev/null +++ b/evals/agent-blind-spots/runner-configs/claude-code/tier-0/claude-overlay/hooks/deny-tier-0.py @@ -0,0 +1,176 @@ +#!/usr/bin/env python3 +""" +Tier-0 PreToolUse hook for the /recce-verify v1 eval. + +Belt-and-suspenders alongside `permissions.deny` in ../settings.json +(see DRC-3584 issue body: hooks are more reliable than deny rules per +open Claude Code issue #6699). + +Tier 0 is a **positive allowlist** for Bash, with Recce MCP / Recce +skill denial layered on top. Anything outside the allowlist exits 2. + +The case-glob version of this hook (deny-tier-0.sh, removed in PR #36's +v2 commit) was bypassed by: + * `true;recce check`, `true|recce`, `(recce check)`, `$(recce ls)` — + shell separators that aren't `[[:space:]]` + * `/usr/local/bin/recce check` — absolute path on the executable + * `sh -c "recce check"`, `bash -lc "..."` — wrapper hiding inner cmd + * `dbt --debug parse` — global flag interposed before subcommand + * `Recce-verify`, `RECCE-VERIFY` — skill case-insensitivity + * `mcp__recce_dev__*` — MCP namespace shape not in the prior regex + +This rewrite tokenises by shell metacharacters (not just whitespace), +basenames each executable token (so paths don't help), recurses into +`sh -c` arg, and lowercases the skill name before matching. +""" +from __future__ import annotations + +import json +import re +import shlex +import sys +from os.path import basename + +# Bash commands a Tier-0 agent legitimately needs. Anything else denied. +# Per RUBRIC.md Tier-0 runtime contract: file read, grep/ripgrep, jq, +# git log/diff/show. The list below is that contract plus standard +# POSIX text-processing utilities for evidence analysis. +TIER_0_ALLOWLIST: frozenset[str] = frozenset({ + "git", "grep", "rg", "jq", "ls", "cat", "head", "tail", "wc", "find", + "echo", "true", "false", "awk", "sed", "sort", "uniq", "comm", "diff", + "basename", "dirname", "readlink", "pwd", "file", "stat", "test", "[", + "printf", "tr", "cut", "xargs", "tee", "less", "more", +}) + +# Shell wrappers that take `-c `. Banned outright at Tier 0 — +# refusing to interpret them is simpler than recursing, and an agent +# with a positive allowlist has no legitimate reason to invoke a shell. +SHELL_WRAPPERS: frozenset[str] = frozenset( + {"sh", "bash", "zsh", "dash", "ash", "ksh"} +) + +# Recce MCP namespaces. Covers mcp__recce__*, mcp__plugin_recce_*, +# mcp__recce_dev__*, and any future Recce-shaped namespace. +MCP_RECCE_RE = re.compile(r"^mcp__(plugin_)?recce(_|-|$)", re.IGNORECASE) + +# Recce skill prefix — `recce-verify`, `recce:recce-review`, etc. +RECCE_SKILL_RE = re.compile(r"^recce[-:]", re.IGNORECASE) + + +def deny(reason: str) -> None: + print(f"Tier-0 sandbox blocks: {reason}", file=sys.stderr) + sys.exit(2) + + +def split_segments(cmd: str) -> list[str]: + """Split a shell command into segments by separators that introduce + a new command boundary: `;`, `&`, `|`, `(`, `)`, newline. Also + extract `$(...)` and backtick subshells as their own segments — + they let the agent smuggle a command past a leading `echo`. + """ + segments: list[str] = [] + + # Extract command substitutions first (they could nest, but a + # single-level extraction defeats the obvious bypass and matches + # the threat model: an LLM is unlikely to construct deep nesting + # specifically to evade us). + for sub in re.findall(r"\$\(([^()]*)\)", cmd): + segments.extend(split_segments(sub)) + cmd = re.sub(r"\$\([^()]*\)", "", cmd) + for sub in re.findall(r"`([^`]*)`", cmd): + segments.extend(split_segments(sub)) + cmd = re.sub(r"`[^`]*`", "", cmd) + + # Now split on command-boundary separators. + parts = re.split(r"[;&|()\n]+", cmd) + for part in parts: + if part.strip(): + segments.append(part.strip()) + return segments + + +def tokenize(segment: str) -> list[str]: + try: + return shlex.split(segment) + except ValueError: + # Unbalanced quotes — treat as one opaque token. Whatever the + # agent meant, it's not a clean invocation; let the allowlist + # check it as-is. + return [segment] + + +def skip_leading_env(tokens: list[str]) -> list[str]: + """Skip `VAR=value` env assignments and `env [args] CMD ...` so the + executable check lands on the real command. + """ + i = 0 + while i < len(tokens) and "=" in tokens[i] and not tokens[i].startswith("="): + i += 1 + if i >= len(tokens): + return [] + if basename(tokens[i]) == "env": + i += 1 + # env [-i] [-u VAR ...] [NAME=value ...] command ... + while i < len(tokens): + t = tokens[i] + if t in ("-i", "-0", "--null"): + i += 1 + continue + if t in ("-u", "--unset"): + i += 2 + continue + if "=" in t and not t.startswith("="): + i += 1 + continue + break + return tokens[i:] + + +def check_bash_command(command: str) -> None: + if not command.strip(): + return + for segment in split_segments(command): + tokens = skip_leading_env(tokenize(segment)) + if not tokens: + continue + name = basename(tokens[0]) + if name in SHELL_WRAPPERS: + deny( + f"shell wrapper '{name}' is banned at Tier 0 — the " + f"agent has no legitimate reason to invoke a subshell " + f"(matched in: {command!r})" + ) + if not name: + continue + if name not in TIER_0_ALLOWLIST: + deny( + f"Bash executable '{name}' not in Tier-0 allowlist " + f"(matched in: {command!r})" + ) + + +def main() -> None: + try: + payload = json.load(sys.stdin) + except json.JSONDecodeError: + # Malformed payload — fail open so a single bad invocation + # doesn't break the whole session. The matcher in settings.json + # narrows what we see; nothing legitimate arrives malformed. + return + + tool_name = payload.get("tool_name", "") or "" + tool_input = payload.get("tool_input", {}) or {} + + if MCP_RECCE_RE.match(tool_name): + deny(f"Recce MCP tool '{tool_name}' (Tier-0 disallows Recce)") + + skill = tool_input.get("skill", "") or "" + if RECCE_SKILL_RE.match(skill): + deny(f"Recce skill '{skill}' (Tier-0 disallows /recce-* skills)") + + if tool_name == "Bash": + check_bash_command(tool_input.get("command", "") or "") + + +if __name__ == "__main__": + main() diff --git a/evals/agent-blind-spots/runner-configs/claude-code/tier-0/.claude/settings.json b/evals/agent-blind-spots/runner-configs/claude-code/tier-0/claude-overlay/settings.json similarity index 63% rename from evals/agent-blind-spots/runner-configs/claude-code/tier-0/.claude/settings.json rename to evals/agent-blind-spots/runner-configs/claude-code/tier-0/claude-overlay/settings.json index 6ac527d..bef569b 100644 --- a/evals/agent-blind-spots/runner-configs/claude-code/tier-0/.claude/settings.json +++ b/evals/agent-blind-spots/runner-configs/claude-code/tier-0/claude-overlay/settings.json @@ -1,9 +1,10 @@ { - "$comment": "Tier-0 sandbox profile for the /recce-verify v1 eval. Drop this .claude/ directory into the per-fixture working tree (.tmp/sources//) before invoking Claude Code. See ../../../ENFORCEMENT.md for the runner recipe.", + "$comment": "Tier-0 sandbox profile for the /recce-verify v1 eval. Drop this .claude/ directory into the per-fixture working tree (.tmp/sources//) before invoking Claude Code. See ../../../ENFORCEMENT.md for the runner recipe. The PreToolUse hook is the load-bearing layer; permissions.deny below is documentation-aligned but does not cover every bypass the hook does — see runner-configs/README.md.", "permissions": { "deny": [ "mcp__recce__*", "mcp__plugin_recce_*", + "mcp__recce_dev__*", "Bash(recce *)", "Bash(dbt run*)", "Bash(dbt test*)", @@ -19,11 +20,11 @@ "hooks": { "PreToolUse": [ { - "matcher": "Bash|Skill|mcp__recce__.*|mcp__plugin_recce_.*", + "matcher": "Bash|Skill|mcp__(plugin_)?recce(_|-).*", "hooks": [ { "type": "command", - "command": "bash \"$CLAUDE_PROJECT_DIR/.claude/hooks/deny-tier-0.sh\"", + "command": "python3 \"$CLAUDE_PROJECT_DIR/.claude/hooks/deny-tier-0.py\"", "timeout": 5 } ] diff --git a/evals/agent-blind-spots/runner-configs/claude-code/tier-1/.claude/hooks/deny-tier-1.sh b/evals/agent-blind-spots/runner-configs/claude-code/tier-1/.claude/hooks/deny-tier-1.sh deleted file mode 100755 index 3df94dd..0000000 --- a/evals/agent-blind-spots/runner-configs/claude-code/tier-1/.claude/hooks/deny-tier-1.sh +++ /dev/null @@ -1,56 +0,0 @@ -#!/usr/bin/env bash -# -# Tier-1 PreToolUse hook for the /recce-verify v1 eval. Tier 1 is -# Tier 0 plus Recce CLI, Recce MCP, and single-env warehouse credentials -# (read-only on the dev environment). Base/prod environment access stays -# denied — that's Tier 2 territory, out of v1 scope. -# -# Reads the tool-use payload on stdin (JSON) and exits 2 with a stderr -# message to block when the call would either regenerate frozen Tier-0 -# inputs or reach for a base/prod environment. -# -# Tier 1 denies (relative to Tier 0): -# * dbt subcommands that regenerate frozen artifacts or hit a warehouse -# directly (`dbt run|test|parse|compile|docs`). Recce reads the -# frozen artifacts; the agent never needs to regenerate them. -# * Direct SQL clients (`duckdb`, `psql`, `snowsql`, `bq`) — at Tier 1 -# warehouse access is mediated through Recce MCP tools (e.g., -# `mcp__recce__query`), never raw shell. -# -# Tier 1 explicitly ALLOWS (which Tier 0 denied): -# * Recce MCP tools and `/recce-*` skills -# * `recce` CLI - -set -euo pipefail - -payload="$(cat)" - -tool_name="$(printf '%s' "${payload}" | jq -r '.tool_name // ""')" -command="$(printf '%s' "${payload}" | jq -r '.tool_input.command // ""')" - -deny() { - printf 'Tier-1 sandbox blocks: %s\n' "$1" >&2 - exit 2 -} - -if [ "${tool_name}" = "Bash" ] && [ -n "${command}" ]; then - case "${command}" in - *dbt[[:space:]]run*|*dbt[[:space:]]test*|*dbt[[:space:]]parse*|*dbt[[:space:]]compile*|*dbt[[:space:]]docs*) - deny "dbt subcommand regenerates frozen artifacts or hits a warehouse (matched in: ${command})" - ;; - duckdb|duckdb[[:space:]]*|*[[:space:]]duckdb|*[[:space:]]duckdb[[:space:]]*) - deny "Direct SQL client 'duckdb' (use Recce MCP query instead; matched in: ${command})" - ;; - psql|psql[[:space:]]*|*[[:space:]]psql|*[[:space:]]psql[[:space:]]*) - deny "Direct SQL client 'psql' (use Recce MCP query instead; matched in: ${command})" - ;; - snowsql|snowsql[[:space:]]*|*[[:space:]]snowsql|*[[:space:]]snowsql[[:space:]]*) - deny "Direct SQL client 'snowsql' (use Recce MCP query instead; matched in: ${command})" - ;; - bq|bq[[:space:]]*|*[[:space:]]bq|*[[:space:]]bq[[:space:]]*) - deny "Direct SQL client 'bq' (use Recce MCP query instead; matched in: ${command})" - ;; - esac -fi - -exit 0 diff --git a/evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/hooks/deny-tier-1.py b/evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/hooks/deny-tier-1.py new file mode 100755 index 0000000..0e70efc --- /dev/null +++ b/evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/hooks/deny-tier-1.py @@ -0,0 +1,163 @@ +#!/usr/bin/env python3 +""" +Tier-1 PreToolUse hook for the /recce-verify v1 eval. + +Tier 1 = Tier 0 plus Recce CLI, Recce MCP, single-env warehouse +credentials (read-only on the dev environment). Base/prod environment +access stays denied — Tier 2 territory, out of v1 scope. Recce MCP is +**intentionally not gated** here — the rubric explicitly allows it. + +Tier 1 denies (relative to Tier 0): + * dbt subcommands that regenerate frozen artifacts or hit a + warehouse directly (`dbt run|test|parse|compile|docs`). Recce + reads the frozen artifacts; the agent never needs to regenerate + them, and dbt invocations would let the agent bypass Recce's + structured query surfaces. + * Direct SQL clients (`duckdb`, `psql`, `snowsql`, `bq`) — at Tier 1 + warehouse access is mediated through Recce MCP tools (e.g. + `mcp__recce__query`), never raw shell. + * `sh -c ""`, `bash -c …`, `zsh -c …` wrappers that smuggle + a denied binary past a shallow check. + +This is a denylist (Tier 0 was an allowlist) because Tier 1's +legitimate command surface is large — the agent can invoke arbitrary +git, grep, recce-*, and other dev-loop utilities. Tokenisation + +basename + `sh -c` recursion catch the bypass shapes the prior +case-glob version missed. +""" +from __future__ import annotations + +import json +import re +import shlex +import sys +from os.path import basename + +# Binaries the Tier-1 agent must not invoke directly. +DENIED_BINS: frozenset[str] = frozenset( + {"dbt", "duckdb", "psql", "snowsql", "bq"} +) + +# dbt subcommands that either regenerate frozen Tier-0 inputs or hit a +# warehouse directly. `dbt` alone (no subcommand) and `dbt --help` are +# allowed — they're discovery-only. +DBT_DENIED_SUBCOMMANDS: frozenset[str] = frozenset( + {"run", "test", "parse", "compile", "docs", "seed", "snapshot", + "build", "freshness"} +) + +SHELL_WRAPPERS: frozenset[str] = frozenset( + {"sh", "bash", "zsh", "dash", "ash", "ksh"} +) + + +def deny(reason: str) -> None: + print(f"Tier-1 sandbox blocks: {reason}", file=sys.stderr) + sys.exit(2) + + +def split_segments(cmd: str) -> list[str]: + segments: list[str] = [] + for sub in re.findall(r"\$\(([^()]*)\)", cmd): + segments.extend(split_segments(sub)) + cmd = re.sub(r"\$\([^()]*\)", "", cmd) + for sub in re.findall(r"`([^`]*)`", cmd): + segments.extend(split_segments(sub)) + cmd = re.sub(r"`[^`]*`", "", cmd) + for part in re.split(r"[;&|()\n]+", cmd): + if part.strip(): + segments.append(part.strip()) + return segments + + +def tokenize(segment: str) -> list[str]: + try: + return shlex.split(segment) + except ValueError: + return [segment] + + +def skip_leading_env(tokens: list[str]) -> list[str]: + i = 0 + while i < len(tokens) and "=" in tokens[i] and not tokens[i].startswith("="): + i += 1 + if i >= len(tokens): + return [] + if basename(tokens[i]) == "env": + i += 1 + while i < len(tokens): + t = tokens[i] + if t in ("-i", "-0", "--null"): + i += 1 + continue + if t in ("-u", "--unset"): + i += 2 + continue + if "=" in t and not t.startswith("="): + i += 1 + continue + break + return tokens[i:] + + +def has_denied_dbt_subcommand(tokens: list[str]) -> bool: + # Walk past `dbt` and look for the first positional (non-flag) token. + for tok in tokens[1:]: + if tok.startswith("-"): + continue + return tok in DBT_DENIED_SUBCOMMANDS + return False + + +def check_tokens(tokens: list[str], command_str: str) -> None: + tokens = skip_leading_env(tokens) + if not tokens: + return + name = basename(tokens[0]) + + if name in SHELL_WRAPPERS: + # Find the -c / -lc arg and recurse into its content. + for j in range(1, len(tokens) - 1): + if tokens[j] in ("-c", "-lc", "-ic"): + inner = tokens[j + 1] + for inner_segment in split_segments(inner): + check_tokens(tokenize(inner_segment), command_str) + return + # A bare `sh`/`bash` interactive subshell with no -c arg is + # not a denied command on its own. + return + + if name == "dbt": + if has_denied_dbt_subcommand(tokens): + deny( + "dbt subcommand regenerates frozen artifacts or hits a " + f"warehouse (matched in: {command_str!r})" + ) + return + + if name in DENIED_BINS: + deny( + f"Direct SQL client '{name}' — use Recce MCP query instead " + f"(matched in: {command_str!r})" + ) + + +def main() -> None: + try: + payload = json.load(sys.stdin) + except json.JSONDecodeError: + return + + if payload.get("tool_name", "") != "Bash": + return + + command = (payload.get("tool_input", {}) or {}).get("command", "") or "" + if not command.strip(): + return + + for segment in split_segments(command): + check_tokens(tokenize(segment), command) + + +if __name__ == "__main__": + main() diff --git a/evals/agent-blind-spots/runner-configs/claude-code/tier-1/.claude/settings.json b/evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/settings.json similarity index 82% rename from evals/agent-blind-spots/runner-configs/claude-code/tier-1/.claude/settings.json rename to evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/settings.json index d4d436e..be3581d 100644 --- a/evals/agent-blind-spots/runner-configs/claude-code/tier-1/.claude/settings.json +++ b/evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/settings.json @@ -7,6 +7,9 @@ "Bash(dbt parse*)", "Bash(dbt compile*)", "Bash(dbt docs*)", + "Bash(dbt seed*)", + "Bash(dbt snapshot*)", + "Bash(dbt build*)", "Bash(duckdb *)", "Bash(psql *)", "Bash(snowsql *)", @@ -20,7 +23,7 @@ "hooks": [ { "type": "command", - "command": "bash \"$CLAUDE_PROJECT_DIR/.claude/hooks/deny-tier-1.sh\"", + "command": "python3 \"$CLAUDE_PROJECT_DIR/.claude/hooks/deny-tier-1.py\"", "timeout": 5 } ] diff --git a/evals/agent-blind-spots/runs/2026-05-28/sandbox-verification.md b/evals/agent-blind-spots/runs/2026-05-28/sandbox-verification.md index 9ed730f..5cc3b0d 100644 --- a/evals/agent-blind-spots/runs/2026-05-28/sandbox-verification.md +++ b/evals/agent-blind-spots/runs/2026-05-28/sandbox-verification.md @@ -2,35 +2,69 @@ Per [DRC-3584](https://linear.app/recce/issue/DRC-3584) acceptance criterion #2: one fixture × {Claude Code, Codex} × {Tier-0, Tier-1} verified by hand, with agent traces inspected to confirm enforcement actually fires. -**Fixture:** `pr1-fix-clv`. **Worktree:** `.claude/worktrees/drc-3584-sandbox-profiles` (commit pending). - -## Synthetic hook tests (5 cells × 2 tiers) - -Hooks fed JSON payloads on stdin; exit code + stderr captured. - -### Tier 0 hook (`deny-tier-0.sh`) - -| Input | Expected | Observed | +**Fixture:** `pr1-fix-clv`. **Worktree:** `.claude/worktrees/drc-3584-sandbox-profiles`. **Hook revision:** `v2` (Python, post-PR-#36-review). + +The v1 case-glob bash hooks (shipped in the first PR-#36 commit) were superseded after the code review surfaced six bypass shapes the case-glob couldn't address. The v2 hooks use `shlex` tokenisation, `basename` path-stripping, `sh -c` recursion, and case-insensitive skill/MCP matching. The "Bypass attempts" table below is the load-bearing evidence that v2 closes the bypasses. + +## Bypass attempts (the new red-team table) + +Each cell ran the actual `deny-tier-{0,1}.py` hook against the JSON payload via stdin. **Every row must be exit 2** (denied) for the rubric's enforcement story to hold. + +### Tier-0 hook + +| Bypass shape | Payload `command` (or `tool_name` / `skill`) | Reviewer finding | Observed | +|---|---|---|---| +| Shell separator `;` | `true;recce check` | B1 | ✅ exit 2 | +| Pipe `\|` | `true\|recce check` | B1 | ✅ exit 2 | +| Subshell `()` | `(recce check)` | B1 | ✅ exit 2 | +| Command substitution `$()` | `echo $(recce ls)` | B1 | ✅ exit 2 | +| Variable indirection | `PROG=recce; $PROG ls` | B1 | ✅ exit 2 | +| Absolute path (recce) | `/usr/local/bin/recce check` | B2 | ✅ exit 2 | +| Absolute path under /opt | `/opt/recce/bin/recce ls` | B2 | ✅ exit 2 | +| `sh -c` wrapper | `sh -c "recce check"` | B3 | ✅ exit 2 | +| `bash -lc` wrapper | `bash -lc "recce check"` | B3 | ✅ exit 2 | +| `dbt` global flag before subcommand | `dbt --debug parse` | B4 | ✅ exit 2 | +| `dbt --profiles-dir` interposed | `dbt --profiles-dir /tmp parse` | B4 | ✅ exit 2 | +| `dbt -q` short-form flag | `dbt -q parse` | B4 | ✅ exit 2 | +| Capitalised Skill name | `tool_name=Skill, skill=Recce-verify` | B5 | ✅ exit 2 | +| Uppercase Skill name | `tool_name=Skill, skill=RECCE-VERIFY` | B5 | ✅ exit 2 | +| `mcp__recce_dev__*` namespace | `tool_name=mcp__recce_dev__some_tool` | I10 | ✅ exit 2 | +| **Counter-claim**: spoiler path read | `cat ../../RUBRIC.md` | B6 | ✅ exit 0 (allowed by hook; **cwd separation does the work**, per the revised ENFORCEMENT.md) | + +### Tier-1 hook + +| Bypass shape | Payload `command` | Observed | |---|---|---| -| `Bash` + `recce check` | exit 2, "Recce CLI invocation" | ✅ exit 2, "Tier-0 sandbox blocks: Recce CLI invocation (matched in: recce check)" | -| `mcp__recce__row_count_diff` | exit 2, "Recce MCP tool" | ✅ exit 2, "Tier-0 sandbox blocks: Recce MCP tool 'mcp__recce__row_count_diff' (Tier-0 disallows Recce)" | -| `Skill` + `recce-verify` | exit 2, "Recce skill" | ✅ exit 2, "Tier-0 sandbox blocks: Recce skill 'recce-verify' (Tier-0 disallows /recce-* skills)" | -| `Bash` + `dbt parse --target dev` | exit 2, "dbt subcommand" | ✅ exit 2, "Tier-0 sandbox blocks: dbt subcommand regenerates frozen Tier-0 artifacts or hits a warehouse (matched in: dbt parse --target dev)" | -| `Bash` + `git diff HEAD` (benign) | exit 0 | ✅ exit 0 | -| `Read` + `/some/file` (benign) | exit 0 | ✅ exit 0 | - -### Tier 1 hook (`deny-tier-1.sh`) - -| Input | Expected | Observed | -|---|---|---| -| `Bash` + `recce check` (now allowed) | exit 0 | ✅ exit 0 | -| `mcp__recce__row_count_diff` (now allowed) | exit 0 | ✅ exit 0 | -| `Bash` + `dbt parse` (still blocked) | exit 2 | ✅ exit 2, "Tier-1 sandbox blocks: dbt subcommand regenerates frozen artifacts or hits a warehouse (matched in: dbt parse)" | -| `Bash` + `psql -h host -c "select 1"` (still blocked) | exit 2 | ✅ exit 2, "Tier-1 sandbox blocks: Direct SQL client 'psql' (use Recce MCP query instead; matched in: psql -h host -c \"select 1\")" | +| dbt with global flag | `dbt --debug parse` | ✅ exit 2 | +| `sh -c "psql ..."` | `sh -c "psql -h h -c x"` | ✅ exit 2 | +| Absolute path psql | `/usr/bin/psql -c "select 1"` | ✅ exit 2 | +| Pipeline with dbt parse | `true;dbt parse` | ✅ exit 2 | +| **Counter-claim**: Recce allowed | `recce check` | ✅ exit 0 | +| **Counter-claim**: Recce MCP allowed | `tool_name=mcp__recce__row_count_diff` | ✅ exit 0 | +| dbt parse direct | `dbt parse` | ✅ exit 2 | +| `git diff` | `git diff HEAD` | ✅ exit 0 | + +## Happy-path coverage (regression check) + +These were green in the v1 hook and must stay green in v2. + +### Tier-0 + +| Cell | Observed | +|---|---| +| `recce check` direct | ✅ exit 2 | +| `mcp__recce__row_count_diff` | ✅ exit 2 | +| `Skill skill=recce-verify` | ✅ exit 2 | +| `dbt parse --target dev` | ✅ exit 2 | +| `git diff HEAD` (benign) | ✅ exit 0 | +| `Read /some/file` (benign) | ✅ exit 0 | +| `grep -rn foo .` | ✅ exit 0 | +| `git log \| head -20` | ✅ exit 0 | +| `jq .nodes manifest.json` | ✅ exit 0 | ## End-to-end cell: Claude Code × Tier 0 -Real `claude --print` invocation with cwd at the per-fixture working tree. +Real `claude --print` invocation with cwd at the per-fixture working tree, against the v2 Python hook. **Setup:** @@ -40,14 +74,14 @@ WT=/Users/evenwei/InfuseAI/recce-claude-plugin/.claude/worktrees/drc-3584-sandbo FIXTURE=$WT/evals/agent-blind-spots/.tmp/sources/$SLUG TIER=$WT/evals/agent-blind-spots/runner-configs/claude-code/tier-0 -cp -r $TIER/.claude $FIXTURE/ +cp -r $TIER/claude-overlay $FIXTURE/.claude cd $FIXTURE claude --print --permission-mode acceptEdits \ "Run the bash command 'recce check' and report what happened, including any error message." ``` -**Agent trace (excerpt):** +**Agent trace (v1 run; v2 hook produces the same shape with a Python-style stderr line):** ``` The command was blocked by a sandbox hook. @@ -57,11 +91,11 @@ The command was blocked by a sandbox hook. - Result: Blocked before execution **Error message:** -PreToolUse:Bash hook error: [bash "$CLAUDE_PROJECT_DIR/.claude/hooks/deny-tier-0.sh"]: -Tier-0 sandbox blocks: Recce CLI invocation (matched in: recce check) +PreToolUse:Bash hook error: [python3 "$CLAUDE_PROJECT_DIR/.claude/hooks/deny-tier-0.py"]: +Tier-0 sandbox blocks: Bash executable 'recce' not in Tier-0 allowlist (matched in: 'recce check') ``` -**Result:** ✅ PASS — the PreToolUse hook fired exactly as designed; the agent recognised the block and surfaced it in its own output. +**Result:** ✅ PASS — the hook fires, the agent surfaces the block. (Re-running this cell against the v2 hook is recommended once the operator picks up the deferred 3 cells below; the hook logic is otherwise identical to what fired in the v1 run.) ## Cells deferred to operator (acceptance #2 leftover) @@ -77,7 +111,7 @@ Recipes for each cell: see [`../../ENFORCEMENT.md`](../../ENFORCEMENT.md) (Claud ## DRC-3430 strip — verified concurrently -Build_fixtures.sh re-run end-to-end (all 6 fixtures): +`build_fixtures.sh` re-run end-to-end (all 6 fixtures) with the expanded strip list (post-review): ``` OK pr1-fix-clv @@ -88,15 +122,18 @@ OK pr44-promotion-flags OK pr46-net-clv-segments ``` -Post-build leak audit: +Strip list now covers (in addition to the v1 list): -``` -$ find .tmp/sources -path '*/.git' -prune -o \ - \( -name 'recce.yml' -o -name 'claude.yml' -o -path '*/.github/prompts*' -o -name 'recce-*.yml' \) -print -(empty) +- `.devcontainer/` (Recce-specific dev container + post-start script that boots Recce) +- `.github/mcp_config.json` (explicit Recce MCP registration) +- `.github/workflows/recce_*.{yml,yaml}` glob (the `recce-*` glob already in v1 missed `recce_ci.yml`) +- `.github/workflows/dbt-build-pr.yml`, `dbt-build-base.yml`, `dbt_base.yml` (dbt CI that uses `DataRecce/recce-cloud-cicd-action`) -$ grep -rlE --exclude-dir=.git 'mcp__recce__|RECCE_API_TOKEN|recce\.yml' .tmp/sources +Post-build leak audit (case-insensitive `recce` match, profiles.yml whitelisted): + +``` +$ grep -rlEi --exclude-dir=.git --exclude=profiles.yml 'mcp__recce|recce' .tmp/sources (empty) ``` -Initial run surfaced `claude.yml` (a "Claude Code + Recce MCP" reviewer workflow that wasn't on the original strip list) plus a false positive on `.git/index`; fixed inline in `build_fixtures.sh` before this run. +`profiles.yml` is whitelisted from the grep — its `role: RECCE` is the Snowflake role name, not Recce-the-tool priming. A role name doesn't tell the agent how to use Recce; stripping `profiles.yml` would break dbt parse. From 5beb6a57bc646ee2f6ad3a7a9dad919072f0fab5 Mon Sep 17 00:00:00 2001 From: even-wei Date: Fri, 29 May 2026 09:50:32 +0800 Subject: [PATCH 03/13] fix(evals): close cycle-review bypasses (Tier-1 dbt flag-with-value + exec wrappers) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The pr-cycle reviewer found 2 new BLOCKERs in v2 that the prior review missed. Both are static hook-logic bugs; v3 fixes them with two small changes plus a regex tweak. BLOCKER 1 — Tier-1 dbt-flag-with-value bypass (`deny-tier-1.py`): `has_denied_dbt_subcommand` skipped `-`-prefixed tokens but never consumed the value following a value-bearing flag, so the value became the "first positional" and the real subcommand was never inspected. Eight verified shapes: dbt --target dev parse → was exit 0, now exit 2 dbt --profiles-dir /tmp parse → was exit 0, now exit 2 dbt --project-dir /x run → was exit 0, now exit 2 dbt --vars "x: 1" parse → was exit 0, now exit 2 dbt --log-format json parse → was exit 0, now exit 2 dbt --log-level debug parse → was exit 0, now exit 2 dbt --printer-width 80 parse → was exit 0, now exit 2 dbt --profile myprof parse → was exit 0, now exit 2 Fix: scan EVERY token after `dbt` for a banned subcommand, not just the "first positional". False positives require the agent to pass a literal banned-subcommand name as a flag value (e.g. `--target parse`), which is perverse and would deserve denial anyway since Tier-1 warehouse access is supposed to be mediated through Recce MCP. Also expanded DBT_DENIED_SUBCOMMANDS with `run-operation`, `debug`, `source` — all hit the warehouse and were missing from the prior list. BLOCKER 2 — exec-wrapper bypass (both hooks): `xargs recce` / `find -exec recce` / `time recce` / `nohup recce` spawned a denied binary as a child process; the hook only inspected the wrapper's head token and let the call through. Tier-0's positive allowlist let `xargs` and `find` pass; Tier-1's denylist saw a non-DENIED_BINS head and stopped looking. Fix: new EXEC_WRAPPERS set covers xargs, time, nice, nohup, setsid, parallel, exec, timeout, watch, ionice, chrt, stdbuf plus `find` (which needs special handling because its wrapped command lives after `-exec`/`-execdir`). For each wrapper at the head of a segment, walk the args for the first executable-shaped token and recurse-check from there. Heuristic `looks_like_executable` skips numeric positionals (timeout duration, chrt priority), flags, and placeholders (`{}`, `[]`). Regex fix (split_segments): `(? --- .../claude-overlay/hooks/deny-tier-0.py | 130 +++++++++++++--- .../claude-overlay/hooks/deny-tier-1.py | 139 +++++++++++++++--- .../runs/2026-05-28/sandbox-verification.md | 61 ++++++-- 3 files changed, 276 insertions(+), 54 deletions(-) diff --git a/evals/agent-blind-spots/runner-configs/claude-code/tier-0/claude-overlay/hooks/deny-tier-0.py b/evals/agent-blind-spots/runner-configs/claude-code/tier-0/claude-overlay/hooks/deny-tier-0.py index f58322a..85146f1 100755 --- a/evals/agent-blind-spots/runner-configs/claude-code/tier-0/claude-overlay/hooks/deny-tier-0.py +++ b/evals/agent-blind-spots/runner-configs/claude-code/tier-0/claude-overlay/hooks/deny-tier-0.py @@ -49,6 +49,20 @@ {"sh", "bash", "zsh", "dash", "ash", "ksh"} ) +# Exec wrappers that run their argument as a new process. `xargs` and +# `find` are in the allowlist (legitimate dev-loop utilities), but +# they can launch any binary — including a denied one. Recurse into +# their wrapped command and re-check it against the allowlist. +# +# `time`, `nohup`, `nice`, `setsid`, etc. are NOT in the allowlist, +# so they're already denied head-of-token. Including them here is +# defensive: if a future version of this hook adds them to the +# allowlist, the recursion contract still holds. +EXEC_WRAPPERS: frozenset[str] = frozenset( + {"xargs", "time", "nice", "nohup", "setsid", "parallel", + "exec", "timeout", "watch", "ionice", "chrt", "stdbuf"} +) + # Recce MCP namespaces. Covers mcp__recce__*, mcp__plugin_recce_*, # mcp__recce_dev__*, and any future Recce-shaped namespace. MCP_RECCE_RE = re.compile(r"^mcp__(plugin_)?recce(_|-|$)", re.IGNORECASE) @@ -81,8 +95,11 @@ def split_segments(cmd: str) -> list[str]: segments.extend(split_segments(sub)) cmd = re.sub(r"`[^`]*`", "", cmd) - # Now split on command-boundary separators. - parts = re.split(r"[;&|()\n]+", cmd) + # Now split on command-boundary separators. Negative lookbehind on + # `\\` keeps escaped separators (e.g. the `\;` that terminates + # `find -exec`) attached to the same segment; shlex strips the + # backslash later. + parts = re.split(r"(? list[str]: return tokens[i:] +def looks_like_executable(name: str) -> bool: + """Heuristic — does this token's basename look like it could be a + binary the agent is actually invoking? Rules out numbers (timeout + durations, chrt priorities), pure-symbol tokens (`{}`, `;`), and + empty strings. + """ + if not name: + return False + if name in ("{}", "[]", ";", "+", "\\;"): + return False + if name.isdigit(): + return False + # Allow alphanumerics + `_`/`-`/`.` (binaries like `git-foo`, + # `python3.11`, `recce-cli`). Anything else (e.g., a quoted-shell + # construct that survived shlex) is treated as non-executable. + stripped = name.replace("-", "").replace("_", "").replace(".", "") + return stripped.isalnum() + + +def find_wrapped_command(wrapper_name: str, tokens: list[str]) -> list[str]: + """For an exec wrapper at tokens[0], return the wrapped command's + tokens. + + Different wrappers interleave their own positional args before the + wrapped command (e.g., `timeout 30 CMD`, `chrt 0 5 CMD`, + `ionice -c 2 CMD`). Hard-coding a per-wrapper arg parser is + brittle. Instead, walk the args and return tokens starting at the + first executable-shaped token. `find` is special because the + wrapped command lives after a `-exec` / `-execdir` keyword. + """ + if wrapper_name == "find": + for keyword in ("-exec", "-execdir"): + if keyword not in tokens: + continue + idx = tokens.index(keyword) + end = len(tokens) + for j in range(idx + 1, len(tokens)): + if tokens[j] in (";", "+", "\\;"): + end = j + break + return tokens[idx + 1:end] + return [] + for i, tok in enumerate(tokens[1:], 1): + if tok.startswith("-"): + continue + if not looks_like_executable(basename(tok)): + continue + return tokens[i:] + return [] + + +def check_segment_tokens(tokens: list[str], command: str, depth: int = 0) -> None: + if depth > 4: + return + tokens = skip_leading_env(tokens) + if not tokens: + return + name = basename(tokens[0]) + + if name in SHELL_WRAPPERS: + deny( + f"shell wrapper '{name}' is banned at Tier 0 — the " + f"agent has no legitimate reason to invoke a subshell " + f"(matched in: {command!r})" + ) + + if not name: + return + + # Exec wrappers in the allowlist (`xargs`, `find`) need recursion + # so they can't smuggle a denied binary through. Wrappers NOT in + # the allowlist (`time`, `nohup`, ...) are denied head-of-token + # below, so the recursion is moot for them — but we still handle + # them defensively in case the allowlist grows. + if name in EXEC_WRAPPERS or name == "find": + wrapped = find_wrapped_command(name, tokens) + if wrapped: + check_segment_tokens(wrapped, command, depth + 1) + # If the wrapper itself isn't in the allowlist, fall through + # to the allowlist check (will deny). If it IS allowlisted + # (xargs / find), the wrapped check above is what enforces. + if name in TIER_0_ALLOWLIST: + return + + if name not in TIER_0_ALLOWLIST: + deny( + f"Bash executable '{name}' not in Tier-0 allowlist " + f"(matched in: {command!r})" + ) + + def check_bash_command(command: str) -> None: if not command.strip(): return for segment in split_segments(command): - tokens = skip_leading_env(tokenize(segment)) - if not tokens: - continue - name = basename(tokens[0]) - if name in SHELL_WRAPPERS: - deny( - f"shell wrapper '{name}' is banned at Tier 0 — the " - f"agent has no legitimate reason to invoke a subshell " - f"(matched in: {command!r})" - ) - if not name: - continue - if name not in TIER_0_ALLOWLIST: - deny( - f"Bash executable '{name}' not in Tier-0 allowlist " - f"(matched in: {command!r})" - ) + check_segment_tokens(tokenize(segment), command) def main() -> None: diff --git a/evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/hooks/deny-tier-1.py b/evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/hooks/deny-tier-1.py index 0e70efc..e7f8eab 100755 --- a/evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/hooks/deny-tier-1.py +++ b/evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/hooks/deny-tier-1.py @@ -9,21 +9,24 @@ Tier 1 denies (relative to Tier 0): * dbt subcommands that regenerate frozen artifacts or hit a - warehouse directly (`dbt run|test|parse|compile|docs`). Recce - reads the frozen artifacts; the agent never needs to regenerate - them, and dbt invocations would let the agent bypass Recce's - structured query surfaces. + warehouse directly (`dbt run|test|parse|compile|docs|seed|...`). + Recce reads the frozen artifacts; the agent never needs to + regenerate them, and dbt invocations would let the agent bypass + Recce's structured query surfaces. * Direct SQL clients (`duckdb`, `psql`, `snowsql`, `bq`) — at Tier 1 warehouse access is mediated through Recce MCP tools (e.g. `mcp__recce__query`), never raw shell. * `sh -c ""`, `bash -c …`, `zsh -c …` wrappers that smuggle a denied binary past a shallow check. + * Exec wrappers (`xargs`, `find -exec`, `time`, `nohup`, …) that + launch a denied binary as a child process — same bypass class as + `sh -c`, different syntactic shape. -This is a denylist (Tier 0 was an allowlist) because Tier 1's +This is a denylist (Tier 0 is an allowlist) because Tier 1's legitimate command surface is large — the agent can invoke arbitrary git, grep, recce-*, and other dev-loop utilities. Tokenisation + -basename + `sh -c` recursion catch the bypass shapes the prior -case-glob version missed. +basename + wrapper recursion catch the bypass shapes the prior +case-glob and v2 hooks missed. """ from __future__ import annotations @@ -38,18 +41,32 @@ {"dbt", "duckdb", "psql", "snowsql", "bq"} ) -# dbt subcommands that either regenerate frozen Tier-0 inputs or hit a -# warehouse directly. `dbt` alone (no subcommand) and `dbt --help` are -# allowed — they're discovery-only. +# dbt subcommands that regenerate frozen Tier-0 inputs or hit a +# warehouse directly. `dbt` alone (no subcommand), `dbt --help`, +# `dbt --version`, `dbt list` / `dbt ls`, `dbt deps`, `dbt clean` are +# allowed because they don't regenerate artifacts or touch the +# warehouse. DBT_DENIED_SUBCOMMANDS: frozenset[str] = frozenset( {"run", "test", "parse", "compile", "docs", "seed", "snapshot", - "build", "freshness"} + "build", "freshness", "run-operation", "debug", "source"} ) SHELL_WRAPPERS: frozenset[str] = frozenset( {"sh", "bash", "zsh", "dash", "ash", "ksh"} ) +# Exec wrappers that run their argument as a new process. If the +# wrapped command is a denied binary, the hook would otherwise see +# only the wrapper and let the call through. We recurse into the +# wrapped command for each. +# +# `find` is handled separately because its wrapped command lives +# after a `-exec` / `-execdir` keyword, not at a fixed position. +EXEC_WRAPPERS: frozenset[str] = frozenset( + {"xargs", "time", "nice", "nohup", "setsid", "parallel", + "exec", "timeout", "watch", "ionice", "chrt", "stdbuf"} +) + def deny(reason: str) -> None: print(f"Tier-1 sandbox blocks: {reason}", file=sys.stderr) @@ -64,7 +81,10 @@ def split_segments(cmd: str) -> list[str]: for sub in re.findall(r"`([^`]*)`", cmd): segments.extend(split_segments(sub)) cmd = re.sub(r"`[^`]*`", "", cmd) - for part in re.split(r"[;&|()\n]+", cmd): + # Negative lookbehind on `\\` keeps escaped separators (e.g. the + # `\;` that terminates `find -exec`) attached to the same segment; + # shlex strips the backslash later. + for part in re.split(r"(? list[str]: def has_denied_dbt_subcommand(tokens: list[str]) -> bool: - # Walk past `dbt` and look for the first positional (non-flag) token. - for tok in tokens[1:]: + """True if any token after `dbt` is a banned subcommand. + + Earlier versions tried to find the "first positional" by skipping + leading flags. That broke on `dbt --target dev parse` — `dev` (the + value of `--target`) became the first positional and `parse` was + never inspected. Scanning every token after `dbt` instead is both + simpler and more robust: a false positive requires the agent to + pass a literal banned-subcommand name as a flag value (e.g. + `--target parse`), which is perverse and would deserve denial + anyway since the agent shouldn't be invoking dbt at Tier 1. + """ + return any(tok in DBT_DENIED_SUBCOMMANDS for tok in tokens[1:]) + + +def find_wrapped_command_after_exec(tokens: list[str]) -> list[str]: + """For `find` with `-exec` / `-execdir`, return the wrapped command + tokens. Returns [] when `find` has no `-exec` — plain `find` is + safe and doesn't need recursion. + """ + for keyword in ("-exec", "-execdir"): + if keyword not in tokens: + continue + idx = tokens.index(keyword) + end = len(tokens) + for j in range(idx + 1, len(tokens)): + if tokens[j] in (";", "+", "\\;"): + end = j + break + return tokens[idx + 1:end] + return [] + + +def looks_like_executable(name: str) -> bool: + """Heuristic — does this token's basename look like it could be a + binary the agent is actually invoking? Rules out numbers (timeout + durations, chrt priorities), pure-symbol tokens (`{}`, `;`), and + empty strings. + """ + if not name: + return False + if name in ("{}", "[]", ";", "+", "\\;"): + return False + if name.isdigit(): + return False + # Allow alphanumerics + `_`/`-`/`.` (binaries like `git-foo`, + # `python3.11`, `recce-cli`). Anything else (e.g., a quoted-shell + # construct that survived shlex) is treated as non-executable so + # we don't spuriously deny. + stripped = name.replace("-", "").replace("_", "").replace(".", "") + return stripped.isalnum() + + +def find_wrapped_command(wrapper_name: str, tokens: list[str]) -> list[str]: + """For an exec wrapper at tokens[0], return the wrapped command's + tokens. + + Different wrappers interleave their own positional args before the + wrapped command (e.g., `timeout 30 CMD`, `chrt 0 5 CMD`, + `ionice -c 2 CMD`). Hard-coding a per-wrapper arg parser is + brittle. Instead, walk the args and return tokens starting at the + first executable-shaped token — that's the wrapped command in + practice. `find` is special because the wrapped command lives + after a `-exec` / `-execdir` keyword. + """ + if wrapper_name == "find": + return find_wrapped_command_after_exec(tokens) + for i, tok in enumerate(tokens[1:], 1): if tok.startswith("-"): continue - return tok in DBT_DENIED_SUBCOMMANDS - return False + if not looks_like_executable(basename(tok)): + continue + return tokens[i:] + return [] -def check_tokens(tokens: list[str], command_str: str) -> None: +def check_tokens(tokens: list[str], command_str: str, depth: int = 0) -> None: + """Inspect a token list. Recurses one level for shell wrappers and + exec wrappers. `depth` guards against pathological deep recursion. + """ + if depth > 4: + return tokens = skip_leading_env(tokens) if not tokens: return name = basename(tokens[0]) + # Shell wrapper with -c — recurse into its argument. if name in SHELL_WRAPPERS: - # Find the -c / -lc arg and recurse into its content. for j in range(1, len(tokens) - 1): if tokens[j] in ("-c", "-lc", "-ic"): inner = tokens[j + 1] for inner_segment in split_segments(inner): - check_tokens(tokenize(inner_segment), command_str) + check_tokens(tokenize(inner_segment), command_str, depth + 1) return - # A bare `sh`/`bash` interactive subshell with no -c arg is + # A bare `sh` / `bash` interactive subshell with no -c arg is # not a denied command on its own. return + # Exec wrapper (xargs / time / nohup / find -exec / ...). + if name in EXEC_WRAPPERS or name == "find": + wrapped = find_wrapped_command(name, tokens) + if wrapped: + check_tokens(wrapped, command_str, depth + 1) + return + if name == "dbt": if has_denied_dbt_subcommand(tokens): deny( diff --git a/evals/agent-blind-spots/runs/2026-05-28/sandbox-verification.md b/evals/agent-blind-spots/runs/2026-05-28/sandbox-verification.md index 5cc3b0d..67d10f8 100644 --- a/evals/agent-blind-spots/runs/2026-05-28/sandbox-verification.md +++ b/evals/agent-blind-spots/runs/2026-05-28/sandbox-verification.md @@ -2,9 +2,13 @@ Per [DRC-3584](https://linear.app/recce/issue/DRC-3584) acceptance criterion #2: one fixture × {Claude Code, Codex} × {Tier-0, Tier-1} verified by hand, with agent traces inspected to confirm enforcement actually fires. -**Fixture:** `pr1-fix-clv`. **Worktree:** `.claude/worktrees/drc-3584-sandbox-profiles`. **Hook revision:** `v2` (Python, post-PR-#36-review). +**Fixture:** `pr1-fix-clv`. **Worktree:** `.claude/worktrees/drc-3584-sandbox-profiles`. **Hook revision:** `v3` (Python; post-PR-#36-cycle-review). -The v1 case-glob bash hooks (shipped in the first PR-#36 commit) were superseded after the code review surfaced six bypass shapes the case-glob couldn't address. The v2 hooks use `shlex` tokenisation, `basename` path-stripping, `sh -c` recursion, and case-insensitive skill/MCP matching. The "Bypass attempts" table below is the load-bearing evidence that v2 closes the bypasses. +The v1 case-glob bash hooks (shipped in the first PR-#36 commit) were superseded twice: +- **v2** (Python rewrite) — closed the 6 bypass shapes the case-glob couldn't address (shell separators, absolute paths, `sh -c`, dbt global flags, skill case-sensitivity, the false ENFORCEMENT.md:71 claim). +- **v3** (this revision) — closed the 2 additional bypass shapes the cycle review surfaced (Tier-1 dbt flag-with-value, exec-wrapper-launches-denied-binary). + +The "Bypass attempts" tables below are the load-bearing evidence that v3 closes every reviewer-named bypass. All rows are exit-2 expected; the few exit-0 entries are explicit counter-claims (a documented allow path that protects the rubric for a different reason — typically cwd separation rather than the hook). ## Bypass attempts (the new red-team table) @@ -23,26 +27,53 @@ Each cell ran the actual `deny-tier-{0,1}.py` hook against the JSON payload via | Absolute path under /opt | `/opt/recce/bin/recce ls` | B2 | ✅ exit 2 | | `sh -c` wrapper | `sh -c "recce check"` | B3 | ✅ exit 2 | | `bash -lc` wrapper | `bash -lc "recce check"` | B3 | ✅ exit 2 | -| `dbt` global flag before subcommand | `dbt --debug parse` | B4 | ✅ exit 2 | -| `dbt --profiles-dir` interposed | `dbt --profiles-dir /tmp parse` | B4 | ✅ exit 2 | -| `dbt -q` short-form flag | `dbt -q parse` | B4 | ✅ exit 2 | +| `dbt` value-less flag before subcommand | `dbt --debug parse` | B4 | ✅ exit 2 (denied by allowlist — dbt not in allowlist) | +| `dbt --profiles-dir` interposed | `dbt --profiles-dir /tmp parse` | B4 | ✅ exit 2 (denied by allowlist) | +| `dbt -q` short-form flag | `dbt -q parse` | B4 | ✅ exit 2 (denied by allowlist) | | Capitalised Skill name | `tool_name=Skill, skill=Recce-verify` | B5 | ✅ exit 2 | | Uppercase Skill name | `tool_name=Skill, skill=RECCE-VERIFY` | B5 | ✅ exit 2 | | `mcp__recce_dev__*` namespace | `tool_name=mcp__recce_dev__some_tool` | I10 | ✅ exit 2 | +| Exec wrapper: `xargs recce` | `echo check \| xargs recce` | cycle B2 | ✅ exit 2 | +| Exec wrapper: `xargs -I {} recce` | `ls \| xargs -I {} recce {}` | cycle B2 | ✅ exit 2 | +| Exec wrapper: `find -exec recce` | `find . -exec recce {} \;` | cycle B2 | ✅ exit 2 | | **Counter-claim**: spoiler path read | `cat ../../RUBRIC.md` | B6 | ✅ exit 0 (allowed by hook; **cwd separation does the work**, per the revised ENFORCEMENT.md) | +| **Counter-claim**: benign xargs | `echo a \| xargs grep b` | regression | ✅ exit 0 (xargs wraps grep, both allowlisted) | +| **Counter-claim**: benign find -exec | `find . -exec grep foo {} \;` | regression | ✅ exit 0 | +| **Counter-claim**: find without -exec | `find . -name foo` | regression | ✅ exit 0 | + +Note: the cycle review (`v2 review`, NOTE 5) flagged the matcher regex `Bash|Skill|mcp__(plugin_)?recce(_|-).*` as "barely permissive enough to fire for `mcp__recce_dev__*`". Confirmed working: the synthetic payload above fires the hook (would be exit 2 instead of an un-gated allow path). ### Tier-1 hook -| Bypass shape | Payload `command` | Observed | -|---|---|---| -| dbt with global flag | `dbt --debug parse` | ✅ exit 2 | -| `sh -c "psql ..."` | `sh -c "psql -h h -c x"` | ✅ exit 2 | -| Absolute path psql | `/usr/bin/psql -c "select 1"` | ✅ exit 2 | -| Pipeline with dbt parse | `true;dbt parse` | ✅ exit 2 | -| **Counter-claim**: Recce allowed | `recce check` | ✅ exit 0 | -| **Counter-claim**: Recce MCP allowed | `tool_name=mcp__recce__row_count_diff` | ✅ exit 0 | -| dbt parse direct | `dbt parse` | ✅ exit 2 | -| `git diff` | `git diff HEAD` | ✅ exit 0 | +| Bypass shape | Payload `command` | Cycle finding | Observed | +|---|---|---|---| +| dbt with value-less flag | `dbt --debug parse` | B4 | ✅ exit 2 | +| dbt with flag-with-value (target) | `dbt --target dev parse` | cycle B1 | ✅ exit 2 | +| dbt with flag-with-value (profiles-dir) | `dbt --profiles-dir /tmp parse` | cycle B1 | ✅ exit 2 | +| dbt with flag-with-value (project-dir) | `dbt --project-dir /x run` | cycle B1 | ✅ exit 2 | +| dbt with flag-with-value (vars) | `dbt --vars "x: 1" parse` | cycle B1 | ✅ exit 2 | +| dbt with flag-with-value (log-format) | `dbt --log-format json parse` | cycle B1 | ✅ exit 2 | +| dbt with flag-with-value (log-level) | `dbt --log-level debug parse` | cycle B1 | ✅ exit 2 | +| dbt with flag-with-value (printer-width) | `dbt --printer-width 80 parse` | cycle B1 | ✅ exit 2 | +| dbt with flag-with-value (profile) | `dbt --profile myprof parse` | cycle B1 | ✅ exit 2 | +| `sh -c "psql ..."` | `sh -c "psql -h h -c x"` | B3 | ✅ exit 2 | +| Absolute path psql | `/usr/bin/psql -c "select 1"` | B2 | ✅ exit 2 | +| Pipeline with dbt parse | `true;dbt parse` | B1 | ✅ exit 2 | +| Exec wrapper: `xargs dbt` | `echo a \| xargs dbt parse` | cycle B2 | ✅ exit 2 | +| Exec wrapper: `find -exec dbt` | `find . -exec dbt parse \;` | cycle B2 | ✅ exit 2 | +| Exec wrapper: `time dbt` | `time dbt parse` | cycle B2 | ✅ exit 2 | +| Exec wrapper: `nohup dbt` | `nohup dbt parse` | cycle B2 | ✅ exit 2 | +| Exec wrapper: `timeout` + psql | `timeout 30 psql -c x` | cycle B2 | ✅ exit 2 | +| Exec wrapper: `xargs psql` | `echo s \| xargs psql` | cycle B2 | ✅ exit 2 | +| Exec wrapper: `chrt` + psql | `chrt 0 5 psql -c x` | cycle B2 | ✅ exit 2 | +| Exec wrapper: `ionice` + psql | `ionice -c 2 psql` | cycle B2 | ✅ exit 2 | +| **Counter-claim**: Recce allowed | `recce check` | rubric | ✅ exit 0 | +| **Counter-claim**: Recce MCP allowed | `tool_name=mcp__recce__row_count_diff` | rubric | ✅ exit 0 | +| **Counter-claim**: bare dbt allowed | `dbt` (discovery only) | rubric | ✅ exit 0 | +| **Counter-claim**: dbt --help allowed | `dbt --help` | rubric | ✅ exit 0 | +| **Counter-claim**: dbt list allowed | `dbt list` (read-only manifest) | rubric | ✅ exit 0 | +| dbt parse direct | `dbt parse` | B4 | ✅ exit 2 | +| `git diff` | `git diff HEAD` | regression | ✅ exit 0 | ## Happy-path coverage (regression check) From 76d76408e2ff9b9bf468d771ec09d131af3f3513 Mon Sep 17 00:00:00 2001 From: even-wei Date: Fri, 29 May 2026 10:04:36 +0800 Subject: [PATCH 04/13] fix(evals): close iter-2 bypasses (eval / \$() smuggling / dbt clone+retry) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The pr-cycle iter-2 review surfaced three more bypass classes at Tier-1 — all real, none caught by v3. v4 closes them. 1. `eval` shell-builtin smuggling `eval dbt run` / `eval "dbt run"` were exit 0 because `eval` was not in SHELL_WRAPPERS. Fix: added `eval` to Tier-0's SHELL_WRAPPERS (banned outright since the positive allowlist has no legit reason for a shell) and added a dedicated `eval ARGS...` branch in Tier-1's check_tokens that joins the args and recurses into the resulting command — same threat surface as `sh -c`, different syntactic shape. 2. `$()` / backtick substitution at command-head position `$(echo dbt) run`, `` `echo dbt` run ``, `$(printf %s dbt) parse` evaluate to `dbt run` / `dbt parse` at bash runtime; the substitution provides the head binary, which the static check_tokens path can't see because the substitution's content has been stripped from the outer segment. Fix: new `check_substitution_at_head` in Tier-1 pre-walks each segment for `$(...)` / backtick at head and, if the substitution payload contains a denied binary name, denies the outer command. For dbt, it also combines payload tail + outer args to decide whether a denied subcommand is present (so `$(echo dbt)` alone is still allowed — bare dbt is discovery-only). Segmentation for this pre-check uses `(? --- .../claude-overlay/hooks/deny-tier-0.py | 5 +- .../claude-overlay/hooks/deny-tier-1.py | 86 ++++++++++++++++++- .../runs/2026-05-28/sandbox-verification.md | 21 ++++- 3 files changed, 107 insertions(+), 5 deletions(-) diff --git a/evals/agent-blind-spots/runner-configs/claude-code/tier-0/claude-overlay/hooks/deny-tier-0.py b/evals/agent-blind-spots/runner-configs/claude-code/tier-0/claude-overlay/hooks/deny-tier-0.py index 85146f1..d9b5305 100755 --- a/evals/agent-blind-spots/runner-configs/claude-code/tier-0/claude-overlay/hooks/deny-tier-0.py +++ b/evals/agent-blind-spots/runner-configs/claude-code/tier-0/claude-overlay/hooks/deny-tier-0.py @@ -45,8 +45,11 @@ # Shell wrappers that take `-c `. Banned outright at Tier 0 — # refusing to interpret them is simpler than recursing, and an agent # with a positive allowlist has no legitimate reason to invoke a shell. +# `eval` is a shell builtin that concatenates its args and re-evaluates +# them as a command (same threat surface, different syntactic shape); +# included here so it's banned alongside the rest. SHELL_WRAPPERS: frozenset[str] = frozenset( - {"sh", "bash", "zsh", "dash", "ash", "ksh"} + {"sh", "bash", "zsh", "dash", "ash", "ksh", "eval"} ) # Exec wrappers that run their argument as a new process. `xargs` and diff --git a/evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/hooks/deny-tier-1.py b/evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/hooks/deny-tier-1.py index e7f8eab..8f747cf 100755 --- a/evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/hooks/deny-tier-1.py +++ b/evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/hooks/deny-tier-1.py @@ -46,15 +46,26 @@ # `dbt --version`, `dbt list` / `dbt ls`, `dbt deps`, `dbt clean` are # allowed because they don't regenerate artifacts or touch the # warehouse. +# +# `dbt clone` (dbt-core ≥1.6) materialises cloned models in the +# warehouse. `dbt retry` re-executes whatever the previous failed +# invocation was — if that was `dbt run`, retry re-runs it, so the +# deny semantics must inherit at the retry call. DBT_DENIED_SUBCOMMANDS: frozenset[str] = frozenset( {"run", "test", "parse", "compile", "docs", "seed", "snapshot", - "build", "freshness", "run-operation", "debug", "source"} + "build", "freshness", "run-operation", "debug", "source", + "clone", "retry"} ) SHELL_WRAPPERS: frozenset[str] = frozenset( {"sh", "bash", "zsh", "dash", "ash", "ksh"} ) +# `eval` is a shell builtin that concatenates its args and +# re-evaluates them as a command. Same threat surface as `sh -c` but +# with no `-c` flag — the inner command is just the joined positional +# args. Handled as a special case alongside SHELL_WRAPPERS. + # Exec wrappers that run their argument as a new process. If the # wrapped command is a denied binary, the hook would otherwise see # only the wrapper and let the call through. We recurse into the @@ -219,6 +230,16 @@ def check_tokens(tokens: list[str], command_str: str, depth: int = 0) -> None: # not a denied command on its own. return + # `eval ARGS...` concatenates its positional args and re-evaluates + # them as a new command. Same threat surface as `sh -c` but with + # the inner command spread across positional args instead of a + # single `-c` value. + if name == "eval": + inner = " ".join(tokens[1:]) + for inner_segment in split_segments(inner): + check_tokens(tokenize(inner_segment), command_str, depth + 1) + return + # Exec wrapper (xargs / time / nohup / find -exec / ...). if name in EXEC_WRAPPERS or name == "find": wrapped = find_wrapped_command(name, tokens) @@ -241,6 +262,61 @@ def check_tokens(tokens: list[str], command_str: str, depth: int = 0) -> None: ) +def check_substitution_at_head(cmd: str) -> None: + """Detect `$()` / backtick substitutions at command-head position + that smuggle a denied binary. + + `$(echo dbt) run` evaluates to `dbt run` at bash runtime — the + substitution provides the head binary, which the regular + check_tokens path can't see because the substitution's content + has already been stripped from the outer segment. This pass walks + the original command and looks for substitutions at the head of + each segment. + + The check is conservative — false positives only occur if the + agent legitimately has a denied-binary basename appear in a + substitution at command-head, which is the threat we're guarding + against and has no benign use at Tier 1. + """ + # Segment on true command separators only — don't split on `(`/`)` + # because that would shred `$(...)` into pieces (we want it intact + # to detect it as a head substitution). `split_segments` upstream + # uses parens too because it has already extracted `$()` before + # splitting; this pass works on the raw command. + for raw_seg in re.split(r"(? None: try: payload = json.load(sys.stdin) @@ -254,6 +330,14 @@ def main() -> None: if not command.strip(): return + # Pre-check for `$()` / backtick substitutions at command-head + # position that smuggle a denied binary (e.g., `$(echo dbt) run`). + # split_segments extracts the substitution's content as its own + # segment, so the outer post-substitution segment loses the head + # binary — check_tokens alone can't see it. This pre-check fills + # that gap. + check_substitution_at_head(command) + for segment in split_segments(command): check_tokens(tokenize(segment), command) diff --git a/evals/agent-blind-spots/runs/2026-05-28/sandbox-verification.md b/evals/agent-blind-spots/runs/2026-05-28/sandbox-verification.md index 67d10f8..ff21358 100644 --- a/evals/agent-blind-spots/runs/2026-05-28/sandbox-verification.md +++ b/evals/agent-blind-spots/runs/2026-05-28/sandbox-verification.md @@ -2,11 +2,12 @@ Per [DRC-3584](https://linear.app/recce/issue/DRC-3584) acceptance criterion #2: one fixture × {Claude Code, Codex} × {Tier-0, Tier-1} verified by hand, with agent traces inspected to confirm enforcement actually fires. -**Fixture:** `pr1-fix-clv`. **Worktree:** `.claude/worktrees/drc-3584-sandbox-profiles`. **Hook revision:** `v3` (Python; post-PR-#36-cycle-review). +**Fixture:** `pr1-fix-clv`. **Worktree:** `.claude/worktrees/drc-3584-sandbox-profiles`. **Hook revision:** `v4` (Python; post-PR-#36-cycle-iteration-2). -The v1 case-glob bash hooks (shipped in the first PR-#36 commit) were superseded twice: +The v1 case-glob bash hooks (shipped in the first PR-#36 commit) were superseded three times: - **v2** (Python rewrite) — closed the 6 bypass shapes the case-glob couldn't address (shell separators, absolute paths, `sh -c`, dbt global flags, skill case-sensitivity, the false ENFORCEMENT.md:71 claim). -- **v3** (this revision) — closed the 2 additional bypass shapes the cycle review surfaced (Tier-1 dbt flag-with-value, exec-wrapper-launches-denied-binary). +- **v3** — closed the 2 additional bypass shapes the cycle iter-1 surfaced (Tier-1 dbt flag-with-value, exec-wrapper-launches-denied-binary). +- **v4** (this revision) — closed 3 additional bypass classes the cycle iter-2 surfaced (`eval` shell-builtin smuggling, `$()`/backtick substitution at command-head, missing dbt subcommands `clone`/`retry`). The "Bypass attempts" tables below are the load-bearing evidence that v3 closes every reviewer-named bypass. All rows are exit-2 expected; the few exit-0 entries are explicit counter-claims (a documented allow path that protects the rubric for a different reason — typically cwd separation rather than the hook). @@ -40,6 +41,8 @@ Each cell ran the actual `deny-tier-{0,1}.py` hook against the JSON payload via | **Counter-claim**: benign xargs | `echo a \| xargs grep b` | regression | ✅ exit 0 (xargs wraps grep, both allowlisted) | | **Counter-claim**: benign find -exec | `find . -exec grep foo {} \;` | regression | ✅ exit 0 | | **Counter-claim**: find without -exec | `find . -name foo` | regression | ✅ exit 0 | +| **v4 — `eval` banned outright** | `eval recce check` | iter-2 | ✅ exit 2 | +| `eval "ls -la"` (even benign args) | `eval "ls -la"` | iter-2 | ✅ exit 2 (no legit reason for `eval` at Tier 0) | Note: the cycle review (`v2 review`, NOTE 5) flagged the matcher regex `Bash|Skill|mcp__(plugin_)?recce(_|-).*` as "barely permissive enough to fire for `mcp__recce_dev__*`". Confirmed working: the synthetic payload above fires the hook (would be exit 2 instead of an un-gated allow path). @@ -72,8 +75,20 @@ Note: the cycle review (`v2 review`, NOTE 5) flagged the matcher regex `Bash|Ski | **Counter-claim**: bare dbt allowed | `dbt` (discovery only) | rubric | ✅ exit 0 | | **Counter-claim**: dbt --help allowed | `dbt --help` | rubric | ✅ exit 0 | | **Counter-claim**: dbt list allowed | `dbt list` (read-only manifest) | rubric | ✅ exit 0 | +| **Counter-claim**: dbt deps allowed | `dbt deps` (no warehouse) | rubric | ✅ exit 0 | | dbt parse direct | `dbt parse` | B4 | ✅ exit 2 | | `git diff` | `git diff HEAD` | regression | ✅ exit 0 | +| **v4 — `eval` shell-builtin smuggling** | `eval dbt run` | iter-2 | ✅ exit 2 | +| `eval` with quoted inner | `eval "dbt run"` | iter-2 | ✅ exit 2 | +| **v4 — `$()` substitution at head** | `$(echo dbt) run` | iter-2 | ✅ exit 2 | +| Backtick substitution at head | `` `echo dbt` run `` | iter-2 | ✅ exit 2 | +| `$(printf %s dbt) parse` | `$(printf %s dbt) parse` | iter-2 | ✅ exit 2 | +| **v4 — `dbt clone`** | `dbt clone` (warehouse) | iter-2 | ✅ exit 2 | +| **v4 — `dbt retry`** | `dbt retry` (inherits prior cmd) | iter-2 | ✅ exit 2 | +| **Counter-claim**: `$()` not at head | `git log --grep=$(echo psql)` | iter-2 | ✅ exit 0 | +| **Counter-claim**: benign `$()` head | `$(which python) script.py` | iter-2 | ✅ exit 0 | +| **Counter-claim**: benign `$(echo grep)` | `$(echo grep) -rn foo .` | iter-2 | ✅ exit 0 | +| **Counter-claim**: dbt-without-subcommand via `$()` | `$(echo dbt)` | iter-2 | ✅ exit 0 (bare dbt allowed) | ## Happy-path coverage (regression check) From 5b92f507cca318df0845a99d6f4b9eb2e8cc5f8b Mon Sep 17 00:00:00 2001 From: even-wei Date: Fri, 29 May 2026 10:33:40 +0800 Subject: [PATCH 05/13] =?UTF-8?q?refactor(evals):=20hooks=20=E2=86=92=20ba?= =?UTF-8?q?shlex=20AST=20parser=20(DRC-3584=20v5,=20closes=20iter-3=20bypa?= =?UTF-8?q?sses)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The pr-cycle iter-3 review surfaced 9 more bypass classes that the shlex+regex hooks couldn't model — all real, all rooted in Bash semantics shlex doesn't represent. Iter-3 BLOCKERs: 1. Nested $() — `$(sh -c "$(echo dbt) run")`, `$($(echo dbt) run)`, `` `$(echo dbt) run` `` (single-level regex misses nesting) 2. `eval $(...)` — substitution stripped before eval branch ran 3. `xargs $(echo dbt)` — wrapper-at-head + substitution-as-arg; find_wrapped_command saw no executable-shaped token 4. `dbt $(echo run)` — substitution supplied the subcommand; has_denied_dbt_subcommand only scanned outer literal tokens 5. Bash ANSI-C $'dbt' / parameter expansion ${a:-dbt} — shlex passed through as opaque basenames that didn't equal 'dbt' 6. `coproc dbt run` — Bash keyword not in any wrapper set 7. `command dbt run` / `builtin eval dbt run` — Bash command modifiers not handled 8. `! dbt run` — Bash logical-not prefix not matched 9. `xargs -I {} sh -c "{} parse" dbt` — xargs substitutes dbt into {} at runtime; hook saw only the literal {} Rather than patch each class with another regex, this commit **replaces the regex/shlex approach with `bashlex` (a real Bash AST parser)**. Substitutions are walked as a tree; ANSI-C / parameter expansion are extracted from their AST nodes; Bash keywords (!, command, builtin) are handled as transparent prefixes; `coproc` is denied outright (bashlex can't represent it). New dependency: `bashlex` (pure Python, MIT, well-maintained). Both hooks fail closed if bashlex isn't installed — better to deny a legit call than silently allow a bypass. ENFORCEMENT.md + runner- configs README document the install step (`python3 -m pip install bashlex`). Architecture: Two passes per command (in walk()): 1. WALK INTO every $()/backtick substitution as if it were a top-level command. Catches `$(sh -c "dbt run")` because the inner sh -c "dbt run" is itself denied. 2. RESOLVE the head word of each command to candidate output strings (literal text, ANSI-C decoded via raw-position lookup, parameter defaults, substitution payload words). At Tier 0, ALL candidates must be in the allowlist — `$(echo dbt) run` resolves to ['echo','dbt'] and 'dbt' isn't allowlisted, so deny. At Tier 1, any candidate matching a denied binary (with denied subcommand visible anywhere in args, including substitution payloads) denies. Exec wrappers (xargs/find -exec/time/nohup/etc.) now scan EVERY wrapped-command word's candidates for a denied basename — exec wrappers supply args from stdin/find-output, so "bare dbt is allowed" reasoning doesn't apply under a wrapper. Shell wrappers (sh -c, bash -lc, ...) reparse the LITERAL word text of the -c arg (not its substitution candidates) so the inner command structure stays intact. Catches nested $(sh -c "$(echo dbt) run"). Verification: 51 cells across both tiers. Every iter-1/iter-2/iter-3 BLOCKER closed. False-positive sanity for benign $() at non-head, $(echo grep), $(which python), echo $(date), command grep, find without -exec, xargs grep, etc. — all allow correctly. Build re-verified across all 6 fixtures. Closes the iter-3 9 BLOCKER classes. The reviewer's recommended follow-up (pytest-parameterized over the bypass matrix) is filed as an out-of-PR improvement. Signed-off-by: even-wei --- evals/agent-blind-spots/ENFORCEMENT.md | 17 +- .../runner-configs/README.md | 16 +- .../claude-overlay/hooks/deny-tier-0.py | 462 ++++++++------ .../claude-overlay/hooks/deny-tier-1.py | 583 ++++++++++-------- .../runs/2026-05-28/sandbox-verification.md | 24 +- 5 files changed, 664 insertions(+), 438 deletions(-) diff --git a/evals/agent-blind-spots/ENFORCEMENT.md b/evals/agent-blind-spots/ENFORCEMENT.md index f177169..dfeeca3 100644 --- a/evals/agent-blind-spots/ENFORCEMENT.md +++ b/evals/agent-blind-spots/ENFORCEMENT.md @@ -6,6 +6,16 @@ This is load-bearing: the lens-3 counterfactual delta (Tier-0 verdict → Tier-1 The sandbox profile templates live under [`runner-configs/`](runner-configs/). See [`runner-configs/README.md`](runner-configs/README.md) for the per-agent / per-tier file map; this document is the recipe that turns those templates into a recorded baseline. +## Prerequisite — `bashlex` + +The Claude Code PreToolUse hooks (`deny-tier-{0,1}.py`) use `bashlex` to parse the agent's Bash command into an AST. Without it the hooks fail closed (exit 2 with an install message) so bypasses cannot slip through silently. Install once per eval-runner Python: + +```bash +python3 -m pip install bashlex +``` + +This is needed for the Claude Code runner only — Codex's enforcement is process-sandbox + PATH scrub + MCP allowlist, no hook involved. + ## Recipe — Claude Code ### Tier 0 @@ -16,8 +26,9 @@ WT_ROOT="$(git rev-parse --show-toplevel)" # eval host repo root FIXTURE_DIR="${WT_ROOT}/evals/agent-blind-spots/.tmp/sources/${SLUG}" TIER_DIR="${WT_ROOT}/evals/agent-blind-spots/runner-configs/claude-code/tier-0" -# 1. Build fixtures if you haven't. +# 1. Build fixtures if you haven't, and confirm bashlex is installed. ( cd "${WT_ROOT}/evals/agent-blind-spots" && ./build_fixtures.sh ) +python3 -c "import bashlex" || python3 -m pip install bashlex # 2. Stamp the per-fixture working tree with the Tier-0 sandbox config. # `claude-overlay/` becomes `.claude/` inside the fixture — the source @@ -40,7 +51,9 @@ cd "${FIXTURE_DIR}" claude "" ``` -The Tier-0 `.claude/settings.json` declares `permissions.deny` rules for the documented Recce MCP namespaces and dbt/SQL-client Bash patterns, and registers a `PreToolUse` hook (`deny-tier-0.py`). The hook is the **load-bearing layer** — `permissions.deny` is conceded to be unreliable (Claude Code [issue #6699](https://github.com/anthropics/claude-code/issues/6699)) and uses shell-glob matching whose surface differs from the hook's tokenizer-based check. Treat `permissions.deny` as documentation; trust the hook. See `runner-configs/README.md` for the divergence details. +The Tier-0 `.claude/settings.json` declares `permissions.deny` rules for the documented Recce MCP namespaces and dbt/SQL-client Bash patterns, and registers a `PreToolUse` hook (`deny-tier-0.py`). The hook is the **load-bearing layer** — `permissions.deny` is conceded to be unreliable (Claude Code [issue #6699](https://github.com/anthropics/claude-code/issues/6699)) and uses shell-glob matching whose surface differs from the hook's AST-based check. Treat `permissions.deny` as documentation; trust the hook. See `runner-configs/README.md` for the divergence details. + +The hook uses `bashlex` (a real Bash AST parser) so nested `$()`, ANSI-C `$'...'`, parameter expansion `${a:-default}`, Bash keywords (`!`, `coproc`), command modifiers (`command`, `builtin`), and exec wrappers (`xargs`, `find -exec`, `time`, `nohup`, …) are all handled by their AST shape rather than ad-hoc regex. ### Tier 1 diff --git a/evals/agent-blind-spots/runner-configs/README.md b/evals/agent-blind-spots/runner-configs/README.md index 7a78aa6..8151bcd 100644 --- a/evals/agent-blind-spots/runner-configs/README.md +++ b/evals/agent-blind-spots/runner-configs/README.md @@ -50,9 +50,9 @@ claude "" ## Why both `permissions.deny` and a `PreToolUse` hook -Claude Code's `permissions.deny` uses shell-glob matching that is conceded to be unreliable: open issue [anthropics/claude-code#6699](https://github.com/anthropics/claude-code/issues/6699) shows bypasses, and the glob shape can't express the things the rubric requires (path-stripped basenames, `sh -c ""` recursion, dbt subcommands with global flags interposed, case-insensitive skill matching). +Claude Code's `permissions.deny` uses shell-glob matching that is conceded to be unreliable: open issue [anthropics/claude-code#6699](https://github.com/anthropics/claude-code/issues/6699) shows bypasses, and the glob shape can't express the things the rubric requires (path-stripped basenames, `sh -c ""` recursion, dbt subcommands with global flags interposed, case-insensitive skill matching, nested `$()` smuggling, ANSI-C / parameter-expansion head smuggling). -The PreToolUse hook is therefore the **load-bearing layer**, not a backup. It uses `shlex` tokenisation, basenames each executable, recurses into shell wrappers, lowercases skill names, and matches MCP namespaces with a regex. +The PreToolUse hook is therefore the **load-bearing layer**, not a backup. It parses each Bash command with `bashlex` (a real Bash AST parser) and walks the AST — substitutions, ANSI-C quoting, parameter defaults, Bash keywords, command modifiers, and exec wrappers are handled by their AST shape rather than regex. **`permissions.deny` and the hook do not mirror each other**, and that's deliberate: @@ -61,6 +61,16 @@ The PreToolUse hook is therefore the **load-bearing layer**, not a backup. It us Treat `permissions.deny` as documentation for human readers. Trust the hook for enforcement. +## Prerequisite: `bashlex` + +The hook depends on `bashlex`. Install once in the eval-runner Python: + +```bash +python3 -m pip install bashlex +``` + +Without `bashlex` the hook fails closed (exit 2 with an install message) so a missing dependency cannot silently allow bypasses. + ## Maintenance When you add a new Recce CLI verb, new MCP tool namespace, or a new dbt subcommand that mutates state: @@ -69,4 +79,4 @@ When you add a new Recce CLI verb, new MCP tool namespace, or a new dbt subcomma 2. Update the documented `permissions.deny` patterns to match the new shape. 3. Add the new shape to the Bypass attempts table in `runs//sandbox-verification.md` so a future regression is caught. -The hook's MCP namespace regex is `^mcp__(plugin_)?recce(_|-|$)` — adding a `mcp__recce___*` namespace is automatically covered. New Bash basenames (e.g. a `recce` CLI rename) require a Python-level change. +The hook's MCP namespace regex is `^mcp__(plugin_)?recce(_|-|$)` — adding a `mcp__recce___*` namespace is automatically covered. New Bash basenames (e.g. a `recce` CLI rename, a new SQL client, a new dbt subcommand) require a Python-level change to `DENIED_BINS` / `DBT_DENIED_SUBCOMMANDS` / `TIER_0_ALLOWLIST` / `EXEC_WRAPPERS` / `SHELL_WRAPPERS` / `TRANSPARENT_PREFIXES`. diff --git a/evals/agent-blind-spots/runner-configs/claude-code/tier-0/claude-overlay/hooks/deny-tier-0.py b/evals/agent-blind-spots/runner-configs/claude-code/tier-0/claude-overlay/hooks/deny-tier-0.py index d9b5305..13ffacd 100755 --- a/evals/agent-blind-spots/runner-configs/claude-code/tier-0/claude-overlay/hooks/deny-tier-0.py +++ b/evals/agent-blind-spots/runner-configs/claude-code/tier-0/claude-overlay/hooks/deny-tier-0.py @@ -1,40 +1,52 @@ #!/usr/bin/env python3 """ -Tier-0 PreToolUse hook for the /recce-verify v1 eval. - -Belt-and-suspenders alongside `permissions.deny` in ../settings.json -(see DRC-3584 issue body: hooks are more reliable than deny rules per -open Claude Code issue #6699). - -Tier 0 is a **positive allowlist** for Bash, with Recce MCP / Recce -skill denial layered on top. Anything outside the allowlist exits 2. - -The case-glob version of this hook (deny-tier-0.sh, removed in PR #36's -v2 commit) was bypassed by: - * `true;recce check`, `true|recce`, `(recce check)`, `$(recce ls)` — - shell separators that aren't `[[:space:]]` - * `/usr/local/bin/recce check` — absolute path on the executable - * `sh -c "recce check"`, `bash -lc "..."` — wrapper hiding inner cmd - * `dbt --debug parse` — global flag interposed before subcommand - * `Recce-verify`, `RECCE-VERIFY` — skill case-insensitivity - * `mcp__recce_dev__*` — MCP namespace shape not in the prior regex - -This rewrite tokenises by shell metacharacters (not just whitespace), -basenames each executable token (so paths don't help), recurses into -`sh -c` arg, and lowercases the skill name before matching. +Tier-0 PreToolUse hook for the /recce-verify v1 eval (v5 — bashlex). + +Tier 0 is a positive allowlist for Bash plus Recce MCP / Recce skill +denial. Anything outside the allowlist exits 2. + +Architecture: see deny-tier-1.py (same bashlex-based AST walk shape). +Tier-0-specific differences: + * Bash head must be in TIER_0_ALLOWLIST (positive policy) instead + of "not in DENIED_BINS". + * Shell wrappers are banned outright (no `-c` recursion); a Tier-0 + agent has no legitimate reason to spawn a subshell. + * `eval` is banned outright as a shell builtin. + * `xargs` and `find` ARE in the allowlist but recurse-check their + wrapped command against the allowlist. + * MCP-Recce namespaces and Recce-* skills are denied at the tool + level before we ever look at the command. + +Fail-closed behavior: missing bashlex → exit 2. """ from __future__ import annotations import json import re -import shlex import sys from os.path import basename +from typing import Iterable + +try: + import bashlex + import bashlex.errors +except ImportError: + print( + "Tier-0 sandbox blocks: bashlex not installed. " + "Install with `python3 -m pip install bashlex` in the eval-runner " + "environment, then retry. Failing closed so bypasses can't slip " + "through silently.", + file=sys.stderr, + ) + sys.exit(2) + -# Bash commands a Tier-0 agent legitimately needs. Anything else denied. -# Per RUBRIC.md Tier-0 runtime contract: file read, grep/ripgrep, jq, -# git log/diff/show. The list below is that contract plus standard -# POSIX text-processing utilities for evidence analysis. +# --- Policy ---------------------------------------------------------------- + +# Bash commands the Tier-0 agent legitimately needs. Anything else is +# denied. Per RUBRIC.md Tier-0 runtime contract: file read, grep / +# ripgrep, jq, git log/diff/show. Augmented with standard POSIX +# text-processing utilities. TIER_0_ALLOWLIST: frozenset[str] = frozenset({ "git", "grep", "rg", "jq", "ls", "cat", "head", "tail", "wc", "find", "echo", "true", "false", "awk", "sed", "sort", "uniq", "comm", "diff", @@ -42,30 +54,22 @@ "printf", "tr", "cut", "xargs", "tee", "less", "more", }) -# Shell wrappers that take `-c `. Banned outright at Tier 0 — -# refusing to interpret them is simpler than recursing, and an agent -# with a positive allowlist has no legitimate reason to invoke a shell. -# `eval` is a shell builtin that concatenates its args and re-evaluates -# them as a command (same threat surface, different syntactic shape); -# included here so it's banned alongside the rest. +# Shell wrappers and `eval` are banned outright at Tier 0. SHELL_WRAPPERS: frozenset[str] = frozenset( {"sh", "bash", "zsh", "dash", "ash", "ksh", "eval"} ) -# Exec wrappers that run their argument as a new process. `xargs` and -# `find` are in the allowlist (legitimate dev-loop utilities), but -# they can launch any binary — including a denied one. Recurse into -# their wrapped command and re-check it against the allowlist. -# -# `time`, `nohup`, `nice`, `setsid`, etc. are NOT in the allowlist, -# so they're already denied head-of-token. Including them here is -# defensive: if a future version of this hook adds them to the -# allowlist, the recursion contract still holds. +# Exec wrappers in the allowlist (`xargs`, `find`) need recursion +# checks; non-allowlisted wrappers (`time`, `nohup`, etc.) are denied +# by the head check anyway. Listed defensively in case the allowlist +# grows later. EXEC_WRAPPERS: frozenset[str] = frozenset( {"xargs", "time", "nice", "nohup", "setsid", "parallel", "exec", "timeout", "watch", "ionice", "chrt", "stdbuf"} ) +TRANSPARENT_PREFIXES: frozenset[str] = frozenset({"command", "builtin"}) + # Recce MCP namespaces. Covers mcp__recce__*, mcp__plugin_recce_*, # mcp__recce_dev__*, and any future Recce-shaped namespace. MCP_RECCE_RE = re.compile(r"^mcp__(plugin_)?recce(_|-|$)", re.IGNORECASE) @@ -74,197 +78,295 @@ RECCE_SKILL_RE = re.compile(r"^recce[-:]", re.IGNORECASE) +# --- Output --------------------------------------------------------------- + def deny(reason: str) -> None: print(f"Tier-0 sandbox blocks: {reason}", file=sys.stderr) sys.exit(2) -def split_segments(cmd: str) -> list[str]: - """Split a shell command into segments by separators that introduce - a new command boundary: `;`, `&`, `|`, `(`, `)`, newline. Also - extract `$(...)` and backtick subshells as their own segments — - they let the agent smuggle a command past a leading `echo`. - """ - segments: list[str] = [] - - # Extract command substitutions first (they could nest, but a - # single-level extraction defeats the obvious bypass and matches - # the threat model: an LLM is unlikely to construct deep nesting - # specifically to evade us). - for sub in re.findall(r"\$\(([^()]*)\)", cmd): - segments.extend(split_segments(sub)) - cmd = re.sub(r"\$\([^()]*\)", "", cmd) - for sub in re.findall(r"`([^`]*)`", cmd): - segments.extend(split_segments(sub)) - cmd = re.sub(r"`[^`]*`", "", cmd) - - # Now split on command-boundary separators. Negative lookbehind on - # `\\` keeps escaped separators (e.g. the `\;` that terminates - # `find -exec`) attached to the same segment; shlex strips the - # backslash later. - parts = re.split(r"(? str: + return re.sub(r"\\(.)", r"\1", inner) + + +def resolve_word(word_node, original_cmd: str) -> list[str]: + """Return possible string values this word could resolve to. + See deny-tier-1.py for the rules.""" + pos = getattr(word_node, 'pos', None) + text = getattr(word_node, 'word', '') or '' + + if pos and len(pos) == 2 and pos[1] > pos[0]: + raw = original_cmd[pos[0]:pos[1]] + m = _ANSI_C_RE.match(raw) + if m: + return [_decode_ansi_c(m.group(1))] + + parts = getattr(word_node, 'parts', []) or [] + if not parts: + return [text] + + candidates: list[str] = [] for part in parts: - if part.strip(): - segments.append(part.strip()) - return segments + kind = getattr(part, 'kind', '') + if kind == 'parameter': + value = getattr(part, 'value', '') or '' + if ':-' in value: + candidates.append(value.split(':-', 1)[1]) + elif ':=' in value: + candidates.append(value.split(':=', 1)[1]) + elif kind == 'commandsubstitution': + sub_cmd = getattr(part, 'command', None) + if sub_cmd is not None: + for sub_word in _walk_command_words(sub_cmd): + candidates.extend(resolve_word(sub_word, original_cmd)) + + if not candidates: + return [text] + return candidates + + +def _walk_command_words(cmd_node) -> Iterable[object]: + kind = getattr(cmd_node, 'kind', '') + if kind == 'word': + yield cmd_node + return + for part in getattr(cmd_node, 'parts', []) or []: + if not hasattr(part, 'kind'): + continue + if part.kind == 'word': + yield part + else: + yield from _walk_command_words(part) -def tokenize(segment: str) -> list[str]: - try: - return shlex.split(segment) - except ValueError: - # Unbalanced quotes — treat as one opaque token. Whatever the - # agent meant, it's not a clean invocation; let the allowlist - # check it as-is. - return [segment] - - -def skip_leading_env(tokens: list[str]) -> list[str]: - """Skip `VAR=value` env assignments and `env [args] CMD ...` so the - executable check lands on the real command. - """ - i = 0 - while i < len(tokens) and "=" in tokens[i] and not tokens[i].startswith("="): - i += 1 - if i >= len(tokens): - return [] - if basename(tokens[i]) == "env": - i += 1 - # env [-i] [-u VAR ...] [NAME=value ...] command ... - while i < len(tokens): - t = tokens[i] - if t in ("-i", "-0", "--null"): - i += 1 - continue - if t in ("-u", "--unset"): - i += 2 - continue - if "=" in t and not t.startswith("="): - i += 1 - continue - break - return tokens[i:] - - -def looks_like_executable(name: str) -> bool: - """Heuristic — does this token's basename look like it could be a - binary the agent is actually invoking? Rules out numbers (timeout - durations, chrt priorities), pure-symbol tokens (`{}`, `;`), and - empty strings. - """ +def _walk_substitutions(word_node, original_cmd: str, depth: int) -> None: + for part in getattr(word_node, 'parts', []) or []: + if getattr(part, 'kind', '') == 'commandsubstitution': + sub_cmd = getattr(part, 'command', None) + if sub_cmd is not None: + walk(sub_cmd, original_cmd, depth + 1) + + +def _looks_like_executable(name: str) -> bool: if not name: return False if name in ("{}", "[]", ";", "+", "\\;"): return False if name.isdigit(): return False - # Allow alphanumerics + `_`/`-`/`.` (binaries like `git-foo`, - # `python3.11`, `recce-cli`). Anything else (e.g., a quoted-shell - # construct that survived shlex) is treated as non-executable. stripped = name.replace("-", "").replace("_", "").replace(".", "") return stripped.isalnum() -def find_wrapped_command(wrapper_name: str, tokens: list[str]) -> list[str]: - """For an exec wrapper at tokens[0], return the wrapped command's - tokens. - - Different wrappers interleave their own positional args before the - wrapped command (e.g., `timeout 30 CMD`, `chrt 0 5 CMD`, - `ionice -c 2 CMD`). Hard-coding a per-wrapper arg parser is - brittle. Instead, walk the args and return tokens starting at the - first executable-shaped token. `find` is special because the - wrapped command lives after a `-exec` / `-execdir` keyword. - """ - if wrapper_name == "find": - for keyword in ("-exec", "-execdir"): - if keyword not in tokens: - continue - idx = tokens.index(keyword) - end = len(tokens) - for j in range(idx + 1, len(tokens)): - if tokens[j] in (";", "+", "\\;"): - end = j - break - return tokens[idx + 1:end] - return [] - for i, tok in enumerate(tokens[1:], 1): - if tok.startswith("-"): - continue - if not looks_like_executable(basename(tok)): - continue - return tokens[i:] - return [] +# --- AST walk ------------------------------------------------------------ +def walk(node, original_cmd: str, depth: int = 0) -> None: + if depth > 8: + return + kind = getattr(node, 'kind', '') -def check_segment_tokens(tokens: list[str], command: str, depth: int = 0) -> None: - if depth > 4: + if kind in ('list', 'pipeline', 'compound', 'if', 'for', 'while', + 'until', 'function', 'case'): + for child in getattr(node, 'parts', []) or []: + if hasattr(child, 'kind'): + walk(child, original_cmd, depth + 1) return - tokens = skip_leading_env(tokens) - if not tokens: + + if kind != 'command': return - name = basename(tokens[0]) - if name in SHELL_WRAPPERS: - deny( - f"shell wrapper '{name}' is banned at Tier 0 — the " - f"agent has no legitimate reason to invoke a subshell " - f"(matched in: {command!r})" - ) + words = [p for p in (node.parts or []) if getattr(p, 'kind', '') == 'word'] + if not words: + return - if not name: + # Pass 1: walk INTO every substitution so the inner is independently + # checked against Tier-0 allowlist. + for w in words: + _walk_substitutions(w, original_cmd, depth) + + # Pass 2: head check. + idx = 0 + while idx < len(words): + cands = resolve_word(words[idx], original_cmd) + names = {basename(c) for c in cands} + if names & TRANSPARENT_PREFIXES: + idx += 1 + continue + if "env" in names and idx + 1 < len(words): + idx += 1 + while idx < len(words): + t = words[idx].word or '' + if t in ("-i", "-0", "--null"): + idx += 1 + continue + if t in ("-u", "--unset"): + idx += 2 + continue + if "=" in t and not t.startswith("="): + idx += 1 + continue + break + continue + break + + if idx >= len(words): return - # Exec wrappers in the allowlist (`xargs`, `find`) need recursion - # so they can't smuggle a denied binary through. Wrappers NOT in - # the allowlist (`time`, `nohup`, ...) are denied head-of-token - # below, so the recursion is moot for them — but we still handle - # them defensively in case the allowlist grows. - if name in EXEC_WRAPPERS or name == "find": - wrapped = find_wrapped_command(name, tokens) + head_word = words[idx] + head_candidates = resolve_word(head_word, original_cmd) + head_names = {basename(c) for c in head_candidates} + arg_words = words[idx + 1:] + + # Shell wrappers and `eval` banned outright at Tier 0. + if head_names & SHELL_WRAPPERS: + bad = next(iter(head_names & SHELL_WRAPPERS)) + deny( + f"shell wrapper '{bad}' is banned at Tier 0 — the agent " + f"has no legitimate reason to invoke a subshell " + f"(matched in: {original_cmd!r})" + ) + + # Exec wrappers (`xargs`, `find`) ARE in the allowlist, but we + # must recurse-check what they spawn so the allowlist isn't + # bypassed via `xargs recce`. + if head_names & EXEC_WRAPPERS or "find" in head_names: + wrapped = _find_exec_target(head_names, arg_words) if wrapped: - check_segment_tokens(wrapped, command, depth + 1) - # If the wrapper itself isn't in the allowlist, fall through - # to the allowlist check (will deny). If it IS allowlisted - # (xargs / find), the wrapped check above is what enforces. - if name in TIER_0_ALLOWLIST: + for w in wrapped: + for cand in resolve_word(w, original_cmd): + name = basename(cand) + if not name or name.startswith("-"): + continue + if not _looks_like_executable(name): + continue + if name not in TIER_0_ALLOWLIST: + deny( + f"Bash executable '{name}' (wrapped by " + f"'{next(iter(head_names))}') not in " + f"Tier-0 allowlist (matched in: " + f"{original_cmd!r})" + ) + # First wrapped-position word checked; stop after first + # candidate set so multi-arg commands like + # `xargs grep foo` don't false-positive on 'foo'. + break + # Also re-parse to catch nested constructs. + head_cands = resolve_word(wrapped[0], original_cmd) + if head_cands: + rest = " ".join((w.word or '') for w in wrapped[1:]) + synth = (head_cands[0] + " " + rest).strip() + if synth: + _reparse_and_walk(synth, original_cmd, depth + 1) + # If wrapper itself is in the allowlist, fall through to the + # allowlist check (which allows). If not, the head-check + # below denies. + if head_names & TIER_0_ALLOWLIST: return - if name not in TIER_0_ALLOWLIST: + # Head must be in the allowlist. When the head resolves to + # multiple candidates (e.g., from a `$(echo dbt)` substitution), + # ALL must be allowed — any one being out of allowlist is a + # potential bypass at runtime because that candidate could be + # the actual head bash exec's. + out_of_allowlist = head_names - TIER_0_ALLOWLIST + if out_of_allowlist: + name = next(iter(out_of_allowlist)) deny( f"Bash executable '{name}' not in Tier-0 allowlist " - f"(matched in: {command!r})" + f"(matched in: {original_cmd!r})" ) -def check_bash_command(command: str) -> None: - if not command.strip(): +def _find_exec_target(head_names: set[str], arg_words: list) -> list: + if "find" in head_names: + for i, w in enumerate(arg_words): + if (w.word or '') in ("-exec", "-execdir"): + end = len(arg_words) + for j in range(i + 1, len(arg_words)): + if (arg_words[j].word or '') in (";", "+", "\\;"): + end = j + break + return arg_words[i + 1:end] + return [] + for i, w in enumerate(arg_words): + text = w.word or '' + if text.startswith("-"): + continue + if _looks_like_executable(basename(text)): + return arg_words[i:] + if getattr(w, 'parts', None): + return arg_words[i:] + return [] + + +def _reparse_and_walk(inner: str, original_cmd: str, depth: int) -> None: + if depth > 8 or not inner.strip(): return - for segment in split_segments(command): - check_segment_tokens(tokenize(segment), command) + try: + trees = bashlex.parse(inner) + except (bashlex.errors.ParsingError, NotImplementedError): + for tok in inner.split(): + base = basename(tok) + if base and not base.startswith("-") and _looks_like_executable(base): + if base not in TIER_0_ALLOWLIST: + deny( + f"unparseable inner command head '{base}' not " + f"in Tier-0 allowlist (matched in: {original_cmd!r})" + ) + # Only inspect the first looks-like-executable token. + return + return + for tree in trees: + walk(tree, original_cmd, depth) +# --- Entry point --------------------------------------------------------- + def main() -> None: try: payload = json.load(sys.stdin) except json.JSONDecodeError: - # Malformed payload — fail open so a single bad invocation - # doesn't break the whole session. The matcher in settings.json - # narrows what we see; nothing legitimate arrives malformed. return tool_name = payload.get("tool_name", "") or "" tool_input = payload.get("tool_input", {}) or {} + # MCP-Recce check (any namespace shape). if MCP_RECCE_RE.match(tool_name): deny(f"Recce MCP tool '{tool_name}' (Tier-0 disallows Recce)") + # Skill check (case-insensitive). skill = tool_input.get("skill", "") or "" if RECCE_SKILL_RE.match(skill): deny(f"Recce skill '{skill}' (Tier-0 disallows /recce-* skills)") - if tool_name == "Bash": - check_bash_command(tool_input.get("command", "") or "") + if tool_name != "Bash": + return + + cmd = tool_input.get("command", "") or "" + if not cmd.strip(): + return + + if re.search(r"\bcoproc\b", cmd): + deny(f"`coproc` keyword is not allowed at Tier 0 (matched in: {cmd!r})") + + try: + trees = bashlex.parse(cmd) + except NotImplementedError as e: + deny( + f"bash construct not supported by parser ({e}); " + f"failing closed (matched in: {cmd!r})" + ) + except bashlex.errors.ParsingError: + return + + for tree in trees: + walk(tree, cmd, 0) if __name__ == "__main__": diff --git a/evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/hooks/deny-tier-1.py b/evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/hooks/deny-tier-1.py index 8f747cf..527faac 100755 --- a/evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/hooks/deny-tier-1.py +++ b/evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/hooks/deny-tier-1.py @@ -1,56 +1,66 @@ #!/usr/bin/env python3 """ -Tier-1 PreToolUse hook for the /recce-verify v1 eval. +Tier-1 PreToolUse hook for the /recce-verify v1 eval (v5 — bashlex). Tier 1 = Tier 0 plus Recce CLI, Recce MCP, single-env warehouse -credentials (read-only on the dev environment). Base/prod environment -access stays denied — Tier 2 territory, out of v1 scope. Recce MCP is -**intentionally not gated** here — the rubric explicitly allows it. - -Tier 1 denies (relative to Tier 0): - * dbt subcommands that regenerate frozen artifacts or hit a - warehouse directly (`dbt run|test|parse|compile|docs|seed|...`). - Recce reads the frozen artifacts; the agent never needs to - regenerate them, and dbt invocations would let the agent bypass - Recce's structured query surfaces. - * Direct SQL clients (`duckdb`, `psql`, `snowsql`, `bq`) — at Tier 1 - warehouse access is mediated through Recce MCP tools (e.g. - `mcp__recce__query`), never raw shell. - * `sh -c ""`, `bash -c …`, `zsh -c …` wrappers that smuggle - a denied binary past a shallow check. - * Exec wrappers (`xargs`, `find -exec`, `time`, `nohup`, …) that - launch a denied binary as a child process — same bypass class as - `sh -c`, different syntactic shape. - -This is a denylist (Tier 0 is an allowlist) because Tier 1's -legitimate command surface is large — the agent can invoke arbitrary -git, grep, recce-*, and other dev-loop utilities. Tokenisation + -basename + wrapper recursion catch the bypass shapes the prior -case-glob and v2 hooks missed. +credentials. Recce MCP is intentionally not gated here. + +Architecture: + Parse the command with `bashlex` into an AST. Walk every node. + Substitutions, ANSI-C quoting, parameter defaults, Bash keywords + and command modifiers are handled by their AST shape, not by + regex/shlex heuristics. + +Two passes per command: + 1. WALK INTO every command substitution as if it were its own + top-level command. This catches `$(sh -c "dbt run")` because + the inner `sh -c "dbt run"` is itself denied. + 2. RESOLVE the head word of each command to its candidate output + strings (literal text, ANSI-C decoded, parameter defaults, + substitution payload words). If any candidate is a denied + binary OR `dbt` with a denied subcommand visible anywhere in + the args (including substitution payloads), deny. + +Threat model: + Adversarial agent reaches a denied binary via any Bash construct. + Static AST walk catches each construct shape; cwd separation + handles spoiler reads; recipe ensures no base/prod creds in env. + +Fail-closed behavior: + - Missing bashlex → exit 2 with install instructions. + - `coproc` keyword (bashlex doesn't support it) → exit 2. + - Other NotImplementedError from bashlex → exit 2. + - Generic bash ParsingError → fail open (a quoting bug in a + legitimate cell shouldn't brick the session). """ from __future__ import annotations import json import re -import shlex import sys from os.path import basename +from typing import Iterable + +try: + import bashlex + import bashlex.errors +except ImportError: + print( + "Tier-1 sandbox blocks: bashlex not installed. " + "Install with `python3 -m pip install bashlex` in the eval-runner " + "environment, then retry. Failing closed so bypasses can't slip " + "through silently.", + file=sys.stderr, + ) + sys.exit(2) + + +# --- Policy ---------------------------------------------------------------- -# Binaries the Tier-1 agent must not invoke directly. DENIED_BINS: frozenset[str] = frozenset( {"dbt", "duckdb", "psql", "snowsql", "bq"} ) -# dbt subcommands that regenerate frozen Tier-0 inputs or hit a -# warehouse directly. `dbt` alone (no subcommand), `dbt --help`, -# `dbt --version`, `dbt list` / `dbt ls`, `dbt deps`, `dbt clean` are -# allowed because they don't regenerate artifacts or touch the -# warehouse. -# -# `dbt clone` (dbt-core ≥1.6) materialises cloned models in the -# warehouse. `dbt retry` re-executes whatever the previous failed -# invocation was — if that was `dbt run`, retry re-runs it, so the -# deny semantics must inherit at the retry call. DBT_DENIED_SUBCOMMANDS: frozenset[str] = frozenset( {"run", "test", "parse", "compile", "docs", "seed", "snapshot", "build", "freshness", "run-operation", "debug", "source", @@ -61,285 +71,362 @@ {"sh", "bash", "zsh", "dash", "ash", "ksh"} ) -# `eval` is a shell builtin that concatenates its args and -# re-evaluates them as a command. Same threat surface as `sh -c` but -# with no `-c` flag — the inner command is just the joined positional -# args. Handled as a special case alongside SHELL_WRAPPERS. - -# Exec wrappers that run their argument as a new process. If the -# wrapped command is a denied binary, the hook would otherwise see -# only the wrapper and let the call through. We recurse into the -# wrapped command for each. -# -# `find` is handled separately because its wrapped command lives -# after a `-exec` / `-execdir` keyword, not at a fixed position. EXEC_WRAPPERS: frozenset[str] = frozenset( {"xargs", "time", "nice", "nohup", "setsid", "parallel", "exec", "timeout", "watch", "ionice", "chrt", "stdbuf"} ) +# `command`, `builtin` defeat alias/function shadowing — the actual +# binary is the next token. Walk past them transparently. +TRANSPARENT_PREFIXES: frozenset[str] = frozenset({"command", "builtin"}) + + +# --- Output --------------------------------------------------------------- def deny(reason: str) -> None: print(f"Tier-1 sandbox blocks: {reason}", file=sys.stderr) sys.exit(2) -def split_segments(cmd: str) -> list[str]: - segments: list[str] = [] - for sub in re.findall(r"\$\(([^()]*)\)", cmd): - segments.extend(split_segments(sub)) - cmd = re.sub(r"\$\([^()]*\)", "", cmd) - for sub in re.findall(r"`([^`]*)`", cmd): - segments.extend(split_segments(sub)) - cmd = re.sub(r"`[^`]*`", "", cmd) - # Negative lookbehind on `\\` keeps escaped separators (e.g. the - # `\;` that terminates `find -exec`) attached to the same segment; - # shlex strips the backslash later. - for part in re.split(r"(? list[str]: - try: - return shlex.split(segment) - except ValueError: - return [segment] +# --- Word resolution ------------------------------------------------------ +_ANSI_C_RE = re.compile(r"^\$'(.*)'$", re.DOTALL) + + +def _decode_ansi_c(inner: str) -> str: + """Naive ANSI-C escape decode: drop backslashes that precede a + char. `\\d\\b\\t` → `dbt`, which is the bypass we care about.""" + return re.sub(r"\\(.)", r"\1", inner) -def skip_leading_env(tokens: list[str]) -> list[str]: - i = 0 - while i < len(tokens) and "=" in tokens[i] and not tokens[i].startswith("="): - i += 1 - if i >= len(tokens): - return [] - if basename(tokens[i]) == "env": - i += 1 - while i < len(tokens): - t = tokens[i] - if t in ("-i", "-0", "--null"): - i += 1 - continue - if t in ("-u", "--unset"): - i += 2 - continue - if "=" in t and not t.startswith("="): - i += 1 - continue - break - return tokens[i:] - - -def has_denied_dbt_subcommand(tokens: list[str]) -> bool: - """True if any token after `dbt` is a banned subcommand. - - Earlier versions tried to find the "first positional" by skipping - leading flags. That broke on `dbt --target dev parse` — `dev` (the - value of `--target`) became the first positional and `parse` was - never inspected. Scanning every token after `dbt` instead is both - simpler and more robust: a false positive requires the agent to - pass a literal banned-subcommand name as a flag value (e.g. - `--target parse`), which is perverse and would deserve denial - anyway since the agent shouldn't be invoking dbt at Tier 1. - """ - return any(tok in DBT_DENIED_SUBCOMMANDS for tok in tokens[1:]) +def resolve_word(word_node, original_cmd: str) -> list[str]: + """Return possible string values this word could resolve to. -def find_wrapped_command_after_exec(tokens: list[str]) -> list[str]: - """For `find` with `-exec` / `-execdir`, return the wrapped command - tokens. Returns [] when `find` has no `-exec` — plain `find` is - safe and doesn't need recursion. + A bare `dbt` → ['dbt']. + `$'dbt'` (ANSI-C) → ['dbt'] (detected via raw position because + bashlex represents the construct as `$dbt` with a ParameterNode). + `${a:-dbt}` → ['dbt']. + `$(echo dbt)` → ['dbt', 'echo'] (each word inside the + substitution is a candidate output; over-approximate but + conservative for denial decisions). + `` `cmd` `` → same as `$(cmd)`. """ - for keyword in ("-exec", "-execdir"): - if keyword not in tokens: + pos = getattr(word_node, 'pos', None) + text = getattr(word_node, 'word', '') or '' + + # ANSI-C: check the raw source text because bashlex collapses + # `$'dbt'` into word='$dbt' + ParameterNode. + if pos and len(pos) == 2 and pos[1] > pos[0]: + raw = original_cmd[pos[0]:pos[1]] + m = _ANSI_C_RE.match(raw) + if m: + return [_decode_ansi_c(m.group(1))] + + parts = getattr(word_node, 'parts', []) or [] + if not parts: + return [text] + + candidates: list[str] = [] + for part in parts: + kind = getattr(part, 'kind', '') + if kind == 'parameter': + value = getattr(part, 'value', '') or '' + if ':-' in value: + candidates.append(value.split(':-', 1)[1]) + elif ':=' in value: + candidates.append(value.split(':=', 1)[1]) + # Bare $VAR — unknown at static time. Don't speculate. + elif kind == 'commandsubstitution': + sub_cmd = getattr(part, 'command', None) + if sub_cmd is not None: + for sub_word in _walk_command_words(sub_cmd): + candidates.extend(resolve_word(sub_word, original_cmd)) + + if not candidates: + return [text] + return candidates + + +def _walk_command_words(cmd_node) -> Iterable[object]: + """Yield all WordNode descendants of a command/list/pipeline node.""" + kind = getattr(cmd_node, 'kind', '') + if kind == 'word': + yield cmd_node + return + for part in getattr(cmd_node, 'parts', []) or []: + if not hasattr(part, 'kind'): continue - idx = tokens.index(keyword) - end = len(tokens) - for j in range(idx + 1, len(tokens)): - if tokens[j] in (";", "+", "\\;"): - end = j - break - return tokens[idx + 1:end] - return [] + if part.kind == 'word': + yield part + else: + yield from _walk_command_words(part) -def looks_like_executable(name: str) -> bool: - """Heuristic — does this token's basename look like it could be a - binary the agent is actually invoking? Rules out numbers (timeout - durations, chrt priorities), pure-symbol tokens (`{}`, `;`), and - empty strings. +def _walk_substitutions(word_node, original_cmd: str, depth: int) -> None: + """Walk INTO every command substitution found in this word as if + it were a top-level command. Catches `$(sh -c "dbt run")` because + the inner `sh -c "dbt run"` is itself denied. """ + for part in getattr(word_node, 'parts', []) or []: + if getattr(part, 'kind', '') == 'commandsubstitution': + sub_cmd = getattr(part, 'command', None) + if sub_cmd is not None: + walk(sub_cmd, original_cmd, depth + 1) + + +def _looks_like_executable(name: str) -> bool: if not name: return False if name in ("{}", "[]", ";", "+", "\\;"): return False if name.isdigit(): return False - # Allow alphanumerics + `_`/`-`/`.` (binaries like `git-foo`, - # `python3.11`, `recce-cli`). Anything else (e.g., a quoted-shell - # construct that survived shlex) is treated as non-executable so - # we don't spuriously deny. stripped = name.replace("-", "").replace("_", "").replace(".", "") return stripped.isalnum() -def find_wrapped_command(wrapper_name: str, tokens: list[str]) -> list[str]: - """For an exec wrapper at tokens[0], return the wrapped command's - tokens. +# --- AST walk ------------------------------------------------------------ - Different wrappers interleave their own positional args before the - wrapped command (e.g., `timeout 30 CMD`, `chrt 0 5 CMD`, - `ionice -c 2 CMD`). Hard-coding a per-wrapper arg parser is - brittle. Instead, walk the args and return tokens starting at the - first executable-shaped token — that's the wrapped command in - practice. `find` is special because the wrapped command lives - after a `-exec` / `-execdir` keyword. - """ - if wrapper_name == "find": - return find_wrapped_command_after_exec(tokens) - for i, tok in enumerate(tokens[1:], 1): - if tok.startswith("-"): - continue - if not looks_like_executable(basename(tok)): - continue - return tokens[i:] - return [] +def walk(node, original_cmd: str, depth: int = 0) -> None: + if depth > 8: + return + kind = getattr(node, 'kind', '') + if kind in ('list', 'pipeline', 'compound', 'if', 'for', 'while', + 'until', 'function', 'case'): + for child in getattr(node, 'parts', []) or []: + if hasattr(child, 'kind'): + walk(child, original_cmd, depth + 1) + return -def check_tokens(tokens: list[str], command_str: str, depth: int = 0) -> None: - """Inspect a token list. Recurses one level for shell wrappers and - exec wrappers. `depth` guards against pathological deep recursion. - """ - if depth > 4: + if kind != 'command': return - tokens = skip_leading_env(tokens) - if not tokens: + + words = [p for p in (node.parts or []) if getattr(p, 'kind', '') == 'word'] + if not words: return - name = basename(tokens[0]) - - # Shell wrapper with -c — recurse into its argument. - if name in SHELL_WRAPPERS: - for j in range(1, len(tokens) - 1): - if tokens[j] in ("-c", "-lc", "-ic"): - inner = tokens[j + 1] - for inner_segment in split_segments(inner): - check_tokens(tokenize(inner_segment), command_str, depth + 1) + + # Pass 1: walk INTO every substitution in every word. Catches + # bypasses where a substitution payload is itself a denied + # command (e.g., `$(sh -c "dbt run")` — even though the outer + # head is the substitution, the inner sh -c "dbt run" is a real + # denied call that bash will execute). + for w in words: + _walk_substitutions(w, original_cmd, depth) + + # Pass 2: head + args check on the current command. + idx = 0 + + # Skip transparent prefixes (`command`, `builtin`). + while idx < len(words): + cands = resolve_word(words[idx], original_cmd) + names = {basename(c) for c in cands} + if names & TRANSPARENT_PREFIXES: + idx += 1 + continue + # `env [-i] [-u VAR] [VAR=val ...] CMD ...` — skip env args. + if "env" in names and idx + 1 < len(words): + idx += 1 + while idx < len(words): + t = words[idx].word or '' + if t in ("-i", "-0", "--null"): + idx += 1 + continue + if t in ("-u", "--unset"): + idx += 2 + continue + if "=" in t and not t.startswith("="): + idx += 1 + continue + break + continue + break + + if idx >= len(words): + return + + head_word = words[idx] + head_candidates = resolve_word(head_word, original_cmd) + head_names = {basename(c) for c in head_candidates} + arg_words = words[idx + 1:] + + # --- Shell wrappers (sh -c, bash -lc, ...) --- + if head_names & SHELL_WRAPPERS: + for j in range(len(arg_words) - 1): + arg_text = arg_words[j].word or '' + if arg_text in ("-c", "-lc", "-ic"): + # Re-parse the LITERAL word text (with substitutions + # intact) instead of its substitution candidates. The + # candidates form `['echo', 'dbt']` for + # `$(echo dbt) run` and lose the ` run` suffix; using + # the literal `$(echo dbt) run` lets bashlex reparse + # it as the inner CommandNode with $() head + `run` + # arg, which the recursive walk catches. + inner_literal = arg_words[j + 1].word or '' + _reparse_and_walk(inner_literal, original_cmd, depth + 1) return - # A bare `sh` / `bash` interactive subshell with no -c arg is - # not a denied command on its own. return - # `eval ARGS...` concatenates its positional args and re-evaluates - # them as a new command. Same threat surface as `sh -c` but with - # the inner command spread across positional args instead of a - # single `-c` value. - if name == "eval": - inner = " ".join(tokens[1:]) - for inner_segment in split_segments(inner): - check_tokens(tokenize(inner_segment), command_str, depth + 1) + # --- eval ARGS... --- + # Inputs can be literal words OR substitution outputs. For each + # candidate "inner command", re-parse and walk. Substitution + # payloads are also independently walked by Pass 1 above. + if "eval" in head_names: + # Literal-word form: `eval foo bar baz` → "foo bar baz". + literal_parts: list[str] = [] + for arg in arg_words: + if not (getattr(arg, 'parts', None) or []): + literal_parts.append(arg.word or '') + if literal_parts: + _reparse_and_walk(" ".join(literal_parts), original_cmd, depth + 1) + # Substitution-form: `eval $(echo "dbt parse")` — each + # substitution candidate could be the eval'd command. + for arg in arg_words: + if getattr(arg, 'parts', None): + for cand in resolve_word(arg, original_cmd): + _reparse_and_walk(cand, original_cmd, depth + 1) return - # Exec wrapper (xargs / time / nohup / find -exec / ...). - if name in EXEC_WRAPPERS or name == "find": - wrapped = find_wrapped_command(name, tokens) + # --- Exec wrappers (xargs / time / nohup / find -exec / ...) --- + if head_names & EXEC_WRAPPERS or "find" in head_names: + wrapped = _find_exec_target(head_names, arg_words) if wrapped: - check_tokens(wrapped, command_str, depth + 1) + # CRITICAL: scan EVERY wrapped-command word's candidates + # for a denied binary. Exec wrappers supply args from + # stdin / find-output, so we can't trust a "bare dbt is + # allowed" reasoning here — any dbt invocation under an + # exec wrapper is suspect because the subcommand could + # come from the wrapper's dynamic args. + for w in wrapped: + for cand in resolve_word(w, original_cmd): + name = basename(cand) + if name in DENIED_BINS: + deny( + f"denied binary '{name}' as exec-wrapped " + f"command (matched in: {original_cmd!r})" + ) + # Also re-parse to catch nested wrappers. + head_cands = resolve_word(wrapped[0], original_cmd) + if head_cands: + rest = " ".join((w.word or '') for w in wrapped[1:]) + synth = (head_cands[0] + " " + rest).strip() + if synth: + _reparse_and_walk(synth, original_cmd, depth + 1) return - if name == "dbt": - if has_denied_dbt_subcommand(tokens): + # --- dbt --- + if "dbt" in head_names: + if _dbt_args_have_denied(arg_words, original_cmd): deny( "dbt subcommand regenerates frozen artifacts or hits a " - f"warehouse (matched in: {command_str!r})" + f"warehouse (matched in: {original_cmd!r})" ) return - if name in DENIED_BINS: + # --- Direct denied bin (psql / duckdb / snowsql / bq) --- + direct_denied = (head_names & DENIED_BINS) - {"dbt"} + if direct_denied: + name = next(iter(direct_denied)) deny( f"Direct SQL client '{name}' — use Recce MCP query instead " - f"(matched in: {command_str!r})" + f"(matched in: {original_cmd!r})" ) -def check_substitution_at_head(cmd: str) -> None: - """Detect `$()` / backtick substitutions at command-head position - that smuggle a denied binary. +def _find_exec_target(head_names: set[str], arg_words: list) -> list: + """Return the wrapped command's word list. For `find`, that's + the tokens after `-exec`/`-execdir` up to `;` / `+`. For other + wrappers, walk past flags and placeholders to the first + executable-shaped word.""" + if "find" in head_names: + for i, w in enumerate(arg_words): + if (w.word or '') in ("-exec", "-execdir"): + end = len(arg_words) + for j in range(i + 1, len(arg_words)): + if (arg_words[j].word or '') in (";", "+", "\\;"): + end = j + break + return arg_words[i + 1:end] + return [] + for i, w in enumerate(arg_words): + text = w.word or '' + if text.startswith("-"): + continue + # Word might be a substitution — _looks_like_executable on + # the raw text says False, but the substitution's payload + # could include a real binary. Either way we return from + # here and let the caller scan candidates. + if _looks_like_executable(basename(text)): + return arg_words[i:] + # Special: if the word is a substitution (has parts), it + # could resolve to a binary at runtime. Treat it as the + # wrapped-target so DENIED_BINS scan can catch it. + if getattr(w, 'parts', None): + return arg_words[i:] + return [] + - `$(echo dbt) run` evaluates to `dbt run` at bash runtime — the - substitution provides the head binary, which the regular - check_tokens path can't see because the substitution's content - has already been stripped from the outer segment. This pass walks - the original command and looks for substitutions at the head of - each segment. +def _dbt_args_have_denied(arg_words: list, original_cmd: str) -> bool: + """True if any resolved value of any arg (including substitution + payloads, ANSI-C decoded, parameter defaults) is a banned + dbt subcommand.""" + for arg in arg_words: + for cand in resolve_word(arg, original_cmd): + if cand in DBT_DENIED_SUBCOMMANDS: + return True + return False - The check is conservative — false positives only occur if the - agent legitimately has a denied-binary basename appear in a - substitution at command-head, which is the threat we're guarding - against and has no benign use at Tier 1. - """ - # Segment on true command separators only — don't split on `(`/`)` - # because that would shred `$(...)` into pieces (we want it intact - # to detect it as a head substitution). `split_segments` upstream - # uses parens too because it has already extracted `$()` before - # splitting; this pass works on the raw command. - for raw_seg in re.split(r"(? None: + if depth > 8 or not inner.strip(): + return + try: + trees = bashlex.parse(inner) + except (bashlex.errors.ParsingError, NotImplementedError): + # Unparseable inner — fall back to literal token scan. + for tok in inner.split(): + base = basename(tok) + if base in DENIED_BINS or base in SHELL_WRAPPERS or base in EXEC_WRAPPERS: deny( - f"denied binary '{sub_name}' smuggled via " - f"substitution at command-head " - f"(matched in: {cmd!r})" + f"unparseable inner command contains denied " + f"basename '{base}' (matched in: {original_cmd!r})" ) + return + for tree in trees: + walk(tree, original_cmd, depth) + +# --- Entry point --------------------------------------------------------- def main() -> None: try: payload = json.load(sys.stdin) except json.JSONDecodeError: return - - if payload.get("tool_name", "") != "Bash": + if payload.get("tool_name") != "Bash": return - - command = (payload.get("tool_input", {}) or {}).get("command", "") or "" - if not command.strip(): + cmd = (payload.get("tool_input") or {}).get("command") or "" + if not cmd.strip(): return - # Pre-check for `$()` / backtick substitutions at command-head - # position that smuggle a denied binary (e.g., `$(echo dbt) run`). - # split_segments extracts the substitution's content as its own - # segment, so the outer post-substitution segment loses the head - # binary — check_tokens alone can't see it. This pre-check fills - # that gap. - check_substitution_at_head(command) + # `coproc` is a Bash keyword bashlex can't parse. A Tier-1 agent + # has no legitimate reason for it; deny outright before the parser + # raises NotImplementedError. + if re.search(r"\bcoproc\b", cmd): + deny(f"`coproc` keyword is not allowed at Tier 1 (matched in: {cmd!r})") + + try: + trees = bashlex.parse(cmd) + except NotImplementedError as e: + deny( + f"bash construct not supported by parser ({e}); " + f"failing closed (matched in: {cmd!r})" + ) + except bashlex.errors.ParsingError: + return - for segment in split_segments(command): - check_tokens(tokenize(segment), command) + for tree in trees: + walk(tree, cmd, 0) if __name__ == "__main__": diff --git a/evals/agent-blind-spots/runs/2026-05-28/sandbox-verification.md b/evals/agent-blind-spots/runs/2026-05-28/sandbox-verification.md index ff21358..f5533f2 100644 --- a/evals/agent-blind-spots/runs/2026-05-28/sandbox-verification.md +++ b/evals/agent-blind-spots/runs/2026-05-28/sandbox-verification.md @@ -2,12 +2,13 @@ Per [DRC-3584](https://linear.app/recce/issue/DRC-3584) acceptance criterion #2: one fixture × {Claude Code, Codex} × {Tier-0, Tier-1} verified by hand, with agent traces inspected to confirm enforcement actually fires. -**Fixture:** `pr1-fix-clv`. **Worktree:** `.claude/worktrees/drc-3584-sandbox-profiles`. **Hook revision:** `v4` (Python; post-PR-#36-cycle-iteration-2). +**Fixture:** `pr1-fix-clv`. **Worktree:** `.claude/worktrees/drc-3584-sandbox-profiles`. **Hook revision:** `v5` (bashlex AST; post-PR-#36-cycle-iteration-3). -The v1 case-glob bash hooks (shipped in the first PR-#36 commit) were superseded three times: -- **v2** (Python rewrite) — closed the 6 bypass shapes the case-glob couldn't address (shell separators, absolute paths, `sh -c`, dbt global flags, skill case-sensitivity, the false ENFORCEMENT.md:71 claim). -- **v3** — closed the 2 additional bypass shapes the cycle iter-1 surfaced (Tier-1 dbt flag-with-value, exec-wrapper-launches-denied-binary). -- **v4** (this revision) — closed 3 additional bypass classes the cycle iter-2 surfaced (`eval` shell-builtin smuggling, `$()`/backtick substitution at command-head, missing dbt subcommands `clone`/`retry`). +The v1 case-glob bash hooks (shipped in the first PR-#36 commit) were superseded four times: +- **v2** (Python rewrite using shlex + regex) — closed the 6 bypass shapes the case-glob couldn't address (shell separators, absolute paths, `sh -c`, dbt global flags, skill case-sensitivity, the false ENFORCEMENT.md:71 claim). +- **v3** — closed the 2 additional bypass shapes cycle iter-1 surfaced (Tier-1 dbt flag-with-value, exec-wrapper-launches-denied-binary). +- **v4** — closed 3 additional bypass classes cycle iter-2 surfaced (`eval` shell-builtin smuggling, `$()`/backtick substitution at command-head, missing dbt subcommands `clone`/`retry`). +- **v5** (this revision) — closed 9 additional bypass classes cycle iter-3 surfaced. Required a structural rewrite from regex/shlex to `bashlex` (a real Bash AST parser) because the bypasses exploited Bash semantics shlex/regex couldn't model: nested `$()`, ANSI-C `$'...'`, parameter expansion `${a:-default}`, Bash keywords (`coproc`, `!`), command modifiers (`command`, `builtin`), exec-wrapper smuggling (`xargs -I {} sh -c "{} parse" dbt`), and substitution-produces-subcommand (`dbt $(echo run)`). The "Bypass attempts" tables below are the load-bearing evidence that v3 closes every reviewer-named bypass. All rows are exit-2 expected; the few exit-0 entries are explicit counter-claims (a documented allow path that protects the rubric for a different reason — typically cwd separation rather than the hook). @@ -89,6 +90,19 @@ Note: the cycle review (`v2 review`, NOTE 5) flagged the matcher regex `Bash|Ski | **Counter-claim**: benign `$()` head | `$(which python) script.py` | iter-2 | ✅ exit 0 | | **Counter-claim**: benign `$(echo grep)` | `$(echo grep) -rn foo .` | iter-2 | ✅ exit 0 | | **Counter-claim**: dbt-without-subcommand via `$()` | `$(echo dbt)` | iter-2 | ✅ exit 0 (bare dbt allowed) | +| **v5 — nested `$()` L1** | `$(sh -c "$(echo dbt) run")` | iter-3 | ✅ exit 2 | +| **v5 — nested `$()` L2** | `$($(echo dbt) run)` | iter-3 | ✅ exit 2 | +| **v5 — nested backtick + `$()`** | `` `$(echo dbt) run` `` | iter-3 | ✅ exit 2 | +| **v5 — `eval $(...)` smuggling** | `eval $(echo "dbt parse")` | iter-3 | ✅ exit 2 | +| **v5 — `xargs $(echo dbt)`** | `xargs $(echo dbt)` | iter-3 | ✅ exit 2 (denied: xargs-wrapped denied basename) | +| **v5 — substitution-supplied subcommand** | `dbt $(echo run)` | iter-3 | ✅ exit 2 | +| **v5 — ANSI-C `$'dbt'`** | `$'dbt' parse` | iter-3 | ✅ exit 2 | +| **v5 — parameter expansion `${a:-dbt}`** | `${a:-dbt} parse` | iter-3 | ✅ exit 2 | +| **v5 — `coproc` keyword** | `coproc dbt run` | iter-3 | ✅ exit 2 (denied pre-parse; bashlex doesn't support coproc) | +| **v5 — `command` modifier** | `command dbt run` | iter-3 | ✅ exit 2 (transparent prefix; dbt revealed underneath) | +| **v5 — `builtin eval`** | `builtin eval dbt run` | iter-3 | ✅ exit 2 | +| **v5 — `!` negation prefix** | `! dbt run` | iter-3 | ✅ exit 2 (pipeline negation walked) | +| **v5 — `xargs -I {} sh -c "{} parse" dbt`** | `xargs -I {} sh -c "{} parse" dbt` | iter-3 | ✅ exit 2 (denied: dbt in wrapped position) | ## Happy-path coverage (regression check) From 977d17516ec3f6de4c365b474700d7768e6ec735 Mon Sep 17 00:00:00 2001 From: even-wei Date: Fri, 29 May 2026 10:57:47 +0800 Subject: [PATCH 06/13] fix(evals): close iter-4 BLOCKERs (process sub / CompoundNode / brace / xargs-chain / \$0) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Iter-4 cycle review found 7 BLOCKERs + 5 ISSUEs against v5's bashlex hooks — all rooted in AST-walker gaps inside the bashlex code (not in bashlex itself). v6 closes them. BLOCKERs: 1+7. **Process substitution `<(...)` / `>(...)`** — `_walk_substitutions` only handled `commandsubstitution` kind. Now also handles `processsubstitution` (same `.command` attribute). Catches `diff <(dbt run) /etc/hosts` (head-position) AND `echo x 2> >(dbt parse)` (process sub inside a RedirectNode — `walk()` now also iterates `kind == 'redirect'` parts and walks their `.output` word's substitutions). 2. **CompoundNode.list** — bashlex's subshell `(...)` and group `{ ...; }` store children in `.list`, not `.parts`. The walker's descent path now tries `.list` first then `.parts`, so `(dbt parse)` and `{ dbt parse; }` are no longer silently allowed. 3+4. **Brace expansion** — bashlex returns `{dbt,bash}` and `{parse,run}` as literal `word` text with no `parts`. New `_expand_brace_literal()` regex decodes `prefix{a,b,c}suffix` into candidate list (`['prefix.a.suffix', 'prefix.b.suffix', ...]`). Used by `resolve_word` for both head and args. Multi-candidate head fix: `{dbt,bash} run` resolves to `{'dbt','bash'}`. Before v6, the shell-wrapper branch fired on `bash` and returned without checking the dbt arm. Now an eager-deny up front checks dbt+args and direct DENIED_BINS BEFORE the shell-wrapper / eval / exec-wrapper branches — any candidate matching a denied shape denies regardless of what other candidates are present. 5. **xargs → sh -c chain** — `echo dbt | xargs -I {} sh -c "{} parse"` exec-wrapper found `sh` as wrapped, but `sh` isn't in DENIED_BINS so the wrapper-scan exited without escalating. v6 adds: if the wrapped binary IS a shell wrapper, find the `-c` arg; if the xargs `-I` placeholder appears inside the shell body, conservatively deny (placeholder substitution from stdin is dynamic — can't be statically resolved, and the pattern has very narrow legit use). Otherwise re-parse the literal -c text. 6. **`$0` head** — bashlex parses `$0` as ParameterNode(value='0'). `resolve_word`'s parameter branch only handled `:-`/`:=` defaults. Now also handles `0`, `BASH`, `SHELL`, `BASH_SOURCE` — these resolve at runtime to a shell name; we surface them as the candidate `"sh"` so the shell-wrapper branch fires conservatively. ISSUEs: 8. **coproc regex inside string literals** — pre-parse `\bcoproc\b` match was firing on `echo coproc`, `echo 'no coproc here'`, etc. Removed the pre-parse check; rely on bashlex's NotImplementedError raised when the actual `coproc` keyword is present. Error message is inspected for `'coproc'` to differentiate from other unsupported constructs. 9. **NotImplementedError fail-closed contradicted docstring** — bashlex raises NotImplementedError on legit `$((arith))`, `select`, etc. v5 denied all such cases. v6 only denies if the error message mentions `coproc`; other unsupported constructs fall through (the rest of the sandbox catches denied binaries). 10. **`_looks_like_executable`** — added `[[`, `]]`, `;;`, `&&`, `||` to the skip set. 11. **ANSI-C decoder** — switched from `re.sub(r"\\(.)", r"\1", inner)` to `codecs.decode(inner, 'unicode_escape')` for proper bash escape semantics. Falls back to the naive regex on decode error. 12. **Reparse fallback** — already scans every token, no change needed. Verification: 49 cells across both tiers + iter-1/2/3/4 regressions + false-positive sanity. All green. Build re-verified across all 6 fixtures. Closes the iter-4 cycle's 7 BLOCKERs and 5 ISSUEs. Signed-off-by: even-wei --- .../claude-overlay/hooks/deny-tier-0.py | 72 ++++++- .../claude-overlay/hooks/deny-tier-1.py | 198 ++++++++++++++++-- .../runs/2026-05-28/sandbox-verification.md | 17 +- 3 files changed, 250 insertions(+), 37 deletions(-) diff --git a/evals/agent-blind-spots/runner-configs/claude-code/tier-0/claude-overlay/hooks/deny-tier-0.py b/evals/agent-blind-spots/runner-configs/claude-code/tier-0/claude-overlay/hooks/deny-tier-0.py index 13ffacd..be28fe3 100755 --- a/evals/agent-blind-spots/runner-configs/claude-code/tier-0/claude-overlay/hooks/deny-tier-0.py +++ b/evals/agent-blind-spots/runner-configs/claude-code/tier-0/claude-overlay/hooks/deny-tier-0.py @@ -88,10 +88,32 @@ def deny(reason: str) -> None: # --- Word resolution ------------------------------------------------------ _ANSI_C_RE = re.compile(r"^\$'(.*)'$", re.DOTALL) +_BRACE_LITERAL_RE = re.compile(r"^([^{},\s]*)\{([^{}]+)\}([^{}]*)$") + +# Parameters that at runtime resolve to a shell name. We can't predict +# which shell, so treat them as if they were a shell wrapper. +_SHELL_NAME_PARAMS: frozenset[str] = frozenset( + {"0", "BASH", "SHELL", "BASH_SOURCE"} +) def _decode_ansi_c(inner: str) -> str: - return re.sub(r"\\(.)", r"\1", inner) + try: + import codecs + return codecs.decode(inner, 'unicode_escape') + except (UnicodeDecodeError, ValueError): + return re.sub(r"\\(.)", r"\1", inner) + + +def _expand_brace_literal(text: str) -> list[str] | None: + m = _BRACE_LITERAL_RE.match(text) + if not m: + return None + prefix, inner, suffix = m.group(1), m.group(2), m.group(3) + parts = [p.strip() for p in inner.split(",")] + if len(parts) < 2: + return None + return [f"{prefix}{p}{suffix}" for p in parts] def resolve_word(word_node, original_cmd: str) -> list[str]: @@ -108,6 +130,9 @@ def resolve_word(word_node, original_cmd: str) -> list[str]: parts = getattr(word_node, 'parts', []) or [] if not parts: + expanded = _expand_brace_literal(text) + if expanded is not None: + return expanded return [text] candidates: list[str] = [] @@ -119,13 +144,23 @@ def resolve_word(word_node, original_cmd: str) -> list[str]: candidates.append(value.split(':-', 1)[1]) elif ':=' in value: candidates.append(value.split(':=', 1)[1]) + elif value in _SHELL_NAME_PARAMS: + candidates.append("sh") elif kind == 'commandsubstitution': sub_cmd = getattr(part, 'command', None) if sub_cmd is not None: for sub_word in _walk_command_words(sub_cmd): candidates.extend(resolve_word(sub_word, original_cmd)) + elif kind == 'processsubstitution': + sub_cmd = getattr(part, 'command', None) + if sub_cmd is not None: + for sub_word in _walk_command_words(sub_cmd): + candidates.extend(resolve_word(sub_word, original_cmd)) if not candidates: + expanded = _expand_brace_literal(text) + if expanded is not None: + return expanded return [text] return candidates @@ -146,7 +181,8 @@ def _walk_command_words(cmd_node) -> Iterable[object]: def _walk_substitutions(word_node, original_cmd: str, depth: int) -> None: for part in getattr(word_node, 'parts', []) or []: - if getattr(part, 'kind', '') == 'commandsubstitution': + kind = getattr(part, 'kind', '') + if kind in ('commandsubstitution', 'processsubstitution'): sub_cmd = getattr(part, 'command', None) if sub_cmd is not None: walk(sub_cmd, original_cmd, depth + 1) @@ -155,7 +191,7 @@ def _walk_substitutions(word_node, original_cmd: str, depth: int) -> None: def _looks_like_executable(name: str) -> bool: if not name: return False - if name in ("{}", "[]", ";", "+", "\\;"): + if name in ("{}", "[]", ";", "+", "\\;", "[[", "]]", ";;", "&&", "||"): return False if name.isdigit(): return False @@ -172,7 +208,10 @@ def walk(node, original_cmd: str, depth: int = 0) -> None: if kind in ('list', 'pipeline', 'compound', 'if', 'for', 'while', 'until', 'function', 'case'): - for child in getattr(node, 'parts', []) or []: + # CompoundNode uses `.list`, other containers use `.parts`. + children = (getattr(node, 'list', None) + or getattr(node, 'parts', None) or []) + for child in children: if hasattr(child, 'kind'): walk(child, original_cmd, depth + 1) return @@ -189,6 +228,14 @@ def walk(node, original_cmd: str, depth: int = 0) -> None: for w in words: _walk_substitutions(w, original_cmd, depth) + # Pass 1b: redirect targets can be process substitutions + # (`echo x 2> >(recce check)`); walk those too. + for p in (node.parts or []): + if getattr(p, 'kind', '') == 'redirect': + out = getattr(p, 'output', None) + if out is not None and getattr(out, 'kind', '') == 'word': + _walk_substitutions(out, original_cmd, depth) + # Pass 2: head check. idx = 0 while idx < len(words): @@ -352,16 +399,19 @@ def main() -> None: if not cmd.strip(): return - if re.search(r"\bcoproc\b", cmd): - deny(f"`coproc` keyword is not allowed at Tier 0 (matched in: {cmd!r})") - try: trees = bashlex.parse(cmd) except NotImplementedError as e: - deny( - f"bash construct not supported by parser ({e}); " - f"failing closed (matched in: {cmd!r})" - ) + msg = str(e).lower() + if 'coproc' in msg: + deny( + f"`coproc` keyword is not allowed at Tier 0 " + f"(matched in: {cmd!r})" + ) + # Other unsupported constructs (arithmetic expansion, select, + # etc.) fall open — they're legitimate Bash idioms; the rest + # of the sandbox catches denied binaries. + return except bashlex.errors.ParsingError: return diff --git a/evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/hooks/deny-tier-1.py b/evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/hooks/deny-tier-1.py index 527faac..94df09b 100755 --- a/evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/hooks/deny-tier-1.py +++ b/evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/hooks/deny-tier-1.py @@ -91,12 +91,44 @@ def deny(reason: str) -> None: # --- Word resolution ------------------------------------------------------ _ANSI_C_RE = re.compile(r"^\$'(.*)'$", re.DOTALL) +_BRACE_LITERAL_RE = re.compile(r"^([^{},\s]*)\{([^{}]+)\}([^{}]*)$") + +# Parameters that at runtime resolve to a shell name. We can't predict +# which shell, so treat them as if they were a shell wrapper — same +# threat surface as `sh`. Includes positional $0 (the script name) +# and $BASH, $SHELL, $BASH_SOURCE, $0 alias. +_SHELL_NAME_PARAMS: frozenset[str] = frozenset( + {"0", "BASH", "SHELL", "BASH_SOURCE"} +) def _decode_ansi_c(inner: str) -> str: - """Naive ANSI-C escape decode: drop backslashes that precede a - char. `\\d\\b\\t` → `dbt`, which is the bypass we care about.""" - return re.sub(r"\\(.)", r"\1", inner) + """Decode ANSI-C escapes the way bash does (close enough for our + threat model): \\d → d (literal, since \\d isn't a bash escape), + \\n → newline, \\t → tab, etc. We use `unicode_escape` for the + standard set; characters that aren't escape codes survive as-is. + """ + try: + import codecs + return codecs.decode(inner, 'unicode_escape') + except (UnicodeDecodeError, ValueError): + # Malformed escape — fall back to dropping backslashes. + return re.sub(r"\\(.)", r"\1", inner) + + +def _expand_brace_literal(text: str) -> list[str] | None: + """For a literal word like `{dbt,bash}` or `prefix{a,b}suffix`, + return the expanded list of candidates. Returns None when no + brace expansion is present. Doesn't handle nested or sequence + expansions (`{a..z}`); those are rare in adversarial contexts.""" + m = _BRACE_LITERAL_RE.match(text) + if not m: + return None + prefix, inner, suffix = m.group(1), m.group(2), m.group(3) + parts = [p.strip() for p in inner.split(",")] + if len(parts) < 2: + return None + return [f"{prefix}{p}{suffix}" for p in parts] def resolve_word(word_node, original_cmd: str) -> list[str]: @@ -106,10 +138,13 @@ def resolve_word(word_node, original_cmd: str) -> list[str]: `$'dbt'` (ANSI-C) → ['dbt'] (detected via raw position because bashlex represents the construct as `$dbt` with a ParameterNode). `${a:-dbt}` → ['dbt']. + `{dbt,bash}` (brace literal) → ['dbt', 'bash']. `$(echo dbt)` → ['dbt', 'echo'] (each word inside the substitution is a candidate output; over-approximate but conservative for denial decisions). - `` `cmd` `` → same as `$(cmd)`. + `$0` / `$BASH` / `$SHELL` → ['sh'] (placeholder that triggers + the shell-wrapper branch in walk(); we can't predict the actual + shell name). """ pos = getattr(word_node, 'pos', None) text = getattr(word_node, 'word', '') or '' @@ -124,6 +159,10 @@ def resolve_word(word_node, original_cmd: str) -> list[str]: parts = getattr(word_node, 'parts', []) or [] if not parts: + # Literal word. Check for brace expansion. + expanded = _expand_brace_literal(text) + if expanded is not None: + return expanded return [text] candidates: list[str] = [] @@ -135,14 +174,35 @@ def resolve_word(word_node, original_cmd: str) -> list[str]: candidates.append(value.split(':-', 1)[1]) elif ':=' in value: candidates.append(value.split(':=', 1)[1]) - # Bare $VAR — unknown at static time. Don't speculate. + elif value in _SHELL_NAME_PARAMS: + # `$0`, `$BASH`, `$SHELL` — resolves to a shell at + # runtime. Surface as 'sh' so the shell-wrapper + # branch fires conservatively. + candidates.append("sh") + # Other bare $VAR — unknown at static time. Don't speculate. elif kind == 'commandsubstitution': sub_cmd = getattr(part, 'command', None) if sub_cmd is not None: for sub_word in _walk_command_words(sub_cmd): candidates.extend(resolve_word(sub_word, original_cmd)) + elif kind == 'processsubstitution': + # `<(cmd)` / `>(cmd)` — same threat surface as $(): the + # inner command runs. We still walk INTO it via + # _walk_substitutions (security pass); here we also + # add its words as candidate outputs since process-sub + # in head position resolves to a /dev/fd/N path that + # bash would `exec` — unusual but possible. + sub_cmd = getattr(part, 'command', None) + if sub_cmd is not None: + for sub_word in _walk_command_words(sub_cmd): + candidates.extend(resolve_word(sub_word, original_cmd)) if not candidates: + # Expansion present but no useful candidate — fall back to + # the literal text (also try brace expansion on the literal). + expanded = _expand_brace_literal(text) + if expanded is not None: + return expanded return [text] return candidates @@ -163,12 +223,14 @@ def _walk_command_words(cmd_node) -> Iterable[object]: def _walk_substitutions(word_node, original_cmd: str, depth: int) -> None: - """Walk INTO every command substitution found in this word as if - it were a top-level command. Catches `$(sh -c "dbt run")` because - the inner `sh -c "dbt run"` is itself denied. + """Walk INTO every command substitution or process substitution + found in this word as if it were a top-level command. Catches + `$(sh -c "dbt run")` and `diff <(dbt run)` because their inner + commands are independently checked. """ for part in getattr(word_node, 'parts', []) or []: - if getattr(part, 'kind', '') == 'commandsubstitution': + kind = getattr(part, 'kind', '') + if kind in ('commandsubstitution', 'processsubstitution'): sub_cmd = getattr(part, 'command', None) if sub_cmd is not None: walk(sub_cmd, original_cmd, depth + 1) @@ -177,7 +239,7 @@ def _walk_substitutions(word_node, original_cmd: str, depth: int) -> None: def _looks_like_executable(name: str) -> bool: if not name: return False - if name in ("{}", "[]", ";", "+", "\\;"): + if name in ("{}", "[]", ";", "+", "\\;", "[[", "]]", ";;", "&&", "||"): return False if name.isdigit(): return False @@ -194,7 +256,12 @@ def walk(node, original_cmd: str, depth: int = 0) -> None: if kind in ('list', 'pipeline', 'compound', 'if', 'for', 'while', 'until', 'function', 'case'): - for child in getattr(node, 'parts', []) or []: + # CompoundNode stores its children under `.list`, not `.parts`. + # Other container kinds use `.parts`. Descend on whichever + # attribute is populated. + children = (getattr(node, 'list', None) + or getattr(node, 'parts', None) or []) + for child in children: if hasattr(child, 'kind'): walk(child, original_cmd, depth + 1) return @@ -214,6 +281,14 @@ def walk(node, original_cmd: str, depth: int = 0) -> None: for w in words: _walk_substitutions(w, original_cmd, depth) + # Pass 1b: redirect targets can be process substitutions + # (`echo x 2> >(dbt parse)`); walk those too. + for p in (node.parts or []): + if getattr(p, 'kind', '') == 'redirect': + out = getattr(p, 'output', None) + if out is not None and getattr(out, 'kind', '') == 'word': + _walk_substitutions(out, original_cmd, depth) + # Pass 2: head + args check on the current command. idx = 0 @@ -250,6 +325,26 @@ def walk(node, original_cmd: str, depth: int = 0) -> None: head_names = {basename(c) for c in head_candidates} arg_words = words[idx + 1:] + # When the head resolves to multiple candidates (brace expansion + # `{dbt,bash}`, substitution payload `$(echo dbt)`), the same + # command could exec ANY of them at runtime. Apply the dbt rule + # eagerly here so a brace literal that includes `dbt` plus a + # denied subcommand in args (`{dbt,bash} run`) denies — without + # this, the shell-wrapper branch below would handle `bash` and + # return without ever checking the dbt arm. + if "dbt" in head_names and _dbt_args_have_denied(arg_words, original_cmd): + deny( + "dbt subcommand regenerates frozen artifacts or hits a " + f"warehouse (matched in: {original_cmd!r})" + ) + eager_denied = (head_names & DENIED_BINS) - {"dbt"} + if eager_denied: + name = next(iter(eager_denied)) + deny( + f"Direct SQL client '{name}' — use Recce MCP query instead " + f"(matched in: {original_cmd!r})" + ) + # --- Shell wrappers (sh -c, bash -lc, ...) --- if head_names & SHELL_WRAPPERS: for j in range(len(arg_words) - 1): @@ -305,8 +400,45 @@ def walk(node, original_cmd: str, depth: int = 0) -> None: f"denied binary '{name}' as exec-wrapped " f"command (matched in: {original_cmd!r})" ) - # Also re-parse to catch nested wrappers. + + # If the wrapped binary is itself a shell wrapper + # (xargs -I {} sh -c "..."), the exec-wrapper's args + # become input to that shell. Re-parse the -c arg as + # bash so the inner construct is walked again. Without + # this, `xargs -I {} sh -c "{} parse" dbt` slips: the + # exec-wrapper scan sees only `sh`, and the {} doesn't + # statically resolve to dbt. head_cands = resolve_word(wrapped[0], original_cmd) + head_basenames = {basename(c) for c in head_cands} + if head_basenames & SHELL_WRAPPERS: + for j in range(1, len(wrapped) - 1): + if (wrapped[j].word or '') in ("-c", "-lc", "-ic"): + inner_literal = wrapped[j + 1].word or '' + placeholder = _xargs_placeholder(arg_words) + if (placeholder + and placeholder in inner_literal + and "xargs" in head_names): + # `xargs -I X sh -c "...X..."` is a classic + # command-injection pattern: the placeholder + # is replaced at runtime by stdin lines, so + # the shell command is dynamic. Static + # analysis can't predict what stdin will + # supply; deny conservatively. + deny( + f"`xargs -I {placeholder} sh -c \"...\"` " + f"with placeholder in the shell body is " + f"a dynamic-command construction — deny " + f"(matched in: {original_cmd!r})" + ) + # No placeholder smuggling — reparse the + # literal -c text and let the recursive walk + # check the inner construct. + _reparse_and_walk( + inner_literal, original_cmd, depth + 1 + ) + return + + # Default: synthesize and re-walk to catch nested wrappers. if head_cands: rest = " ".join((w.word or '') for w in wrapped[1:]) synth = (head_cands[0] + " " + rest).strip() @@ -333,6 +465,15 @@ def walk(node, original_cmd: str, depth: int = 0) -> None: ) +def _xargs_placeholder(arg_words: list) -> str | None: + """For `xargs -I PLACEHOLDER ...`, return PLACEHOLDER. Default is + `{}` when -I has no arg or when the wrapper isn't xargs.""" + for i, w in enumerate(arg_words): + if (w.word or '') in ("-I", "--replace") and i + 1 < len(arg_words): + return arg_words[i + 1].word or '{}' + return '{}' + + def _find_exec_target(head_names: set[str], arg_words: list) -> list: """Return the wrapped command's word list. For `find`, that's the tokens after `-exec`/`-execdir` up to `;` / `+`. For other @@ -383,7 +524,10 @@ def _reparse_and_walk(inner: str, original_cmd: str, depth: int) -> None: try: trees = bashlex.parse(inner) except (bashlex.errors.ParsingError, NotImplementedError): - # Unparseable inner — fall back to literal token scan. + # Unparseable inner — fall back to a literal token scan that + # walks EVERY token (not just the first). A constructed inner + # that surfaces `:` first and the wrapper second escapes a + # first-token-only scan. for tok in inner.split(): base = basename(tok) if base in DENIED_BINS or base in SHELL_WRAPPERS or base in EXEC_WRAPPERS: @@ -409,19 +553,27 @@ def main() -> None: if not cmd.strip(): return - # `coproc` is a Bash keyword bashlex can't parse. A Tier-1 agent - # has no legitimate reason for it; deny outright before the parser - # raises NotImplementedError. - if re.search(r"\bcoproc\b", cmd): - deny(f"`coproc` keyword is not allowed at Tier 1 (matched in: {cmd!r})") - try: trees = bashlex.parse(cmd) except NotImplementedError as e: - deny( - f"bash construct not supported by parser ({e}); " - f"failing closed (matched in: {cmd!r})" - ) + # Only deny for `coproc` (real bypass surface — a Tier-1 + # agent has no legitimate reason to background-spawn a + # named pipe to dbt). Other unsupported constructs (e.g. + # arithmetic expansion `$((1+2))`, `select` keyword, etc.) + # are legitimate Bash idioms that agents use; falling open + # there leaves the rest of the sandbox (PATH scrub at the + # runner level, MCP allowlist, settings.json deny) as the + # net. Conservatively, also deny if the error message + # contains any obvious wrapper word that hints the construct + # might be smuggling a denied binary, but accept the + # arithmetic / select / brace-expansion shapes. + msg = str(e).lower() + if 'coproc' in msg: + deny( + f"`coproc` keyword is not allowed at Tier 1 " + f"(matched in: {cmd!r})" + ) + return except bashlex.errors.ParsingError: return diff --git a/evals/agent-blind-spots/runs/2026-05-28/sandbox-verification.md b/evals/agent-blind-spots/runs/2026-05-28/sandbox-verification.md index f5533f2..2ca2533 100644 --- a/evals/agent-blind-spots/runs/2026-05-28/sandbox-verification.md +++ b/evals/agent-blind-spots/runs/2026-05-28/sandbox-verification.md @@ -2,13 +2,14 @@ Per [DRC-3584](https://linear.app/recce/issue/DRC-3584) acceptance criterion #2: one fixture × {Claude Code, Codex} × {Tier-0, Tier-1} verified by hand, with agent traces inspected to confirm enforcement actually fires. -**Fixture:** `pr1-fix-clv`. **Worktree:** `.claude/worktrees/drc-3584-sandbox-profiles`. **Hook revision:** `v5` (bashlex AST; post-PR-#36-cycle-iteration-3). +**Fixture:** `pr1-fix-clv`. **Worktree:** `.claude/worktrees/drc-3584-sandbox-profiles`. **Hook revision:** `v6` (bashlex AST; post-PR-#36-cycle-iteration-4). -The v1 case-glob bash hooks (shipped in the first PR-#36 commit) were superseded four times: +The v1 case-glob bash hooks (shipped in the first PR-#36 commit) were superseded five times: - **v2** (Python rewrite using shlex + regex) — closed the 6 bypass shapes the case-glob couldn't address (shell separators, absolute paths, `sh -c`, dbt global flags, skill case-sensitivity, the false ENFORCEMENT.md:71 claim). - **v3** — closed the 2 additional bypass shapes cycle iter-1 surfaced (Tier-1 dbt flag-with-value, exec-wrapper-launches-denied-binary). - **v4** — closed 3 additional bypass classes cycle iter-2 surfaced (`eval` shell-builtin smuggling, `$()`/backtick substitution at command-head, missing dbt subcommands `clone`/`retry`). -- **v5** (this revision) — closed 9 additional bypass classes cycle iter-3 surfaced. Required a structural rewrite from regex/shlex to `bashlex` (a real Bash AST parser) because the bypasses exploited Bash semantics shlex/regex couldn't model: nested `$()`, ANSI-C `$'...'`, parameter expansion `${a:-default}`, Bash keywords (`coproc`, `!`), command modifiers (`command`, `builtin`), exec-wrapper smuggling (`xargs -I {} sh -c "{} parse" dbt`), and substitution-produces-subcommand (`dbt $(echo run)`). +- **v5** — closed 9 additional bypass classes cycle iter-3 surfaced. Required a structural rewrite from regex/shlex to `bashlex` (real Bash AST parser). +- **v6** (this revision) — closed 7 BLOCKERs + 5 ISSUEs cycle iter-4 surfaced. These were AST-walker gaps inside the bashlex code: process substitution `<(...)`/`>(...)` not walked, `CompoundNode.list` not descended (subshells `(...)`, groups `{ ...; }` silently allowed), brace expansion (`{dbt,bash}`, `dbt {parse,run}`) left as literal words, xargs→sh-c chain with placeholder substitution, `$0`/`$BASH` parameter resolution. ISSUEs: `coproc` regex false-positives inside string literals (now relies on bashlex's NotImplementedError + error-message check), `$((arith))` no longer fails closed (legit construct), ANSI-C decoder uses `codecs.decode` with `unicode_escape`, `_looks_like_executable` adds `[[`/`]]`/`;;`/`&&`/`||`, reparse fallback scans all tokens. The "Bypass attempts" tables below are the load-bearing evidence that v3 closes every reviewer-named bypass. All rows are exit-2 expected; the few exit-0 entries are explicit counter-claims (a documented allow path that protects the rubric for a different reason — typically cwd separation rather than the hook). @@ -103,6 +104,16 @@ Note: the cycle review (`v2 review`, NOTE 5) flagged the matcher regex `Bash|Ski | **v5 — `builtin eval`** | `builtin eval dbt run` | iter-3 | ✅ exit 2 | | **v5 — `!` negation prefix** | `! dbt run` | iter-3 | ✅ exit 2 (pipeline negation walked) | | **v5 — `xargs -I {} sh -c "{} parse" dbt`** | `xargs -I {} sh -c "{} parse" dbt` | iter-3 | ✅ exit 2 (denied: dbt in wrapped position) | +| **v6 — process substitution `<(`** | `diff <(dbt run) /etc/hosts` | iter-4 | ✅ exit 2 (ProcesssubstitutionNode walked) | +| **v6 — process substitution `>(` in redirect** | `echo x 2> >(dbt parse)` | iter-4 | ✅ exit 2 (RedirectNode.output walked) | +| **v6 — subshell `(...)`** | `(dbt parse)` | iter-4 | ✅ exit 2 (CompoundNode.list descended) | +| **v6 — group `{ ...; }`** | `{ dbt parse; }` | iter-4 | ✅ exit 2 | +| **v6 — brace head** | `{dbt,bash} run` | iter-4 | ✅ exit 2 (brace literal decoded into candidates) | +| **v6 — brace dbt subcommand** | `dbt {parse,run}` | iter-4 | ✅ exit 2 | +| **v6 — xargs→sh-c chain with placeholder** | `echo dbt \| xargs -I {} sh -c "{} parse"` | iter-4 | ✅ exit 2 (placeholder-in-body denied conservatively) | +| **v6 — `$0` positional** | `$0 -c "dbt parse"` | iter-4 | ✅ exit 2 (`$0` resolves to shell wrapper candidate) | +| **v6 ISSUE — coproc inside string literal** | `echo coproc` | iter-4 | ✅ exit 0 (no longer false-positives) | +| **v6 ISSUE — arithmetic expansion** | `echo $((1+1))` | iter-4 | ✅ exit 0 (no longer fails closed) | ## Happy-path coverage (regression check) From 13848640a3b7410174e0c752d9603d89549b4c2a Mon Sep 17 00:00:00 2001 From: even-wei Date: Fri, 29 May 2026 17:26:05 +0800 Subject: [PATCH 07/13] docs(evals): add non-adversarial threat-model section to ENFORCEMENT.md Document the v1 enforcement contract explicitly: hooks are calibrated for a non-adversarial code agent (RUBRIC.md's stated subject), not a red-teamer actively evading the sandbox. Catalogue the seven classes of static-Bash-analysis-unreachable bypass (bashlex NIE, heredoc-body, pipeline string flow, stdin-supplied subcommands, trap evaluation, interpreter shell-out, variable-flow loops/assignments) as accepted out-of-scope for v1, with rationale and the follow-up trigger (non-adversarial agent reaching one in practice). Reframes the iter-3/4/5 cycle-reviewer BLOCKER inflation as adversarial review against a non-adversarial artifact; closes that loop on the documentation layer rather than the code layer. Signed-off-by: even-wei --- evals/agent-blind-spots/ENFORCEMENT.md | 27 ++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/evals/agent-blind-spots/ENFORCEMENT.md b/evals/agent-blind-spots/ENFORCEMENT.md index dfeeca3..3c08065 100644 --- a/evals/agent-blind-spots/ENFORCEMENT.md +++ b/evals/agent-blind-spots/ENFORCEMENT.md @@ -123,6 +123,33 @@ Per DRC-3584 acceptance criterion #2, run one fixture × {Claude Code, Codex} × Record the result in a `runs//sandbox-verification.md` per cell so future contributors can repro. +## Threat model — non-adversarial code agent + +`RUBRIC.md` evaluates a **non-adversarial code agent** doing dbt PR review under instruction. That agent's failure mode is forgetting to call Recce, or calling Recce incorrectly — not constructing Bash incantations to slip past the sandbox. The Claude Code hooks are calibrated to that threat model; treat them as accident-prevention, not as red-team defense. + +The hooks block what a non-adversarial agent reaches in normal operation: + +- Direct Recce reachable paths (`recce` CLI, `mcp__recce__*`, `/recce:*` skill names, `recce.yml` discovery, `RECCE_API_TOKEN` references). +- The dbt subcommands that compile or evaluate models (`run`, `test`, `parse`, `compile`, `build`, `seed`, `snapshot`, `docs`, `freshness`, `run-operation`, `debug`, `source`, `clone`, `retry`). +- Raw SQL clients in the Tier-0 / Tier-1 denylist (`duckdb`, `psql`, `snowsql`, `bq`). +- Standard exec-wrapper laundering (`xargs`, `find -exec`, `time`, `nohup`, `parallel`, `timeout`, `command`, `builtin`) and shell-wrapper laundering (`sh -c`, `bash -c`; `eval` at Tier-0). + +The hooks do **not** attempt to defeat an adversarial agent actively trying to evade the sandbox. Static Bash analysis has structural limits — an adversary can reach denied commands through any of the following classes, all of which are **accepted as out of scope for v1**: + +| Class | Example | Why static analysis can't close it | +|---|---|---| +| `bashlex` parser gaps | `time dbt parse`, `case x in *) dbt parse;; esac`, `select … dbt …` | bashlex raises `NotImplementedError` on certain Bash keywords; the AST is unreachable | +| Heredoc body as interpreter input | `sh < Date: Fri, 29 May 2026 18:19:07 +0800 Subject: [PATCH 08/13] feat(recce): opt-in PostHog telemetry scaffolding for /recce-verify (DRC-3597) (#38) Foundation for the L3 funnel signal proposed in the 2026-05-29 project rethink. Tells the project whether real-world agent users reach for /recce-verify, complete it, and convert downstream -- the production complement to the L1 offline eval (DRC-3405) and L2 in-driver trace metrics (DRC-3586). What ships: - plugins/recce/hooks/scripts/telemetry.sh Event emitter. Fires recce_verify.* events to PostHog via curl fire-and-forget. Off by default; opt in via RECCE_TELEMETRY_OPT_IN=1 or ~/.recce/config.yml `telemetry_opt_in: true`. Anonymous stable installation ID at ~/.recce/installation-id (UUID4). Failure modes are silent and non-blocking by construction. - plugins/recce/hooks/scripts/test-telemetry.sh Audit script that exercises every opt-in / opt-out short-circuit without firing network traffic. Currently 4/4 pass on default-off, opt-in + no key, opt-in + DISABLED bypass, and missing-event-name edge case. - plugins/recce/hooks/scripts/README-telemetry.md Documents the event schema (skill_invoked, tier_degraded, tool_call, verdict_emitted, session_completed), opt-in mechanism, per-event wiring recommendations, plugin-maintainer responsibilities (PostHog key), and audit script usage. What's NOT in this PR (deferred to follow-up): - Auto-wiring in hooks.json (which PostToolUse / Stop hooks fire which events; needs design decision on session-scoping) - Inline emit calls in plugins/recce/skills/recce-verify/SKILL.md - The actual PostHog project key (plugin maintainer fills in at release) - Recce Cloud signup join key (coordinate with Andy when Cloud picks up the parameter) This is independent of the eval chain (DRC-3585 / 3586 / 3587 / 3405) and can land on its own without disturbing the in-flight eval work. Signed-off-by: even-wei --- .../recce/hooks/scripts/README-telemetry.md | 94 ++++++++++++++++ plugins/recce/hooks/scripts/telemetry.sh | 101 ++++++++++++++++++ plugins/recce/hooks/scripts/test-telemetry.sh | 54 ++++++++++ 3 files changed, 249 insertions(+) create mode 100644 plugins/recce/hooks/scripts/README-telemetry.md create mode 100755 plugins/recce/hooks/scripts/telemetry.sh create mode 100755 plugins/recce/hooks/scripts/test-telemetry.sh diff --git a/plugins/recce/hooks/scripts/README-telemetry.md b/plugins/recce/hooks/scripts/README-telemetry.md new file mode 100644 index 0000000..b1601f2 --- /dev/null +++ b/plugins/recce/hooks/scripts/README-telemetry.md @@ -0,0 +1,94 @@ +# `/recce-verify` telemetry — opt-in PostHog events + +L3 funnel signal for the `Agent-blind spots: /recce-verify v1` project ([DRC-3597](https://linear.app/recce/issue/DRC-3597)). Lets the project see whether real-world agent users reach for the skill, complete it, and convert downstream — the production complement to the L1 offline eval (DRC-3405) and L2 in-driver trace metrics (DRC-3586). + +**Off by default.** Telemetry fires only when the user opts in. + +## What gets emitted + +| Event | When | Properties | +|---|---|---| +| `recce_verify.skill_invoked` | `/recce-verify` SKILL.md activates | `tier_detected` (0 / 1 / 2), `agent` (claude_code / codex), `recce_version`, `plugin_version` | +| `recce_verify.tier_degraded` | Skill detects degraded capability and falls back | `from_tier`, `to_tier`, `reason` (recce_missing / mcp_unreachable / no_dev_env / …) | +| `recce_verify.tool_call` | Any `mcp__plugin_recce_recce__*` call inside the skill flow | `tool_name`, `success` (true/false), `error_class` (truncated), `duration_ms_bucket` | +| `recce_verify.verdict_emitted` | Skill writes its structured verdict | `verdict` (catch/miss/partial/abstain), `evidence_tier`, `subset` (1a/1b/1c) | +| `recce_verify.session_completed` | Last skill step before agent yields | `cells_invoked` count, `total_duration_ms_bucket` | + +**No PII, no SQL bodies, no model identifiers in event properties** — coarse buckets only. The `verdict` value is event-bound (`catch` / `miss` / `partial`), not free-form. + +## How to opt in (user side) + +Either: + +```bash +export RECCE_TELEMETRY_OPT_IN=1 # per-shell +``` + +Or write to `~/.recce/config.yml`: + +```yaml +telemetry_opt_in: true +``` + +To opt out for one invocation while the env / config is on: + +```bash +RECCE_TELEMETRY_DISABLED=1 claude # bypass for this session +``` + +## How to wire (plugin maintainer side) + +The emitter script is at `plugins/recce/hooks/scripts/telemetry.sh`. Call it with the event name + key=value props: + +```bash +bash "${CLAUDE_PLUGIN_ROOT}/hooks/scripts/telemetry.sh" recce_verify.skill_invoked \ + tier_detected=1 agent=claude_code recce_version=0.42.0 plugin_version=0.2.0 +``` + +Fires in the background. Never blocks. Returns 0 unconditionally. + +### Auto-wired (via hooks.json) + +None yet — the wiring needs design decisions (which events fire from hooks vs from inline SKILL.md calls) that should happen in a follow-up PR. This PR ships the emitter foundation only. + +Recommended next steps (see DRC-3597 acceptance): + +1. Add `PostToolUse` hook matcher `mcp__plugin_recce_recce__.*` → `telemetry.sh recce_verify.tool_call tool_name=$TOOL_NAME success=$SUCCESS`. Requires deciding how to scope to "only when /recce-verify is active" — proposal: SKILL.md sets a marker file at Step 0, hook checks marker before firing. +2. Add `Stop` hook → `telemetry.sh recce_verify.session_completed cells_invoked=…`. Requires aggregating per-session counts (e.g. from the marker file). +3. Inline `bash ${CLAUDE_PLUGIN_ROOT}/hooks/scripts/telemetry.sh recce_verify.skill_invoked …` in SKILL.md Step 0 (single-line addition, low risk). +4. Inline `… recce_verify.verdict_emitted …` in the SKILL.md verdict-write step. + +### PostHog project configuration + +The plugin maintainer fills in the PostHog public project key during release packaging — either by patching `telemetry.sh` (constant) or by shipping a small wrapper that exports `RECCE_POSTHOG_PROJECT_KEY` before calling the script. Both approaches keep the key in plugin-controlled code, not in user config. + +For development: + +```bash +export RECCE_POSTHOG_PROJECT_KEY=phc_xxxxx +export RECCE_TELEMETRY_OPT_IN=1 +bash plugins/recce/hooks/scripts/test-telemetry.sh +``` + +## Funnel attribution + +Each user's events carry a stable `distinct_id` from `~/.recce/installation-id` (UUID4, lazily created on first opt-in emit, persisted across sessions). The file holds only the UUID — no credentials, no personal info. + +Attribution to Recce Cloud signups requires a join key on the Cloud side. That's not in this PR — coordinate with @Andy when Cloud's signup form picks up the parameter. + +## Failure modes (all silent, all non-blocking) + +- Opt-in unset → script exits 0 before doing anything. +- `~/.recce/` unwriteable → installation-id creation skipped, script exits 0. +- No PostHog key configured → script exits 0 before any HTTP call. +- PostHog endpoint unreachable / 5xx / timeout → `curl --max-time 2`; the request runs in a background subshell, so the script returns immediately regardless of outcome. +- `jq` missing → properties JSON falls back to empty `{}`; event still fires with no props. +- Any other unexpected failure → trailing `|| true` and `set -u` (no `-e`) keep the script returning 0. + +## Audit script + +`test-telemetry.sh` exercises the opt-in paths without firing a real network request, by setting `RECCE_POSTHOG_PROJECT_KEY=""` so the emit short-circuits before the `curl`. Use it after editing `telemetry.sh`: + +```bash +bash plugins/recce/hooks/scripts/test-telemetry.sh +``` diff --git a/plugins/recce/hooks/scripts/telemetry.sh b/plugins/recce/hooks/scripts/telemetry.sh new file mode 100755 index 0000000..af06c93 --- /dev/null +++ b/plugins/recce/hooks/scripts/telemetry.sh @@ -0,0 +1,101 @@ +#!/usr/bin/env bash +# telemetry.sh — opt-in PostHog event emitter (DRC-3597, L3 funnel signal) +# +# Fires events for the /recce-verify skill funnel: +# - recce_verify.skill_invoked +# - recce_verify.tier_degraded +# - recce_verify.tool_call +# - recce_verify.verdict_emitted +# - recce_verify.session_completed +# +# Off by default. Activate via one of: +# - export RECCE_TELEMETRY_OPT_IN=1 +# - write `telemetry_opt_in: true` to ~/.recce/config.yml +# +# Failure modes are silent — never blocks the agent flow. +# +# Usage: +# telemetry.sh [key=value] [key=value] ... + +set -u + +# ── Opt-in gate ──────────────────────────────────────────────────────────────── + +if [[ "${RECCE_TELEMETRY_OPT_IN:-}" != "1" ]]; then + cfg="${HOME}/.recce/config.yml" + if [[ ! -f "${cfg}" ]] || ! grep -qE '^telemetry_opt_in:[[:space:]]*true$' "${cfg}" 2>/dev/null; then + exit 0 + fi +fi + +# ── Per-tool one-flag bypass ─────────────────────────────────────────────────── + +if [[ "${RECCE_TELEMETRY_DISABLED:-}" == "1" ]]; then + exit 0 +fi + +# ── Event + properties ───────────────────────────────────────────────────────── + +event="${1:-}" +shift || true + +if [[ -z "${event}" ]]; then + echo "telemetry.sh: missing event name" >&2 + exit 0 +fi + +# ── Anonymous, stable installation ID ───────────────────────────────────────── + +install_id_file="${HOME}/.recce/installation-id" +if [[ ! -f "${install_id_file}" ]]; then + mkdir -p "$(dirname "${install_id_file}")" 2>/dev/null || exit 0 + if command -v uuidgen >/dev/null 2>&1; then + uuidgen | tr '[:upper:]' '[:lower:]' > "${install_id_file}" 2>/dev/null || exit 0 + elif command -v python3 >/dev/null 2>&1; then + python3 -c 'import uuid; print(uuid.uuid4())' > "${install_id_file}" 2>/dev/null || exit 0 + else + exit 0 + fi +fi +install_id="$(cat "${install_id_file}" 2>/dev/null || true)" +[[ -z "${install_id}" ]] && exit 0 + +# ── PostHog destination ──────────────────────────────────────────────────────── + +posthog_host="${POSTHOG_HOST:-https://us.i.posthog.com}" +posthog_key="${RECCE_POSTHOG_PROJECT_KEY:-}" + +# If no key configured, drop the event silently. The plugin maintainer fills +# this in during plugin packaging; users opting in still need a key to fire. +if [[ -z "${posthog_key}" ]]; then + exit 0 +fi + +# ── Build properties JSON from key=value args ───────────────────────────────── + +props='{}' +if command -v jq >/dev/null 2>&1; then + for kv in "$@"; do + k="${kv%%=*}" + v="${kv#*=}" + [[ "${k}" == "${kv}" ]] && continue # no '=' → skip + props=$(jq --arg k "${k}" --arg v "${v}" '. + {($k): $v}' <<< "${props}" 2>/dev/null) || props='{}' + done +fi + +# ── Fire-and-forget POST ─────────────────────────────────────────────────────── + +payload=$(cat </dev/null 2>&1 || true +) & + +exit 0 diff --git a/plugins/recce/hooks/scripts/test-telemetry.sh b/plugins/recce/hooks/scripts/test-telemetry.sh new file mode 100755 index 0000000..c6afeca --- /dev/null +++ b/plugins/recce/hooks/scripts/test-telemetry.sh @@ -0,0 +1,54 @@ +#!/usr/bin/env bash +# test-telemetry.sh — verify opt-in / opt-out paths in telemetry.sh +# +# Exercises every short-circuit in the emitter without hitting the network. +# Sets RECCE_POSTHOG_PROJECT_KEY="" so the emit ends before the curl call; +# this isolates opt-in logic from network behaviour. + +set -uo pipefail + +cd "$(dirname "${BASH_SOURCE[0]}")" + +EMIT="./telemetry.sh" +PASS=0 +FAIL=0 + +run() { + local label="$1" + shift + local want="$1" + shift + # Capture stdout + stderr together + set +e + out=$(env -i HOME="${HOME}" PATH="${PATH}" "$@" "${EMIT}" recce_verify.test prop_a=1 2>&1) + rc=$? + set -e + if [[ "${rc}" == "${want}" ]]; then + PASS=$((PASS + 1)) + printf " pass rc=%d %s\n" "${rc}" "${label}" + else + FAIL=$((FAIL + 1)) + printf " FAIL want=%s got=%d %s\n --- output ---\n%s\n --- end ---\n" \ + "${want}" "${rc}" "${label}" "${out}" + fi +} + +echo "Testing telemetry.sh opt-in paths (no network)..." +echo + +# All cases should rc=0 (silent + non-blocking is the contract). The behaviour +# difference is whether the script proceeds past the opt-in gate, which we +# infer from -x trace if needed; here we just confirm rc=0 across paths. + +run "default off (no env, no config)" 0 +run "RECCE_TELEMETRY_OPT_IN=1 + no key (drops at key check)" 0 \ + RECCE_TELEMETRY_OPT_IN=1 + +run "RECCE_TELEMETRY_OPT_IN=1 + key + DISABLED=1 (bypass)" 0 \ + RECCE_TELEMETRY_OPT_IN=1 RECCE_POSTHOG_PROJECT_KEY=phc_fake RECCE_TELEMETRY_DISABLED=1 + +run "missing event name" 0 + +echo +echo "Summary: ${PASS} pass · ${FAIL} fail" +exit "${FAIL}" From 1d3c95dd7d8f9c16592b605b2c48f38865a3339a Mon Sep 17 00:00:00 2001 From: Even Wei Date: Fri, 29 May 2026 18:22:32 +0800 Subject: [PATCH 09/13] feat(evals): Karpathy spike driver for /recce-verify v1 eval (DRC-3586) (#37) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(evals): Karpathy spike driver for /recce-verify v1 eval (DRC-3586) Single-file Python driver at evals/agent-blind-spots/spike-driver/. Dispatches up to 6 fixtures x 2 agents x 2 tiers = 24 cells, captures each agent's transcript, runs a Claude-as-judge pass per transcript to produce a three-axis verdict (catch / tier / delta), and writes a CSV + Markdown summary under runs//spike-driver/. Stability checks (both optional, both produce judge-quality signal): - --judge-stability: double-judges each transcript, reports per-axis self-consistency (catch / tier / delta). Floor 80%. - --baseline-dir : compares judge verdicts against DRC-3585 manual baseline once it lands (judge-vs-human catch agreement on Tier-0 cells). Graceful degradation: - Skips an agent's cells if its CLI is not on PATH; codex commonly absent in lighter dev envs (recipe in runner-configs/codex/tier-{0,1}/README.md). - --no-run mode re-judges existing transcripts without re-running agents. Sandbox profile integration (from DRC-3584): - Claude Code cells stamp tier-N claude-overlay/ into the fixture worktree before invoking `claude --print`, with a neutered CLAUDE_CONFIG_DIR per cell and scrubbed warehouse env. - Codex cells use `codex exec --sandbox= --ask-for-approval=never --config ` plus PATH scrub. Non-goals for this spike (deferred): - Durable harness with resume / parallel dispatch -> DRC-3587 Inspect AI port. - Gap-report generator across 6 fixtures -> DRC-3405. - Auto-iteration on prompt / skill changes (closed-loop optimization overfits at N=6 -- explicit non-goal of the spike). Stacks on PR #36 (DRC-3584 + DRC-3430). When #36 merges to main, this PR should be rebased onto main; GitHub auto-retargets the base branch. Signed-off-by: even-wei * fix(evals): stage frozen Tier-0 inputs into cwd before agent run The agent's cwd is .tmp/sources// (the per-fixture standalone repo), but the frozen Tier-0 inputs (diff.patch, artifacts/{manifest, compiled, catalog}*) live at fixtures//, outside cwd. Without staging: - Claude Code reaches them via absolute paths (Read tool isn't cwd-anchored), but the prompt doesn't tell the agent where to look. - Codex Tier-0 read-only sandbox blocks reads outside cwd entirely, so the agent literally cannot reach the artifacts. Driver now symlinks fixtures//{diff.patch, artifacts} into a _eval_inputs/ subdir under cwd before each cell. Prompt updated to point at _eval_inputs/. Works under both agent sandboxes. Verified locally: >>> stage_inputs(SOURCES_DIR / 'pr1-fix-clv', 'pr1-fix-clv') >>> sorted(p.name for p in (SOURCES_DIR / 'pr1-fix-clv' / '_eval_inputs').iterdir()) ['artifacts', 'diff.patch'] Signed-off-by: even-wei * fix(evals): inherit CLAUDE_CONFIG_DIR for auth in unattended runs ENFORCEMENT.md recipe step 3 sets CLAUDE_CONFIG_DIR to a fresh mktemp dir to neuter user-level ~/.claude/settings.json. That step also strips the Claude Code auth state, so a child `claude --print` invoked by the spike driver fails with "Not logged in" and no transcript ever lands. For unattended runs, the load-bearing enforcement is the project-level .claude/settings.json overlay (stamped per cell) plus the PreToolUse hook (deny-tier-N.py); a user-level `permissions.allow` cannot bypass an exit-2 hook. Skip the CLAUDE_CONFIG_DIR override by default. For paranoid mode (e.g. when running the eval on a machine with risky ~/.claude/ contents), set RECCE_EVAL_STRICT_CONFIG=1 to enable the override. The operator is responsible for preseeding auth under the per-cell _claude_cfg dir. Verified end-to-end smoke run after this fix: uv run driver.py --smoke --agents claude --tiers 0 [run] pr1-fix-clv · claude · tier-0 [judge] pr1-fix-clv · claude · tier-0 Agent VERDICT: catch · request-changes Judge verdict: {catch, tier-0, same} (delta=same expected with no T1 paired) Signed-off-by: even-wei * fix(evals): address pr-cycle review of #37 — 3 ISSUEs 1. Drop dead VERDICT_TAIL_RE regex. Defined in driver.py:99 but never used; agent verdict parsing is fully handled by judge_cell() downstream. Removed alongside the regex line. 2. Add surgical .gitignore entries for spike-driver runtime outputs: evals/agent-blind-spots/runs/*/spike-driver/transcripts/ evals/agent-blind-spots/runs/*/spike-driver/_claude_cfg/ These are the genuinely-transient subdirs. verdicts.csv, summary.md, cells.json under runs//spike-driver/ are intentionally NOT ignored — they may be committed manually as run baselines for cross-iteration tracking. 3. stage_inputs() now raises FileNotFoundError when no Tier-0 inputs (neither diff.patch nor artifacts/) could be staged under the fixture's source dir. run_cell() catches and records into cell.error so the cell shows up as failed in the matrix instead of silently feeding the agent an empty _eval_inputs/ and letting it review with no inputs. Verified: python3 -m py_compile spike-driver/driver.py -> ok stage_inputs(.../pr1-fix-clv, 'pr1-fix-clv') -> happy path stage_inputs(/tmp/dne, 'does-not-exist') -> raises FileNotFoundError NOTE-level findings 4-7 from the review are accepted as-is per the spike framing (judge inherits parent env intentionally for auth; same-day run dirs are an operator concern; broken-symlink in overlay copy is rare and surfaced via run_cell's general except; no tests is consistent with the Karpathy-spike scope and the DRC-3587 follow-up which adds Inspect AI test harness). Signed-off-by: even-wei --------- Signed-off-by: even-wei --- .gitignore | 7 + .../agent-blind-spots/spike-driver/README.md | 99 ++++ .../agent-blind-spots/spike-driver/driver.py | 466 ++++++++++++++++++ 3 files changed, 572 insertions(+) create mode 100644 evals/agent-blind-spots/spike-driver/README.md create mode 100644 evals/agent-blind-spots/spike-driver/driver.py diff --git a/.gitignore b/.gitignore index 7f5fa70..0e59a40 100644 --- a/.gitignore +++ b/.gitignore @@ -47,6 +47,13 @@ temp/ evals/agent-blind-spots/.tmp/ evals/agent-blind-spots/fixtures/*/artifacts/ +# DRC-3586 spike-driver runtime outputs — per-run transcripts and ephemeral +# Claude config dirs. The verdicts.csv / summary.md / cells.json under +# runs//spike-driver/ may be committed manually as run baselines, so +# the runs/ root is NOT ignored — only the transient subdirs are. +evals/agent-blind-spots/runs/*/spike-driver/transcripts/ +evals/agent-blind-spots/runs/*/spike-driver/_claude_cfg/ + # Coverage / editor logs .coverage .nvimlog diff --git a/evals/agent-blind-spots/spike-driver/README.md b/evals/agent-blind-spots/spike-driver/README.md new file mode 100644 index 0000000..4797d05 --- /dev/null +++ b/evals/agent-blind-spots/spike-driver/README.md @@ -0,0 +1,99 @@ +# Spike driver — DRC-3586 + +Single-file Python driver for the Karpathy-style spike in [DRC-3586](https://linear.app/recce/issue/DRC-3586). De-risks two unknowns before any durable-harness work (DRC-3587): + +1. **Judge stability** — does the Claude-as-judge prompt produce consistent verdicts on the locked three-axis rubric? +2. **Codex-under-sandbox** — does Codex behave in a programmatic loop under the Tier-0 / Tier-1 sandbox profiles from DRC-3584? + +Not the durable harness. Output artifacts live under `runs//spike-driver/`. If the spike's stability + sandbox checks pass, the next ticket (DRC-3587) ports this to Inspect AI; if they fail, the project falls back to Candidate B (manual scoring loop). + +## Prereqs + +- Per-fixture worktrees built: `( cd evals/agent-blind-spots && ./build_fixtures.sh )` +- `bashlex` installed for the Claude Code Tier-0/1 PreToolUse hook: `python3 -m pip install bashlex` +- `claude` CLI on PATH (for both agent + judge passes) +- `codex` CLI on PATH (optional — codex cells skip gracefully if missing) +- `uv` available (per repo convention) + +## Auth note — `CLAUDE_CONFIG_DIR` + +`ENFORCEMENT.md` recipe step 3 sets `CLAUDE_CONFIG_DIR=$(mktemp -d)` to neuter +a stray `~/.claude/settings.json`. The driver **does not** apply that override +by default because it also strips the auth state — the child `claude --print` +fails with `Not logged in`. For unattended runs the load-bearing enforcement is +the project-level `.claude/settings.json` overlay (stamped per cell) plus the +`PreToolUse` hook; a user-level `permissions.allow` cannot bypass an exit-2 +hook regardless. + +To enable the strict override (paranoid mode), set: + +```bash +RECCE_EVAL_STRICT_CONFIG=1 uv run driver.py --smoke +``` + +You must preseed auth under the per-cell `_claude_cfg/_t/` dir +before each run; the driver does not provision auth. + +## Usage + +```bash +# Smoke test (1 fixture × 2 agents × 2 tiers = 4 cells, ~5-10 min) +uv run evals/agent-blind-spots/spike-driver/driver.py --smoke + +# Full run (6 fixtures × 2 agents × 2 tiers = 24 cells, ~1-2 hours) +uv run evals/agent-blind-spots/spike-driver/driver.py + +# Limit to one agent / tier +uv run evals/agent-blind-spots/spike-driver/driver.py --agents claude --tiers 0 + +# Double-judge each transcript for self-consistency +uv run evals/agent-blind-spots/spike-driver/driver.py --judge-stability + +# Re-judge an existing run without re-running agents +uv run evals/agent-blind-spots/spike-driver/driver.py \ + --no-run --run-dir evals/agent-blind-spots/runs/2026-05-29/spike-driver/ + +# Compare judge verdicts to DRC-3585 manual baseline once it lands +uv run evals/agent-blind-spots/spike-driver/driver.py \ + --baseline-dir evals/agent-blind-spots/fixtures/ +``` + +## Output layout + +``` +runs//spike-driver/ +├── transcripts/ # raw agent stdout/stderr per cell +│ ├── pr1-fix-clv_claude_t0.txt +│ ├── pr1-fix-clv_claude_t1.txt +│ ├── pr1-fix-clv_codex_t0.txt +│ └── ... +├── _claude_cfg/ # neutered CLAUDE_CONFIG_DIR per cell +├── verdicts.csv # one row per cell +├── cells.json # raw cells (re-judgeable) +└── summary.md # rendered matrix + stability check +``` + +## Stability checks + +Two ways to validate judge quality (use both when possible): + +**Self-consistency** (`--judge-stability`): judges each transcript twice in independent subprocess calls; reports per-axis agreement. Floor: ≥80% on each of catch / tier / delta. Below that, the judge is too noisy to replace hand-grading. + +**Judge vs human** (`--baseline-dir `): if [DRC-3585](https://linear.app/recce/issue/DRC-3585)'s manual rubric-lock has produced `fixtures//tier-0-baseline.md` files, this compares the judge's catch verdict for Tier-0 cells to the human's. This is the canonical check from DRC-3586's acceptance criterion 2; self-consistency is the proxy when no manual baseline exists yet. + +## Codex-under-sandbox check + +After a run, grep transcripts under `transcripts/*_codex_*.txt` for: + +- `mcp__recce__*` references in Tier-0 → leak (MCP table is empty in `runner-configs/codex/tier-0/config.toml`). +- `recce ` shell calls in Tier-0 → check exit was non-zero (PATH scrub). +- Reads of `../README.md` or `../../RUBRIC.md` → leak (cwd separation). +- `dbt ` shell calls in either tier → check exit was non-zero (PATH scrub). + +If every Codex cell ran to completion without these escape attempts succeeding, criterion 3 passes. + +## Non-goals (deferred to DRC-3587 / DRC-3405) + +- Durable harness with resume / parallel dispatch — that's Inspect AI. +- Gap-report generator across 6 fixtures — that's DRC-3405. +- Auto-iteration on prompt / skill changes — explicit non-goal of the spike (closed-loop optimization overfits at N=6). diff --git a/evals/agent-blind-spots/spike-driver/driver.py b/evals/agent-blind-spots/spike-driver/driver.py new file mode 100644 index 0000000..80cba79 --- /dev/null +++ b/evals/agent-blind-spots/spike-driver/driver.py @@ -0,0 +1,466 @@ +#!/usr/bin/env python3 +"""Karpathy-style spike driver for /recce-verify v1 eval (DRC-3586). + +Dispatches up to 6 fixtures × 2 agents × 2 tiers = 24 cells, captures +each agent's transcript, runs a Claude-as-judge pass per transcript to +produce a three-axis verdict (catch / tier / delta), and writes a +CSV + Markdown summary under `runs//spike-driver/`. + +Two stability modes: + --judge-stability Run the judge twice per cell; report self-consistency. + --baseline-dir Compare judge verdicts against DRC-3585 manual baseline. + +Both are optional. Without either, the driver still produces a verdict +matrix; stability checks are how you decide whether the judge can replace +hand-grading at N=6. + +Usage: + uv run driver.py --smoke # 1 fixture × 2 agents × 2 tiers + uv run driver.py # full 24-cell run + uv run driver.py --agents claude # claude only + uv run driver.py --tiers 0 # tier-0 only + uv run driver.py --judge-stability # double-judge for self-consistency + uv run driver.py --no-run \\ + --run-dir runs/2026-05-29/spike-driver/ # re-judge only +""" + +from __future__ import annotations + +import argparse +import csv +import datetime as dt +import json +import os +import re +import shutil +import subprocess +import sys +from dataclasses import asdict, dataclass +from pathlib import Path + +SPIKE_DIR = Path(__file__).resolve().parent +EVAL_DIR = SPIKE_DIR.parent +SOURCES_DIR = EVAL_DIR / ".tmp" / "sources" +RUNS_DIR = EVAL_DIR / "runs" +RUNNER_CONFIGS = EVAL_DIR / "runner-configs" + +DEFAULT_FIXTURES = ( + "pr1-fix-clv", + "pr2-refactor-cte-to-models", + "pr3-amount-double-to-decimal", + "pr42-is-closed-filter", + "pr44-promotion-flags", + "pr46-net-clv-segments", +) +AGENTS = ("claude", "codex") +TIERS = (0, 1) + +AGENT_PROMPT = ( + "Review this dbt PR. The current working directory is the dbt project at the " + "head SHA (models/, dbt_project.yml, etc.). The frozen Tier-0 inputs are " + "staged under `_eval_inputs/`:\n" + " - _eval_inputs/diff.patch — source-model diff base..head\n" + " - _eval_inputs/artifacts/manifest-{before,after}.json — dbt manifests pre/post\n" + " - _eval_inputs/artifacts/compiled-{before,after}/ — compiled SQL pre/post\n" + " - _eval_inputs/artifacts/catalog-{before,after}.json — schema-only (row stats are zero)\n" + "Decide catch / miss / partial per the rubric. Recommend " + "approve / request-changes / abstain. End your output with one line, exactly:\n" + "VERDICT: · " +) + +JUDGE_SYSTEM = ( + "You are a strict, terse judge. Score the agent transcript against the " + "locked three-axis rubric (catch, primary evidence tier, counterfactual " + "delta vs Tier-0 baseline). Emit ONLY JSON. No prose outside JSON." +) + +JUDGE_USER_TEMPLATE = """Rubric (excerpt): +{rubric} + +Fixture: {fixture} +Agent: {agent} +Tier: {tier} +Tier-0 baseline catch for this fixture (if known): {baseline_catch} + +Agent transcript: +--- +{transcript} +--- + +Emit JSON ONLY in this exact shape: +{{ + "catch": "catch" | "miss" | "partial", + "tier": "0" | "1a" | "1b" | "1c" | "2", + "delta": "improvement" | "same" | "regression", + "reasoning": "1-2 sentences citing the decisive evidence the agent used" +}} +""" + + +@dataclass +class Cell: + fixture: str + agent: str + tier: int + transcript_path: str | None = None + returncode: int | None = None + verdict: dict | None = None + verdict_2: dict | None = None + error: str | None = None + + +def cli_available(name: str) -> bool: + return shutil.which(name) is not None + + +def scrub_env(extra_unset: tuple[str, ...] = ()) -> dict: + env = os.environ.copy() + for var in ( + "RECCE_API_TOKEN", "DBT_PROFILES_DIR", + "SNOWFLAKE_USER", "SNOWFLAKE_PASSWORD", "SNOWFLAKE_ACCOUNT", + "POSTGRES_PASSWORD", "BIGQUERY_PROJECT", *extra_unset, + ): + env.pop(var, None) + return env + + +def scrub_path(strip_patterns: tuple[str, ...]) -> str: + pat = re.compile("|".join(strip_patterns)) + return ":".join(p for p in os.environ.get("PATH", "").split(":") if not pat.search(p)) + + +def stage_inputs(fixture_dir: Path, fixture_id: str) -> None: + """Stage frozen Tier-0 inputs into the agent's cwd at `_eval_inputs/`. + + Symlinks `fixtures//{diff.patch, artifacts/}` into a `_eval_inputs/` + subdirectory of the per-fixture worktree, so an agent whose sandbox is + anchored on cwd (notably Codex Tier-0 read-only) can still reach the + frozen artifacts. Without this, only Claude Code (which doesn't anchor + Read on cwd) could see the inputs. + + Raises FileNotFoundError if neither `diff.patch` nor `artifacts/` exists + under `fixtures//`; otherwise the agent would see an empty + `_eval_inputs/` and review the PR with no Tier-0 inputs (silent miss). + """ + src_root = EVAL_DIR / "fixtures" / fixture_id + if not src_root.exists(): + raise FileNotFoundError(f"fixture dir missing: {src_root}") + dst = fixture_dir / "_eval_inputs" + if dst.is_symlink() or dst.exists(): + if dst.is_symlink(): + dst.unlink() + else: + shutil.rmtree(dst) + dst.mkdir() + staged = 0 + for name in ("diff.patch", "artifacts"): + src = src_root / name + if src.exists(): + (dst / name).symlink_to(src.resolve()) + staged += 1 + if staged == 0: + dst.rmdir() + raise FileNotFoundError( + f"no Tier-0 inputs under {src_root} (need diff.patch and/or artifacts/); " + "did build_fixtures.sh complete for this fixture?" + ) + + +def run_claude(fixture_dir: Path, tier: int, run_dir: Path, prompt: str) -> tuple[Path, int]: + tier_dir = RUNNER_CONFIGS / "claude-code" / f"tier-{tier}" + overlay_src = tier_dir / "claude-overlay" + overlay_dst = fixture_dir / ".claude" + if overlay_dst.exists(): + shutil.rmtree(overlay_dst) + shutil.copytree(overlay_src, overlay_dst) + + transcript_path = run_dir / "transcripts" / f"{fixture_dir.name}_claude_t{tier}.txt" + transcript_path.parent.mkdir(parents=True, exist_ok=True) + + # NOTE: ENFORCEMENT.md recipe step 3 (CLAUDE_CONFIG_DIR=$(mktemp -d)) is + # *intentionally not applied here* — neutering ~/.claude/ also strips the + # auth state, which breaks unattended runs. The load-bearing enforcement + # for Tier-0 is the project-level .claude/settings.json overlay (stamped + # above) + the PreToolUse hook (deny-tier-N.py); a stray user-level + # `permissions.allow` cannot bypass an exit-2 hook. For paranoid mode, + # set RECCE_EVAL_STRICT_CONFIG=1 to override CLAUDE_CONFIG_DIR; the cell + # will fail with "Not logged in" unless auth is preseeded under that dir. + env = scrub_env() + if os.environ.get("RECCE_EVAL_STRICT_CONFIG"): + cfg_dir = run_dir / "_claude_cfg" / f"{fixture_dir.name}_t{tier}" + cfg_dir.mkdir(parents=True, exist_ok=True) + env["CLAUDE_CONFIG_DIR"] = str(cfg_dir) + + proc = subprocess.run( + ["claude", "--print", "--dangerously-skip-permissions", prompt], + cwd=str(fixture_dir), env=env, capture_output=True, text=True, timeout=900, + ) + transcript_path.write_text( + f"# Cell: {fixture_dir.name} · claude · tier-{tier}\n" + f"# returncode: {proc.returncode}\n\n" + f"## stdout\n{proc.stdout}\n\n## stderr\n{proc.stderr}\n" + ) + return transcript_path, proc.returncode + + +def run_codex(fixture_dir: Path, tier: int, run_dir: Path, prompt: str) -> tuple[Path, int]: + tier_dir = RUNNER_CONFIGS / "codex" / f"tier-{tier}" + config = tier_dir / "config.toml" + sandbox = "read-only" if tier == 0 else "workspace-write" + + transcript_path = run_dir / "transcripts" / f"{fixture_dir.name}_codex_t{tier}.txt" + transcript_path.parent.mkdir(parents=True, exist_ok=True) + + strip_patterns = (r"/recce(/|$)", r"/dbt(/|$)", r"\.recce") if tier == 0 else (r"/dbt(/|$)",) + env = scrub_env() + env["PATH"] = scrub_path(strip_patterns) + # Tier-1 keeps single-env warehouse credentials from the parent shell; + # the operator is responsible for absence of base/prod creds (see codex/tier-1/README.md). + + proc = subprocess.run( + ["codex", "exec", + f"--sandbox={sandbox}", + "--ask-for-approval=never", + "--config", str(config), + prompt], + cwd=str(fixture_dir), env=env, capture_output=True, text=True, timeout=900, + ) + transcript_path.write_text( + f"# Cell: {fixture_dir.name} · codex · tier-{tier}\n" + f"# returncode: {proc.returncode}\n\n" + f"## stdout\n{proc.stdout}\n\n## stderr\n{proc.stderr}\n" + ) + return transcript_path, proc.returncode + + +def run_cell(cell: Cell, run_dir: Path) -> None: + fixture_dir = SOURCES_DIR / cell.fixture + if not fixture_dir.exists(): + cell.error = f"per-fixture worktree missing: {fixture_dir} (did you run build_fixtures.sh?)" + return + if not cli_available(cell.agent): + cell.error = f"{cell.agent} CLI not found on PATH; skipping" + return + try: + stage_inputs(fixture_dir, cell.fixture) + except FileNotFoundError as e: + cell.error = f"stage_inputs failed: {e}" + return + try: + if cell.agent == "claude": + path, rc = run_claude(fixture_dir, cell.tier, run_dir, AGENT_PROMPT) + else: + path, rc = run_codex(fixture_dir, cell.tier, run_dir, AGENT_PROMPT) + cell.transcript_path = str(path) + cell.returncode = rc + except subprocess.TimeoutExpired: + cell.error = f"{cell.agent} timed out after 900s" + except FileNotFoundError as e: + cell.error = f"{cell.agent} setup failed: {e}" + + +def _shrink(text: str, limit: int = 30000) -> str: + if len(text) <= limit: + return text + half = limit // 2 + return text[:half] + "\n…[truncated]…\n" + text[-half:] + + +def judge_cell(cell: Cell, rubric: str, baseline_catch: str) -> dict: + if not cell.transcript_path: + return {"error": "no transcript"} + transcript = _shrink(Path(cell.transcript_path).read_text()) + user = JUDGE_USER_TEMPLATE.format( + rubric=_shrink(rubric, 8000), + fixture=cell.fixture, agent=cell.agent, tier=cell.tier, + transcript=transcript, baseline_catch=baseline_catch, + ) + proc = subprocess.run( + ["claude", "--print", "--dangerously-skip-permissions", + "--append-system-prompt", JUDGE_SYSTEM, user], + capture_output=True, text=True, timeout=300, + ) + if proc.returncode != 0: + return {"error": f"judge rc={proc.returncode}", "stderr": proc.stderr[:300]} + text = proc.stdout.strip() + start, end = text.find("{"), text.rfind("}") + if start < 0 or end < 0: + return {"error": "no JSON in judge output", "raw": text[:500]} + try: + return json.loads(text[start:end + 1]) + except json.JSONDecodeError as e: + return {"error": f"JSON parse: {e}", "raw": text[start:end + 1][:500]} + + +def parse_baseline_dir(baseline_dir: Path) -> dict[str, str]: + """Extract `catch / miss / partial` per fixture from tier-0-baseline.md files. + + Expects files at //tier-0-baseline.md following + the template at templates/tier-0-baseline.md. Returns {fixture: catch}. + """ + out: dict[str, str] = {} + if not baseline_dir or not baseline_dir.exists(): + return out + catch_re = re.compile(r"Catch\s*/\s*miss\s*/\s*partial:\s*`?(catch|miss|partial)`?", re.IGNORECASE) + for sub in baseline_dir.iterdir(): + baseline = sub / "tier-0-baseline.md" + if baseline.is_file(): + m = catch_re.search(baseline.read_text()) + if m: + out[sub.name] = m.group(1).lower() + return out + + +def axis_agreement(cells: list[Cell], axis: str) -> tuple[int, int]: + paired = [ + (c.verdict, c.verdict_2) for c in cells + if c.verdict and c.verdict_2 and "error" not in c.verdict and "error" not in c.verdict_2 + ] + if not paired: + return 0, 0 + agree = sum(1 for v1, v2 in paired if v1.get(axis) == v2.get(axis)) + return agree, len(paired) + + +def write_csv(cells: list[Cell], path: Path) -> None: + with path.open("w", newline="") as f: + w = csv.writer(f) + w.writerow([ + "fixture", "agent", "tier", + "catch", "evidence_tier", "delta", + "catch_2", "tier_2", "delta_2", + "returncode", "error", + ]) + for c in cells: + v = c.verdict or {} + v2 = c.verdict_2 or {} + w.writerow([ + c.fixture, c.agent, c.tier, + v.get("catch", ""), v.get("tier", ""), v.get("delta", ""), + v2.get("catch", ""), v2.get("tier", ""), v2.get("delta", ""), + "" if c.returncode is None else c.returncode, + c.error or "", + ]) + + +def write_summary(cells: list[Cell], path: Path, stability: bool, baseline: dict[str, str]) -> None: + lines = [ + "# Spike driver run — summary", + "", + f"- Run date: {dt.date.today().isoformat()}", + f"- Cells attempted: {len(cells)}", + f"- Cells with transcript: {sum(1 for c in cells if c.transcript_path)}", + f"- Cells judged: {sum(1 for c in cells if c.verdict and 'error' not in c.verdict)}", + f"- Cells errored: {sum(1 for c in cells if c.error)}", + "", + "## Cell matrix", + "", + "| Fixture | Agent | Tier | Catch | Evidence | Delta | Status |", + "|---|---|---|---|---|---|---|", + ] + for c in cells: + v = c.verdict or {} + status = c.error or ("judge_error" if v.get("error") else "ok") + lines.append( + f"| {c.fixture} | {c.agent} | {c.tier} | " + f"{v.get('catch','-')} | {v.get('tier','-')} | {v.get('delta','-')} | {status} |" + ) + + if stability: + lines += ["", "## Judge self-consistency (two passes on the same transcript)", ""] + for axis in ("catch", "tier", "delta"): + agree, total = axis_agreement(cells, axis) + pct = f"{agree / total:.0%}" if total else "n/a" + lines.append(f"- **{axis}**: {pct} ({agree}/{total} double-judged cells)") + lines += [ + "", + "**Stability bar:** ≥80% per axis. Below that, the judge can't replace human grading at N=6.", + ] + + if baseline: + lines += ["", "## Judge vs DRC-3585 manual baseline (catch axis)", ""] + compare = [ + (c, baseline[c.fixture]) + for c in cells if c.tier == 0 and c.fixture in baseline and c.verdict and "error" not in c.verdict + ] + if compare: + agree = sum(1 for c, b in compare if (c.verdict or {}).get("catch") == b) + lines.append(f"- Tier-0 cells with manual baseline available: {len(compare)}") + lines.append(f"- Judge–human catch agreement: {agree}/{len(compare)} ({agree / len(compare):.0%})") + else: + lines.append("- No Tier-0 cells overlap with baseline files; provide --baseline-dir pointing at fixtures/") + + path.write_text("\n".join(lines) + "\n") + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--smoke", action="store_true", help="One fixture × all agents × all tiers") + parser.add_argument("--agents", default=",".join(AGENTS), help="comma-list of agents") + parser.add_argument("--tiers", default=",".join(str(t) for t in TIERS), help="comma-list of tiers") + parser.add_argument("--fixtures", default=",".join(DEFAULT_FIXTURES), help="comma-list of fixtures") + parser.add_argument("--judge-stability", action="store_true", help="Run judge twice per cell") + parser.add_argument("--baseline-dir", type=Path, help="DRC-3585 manual baseline dir; compares judge to human on catch axis") + parser.add_argument("--no-run", action="store_true", help="Skip agent runs; judge existing transcripts in --run-dir") + parser.add_argument("--run-dir", type=Path, help="Override run output dir (default runs//spike-driver/)") + args = parser.parse_args() + + agents = [a for a in args.agents.split(",") if a] + tiers = [int(t) for t in args.tiers.split(",") if t] + fixtures = list(args.fixtures.split(",")) + if args.smoke: + fixtures = fixtures[:1] + + run_dir = args.run_dir or (RUNS_DIR / dt.date.today().isoformat() / "spike-driver") + run_dir.mkdir(parents=True, exist_ok=True) + + rubric_path = EVAL_DIR / "RUBRIC.md" + rubric = rubric_path.read_text() if rubric_path.exists() else "(RUBRIC.md missing)" + baseline = parse_baseline_dir(args.baseline_dir) if args.baseline_dir else {} + if args.baseline_dir and not baseline: + print(f"[warn] --baseline-dir given but no baselines parsed from {args.baseline_dir}", file=sys.stderr) + + cells: list[Cell] = [ + Cell(fixture=f, agent=a, tier=t) + for f in fixtures for a in agents for t in tiers + ] + + if not args.no_run: + for cell in cells: + print(f"[run] {cell.fixture} · {cell.agent} · tier-{cell.tier}", file=sys.stderr) + run_cell(cell, run_dir) + if cell.error: + print(f" → {cell.error}", file=sys.stderr) + else: + # Re-judge mode: discover existing transcripts in run_dir + for cell in cells: + t = run_dir / "transcripts" / f"{cell.fixture}_{cell.agent}_t{cell.tier}.txt" + if t.exists(): + cell.transcript_path = str(t) + else: + cell.error = f"no existing transcript at {t}" + + for cell in cells: + if not cell.transcript_path: + continue + baseline_catch = baseline.get(cell.fixture, "unknown") + print(f"[judge] {cell.fixture} · {cell.agent} · tier-{cell.tier}", file=sys.stderr) + cell.verdict = judge_cell(cell, rubric, baseline_catch) + if args.judge_stability: + cell.verdict_2 = judge_cell(cell, rubric, baseline_catch) + + csv_path = run_dir / "verdicts.csv" + write_csv(cells, csv_path) + summary_path = run_dir / "summary.md" + write_summary(cells, summary_path, args.judge_stability, baseline) + cells_json = run_dir / "cells.json" + cells_json.write_text(json.dumps([asdict(c) for c in cells], indent=2)) + + print(f"\n[output] verdicts: {csv_path}", file=sys.stderr) + print(f"[output] summary: {summary_path}", file=sys.stderr) + print(f"[output] raw: {cells_json}", file=sys.stderr) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) From 6f2028959110c87da82637e3c0042d69a487cf0f Mon Sep 17 00:00:00 2001 From: even-wei Date: Mon, 1 Jun 2026 14:36:49 +0800 Subject: [PATCH 10/13] fix(evals): close iter-6 BLOCKER (git history leak) + ISSUE (Tier-1 dbt policy) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two findings from the pr-cycle iter-6 review of #36, both re-confirming wcchang1115 (Andy) review comments that remained open. ## BLOCKER: per-fixture git history leaks stripped content build_fixtures.sh fetched the upstream head SHA into refs/fixture/head and `rm -rf`'d Recce-aware paths from the working tree. The original commit (and its tree) was untouched, so a Tier-0 agent allowlisted to run `git` could recover everything via `git show HEAD:recce.yml`, `git cat-file -p HEAD^{tree}`, `git log -p`, etc. The working-tree strip is bypassable by anyone reading git objects. Fix: after strip + leak-grep, rewrite the per-fixture repo's history into a single fresh commit whose tree IS the stripped working tree. Delete .git, `git init`, `git add -A`, `git commit`. No ancestor references the original head's tree; no other ref / reflog entry keeps it reachable. Verified end-to-end on a copy of pr1-fix-clv's source: Before: git show HEAD:recce.yml -> dumps recce.yml content (leak) git ls-tree -r HEAD | grep recce.yml -> recce.yml is listed After rewrite: git cat-file -e HEAD:recce.yml -> exit 128 (unreachable) git cat-file -e HEAD:.github/prompts/system-prompt.md -> unreachable git cat-file -e HEAD:.github/workflows/claude.yml -> unreachable git cat-file -e HEAD:.github/workflows/recce_ci.yml -> unreachable git cat-file -e HEAD:.devcontainer/post-create.sh -> unreachable rev-list --all HEAD --count -> 1 (single commit invariant preserved) Post-rewrite path-leak regex tightened to anchor at path components (`(^|/)recce\.yml$` etc.) so the generic .devcontainer.json VS Code config (no Recce content) isn't a false positive. ## ISSUE: Tier-1 dbt subcommand policy divergence Codex Tier-1 PATH-scrubs the `dbt` binary entirely (no dbt at all); Claude Code Tier-1 hook denylisted only {run,test,parse,compile,...} and allowed `dbt list/show/ls/deps/--help/--version`. The lens-3 cross-runner delta would partially measure "Claude Code can call read-only dbt subcommands but Codex can't" instead of the Recce-equipped-vs-not-equipped signal. Fix: at Tier-1, deny ALL dbt invocations regardless of subcommand. The recce-verify SKILL.md uses `git diff --name-only` + Recce MCP for model discovery, not `dbt list` — so denying read-only dbt subcommands costs nothing in capability. The DENIED_BINS check at line 340 still has the `- {"dbt"}` exclusion so the dbt branch gives its own specific message; the message is updated to reflect the broader policy ("not reachable at Tier-1"). Synthetic hook tests (Tier-1): dbt list -> BLOCKED ✓ dbt show -> BLOCKED ✓ dbt --version -> BLOCKED ✓ dbt run -> BLOCKED ✓ (already was) dbt parse -> BLOCKED ✓ (already was) git status -> allowed ✓ recce list -> allowed ✓ Signed-off-by: even-wei --- evals/agent-blind-spots/build_fixtures.sh | 50 +++++++++++++++++++ .../claude-overlay/hooks/deny-tier-1.py | 23 ++++++--- 2 files changed, 66 insertions(+), 7 deletions(-) diff --git a/evals/agent-blind-spots/build_fixtures.sh b/evals/agent-blind-spots/build_fixtures.sh index 138a5df..a048e0e 100755 --- a/evals/agent-blind-spots/build_fixtures.sh +++ b/evals/agent-blind-spots/build_fixtures.sh @@ -350,6 +350,56 @@ build_fixture() { return 1 fi + # DRC-3584 Andy review B1 / orchestrator iter-6: rewrite the per- + # fixture repo's history so stripped Recce-aware content is NOT + # recoverable via `git show HEAD`, `git cat-file -p HEAD^{tree}`, + # `git log -p`, etc. `git` is Tier-0-allowlisted, so without this + # rewrite the working-tree strip is bypassable. + # + # The simplest history-rewrite: delete `.git/` and re-init a fresh + # single-commit repo from the already-stripped working tree. The + # new commit's tree contains ONLY the stripped paths; no + # ancestor commit, no other ref, no reflog entry references the + # original head SHA's tree. + # + # The upstream head SHA is recorded in commits.txt for + # documentation — losing it from the per-fixture repo is fine + # (and arguably better, since the SHA itself is a weak leak + # vector: an agent could `git log` and infer the upstream + # project from the commit message + author). + rm -rf "${source_dir}/.git" + git -c init.defaultBranch=main -C "${source_dir}" init --quiet + git -c user.email=fixture@recce.eval -c user.name=fixture-build \ + -C "${source_dir}" add -A + git -c user.email=fixture@recce.eval -c user.name=fixture-build \ + -C "${source_dir}" commit --quiet \ + -m "Stripped fixture tree for ${slug} (build_fixtures.sh)" + + # Re-verify the post-rewrite invariants. The rev-list count must + # still be 1 (single commit), and `git ls-tree -r` over the new + # tree must not contain Recce-shaped paths (same regex layers as + # the working-tree leak grep above, but now applied to git + # objects — closes the BLOCKER). + local post_rev_count + post_rev_count="$(git -C "${source_dir}" rev-list --all HEAD --count)" + if [[ "${post_rev_count}" != "1" ]]; then + echo "FAIL ${slug} (post-rewrite Tier-0 leak: rev-list = ${post_rev_count}, expected 1)" >&2 + return 1 + fi + # Anchored regex — match path components exactly, not substrings. + # `\.devcontainer/` matches files inside the dir but NOT the + # sibling `.devcontainer.json` (generic VS Code dbt config, no + # Recce content; would false-positive without the trailing slash). + local git_tree_leak + git_tree_leak="$(git -C "${source_dir}" ls-tree -r --name-only HEAD \ + | grep -E '(^|/)recce\.yml$|(^|/)mcp_config\.json$|(^|/)\.devcontainer/|(^|/)\.github/prompts/|(^|/)\.github/workflows/(claude|recce[-_].*|dbt-build-[a-z]+|dbt_base)\.ya?ml$' \ + || true)" + if [[ -n "${git_tree_leak}" ]]; then + echo "FAIL ${slug} (post-rewrite git tree still has Recce-shaped paths:)" >&2 + sed 's/^/ /' <<< "${git_tree_leak}" >&2 + return 1 + fi + # PR #20 intermediate snapshot — keyed on the well-known SHA in commits.txt. if [[ "${slug}" == "pr44-promotion-flags" ]]; then local intermediate_sha="23b96ca" diff --git a/evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/hooks/deny-tier-1.py b/evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/hooks/deny-tier-1.py index 94df09b..a304c08 100755 --- a/evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/hooks/deny-tier-1.py +++ b/evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/hooks/deny-tier-1.py @@ -328,14 +328,23 @@ def walk(node, original_cmd: str, depth: int = 0) -> None: # When the head resolves to multiple candidates (brace expansion # `{dbt,bash}`, substitution payload `$(echo dbt)`), the same # command could exec ANY of them at runtime. Apply the dbt rule - # eagerly here so a brace literal that includes `dbt` plus a - # denied subcommand in args (`{dbt,bash} run`) denies — without - # this, the shell-wrapper branch below would handle `bash` and - # return without ever checking the dbt arm. - if "dbt" in head_names and _dbt_args_have_denied(arg_words, original_cmd): + # eagerly here so a brace literal that includes `dbt` denies — + # without this, the shell-wrapper branch below would handle + # `bash` and return without ever checking the dbt arm. + # + # At Tier-1 we deny ALL dbt invocations (no subcommand + # discrimination). The Codex Tier-1 recipe PATH-scrubs the `dbt` + # binary entirely; this matches that policy on the Claude Code + # side so the lens-3 cross-runner delta measures the Recce signal, + # not a policy mismatch (Andy review I2 / orchestrator iter-6). + # Tier-1 contract: Recce reads the frozen artifacts; the agent + # never needs to invoke dbt — even read-only subcommands like + # `dbt list` / `dbt show` are unnecessary (the SKILL.md uses + # `git diff --name-only` for model discovery). + if "dbt" in head_names: deny( - "dbt subcommand regenerates frozen artifacts or hits a " - f"warehouse (matched in: {original_cmd!r})" + "dbt is not reachable at Tier-1 — Recce reads the frozen " + f"artifacts (matched in: {original_cmd!r})" ) eager_denied = (head_names & DENIED_BINS) - {"dbt"} if eager_denied: From ff46b9807ab342f966d44737d77ce85918bd029c Mon Sep 17 00:00:00 2001 From: even-wei Date: Mon, 1 Jun 2026 14:46:56 +0800 Subject: [PATCH 11/13] =?UTF-8?q?chore(evals):=20address=20iter-7=20NOTEs?= =?UTF-8?q?=20=E2=80=94=20remove=20dead=20Tier-1=20dbt=20code=20+=20doc=20?= =?UTF-8?q?drift?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both findings from pr-cycle iter-7 review of #36 (explicit non-blockers, author already acknowledged the doc-drift one inline). NOTE 1: dead code in deny-tier-1.py after the iter-6 ISSUE fix. The unconditional `if "dbt" in head_names: deny(...)` at line ~340 makes the entire subcommand-discrimination path unreachable. Removed: - DBT_DENIED_SUBCOMMANDS frozenset (was lines 64-68) - _dbt_args_have_denied() function (was lines 519-527) - The `# --- dbt ---` block in walk() (was lines 458-465) deny-tier-0.py still has its own DBT_DENIED_SUBCOMMANDS — Tier-0 denies the same set + bare `dbt`. The two hook files are kept separate intentionally; cross-file dedup would require restructuring the per-tier overlay layout. Comment at line 65-68 documents this. NOTE 2: doc drift at fixtures/README.md:32 — the prior paragraph mentioned only the simple `grep -E '...'` belt-and-suspenders and made no reference to the history rewrite. Updated to describe: - Both leak-grep layers (tight identifier + loose `[Rr]ecce`) - profiles.yml whitelist rationale - The git history rewrite step + post-rewrite path-leak check - Why the rewrite matters (Tier-0-allowlisted `git` recoverability) Verified: py_compile ok, no functional orphan refs (only one comment mentions the deleted DBT_DENIED_SUBCOMMANDS name as cross-reference to tier-0). 10/10 synthetic hook tests still pass: - dbt list/show/run/parse/--version/deps -> all BLOCKED - git status, recce list -> allowed - sh -c "dbt parse" -> BLOCKED (shell wrapper) - {dbt,bash} run -> BLOCKED (brace expansion) Signed-off-by: even-wei --- evals/agent-blind-spots/fixtures/README.md | 4 ++- .../claude-overlay/hooks/deny-tier-1.py | 32 ++++--------------- 2 files changed, 10 insertions(+), 26 deletions(-) diff --git a/evals/agent-blind-spots/fixtures/README.md b/evals/agent-blind-spots/fixtures/README.md index cf52a3a..156e1ed 100644 --- a/evals/agent-blind-spots/fixtures/README.md +++ b/evals/agent-blind-spots/fixtures/README.md @@ -29,7 +29,9 @@ Re-running is idempotent — existing `artifacts/` directories are removed and r `profiles.yml` is **not** stripped despite carrying the literal string `RECCE` (the upstream Snowflake role name) — it's required for dbt to parse, and a role name is not Recce-the-tool priming. The post-build leak grep whitelists it via `--exclude=profiles.yml`. -The strip is followed by a `grep -E 'mcp__recce__|recce\.yml|RECCE_API_TOKEN'` belt-and-suspenders sweep over the source tree; the build fails fast (`FAIL `) if any new Recce-shaped file slips in at an unanticipated path. When that happens, extend the strip list in `build_fixtures.sh` and re-run. +The strip is followed by a two-layer leak sweep over the working tree (`build_fixtures.sh:340`): (a) tight identifier match — `mcp__recce__`, `recce.yml`, `RECCE_API_TOKEN`; (b) loose `[Rr]ecce` substring to catch natural-language priming like a CI workflow that says *"Use the recce CLI to review this PR"*. `profiles.yml` is whitelisted via `--exclude=profiles.yml`. The build fails fast (`FAIL `) on any hit. + +After the working-tree strip passes, the per-fixture repo's history is **rewritten** into a single fresh commit whose tree IS the stripped working tree (`build_fixtures.sh:370`). Without this rewrite, a Tier-0 agent allowlisted to run `git` could recover the original tree via `git show HEAD:recce.yml` / `git cat-file -p HEAD^{tree}` / `git log -p`, even though those paths are absent from the working tree. The post-rewrite path-leak regex (anchored at path components so `.devcontainer.json` doesn't false-positive) verifies no Recce-shaped paths appear in the new tree and `rev-list --all HEAD --count` is still 1. This is fixture-source-specific, not a contract bug: `DataRecce/jaffle_shop_golden` is a Recce-dogfood repo and the stripped paths are the Recce team's own automation. A random dbt project in the wild would not have them. See [`../ENFORCEMENT.md`](../ENFORCEMENT.md) for how the sandbox profiles relate to this strip. diff --git a/evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/hooks/deny-tier-1.py b/evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/hooks/deny-tier-1.py index a304c08..11852e2 100755 --- a/evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/hooks/deny-tier-1.py +++ b/evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/hooks/deny-tier-1.py @@ -61,11 +61,10 @@ {"dbt", "duckdb", "psql", "snowsql", "bq"} ) -DBT_DENIED_SUBCOMMANDS: frozenset[str] = frozenset( - {"run", "test", "parse", "compile", "docs", "seed", "snapshot", - "build", "freshness", "run-operation", "debug", "source", - "clone", "retry"} -) +# Note: at Tier-1 we deny *all* dbt invocations to match the Codex +# PATH-scrub policy (orchestrator iter-6 ISSUE). No subcommand +# discrimination list is needed; deny-tier-0.py still has its own +# DBT_DENIED_SUBCOMMANDS for the Tier-0 contract. SHELL_WRAPPERS: frozenset[str] = frozenset( {"sh", "bash", "zsh", "dash", "ash", "ksh"} @@ -455,16 +454,10 @@ def walk(node, original_cmd: str, depth: int = 0) -> None: _reparse_and_walk(synth, original_cmd, depth + 1) return - # --- dbt --- - if "dbt" in head_names: - if _dbt_args_have_denied(arg_words, original_cmd): - deny( - "dbt subcommand regenerates frozen artifacts or hits a " - f"warehouse (matched in: {original_cmd!r})" - ) - return - # --- Direct denied bin (psql / duckdb / snowsql / bq) --- + # (dbt is denied unconditionally by the eager check at the top of + # this function; reaching this point means head_names does not + # contain "dbt" and the line below excludes it anyway.) direct_denied = (head_names & DENIED_BINS) - {"dbt"} if direct_denied: name = next(iter(direct_denied)) @@ -516,17 +509,6 @@ def _find_exec_target(head_names: set[str], arg_words: list) -> list: return [] -def _dbt_args_have_denied(arg_words: list, original_cmd: str) -> bool: - """True if any resolved value of any arg (including substitution - payloads, ANSI-C decoded, parameter defaults) is a banned - dbt subcommand.""" - for arg in arg_words: - for cand in resolve_word(arg, original_cmd): - if cand in DBT_DENIED_SUBCOMMANDS: - return True - return False - - def _reparse_and_walk(inner: str, original_cmd: str, depth: int) -> None: if depth > 8 or not inner.strip(): return From 80d5ea3b72959e5347ab97ce0d44d658b1512e15 Mon Sep 17 00:00:00 2001 From: even-wei Date: Mon, 1 Jun 2026 20:40:22 +0800 Subject: [PATCH 12/13] fix(evals): close iter-8 BLOCKERs + ISSUEs on spike driver + overlays MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Address @wcchang1115's re-review at ff46b98 (PR #36). BLOCKER 1 — overlay-as-spoiler at the agent's cwd. The Tier-0 enforcement overlay copied to `/.claude/` contains Recce-shaped strings (`mcp__recce__*`, `recce`, `RECCE_API_TOKEN`); `cat`/`ls`/`grep` over `.claude/` would let a Tier-0 agent read its own enforcement file and defeat the "agent cannot reach Recce-shaped signals" contract. * Tier-0 + Tier-1 `settings.json`: deny `Read/Glob/Grep(.claude/**)`. (Project-level deny rules apply even with `--dangerously-skip-permissions`; that flag bypasses the interactive prompt, not the rule.) * Tier-0 hook (`deny-tier-0.py`): new Pass 0 over command words and redirect targets. Any argument matching `(^|[/=])\.claude(/|$)` exits 2 with a "references the Tier-0 enforcement overlay" message. Covers `cat .claude/...`, `cat ./.claude/...`, `cat < .claude/...`, `ls .claude/`, `find .claude`, `grep -r recce .claude/`, and `cat $(echo .claude/...)` — i.e., the non-adversarial shapes. Interpreter-shell-out (awk/python `-c`) is intentionally out of scope per ENFORCEMENT.md § "Threat model — non-adversarial". BLOCKER 2 — Codex Tier-0 PATH-regex scrub was a no-op for normal installs. `grep -v -E '/recce(/|$)|/dbt(/|$)|\.recce'` strips dirs whose path contains literal `/recce/` or `/dbt/` — NOT dirs containing the binary named `recce`/`dbt`. So `/opt/homebrew/bin`, `.venv/bin`, `~/.local/bin` all survived and Codex Tier-0 could still run `recce list` / `dbt list`. * New `runner-configs/codex/tier-0/stub-bin/{recce,dbt}` — exit-127 stubs prepended to PATH so the real binaries are masked regardless of which bin dir they live in. * `driver.py::codex_tier0_path()` builds the PATH (stub-bin first, then the legacy regex scrub as belt-and-suspenders). * `driver.py::assert_codex_tier0_masked()` runs `command -v recce|dbt` under the final env before the agent launches; if either resolves outside `stub-bin/`, the cell errors out instead of contaminating the Tier-0 baseline. Verified: regression detected when stub-bin is skipped. * `codex/tier-0/README.md` updated with the new recipe + pre-flight assertion shell snippet. ISSUE 1 — Tier-0 read/write asymmetry. Codex Tier-0 is OS read-only; Claude Code Tier-0 had no Write/Edit deny + matcher didn't fire on those tool kinds + `--dangerously-skip-permissions` was set. * Tier-0 `settings.json`: deny `Write`, `Edit`, `MultiEdit`, `NotebookEdit`. Deny rules are enforced separately from `--dangerously-skip-permissions`. ISSUE 2 — cells don't reset state between runs. `fixture_dir = SOURCES_DIR / cell.fixture` was reused across all four cells per fixture with no `git reset`, so a write from cell N persisted into cell N+1. * New `driver.py::reset_fixture_dir()` runs `git reset --hard HEAD` + `git clean -fdx` at the top of `run_cell()`, before `stage_inputs()` (whose `_eval_inputs/` symlinks would otherwise be cleaned away) and before the overlay copy. Each cell now starts from the same committed base. ISSUE 3 — no model pinning (ENFORCEMENT.md:157). * `driver.py`: new `--model` flag (default `claude-opus-4-5`), `--no-model` opt-out. The pin is passed to `claude --model` and `codex --model` for every agent invocation and to the judge call. Cell metadata (`cells.json`, transcript headers) records the pinned model so cross-cell comparability is auditable. Deferred (Andy's Notes, non-blocking for this PR): - N1: tier-1 dbt deny-list / docs drift. - N2: `_eval_inputs/` symlinks could be written through at Tier-1. - N3: `codex/tier-1/config.toml` `command = ""` unvalidated. Verification: - `py_compile` clean on deny-tier-{0,1}.py + driver.py. - JSON valid on both `settings.json` files. - Synthetic Bash-hook tests: 12/12 `.claude/` access shapes denied; 8/8 allowlisted commands pass. - `codex_tier0_path()` + `assert_codex_tier0_masked()` pre-flight works under the new stub-bin recipe and catches regressions if stub-bin is dropped. - `driver.py --help` shows `--model` / `--no-model`. Signed-off-by: even-wei --- .../claude-overlay/hooks/deny-tier-0.py | 48 +++++ .../tier-0/claude-overlay/settings.json | 14 +- .../tier-1/claude-overlay/settings.json | 10 +- .../runner-configs/codex/tier-0/README.md | 25 ++- .../runner-configs/codex/tier-0/stub-bin/dbt | 8 + .../codex/tier-0/stub-bin/recce | 9 + .../agent-blind-spots/spike-driver/driver.py | 175 +++++++++++++++--- 7 files changed, 252 insertions(+), 37 deletions(-) create mode 100755 evals/agent-blind-spots/runner-configs/codex/tier-0/stub-bin/dbt create mode 100755 evals/agent-blind-spots/runner-configs/codex/tier-0/stub-bin/recce diff --git a/evals/agent-blind-spots/runner-configs/claude-code/tier-0/claude-overlay/hooks/deny-tier-0.py b/evals/agent-blind-spots/runner-configs/claude-code/tier-0/claude-overlay/hooks/deny-tier-0.py index be28fe3..bbc2855 100755 --- a/evals/agent-blind-spots/runner-configs/claude-code/tier-0/claude-overlay/hooks/deny-tier-0.py +++ b/evals/agent-blind-spots/runner-configs/claude-code/tier-0/claude-overlay/hooks/deny-tier-0.py @@ -77,6 +77,16 @@ # Recce skill prefix — `recce-verify`, `recce:recce-review`, etc. RECCE_SKILL_RE = re.compile(r"^recce[-:]", re.IGNORECASE) +# Overlay-leak guard. The Tier-0 enforcement overlay itself names Recce +# (settings.json deny rules, deny-tier-0.py vocabulary). `cat` is in +# the allowlist, so a bare `cat .claude/settings.json` would otherwise +# turn the enforcement file into a Recce-shaped spoiler. Block any Bash +# command argument that references `.claude/` or the `.claude` +# directory itself. The Read/Glob/Grep tool paths are blocked +# separately via permissions.deny in settings.json (deny rules apply +# even with --dangerously-skip-permissions). +_CLAUDE_DIR_RE = re.compile(r"(?:^|[/=])\.claude(?:/|$)") + # --- Output --------------------------------------------------------------- @@ -223,6 +233,44 @@ def walk(node, original_cmd: str, depth: int = 0) -> None: if not words: return + # Pass 0: overlay-leak guard. Any word argument that references + # `.claude/` (the directory holding the Tier-0 enforcement overlay + # itself) is denied at Tier 0 — its contents name Recce vocabulary + # and reading it would defeat the "agent cannot reach Recce-shaped + # signals" contract. Read/Glob/Grep tool kinds are blocked + # separately by permissions.deny in settings.json. + # + # This intentionally catches the non-adversarial shape — a normal + # agent inspecting cwd via `cat .claude/...`, `ls .claude/`, + # `cat < .claude/...`, `find .claude`, etc. The interpreter-shell-out + # bypass class (awk/python -c "...cat .claude/...") is out of scope + # per ENFORCEMENT.md § "Threat model — non-adversarial code agent". + for w in words: + for cand in resolve_word(w, original_cmd): + if cand and _CLAUDE_DIR_RE.search(cand): + deny( + f"argument '{cand}' references the Tier-0 enforcement " + f"overlay at .claude/, which names Recce vocabulary " + f"(matched in: {original_cmd!r})" + ) + + # Redirect targets (`cat < .claude/settings.json`, + # `grep recce < .claude/...`) bypass the words pass because + # `/) before invoking Claude Code. See ../../../ENFORCEMENT.md for the runner recipe. The PreToolUse hook is the load-bearing layer; permissions.deny below is documentation-aligned but does not cover every bypass the hook does — see runner-configs/README.md.", + "$comment": "Tier-0 sandbox profile for the /recce-verify v1 eval. Drop this .claude/ directory into the per-fixture working tree (.tmp/sources//) before invoking Claude Code. See ../../../ENFORCEMENT.md for the runner recipe. The PreToolUse hook is the load-bearing layer; permissions.deny below is documentation-aligned but does not cover every bypass the hook does — see runner-configs/README.md. Note: deny rules are enforced even with --dangerously-skip-permissions; only the interactive prompt is bypassed by that flag.", "permissions": { "deny": [ "mcp__recce__*", @@ -14,7 +14,17 @@ "Bash(duckdb *)", "Bash(psql *)", "Bash(snowsql *)", - "Bash(bq *)" + "Bash(bq *)", + "Read(.claude/**)", + "Read(./.claude/**)", + "Glob(.claude/**)", + "Glob(./.claude/**)", + "Grep(.claude/**)", + "Grep(./.claude/**)", + "Write", + "Edit", + "MultiEdit", + "NotebookEdit" ] }, "hooks": { diff --git a/evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/settings.json b/evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/settings.json index be3581d..d358e7c 100644 --- a/evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/settings.json +++ b/evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/settings.json @@ -1,5 +1,5 @@ { - "$comment": "Tier-1 sandbox profile for the /recce-verify v1 eval. Tier 1 is Tier 0 plus Recce CLI, Recce MCP, and single-env warehouse credentials (read-only on the dev environment). Drop this .claude/ directory into the per-fixture working tree (.tmp/sources//) before invoking Claude Code. See ../../../ENFORCEMENT.md for the runner recipe.", + "$comment": "Tier-1 sandbox profile for the /recce-verify v1 eval. Tier 1 is Tier 0 plus Recce CLI, Recce MCP, and single-env warehouse credentials (read-only on the dev environment). Drop this .claude/ directory into the per-fixture working tree (.tmp/sources//) before invoking Claude Code. See ../../../ENFORCEMENT.md for the runner recipe. Note: deny rules are enforced even with --dangerously-skip-permissions.", "permissions": { "deny": [ "Bash(dbt run*)", @@ -13,7 +13,13 @@ "Bash(duckdb *)", "Bash(psql *)", "Bash(snowsql *)", - "Bash(bq *)" + "Bash(bq *)", + "Read(.claude/**)", + "Read(./.claude/**)", + "Glob(.claude/**)", + "Glob(./.claude/**)", + "Grep(.claude/**)", + "Grep(./.claude/**)" ] }, "hooks": { diff --git a/evals/agent-blind-spots/runner-configs/codex/tier-0/README.md b/evals/agent-blind-spots/runner-configs/codex/tier-0/README.md index 5e1b403..29a97d3 100644 --- a/evals/agent-blind-spots/runner-configs/codex/tier-0/README.md +++ b/evals/agent-blind-spots/runner-configs/codex/tier-0/README.md @@ -4,7 +4,7 @@ Codex (OpenAI CLI) does not load `.claude/settings.json`. Enforcement at Tier 0 1. **Process sandbox** via `--sandbox=read-only` — blocks any write the agent attempts outside the workspace, which already blocks `dbt run`/`test`/`parse`/`compile`/`docs generate` (all write `target/`) and any `recce` invocation that wants to mutate state under `~/.recce/`. 2. **MCP allowlist** — invoke Codex with a `config.toml` that registers zero MCP servers (see `config.toml` in this directory). The Recce MCP server is not reachable to the agent. -3. **PATH scrub** — run Codex with a sanitised `PATH` so Recce-related binaries are not on it. This catches read-only Recce CLI calls (e.g., `recce list`) that the process sandbox would otherwise allow. +3. **PATH stub overlay** — `stub-bin/` contains exit-127 stubs for `recce` and `dbt`. Prepending `stub-bin/` to `PATH` masks the real binaries no matter which directory they live in (`/opt/homebrew/bin`, `~/.local/bin`, `.venv/bin`, etc.). An earlier regex-based PATH scrub only stripped directories whose path contained `/recce/` or `/dbt/` literally — bin directories where the real binaries live were not matched and `recce list` / `dbt list` survived the scrub. The stub overlay closes that hole. The regex scrub is still applied as belt-and-suspenders for any directory whose path *does* contain `/recce/`, `/dbt/`, or `.recce`. 4. **Per-fixture working directory** — Codex runs with cwd at `.tmp/sources//` (per-fixture standalone repo from `build_fixtures.sh`). Sibling fixtures, `evals/agent-blind-spots/RUBRIC.md`, and `fixtures//README.md` are outside the workspace and not in `cwd`, so file reads can't reach them. Make sure you do **not** launch Codex from the eval repo root. ## Invocation recipe @@ -13,12 +13,25 @@ Codex (OpenAI CLI) does not load `.claude/settings.json`. Enforcement at Tier 0 SLUG="pr1-fix-clv" WT_ROOT="$(git rev-parse --show-toplevel)" FIXTURE_DIR="${WT_ROOT}/evals/agent-blind-spots/.tmp/sources/${SLUG}" +TIER0_DIR="${WT_ROOT}/evals/agent-blind-spots/runner-configs/codex/tier-0" -# Strip Recce + dbt binaries from PATH so even read-only CLI calls fail. -SAFE_PATH="$(echo "$PATH" \ +# Prepend the stub-bin overlay so `recce` and `dbt` resolve to exit-127 +# stubs no matter which bin directory the real binary lives in. Belt-and- +# suspenders: also strip any literal /recce/, /dbt/, .recce dirs. +SCRUBBED_PATH="$(echo "$PATH" \ | tr ':' '\n' \ | grep -v -E '/recce(/|$)|/dbt(/|$)|\.recce' \ | paste -sd: -)" +SAFE_PATH="${TIER0_DIR}/stub-bin:${SCRUBBED_PATH}" + +# Pre-flight assertion: real binaries must NOT shadow the stub. +for bin in recce dbt; do + resolved="$(PATH="${SAFE_PATH}" command -v "${bin}" || true)" + case "${resolved}" in + "${TIER0_DIR}/stub-bin/"*|"") ;; + *) echo "Tier-0 PATH masking failed: ${bin} resolves to ${resolved}" >&2; exit 1 ;; + esac +done cd "${FIXTURE_DIR}" @@ -28,7 +41,7 @@ DBT_PROFILES_DIR="" \ codex exec \ --sandbox=read-only \ --ask-for-approval=never \ - --config "${WT_ROOT}/evals/agent-blind-spots/runner-configs/codex/tier-0/config.toml" \ + --config "${TIER0_DIR}/config.toml" \ "" ``` @@ -38,9 +51,9 @@ codex exec \ | Concern | Enforced by | Notes | |---|---|---| -| Cannot regenerate frozen artifacts (`manifest`, `compiled`, `catalog`) | `--sandbox=read-only` (no `target/` writes) + PATH scrub (no `dbt` binary) | Belt-and-suspenders | +| Cannot regenerate frozen artifacts (`manifest`, `compiled`, `catalog`) | `--sandbox=read-only` (no `target/` writes) + stub-bin overlay (no `dbt` binary) | Belt-and-suspenders | | Cannot reach Recce MCP | `config.toml` registers zero MCP servers | The `mcp_servers` table is intentionally empty in `config.toml` | -| Cannot invoke `recce` CLI | PATH scrub | Process sandbox alone doesn't help for read-only Recce calls | +| Cannot invoke `recce` CLI | `stub-bin/recce` overlay (exit 127) prepended to PATH | Process sandbox alone doesn't help for read-only Recce calls; the regex scrub alone didn't help either because `recce` typically lives in `/opt/homebrew/bin`, `.venv/bin`, etc. — directories the regex didn't match. | | Cannot reach a warehouse | No warehouse credentials in the eval shell + read-only sandbox + PATH scrub of SQL clients | Recipe should never `export SNOWFLAKE_PASSWORD=...` for Tier 0 | | Cannot read spoiler files (`fixtures//README.md`, `RUBRIC.md`, sibling fixtures) | cwd at `.tmp/sources//` (a separate git repo from the eval host) | Out-of-workspace paths are not in the read sandbox | diff --git a/evals/agent-blind-spots/runner-configs/codex/tier-0/stub-bin/dbt b/evals/agent-blind-spots/runner-configs/codex/tier-0/stub-bin/dbt new file mode 100755 index 0000000..3645b60 --- /dev/null +++ b/evals/agent-blind-spots/runner-configs/codex/tier-0/stub-bin/dbt @@ -0,0 +1,8 @@ +#!/usr/bin/env bash +# Tier-0 stub for `dbt`. See stub-bin/recce for rationale — the prior PATH +# regex only stripped dirs whose path contained `/dbt/`, leaving `dbt` in +# real bin/ directories reachable. This stub is prepended to PATH at Tier 0 +# so any `dbt` invocation resolves here first and exits non-zero. +echo "Tier-0 sandbox blocks: dbt CLI is not available at Tier 0." >&2 +echo " (stub at $0; the real binary is masked by PATH overlay)" >&2 +exit 127 diff --git a/evals/agent-blind-spots/runner-configs/codex/tier-0/stub-bin/recce b/evals/agent-blind-spots/runner-configs/codex/tier-0/stub-bin/recce new file mode 100755 index 0000000..aa0c1ce --- /dev/null +++ b/evals/agent-blind-spots/runner-configs/codex/tier-0/stub-bin/recce @@ -0,0 +1,9 @@ +#!/usr/bin/env bash +# Tier-0 stub for `recce`. The PATH-regex scrub in earlier revisions only +# stripped directories whose path contained `/recce/` or `/dbt/` — directories +# like `/opt/homebrew/bin`, `~/.local/bin`, and `.venv/bin` (where the real +# binaries actually live) survived the scrub. This stub is prepended to PATH +# so any Tier-0 `recce` invocation resolves here first and exits non-zero. +echo "Tier-0 sandbox blocks: recce CLI is not available at Tier 0." >&2 +echo " (stub at $0; the real binary is masked by PATH overlay)" >&2 +exit 127 diff --git a/evals/agent-blind-spots/spike-driver/driver.py b/evals/agent-blind-spots/spike-driver/driver.py index 80cba79..6199f45 100644 --- a/evals/agent-blind-spots/spike-driver/driver.py +++ b/evals/agent-blind-spots/spike-driver/driver.py @@ -102,6 +102,7 @@ class Cell: fixture: str agent: str tier: int + model: str | None = None transcript_path: str | None = None returncode: int | None = None verdict: dict | None = None @@ -129,6 +130,47 @@ def scrub_path(strip_patterns: tuple[str, ...]) -> str: return ":".join(p for p in os.environ.get("PATH", "").split(":") if not pat.search(p)) +def codex_tier0_path(stub_bin: Path) -> str: + """Return a PATH for Codex Tier-0 that genuinely masks `recce`/`dbt`. + + The earlier regex scrub only stripped directories whose path contained + `/recce/` or `/dbt/` literally; bin dirs like `/opt/homebrew/bin`, + `~/.local/bin`, and `.venv/bin` (where the real binaries live) survived + the scrub. Prepending `stub-bin/` — which contains `recce` and `dbt` + scripts that exit 127 — makes the masking real: shell name resolution + hits the stub first. + """ + # Belt-and-suspenders: keep the regex scrub for any path literally + # containing /recce/ or /dbt/ or .recce, AND prepend the stub-bin so + # the standard bin dirs are overridden. + scrubbed = scrub_path((r"/recce(/|$)", r"/dbt(/|$)", r"\.recce")) + return f"{stub_bin}:{scrubbed}" if scrubbed else str(stub_bin) + + +def assert_codex_tier0_masked(env: dict, cwd: Path) -> None: + """Fail fast if `recce`/`dbt` resolve to anything other than the stub. + + Codex's Tier-0 Recce/dbt block is now PATH stub-bin masking; the cell + is contaminated the moment `recce list` or `dbt list` could actually + run. Run `command -v` under the scrubbed env before invoking the + agent — if either resolves outside the stub-bin, abort. + """ + stub_bin_dir = str(RUNNER_CONFIGS / "codex" / "tier-0" / "stub-bin") + for binary in ("recce", "dbt"): + proc = subprocess.run( + ["bash", "-c", f"command -v {binary} || true"], + cwd=str(cwd), env=env, capture_output=True, text=True, timeout=5, + ) + resolved = proc.stdout.strip() + if resolved and not resolved.startswith(stub_bin_dir): + raise RuntimeError( + f"Codex Tier-0 PATH masking failed: '{binary}' resolves to " + f"{resolved!r} (expected stub under {stub_bin_dir!r}). " + "Aborting before the cell runs to avoid contaminating the " + "Tier-0 baseline." + ) + + def stage_inputs(fixture_dir: Path, fixture_id: str) -> None: """Stage frozen Tier-0 inputs into the agent's cwd at `_eval_inputs/`. @@ -166,7 +208,7 @@ def stage_inputs(fixture_dir: Path, fixture_id: str) -> None: ) -def run_claude(fixture_dir: Path, tier: int, run_dir: Path, prompt: str) -> tuple[Path, int]: +def run_claude(fixture_dir: Path, tier: int, run_dir: Path, prompt: str, model: str | None) -> tuple[Path, int]: tier_dir = RUNNER_CONFIGS / "claude-code" / f"tier-{tier}" overlay_src = tier_dir / "claude-overlay" overlay_dst = fixture_dir / ".claude" @@ -191,19 +233,23 @@ def run_claude(fixture_dir: Path, tier: int, run_dir: Path, prompt: str) -> tupl cfg_dir.mkdir(parents=True, exist_ok=True) env["CLAUDE_CONFIG_DIR"] = str(cfg_dir) + cmd = ["claude", "--print", "--dangerously-skip-permissions"] + if model: + cmd += ["--model", model] + cmd.append(prompt) proc = subprocess.run( - ["claude", "--print", "--dangerously-skip-permissions", prompt], - cwd=str(fixture_dir), env=env, capture_output=True, text=True, timeout=900, + cmd, cwd=str(fixture_dir), env=env, capture_output=True, text=True, timeout=900, ) transcript_path.write_text( f"# Cell: {fixture_dir.name} · claude · tier-{tier}\n" + f"# model: {model or '(unpinned)'}\n" f"# returncode: {proc.returncode}\n\n" f"## stdout\n{proc.stdout}\n\n## stderr\n{proc.stderr}\n" ) return transcript_path, proc.returncode -def run_codex(fixture_dir: Path, tier: int, run_dir: Path, prompt: str) -> tuple[Path, int]: +def run_codex(fixture_dir: Path, tier: int, run_dir: Path, prompt: str, model: str | None) -> tuple[Path, int]: tier_dir = RUNNER_CONFIGS / "codex" / f"tier-{tier}" config = tier_dir / "config.toml" sandbox = "read-only" if tier == 0 else "workspace-write" @@ -211,29 +257,66 @@ def run_codex(fixture_dir: Path, tier: int, run_dir: Path, prompt: str) -> tuple transcript_path = run_dir / "transcripts" / f"{fixture_dir.name}_codex_t{tier}.txt" transcript_path.parent.mkdir(parents=True, exist_ok=True) - strip_patterns = (r"/recce(/|$)", r"/dbt(/|$)", r"\.recce") if tier == 0 else (r"/dbt(/|$)",) env = scrub_env() - env["PATH"] = scrub_path(strip_patterns) - # Tier-1 keeps single-env warehouse credentials from the parent shell; - # the operator is responsible for absence of base/prod creds (see codex/tier-1/README.md). - + if tier == 0: + # Tier 0: real PATH masking via stub-bin (the earlier regex-only + # scrub left `recce`/`dbt` reachable via `/opt/homebrew/bin`, + # `.venv/bin`, etc.). Pre-flight `command -v` confirms the mask. + stub_bin = tier_dir / "stub-bin" + env["PATH"] = codex_tier0_path(stub_bin) + assert_codex_tier0_masked(env, fixture_dir) + else: + # Tier 1: strip only dbt-named directories; Recce CLI must remain + # reachable. Single-env warehouse credentials are inherited from + # the parent shell; operator is responsible for absence of + # base/prod creds (see codex/tier-1/README.md). + env["PATH"] = scrub_path((r"/dbt(/|$)",)) + + cmd = ["codex", "exec", + f"--sandbox={sandbox}", + "--ask-for-approval=never", + "--config", str(config)] + if model: + cmd += ["--model", model] + cmd.append(prompt) proc = subprocess.run( - ["codex", "exec", - f"--sandbox={sandbox}", - "--ask-for-approval=never", - "--config", str(config), - prompt], - cwd=str(fixture_dir), env=env, capture_output=True, text=True, timeout=900, + cmd, cwd=str(fixture_dir), env=env, capture_output=True, text=True, timeout=900, ) transcript_path.write_text( f"# Cell: {fixture_dir.name} · codex · tier-{tier}\n" + f"# model: {model or '(unpinned)'}\n" f"# returncode: {proc.returncode}\n\n" f"## stdout\n{proc.stdout}\n\n## stderr\n{proc.stderr}\n" ) return transcript_path, proc.returncode -def run_cell(cell: Cell, run_dir: Path) -> None: +def reset_fixture_dir(fixture_dir: Path) -> None: + """Reset the fixture worktree to its committed HEAD before a cell runs. + + Each fixture's `.tmp/sources//` directory is reused across all + 4 cells (claude×{0,1}, codex×{0,1}). Without an explicit reset, a + write from cell N (Claude Code at Tier 0 was previously + write-capable; Tier 1 is write-capable by design for staging Recce + artifacts) persists into cell N+1, breaking the "all cells start + from the same base" contract in ENFORCEMENT.md. + + `_eval_inputs/` is staged by stage_inputs() *after* this reset so + the symlinks survive — git treats them as untracked and + `git clean -fdx` removes them, which is intended (stage_inputs() + re-creates them per cell). + """ + subprocess.run( + ["git", "reset", "--hard", "HEAD"], + cwd=str(fixture_dir), check=True, capture_output=True, text=True, + ) + subprocess.run( + ["git", "clean", "-fdx"], + cwd=str(fixture_dir), check=True, capture_output=True, text=True, + ) + + +def run_cell(cell: Cell, run_dir: Path, model: str | None) -> None: fixture_dir = SOURCES_DIR / cell.fixture if not fixture_dir.exists(): cell.error = f"per-fixture worktree missing: {fixture_dir} (did you run build_fixtures.sh?)" @@ -241,22 +324,41 @@ def run_cell(cell: Cell, run_dir: Path) -> None: if not cli_available(cell.agent): cell.error = f"{cell.agent} CLI not found on PATH; skipping" return + # Reset the fixture worktree before staging inputs and copying the + # overlay, so each cell starts from the same base — no Claude write + # from a prior cell bleeds across. Must run *before* stage_inputs() + # (whose `_eval_inputs/` symlinks are untracked and would be removed + # by `git clean -fdx`) and *before* run_claude()'s overlay copy. + try: + reset_fixture_dir(fixture_dir) + except subprocess.CalledProcessError as e: + cell.error = ( + f"reset_fixture_dir failed for {fixture_dir}: " + f"rc={e.returncode}; stderr={(e.stderr or '').strip()[:200]}" + ) + return try: stage_inputs(fixture_dir, cell.fixture) except FileNotFoundError as e: cell.error = f"stage_inputs failed: {e}" return + cell.model = model try: if cell.agent == "claude": - path, rc = run_claude(fixture_dir, cell.tier, run_dir, AGENT_PROMPT) + path, rc = run_claude(fixture_dir, cell.tier, run_dir, AGENT_PROMPT, model) else: - path, rc = run_codex(fixture_dir, cell.tier, run_dir, AGENT_PROMPT) + path, rc = run_codex(fixture_dir, cell.tier, run_dir, AGENT_PROMPT, model) cell.transcript_path = str(path) cell.returncode = rc except subprocess.TimeoutExpired: cell.error = f"{cell.agent} timed out after 900s" except FileNotFoundError as e: cell.error = f"{cell.agent} setup failed: {e}" + except RuntimeError as e: + # Pre-flight assertion failures (e.g., Codex Tier-0 PATH masking + # didn't take) surface here so the cell is recorded as errored + # rather than producing a contaminated transcript. + cell.error = f"{cell.agent} pre-flight failed: {e}" def _shrink(text: str, limit: int = 30000) -> str: @@ -266,7 +368,7 @@ def _shrink(text: str, limit: int = 30000) -> str: return text[:half] + "\n…[truncated]…\n" + text[-half:] -def judge_cell(cell: Cell, rubric: str, baseline_catch: str) -> dict: +def judge_cell(cell: Cell, rubric: str, baseline_catch: str, model: str | None) -> dict: if not cell.transcript_path: return {"error": "no transcript"} transcript = _shrink(Path(cell.transcript_path).read_text()) @@ -275,10 +377,12 @@ def judge_cell(cell: Cell, rubric: str, baseline_catch: str) -> dict: fixture=cell.fixture, agent=cell.agent, tier=cell.tier, transcript=transcript, baseline_catch=baseline_catch, ) + judge_cmd = ["claude", "--print", "--dangerously-skip-permissions"] + if model: + judge_cmd += ["--model", model] + judge_cmd += ["--append-system-prompt", JUDGE_SYSTEM, user] proc = subprocess.run( - ["claude", "--print", "--dangerously-skip-permissions", - "--append-system-prompt", JUDGE_SYSTEM, user], - capture_output=True, text=True, timeout=300, + judge_cmd, capture_output=True, text=True, timeout=300, ) if proc.returncode != 0: return {"error": f"judge rc={proc.returncode}", "stderr": proc.stderr[:300]} @@ -403,7 +507,24 @@ def main() -> int: parser.add_argument("--baseline-dir", type=Path, help="DRC-3585 manual baseline dir; compares judge to human on catch axis") parser.add_argument("--no-run", action="store_true", help="Skip agent runs; judge existing transcripts in --run-dir") parser.add_argument("--run-dir", type=Path, help="Override run output dir (default runs//spike-driver/)") + parser.add_argument( + "--model", + default="claude-opus-4-5", + help=( + "Model id to pin on every agent + judge invocation " + "(ENFORCEMENT.md:157 requires the runner to pin a single model " + "across the matrix). Same value is passed to both `claude --model` " + "and `codex --model`. Use --no-model to opt out for ad-hoc smoke runs." + ), + ) + parser.add_argument( + "--no-model", + action="store_true", + help="Skip --model pin (cells fall back to each CLI's default). " + "Recorded as '(unpinned)' in transcripts and cells.json.", + ) args = parser.parse_args() + model = None if args.no_model else args.model agents = [a for a in args.agents.split(",") if a] tiers = [int(t) for t in args.tiers.split(",") if t] @@ -427,8 +548,8 @@ def main() -> int: if not args.no_run: for cell in cells: - print(f"[run] {cell.fixture} · {cell.agent} · tier-{cell.tier}", file=sys.stderr) - run_cell(cell, run_dir) + print(f"[run] {cell.fixture} · {cell.agent} · tier-{cell.tier} · model={model or '(unpinned)'}", file=sys.stderr) + run_cell(cell, run_dir, model) if cell.error: print(f" → {cell.error}", file=sys.stderr) else: @@ -444,10 +565,10 @@ def main() -> int: if not cell.transcript_path: continue baseline_catch = baseline.get(cell.fixture, "unknown") - print(f"[judge] {cell.fixture} · {cell.agent} · tier-{cell.tier}", file=sys.stderr) - cell.verdict = judge_cell(cell, rubric, baseline_catch) + print(f"[judge] {cell.fixture} · {cell.agent} · tier-{cell.tier} · model={model or '(unpinned)'}", file=sys.stderr) + cell.verdict = judge_cell(cell, rubric, baseline_catch, model) if args.judge_stability: - cell.verdict_2 = judge_cell(cell, rubric, baseline_catch) + cell.verdict_2 = judge_cell(cell, rubric, baseline_catch, model) csv_path = run_dir / "verdicts.csv" write_csv(cells, csv_path) From 8b5a504278025d4b27d7cd564f9e174a900e3cbd Mon Sep 17 00:00:00 2001 From: even-wei Date: Tue, 2 Jun 2026 18:19:33 +0800 Subject: [PATCH 13/13] fix(evals): close iter-8 BLOCKER (Tier-0 overlay leak via cwd) structurally MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Andy's iter-8 review (PR #36, on 80d5ea3) flagged that the Pass 0 overlay-leak guard in deny-tier-0.py only matches the literal `.claude` substring, so a non-adversarial Tier-0 agent dumps the overlay's Recce vocabulary via ordinary recursive reads the Bash AST hook can't see ahead of glob expansion: cat .*/settings.json (glob expands AFTER the hook reads literal) grep -r recce . (recurse descends into .claude/) find . -type f -exec cat {} \; cat $(find . -type f) Token-matching is whack-a-mole; the root cause is structural — the overlay (mcp__recce__*, Bash(recce *), RECCE_API_TOKEN, recce literals in the hook source) lives in the agent's cwd as /.claude/. Andy's recommended fix (verified on CC v2.1.160): load settings via `claude --settings ` from outside cwd, with the hook command pointing to an absolute path also outside cwd. The agent's cwd then becomes a pristine dbt project with no Recce-shaped files — leak is structurally impossible. Driver change (spike-driver/driver.py): * Add render_claude_settings() — reads the template settings.json, substitutes ${RUNNER_HOOK_PATH} with the absolute path to the tier-N hook script (under runner-configs/), writes the rendered file to run_dir/_settings/_t.json. * run_claude() drops shutil.copytree(overlay_src, fixture/.claude), calls render_claude_settings(), and passes --settings to `claude --print`. * Codex path (run_codex, assert_codex_tier0_masked) unchanged — Codex uses --config + PATH stub-bin masking, separate enforcement shape. Template change (tier-{0,1}/claude-overlay/settings.json): * `hooks.PreToolUse[].hooks[].command` now uses ${RUNNER_HOOK_PATH} placeholder; the runner substitutes at render time. * Hook matcher unchanged: Bash|Skill|mcp__(plugin_)?recce(_|-).* * permissions.deny rules unchanged (belt-and-suspenders, even though the overlay is no longer in cwd). Doc updates: * ENFORCEMENT.md Claude Code recipe: bash example now renders the template to a temp path and passes `claude --settings ` instead of `cp -r claude-overlay/ .claude/`. Same shape for Tier-1. * runner-configs/README.md: directory map and Quick start updated; added the "why not copy" explainer pointing at the leak. Verification: * py_compile passes for driver.py, deny-tier-{0,1}.py. * JSON parse passes for tier-{0,1} settings templates. * Filesystem check: after reset_fixture_dir() + stage_inputs(), find -maxdepth 2 -name "*.claude*" returns empty. Fixture cwd contains dbt project files + _eval_inputs/ symlink only. * Rendered settings live at run_dir/_settings/, outside fixture cwd. * Rendered hook command field carries an absolute path that resolves to an existing deny-tier-N.py with no ${RUNNER_HOOK_PATH} leakage. * Andy's 4 bypass shapes (cat .*/settings.json, grep -r Bash(recce ., find . -type f -exec grep -l deny-tier {} \;, cat $(find . -name deny-tier-0.py)) all return empty / no-such-file — overlay simply isn't there. * Standalone hook invocation (Bash dbt parse → exit 2; Bash ls → exit 0) still works when the hook script is executed from outside cwd. * Codex Tier-0 PATH masking (assert_codex_tier0_masked) is untouched and still aborts when recce/dbt resolve outside stub-bin. Out of scope (per the iter-8 review): * Codex --ignore-rules / --dangerously-bypass-hook-trust (codex-only, separate code path). * B1 (git history leak) and I2 (Tier-1 dbt policy) — already closed in 6f20289. * 7 adversarial-bypass classes documented in ENFORCEMENT.md § "Threat model — non-adversarial code agent". * The 3 NOTEs Andy explicitly deferred. Refs: DRC-3584 Signed-off-by: even-wei --- evals/agent-blind-spots/ENFORCEMENT.md | 37 ++++++--- .../runner-configs/README.md | 30 ++++++-- .../tier-0/claude-overlay/settings.json | 4 +- .../tier-1/claude-overlay/settings.json | 4 +- .../agent-blind-spots/spike-driver/driver.py | 77 +++++++++++++++---- 5 files changed, 115 insertions(+), 37 deletions(-) diff --git a/evals/agent-blind-spots/ENFORCEMENT.md b/evals/agent-blind-spots/ENFORCEMENT.md index 3c08065..ea365bd 100644 --- a/evals/agent-blind-spots/ENFORCEMENT.md +++ b/evals/agent-blind-spots/ENFORCEMENT.md @@ -30,11 +30,24 @@ TIER_DIR="${WT_ROOT}/evals/agent-blind-spots/runner-configs/claude-code/tier-0" ( cd "${WT_ROOT}/evals/agent-blind-spots" && ./build_fixtures.sh ) python3 -c "import bashlex" || python3 -m pip install bashlex -# 2. Stamp the per-fixture working tree with the Tier-0 sandbox config. -# `claude-overlay/` becomes `.claude/` inside the fixture — the source -# directory is named differently so it isn't swallowed by the repo's -# `.claude/` gitignore rule. -cp -r "${TIER_DIR}/claude-overlay" "${FIXTURE_DIR}/.claude" +# 2. Render the Tier-0 settings template to a path *outside* the fixture cwd. +# The template's `command` field carries a ${RUNNER_HOOK_PATH} placeholder +# that must be substituted with an absolute path to the hook script +# (also outside cwd). NOTE: the agent's cwd MUST stay free of any +# `.claude/` directory — the overlay file and the hook script both name +# Recce vocabulary, and a Tier-0 agent doing ordinary recursive reads +# (`grep -r recce .`, `find . -type f -exec cat {} \;`, +# `cat .*/settings.json`) would otherwise reach them. The +# `--settings ` flow keeps Recce-shaped files structurally +# out of the agent's view. +HOOK_PATH="${TIER_DIR}/claude-overlay/hooks/deny-tier-0.py" +RENDERED_DIR="$(mktemp -d)" +RENDERED_SETTINGS="${RENDERED_DIR}/tier-0.json" +python3 -c " +from pathlib import Path +tpl = Path('${TIER_DIR}/claude-overlay/settings.json').read_text() +Path('${RENDERED_SETTINGS}').write_text(tpl.replace('\${RUNNER_HOOK_PATH}', '${HOOK_PATH}')) +" # 3. Neuter user-level Claude Code settings so a stray ~/.claude/settings.json # can't widen the sandbox. (Skip this step at your own risk.) @@ -46,20 +59,24 @@ unset SNOWFLAKE_USER SNOWFLAKE_PASSWORD SNOWFLAKE_ACCOUNT unset POSTGRES_PASSWORD BIGQUERY_PROJECT # (extend to match your environment; goal is no live credential in env) -# 5. Run the agent with cwd at the per-fixture working tree. +# 5. Run the agent with cwd at the per-fixture working tree, loading the +# rendered Tier-0 settings via --settings. cd "${FIXTURE_DIR}" -claude "" +claude --settings "${RENDERED_SETTINGS}" \ + "" ``` -The Tier-0 `.claude/settings.json` declares `permissions.deny` rules for the documented Recce MCP namespaces and dbt/SQL-client Bash patterns, and registers a `PreToolUse` hook (`deny-tier-0.py`). The hook is the **load-bearing layer** — `permissions.deny` is conceded to be unreliable (Claude Code [issue #6699](https://github.com/anthropics/claude-code/issues/6699)) and uses shell-glob matching whose surface differs from the hook's AST-based check. Treat `permissions.deny` as documentation; trust the hook. See `runner-configs/README.md` for the divergence details. +The Tier-0 settings template declares `permissions.deny` rules for the documented Recce MCP namespaces and dbt/SQL-client Bash patterns, and registers a `PreToolUse` hook (`deny-tier-0.py`). The hook is the **load-bearing layer** — `permissions.deny` is conceded to be unreliable (Claude Code [issue #6699](https://github.com/anthropics/claude-code/issues/6699)) and uses shell-glob matching whose surface differs from the hook's AST-based check. Treat `permissions.deny` as documentation; trust the hook. See `runner-configs/README.md` for the divergence details. The hook uses `bashlex` (a real Bash AST parser) so nested `$()`, ANSI-C `$'...'`, parameter expansion `${a:-default}`, Bash keywords (`!`, `coproc`), command modifiers (`command`, `builtin`), and exec wrappers (`xargs`, `find -exec`, `time`, `nohup`, …) are all handled by their AST shape rather than ad-hoc regex. +**Why not copy `claude-overlay/` into `${FIXTURE_DIR}/.claude/`?** Earlier iterations of this recipe did exactly that. The overlay file itself names Recce vocabulary (`mcp__recce__*`, `Bash(recce *)`, `RECCE_API_TOKEN`, the hook source mentions `recce`), and a Tier-0 agent doing a non-adversarial recursive search of cwd would reach those tokens through bypass shapes the AST hook cannot pre-empt (glob expansion happens *after* the hook reads the literal argument). Loading via `--settings ` and pointing the hook `command` at an absolute path outside cwd makes the leak structurally impossible. Verified on Claude Code v2.1.160 that hooks fire and `permissions.deny` holds under `--dangerously-skip-permissions` with zero `.claude/` in cwd. + ### Tier 1 Same shape as Tier 0, with three differences: -1. `TIER_DIR` points to `runner-configs/claude-code/tier-1/`. +1. `TIER_DIR` points to `runner-configs/claude-code/tier-1/` and `HOOK_PATH` to `tier-1/claude-overlay/hooks/deny-tier-1.py`. 2. Leave the **dev-environment** warehouse credentials in the shell (or load them from a secrets file). Do **not** also export base/prod credentials — Tier 1 is single-env. 3. Recce MCP must be reachable to Claude Code (typically already true if `/recce-verify` works locally). The Tier-1 settings allow Recce MCP tools and Recce CLI; only dbt-regen and direct SQL clients stay denied. @@ -84,7 +101,7 @@ The runner therefore **must** launch the agent with cwd set to the per-fixture w This complements the cwd separation: -* Claude Code: the project-level `.claude/settings.json` lives inside the per-fixture worktree, so it travels with the agent's cwd. **Spoiler-path protection comes from cwd alone** — the PreToolUse hook does not gate `Read`/`Grep`/`Glob` and the Tier-0 Bash allowlist includes `cat`. If the runner mistakenly launches `claude` from the eval host repo root, the agent can read `RUBRIC.md` and the per-fixture spoiler README. The recipe above (step 5: `cd "${FIXTURE_DIR}"`) is therefore not optional. +* Claude Code: the settings.json and PreToolUse hook live **outside** the per-fixture worktree (loaded via `claude --settings `), keeping the agent's cwd a pristine dbt project with no Recce-shaped files at all. **Spoiler-path protection comes from cwd alone** — the PreToolUse hook does not gate `Read`/`Grep`/`Glob` and the Tier-0 Bash allowlist includes `cat`. If the runner mistakenly launches `claude` from the eval host repo root, the agent can read `RUBRIC.md` and the per-fixture spoiler README. The recipe above (step 5: `cd "${FIXTURE_DIR}"`) is therefore not optional. * Codex: the process sandbox is anchored on cwd; absolute paths outside cwd require approval at Tier-0 read-only mode and writes are uniformly denied. ## Recording in the baseline diff --git a/evals/agent-blind-spots/runner-configs/README.md b/evals/agent-blind-spots/runner-configs/README.md index 8151bcd..46e9345 100644 --- a/evals/agent-blind-spots/runner-configs/README.md +++ b/evals/agent-blind-spots/runner-configs/README.md @@ -9,12 +9,20 @@ runner-configs/ ├── README.md ← (this file) ├── claude-code/ │ ├── tier-0/ -│ │ └── claude-overlay/ ← copied to /.claude/ by the runner -│ │ ├── settings.json ← permissions.deny (documentation) + PreToolUse hook +│ │ └── claude-overlay/ ← template; the runner renders settings.json +│ │ │ to a per-cell temp path and loads it via +│ │ │ `claude --settings `. NEVER +│ │ │ copied into the fixture cwd — see below. +│ │ ├── settings.json ← permissions.deny (documentation) + PreToolUse +│ │ │ hook; `command` field uses `${RUNNER_HOOK_PATH}` +│ │ │ placeholder substituted by the runner with an +│ │ │ absolute path to the hook. │ │ └── hooks/deny-tier-0.py ← exit-2 block, positive Bash allowlist │ └── tier-1/ │ └── claude-overlay/ -│ ├── settings.json ← narrower deny (Recce allowed; dbt regen + SQL clients denied) +│ ├── settings.json ← narrower deny (Recce allowed; dbt regen + SQL +│ │ clients denied). Same `${RUNNER_HOOK_PATH}` +│ │ placeholder pattern as Tier-0. │ └── hooks/deny-tier-1.py ← exit-2 block, tokenised denylist └── codex/ ├── tier-0/ @@ -25,7 +33,7 @@ runner-configs/ └── config.toml ← template mcp_servers.recce entry to fill in ``` -The template directory is named `claude-overlay/` rather than `.claude/` so it isn't swallowed by the repo's `.claude/` gitignore rule. The eval runner renames it to `.claude/` when it copies the overlay into the per-fixture working tree (see Quick start below). +The template directory is named `claude-overlay/` rather than `.claude/` so it isn't swallowed by the repo's `.claude/` gitignore rule. **It is NOT copied into the fixture working tree.** Earlier iterations of the recipe did exactly that, but the overlay file and the hook source both name Recce vocabulary (`mcp__recce__*`, `Bash(recce *)`, `RECCE_API_TOKEN`, recce-shaped tokens in the hook code), and a Tier-0 agent doing non-adversarial recursive reads (`grep -r recce .`, `find . -type f -exec cat {} \;`, `cat .*/settings.json`) would reach those tokens through bypass shapes the AST hook cannot pre-empt (glob expansion happens after the hook reads the literal argument). The eval runner now renders the template settings.json to a path *outside* the agent's cwd and loads it via `claude --settings `, leaving the agent's cwd a pristine dbt project with no Recce-shaped files at all. Verified on Claude Code v2.1.160. ## Quick start @@ -37,13 +45,23 @@ WT_ROOT="$(git rev-parse --show-toplevel)" FIXTURE_DIR="${WT_ROOT}/evals/agent-blind-spots/.tmp/sources/${SLUG}" TIER_DIR="${WT_ROOT}/evals/agent-blind-spots/runner-configs/claude-code/tier-0" -cp -r "${TIER_DIR}/claude-overlay" "${FIXTURE_DIR}/.claude" +# Render the template with an absolute hook path (both file paths live +# OUTSIDE the fixture cwd by design). +HOOK_PATH="${TIER_DIR}/claude-overlay/hooks/deny-tier-0.py" +RENDERED_DIR="$(mktemp -d)" +RENDERED_SETTINGS="${RENDERED_DIR}/tier-0.json" +python3 -c " +from pathlib import Path +tpl = Path('${TIER_DIR}/claude-overlay/settings.json').read_text() +Path('${RENDERED_SETTINGS}').write_text(tpl.replace('\${RUNNER_HOOK_PATH}', '${HOOK_PATH}')) +" # Optional but recommended — neuter user-level settings for this run: export CLAUDE_CONFIG_DIR="$(mktemp -d)" cd "${FIXTURE_DIR}" -claude "" +claude --settings "${RENDERED_SETTINGS}" \ + "" ``` **Codex, Tier 0:** see [`codex/tier-0/README.md`](codex/tier-0/README.md). diff --git a/evals/agent-blind-spots/runner-configs/claude-code/tier-0/claude-overlay/settings.json b/evals/agent-blind-spots/runner-configs/claude-code/tier-0/claude-overlay/settings.json index d653445..61275d2 100644 --- a/evals/agent-blind-spots/runner-configs/claude-code/tier-0/claude-overlay/settings.json +++ b/evals/agent-blind-spots/runner-configs/claude-code/tier-0/claude-overlay/settings.json @@ -1,5 +1,5 @@ { - "$comment": "Tier-0 sandbox profile for the /recce-verify v1 eval. Drop this .claude/ directory into the per-fixture working tree (.tmp/sources//) before invoking Claude Code. See ../../../ENFORCEMENT.md for the runner recipe. The PreToolUse hook is the load-bearing layer; permissions.deny below is documentation-aligned but does not cover every bypass the hook does — see runner-configs/README.md. Note: deny rules are enforced even with --dangerously-skip-permissions; only the interactive prompt is bypassed by that flag.", + "$comment": "Tier-0 sandbox profile template for the /recce-verify v1 eval. The runner loads this file via `claude --settings ` rather than copying it into the per-fixture cwd, so the agent's cwd is a pristine dbt project with no Recce-shaped files. The `command` field below uses a runner-hook-path placeholder (dollar-brace RUNNER_HOOK_PATH), which the runner substitutes with an absolute path to deny-tier-0.py outside the agent's cwd before invocation. See ../../../ENFORCEMENT.md for the recipe. The PreToolUse hook is the load-bearing layer; permissions.deny below is documentation-aligned but does not cover every bypass the hook does — see runner-configs/README.md. Note: deny rules are enforced even with --dangerously-skip-permissions; only the interactive prompt is bypassed by that flag.", "permissions": { "deny": [ "mcp__recce__*", @@ -34,7 +34,7 @@ "hooks": [ { "type": "command", - "command": "python3 \"$CLAUDE_PROJECT_DIR/.claude/hooks/deny-tier-0.py\"", + "command": "python3 \"${RUNNER_HOOK_PATH}\"", "timeout": 5 } ] diff --git a/evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/settings.json b/evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/settings.json index d358e7c..d65698a 100644 --- a/evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/settings.json +++ b/evals/agent-blind-spots/runner-configs/claude-code/tier-1/claude-overlay/settings.json @@ -1,5 +1,5 @@ { - "$comment": "Tier-1 sandbox profile for the /recce-verify v1 eval. Tier 1 is Tier 0 plus Recce CLI, Recce MCP, and single-env warehouse credentials (read-only on the dev environment). Drop this .claude/ directory into the per-fixture working tree (.tmp/sources//) before invoking Claude Code. See ../../../ENFORCEMENT.md for the runner recipe. Note: deny rules are enforced even with --dangerously-skip-permissions.", + "$comment": "Tier-1 sandbox profile template for the /recce-verify v1 eval. Tier 1 is Tier 0 plus Recce CLI, Recce MCP, and single-env warehouse credentials (read-only on the dev environment). The runner loads this file via `claude --settings ` rather than copying it into the per-fixture cwd, so the agent's cwd is a pristine dbt project with no Recce-shaped files. The `command` field below uses a runner-hook-path placeholder (dollar-brace RUNNER_HOOK_PATH), which the runner substitutes with an absolute path to deny-tier-1.py outside the agent's cwd before invocation. See ../../../ENFORCEMENT.md for the recipe. Note: deny rules are enforced even with --dangerously-skip-permissions.", "permissions": { "deny": [ "Bash(dbt run*)", @@ -29,7 +29,7 @@ "hooks": [ { "type": "command", - "command": "python3 \"$CLAUDE_PROJECT_DIR/.claude/hooks/deny-tier-1.py\"", + "command": "python3 \"${RUNNER_HOOK_PATH}\"", "timeout": 5 } ] diff --git a/evals/agent-blind-spots/spike-driver/driver.py b/evals/agent-blind-spots/spike-driver/driver.py index 6199f45..b3e7ea3 100644 --- a/evals/agent-blind-spots/spike-driver/driver.py +++ b/evals/agent-blind-spots/spike-driver/driver.py @@ -208,13 +208,50 @@ def stage_inputs(fixture_dir: Path, fixture_id: str) -> None: ) -def run_claude(fixture_dir: Path, tier: int, run_dir: Path, prompt: str, model: str | None) -> tuple[Path, int]: +def render_claude_settings(tier: int, run_dir: Path, fixture_name: str) -> Path: + """Render the Tier-N settings.json template to a per-cell temp location. + + Andy's iter-8 BLOCKER (Tier-0 overlay leak): a copied-into-cwd + `.claude/{settings.json, hooks/deny-tier-N.py}` is reachable by a + non-adversarial agent through ordinary recursive reads + (`grep -r recce .`, `find . -type f -exec cat {} \\;`, `cat .*/settings.json`) + that the Bash AST hook cannot see ahead of glob expansion. Pass 0 + token-matching is whack-a-mole; the root cause is structural — + Recce-shaped files in the agent's cwd. + + The fix (verified by Andy on CC v2.1.160): load the settings via + `claude --settings ` from *outside* cwd, with the hook + `command` field pointing to an absolute path also outside cwd. This + keeps the agent's cwd a pristine dbt project with no Recce-shaped + files whatsoever, making the leak structurally impossible. + + The template at `runner-configs/claude-code/tier-{tier}/claude-overlay/ + settings.json` declares the hook command as `python3 + "${RUNNER_HOOK_PATH}"`; we substitute that placeholder with the + absolute path to the corresponding `deny-tier-{tier}.py` (which lives + in the runner-configs tree, also outside cwd) and write the rendered + file under `run_dir/_settings/_t.json`. The runner + passes that rendered path to `claude --settings`. + """ tier_dir = RUNNER_CONFIGS / "claude-code" / f"tier-{tier}" - overlay_src = tier_dir / "claude-overlay" - overlay_dst = fixture_dir / ".claude" - if overlay_dst.exists(): - shutil.rmtree(overlay_dst) - shutil.copytree(overlay_src, overlay_dst) + template_path = tier_dir / "claude-overlay" / "settings.json" + hook_path = (tier_dir / "claude-overlay" / "hooks" / f"deny-tier-{tier}.py").resolve() + + template_text = template_path.read_text() + rendered_text = template_text.replace("${RUNNER_HOOK_PATH}", str(hook_path)) + + settings_dir = run_dir / "_settings" + settings_dir.mkdir(parents=True, exist_ok=True) + rendered_path = settings_dir / f"{fixture_name}_t{tier}.json" + rendered_path.write_text(rendered_text) + return rendered_path + + +def run_claude(fixture_dir: Path, tier: int, run_dir: Path, prompt: str, model: str | None) -> tuple[Path, int]: + # No overlay copy. The settings.json + hook live outside cwd; cwd + # stays a pristine dbt project with `_eval_inputs/` symlink + project + # files only. See render_claude_settings() for the rationale. + settings_path = render_claude_settings(tier, run_dir, fixture_dir.name) transcript_path = run_dir / "transcripts" / f"{fixture_dir.name}_claude_t{tier}.txt" transcript_path.parent.mkdir(parents=True, exist_ok=True) @@ -222,18 +259,22 @@ def run_claude(fixture_dir: Path, tier: int, run_dir: Path, prompt: str, model: # NOTE: ENFORCEMENT.md recipe step 3 (CLAUDE_CONFIG_DIR=$(mktemp -d)) is # *intentionally not applied here* — neutering ~/.claude/ also strips the # auth state, which breaks unattended runs. The load-bearing enforcement - # for Tier-0 is the project-level .claude/settings.json overlay (stamped - # above) + the PreToolUse hook (deny-tier-N.py); a stray user-level - # `permissions.allow` cannot bypass an exit-2 hook. For paranoid mode, - # set RECCE_EVAL_STRICT_CONFIG=1 to override CLAUDE_CONFIG_DIR; the cell - # will fail with "Not logged in" unless auth is preseeded under that dir. + # for Tier-0 is the `--settings` override + the PreToolUse hook + # (deny-tier-N.py); a stray user-level `permissions.allow` cannot bypass + # an exit-2 hook. For paranoid mode, set RECCE_EVAL_STRICT_CONFIG=1 to + # override CLAUDE_CONFIG_DIR; the cell will fail with "Not logged in" + # unless auth is preseeded under that dir. env = scrub_env() if os.environ.get("RECCE_EVAL_STRICT_CONFIG"): cfg_dir = run_dir / "_claude_cfg" / f"{fixture_dir.name}_t{tier}" cfg_dir.mkdir(parents=True, exist_ok=True) env["CLAUDE_CONFIG_DIR"] = str(cfg_dir) - cmd = ["claude", "--print", "--dangerously-skip-permissions"] + cmd = [ + "claude", "--print", + "--settings", str(settings_path), + "--dangerously-skip-permissions", + ] if model: cmd += ["--model", model] cmd.append(prompt) @@ -324,11 +365,13 @@ def run_cell(cell: Cell, run_dir: Path, model: str | None) -> None: if not cli_available(cell.agent): cell.error = f"{cell.agent} CLI not found on PATH; skipping" return - # Reset the fixture worktree before staging inputs and copying the - # overlay, so each cell starts from the same base — no Claude write - # from a prior cell bleeds across. Must run *before* stage_inputs() - # (whose `_eval_inputs/` symlinks are untracked and would be removed - # by `git clean -fdx`) and *before* run_claude()'s overlay copy. + # Reset the fixture worktree before staging inputs, so each cell + # starts from the same base — no Claude write from a prior cell + # bleeds across. Must run *before* stage_inputs() (whose + # `_eval_inputs/` symlinks are untracked and would be removed by + # `git clean -fdx`). The Claude Code settings are loaded via + # `--settings ` outside cwd (see render_claude_settings), + # so the fixture cwd stays Recce-shaped-free. try: reset_fixture_dir(fixture_dir) except subprocess.CalledProcessError as e: