Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 48 additions & 0 deletions .changeset/verify-indirection.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
---
"@stainless-code/verdict": patch
---

The verify pre-exec gate refuses a command word it cannot read: `E=rm; $E -rf
./target` and `E=curl; $E -o ./f http://host/x` passed every arm because the
tool word sat in the variable, and the resolved command then ran. `verify` now
refuses the shape as `verify-indirection`, quoting the spelling it saw and
naming the remedy (write the tool word plainly; `verify --sandbox` refuses it
too, because the gate runs before the sandbox wrapper). The braced (`${E}`), quoted (`"$E"`), `export`ed,
assignment-prefix, loop (`for E in curl; do $E ...`), concatenated (`$A$B`),
and own-line (`E=rm\n$E -rf ./target`) spellings close with it, and so do the
wrappers that run a later word (`env -u HOME $E`, `timeout -s KILL 5 $E`,
`timeout 5s $E`, `xargs -I {} $E`, `command`, `nice`, `time`, `nohup`,
`stdbuf`, `exec`, `then`, `do`). Variables in argument position still execute
(`echo $USER`, `PORT=9 curl http://127.0.0.1:9/$PORT/f`, `$BIN/tool --version`),
so the false-positive boundary is unchanged for a command written plainly.

The gate does not resolve variables — the value can come from the command's own
environment, or from a later or conditional assignment — so it refuses rather
than guessing, and a variable holding a whole command line (`sh "$V"`) stays a
documented residual of the tripwire. A wrapper's own option grammar decides
which later word bash runs, so after `builtin`/`command`/`env`/`exec`/`nice`/
`nohup`/`stdbuf`/`time`/`timeout`/`xargs` any variable in that command refuses:
that is the deliberate cost of not modelling each wrapper's flags, so
`timeout 300 make test` stays clean while `timeout 300 make test $TARGET`
refuses. A shell carrying an inline command (`sh -c "$E pwned"`, `bash -ec`, `sh -cu`)
has that command string judged as its own command position, so
`sh -c "echo $E"` stays clean; `eval` and `sudo` refuse earlier as
`verify-destructive`.
The three arms that read a command word — glob fragments, indirection and
command position — now read the segmented text, which splits at newlines, so a
later line is a command position of its own: `echo ok\nr? -rf x` and
`echo ok\n$(echo rm) -rf x` refuse instead of joining the first segment. (The
whole-string greps already saw every line.) A `\`-newline continuation is one
command and joins before segmentation, so `echo \` + newline + `"$USER"` stays
clean. Leading redirections (`</dev/null`, `2>&1`, `>> f`) are skipped to reach
the command word; a quoted word stays one word, so `LABEL="hello world" $E ok`
and `>"out file" $E` refuse rather than putting `world` in command position; a
substitution body (`$($E pwned)`, backticks, `cat <($E pwned)`) is judged as its
own command position — quote-aware, so `$($E "(")` closes where bash closes it —
while `$((…))` arithmetic and single-quoted literals (`awk '{print $($NF)}'`)
are left alone.

`SECURITY.md` now says the same thing
the README and the manual say: the refusals are a tripwire, `verify --sandbox`
is the boundary, and a spelling outside the gate's reach is a gate gap to arm,
not a vulnerability.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,7 @@ On the other three hosts, `record`, `report`, `show`, `next`, `triage`, and `thr

## Security

- **The skill runs with your privileges.** `verify` runs the agent's `verifyCmd` through `bash -c` with a 300 second limit. A pre-exec gate refuses destructive, networked, credential-reading, and repo-escaping commands before anything spawns. The gate is a tripwire, not a sandbox. For a hostile PR, `verify --sandbox` enforces the sandbox: `bubblewrap` on Linux (mounts, own PID namespace, no network) and `sandbox-exec` on macOS (no network); elsewhere follow the [recipe](docs/manual.md#hostile-pr-sandbox-recipe).
- **The skill runs with your privileges.** `verify` runs the agent's `verifyCmd` through `bash -c` with a 300 second limit. A pre-exec gate refuses destructive, networked, credential-reading, and repo-escaping commands before anything spawns, and refuses a command word built from a variable (`E=curl; $E -o ./f http://host/x`) as `verify-indirection`. The gate is a tripwire, not a sandbox. For a hostile PR, `verify --sandbox` enforces the sandbox: `bubblewrap` on Linux (mounts, own PID namespace, no network) and `sandbox-exec` on macOS (no network); elsewhere follow the [recipe](docs/manual.md#hostile-pr-sandbox-recipe).
- **Review controls come from the base commit.** A PR cannot edit the panel, budget, or knowledge-base rules it is judged by. See [Review controls are base-owned](docs/manual.md#review-controls-are-base-owned).
- **Diffs and comments are untrusted input.** The prompts treat them as evidence, never as instructions.
- **Secrets stay local.** `GH_TOKEN` never leaves your machine, and provider tokens go only to their own hosts. Nothing phones home: no telemetry, no background updater.
Expand Down
26 changes: 20 additions & 6 deletions SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,11 +30,22 @@ before reporting: many findings are already fixed on the latest tag.

In scope (highest priority first):

- The `verify` gate: `verify` executes the agent's claimed proof
(`verifyCmd` via `bash -c`, 300s watchdog, tail recorded into the
patch record). Its pre-exec refusals (destructive, networked,
credential-reading, repo-escaping commands) are a security boundary;
any bypass is a vulnerability.
- `verify --sandbox`, the enforced boundary: mount and namespace isolation
under `bubblewrap` on Linux, network denial under `sandbox-exec` on macOS,
the credential-path hiding, and the `sandbox-unavailable` refusal that never
degrades to an unwrapped run. An escape from that sandbox, or a run that
records `"sandboxed": true` without the enforcement, is a vulnerability.
- The `verify` gate's own contract: `verify` executes the agent's claimed
proof (`verifyCmd` via `bash -c`, 300s watchdog, tail recorded into the
patch record). The pre-exec refusals are a **tripwire, not a boundary**: the
gate refuses the spellings it knows, and a `verifyCmd` spelling it does not
know executes inside whatever boundary you gave it — that ceiling is
documented in the [manual](docs/manual.md#hostile-pr-sandbox-recipe) and a
hit on it is a gate gap to report and arm, not a vulnerability. What is in
scope is the tripwire's own promise: a command that runs after the gate
refused it, a spawn that happens before the refusal, a refusal that names
the wrong reason, or a spelling the docs and the fuzz corpus claim refuses
while it executes.
- The installers (`install.sh`, `install.ps1`): archive checksum, cosign
verification pinned to the release's published build commit and to a
push-triggered run, atomic replace, no execution before verification. A
Expand All @@ -59,7 +70,10 @@ In scope (highest priority first):
head bytes steer a review without the recorded
`--allow-head-controls` opt-out is a vulnerability.

Out of scope: documentation typos (open a regular issue), missing
Out of scope: a `verifyCmd` spelling the string gate does not know (the
documented tripwire ceiling above — send it, it gets armed and added to
`evals/fixtures/verify-fuzz-corpus.tsv`), documentation typos (open a regular
issue), missing
hardening wishes with no exploit path (open a feature request), and
vulnerabilities in `apps/docs` build-time dependencies that never
reach the published static site (still reportable, assessed by
Expand Down
1 change: 1 addition & 0 deletions clients/ts/src/refusal-codes.gen.ts
Original file line number Diff line number Diff line change
Expand Up @@ -226,6 +226,7 @@ export type RefusalCode =
| "verify-credential-read"
| "verify-destructive"
| "verify-failed"
| "verify-indirection"
| "verify-networked"
| "verify-repo-escape"
| "window-too-large"
Expand Down
3 changes: 2 additions & 1 deletion cmd/verdict/usage.go
Original file line number Diff line number Diff line change
Expand Up @@ -205,7 +205,8 @@ func usageText() string {
" Runs the agent's claimed proof — exit code becomes machine fact. No GitHub I/O.\n" +
" Destructive, networked, credential-reading, and repo-escaping commands\n" +
" refuse before execution as verify-destructive, verify-networked,\n" +
" verify-credential-read, and verify-repo-escape.\n" +
" verify-credential-read, and verify-repo-escape; a command word built from\n" +
" a variable refuses as verify-indirection.\n" +
" doctor [--provider github|gitlab|gitcafe|forgejo|bitbucket|azure] [--json]\n" +
" preflight the configured host: its CLI and auth, plus .verdict storage\n" +
" writability, hard-fail the run; scopes, git, ps, and API reachability warn\n" +
Expand Down
16 changes: 13 additions & 3 deletions docs/manual.md
Original file line number Diff line number Diff line change
Expand Up @@ -3147,7 +3147,8 @@ step — not the whole review — inside a throwaway sandbox so a malicious
command finds no credentials and no route out. The `verify` verb already
refuses hostile shapes before spawning anything — destructive
(`verify-destructive`), networked (`verify-networked`), credential-reading
(`verify-credential-read`), repo-escaping (`verify-repo-escape`) — each
(`verify-credential-read`), repo-escaping (`verify-repo-escape`), and a
command word built from a variable (`verify-indirection`) — each
typed refusal quoting the matched fragment with a `try:` rewrite hint, and
the refusal leaves `patch.json` and `.verify.out` untouched. The sandbox
below is the second layer for what the static gate cannot see (a plain
Expand All @@ -3156,7 +3157,15 @@ replacement.

The static gate is a tripwire, not a boundary. It matches the spellings it
knows, on one command string at a time, and a shape it has not learned
executes. The boundary is the enforced sandbox below: `verify --sandbox`
executes. It does not resolve variables, so it refuses a command word it
cannot read rather than guessing what the variable holds — including on a
later line of a multi-line `verifyCmd`, after a leading redirection, inside a
substitution (`$(...)`, backticks, `<(...)`), and behind a wrapper (`builtin`,
`command`, `env`, `exec`, `nice`, `nohup`, `stdbuf`, `time`, `timeout`,
`xargs`, or a shell carrying `-c`) whose own option grammar decides which word
runs. The boundary is
the enforced sandbox below, which runs after this gate — a refused spelling
refuses with or without `--sandbox`: `verify --sandbox`
runs the command under `bubblewrap` on Linux (the full mount and namespace
shape) and under `sandbox-exec` on macOS (no network). A host that can
enforce neither refuses `sandbox-unavailable` and runs nothing, so
Expand Down Expand Up @@ -3283,7 +3292,8 @@ through any of these reopens the mandate discussion.
`GEM_HOME`, `CARGO_HOME`, …) including innocent-command hijack;
- env-directed fetch destinations (index/registry envs; registry _names_ have no URL
to key on);
- dynamic construction (`$'…'`-hex, base64, variable indirection, alias-`!`);
- dynamic construction (`$'…'`-hex, base64, alias-`!`, a variable holding a whole
command line — a bare variable _command word_ refuses as `verify-indirection`);
- interpreter-mediated execution (`python|perl|ruby|node|php …`, `curl | python3`);
- fetch-then-exec splits across the `/tmp` carve-out (I3);
- TOCTOU/meaning-change (symlink swaps, concurrent-run shared state);
Expand Down
33 changes: 33 additions & 0 deletions docs/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -1404,6 +1404,39 @@ Cause: the verification command (or the receipt/head read around it) failed —

Fix: fix the command or the evidence named above, then re-run `verify`.

## `verify-indirection`

Cause: `verifyCmd` invokes a command word the gate cannot read — a plain
variable (`E=curl; $E -o ./f http://host/x`), the braced (`${E}`) or quoted
(`"$E"`) spelling, an `export`, an assignment prefix, a loop variable, the
same spelling on its own line, after a leading redirection (`</dev/null $E`,
`2>&1 $E`, `>> f $E`), or inside a substitution (`echo $($E pwned)`, backticks,
`cat <($E pwned)`), whose body is judged as its own command — quotes included,
so `$($E "(")` is read the way bash reads it. The value may
come from the command's own environment, so the gate refuses the shape instead
of guessing what the variable holds.

A wrapper's own option grammar decides which later word bash runs, so after
`builtin`, `command`, `env`, `exec`, `nice`, `nohup`, `stdbuf`, `time`,
`timeout`, or `xargs` any variable in that command refuses: `timeout 300 make
test` stays clean, `timeout 300 make test $TARGET` does not. A shell counts
the same way when it carries an inline command — any short-flag cluster with a
`c` in it (`sh -c`, `bash -c`, `bash -ec`, `sh -cu`): the inline command string
is judged as its own command, so `sh -c "$E pwned"` refuses while
`sh -c "echo $E"` stays clean. `eval` and `sudo` refuse earlier, as
`verify-destructive`, whatever their arguments. A quoted word stays one word:
`LABEL="hello world" $E ok` and `>"out file" $E` refuse, because quote removal
would otherwise put `world` in command position.

The gate is a tripwire, not a boundary, and it runs before the sandbox
wrapper: `verify --sandbox` refuses an indirection the same way, so the remedy
is to write the command plainly. A variable holding a whole command line
(`sh "$V"`, a file written and then run) is a shape the gate has not learned.

Fix: write the tool word plainly (`E=curl; $E -o ./f http://host/x` becomes
`curl -o ./f http://host/x`, or the test runner directly) and keep variables
for arguments of a command written plainly, then re-run.

## `verify-networked`

Cause: `verifyCmd` fetches the network or pipes to a shell.
Expand Down
7 changes: 4 additions & 3 deletions evals/cases/usage-doc-gates.sh
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ if [ "$EVAL_STATUS" -ne 0 ]; then
bad "--help exits $EVAL_STATUS"
fi

for code in verify-destructive verify-networked verify-credential-read verify-repo-escape; do
for code in verify-destructive verify-networked verify-credential-read verify-repo-escape verify-indirection; do
if grep -q "$code" "$EVAL_OUT"; then
good "verify usage names $code"
else
Expand All @@ -27,8 +27,9 @@ fi
if grep -q "verify-destructive" "$EVAL_ROOT/docs/manual.md" \
&& grep -q "verify-networked" "$EVAL_ROOT/docs/manual.md" \
&& grep -q "verify-credential-read" "$EVAL_ROOT/docs/manual.md" \
&& grep -q "verify-repo-escape" "$EVAL_ROOT/docs/manual.md"; then
good "manual names the four verify refusal codes"
&& grep -q "verify-repo-escape" "$EVAL_ROOT/docs/manual.md" \
&& grep -q "verify-indirection" "$EVAL_ROOT/docs/manual.md"; then
good "manual names the five verify refusal codes"
else
bad "manual misses a verify refusal code"
fi
Expand Down
54 changes: 53 additions & 1 deletion evals/fixtures/verify-fuzz-corpus.tsv
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@
#
# id stable case id (F-R* refuse, F-X* expected-fail, F-P* pass).
# expect what the gate must do: one of verify-destructive,
# verify-networked, verify-credential-read, verify-repo-escape
# verify-networked, verify-credential-read, verify-repo-escape,
# verify-indirection
# (refuse with exactly this typed code), or PASS (execute).
# For xfail rows expect is aspirational: the code the gate SHOULD
# refuse with once the gap is closed.
Expand Down Expand Up @@ -66,6 +67,36 @@ F-R34 verify-destructive strict tar -Itouch-fz -cf /tmp/verify-fuzz-attached.tar
F-R35 verify-destructive strict tar -cIf touch-fz /tmp/verify-fuzz-cluster.tar fz-notes.txt
F-R36 verify-networked strict wget -i/tmp/verify-fuzz-wgeturls http://127.0.0.1:9/f
F-R37 verify-networked strict wget -qi /tmp/verify-fuzz-wgeturls http://127.0.0.1:9/f
F-R38 verify-indirection strict E=rm; $E -rf ./fz-target
F-R39 verify-indirection strict E=curl; $E -o ./f http://127.0.0.1:9/x
F-R40 verify-indirection strict E=curl; ${E} -o ./f http://127.0.0.1:9/x
F-R41 verify-indirection strict E=curl; "$E" -o ./f http://127.0.0.1:9/x
F-R42 verify-indirection strict export E=curl; $E -o ./f http://127.0.0.1:9/x
F-R43 verify-indirection strict A=curl B=1; $A -o ./f http://127.0.0.1:9/x
F-R44 verify-indirection strict for E in curl; do $E -o ./f http://127.0.0.1:9/x; done
F-R45 verify-indirection strict A=cu B=rl; $A$B -o ./f http://127.0.0.1:9/x
F-R46 verify-indirection strict env -u HOME $E -rf ./fz-target
F-R47 verify-indirection strict timeout 5s $E -rf ./fz-target
F-R48 verify-indirection strict timeout -s KILL 5 $E -rf ./fz-target
F-R49 verify-indirection strict xargs -I {} $E -rf ./fz-target
F-R50 verify-indirection strict command $E -rf ./fz-target
F-R51 verify-indirection strict nice -n 5 $E -rf ./fz-target
F-R52 verify-indirection strict E=echo; </dev/null $E pwned
F-R53 verify-indirection strict E=echo; >/dev/null $E pwned
F-R54 verify-indirection strict E=echo; 2>&1 $E pwned
F-R55 verify-indirection strict E=echo; <<<x $E pwned
F-R56 verify-indirection strict E=echo; >> fz-notes.txt $E pwned
F-R57 verify-indirection strict E=echo; echo $($E pwned)
F-R58 verify-indirection strict E=echo; echo `$E pwned`
F-R59 verify-indirection strict E=echo; cat <($E pwned)
F-R60 verify-indirection strict E=echo; builtin $E pwned
F-R61 verify-indirection strict E=echo; sh -c "$E pwned"
F-R62 verify-destructive strict E=echo; eval $E pwned
F-R63 verify-destructive strict E=echo; sudo $E pwned
F-R64 verify-indirection strict E=echo; sh -cu "$E pwned"
F-R65 verify-indirection strict E=echo; echo $($E "(")
F-R66 verify-indirection strict E=printf; LABEL="hello world" $E ok
F-R67 verify-indirection strict E=printf; >"out file" $E ok
F-X01 verify-networked strict wget --remote-name http://127.0.0.1:9/f
F-X02 verify-networked strict curl -K /tmp/verify-fuzz-curlcfg http://127.0.0.1:9/
F-X03 verify-networked strict wget -i /tmp/verify-fuzz-wgeturls http://127.0.0.1:9/f
Expand Down Expand Up @@ -114,3 +145,24 @@ F-P26 PASS strict git --exec-path
F-P27 PASS strict tar -czf /tmp/verify-fuzz-ok.tar fz-notes.txt
F-P28 PASS strict tar -cf /tmp/verify-fuzz-plain.tar fz-notes.txt
F-P29 PASS strict wget -q --version
F-P30 PASS strict printf '%s' $FOO
F-P31 PASS strict $BIN/fz-tool --version
F-P32 PASS strict printf ok --prefix=$FOO/x
F-P33 PASS strict X=$(date); printf hi
F-P34 PASS strict timeout 300 printf ok
F-P35 PASS strict env CI=1 printf ok
F-P36 PASS strict xargs -n1 echo ok
F-P37 PASS strict { echo $USER; }
F-P38 PASS strict if true; then echo $USER; fi
F-P39 PASS strict cat </dev/null
F-P40 PASS strict echo $(date)
F-P41 PASS strict echo `date`
F-P42 PASS strict builtin echo ok
F-P43 PASS strict diff <(sort fz-notes.txt) <(sort fz-notes.txt)
F-P44 PASS strict bash ./fz-missing.sh $TARGET
F-P45 PASS strict echo $(( $X + 1 ))
F-P46 PASS strict echo $(printf "(%s)" x)
F-P47 PASS strict awk '{print $($NF)}' fz-notes.txt
F-P48 PASS strict LABEL="hello world" printf ok
F-P49 PASS strict echo "hello $USER"
F-P50 PASS strict sh -c 'echo $TARGET'
2 changes: 2 additions & 0 deletions internal/core/hint.go
Original file line number Diff line number Diff line change
Expand Up @@ -205,6 +205,8 @@ var hintRules = []HintRule{
{codes: []string{"verify-credential-read"}, hint: "try: rewrite verifyCmd without reading credential paths " +
"or piping env into the network, then re-run"},
{codes: []string{"verify-repo-escape"}, hint: "try: rewrite verifyCmd with repo-relative paths (/tmp for scratch), then re-run"},
{codes: []string{"verify-indirection"}, hint: "try: write the tool word plainly instead of " +
"$VAR, then re-run"},
{codes: []string{"sandbox-unavailable"}, hint: "try: on Linux install bubblewrap and allow unprivileged " +
"user namespaces; on macOS sandbox-exec ships with the OS — or re-run without --sandbox on a command you trust"},
{codes: []string{"agent-missing"}, hint: "try: install the agent CLI (or put it on PATH), or re-run with " +
Expand Down
1 change: 1 addition & 0 deletions internal/core/refusalcodes.go
Original file line number Diff line number Diff line change
Expand Up @@ -249,6 +249,7 @@ var refusalCodes = []string{
"verify-credential-read",
"verify-destructive",
"verify-failed",
"verify-indirection",
"verify-networked",
"verify-repo-escape",
"window-too-large",
Expand Down
1 change: 1 addition & 0 deletions internal/core/testdata/hints.golden
Original file line number Diff line number Diff line change
Expand Up @@ -222,6 +222,7 @@ validation-stale try: re-run the validation pass against the current head, or de
verify-credential-read try: rewrite verifyCmd without reading credential paths or piping env into the network, then re-run
verify-destructive try: rewrite verifyCmd to stay in-repo and non-destructive (run the test runner, not rm/mkfs/dd), then re-run
verify-failed try: address the cause above, then re-run the same command
verify-indirection try: write the tool word plainly instead of $VAR, then re-run
verify-networked try: rewrite verifyCmd without network fetches or pipe-to-shell (fetch fixtures first, run offline), then re-run
verify-repo-escape try: rewrite verifyCmd with repo-relative paths (/tmp for scratch), then re-run
window-too-large try: narrow --since (a shorter span or a later ref), then re-run
Expand Down
Loading
Loading