Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
d104296
run: probe agent identity before launch
SutuSebastian Nov 17, 2026
0ed83e8
run: add 32 headless agent rows
SutuSebastian Nov 17, 2026
98d8bec
docs: document new agents and identity probes
SutuSebastian Nov 17, 2026
bfe85b8
run: add headless flags to kode, neovate, qoder, qodercn
SutuSebastian Nov 17, 2026
a94421c
run: remove bob (clashing tool)
SutuSebastian Nov 17, 2026
8627c48
config: remove bob from builtin names
SutuSebastian Nov 17, 2026
d896a9c
run: remove clashing or non-distinctive agents
SutuSebastian Nov 17, 2026
ffcc002
Revert "run: remove clashing or non-distinctive agents"
SutuSebastian Nov 17, 2026
d44e751
Revert "config: remove bob from builtin names"
SutuSebastian Nov 17, 2026
9c7bc60
Revert "run: remove bob (clashing tool)"
SutuSebastian Nov 17, 2026
e571f43
run: use distinctive probe matches for clashing agents
SutuSebastian Nov 17, 2026
8ba82cb
run: test that clashing binaries need documented probe flags
SutuSebastian Nov 17, 2026
87d3697
eval: probe fx by its documented ultrafast flag
SutuSebastian Nov 17, 2026
4076646
docs: align agent flags and probe notes with the shipped rows
SutuSebastian Nov 17, 2026
f8c823c
test: satisfy lint in the clash probe fixtures
SutuSebastian Nov 17, 2026
d31730f
run: search the head of the identity probe capture
SutuSebastian Nov 17, 2026
25915a1
config: refuse an empty identity probe argument list
SutuSebastian Nov 17, 2026
88321b6
run: inject the identity probe budget for process fixtures
SutuSebastian Nov 17, 2026
76fb682
run: keep parent cancellation ahead of probe classification
SutuSebastian Nov 17, 2026
39259aa
run: record the identity probe in every repair round
SutuSebastian Nov 17, 2026
40edae3
config: keep the probe argument check within the lint budget
SutuSebastian Nov 17, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/sta-440-agents-2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@stainless-code/verdict": minor
---

`run --agent` now offers 32 more cited headless agents, including Cline and Junie after CLI-reference checks. Rows whose binary shares a name with another tool run a harmless identity probe before launch; a non-zero exit, a five-second timeout, or a missing match refuses the new `agent-mismatch` code before any agent starts, and `launcher.json` records `probe: {args, ok}` for a passing check. Operator agents can set `probe_args` and `probe_match` under settings.json `agents`, with `probe_match` required when `probe_args` is present.
1 change: 1 addition & 0 deletions clients/ts/src/refusal-codes.gen.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
export type RefusalCode =
| "adopt-failed"
| "agent-failed"
| "agent-mismatch"
| "agent-missing"
| "agent-record-failed"
| "agent-spawn-failed"
Expand Down
8 changes: 7 additions & 1 deletion cmd/verdict/usage.go
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,11 @@ func usageText() string {
" to stdout (human; progress moves to stderr) or data.prompt_dump (--json).\n" +
" No agent runs, no network beyond the fetch, no store writes beyond the bundle.\n" +
" run <owner/repo> <pr|local-name>\n" +
" --agent <agy|amp|claude|codex|copilot|crush|cursor|droid|gemini|goose|kiro|opencode|pi|qwen>\n" +
" --agent <adal|agy|amp|auggie|autohand|bob|claude|cline|codearts|codebuddy|\n" +
" codex|commandcode|continue|copilot|cortex|crush|cursor|dcode|devin|dexto|\n" +
" droid|forge|fx|gemini|goose|grok|hermes|jazz|junie|kilo|kimchi|kimi|kiro|\n" +
" kode|mcode|neovate|openclaw|opencode|pa|pi|qoder|qodercn|qwen|rovodev|\n" +
" vibe|zcode>\n" +
" [--recipe <slug>] [--stage <n>] [--timeout <secs, default 300>] [--rounds <n, default 1>]\n" +
" [--print] [-- <agent args>...] [--json]\n" +
" launch YOUR agent against an existing bundle: one process per round, one shot by default.\n" +
Expand All @@ -63,6 +67,8 @@ func usageText() string {
" Operators can define more names under the 'agents' key of their own settings\n" +
" file (XDG_CONFIG_HOME/verdict/settings.json); a repository config cannot.\n" +
" A flag that takes the prompt places it immediately before the directive.\n" +
" A binary whose name clashes with another tool answers a harmless help probe\n" +
" before launch; a mismatch refuses agent-mismatch without starting an agent.\n" +
" A non-zero agent exit is the process exit code (agent-failed); the watchdog\n" +
" timeout (agent-timeout), a missing binary (agent-missing), and a spawn failure\n" +
" (agent-spawn-failed) refuse typed with a hint. Provenance lands in\n" +
Expand Down
89 changes: 71 additions & 18 deletions docs/manual.md

Large diffs are not rendered by default.

6 changes: 6 additions & 0 deletions docs/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,12 @@ Cause: the launched agent exited non-zero.

Fix: read the recorded tail in `.verdict/state/<bundle>/launcher.json`, fix the invocation (the agent's own flags go after `--`), then re-run `verdict run`.

## `agent-mismatch`

Cause: the binary resolved for a row with a binary-name clash failed its identity probe (non-zero exit, timeout, or missing match text).

Fix: define the agent in settings.json `agents` with an absolute `binary` path, or re-run `verdict run` with `--agent` for the one you meant.

## `agent-missing`

Cause: the named agent binary is not on `PATH`.
Expand Down
1 change: 1 addition & 0 deletions evals/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -180,6 +180,7 @@ the build.
| `rule-exposure` | Per-rule `recalled_in`, `cited_in`, `exposure`, and `acceptance` on `learn --status` | Real CLI `learn --status` (text and `--json`) over seeded bundles: missing `kb-recall.json` reads `null`/`unmeasured`; two measured bundles recalling both rules where one rule is never cited and the other is cited once addressed read `recalled_in=2 cited_in=0 exposure=0 acceptance=n/a` and `recalled_in=2 cited_in=1 exposure=0.5 acceptance=1`. Zero host calls |
| `self-review` | Bounded local self-review recipe | Claude, Codex, and OpenCode stubs fix on round 2 or exhaust two rounds; assert exit, last gate, receipts, one spawn per round, zero host calls, and recipe unit checks. |
| `run-agent` | Shipped agent-launch recipe: one fresh review.json, then handoff | Committed stub agent on PATH as `claude` (no model, no network, no secret): `--print` renders the exact argv and writes nothing; a launch spawns one process in the repo root, is accepted only for one fresh `review.json`, and `launcher.json` records the redacted argv, exit, and freshness; `record` merges the launched review byte-identically; on a PR bundle `post --dry-run` accepts it at `COMMENT` with no `post.json`; stale, missing, failed, timed-out, and not-installed all refuse typed, and the docs recipe plus the CI step that proves the case ran are graded too |
| `run-agent-probe` | Identity probe for a clashing agent binary | Two `fx` stubs derive their behavior from argv: the agent answers `ask --help` and writes the directive review, the JSON viewer refuses `agent-mismatch`; `--print` spawns nothing, a passing launch records `probe.args`/`probe.ok` in `launcher.json`, and the mismatch starts no agent and writes no artifact |
| `run-custom-agent` | Operator-defined `run --agent` row from prefs | A stub generated in the sandbox is defined only under a sandboxed `XDG_CONFIG_HOME`, derives its behavior from the argv (subcommand, prompt flag immediately before the directive, review path in the directive), and `--print` matches the launch byte for byte; `launcher.json` marks `agent_source: "prefs"`, a missing binary carries the install hint from its definition, `unknown-agent` labels both sources, a repo-local config with `agents` does not define an agent, and a shipped-name collision refuses `bad-settings` |
| `run-recipe` | Declared review pipelines resolve from the base recipe control, never the PR's own edit | Real git base and head, `fetch-local --base` bundles, committed stub agent as `claude`: `--print` fills `data.recipe` (base version, sha256, four stages, its lens, the budget), named in the argv, writing nothing; an absent lens and an over-limit budget each warn once, then run; no `--recipe` resolves `builtin/default` and warns about nothing; the launch's `launcher.json.recipe` equals `data.recipe` exactly, the directive carries the pipeline never the description, `record` accepts the review; a head rewrite never reaches the plan; `--allow-head-controls` reads the head's copy instead; tampered snapshot bytes refuse `controls-stale`; `unknown-recipe` and `bad-recipe` refuse before any spawn; `--print --stage <n>` renders one stage's argv (the declared hint as the model argument, lens hints and caps, the `review.json` the stages chain through) plus its own refusals; zero host calls |
| `run-repair` | Structured-output repair loop in `run --rounds N` | Committed repair stub on PATH as `claude` (no model, no network, no secret): round 1 invalid then round 2 valid exits 0 with exactly 2 spawns, `rounds_to_valid` 2, round 2's bytes in place, and per-round evidence under `.run.rounds/`; always-invalid with `--rounds 3` refuses `repair-exhausted` with exactly 3 spawns and non-empty `error.problems`; the default spawns once with no `data.repair`; `--print --rounds` and `--rounds 0/6/x` refuse `bad-flags` |
Expand Down
112 changes: 112 additions & 0 deletions evals/cases/run-agent-probe.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
#!/usr/bin/env bash
#
# STA-440: a binary-name clash is settled by a help probe before a launch. The
# two `fx` stubs below derive every answer from argv: the agent answers its own
# ask --help and writes the directive's review, while the JSON viewer prints the
# wrong identity. No model, no secret, no host.
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
# shellcheck source=../lib/common.sh
. "$HERE/../lib/common.sh"

SANDBOX="$(cd "$EVAL_SANDBOX" && pwd)"
LOCAL=local-probe
DIR="$SANDBOX/.verdict/state/$LOCAL"
LAUNCHER="$DIR/launcher.json"
AGENTBIN="$SANDBOX/probebin"
STUB_LOG="$SANDBOX/fx-probe.log"
mkdir -p "$AGENTBIN"
export STUB_LOG

cat >"$AGENTBIN/fx" <<'STUB'
#!/usr/bin/env bash
set -euo pipefail
printf 'argc=%s argv=%s\n' "$#" "$*" >>"$STUB_LOG"
if [ "${1:-}" != "ask" ]; then
echo "fx: expected ask <directive>" >&2
exit 2
fi
if [ "${2:-}" = "--help" ]; then
echo "fx ask: runs one noninteractive request and exits"
echo " --model <model> choose the model"
echo " --effort <level> set the reasoning effort"
echo " --fast, --ultrafast request a faster answer"
exit 0
fi
directive="${!#}"
rest="${directive#*Write }"
artifact="${rest%% with \{*}"
if [ -z "$artifact" ] || [ "$artifact" = "$directive" ]; then
echo "fx: the directive named no review path" >&2
exit 2
fi
cat >"$artifact" <<'JSON'
{"schema_version":"1","body":"Nit: one note.","approved":false,"comments":[]}
JSON
STUB
chmod +x "$AGENTBIN/fx"
export PATH="$AGENTBIN:$PATH"

eval_mk_bundle "$LOCAL"

# --- --print renders argv and spawns nothing, so it cannot probe -------------
: >"$STUB_LOG"
run_review run acme/app "$LOCAL" --agent fx --print --json -- --model M
assert_exit "fx --print renders" 0
assert_envelope "fx --print envelope"
assert_json_field "--print uses fx ask" "fx ask --model M" \
'.data.argv | .[0:4] | join(" ")'
if [ -s "$STUB_LOG" ]; then
echo "--print spawned a probe: $(cat "$STUB_LOG")" >&2
exit 1
fi

# --- the real agent passes the probe and records it before the launch --------
: >"$STUB_LOG"
run_review run acme/app "$LOCAL" --agent fx --json -- --model M
assert_exit "fx launches after a matching probe" 0
assert_envelope "fx launch envelope"
assert_json_field "the launch accepts one fresh review" "true" '.data.artifact_fresh'
assert_json_field "launcher.json records the probe args" "ask,--help" \
'.probe.args | join(",")' "$LAUNCHER"
assert_json_field "launcher.json records the passing probe" "true" \
'.probe.ok' "$LAUNCHER"
assert_file_grep "the probe ran first" "$STUB_LOG" '^argc=2 argv=ask --help$'
assert_file_grep "the launch followed it" "$STUB_LOG" '^argc=4 argv=ask --model M Review acme/app'

# --- a different fx refuses before starting any agent -----------------------
cat >"$AGENTBIN/fx" <<'STUB'
#!/usr/bin/env bash
set -euo pipefail
printf 'wrong argc=%s argv=%s\n' "$#" "$*" >>"$STUB_LOG"
if [ "${2:-}" = "--help" ]; then
echo "fx 1.0: terminal JSON viewer"
exit 0
fi
echo "fx: JSON viewer has no ask command" >&2
exit 1
STUB
chmod +x "$AGENTBIN/fx"
rm -f "$DIR/review.json" "$DIR/launcher.json"
: >"$STUB_LOG"
run_review run acme/app "$LOCAL" --agent fx --json -- --model M
assert_exit "a clashing binary refuses" 1
assert_refusal "clash" "agent-mismatch"
assert_json_field "the refusal names the documented flag" "true" \
'.error.message | contains("--ultrafast")'
assert_json_field "the refusal names the resolved path" "true" \
'.error.message | contains("probe")'
assert_json_field "the refusal hint names settings.json" "true" \
'(.error.message // "") | contains("settings.json")'
assert_file_grep "only the probe ran" "$STUB_LOG" '^wrong argc=2 argv=ask --help$'
if [ -e "$DIR/review.json" ] || [ -e "$LAUNCHER" ]; then
echo "the mismatch wrote a launch artifact" >&2
exit 1
fi
if [ "$(wc -l <"$STUB_LOG" | tr -d ' ')" != "1" ]; then
echo "the mismatch started an agent: $(cat "$STUB_LOG")" >&2
exit 1
fi
assert_stream_nogrep "run made no host call" "$EVAL_STUBLOG" "stub-gh"

eval_end
57 changes: 57 additions & 0 deletions internal/archtest/agent_sources_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,63 @@ func TestBuiltinAgentsCarrySourceComments(t *testing.T) {
}
}

// TestAgentNamesMatchTheirSurfaces pins the two human lists of --agent names
// against the shipped table. A row that exists in code but not in usage or the
// docs is undiscoverable; a name in the docs that no row answers is a lie.
func TestAgentNamesMatchTheirSurfaces(t *testing.T) {
t.Parallel()
root := moduleRoot(t)
names := shippedAgentNames(t, filepath.Join(root, "internal", "core", "run", "agents.go"))
for _, surface := range []struct {
path string
quoted bool
}{
{path: filepath.Join(root, "cmd", "verdict", "usage.go")},
{path: filepath.Join(root, "docs", "manual.md"), quoted: true},
{path: filepath.Join(root, "skills", "verdict", "references", "commands.md"), quoted: true},
} {
assertSurfaceNames(t, surface.path, surface.quoted, names)
}
}

func shippedAgentNames(t *testing.T, path string) []string {
t.Helper()
raw, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
matches := agentNameLiteral.FindAllStringSubmatch(string(raw), -1)
if len(matches) == 0 {
t.Fatal("agent table scan found no names; the guard would pass vacuously")
}
names := make([]string, 0, len(matches))
for _, match := range matches {
names = append(names, match[1])
}
return names
}

func assertSurfaceNames(t *testing.T, path string, quoted bool, names []string) {
t.Helper()
doc, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
for _, name := range names {
want := surfaceWant(name, quoted)
if !strings.Contains(string(doc), want) {
t.Errorf("%s does not name agent %q (want %q)", path, name, want)
}
}
}

func surfaceWant(name string, quoted bool) string {
if quoted {
return "`" + name + "`"
}
return name
}

func checkAgentSourceComments(t *testing.T, path string, lines []string) int {
t.Helper()
seen := 0
Expand Down
2 changes: 1 addition & 1 deletion internal/config/fields.go
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ var (
personalTopFields = core.SettingsTopKeys()
personalSpendFields = []string{"budget_usd", "require_known_estimate", "enforce_cap"}
personalPrivacyFields = []string{"exclusions"}
personalAgentFields = []string{"binary", "args", "prompt_flag", "model_flag", "install"}
personalAgentFields = []string{"binary", "args", "prompt_flag", "model_flag", "install", "probe_args", "probe_match"}
)

// The typed operator-settings sections: named once so the field contract,
Expand Down
Loading
Loading