diff --git a/.changeset/run-agent-breadth.md b/.changeset/run-agent-breadth.md new file mode 100644 index 000000000..ebc05ebe1 --- /dev/null +++ b/.changeset/run-agent-breadth.md @@ -0,0 +1,5 @@ +--- +"@stainless-code/verdict": minor +--- + +`run --agent` now supports fourteen verified official agent CLIs, including `agy`, `amp`, `copilot`, `crush`, `droid`, `gemini`, `goose`, `kiro`, `pi`, and `qwen`, with prompt flags placed immediately before the directive for agents that take it as a flag value. Operators can define more agents under the new `agents` key of their own settings file; the shape is prefs-only, closed, and refuses name collisions and invalid binaries as `bad-settings`. diff --git a/cmd/verdict/usage.go b/cmd/verdict/usage.go index 17dc4bcbe..b6de37fc2 100644 --- a/cmd/verdict/usage.go +++ b/cmd/verdict/usage.go @@ -49,9 +49,10 @@ func usageText() string { " kb.txt/sensitive.txt/panel when present (one fence per file, in that order) —\n" + " to stdout (human; progress moves to stderr) or data.prompt_dump (--json).\n" + " No agent runs, no network beyond the fetch, no store writes beyond the bundle.\n" + - " run --agent [--recipe ]\n" + - " [--stage ] [--timeout ] [--rounds ] [--print]\n" + - " [-- ...] [--json]\n" + + " run \n" + + " --agent \n" + + " [--recipe ] [--stage ] [--timeout ] [--rounds ]\n" + + " [--print] [-- ...] [--json]\n" + " launch YOUR agent against an existing bundle: one process per round, one shot by default.\n" + " It requires exactly one fresh review.json (absent refuses no-review-artifact, older\n" + " than the launch refuses stale-review-artifact), names the next command, and by default never\n" + @@ -59,6 +60,9 @@ func usageText() string { " invalid review.json re-prompts the agent with the shape checker's fix list, up to\n" + " N launch-to-check rounds; exhaustion refuses repair-exhausted. The agent inherits your\n" + " environment (your key) and may use the network; verdict itself makes no calls.\n" + + " Operators can define more names under the 'agents' key of their own settings\n" + + " file (XDG_CONFIG_HOME/verdict/settings.json); a repository config cannot.\n" + + " A flag that takes the prompt places it immediately before the directive.\n" + " A non-zero agent exit is the process exit code (agent-failed); the watchdog\n" + " timeout (agent-timeout), a missing binary (agent-missing), and a spawn failure\n" + " (agent-spawn-failed) refuse typed with a hint. Provenance lands in\n" + diff --git a/docs/manual.md b/docs/manual.md index 197151a57..97b6bc8bd 100644 --- a/docs/manual.md +++ b/docs/manual.md @@ -2142,19 +2142,77 @@ bundle's `.run.rounds/` files. `run --print` accepts only `--model`, `-m`, `--max-budget-usd`, `--max-cost`, and `--max-turns` with simple model identifiers or numeric values. Other arguments refuse printing; launch them directly. Credentials belong in the environment. Provenance hides arguments outside this allowlist. Missing agent binaries are checked before directive assembly and report an install step. -Agent definitions are data: a name, a binary, and the non-interactive argument -shape (`claude -p`, `codex exec`, `cursor-agent -p`, `opencode run`). Adding an -agent is a new table row, not a new code path; the launcher branches on none of -it. The argv it builds is -` [...] `. - -| Piece | Meaning | -| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- | -| `--agent ` | Required. An unknown name refuses `unknown-agent` and lists the known ones. | -| `--timeout ` | Watchdog budget, default `300` (the same default `verify` uses). `--timeout` is per round. | -| `--rounds ` | Total launch→check rounds, default `1` (today's one-shot), range 1–5. `-- ` passes through every round. | -| `-- ` | Everything after the bare `--` goes to the agent unchanged: its own model, cost, and turn flags are yours. | -| `--print` | Renders the exact command (POSIX-quoted) and exits `0` without spawning or writing. The escape hatch for harnesses that build their own invocation. | +Agent definitions are data: a name, a binary, the non-interactive argument +shape, and optionally a flag that takes the directive as its value. The launcher +branches on none of it. With no prompt flag the argv is +` [...] `; with one it is +` [...] `, so a +pass-through flag can never be swallowed as the prompt flag's value. +`--print` renders the same bytes a launch spawns. + +| Name | Binary and agent flags | Prompt placement | Official install | Flag that lets it write the artifact | Source | +| ---------- | -------------------------------- | ---------------- | -------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | +| `agy` | `agy` | `-p ` | `curl -fsSL https://antigravity.google/cli/install.sh \| bash` | `--dangerously-skip-permissions`, or `--mode accept-edits` | [Antigravity CLI](https://github.com/google-antigravity/antigravity-cli) | +| `amp` | `amp` | `-x ` | `npm install -g @ampcode/cli` | none: tools run without an approval prompt by default | [Amp execute mode](https://ampcode.com/docs/markdown/cli/execute-mode) | +| `claude` | `claude -p` | last argument | `curl -fsSL https://claude.ai/install.sh \| bash` | `--permission-mode acceptEdits`, `--allowedTools`, or `--dangerously-skip-permissions` | [Claude CLI reference](https://code.claude.com/docs/en/cli-reference.md) | +| `codex` | `codex exec` | last argument | `npm install -g @openai/codex` | `--sandbox workspace-write` (the default sandbox is read-only) | [Codex README](https://github.com/openai/codex/blob/main/README.md) | +| `copilot` | `copilot` | `-p ` | `npm install -g @github/copilot` | `--allow-tool=write` or `--allow-all-tools` | [Copilot CLI automation](https://docs.github.com/en/copilot/how-tos/copilot-cli/automate-copilot-cli/run-cli-programmatically) | +| `crush` | `crush run` | last argument | `brew install charmbracelet/tap/crush` | none: `run` auto-approves permission requests | [Crush run command](https://github.com/charmbracelet/crush/blob/main/internal/cmd/run.go) | +| `cursor` | `cursor-agent -p` | last argument | `curl https://cursor.com/install -fsS \| bash` | none for workspace writes; `--force` for shell or writes outside the workspace | [Cursor headless](https://cursor.com/docs/cli/headless) | +| `droid` | `droid exec` | last argument | `curl -fsSL https://app.factory.ai/cli \| sh` | `--auto ` | [Droid CLI reference](https://docs.factory.com/droid-cli/cli-reference.md) | +| `gemini` | `gemini` | `-p ` | `npm install -g @google/gemini-cli` | `--approval-mode=auto_edit`, plus `--skip-trust` for an untrusted folder | [Gemini CLI README](https://github.com/google-gemini/gemini-cli/blob/main/README.md) | +| `goose` | `goose run` | `-t ` | `brew install block-goose-cli` | none: autonomous mode is the default; `--model` usually needs `--provider` or `GOOSE_PROVIDER` | [Goose running tasks](https://github.com/aaif-goose/goose/blob/main/documentation/docs/guides/running-tasks.md) | +| `kiro` | `kiro-cli chat --no-interactive` | last argument | `curl -fsSL https://cli.kiro.dev/install \| bash` | `--trust-all-tools`; name `--v2` or `--v3` yourself so the harness stays explicit | [Kiro headless](https://kiro.dev/docs/cli/headless.md) | +| `opencode` | `opencode run` | last argument | `npm install -g @opencode/cli` | `--auto` when custom ask rules or `.env` reads could block | [OpenCode CLI](https://opencode.ai/docs/cli/) | +| `pi` | `pi -p` | last argument | `curl -fsSL https://pi.dev/install.sh \| sh` | none: tools are enabled by default; `--tools` narrows the allowlist | [Pi quickstart](https://github.com/badlogic/pi-mono/blob/main/packages/coding-agent/docs/quickstart.md) | +| `qwen` | `qwen` | last argument | `npm install -g @qwen-code/qwen-code@latest` | `--approval-mode=yolo` in a trusted folder | [Qwen top-level options](https://github.com/QwenLM/qwen-code/blob/main/packages/cli/src/config/top-level-options.ts) | + +Permission and approval flags are yours to pass after `--`, never baked into a +row. Gemini CLI's maintained scope is Enterprise and Code Assist; free, Pro, +and Ultra subscriptions use `agy`. Qwen Code's positional prompt is the +supported one-shot form (`-p` is deprecated). `cursor` keeps `cursor-agent` +because that is the binary its installer still puts on `PATH`. + +Aider is deliberately absent. Its positional arguments are files to edit, not +a prompt, so it cannot read a verdict bundle with its own tools; it also +auto-commits by default. An operator can still define it below. + +#### Operator-defined agents + +Names can also come from your own settings file, +`${XDG_CONFIG_HOME:-$HOME/.config}/verdict/settings.json`, under `agents`. +The file accepts ten top-level keys in total (including `$schema`): `$schema`, +`output`, `layout`, `color`, `publish`, `review`, `spend`, `privacy`, `packs`, +and `agents`: + +```json +{ + "agents": { + "local": { + "binary": "local-agent", + "args": ["exec"], + "prompt_flag": "-p", + "model_flag": "--model", + "install": "brew install local-agent" + } + } +} +``` + +`binary` is required and is either a name resolved on `PATH` or an absolute +path; `args` defaults to `[]`, and the other three fields are optional strings. +A name matches `[a-z0-9][a-z0-9-]*` and must not collide with a shipped row. +Unknown keys at any level, a blank binary, a relative binary containing a path +separator, a bad name, and a collision all refuse `bad-settings` naming the +path. The file is prefs-only: a repository `.verdict/config.json` carrying an +`agents` key refuses `bad-config` from the typed config parser, and a PR cannot +define an agent. The settings parser owns the same closed shape. + +A defined agent behaves like any shipped row: its own flags come first, your +`-- ` follow, and the directive comes last (or directly after its +`prompt_flag`). `unknown-agent` lists both groups with labels. `launcher.json` +records `agent_source: "prefs"` so provenance says where the definition came +from. ### Recipes (declared review pipelines) @@ -2504,8 +2562,10 @@ The prefs file ends env sprawl without forcing migration. Path: respected, `$schema` key carries a schema hint that is never fetched). The file carries the four presentation namespaces below plus the typed personal layer (`review`, `spend`, `privacy`) described under "Typed repository -configuration" above, so it accepts eight top-level keys in total: `$schema`, `output`, -`layout`, `color`, `publish`, `review`, `spend`, `privacy`. Anything else — +configuration" above, operator-defined agents under `agents`, and the org pack +list under `packs`, so it accepts ten top-level keys in total (including +`$schema`): `$schema`, `output`, `layout`, `color`, `publish`, `review`, +`spend`, `privacy`, `packs`, `agents`. Anything else — server-shaped or `org.*` keys included — refuses typed `bad-settings`. The presentation sample: diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 23f56b01e..b49df76c5 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -344,9 +344,9 @@ Fix: pass a comma-separated glob list (e.g. `src/*.ts`) together with `--rule`, ## `bad-settings` -Cause: the settings file is malformed (or carries an unknown placeholder). +Cause: the settings file is malformed, carries an unknown placeholder, or names an invalid operator-defined agent under `agents` — an unknown key, a blank or relative binary, a name outside `[a-z0-9][a-z0-9-]*`, or a collision with a shipped agent. -Fix: keep only `output/layout/color/publish/review/spend/privacy` keys (see `docs/manual.md`), then re-run. +Fix: keep only the ten accepted top-level keys (including `$schema`): `$schema/output/layout/color/publish/review/spend/privacy/packs/agents`. Fix the named `$.agents.*` field (or choose a non-colliding name), then re-run. Agent definitions belong only in the prefs file, never in `.verdict/config.json` (see `docs/manual.md`). ## `bad-settings-credentials` diff --git a/evals/README.md b/evals/README.md index f2d36a5a1..e4d106b5c 100644 --- a/evals/README.md +++ b/evals/README.md @@ -180,6 +180,7 @@ the build. | `rule-exposure` | Per-rule `recalled_in`, `cited_in`, `exposure`, and `acceptance` on `learn --status` | Real CLI `learn --status` (text and `--json`) over seeded bundles: missing `kb-recall.json` reads `null`/`unmeasured`; two measured bundles recalling both rules where one rule is never cited and the other is cited once addressed read `recalled_in=2 cited_in=0 exposure=0 acceptance=n/a` and `recalled_in=2 cited_in=1 exposure=0.5 acceptance=1`. Zero host calls | | `self-review` | Bounded local self-review recipe | Claude, Codex, and OpenCode stubs fix on round 2 or exhaust two rounds; assert exit, last gate, receipts, one spawn per round, zero host calls, and recipe unit checks. | | `run-agent` | Shipped agent-launch recipe: one fresh review.json, then handoff | Committed stub agent on PATH as `claude` (no model, no network, no secret): `--print` renders the exact argv and writes nothing; a launch spawns one process in the repo root, is accepted only for one fresh `review.json`, and `launcher.json` records the redacted argv, exit, and freshness; `record` merges the launched review byte-identically; on a PR bundle `post --dry-run` accepts it at `COMMENT` with no `post.json`; stale, missing, failed, timed-out, and not-installed all refuse typed, and the docs recipe plus the CI step that proves the case ran are graded too | +| `run-custom-agent` | Operator-defined `run --agent` row from prefs | A stub generated in the sandbox is defined only under a sandboxed `XDG_CONFIG_HOME`, derives its behavior from the argv (subcommand, prompt flag immediately before the directive, review path in the directive), and `--print` matches the launch byte for byte; `launcher.json` marks `agent_source: "prefs"`, a missing binary carries the install hint from its definition, `unknown-agent` labels both sources, a repo-local config with `agents` does not define an agent, and a shipped-name collision refuses `bad-settings` | | `run-recipe` | Declared review pipelines resolve from the base recipe control, never the PR's own edit | Real git base and head, `fetch-local --base` bundles, committed stub agent as `claude`: `--print` fills `data.recipe` (base version, sha256, four stages, its lens, the budget), named in the argv, writing nothing; an absent lens and an over-limit budget each warn once, then run; no `--recipe` resolves `builtin/default` and warns about nothing; the launch's `launcher.json.recipe` equals `data.recipe` exactly, the directive carries the pipeline never the description, `record` accepts the review; a head rewrite never reaches the plan; `--allow-head-controls` reads the head's copy instead; tampered snapshot bytes refuse `controls-stale`; `unknown-recipe` and `bad-recipe` refuse before any spawn; `--print --stage ` renders one stage's argv (the declared hint as the model argument, lens hints and caps, the `review.json` the stages chain through) plus its own refusals; zero host calls | | `run-repair` | Structured-output repair loop in `run --rounds N` | Committed repair stub on PATH as `claude` (no model, no network, no secret): round 1 invalid then round 2 valid exits 0 with exactly 2 spawns, `rounds_to_valid` 2, round 2's bytes in place, and per-round evidence under `.run.rounds/`; always-invalid with `--rounds 3` refuses `repair-exhausted` with exactly 3 spawns and non-empty `error.problems`; the default spawns once with no `data.repair`; `--print --rounds` and `--rounds 0/6/x` refuse `bad-flags` | | `sandbox-recipe-docs` | Hostile-PR sandbox recipe docs presence | The manual carries the bwrap one-liner (ro-bind `/`, rw-bind repo, `--unshare-net` variant), the systemd-nspawn alternative, the when-to-use rule, the tmpfs plus `~/.aws`/`~/.ssh`/`~/.config/gh` secret paths, and the recipe-not-enforcement line; `system.md` names the enforced typed verify gate while the untrusted-data preamble stays untouched; the recipe pins its known residuals | diff --git a/evals/cases/run-custom-agent.sh b/evals/cases/run-custom-agent.sh new file mode 100644 index 000000000..a627fafdf --- /dev/null +++ b/evals/cases/run-custom-agent.sh @@ -0,0 +1,152 @@ +#!/usr/bin/env bash +# +# STA-440: a run agent can also come from the operator's own settings file. +# The stub agent below is generated in the sandbox, defined only under +# XDG_CONFIG_HOME, and derives everything it does from the argv the launcher +# hands it: a subcommand, the prompt flag immediately before the directive, +# and the review path quoted in that directive. No model, no secret, no host. +# +# The second half pins the trust boundary: a repo-local .verdict/config.json +# carrying an `agents` key does not define an agent for run, and a collision +# with a shipped name in the prefs file refuses bad-settings. +set -euo pipefail +HERE="$(cd "$(dirname "$0")" && pwd)" +# shellcheck source=../lib/common.sh +. "$HERE/../lib/common.sh" + +SANDBOX="$(cd "$EVAL_SANDBOX" && pwd)" +LOCAL=local-custom +DIR="$SANDBOX/.verdict/state/$LOCAL" +LAUNCHER="$DIR/launcher.json" +SETTINGS_DIR="$SANDBOX/xdg/verdict" +SETTINGS="$SETTINGS_DIR/settings.json" +AGENTBIN="$SANDBOX/custombin" +STUB_LOG="$SANDBOX/custom-agent.log" +mkdir -p "$SETTINGS_DIR" "$AGENTBIN" +export XDG_CONFIG_HOME="$SANDBOX/xdg" +export STUB_LOG + +write_settings() { + printf '%s\n' "$1" >"$SETTINGS" +} + +write_settings '{"agents":{"local":{"binary":"custom-agent","args":["exec"],"prompt_flag":"-p","model_flag":"--model","install":"install custom-agent"}}}' + +# The stub is a cooperating agent, not a recording of one expected argv: it +# refuses a wrong subcommand or a prompt flag that is not immediately before +# the directive, then writes only the review file the directive names. +cat >"$AGENTBIN/custom-agent" <<'STUB' +#!/usr/bin/env bash +set -euo pipefail +printf 'argc=%s\n' "$#" >>"$STUB_LOG" +i=0 +for arg in "$@"; do + printf 'arg[%s]=%s\n' "$i" "$(printf '%s' "$arg" | tr '\n' ' ')" >>"$STUB_LOG" + i=$((i + 1)) +done +if [ "${1:-}" != "exec" ] || [ "$#" -lt 3 ]; then + echo "custom-agent: expected 'exec -p '" >&2 + exit 2 +fi +prompt_at=0 +for ((i = 1; i <= $#; i++)); do + if [ "${!i}" = "-p" ]; then + prompt_at=$i + fi +done +if [ "$prompt_at" -ne "$(( $# - 1 ))" ]; then + echo "custom-agent: -p must sit immediately before the directive" >&2 + exit 2 +fi +directive="${!#}" +rest="${directive#*Write }" +artifact="${rest%% with \{*}" +if [ -z "$artifact" ] || [ "$artifact" = "$directive" ]; then + echo "custom-agent: the directive named no review path" >&2 + exit 2 +fi +cat >"$artifact" <<'JSON' +{"schema_version":"1","body":"Nit: one note.","approved":false,"comments":[]} +JSON +STUB +chmod +x "$AGENTBIN/custom-agent" +export PATH="$AGENTBIN:$PATH" + +eval_mk_bundle "$LOCAL" + +# --- --print: the custom row renders the exact argv and spawns nothing ------- +run_review run acme/app "$LOCAL" --agent local --print --json -- --model test-model +assert_exit "a prefs-defined agent prints" 0 +assert_envelope "custom agent --print" +assert_json_field "--print names the custom agent" "local" '.data.agent' +assert_json_field "--print names the custom binary" "custom-agent" '.data.binary' +assert_json_field "--print puts the prompt flag before the directive" \ + "custom-agent exec --model test-model -p [prompt]" \ + '.data.argv | "\(.[0:5] | join(" ")) [prompt]"' +assert_json_field "--print keeps the directive last" "true" \ + '.data.argv[-1] | startswith("Review acme/app local change local-custom")' + +# --- the launch: one process, provenance marked as prefs, one review --------- +run_review run acme/app "$LOCAL" --agent local --json -- --model test-model +assert_exit "a prefs-defined agent launches" 0 +assert_envelope "custom agent launch" +assert_json_field "the launch accepts one fresh review" "true" '.data.artifact_fresh' +assert_json_field "the recorded argv replaces the directive" "true" \ + '(.data.argv | map(select(. == "")) | length) == 1' +assert_json_field "launcher.json names the prefs source" "prefs" '.agent_source' "$LAUNCHER" +assert_json_field "launcher.json names the custom agent" "local" '.agent' "$LAUNCHER" +assert_json_field "launcher.json keeps the operator's model" "test-model" \ + '.argv | .[3]' "$LAUNCHER" +assert_file_grep "the agent got its subcommand" "$STUB_LOG" '^arg\[0\]=exec$' +assert_file_grep "the prompt flag sits before the directive" "$STUB_LOG" '^arg\[3\]=-p$' +assert_file_grep "the directive arrived last" "$STUB_LOG" \ + '^arg\[4\]=Review acme/app local change local-custom' +assert_file_grep "the agent wrote no extra argument" "$STUB_LOG" '^argc=5$' +assert_stream_nogrep "run made no host call" "$EVAL_STUBLOG" "stub-gh" + +# --- the missing-binary refusal carries the definition's install hint -------- +MINBIN="$SANDBOX/minbin" +mkdir -p "$MINBIN" +for tool in git jq sed grep cat dirname mkdir date env; do + found="$(command -v "$tool" 2>/dev/null || true)" + [ -n "$found" ] || continue + ln -sf "$found" "$MINBIN/$tool" +done +set +e +( + cd "$EVAL_SANDBOX" || exit 99 + PATH="$MINBIN" "$EVAL_CLI" run acme/app "$LOCAL" --agent local --json >"$EVAL_OUT" 2>"$EVAL_ERR" +) +# shellcheck disable=SC2034 # EVAL_STATUS is read by assert_exit in the caller +EVAL_STATUS=$? +set -e +assert_exit "a missing custom binary refuses" 1 +assert_refusal "missing custom binary" "agent-missing" +assert_json_field "the refusal names the prefs install hint" "true" \ + '.error.message | contains("install custom-agent")' + +# --- discovery and the prefs-only boundary ---------------------------------- +run_review run acme/app "$LOCAL" --agent nope --json +assert_exit "an unknown name refuses" 1 +assert_refusal "unknown custom name" "unknown-agent" +assert_json_field "the refusal labels the prefs source" "true" \ + '.error.message | contains("operator-defined in prefs: local")' + +rm -f "$DIR/review.json" "$DIR/launcher.json" +printf '%s\n' '{"schema_version":"1","agents":{"local":{"binary":"custom-agent"}}}' >"$EVAL_SANDBOX/.verdict/config.json" +write_settings '{}' +run_review run acme/app "$LOCAL" --agent local --json +assert_exit "a repo config cannot define an agent" 1 +assert_refusal "repo config is not an agent source" "unknown-agent" +assert_json_field "the refusal says no operator agent is defined" "true" \ + '.error.message | contains("operator-defined in prefs: none")' +rm -f "$EVAL_SANDBOX/.verdict/config.json" + +write_settings '{"agents":{"claude":{"binary":"custom-agent"}}}' +run_review run acme/app "$LOCAL" --agent claude --print --json +assert_exit "a colliding name refuses bad-settings" 1 +assert_refusal "collision" "bad-settings" +assert_json_field "the refusal names the agents key" "true" \ + '.error.message | contains("agents.claude")' + +eval_end diff --git a/internal/archtest/agent_sources_test.go b/internal/archtest/agent_sources_test.go new file mode 100644 index 000000000..72fefbf25 --- /dev/null +++ b/internal/archtest/agent_sources_test.go @@ -0,0 +1,71 @@ +package archtest + +import ( + "os" + "path/filepath" + "regexp" + "strings" + "testing" +) + +var ( + agentRowLine = regexp.MustCompile(`^\s*Name: "`) + agentSourceLine = regexp.MustCompile(`^// https://\S+ — read 2026-11-17$`) + agentNameLiteral = regexp.MustCompile(`Name: "([a-z0-9-]+)"`) + agentBlockLine = regexp.MustCompile(`^(//.*|\{)$`) +) + +// TestBuiltinAgentsCarrySourceComments reads the shipped agent table and fails +// when a row loses the one-line primary-source citation above it. The citation +// is the evidence trail for a headless argv and a model flag; a row added from +// memory is not a row this repo can defend. +func TestBuiltinAgentsCarrySourceComments(t *testing.T) { + t.Parallel() + path := filepath.Join(moduleRoot(t), "internal", "core", "run", "agents.go") + raw, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + if checkAgentSourceComments(t, path, strings.Split(string(raw), "\n")) == 0 { + t.Fatalf("%s: scan found no agent rows; the guard would pass vacuously", path) + } +} + +func checkAgentSourceComments(t *testing.T, path string, lines []string) int { + t.Helper() + seen := 0 + for i, line := range lines { + if !agentRowLine.MatchString(line) { + continue + } + seen++ + assertAgentSourceRow(t, path, lines, i, line) + } + return seen +} + +func assertAgentSourceRow(t *testing.T, path string, lines []string, i int, line string) { + t.Helper() + name := agentNameLiteral.FindStringSubmatch(line) + if len(name) != 2 { + t.Fatalf("%s:%d: row has no name literal: %s", path, i+1, line) + } + if !sourceCommentAbove(lines, i) { + t.Errorf("%s:%d: agent %q has no source comment with a URL and read date", path, i+1, name[1]) + } +} + +// sourceCommentAbove walks through the opening brace and the row's own comment +// block, stopping at the blank line that separates one row from the next. +func sourceCommentAbove(lines []string, row int) bool { + for j := row - 1; j >= 0; j-- { + trimmed := strings.TrimSpace(lines[j]) + if agentSourceLine.MatchString(trimmed) { + return true + } + if !agentBlockLine.MatchString(trimmed) { + return false + } + } + return false +} diff --git a/internal/cli/agent_prefs_test.go b/internal/cli/agent_prefs_test.go new file mode 100644 index 000000000..fa0009c62 --- /dev/null +++ b/internal/cli/agent_prefs_test.go @@ -0,0 +1,51 @@ +package cli_test + +import ( + "encoding/json" + "errors" + "testing" + + "github.com/stainless-code/verdict/internal/cli" +) + +// TestPrefsAgentNamesAreNotCredentialKeys pins the same boundary as the +// personal parser: an agent name is data, but a field inside its definition +// still refuses a credential key. +func TestPrefsAgentNamesAreNotCredentialKeys(t *testing.T) { + for _, name := range []string{"token", "patchbot"} { + t.Run(name, func(t *testing.T) { assertPrefsAgentNameAccepted(t, name) }) + } +} + +func assertPrefsAgentNameAccepted(t *testing.T, name string) { + t.Helper() + body, err := json.Marshal(map[string]any{"agents": map[string]any{ + name: map[string]any{"binary": "local-agent"}, + }}) + if err != nil { + t.Fatal(err) + } + _, err = cli.ResolvePrefs(&cli.Runner{}, map[string]string{"HOME": t.TempDir()}, + func(string) ([]byte, error) { return body, nil }) + if err != nil { + t.Fatalf("ResolvePrefs refused agent name %q: %v", name, err) + } +} + +func TestPrefsCredentialFieldInsideAgentStillRefuses(t *testing.T) { + body, err := json.Marshal(map[string]any{"agents": map[string]any{ + "local": map[string]any{"binary": "local-agent", "token": "abc"}, + }}) + if err != nil { + t.Fatal(err) + } + _, err = cli.ResolvePrefs(&cli.Runner{}, map[string]string{"HOME": t.TempDir()}, + func(string) ([]byte, error) { return body, nil }) + var ref *cli.RefusalError + if !errors.As(err, &ref) { + t.Fatalf("err = %T; want a typed settings refusal", err) + } + if ref.Code != "bad-settings-credentials" { + t.Errorf("code = %q; want bad-settings-credentials", ref.Code) + } +} diff --git a/internal/cli/messages.go b/internal/cli/messages.go index aa44be63a..1aec8641e 100644 --- a/internal/cli/messages.go +++ b/internal/cli/messages.go @@ -3,8 +3,8 @@ package cli const ( msgCredKey = "settings file %s names a credential key " + "(token/key/auth/secret/password/pat/credential/bearer) — credentials never live in the settings file (export them as env instead)" - msgTopKey = "settings file %s carries unknown top-level key '%s' (want only " + - "$schema, output, layout, color, publish; no server-shaped or org.* namespaces)" + msgTopKey = "settings file %s carries unknown top-level key '%s' (want only %s; " + + "no server-shaped or org.* namespaces)" msgSchemaType = "settings file %s: $schema must be a string (schema hint only, never fetched)" msgOutputObj = "settings file %s: output must be an object (want {\"format\": \"text\"|\"json\"})" msgOutputKey = "settings file %s: unknown output key '%s' (want only format)" @@ -35,6 +35,5 @@ const ( const ( msgNotJSON = "settings file %s is not valid JSON (fix it or delete it)" - msgNotObject = "settings file %s must hold a JSON object " + - "(want only $schema, output, layout, color, publish)" + msgNotObject = "settings file %s must hold a JSON object (want only %s)" ) diff --git a/internal/cli/prefs.go b/internal/cli/prefs.go index 2c7521031..f0c0b043f 100644 --- a/internal/cli/prefs.go +++ b/internal/cli/prefs.go @@ -3,8 +3,11 @@ package cli import ( "fmt" "regexp" + "slices" + "strings" "github.com/stainless-code/verdict/internal/config" + "github.com/stainless-code/verdict/internal/core" "github.com/stainless-code/verdict/internal/jq" "github.com/stainless-code/verdict/internal/readcommon" ) @@ -46,7 +49,8 @@ func ResolvePrefs(r *Runner, env map[string]string, readFile func(string) ([]byt } obj, ok := jq.AsMap(doc) if !ok { - return nil, &RefusalError{Code: codeBadSettings, Message: fmt.Sprintf(msgNotObject, path)} + return nil, &RefusalError{Code: codeBadSettings, Message: fmt.Sprintf(msgNotObject, path, + strings.Join(core.SettingsTopKeys(), ", "))} } return resolveSettingsFile(r, path, obj, doc, &eff) } @@ -127,11 +131,11 @@ func resolveSettingsFile(r *Runner, path string, obj map[string]any, doc any, ef } func checkTopKeys(path string, obj map[string]any) error { + accepted := core.SettingsTopKeys() for _, k := range SortedKeys(obj) { - switch k { - case "$schema", "output", "layout", "color", "publish", "review", "spend", "privacy", "packs": - default: - return &RefusalError{Code: codeBadSettings, Message: fmt.Sprintf(msgTopKey, path, k)} + if !slices.Contains(accepted, k) { + return &RefusalError{Code: codeBadSettings, Message: fmt.Sprintf(msgTopKey, path, k, + strings.Join(accepted, ", "))} } } return nil @@ -272,21 +276,41 @@ type credWalker struct { } func (w *credWalker) walk(v any) { + w.walkAt(v, true) +} + +func (w *credWalker) walkAt(v any, root bool) { switch t := v.(type) { case map[string]any: - w.walkMap(t) + w.walkMap(t, root) case []any: for _, child := range t { - w.walk(child) + w.walkAt(child, false) } } } -func (w *credWalker) walkMap(t map[string]any) { +func (w *credWalker) walkMap(t map[string]any, root bool) { for k, child := range t { - if credRe.MatchString(readcommon.ASCIILower(k)) { - w.n++ + w.countCredentialKey(k) + w.walkChild(k, child, root) + } +} + +func (w *credWalker) countCredentialKey(key string) { + if credRe.MatchString(readcommon.ASCIILower(key)) { + w.n++ + } +} + +func (w *credWalker) walkChild(key string, child any, root bool) { + if root && key == "agents" { + if defs, ok := child.(map[string]any); ok { + for _, def := range defs { + w.walkAt(def, false) + } + return } - w.walk(child) } + w.walkAt(child, false) } diff --git a/internal/cli/settings_keys_test.go b/internal/cli/settings_keys_test.go new file mode 100644 index 000000000..782ca593c --- /dev/null +++ b/internal/cli/settings_keys_test.go @@ -0,0 +1,47 @@ +package cli_test + +import ( + "errors" + "strings" + "testing" + + "github.com/stainless-code/verdict/internal/cli" + "github.com/stainless-code/verdict/internal/core" +) + +// TestSettingsKeySurfacesListEveryAcceptedKey pins the surfaces that tell an +// operator which settings keys exist. The expected set comes from the same list +// the loader uses, so adding a key without updating the hint or either settings +// refusal message fails here. +func TestSettingsKeySurfacesListEveryAcceptedKey(t *testing.T) { + accepted := core.SettingsTopKeys() + if len(accepted) == 0 { + t.Fatal("SettingsTopKeys is empty; the check would pass vacuously") + } + assertListsEveryKey(t, "bad-settings hint", core.HintFor("bad-settings"), accepted) + assertListsEveryKey(t, "unknown-key message", settingsRefusal(t, `{"bogus":1}`).Message, accepted) + assertListsEveryKey(t, "non-object message", settingsRefusal(t, `[]`).Message, accepted) +} + +func settingsRefusal(t *testing.T, body string) *cli.RefusalError { + t.Helper() + _, err := cli.ResolvePrefs(&cli.Runner{}, map[string]string{"HOME": t.TempDir()}, + func(string) ([]byte, error) { return []byte(body), nil }) + var ref *cli.RefusalError + if !errors.As(err, &ref) { + t.Fatalf("err = %T; want a typed settings refusal", err) + } + if ref.Code != "bad-settings" { + t.Fatalf("code = %q; want bad-settings", ref.Code) + } + return ref +} + +func assertListsEveryKey(t *testing.T, surface, message string, accepted []string) { + t.Helper() + for _, key := range accepted { + if !strings.Contains(message, key) { + t.Errorf("%s %q omits accepted key %q", surface, message, key) + } + } +} diff --git a/internal/config/errors.go b/internal/config/errors.go index 1a1102101..3d3bbae63 100644 --- a/internal/config/errors.go +++ b/internal/config/errors.go @@ -14,6 +14,9 @@ const ( CodeTooLarge = "config-too-large" // CodeBadConfig refuses malformed or semantically invalid configuration. CodeBadConfig = "bad-config" + // CodeBadSettings refuses a malformed operator settings file. The agents + // section uses it so the settings loader's remediation covers a bad agent. + CodeBadSettings = "bad-settings" // CodeBadCredentials refuses a configuration carrying a credential key. CodeBadCredentials = "bad-config-credentials" //nolint:gosec // G101: refusal code, not a credential. // CodeBadRequirement refuses an unsatisfiable or invalid requirement. diff --git a/internal/config/fields.go b/internal/config/fields.go index 93b2f6a09..4e8a89ed7 100644 --- a/internal/config/fields.go +++ b/internal/config/fields.go @@ -37,12 +37,10 @@ var ( // typed personal sections plus the CLI prefs keys the prefs layer // owns (output, layout, color, publish), which parse but never feed // the resolver. - personalTopFields = []string{ - "$schema", personalSectionReview, personalSectionSpend, personalSectionPrivacy, - "packs", "output", "layout", "color", "publish", - } + personalTopFields = core.SettingsTopKeys() personalSpendFields = []string{"budget_usd", "require_known_estimate", "enforce_cap"} personalPrivacyFields = []string{"exclusions"} + personalAgentFields = []string{"binary", "args", "prompt_flag", "model_flag", "install"} ) // The typed operator-settings sections: named once so the field contract, @@ -52,6 +50,7 @@ const ( personalSectionSpend = "spend" personalSectionPrivacy = "privacy" personalSectionPacks = "packs" + personalSectionAgents = "agents" ) // checkUnknownFields refuses any key outside the contract on fixed-shape diff --git a/internal/config/personal.go b/internal/config/personal.go index 261b81255..e79bdd4fa 100644 --- a/internal/config/personal.go +++ b/internal/config/personal.go @@ -2,8 +2,13 @@ package config import ( "errors" + "fmt" + "maps" "os" "path/filepath" + "regexp" + "slices" + "strings" "github.com/stainless-code/verdict/internal/core" ) @@ -40,9 +45,22 @@ type PersonalPrivacy struct { Exclusions []string } +// PersonalAgent is one operator-defined run agent from the prefs file. It has +// the same launch fields as a shipped row; Source is set by the run service, +// never by the document. +type PersonalAgent struct { + Name string + Binary string + PromptFlag string + ModelFlag string + Install string + Args []string +} + // PersonalSettings is the parsed personal layer: review defaults (every // defaults key except budget_usd, which lives in spend), the spend -// constraint, the privacy constraint, and the explicit org pack paths. +// constraint, the privacy constraint, the explicit org pack paths, and the +// operator-defined run agents. // //nolint:govet // the sections read in resolver order type PersonalSettings struct { @@ -50,6 +68,7 @@ type PersonalSettings struct { Spend PersonalSpend Privacy PersonalPrivacy Packs []string + Agents []PersonalAgent } // spendPath addresses the personal spend constraint section. @@ -58,6 +77,23 @@ const spendPath = "$.spend" // packsPath addresses the explicit org pack list: directories, never a scan. const packsPath = "$.packs" +// agentsPath addresses the operator-defined run agents: prefs only. +const ( + agentsPath = "$.agents" + pathDot = "." +) + +// Shared diagnostics for the closed agents shape. +const ( + wantObjectMsg = "want an object, got %s" + binaryField = ".binary" + argsField = ".args" +) + +// agentNamePattern is the published name vocabulary: a lowercase slug that +// starts with an alphanumeric byte. +var agentNamePattern = regexp.MustCompile(`^[a-z0-9][a-z0-9-]*$`) + // ParsePersonal reads the personal sections in the same fixed order as // the repository parser: size, JSON shape, credential scan, unknown // fields, then semantic validation. A budget_usd inside review refuses as @@ -120,9 +156,36 @@ func checkPersonalSections(root map[string]any) *core.RefusalError { return ref } } + return checkPresent(root, personalSectionAgents, checkAgentKeys) +} + +// checkAgentKeys refuses unknown keys and wrong shapes inside the agents +// section before the semantic pass reads any field. +func checkAgentKeys(raw any) *core.RefusalError { + obj, ok := raw.(map[string]any) + if !ok { + return agentRefuse(agentsPath, wantObjectMsg, jsonKind(raw)) + } + for _, name := range sortedObjectKeys(obj) { + value := obj[name] + path := agentsPath + pathDot + name + entry, ok := value.(map[string]any) + if !ok { + return agentRefuse(path, wantObjectMsg, jsonKind(value)) + } + if ref := checkKeys(path, entry, personalAgentFields); ref != nil { + return refuse(CodeBadSettings, "%s", ref.Message) + } + } return nil } +// agentRefuse keeps every agents-section diagnostic in the settings refusal +// family, with the JSON path the operator has to edit. +func agentRefuse(path, format string, args ...any) *core.RefusalError { + return refuse(CodeBadSettings, "%s: %s", path, fmt.Sprintf(format, args...)) +} + // checkPresent runs one section check when the key is present; absent reads // clean and the semantic pass names anything else. func checkPresent(root map[string]any, key string, check func(any) *core.RefusalError) *core.RefusalError { @@ -169,6 +232,10 @@ func buildPersonal(root map[string]any) (*PersonalSettings, *core.RefusalError) val, ref := buildPersonalPacks(raw) return assignSection(&out.Packs, val, ref) }}, + {key: personalSectionAgents, build: func(raw any) *core.RefusalError { + val, ref := buildPersonalAgents(raw) + return assignSection(&out.Agents, val, ref) + }}, } // The section order is fixed: it decides which refusal a document // with several violations reports first. @@ -247,6 +314,133 @@ func presentBool(obj map[string]any, key, path string) (bool, *core.RefusalError return *val, nil } +// BuiltinAgentNames lists the names a prefs-defined agent may not shadow. +// The run table is the authority; a test in that package keeps this list equal +// to it, so a new shipped row cannot become a silently shadowed operator name. +func BuiltinAgentNames() []string { + return []string{ + "agy", "amp", "claude", "codex", "copilot", "crush", "cursor", + "droid", "gemini", "goose", "kiro", "opencode", "pi", "qwen", + } +} + +// buildPersonalAgents validates the closed operator-agent shape. Names are +// read in sorted order so one malformed document always yields the same first +// diagnostic. Every failure names the agents key and the offending field. +func buildPersonalAgents(raw any) ([]PersonalAgent, *core.RefusalError) { + obj, ok := raw.(map[string]any) + if !ok { + return nil, agentRefuse(agentsPath, wantObjectMsg, jsonKind(raw)) + } + builtins := BuiltinAgentNames() + out := make([]PersonalAgent, 0, len(obj)) + for _, name := range slices.Sorted(maps.Keys(obj)) { + row, ref := buildAgentRow(name, obj[name], builtins) + if ref != nil { + return nil, ref + } + out = append(out, row) + } + return out, nil +} + +// buildAgentRow validates one named row and fills its defaults. +func buildAgentRow(name string, raw any, builtins []string) (PersonalAgent, *core.RefusalError) { + path := agentsPath + pathDot + name + if !agentNamePattern.MatchString(name) { + return PersonalAgent{}, agentRefuse(path, "want a name matching [a-z0-9][a-z0-9-]*") + } + if slices.Contains(builtins, name) { + return PersonalAgent{}, agentRefuse(path, "collides with a built-in agent; pick another name") + } + entry, ok := raw.(map[string]any) + if !ok { + return PersonalAgent{}, agentRefuse(path, wantObjectMsg, jsonKind(raw)) + } + binary, ref := agentBinary(entry, path) + if ref != nil { + return PersonalAgent{}, ref + } + args, ref := agentArgs(entry, path) + if ref != nil { + return PersonalAgent{}, ref + } + prompt, model, install, ref := agentOptionalFields(entry, path) + if ref != nil { + return PersonalAgent{}, ref + } + return PersonalAgent{ + Name: name, Binary: binary, Args: args, PromptFlag: prompt, + ModelFlag: model, Install: install, + }, nil +} + +func agentOptionalFields(entry map[string]any, path string) (prompt, model, install string, ref *core.RefusalError) { + if prompt, ref = agentOptionalString(entry, "prompt_flag", path); ref != nil { + return "", "", "", ref + } + if model, ref = agentOptionalString(entry, "model_flag", path); ref != nil { + return "", "", "", ref + } + install, ref = agentOptionalString(entry, "install", path) + return prompt, model, install, ref +} + +// agentBinary requires a non-blank name on PATH or an absolute path, so a +// prefs file cannot name a relative executable the agent might resolve inside +// a reviewed repository. +func agentBinary(entry map[string]any, path string) (string, *core.RefusalError) { + raw, ok := entry["binary"] + if !ok { + return "", agentRefuse(path+binaryField, "is required") + } + binary, ok := raw.(string) + if !ok { + return "", agentRefuse(path+binaryField, "want a string, got %s", jsonKind(raw)) + } + if strings.TrimSpace(binary) == "" { + return "", agentRefuse(path+binaryField, "want a non-blank string") + } + if strings.ContainsAny(binary, `/\`) && !filepath.IsAbs(binary) { + return "", agentRefuse(path+binaryField, "must be absolute when it names a path") + } + return binary, nil +} + +// agentArgs reads the optional argument list, defaulting to an empty one. +func agentArgs(entry map[string]any, path string) ([]string, *core.RefusalError) { + raw, ok := entry["args"] + if !ok { + return []string{}, nil + } + items, ok := raw.([]any) + if !ok { + return nil, agentRefuse(path+argsField, "want an array of strings, got %s", jsonKind(raw)) + } + out := make([]string, 0, len(items)) + for i, item := range items { + arg, ok := item.(string) + if !ok { + return nil, agentRefuse(arrayPath(path+argsField, i), "want a string, got %s", jsonKind(item)) + } + out = append(out, arg) + } + return out, nil +} + +// agentOptionalString reads one optional string field, defaulting to empty. +func agentOptionalString(entry map[string]any, key, path string) (string, *core.RefusalError) { + raw, ok := entry[key] + if !ok { + return "", nil + } + value, ok := raw.(string) + if !ok { + return "", agentRefuse(path+"."+key, "want a string, got %s", jsonKind(raw)) + } + return value, nil +} + // buildPersonalPacks validates the explicit org pack directory list. An // absent key reads as no org layer; there is no directory scan. func buildPersonalPacks(raw any) ([]string, *core.RefusalError) { @@ -299,7 +493,8 @@ func LoadPersonal(path string) (*PersonalSettings, *core.RefusalError) { // personalSections are the settings sections this parser owns. var personalSections = []string{ - personalSectionReview, personalSectionSpend, personalSectionPrivacy, personalSectionPacks, + personalSectionReview, personalSectionSpend, personalSectionPrivacy, + personalSectionPacks, personalSectionAgents, } // ValidatePersonalSections checks the four typed personal sections of an @@ -322,14 +517,53 @@ func ValidatePersonalSections(doc map[string]any) *core.RefusalError { return ref } -// credentialScanPersonal runs the credential scan over a personal -// document with no exemptions: settings carry no user-defined map keys. +// credentialScanPersonal runs the credential scan over a personal document. +// Agent names are user-defined keys: their direct keys are exempt, but each +// definition still scans as a fixed-shape object. func credentialScanPersonal(root map[string]any) *core.RefusalError { creds := make(map[string]bool, len(CredentialKeys)) for _, k := range CredentialKeys { creds[k] = true } - return scanCredentialObject(root, "$", creds, false) + for _, key := range sortedObjectKeys(root) { + if ref := scanPersonalCredentialKey(key, root[key], creds); ref != nil { + return ref + } + } + return nil +} + +func scanPersonalCredentialKey(key string, value any, creds map[string]bool) *core.RefusalError { + path := "$" + pathDot + key + if creds[key] { + return refuse(CodeBadCredentials, "%s: credential key %q is never stored in configuration", path, key) + } + if key == personalSectionAgents { + return scanAgentCredentialDefinitions(value, path, creds) + } + return scanCredentialValue(value, path, creds, false) +} + +func scanAgentCredentialDefinitions(raw any, path string, creds map[string]bool) *core.RefusalError { + defs, ok := raw.(map[string]any) + if !ok { + return scanCredentialValue(raw, path, creds, false) + } + for _, name := range sortedObjectKeys(defs) { + if ref := scanAgentDefinitionCredentials(name, defs[name], path, creds); ref != nil { + return ref + } + } + return nil +} + +func scanAgentDefinitionCredentials(name string, raw any, path string, creds map[string]bool) *core.RefusalError { + childPath := path + pathDot + name + def, ok := raw.(map[string]any) + if !ok { + return scanCredentialValue(raw, childPath, creds, false) + } + return scanCredentialObject(def, childPath, creds, false) } // RegistryPath returns the derived discovery registry path. diff --git a/internal/config/personal_agents_test.go b/internal/config/personal_agents_test.go new file mode 100644 index 000000000..e54f8ba4f --- /dev/null +++ b/internal/config/personal_agents_test.go @@ -0,0 +1,214 @@ +package config + +import ( + "encoding/json" + "path/filepath" + "slices" + "strings" + "testing" +) + +// TestParsePersonalAgents pins the operator-defined agent shape: the binary is +// required, args defaults to empty, the other fields are optional strings, and +// the definitions are returned in name order for stable discovery. +func TestParsePersonalAgents(t *testing.T) { + binary := filepath.Join(t.TempDir(), "min") + give := settingsJSON(t, map[string]any{ + "agents": map[string]any{ + "z-local": map[string]any{ + "binary": "local-agent", "args": []string{"exec", "-q"}, + "prompt_flag": "-p", "model_flag": "-m", + "install": "brew install local-agent", + }, + "a-min": map[string]any{"binary": binary}, + }, + }) + got, ref := ParsePersonal(give) + if ref != nil { + t.Fatalf("ParsePersonal: %v", ref) + } + if len(got.Agents) != 2 { + t.Fatalf("agents = %+v, want two rows", got.Agents) + } + assertMinimalAgent(t, &got.Agents[0], binary) + assertFullAgent(t, &got.Agents[1]) +} + +// settingsJSON marshals a fixture so absolute paths carry the OS separator +// without hand-escaping backslashes in a JSON literal. +func settingsJSON(t *testing.T, doc any) []byte { + t.Helper() + raw, err := json.Marshal(doc) + if err != nil { + t.Fatal(err) + } + return raw +} + +func assertMinimalAgent(t *testing.T, row *PersonalAgent, binary string) { + t.Helper() + if row.Name != "a-min" || row.Binary != binary || + len(row.Args) != 0 || row.PromptFlag != "" || row.ModelFlag != "" { + t.Errorf("row = %+v, want the defaults-filled minimal row with binary %q", row, binary) + } +} + +func assertFullAgent(t *testing.T, row *PersonalAgent) { + t.Helper() + if row.Name != "z-local" || row.Binary != "local-agent" || + !slices.Equal(row.Args, []string{"exec", "-q"}) || + row.PromptFlag != "-p" || row.ModelFlag != "-m" || + row.Install != "brew install local-agent" { + t.Errorf("row = %+v, want every field read verbatim", row) + } +} + +// TestPersonalAgentsRefusals pins every closed-shape rule. Each refusal names +// the prefs key and the offending field, and reuses bad-settings so the CLI +// renders it with the settings remediation. +func TestPersonalAgentsRefusals(t *testing.T) { + tests := []struct { + name string + want string + give []byte + }{ + { + name: "unknown agent key", + give: []byte(`{"agents":{"local":{"binary":"local","promt_flag":"-p"}}}`), + want: "$.agents.local.promt_flag", + }, + { + name: "empty binary", + give: []byte(`{"agents":{"local":{"binary":" "}}}`), + want: "$.agents.local.binary", + }, + { + name: "missing binary", + give: []byte(`{"agents":{"local":{"install":"brew install local"}}}`), + want: "$.agents.local.binary", + }, + { + name: "relative binary with a separator", + give: settingsJSON(t, map[string]any{"agents": map[string]any{ + "local": map[string]any{"binary": filepath.Join("bin", "x")}, + }}), + want: "$.agents.local.binary", + }, + { + name: "built-in collision", + give: []byte(`{"agents":{"claude":{"binary":"other"}}}`), + want: "$.agents.claude", + }, + { + name: "uppercase name", + give: []byte(`{"agents":{"Local":{"binary":"local"}}}`), + want: "$.agents.Local", + }, + { + name: "leading dash name", + give: []byte(`{"agents":{"-local":{"binary":"local"}}}`), + want: "$.agents.-local", + }, + { + name: "empty name", + give: []byte(`{"agents":{"":{"binary":"local"}}}`), + want: "$.agents.", + }, + { + name: "wrong agents shape", + give: []byte(`{"agents":[]}`), + want: "$.agents", + }, + { + name: "wrong binary type", + give: []byte(`{"agents":{"local":{"binary":3}}}`), + want: "$.agents.local.binary", + }, + { + name: "wrong args type", + give: []byte(`{"agents":{"local":{"binary":"local","args":"exec"}}}`), + want: "$.agents.local.args", + }, + { + name: "wrong optional string type", + give: []byte(`{"agents":{"local":{"binary":"local","prompt_flag":true}}}`), + want: "$.agents.local.prompt_flag", + }, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { assertAgentRefusal(t, tc.give, tc.want) }) + } +} + +func assertAgentRefusal(t *testing.T, give []byte, want string) { + t.Helper() + _, ref := ParsePersonal(give) + if ref == nil || ref.Code != "bad-settings" { + t.Fatalf("refusal = %+v, want bad-settings", ref) + } + if !strings.Contains(ref.Message, want) { + t.Errorf("message = %q, want it to name %q", ref.Message, want) + } +} + +// TestAgentsArePrefsOnly pins the boundary: the same key in a repository +// configuration refuses as an unknown field, so a PR cannot define an agent. +func TestAgentsArePrefsOnly(t *testing.T) { + _, ref := Parse([]byte(`{"schema_version":"1","agents":{"local":{"binary":"local"}}}`)) + if ref == nil || ref.Code != CodeBadConfig { + t.Fatalf("repo config refusal = %+v, want bad-config", ref) + } + if !strings.Contains(ref.Message, "agents") { + t.Errorf("message = %q, want it to name the agents key", ref.Message) + } +} + +// TestAgentDiagnosticsPreferLexicalName pins the first refusal when more than +// one agent row is malformed. Map iteration order must never decide the path. +func TestAgentDiagnosticsPreferLexicalName(t *testing.T) { + give := settingsJSON(t, map[string]any{"agents": map[string]any{ + "z": []any{}, "a": []any{}, + }}) + _, ref := ParsePersonal(give) + if ref == nil || ref.Code != CodeBadSettings { + t.Fatalf("refusal = %+v, want bad-settings", ref) + } + if !strings.Contains(ref.Message, "$.agents.a") { + t.Errorf("message = %q, want the lexically first agent a", ref.Message) + } +} + +// TestAgentNamesAreNotCredentialKeys pins the boundary between a user-defined +// agent name and a field name. The name is data; fields still scan. +func TestAgentNamesAreNotCredentialKeys(t *testing.T) { + for _, name := range []string{"token", "patchbot"} { + t.Run(name, func(t *testing.T) { assertAgentNameAccepted(t, name) }) + } +} + +func assertAgentNameAccepted(t *testing.T, name string) { + t.Helper() + give := settingsJSON(t, map[string]any{"agents": map[string]any{ + name: map[string]any{"binary": "local-agent"}, + }}) + got, ref := ParsePersonal(give) + if ref != nil { + t.Fatalf("ParsePersonal refused agent name %q: %v", name, ref) + } + if len(got.Agents) != 1 || got.Agents[0].Name != name { + t.Errorf("agents = %+v, want the %q definition", got.Agents, name) + } +} + +func TestCredentialFieldInsideAgentStillRefuses(t *testing.T) { + give := settingsJSON(t, map[string]any{"agents": map[string]any{ + "local": map[string]any{"binary": "local-agent", "token": "abc"}, + }}) + _, ref := ParsePersonal(give) + if ref == nil || ref.Code != CodeBadCredentials { + t.Fatalf("refusal = %+v, want bad-config-credentials", ref) + } + if !strings.Contains(ref.Message, "$.agents.local.token") { + t.Errorf("message = %q, want the field inside the agent", ref.Message) + } +} diff --git a/internal/core/hint.go b/internal/core/hint.go index ea91163c1..9babe4e12 100644 --- a/internal/core/hint.go +++ b/internal/core/hint.go @@ -156,8 +156,8 @@ var hintRules = []HintRule{ {codes: []string{"needs-confirm"}, hint: "try: re-run with --yes after reviewing the preview, then re-run"}, {codes: []string{"bad-settings-credentials"}, hint: "try: remove the credential key from the settings file " + "(credentials never live there), then re-run"}, - {codes: []string{"bad-settings"}, hint: "try: keep only output/layout/color/publish/review/spend/privacy keys " + - "(see docs/manual.md), then re-run"}, + {codes: []string{"bad-settings"}, hint: "try: keep only " + + strings.Join(SettingsTopKeys(), "/") + " keys (see docs/manual.md), then re-run"}, {codes: []string{"controls-unavailable"}, hint: "try: check out the PR base (actions/checkout fetch-depth: 0, " + "or git fetch --depth=1 origin ), then re-fetch; fetch --allow-head-controls is the explicit opt-out"}, {codes: []string{"controls-stale"}, hint: "try: re-fetch the bundle (controls changed under it), then re-run"}, diff --git a/internal/core/run/agents.go b/internal/core/run/agents.go index cf0907fc8..4f5ce69ca 100644 --- a/internal/core/run/agents.go +++ b/internal/core/run/agents.go @@ -1,11 +1,13 @@ package run import ( + "slices" "strings" ) -// agentDef is one launch definition: the --agent name an operator types, the -// binary it spawns, the flag that binary takes a model on, and the +// agentDef is one launch definition: the --agent name an operator types, +// the binary it spawns, the flag that binary takes a model on, the flag +// that takes its prompt as a value (empty for a positional prompt), and the // non-interactive argument shape that binary uses to answer once and exit. // // The launcher reads every field and branches on none, so adding an agent is @@ -13,44 +15,119 @@ import ( // row obeys is the argument order the launcher builds (see [argvFor]): // // [ ] [] +// [ ] [] // // The agent's own non-interactive flags come first so the operator's // pass-through arguments cannot cancel them, the operator's flags come next so // per-agent model/cost selection stays the operator's business, and the -// tool-generated directive is the last argument. A binary that wants its prompt -// somewhere else (a file path, stdin) does not fit this shape yet. ModelFlag is -// read only by a --print --stage render, which prints the stage's declared hint -// as that argument; a launch never composes one. +// tool-generated directive is the last argument. A PromptFlag sits immediately +// before that directive, so a pass-through flag can never be swallowed as its +// value. A binary that wants its prompt somewhere else (a file path, stdin) +// does not fit this shape yet. ModelFlag is read only by a --print --stage +// render, which prints the stage's declared hint as that argument; a launch +// never composes one. type agentDef struct { - Name string - Binary string - Install string - ModelFlag string - Args []string + Name string + Binary string + Install string + ModelFlag string + PromptFlag string + Source string + Args []string } // modelFlag is the argument every shipped row takes a model name on; a row // whose binary spells it differently carries its own value. const modelFlag = "--model" +// agentSourcePrefs marks a definition read from the operator's prefs file in +// the provenance record. A shipped row leaves Source empty. +const agentSourcePrefs = "prefs" + +// The four names whose prefix stays first in discovery messages, and the +// subcommand shared by the rows that launch one prompt through a run verb. +const ( + nameClaude = "claude" + nameCodex = "codex" + nameCursor = "cursor" + nameOpenCode = "opencode" + argsRun = "run" +) + // agentDefs is the shipped agent table, sorted by name. Every row is data: a // new agent is a new entry here, and nothing else in the launcher changes. +// Permission and approval flags stay with the operator via `-- `. var agentDefs = []agentDef{ + // https://github.com/google-antigravity/antigravity-cli — read 2026-11-17 + { + Name: "agy", Binary: "agy", ModelFlag: modelFlag, PromptFlag: "-p", + Install: "curl -fsSL https://antigravity.google/cli/install.sh | bash", + }, + // https://ampcode.com/docs/markdown/cli/execute-mode — read 2026-11-17 + { + Name: "amp", Binary: "amp", ModelFlag: modelFlag, PromptFlag: "-x", + Install: "npm install -g @ampcode/cli", + }, + // https://code.claude.com/docs/en/cli-reference.md — read 2026-11-17 { - Name: "claude", Binary: "claude", ModelFlag: modelFlag, Args: []string{"-p"}, - Install: "install claude using its official installer", + Name: nameClaude, Binary: nameClaude, ModelFlag: modelFlag, Args: []string{"-p"}, + Install: "curl -fsSL https://claude.ai/install.sh | bash", }, + // https://github.com/openai/codex/blob/main/README.md — read 2026-11-17 { - Name: "codex", Binary: "codex", ModelFlag: modelFlag, Args: []string{"exec"}, + Name: nameCodex, Binary: nameCodex, ModelFlag: modelFlag, Args: []string{"exec"}, Install: "npm install -g @openai/codex", }, + // https://docs.github.com/en/copilot/how-tos/copilot-cli/automate-copilot-cli/run-cli-programmatically — read 2026-11-17 + { + Name: "copilot", Binary: "copilot", ModelFlag: modelFlag, PromptFlag: "-p", + Install: "npm install -g @github/copilot", + }, + // https://github.com/charmbracelet/crush/blob/main/internal/cmd/run.go — read 2026-11-17 + { + Name: "crush", Binary: "crush", ModelFlag: modelFlag, Args: []string{argsRun}, + Install: "brew install charmbracelet/tap/crush", + }, + // https://cursor.com/docs/cli/headless — read 2026-11-17 { Name: "cursor", Binary: "cursor-agent", ModelFlag: modelFlag, Args: []string{"-p"}, - Install: "install cursor-agent using the Cursor CLI installer", + Install: "curl https://cursor.com/install -fsS | bash", }, + // https://docs.factory.com/droid-cli/cli-reference.md — read 2026-11-17 { - Name: "opencode", Binary: "opencode", ModelFlag: modelFlag, Args: []string{"run"}, - Install: "npm install -g opencode-ai", + Name: "droid", Binary: "droid", ModelFlag: modelFlag, Args: []string{"exec"}, + Install: "curl -fsSL https://app.factory.ai/cli | sh", + }, + // https://github.com/google-gemini/gemini-cli/blob/main/README.md — read 2026-11-17 + { + Name: "gemini", Binary: "gemini", ModelFlag: modelFlag, PromptFlag: "-p", + Install: "npm install -g @google/gemini-cli", + }, + // https://github.com/aaif-goose/goose/blob/main/documentation/docs/guides/running-tasks.md — read 2026-11-17 + { + Name: "goose", Binary: "goose", ModelFlag: modelFlag, PromptFlag: "-t", Args: []string{argsRun}, + Install: "brew install block-goose-cli", + }, + // https://kiro.dev/docs/cli/headless.md — read 2026-11-17 + { + Name: "kiro", Binary: "kiro-cli", ModelFlag: modelFlag, + Args: []string{"chat", "--no-interactive"}, + Install: "curl -fsSL https://cli.kiro.dev/install | bash", + }, + // https://opencode.ai/docs/cli/ — read 2026-11-17 + { + Name: nameOpenCode, Binary: nameOpenCode, ModelFlag: modelFlag, Args: []string{argsRun}, + Install: "npm install -g @opencode/cli", + }, + // https://github.com/badlogic/pi-mono/blob/main/packages/coding-agent/docs/quickstart.md — read 2026-11-17 + { + Name: "pi", Binary: "pi", ModelFlag: modelFlag, Args: []string{"-p"}, + Install: "curl -fsSL https://pi.dev/install.sh | sh", + }, + // https://github.com/QwenLM/qwen-code/blob/main/packages/cli/src/config/top-level-options.ts — read 2026-11-17 + { + Name: "qwen", Binary: "qwen", ModelFlag: modelFlag, + Install: "npm install -g @qwen-code/qwen-code@latest", }, } @@ -64,28 +141,42 @@ func lookupAgent(name string) (agentDef, bool) { return agentDef{}, false } -// AgentNames returns the known agent names in table order, for the -// unknown-agent refusal and the usage block. +// AgentNames returns the known agent names, for the unknown-agent refusal and +// the other verbs that list them. The four rows shipped before STA-440 keep +// their prefix, so a caller that matched that fragment still finds it; the rest +// follow in table order. func AgentNames() []string { + first := []string{nameClaude, nameCodex, nameCursor, nameOpenCode} names := make([]string, 0, len(agentDefs)) + for _, name := range first { + if _, ok := lookupAgent(name); ok { + names = append(names, name) + } + } for _, def := range agentDefs { - names = append(names, def.Name) + if !slices.Contains(first, def.Name) { + names = append(names, def.Name) + } } return names } -// argvFor builds the exact argv the launcher spawns, and the same argv --print -// renders: the binary, the definition's non-interactive flags, the model hint a -// --print --stage render declares (empty for everything else), the operator's -// pass-through arguments unchanged, then the directive. +// argvFor builds the exact argv the launcher spawns, and the same argv +// --print renders: the binary, the definition's non-interactive flags, the +// model hint a --print --stage render declares (empty for everything else), +// the operator's pass-through arguments unchanged, an optional prompt flag, +// then the directive. func argvFor(def *agentDef, model string, passthrough []string, prompt string) []string { var argv []string argv = append(argv, def.Binary) argv = append(argv, def.Args...) - if model != "" { + if model != "" && def.ModelFlag != "" { argv = append(argv, def.ModelFlag, model) } argv = append(argv, passthrough...) + if def.PromptFlag != "" { + argv = append(argv, def.PromptFlag) + } return append(argv, prompt) } diff --git a/internal/core/run/agents_internal_test.go b/internal/core/run/agents_internal_test.go new file mode 100644 index 000000000..812284b77 --- /dev/null +++ b/internal/core/run/agents_internal_test.go @@ -0,0 +1,196 @@ +package run + +import ( + "slices" + "testing" +) + +type builtinRow struct { + name string + binary string + args []string + promptFlag string + install string + wantArgv []string +} + +// TestBuiltinAgentRows pins every shipped row: its binary, its non-interactive +// flags, its official install hint, and the exact argv a model hint and a +// pass-through flag produce. Prompt-bearing rows put their flag immediately +// before the directive; the others keep the positional shape. The name set is +// literal, so a new row cannot join the table without a reviewed expectation. +func TestBuiltinAgentRows(t *testing.T) { + tests := []builtinRow{ + { + name: "agy", binary: "agy", promptFlag: "-p", + install: "curl -fsSL https://antigravity.google/cli/install.sh | bash", + wantArgv: []string{"agy", "--model", "M", "--x", "-p", "PROMPT"}, + }, + { + name: "amp", binary: "amp", promptFlag: "-x", + install: "npm install -g @ampcode/cli", + wantArgv: []string{"amp", "--model", "M", "--x", "-x", "PROMPT"}, + }, + { + name: "claude", binary: "claude", args: []string{"-p"}, + install: "curl -fsSL https://claude.ai/install.sh | bash", + wantArgv: []string{"claude", "-p", "--model", "M", "--x", "PROMPT"}, + }, + { + name: "codex", binary: "codex", args: []string{"exec"}, + install: "npm install -g @openai/codex", + wantArgv: []string{"codex", "exec", "--model", "M", "--x", "PROMPT"}, + }, + { + name: "copilot", binary: "copilot", promptFlag: "-p", + install: "npm install -g @github/copilot", + wantArgv: []string{"copilot", "--model", "M", "--x", "-p", "PROMPT"}, + }, + { + name: "crush", binary: "crush", args: []string{"run"}, + install: "brew install charmbracelet/tap/crush", + wantArgv: []string{"crush", "run", "--model", "M", "--x", "PROMPT"}, + }, + { + name: "cursor", binary: "cursor-agent", args: []string{"-p"}, + install: "curl https://cursor.com/install -fsS | bash", + wantArgv: []string{"cursor-agent", "-p", "--model", "M", "--x", "PROMPT"}, + }, + { + name: "droid", binary: "droid", args: []string{"exec"}, + install: "curl -fsSL https://app.factory.ai/cli | sh", + wantArgv: []string{"droid", "exec", "--model", "M", "--x", "PROMPT"}, + }, + { + name: "gemini", binary: "gemini", promptFlag: "-p", + install: "npm install -g @google/gemini-cli", + wantArgv: []string{"gemini", "--model", "M", "--x", "-p", "PROMPT"}, + }, + { + name: "goose", binary: "goose", args: []string{"run"}, promptFlag: "-t", + install: "brew install block-goose-cli", + wantArgv: []string{"goose", "run", "--model", "M", "--x", "-t", "PROMPT"}, + }, + { + name: "kiro", binary: "kiro-cli", args: []string{"chat", "--no-interactive"}, + install: "curl -fsSL https://cli.kiro.dev/install | bash", + wantArgv: []string{"kiro-cli", "chat", "--no-interactive", "--model", "M", "--x", "PROMPT"}, + }, + { + name: "opencode", binary: "opencode", args: []string{"run"}, + install: "npm install -g @opencode/cli", + wantArgv: []string{"opencode", "run", "--model", "M", "--x", "PROMPT"}, + }, + { + name: "pi", binary: "pi", args: []string{"-p"}, + install: "curl -fsSL https://pi.dev/install.sh | sh", + wantArgv: []string{"pi", "-p", "--model", "M", "--x", "PROMPT"}, + }, + { + name: "qwen", binary: "qwen", + install: "npm install -g @qwen-code/qwen-code@latest", + wantArgv: []string{"qwen", "--model", "M", "--x", "PROMPT"}, + }, + } + wantNames := make([]string, 0, len(tests)) + for _, tc := range tests { + wantNames = append(wantNames, tc.name) + t.Run(tc.name, func(t *testing.T) { assertBuiltinRow(t, &tc) }) + } + names := make([]string, 0, len(agentDefs)) + for _, def := range agentDefs { + names = append(names, def.Name) + } + if !slices.Equal(names, wantNames) { + t.Errorf("table names = %q, want %q", names, wantNames) + } +} + +func assertBuiltinRow(t *testing.T, tc *builtinRow) { + t.Helper() + def, ok := lookupAgent(tc.name) + if !ok { + t.Fatalf("lookupAgent(%q) found no row", tc.name) + } + if def.Binary != tc.binary || def.PromptFlag != tc.promptFlag || def.Install != tc.install { + t.Errorf("row = %+v, want binary %q, prompt flag %q, install %q", + def, tc.binary, tc.promptFlag, tc.install) + } + if !slices.Equal(def.Args, tc.args) { + t.Errorf("args = %q, want %q", def.Args, tc.args) + } + if def.ModelFlag != modelFlag { + t.Errorf("model flag = %q, want %q", def.ModelFlag, modelFlag) + } + got := argvFor(&def, "M", []string{"--x"}, "PROMPT") + if !slices.Equal(got, tc.wantArgv) { + t.Errorf("argv = %q, want %q", got, tc.wantArgv) + } +} + +// TestArgvForPromptPlacement pins the one shape change in STA-440: a row that +// takes its prompt as a flag value puts that flag immediately before the +// directive, after every operator argument, so a pass-through flag cannot be +// swallowed as its value. Rows without PromptFlag keep the positional shape. +func TestArgvForPromptPlacement(t *testing.T) { + tests := []struct { + name string + def agentDef + want []string + }{ + { + name: "positional prompt", + def: agentDef{ + Name: "claude", Binary: "claude", Args: []string{"-p"}, ModelFlag: modelFlag, + }, + want: []string{"claude", "-p", "--model", "M", "--x", "PROMPT"}, + }, + { + name: "flag value prompt", + def: agentDef{ + Name: "amp", Binary: "amp", Args: nil, PromptFlag: "-x", ModelFlag: modelFlag, + }, + want: []string{"amp", "--model", "M", "--x", "-x", "PROMPT"}, + }, + { + name: "flag value prompt after a subcommand", + def: agentDef{ + Name: "goose", Binary: "goose", Args: []string{"run"}, PromptFlag: "-t", ModelFlag: modelFlag, + }, + want: []string{"goose", "run", "--model", "M", "--x", "-t", "PROMPT"}, + }, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + def := tc.def + got := argvFor(&def, "M", []string{"--x"}, "PROMPT") + if !slices.Equal(got, tc.want) { + t.Fatalf("argv = %q, want %q", got, tc.want) + } + }) + } +} + +// TestArgvForOmitsEmptyFields pins the no-model, no-passthrough shape: no +// empty argument is ever handed to a binary, with or without PromptFlag. +func TestArgvForOmitsEmptyFields(t *testing.T) { + tests := []struct { + def agentDef + want []string + }{ + { + def: agentDef{Name: "codex", Binary: "codex", Args: []string{"exec"}, ModelFlag: modelFlag}, + want: []string{"codex", "exec", "PROMPT"}, + }, + { + def: agentDef{Name: "amp", Binary: "amp", PromptFlag: "-x", ModelFlag: modelFlag}, + want: []string{"amp", "-x", "PROMPT"}, + }, + } + for _, tc := range tests { + def := tc.def + if got := argvFor(&def, "", nil, "PROMPT"); !slices.Equal(got, tc.want) { + t.Errorf("%s argv = %q, want %q", def.Name, got, tc.want) + } + } +} diff --git a/internal/core/run/operators_internal_test.go b/internal/core/run/operators_internal_test.go new file mode 100644 index 000000000..3d331c6b4 --- /dev/null +++ b/internal/core/run/operators_internal_test.go @@ -0,0 +1,129 @@ +package run + +import ( + "slices" + "strings" + "testing" + + "github.com/stainless-code/verdict/internal/config" + "github.com/stainless-code/verdict/internal/jq" +) + +func operatorDef(name string) config.PersonalAgent { + return config.PersonalAgent{ + Name: name, Binary: "local-agent", Args: []string{"exec"}, + PromptFlag: "-t", ModelFlag: "-m", Install: "brew install local-agent", + } +} + +// TestBuiltinNamesMatchTheConfigList keeps the prefs parser's collision list +// equal to the shipped table: a new row that the parser does not refuse as a +// name would let a prefs file shadow it. +func TestBuiltinNamesMatchTheConfigList(t *testing.T) { + got, want := config.BuiltinAgentNames(), AgentNames() + slices.Sort(got) + slices.Sort(want) + if !slices.Equal(got, want) { + t.Errorf("config collision list = %q, run table = %q", got, want) + } +} + +// TestOperatorAgentLaunchAndPrint pins the custom-row contract: argv shape, +// pass-through, prompt placement, print parity, and the provenance marker that +// says where the definition came from. +func TestOperatorAgentLaunchAndPrint(t *testing.T) { + defs := []config.PersonalAgent{operatorDef("local")} + root := fixtureRoot(t) + dir := bundleDir(root) + printReq := request(root, &scripted{}) + printReq.Agent = "local" + printReq.PassThrough = []string{"--max-turns", "3"} + printReq.Print = true + printed, ref := WithAgents(t.Context(), printReq, defs) + if ref != nil { + t.Fatalf("print refused: %v", ref) + } + want := []string{"local-agent", "exec", "--max-turns", "3", "-t", printed.Data.Argv[len(printed.Data.Argv)-1]} + if !slices.Equal(printed.Data.Argv, want) { + t.Errorf("printed argv = %q, want %q", printed.Data.Argv, want) + } + + cmdr := &scripted{write: func() { writeReview(t, dir) }} + launchReq := request(root, cmdr) + launchReq.Agent = "local" + launchReq.PassThrough = []string{"--max-turns", "3"} + if _, ref := WithAgents(t.Context(), launchReq, defs); ref != nil { + t.Fatalf("launch refused: %v", ref) + } + if cmdr.cmd == nil || !slices.Equal(printed.Data.Argv, cmdr.cmd.Argv) { + t.Fatalf("printed argv = %q, spawned %+v", printed.Data.Argv, cmdr.cmd) + } + rec := readLauncher(t, dir) + if got := jq.StrOf(rec["agent_source"]); got != "prefs" { + t.Errorf("launcher agent_source = %q, want prefs", got) + } + if got := jq.StrOf(rec["agent"]); got != "local" { + t.Errorf("launcher agent = %q, want local", got) + } +} + +// TestOperatorAgentMissingBinaryKeepsItsInstallHint proves a custom row uses +// the built-in missing-binary refusal and carries its own install step. +func TestOperatorAgentMissingBinaryKeepsItsInstallHint(t *testing.T) { + root := fixtureRoot(t) + req := request(root, &scripted{}) + req.Agent = "local" + req.LookPath = missingLookPath + _, ref := WithAgents(t.Context(), req, []config.PersonalAgent{operatorDef("local")}) + if ref == nil || ref.Code != codeAgentMissing { + t.Fatalf("refusal = %+v, want agent-missing", ref) + } + if !strings.Contains(ref.Message, "brew install local-agent") { + t.Errorf("message = %q, want the definition's install hint", ref.Message) + } +} + +// TestOperatorAgentDiscoveryLabelsBothSources pins the unknown-agent list: +// built-in names keep their prefix and the prefs names are labeled separately. +func TestOperatorAgentDiscoveryLabelsBothSources(t *testing.T) { + root := fixtureRoot(t) + req := request(root, &scripted{}) + req.Agent = "nope" + _, ref := WithAgents(t.Context(), req, []config.PersonalAgent{operatorDef("local")}) + if ref == nil || ref.Code != codeUnknownAgent { + t.Fatalf("refusal = %+v, want unknown-agent", ref) + } + for _, fragment := range []string{ + "known: claude, codex, cursor, opencode", + "operator-defined in prefs: local", + } { + if !strings.Contains(ref.Message, fragment) { + t.Errorf("message = %q, want %q", ref.Message, fragment) + } + } +} + +// TestOperatorAgentCollisionRefusesSettings pins the no-shadowing rule in the +// service too, for callers that supply definitions without the prefs parser. +func TestOperatorAgentCollisionRefusesSettings(t *testing.T) { + root := fixtureRoot(t) + req := request(root, &scripted{}) + req.Agent = "claude" + _, ref := WithAgents(t.Context(), req, []config.PersonalAgent{operatorDef("claude")}) + if ref == nil || ref.Code != codeBadSettings { + t.Fatalf("refusal = %+v, want bad-settings", ref) + } + if !strings.Contains(ref.Message, "claude") { + t.Errorf("message = %q, want the colliding name", ref.Message) + } +} + +// TestOperatorAgentWithoutModelFlag proves an optional model flag renders no +// model argument, rather than an empty argument the binary would misread. +func TestOperatorAgentWithoutModelFlag(t *testing.T) { + def := agentDef{Name: "local", Binary: "local-agent", ModelFlag: "", PromptFlag: "-t"} + if got := argvFor(&def, "M", []string{"--x"}, "PROMPT"); !slices.Equal(got, + []string{"local-agent", "--x", "-t", "PROMPT"}) { + t.Errorf("argv = %q, want no model argument", got) + } +} diff --git a/internal/core/run/record.go b/internal/core/run/record.go index 039b9574e..4e217e289 100644 --- a/internal/core/run/record.go +++ b/internal/core/run/record.go @@ -68,6 +68,7 @@ type launcherRepair struct { type launcher struct { //nolint:govet // field order is the published property order SchemaVersion string `json:"schema_version"` Agent string `json:"agent"` + AgentSource string `json:"agent_source,omitempty"` Binary string `json:"binary"` Argv []string `json:"argv"` Bundle string `json:"bundle"` diff --git a/internal/core/run/redact_internal_test.go b/internal/core/run/redact_internal_test.go index 662eacbd2..449bf4f19 100644 --- a/internal/core/run/redact_internal_test.go +++ b/internal/core/run/redact_internal_test.go @@ -249,20 +249,38 @@ func TestDirectiveKeepsAPercentPathLiteral(t *testing.T) { // TestAgentTableIsSortedAndUnique keeps the data table honest: duplicate or // unsorted rows would make the known-name list a lie. func TestAgentTableIsSortedAndUnique(t *testing.T) { - names := AgentNames() - if !slices.IsSorted(names) { - t.Errorf("agent table is not sorted: %v", names) + table := tableNames() + if !slices.IsSorted(table) { + t.Errorf("agent table is not sorted: %v", table) } - if len(slices.Compact(slices.Clone(names))) != len(names) { - t.Errorf("agent table repeats a name: %v", names) + if len(slices.Compact(slices.Clone(table))) != len(table) { + t.Errorf("agent table repeats a name: %v", table) } - for _, name := range names { + assertKnownNamesEnumerateTable(t, table) + for _, name := range table { def, ok := lookupAgent(name) - if !ok || def.Binary == "" || len(def.Args) == 0 { - t.Errorf("agent %q has no binary or no non-interactive argument shape: %+v", name, def) + if !ok || def.Binary == "" || def.ModelFlag == "" || def.Install == "" { + t.Errorf("agent %q has no binary, model flag, or install hint: %+v", name, def) } } if _, ok := lookupAgent("gpt"); ok { t.Error("lookupAgent resolved an agent that is not in the table") } } + +func tableNames() []string { + names := make([]string, 0, len(agentDefs)) + for _, def := range agentDefs { + names = append(names, def.Name) + } + return names +} + +func assertKnownNamesEnumerateTable(t *testing.T, table []string) { + t.Helper() + known := AgentNames() + slices.Sort(known) + if !slices.Equal(known, table) { + t.Errorf("known names %v do not enumerate the table %v", known, table) + } +} diff --git a/internal/core/run/repair.go b/internal/core/run/repair.go index 735b24e70..62533e075 100644 --- a/internal/core/run/repair.go +++ b/internal/core/run/repair.go @@ -77,6 +77,7 @@ func (st *state) repairLoop(ctx context.Context, args *launchArgs, n int) (*core rec := &launcher{ SchemaVersion: launcherSchema, Agent: args.def.Name, + AgentSource: args.def.Source, Binary: bin, Bundle: args.bundle, Artifact: args.artifact, diff --git a/internal/core/run/service.go b/internal/core/run/service.go index ae44f4f00..2cd947273 100644 --- a/internal/core/run/service.go +++ b/internal/core/run/service.go @@ -17,6 +17,7 @@ import ( "os" "os/exec" "path/filepath" + "slices" "strconv" "strings" "time" @@ -39,6 +40,9 @@ const ( codeStaleArtifact = "stale-review-artifact" codeUnknownAgent = "unknown-agent" codeRecordFailed = "agent-record-failed" + // codeBadSettings is reused from the settings vocabulary: a prefs-defined + // agent that collides with a shipped name is a bad settings file. + codeBadSettings = "bad-settings" ) // Reused refusal codes: the shared vocabulary for "you canceled", "there is no @@ -69,7 +73,33 @@ const ( // Run launches the configured agent against the bundle, one shot by default // and up to --rounds launch-to-check rounds when the operator asks. func Run(ctx context.Context, req *core.RunRequest) (resp *core.RunResponse, ref *core.RefusalError) { - st := &state{req: req, warnings: append([]string{}, req.Warnings...)} + return WithAgents(ctx, req, nil) +} + +// WithAgents adds the operator-defined rows read from the prefs file to the +// shipped table for one invocation. Definitions are values: the caller owns +// the slice, and the service never mutates it. +func WithAgents( + ctx context.Context, req *core.RunRequest, agents []config.PersonalAgent, +) (resp *core.RunResponse, ref *core.RefusalError) { + seen := make(map[string]bool, len(agents)) + for _, def := range agents { + path := "$.agents." + def.Name + if _, ok := lookupAgent(def.Name); ok { + return nil, &core.RefusalError{ + Code: codeBadSettings, + Message: path + ": collides with a built-in agent; pick another name", + } + } + if seen[def.Name] { + return nil, &core.RefusalError{ + Code: codeBadSettings, + Message: path + ": defined twice; use one agent per name", + } + } + seen[def.Name] = true + } + st := &state{req: req, warnings: append([]string{}, req.Warnings...), defs: agents} return execute(ctx, st) } @@ -262,6 +292,7 @@ type state struct { argSecrets []string drainDigest string warnings []string + defs []config.PersonalAgent } // launchArgs is the resolved invocation one launch works from: the definition, @@ -389,17 +420,55 @@ func (st *state) bundle() (string, *core.RefusalError) { readcommon.DisplayPath(st.req.RepoRootPath(), dir), st.req.Repo, st.req.Pr)) } -// agent resolves the --agent value against the shipped definition table. An -// absent or unknown name refuses with the known list: the table is data, so the -// message is the only place an operator learns it can grow. +// lookupAgent resolves one name against the shipped table, then the prefs +// definitions read for this invocation. Collisions refuse before this point. +func (st *state) lookupAgent(name string) (agentDef, bool) { + if def, ok := lookupAgent(name); ok { + return def, true + } + for _, def := range st.defs { + if def.Name == name { + return operatorAgent(&def), true + } + } + return agentDef{}, false +} + +// operatorAgent projects one prefs row onto the same data shape as a shipped +// row. The source marker travels to launcher.json so provenance can say where +// the definition came from. +func operatorAgent(def *config.PersonalAgent) agentDef { + return agentDef{ + Name: def.Name, Binary: def.Binary, Install: def.Install, + ModelFlag: def.ModelFlag, PromptFlag: def.PromptFlag, + Args: slices.Clone(def.Args), Source: agentSourcePrefs, + } +} + +// knownAgents labels both sources in the refusal and the missing-name hint. +func (st *state) knownAgents() string { + message := "known: " + knownAgents() + "; operator-defined in prefs: " + if len(st.defs) == 0 { + return message + "none" + } + names := make([]string, 0, len(st.defs)) + for _, def := range st.defs { + names = append(names, def.Name) + } + return message + strings.Join(names, ", ") +} + +// agent resolves the --agent value against the shipped table and the operator +// definitions. An absent or unknown name refuses with both sources labeled: the +// message is the only place an operator learns the table can grow. func (st *state) agent() (agentDef, *core.RefusalError) { if st.req.Agent == "" { - return agentDef{}, st.refuse(codeBadFlags, "run needs --agent (known: "+knownAgents()+")") + return agentDef{}, st.refuse(codeBadFlags, "run needs --agent ("+st.knownAgents()+")") } - def, ok := lookupAgent(st.req.Agent) + def, ok := st.lookupAgent(st.req.Agent) if !ok { return agentDef{}, st.refuse(codeUnknownAgent, fmt.Sprintf( - "unknown agent '%s' (known: %s)", st.req.Agent, knownAgents())) + "unknown agent '%s' (%s)", st.req.Agent, st.knownAgents())) } return def, nil } @@ -481,6 +550,7 @@ func (st *state) launch(ctx context.Context, args *launchArgs) (*core.RunRespons rec := &launcher{ SchemaVersion: launcherSchema, Agent: args.def.Name, + AgentSource: args.def.Source, Binary: bin, Recipe: args.plan, Bundle: args.bundle, @@ -509,8 +579,13 @@ func (st *state) resolve(def *agentDef) (string, *core.RefusalError) { } bin, err := look(def.Binary) if err != nil { - return "", st.refuse(codeAgentMissing, fmt.Sprintf( - "agent '%s' is not installed: no '%s' on PATH", def.Name, def.Binary)+"; "+def.Install) + message := fmt.Sprintf("agent '%s' is not installed: no '%s' on PATH", def.Name, def.Binary) + if def.Install != "" { + message += "; " + def.Install + } else { + message += "; put it on PATH" + } + return "", st.refuse(codeAgentMissing, message) } return bin, nil } diff --git a/internal/core/settingskeys.go b/internal/core/settingskeys.go new file mode 100644 index 000000000..6dce6a77e --- /dev/null +++ b/internal/core/settingskeys.go @@ -0,0 +1,11 @@ +package core + +// SettingsTopKeys lists every top-level key the operator settings file +// accepts. The prefs loader, the typed personal parser, the unknown-key +// message, and the bad-settings hint all render from this one list. +func SettingsTopKeys() []string { + return []string{ + "$schema", "output", "layout", "color", "publish", + "review", "spend", "privacy", "packs", "agents", + } +} diff --git a/internal/core/testdata/hints.golden b/internal/core/testdata/hints.golden index 691931a8b..5082c5b78 100644 --- a/internal/core/testdata/hints.golden +++ b/internal/core/testdata/hints.golden @@ -54,7 +54,7 @@ bad-root try: pass --root , or run serve inside a checkout bad-rule try: re-run with one of the valid values named above bad-sarif try: pass a valid UTF-8 SARIF 2.1.0 JSON file with repository-relative URIs bad-scope try: re-run with one of the valid values named above -bad-settings try: keep only output/layout/color/publish/review/spend/privacy keys (see docs/manual.md), then re-run +bad-settings try: keep only $schema/output/layout/color/publish/review/spend/privacy/packs/agents keys (see docs/manual.md), then re-run bad-settings-credentials try: remove the credential key from the settings file (credentials never live there), then re-run bad-sig try: list open inline sigs with report --status open --kind inline, then re-run with the full prefix bad-since try: pass --since <90d|12w|18mo|2y|ref> (a bare m is not a unit, and the ref must precede --until) diff --git a/internal/record/testdata/overrides/bad-placeholder.golden b/internal/record/testdata/overrides/bad-placeholder.golden index 0c98a549f..3f89d0026 100644 --- a/internal/record/testdata/overrides/bad-placeholder.golden +++ b/internal/record/testdata/overrides/bad-placeholder.golden @@ -5,7 +5,7 @@ exit: 1 warn: no fetch-time hunk snapshot (.verdict/state/local-ov/hunks.tsv missing) — rebuilding from full.diff, validation is best-effort (re-fetch to pin the snapshot) warn: $.body: no Basis line in the leading quote block — inserted the computed counts (basis-replaced) error: override .verdict/overrides/preamble.md carries unknown placeholder '{frobnicate}' (want one of {version}, {tier}, {product_url}, {files_reviewed}, {head_sha_short}, {model}, {cost}, {attribution}) -try: keep only output/layout/color/publish/review/spend/privacy keys (see docs/manual.md), then re-run +try: keep only $schema/output/layout/color/publish/review/spend/privacy/packs/agents keys (see docs/manual.md), then re-run re-enter: verdict record acme/app local-ov --- files --- --- file C/.verdict/state/local-ov/hunks.tsv --- diff --git a/internal/run/run.go b/internal/run/run.go index 19ac2a123..1372203c0 100644 --- a/internal/run/run.go +++ b/internal/run/run.go @@ -21,6 +21,7 @@ import ( "strings" "github.com/stainless-code/verdict/internal/cli" + "github.com/stainless-code/verdict/internal/config" "github.com/stainless-code/verdict/internal/core" coreexec "github.com/stainless-code/verdict/internal/core/exec" runcore "github.com/stainless-code/verdict/internal/core/run" @@ -69,7 +70,11 @@ func Run(ctx context.Context, r *cli.Runner, args []string, req Request) int { if code, done := r.Canceled(ctx, "run"); done { return code } - resp, ref := runcore.Run(ctx, runRequest(r, req, &flags)) + agents, ref := personalAgents(req.Env) + if ref != nil { + return refuse(r, nil, &core.RefusalError{Code: ref.Code, Message: ref.Message}) + } + resp, ref := runcore.WithAgents(ctx, runRequest(r, req, &flags), agents) if ref != nil { return refuse(r, resp, ref) } @@ -85,6 +90,17 @@ func Run(ctx context.Context, r *cli.Runner, args []string, req Request) int { return cli.ExitOK } +// personalAgents reads operator-defined agents from the prefs file only. A +// repository config can never define one: the typed repo parser refuses the +// key, and this read addresses only the XDG/HOME settings path. +func personalAgents(env map[string]string) ([]config.PersonalAgent, *core.RefusalError) { + prefs, ref := config.LoadPersonal(config.SettingsPath(env["HOME"], env["XDG_CONFIG_HOME"])) + if ref != nil { + return nil, ref + } + return prefs.Agents, nil +} + // refuse renders a run refusal, preserving the agent's own exit code when the // service reports one (only the agent-failed refusal carries it). func refuse(r *cli.Runner, resp *core.RunResponse, ref *core.RefusalError) int { diff --git a/internal/run/run_test.go b/internal/run/run_test.go index 37071a471..f4351fd88 100644 --- a/internal/run/run_test.go +++ b/internal/run/run_test.go @@ -1,14 +1,26 @@ package run import ( + "os" + "path/filepath" "slices" "strings" "testing" "time" "github.com/stainless-code/verdict/internal/cli" + "github.com/stainless-code/verdict/internal/config" ) +func readFile(t *testing.T, path string) []byte { + t.Helper() + raw, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + return raw +} + // testRunner builds the run verb's runner around buffers the caller can // inspect. A test that drives another verb, or JSON mode, adjusts the runner. func testRunner() (*cli.Runner, *strings.Builder, *strings.Builder) { @@ -346,3 +358,38 @@ func TestRentryKeepsStage(t *testing.T) { t.Fatalf("rentry named a stage nobody asked for: %q", plain) } } + +// TestPersonalAgentsReadsPrefsOnly pins the adapter's source: definitions come +// from the XDG/HOME settings file, and a repository config carrying the same +// key cannot define one. +func TestPersonalAgentsReadsPrefsOnly(t *testing.T) { + xdg := t.TempDir() + settings := filepath.Join(xdg, "verdict", "settings.json") + if err := os.MkdirAll(filepath.Dir(settings), 0o700); err != nil { + t.Fatal(err) + } + doc := `{"agents":{"local":{"binary":"local-agent","args":["exec"],"prompt_flag":"-p"}}}` + if err := os.WriteFile(settings, []byte(doc), 0o600); err != nil { + t.Fatal(err) + } + defs, ref := personalAgents(map[string]string{"XDG_CONFIG_HOME": xdg}) + if ref != nil { + t.Fatalf("personalAgents: %v", ref) + } + assertSinglePrefsAgent(t, defs) + repo := filepath.Join(t.TempDir(), "config.json") + if err := os.WriteFile(repo, []byte(`{"schema_version":"1","agents":{"local":{"binary":"local"}}}`), 0o600); err != nil { + t.Fatal(err) + } + if _, ref := config.Parse(readFile(t, repo)); ref == nil || ref.Code != config.CodeBadConfig { + t.Fatalf("repo config with agents = %v, want bad-config", ref) + } +} + +func assertSinglePrefsAgent(t *testing.T, defs []config.PersonalAgent) { + t.Helper() + if len(defs) != 1 || defs[0].Name != "local" || defs[0].Binary != "local-agent" || + !slices.Equal(defs[0].Args, []string{"exec"}) || defs[0].PromptFlag != "-p" { + t.Fatalf("definitions = %+v, want the prefs row", defs) + } +} diff --git a/scripts/size-baseline.txt b/scripts/size-baseline.txt index dd324dc48..b2ec03c80 100644 --- a/scripts/size-baseline.txt +++ b/scripts/size-baseline.txt @@ -1,7 +1,7 @@ deps 0 -darwin/amd64 17540000 -darwin/arm64 15940000 -linux/amd64 17240000 +darwin/amd64 17570000 +darwin/arm64 15970000 +linux/amd64 17270000 linux/arm64 15600000 -windows/amd64 17630000 -windows/arm64 15720000 +windows/amd64 17660000 +windows/arm64 15750000 diff --git a/skills/verdict/references/commands.md b/skills/verdict/references/commands.md index a7868564d..7a1b39229 100644 --- a/skills/verdict/references/commands.md +++ b/skills/verdict/references/commands.md @@ -59,6 +59,25 @@ GitHub and GitLab only; a write failure records `failed[]` and refuses `drain-incomplete`, after which the same `--post` command retries only the threads without a ledger entry. +## `run --agent` — launch your own agent + +`verdict run owner/repo --agent [--recipe ] +[--stage ] [--timeout ] [--rounds ] [--print] [-- ...]` +launches one agent process against an existing bundle and hands the reviewed +artifact back to `record` or `post`. It never posts, and `run --print` renders +the exact argv without spawning or writing. Everything after `--` reaches the +agent first; a flag that takes the prompt as its value sits immediately before +the directive, so your flags cannot be swallowed as that value. + +Names come from the shipped table (`agy`, `amp`, `claude`, `codex`, `copilot`, +`crush`, `cursor`, `droid`, `gemini`, `goose`, `kiro`, `opencode`, `pi`, `qwen`) +or from the `agents` key of your own `${XDG_CONFIG_HOME:-$HOME/.config}/verdict/settings.json`. +The prefs shape is `{"agents": {"": {"binary": "...", "args": [], +"prompt_flag": "...", "model_flag": "...", "install": "..."}}}`; `binary` is +required, `args` defaults to `[]`, and names cannot collide with a shipped row. +A repository config cannot define one. `unknown-agent` lists both groups, and +`launcher.json` marks prefs-defined rows with `agent_source: "prefs"`. + ## Suppression receipts Read `excluded.json` and the TOC exclusion section before judging review coverage.