Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .changeset/sta-445-schemas.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
"@stainless-code/verdict": minor
---

**Breaking changes**

Published JSON schemas moved from `skills/verdict/schemas/` to `/schemas/`; every `$id` changed accordingly.
4 changes: 2 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,11 +70,11 @@ rename in the tree. The binary installs its own embedded skill copy with

```
verdict/
skills/verdict/ # the skill payload, exactly what `verdict skill files` lists:
skills/verdict/ # the skill payload copied by `npx skills add`:
SKILL.md # skill entry point (feed to any agent)
prompts/ # system, review, incremental, address, fix-ci, build, ci, grade, diagrams, subagent
schemas/ # JSON Schemas for the envelope and store files
scripts/verdict # POSIX skill shim: VERDICT_BIN, bundled bin, PATH, then bootstrap
schemas/ # published JSON Schemas, served by `verdict skill files` at schemas/
install.sh # verified binary installer (macOS/Linux release asset)
install.ps1 # verified binary installer (Windows release asset)
install-skill.sh # doctor-gated symlink installer for checkouts
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,7 @@ Inputs are `version` (default `latest`) and `require-signature` (default `true`)
- **MCP.** `verdict mcp` serves the review verbs as tools over stdio. `verdict mcp config --agent claude` prints the config entry, and `--write` merges it into the file. It supports `claude`, `cursor`, and `codex`, and is experimental until 1.0. See the [manual](docs/manual.md#mcp-server-stdio-experimental-until-10).
- **Go SDK.** [`pkg/verdict`](pkg/verdict) runs the verbs in-process for Go programs, with typed requests and responses — no `verdict` subprocess, no `--json` parsing.
- **TypeScript client.** `import { createClient } from "@stainless-code/verdict/client"`. It ships in the npm package. See [`clients/ts`](clients/ts/README.md).
- **Other languages.** Every `--json` envelope and store file has a JSON Schema in [`skills/verdict/schemas/`](skills/verdict/schemas). The [Build a client guide](https://stainless-code.com/verdict/build-a-client) covers exit codes and codegen, with a tested [Python example](examples/python-client/).
- **Other languages.** Every `--json` envelope and store file has a JSON Schema in [`schemas/`](schemas). The [Build a client guide](https://stainless-code.com/verdict/build-a-client) covers exit codes and codegen, with a tested [Python example](examples/python-client/).

## Hosts

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ verdict report acme/app 123 --json

Every successful `--json` run prints one object with these nine keys, in this
order. The [JSON envelope and exit codes](/docs/json-envelope) reference pins this table with the exit codes. The
[envelope schema](https://github.com/stainless-code/verdict/blob/main/skills/verdict/schemas/envelope.schema.json)
[envelope schema](https://github.com/stainless-code/verdict/blob/main/schemas/envelope.schema.json)
is generated from the Go shape and is the machine-readable source of truth.

| Key | Type | Meaning |
Expand Down Expand Up @@ -153,7 +153,7 @@ release version.
- **A major change needs a new `schema_version`.** A client must reject an
unknown major instead of guessing. The schemas README
(`make schemas`) remains the source of truth:
[schemas/README.md](https://github.com/stainless-code/verdict/blob/main/skills/verdict/schemas/README.md).
[schemas/README.md](https://github.com/stainless-code/verdict/blob/main/schemas/README.md).
- **The clients, the Go SDK, and the MCP tool shapes are experimental until
1.0.** Their ergonomics may change without a major schema bump. The envelope
contract does not.
Expand Down Expand Up @@ -253,7 +253,7 @@ The profiles are `$defs` entries, not a `oneOf` on `data`: the envelope keeps
one open `data` object so a client that does not know a verb can still read it.

The schemas are published under
[`skills/verdict/schemas/`](https://github.com/stainless-code/verdict/tree/main/skills/verdict/schemas). They
[`schemas/`](https://github.com/stainless-code/verdict/tree/main/schemas). They
also ship in the npm package and inside the binary. `verdict skill files` lists
them. There is no hosted schema URL yet, so link the repo path.
`verdict mcp` serves the verbs over stdio and has no `--json` envelope of its
Expand Down Expand Up @@ -302,7 +302,7 @@ python3 -m venv .venv
pip install -r examples/python-client/requirements-dev.txt

datamodel-codegen \
--input skills/verdict/schemas/envelope.schema.json \
--input schemas/envelope.schema.json \
--input-file-type jsonschema \
--output examples/python-client/models.py \
--output-model-type dataclasses.dataclass \
Expand Down
2 changes: 1 addition & 1 deletion apps/docs/content/01-docs/(reference)/json-envelope.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ A failure keeps `data` null and names a typed `error.code` with its cause and fi

## Schemas

The shapes live in `skills/verdict/schemas/`, generated from `internal/schemagen/model.go` and shipped in the npm tarball. `envelope.schema.json` covers the envelope plus the 21 pinned `<verb>Data` profiles (including `report`, `post`, and `record`). Pin one by vendoring the file at a release tag and generating types from it — the Python client example checks in `models.py` generated exactly this way — and refuse at runtime when `schema_version` is not `"1"`.
The shapes live in `schemas/`, generated from `internal/schemagen/model.go` and shipped in the npm tarball. `envelope.schema.json` covers the envelope plus the 21 pinned `<verb>Data` profiles (including `report`, `post`, and `record`). Pin one by vendoring the file at a release tag and generating types from it — the Python client example checks in `models.py` generated exactly this way — and refuse at runtime when `schema_version` is not `"1"`.

## Worked example

Expand Down
6 changes: 3 additions & 3 deletions apps/docs/content/01-docs/(start)/glossary.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -46,9 +46,9 @@ tools. Commands accept a unique short prefix.

## Surfaces

**Skill.** The instructions an agent loads to run a review: `SKILL.md`,
`prompts/`, and `schemas/`. The binary carries its own copy, and
`verdict skill install` writes it into an agent's skills directory.
**Skill.** The instructions an agent loads to run a review: `SKILL.md` and
`prompts/`. The binary carries its own copy and the published `schemas/`;
`verdict skill install` writes them into an agent's skills directory.

**Envelope.** The one JSON object a `--json` run prints to stdout. It has nine
fixed keys and `schema_version: "1"`. New information goes inside `data`. See
Expand Down
2 changes: 1 addition & 1 deletion apps/docs/content/04-roadmap/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ Planned means it is not built yet, and says so on its own line.
| Housekeeping | `init`, `config`, `pack`, `learn`, `upgrade`, `uninstall` | Repo config, explain policy, lens packs (`pack list`, `pack test`, `pack add`), project rules with provenance, and channel-aware upgrades and removals. |

Clients and actions over those verbs: the Go SDK (`pkg/verdict`), the TypeScript
client (`@stainless-code/verdict/client`), the JSON schemas in `skills/verdict/schemas/`, and
client (`@stainless-code/verdict/client`), the JSON schemas in `schemas/`, and
`setup-verdict` for GitHub Actions.

<Callout type="warning" title="Experimental until 1.0">
Expand Down
4 changes: 2 additions & 2 deletions bench/corpus/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ bundle with ground truth:

```
bench/corpus/<case-id>/
case.json # ground truth + provenance (skills/verdict/schemas/corpus.schema.json)
case.json # ground truth + provenance (schemas/corpus.schema.json)
full.diff # real git diff of the reviewed change, with the bug lines in hunks
snapshots/<file>.before # the reviewed base tree
snapshots/<file>.after # the reviewed head tree
Expand Down Expand Up @@ -124,7 +124,7 @@ bench/corpus/<repo>-<pr>-<slug>/
real case never enters the hermetic rates. Real cases are scored by live runs
under `.verdict/state/`, as `docs/benchmark.md` §2 describes.

Do not edit `skills/verdict/schemas/corpus.schema.json` by hand; run `make schemas`.
Do not edit `schemas/corpus.schema.json` by hand; run `make schemas`.

The operator runbook for real PRs, real agents, cost estimates, and
publishing measured rates lives in [`docs/benchmark.md`](../../docs/benchmark.md).
2 changes: 1 addition & 1 deletion clients/ts/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ createClient({

`src/types.gen.ts` (envelope `$defs`) and the platform → npm package map
(`src/platforms.gen.ts` for the client, `scripts/npm-shim/platforms.gen.json`
for the install shim) are rendered from `skills/verdict/schemas/envelope.schema.json` and
for the install shim) are rendered from `schemas/envelope.schema.json` and
`release/contract.json`:

```sh
Expand Down
7 changes: 2 additions & 5 deletions clients/ts/scripts/gen-ts-types.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -195,10 +195,7 @@ function mapTS(node, defs) {

function genTypes() {
const schema = JSON.parse(
readFileSync(
join(REPO_ROOT, "skills", "verdict", "schemas", "envelope.schema.json"),
"utf8",
),
readFileSync(join(REPO_ROOT, "schemas", "envelope.schema.json"), "utf8"),
);
const defs = hoist(schema.$defs ?? {});
const chunks = [];
Expand All @@ -215,7 +212,7 @@ function genTypes() {
);
}
}
return `${header("skills/verdict/schemas/envelope.schema.json ($defs)")}\n${chunks.join("\n\n")}\n`;
return `${header("schemas/envelope.schema.json ($defs)")}\n${chunks.join("\n\n")}\n`;
}

function genPlatforms() {
Expand Down
2 changes: 1 addition & 1 deletion clients/ts/src/types.gen.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// GENERATED from skills/verdict/schemas/envelope.schema.json ($defs) by clients/ts/scripts/gen-ts-types.mjs. Do not edit.
// GENERATED from schemas/envelope.schema.json ($defs) by clients/ts/scripts/gen-ts-types.mjs. Do not edit.
export interface Truncation {
truncated: boolean;
truncation_reason:
Expand Down
8 changes: 4 additions & 4 deletions cmd/schemagen/main.go
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
// Command schemagen writes (or checks) the published schemas from the Go
// shape types in internal/schemagen. Run it through `make schemas` or
// `go generate ./...`; never edit skills/verdict/schemas/*.schema.json by hand.
// `go generate ./...`; never edit schemas/*.schema.json by hand.
// In check mode it also re-checks the additivity baseline under
// internal/schemagen/testdata/baseline byte for byte.
package main
Expand All @@ -25,7 +25,7 @@ const storeDirMode = 0o755
// schemas, relative to the repository root, each with the target that rewrites
// it.
const (
shippedDir = "skills/verdict/schemas"
shippedDir = "schemas"
baselineDir = "internal/schemagen/testdata/baseline"
shippedHint = "run `make schemas` from the repo root"
baselineHint = "run `make schemas-baseline` from the repo root"
Expand Down Expand Up @@ -141,8 +141,8 @@ func samePath(a, b string) bool {
func parseFlags(args []string, stderr io.Writer) (options, error) {
flags := flag.NewFlagSet("schemagen", flag.ContinueOnError)
flags.SetOutput(stderr)
write := flags.Bool("write", false, "rewrite skills/verdict/schemas/*.schema.json instead of checking them")
dir := flags.String("dir", "", "schemas directory (default: <repo>/skills/verdict/schemas)")
write := flags.Bool("write", false, "rewrite schemas/*.schema.json instead of checking them")
dir := flags.String("dir", "", "schemas directory (default: <repo>/schemas)")
if err := flags.Parse(args); err != nil {
return options{}, wrapError("parse flags", err)
}
Expand Down
2 changes: 1 addition & 1 deletion docs/adding-a-verb.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ A verb served before argument parsing (`mcp`) belongs in `run` and in the
(`model.go`, `overlay.json` descriptions) and pin it in
`conformance_verbs_test.go`; the envelope's `data` stays free-form for a
verb that adds no contract.
- `make schemas` rewrites `skills/verdict/schemas/*.schema.json`;
- `make schemas` rewrites `schemas/*.schema.json`;
`make schemas-check` proves no drift. Any shape the generator emits, additive
or breaking, also needs `make schemas-baseline` and a reviewed baseline diff.
- `make tsgen` rewrites the Go-derived `clients/ts/src/refusal-codes.gen.ts`;
Expand Down
2 changes: 1 addition & 1 deletion docs/benchmark.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ attributable per-model rates.
## Corpus format

One case, one manifest, published as
[`schemas/corpus.schema.json`](../skills/verdict/schemas/corpus.schema.json):
[`schemas/corpus.schema.json`](../schemas/corpus.schema.json):

```json
{
Expand Down
2 changes: 1 addition & 1 deletion docs/go-rules.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ this repo, and the repo files stay the truth for what is gated:
- `os.Root` confinement; `crypto/rand` secrets; validated edge input (GO-SEC-01/02/04). — partly: `lint:depguard` (`math/rand`), `lint:gosec`, `lint:nosprintfhostport`; `os.Root` is review-only, and operator-supplied bundle paths are exempt by decision.
- Exit-code contract; `--json/--jq` exporters; generated completions/manpages (GO-CLI-02/06/07). — partly: exit codes are frozen constants in `internal/cli` with tests; no generated completions or manpages.
- Start std-`flag` under a zero-dep policy; graduate to cobra when justified (GO-CLI-08). — partly: `go.mod` carries no third-party requirements; argv parsing is hand-rolled in `internal/cli` to keep the frozen contract byte-identical.
- `embed` prompts; stamped `--version` + `-trimpath` builds (GO-CLI-09). — partly: `skillfs.go` embeds `skills/verdict/` (`SKILL.md`, `prompts`, `schemas`, `scripts/verdict`); `make build` and goreleaser pass `-trimpath`.
- `embed` prompts; stamped `--version` + `-trimpath` builds (GO-CLI-09). — partly: `skillfs.go` embeds `skills/verdict/` (`SKILL.md`, `prompts`, `scripts/verdict`) and the root `schemas/`, serving both at payload paths; `make build` and goreleaser pass `-trimpath`.
- flags > env > file > defaults; report secret sources; lenient config reads (GO-CONFIG-01/02/03). — partly: `internal/cli` precedence + `internal/cli/flags_test.go`.
- Advisory `flock` across read-modify-write; JSON schema libs where beyond `DisallowUnknownFields` (GO-ATOMIC-02, GO-JSON-04); tag linters apply to new types only, legacy wire fields frozen (GO-JSON-03). — partly: `lint:musttag,tagalign`; the rest is review-only.
- Progress as stderr attrs + JSON error shapes; log after flag parsing (GO-OBS-01/02). — partly: envelope error-shape goldens.
Expand Down
20 changes: 10 additions & 10 deletions docs/manual.md
Original file line number Diff line number Diff line change
Expand Up @@ -288,7 +288,7 @@ whole store — plugins read it instead of sniffing shapes. GitHub bundles keep
unversioned); the other hosts reshape `reviews.json`/`inline.json` per host
and normalize `PR.json` to one schema. Every provider bundle also writes
`threads.json` (`schema_version: "1"`, published as
[`schemas/threads.schema.json`](https://github.com/stainless-code/verdict/blob/main/skills/verdict/schemas/threads.schema.json)): the raw
[`schemas/threads.schema.json`](https://github.com/stainless-code/verdict/blob/main/schemas/threads.schema.json)): the raw
files stay as-is, and the feed is the authoritative PR conversation. Review
threads come from each host's native source — GitHub GraphQL `reviewThreads`
(REST has no resolution), GitLab discussions (`resolved`/`resolvable`), Azure
Expand Down Expand Up @@ -867,7 +867,7 @@ posture and never gate alone.
## Typed repository configuration

`.verdict/config.json` is the only repository configuration file:
one strict document (schema `skills/verdict/schemas/config.schema.json`) for
one strict document (schema `schemas/config.schema.json`) for
defaults (`panel`, `budget_usd`, `max_bytes`, `chunk`, `kb_policy`, `effort`,
the two severity floors), the local `index` caps (`files`, `file_bytes`,
`tree_bytes`, `shards_bytes`, `keep_trees`, `refs_per_file`), blocking
Expand Down Expand Up @@ -2767,12 +2767,12 @@ platform package. A plain `verdict` call consults neither variable.
`install-skill.sh` refuses outside a git checkout. `hooks/pre-commit`
wiring applies to the checkout only.
- **Embedded skill** (`verdict skill`): the binary carries its own `SKILL.md`,
`prompts/`, `schemas/`, `packs/`, and `scripts/verdict`, so the served skill can
never skew from the binary. In the repository they are the one
`skills/verdict/` directory, which holds nothing else. The skills CLI
(`npx skills add stainless-code/verdict`) copies that directory and no
more, so the registry install and `verdict skill install` write the same
files. `verdict skill` prints `SKILL.md` (default
`prompts/`, `packs/`, and `scripts/verdict` from `skills/verdict/`, plus
`schemas/` from the repository root, so the served skill can never skew from
the binary. The skills CLI
(`npx skills add stainless-code/verdict`) copies only `skills/verdict/`;
`verdict skill install` writes that payload and the served `schemas/` files.
`verdict skill` prints `SKILL.md` (default
`--format md`); `--format json` is one standard envelope with
`name`/`version`/`files`; `verdict skill files` lists every embedded path.
`verdict skill show <path>` prints one inventory file byte-identical (a
Expand Down Expand Up @@ -3122,7 +3122,7 @@ live ground truth.
Static-analysis ingestion: `ingest owner/repo N --sarif <file|->` maps SARIF 2.1.0
results into `.verdict/state/<bundle>/findings.json` with `source: "tool"`,
sharing one store, one dedupe path, and one suppression/withheld path with model
reviews. Severity is mapped via `skills/verdict/schemas/sarif-map.json`
reviews. Severity is mapped via `schemas/sarif-map.json`
(`security-severity` threshold or SARIF `level`), anchors are resolved against
the bundle diff (`anchored: false` when outside hunks so out-of-diff results
never fail inline posting), excluded paths land withheld with `source: "exclusion"`,
Expand Down Expand Up @@ -4641,7 +4641,7 @@ The metrics:
`--json` `data` is `{schema_version, run, mode, prs, not_replayed, bugs,
reported, matched, unclassified, catch_rate, off_target_per_pr,
cost_per_review_usd, spend, signals}` — and `report.json` is that same object,
so [`schemas/backtest.schema.json`](../skills/verdict/schemas/backtest.schema.json)
so [`schemas/backtest.schema.json`](../schemas/backtest.schema.json)
describes both. `spend` is `{usd, priced_reviews, unpriced_reviews}`; a signal
row is `{signal, bugs, matched, catch_rate}`. Human output is the scope header,
the catch rate, the off-target rate, the cost line, the unclassified count, one
Expand Down
2 changes: 1 addition & 1 deletion evals/cases/bench-corpus.sh
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ stub_before="$(wc -l <"$EVAL_STUBLOG" | tr -d ' ')"
schema_bad=0
for case_dir in "$EVAL_ROOT"/bench/corpus/*/; do
[ -f "$case_dir/case.json" ] || continue
if ! "$EVAL_SKILL/schemas/validate.sh" "$EVAL_SKILL/schemas/corpus.schema.json" "$case_dir/case.json" >/dev/null 2>&1; then
if ! "$EVAL_ROOT/schemas/validate.sh" "$EVAL_ROOT/schemas/corpus.schema.json" "$case_dir/case.json" >/dev/null 2>&1; then
bad "corpus.schema.json rejects $case_dir/case.json"
schema_bad=$((schema_bad + 1))
fi
Expand Down
4 changes: 2 additions & 2 deletions evals/cases/drain-plan.sh
Original file line number Diff line number Diff line change
Expand Up @@ -105,10 +105,10 @@ if jq -e --arg sig "$SIG" '
else
bad "P2 plan: $(jq -c '{pending,skipped,other_threads}' "$PLAN" 2>/dev/null | head -c 600)"
fi
if "$EVAL_SKILL/schemas/validate.sh" "$EVAL_SKILL/schemas/drain.schema.json" "$PLAN" >/dev/null 2>&1; then
if "$EVAL_ROOT/schemas/validate.sh" "$EVAL_ROOT/schemas/drain.schema.json" "$PLAN" >/dev/null 2>&1; then
good "P3 the plan validates against schemas/drain.schema.json"
else
bad "P3 schema: $("$EVAL_SKILL/schemas/validate.sh" "$EVAL_SKILL/schemas/drain.schema.json" "$PLAN" 2>&1 | head -3)"
bad "P3 schema: $("$EVAL_ROOT/schemas/validate.sh" "$EVAL_ROOT/schemas/drain.schema.json" "$PLAN" 2>&1 | head -3)"
fi
assert_stream_nogrep "P4 zero host writes on the stub log" "$EVAL_STUBLOG" \
'POST|PATCH|mutation|method[[:space:]]+:([[:space:]]*)?"(POST|PATCH|DELETE)' -Eqi
Expand Down
6 changes: 5 additions & 1 deletion evals/cases/install-routes.sh
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,11 @@ done
for root in "$TARGET/.agents/skills" "$HOME/.agents/skills"; do
while IFS= read -r rel; do
[ -n "$rel" ] || continue
cmp -s "$PKG/skills/verdict/$rel" "$root/verdict/$rel" || copy_ok="false"
case "$rel" in
schemas/*) src="$PKG/$rel" ;;
*) src="$PKG/skills/verdict/$rel" ;;
esac
cmp -s "$src" "$root/verdict/$rel" || copy_ok="false"
done <<<"$inventory"
[ "$(jq -r '.tool // empty' "$root/verdict/.verdict-managed" 2>/dev/null)" = "verdict" ] || copy_ok="false"
[ "$(jq -r '.version // empty' "$root/verdict/.verdict-managed" 2>/dev/null)" = "$(jq -r .version "$EVAL_ROOT/package.json")" ] || copy_ok="false"
Expand Down
Loading
Loading