Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .changeset/03-review-quality-and-provenance.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,3 +25,7 @@
- **Validation pass** — a trusted `prompts/validate.md` pass writes `validation.json` per sig, `rigor_policy` floors each severity tier, and `post --require-rigor` narrows it per invocation: advisory by default, a receipted withhold in gate mode. [Rigor floors](docs/manual.md#rigor-floors-and-the-validation-pass) <!-- changesets: rigor-validation.md -->
- **Published key names** — `prompts/review.md` tags the evidence rung as `rigor` and the in-bounds flag as `in-bounds`, and an archtest holds every published key to `review.schema.json`. [Rigor floors](docs/manual.md#rigor-floors-and-the-validation-pass) <!-- changesets: prompt-rigor-key.md, prompt-key-shapes.md -->
- **Verdict reconciliation** — binding verdicts reconcile separately from summary and inline content, so a retried approval does not duplicate a GitLab summary note. [Submit per host](docs/manual.md#submit-post-per-host) <!-- changesets: post-verdict-reconciliation.md -->
- **Plain-body Basis line** — `record` leaves a `review.json` body with no leading quote block byte-identical and reports no `basis-replaced` warning, because there is no confidence lead to insert under. [Review-input problems](docs/manual.md#review-input-problems) <!-- changesets: sta-425-basis-plain-body.md -->
- **Array-valued coverage ranges** — `next --coverage` refuses a `coverage[].ranges` array on plain and chunked bundles with the same `bad-review-input` problem `record` reports, instead of reading it as a silent 0% attestation. [Diff coverage](docs/manual.md#diff-coverage-agent-attestation) <!-- changesets: sta-425-coverage-ranges.md -->
- **Passing gate next step** — a passing `report --check <tier>` names `merge — gate passes` in the envelope's `next` and human footer, leaving `address open threads` for a failing gate. [Machine surface](docs/manual.md#machine-surface) <!-- changesets: sta-425-gate-next.md -->
- **More generated paths** — `*.gen.*` and `zz_generated.*` now count as generated in the shared role heuristic, so `next --coverage` and the risk floor exempt them like the other generated patterns. [Diff coverage](docs/manual.md#diff-coverage-agent-attestation) <!-- changesets: sta-425-generated-exemption.md -->
2 changes: 2 additions & 0 deletions .changeset/11-fixes.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,5 @@
- **`pr-wait` inconsistent reads** — a head that is not yet the same on both APIs is retried instead of failing the bounded wait, and `inconsistent_observations` keeps the retry visible. [Bounded foreground waits](docs/manual.md#bounded-foreground-waits) <!-- changesets: pr-wait-inconsistent-reads.md -->
- **Windows watchdog resume** — a command that failed before starting under load retries the thread lookup instead of refusing `resume suspended verify process`. [Bounded foreground waits](docs/manual.md#bounded-foreground-waits) <!-- changesets: windows-watchdog-resume-retry.md -->
- **`backtest mine` in a partial clone** — a `--filter=blob:none` clone refuses `partial-clone` with the step that fetches the missing objects, instead of stalling. [Backtest](docs/manual.md#backtest-szz-candidates) <!-- changesets: backtest-partial-clone.md -->
- **Quiet local live skip** — a local bundle's live recheck writes one `info:` line to the human stream instead of a warning or a `--json` `warnings[]` entry; failed remote reads still warn. [Machine surface](docs/manual.md#machine-surface) <!-- changesets: sta-425-local-skip-info.md -->
- **Optional review schema version** — `record` accepts a `review.json` without `schema_version` and reads it as version 1, while a present value other than `"1"` still refuses. [Review-input problems](docs/manual.md#review-input-problems) <!-- changesets: sta-425-optional-schema-version.md -->
4 changes: 2 additions & 2 deletions apps/docs/pages/_home/HowItWorks.astro
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,8 @@ const sample = [
{ kind: "out", text: " 878a93e0 open true src/retry.ts" },
{ kind: "cmd", text: "verdict report acme/app local-work --check important" },
{
kind: "dim",
text: "warn: live recheck skipped: local bundle threads unreadable here (gating on local dimensions only)",
kind: "out",
text: "info: live recheck skipped: local bundle threads unreadable here (gating on local dimensions only)",
},
{ kind: "out", text: "report: .verdict/state/local-work (1 shown) open=1 head=" },
{ kind: "out", text: "1. [open] src/retry.ts:3 (878a93e0)" },
Expand Down
91 changes: 91 additions & 0 deletions cmd/verdict/stdout_contract_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
package main

import (
"encoding/json"
"errors"
"io"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
)

func TestJSONVerbsWriteOneEnvelopeToStdout(t *testing.T) {
hermeticEnv(t)
root := t.TempDir()
t.Chdir(root)
base, head := gitRangeFixture(t, root)
if got := runOne(t, "fetch-local", "acme/app", "stdout", "--range", base+".."+head, "--json"); got.code != 0 {
t.Fatalf("fetch-local exit = %d\nstdout: %s\nstderr: %s", got.code, got.out, got.err)
}
writeReviewJSON(t, ".verdict/state/local-stdout/review.json")
for _, args := range [][]string{
{"next", "acme/app", "local-stdout", "--coverage", "--json"},
{"record", "acme/app", "local-stdout", "--json"},
{"report", "acme/app", "local-stdout", "--check", "important", "--max-risk", "medium", "--json"},
} {
got := runOne(t, args...)
if got.code != 0 {
t.Fatalf("%v exit = %d\nstderr: %s", args, got.code, got.err)
}
assertOneJSONValue(t, args[0], got.out)
}
}

func assertOneJSONValue(t *testing.T, verb, out string) {
t.Helper()
dec := json.NewDecoder(strings.NewReader(out))
var first any
if err := dec.Decode(&first); err != nil {
t.Fatalf("%s stdout is not a JSON value: %v\n%s", verb, err, out)
}
var extra any
if err := dec.Decode(&extra); !errors.Is(err, io.EOF) {
t.Fatalf("%s stdout carries more than one JSON value: %v\n%s", verb, err, out)
}
}

func writeReviewJSON(t *testing.T, path string) {
t.Helper()
body := `{"schema_version":"1","body":"## Summary\nA clean review body with enough text and no quote lead.","approved":false,"comments":[]}`
if err := os.WriteFile(path, []byte(body), 0o644); err != nil {
t.Fatal(err)
}
}

func gitRangeFixture(t *testing.T, dir string) (base, head string) {
t.Helper()
git := func(args ...string) string {
cmd := exec.CommandContext(t.Context(), "git", args...)
cmd.Dir = dir
cmd.Env = append(os.Environ(), "GIT_CONFIG_NOSYSTEM=1")
out, err := cmd.CombinedOutput()
if err != nil {
t.Fatalf("git %v: %v\n%s", args, err, out)
}
return string(out)
}
git("init", "-q")
notes := filepath.Join(dir, "notes.txt")
if err := os.WriteFile(notes, []byte("one\ntwo\nthree\n"), 0o644); err != nil {
t.Fatal(err)
}
git("add", "notes.txt")
git("-c", "user.name=t", "-c", "user.email=t@example.com", "commit", "-qm", "one")
base = git("rev-parse", "HEAD")
if err := os.WriteFile(notes, []byte("one\nTWO\nthree\n"), 0o644); err != nil {
t.Fatal(err)
}
git("add", "notes.txt")
git("-c", "user.name=t", "-c", "user.email=t@example.com", "commit", "-qm", "two")
head = git("rev-parse", "HEAD")
return trimNewline(base), trimNewline(head)
}

func trimNewline(s string) string {
for len(s) > 0 && (s[len(s)-1] == '\n' || s[len(s)-1] == '\r') {
s = s[:len(s)-1]
}
return s
}
78 changes: 44 additions & 34 deletions docs/manual.md
Original file line number Diff line number Diff line change
Expand Up @@ -239,28 +239,30 @@ an unparsable or non-object document, a comment with no recoverable string
coverage still refuses. `record --dry-run` adds `data.normalized` (the exact
object the checker read) and `data.repairs`; verdict never rewrites
`review.json` on disk. The CLI also owns the body's `> Basis:` line: it
replaces a disagreeing line, or inserts a missing one, with the counts computed
from the review and reports `basis-replaced`.

| Code | Level | Meaning |
| ---------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| `invalid-json` | error | The file is not one JSON document; the message carries line:col, and the fix is to emit only the object. |
| `wrong-type` | mixed | Error for a wrong type; warning for a non-numeric line/start_line or a fenced suggestion. |
| `missing-field` | mixed | Error for a required field; warning for missing schema_version, comments, or line/start_line. |
| `bad-enum` | mixed | Error for an unknown closed-set value; warning for a malformed KB id or unknown chunk name. |
| `out-of-range` | error | A length or numeric bound is outside its published range (confidence, brief approach/question, caps). |
| `path-not-in-diff` | mixed | Error for a comment path; warning for a brief site outside the diff. |
| `unknown-field` | warning | A key is not in the published shape; the fix names the nearest known key. |
| `risk-stale-head` | warning | `review.json.risk.assessed_head` disagrees with the bundle's current HEAD SHA; the stored claim is marked `stale: true`. |
| `anchor-outside-hunk` | warning | The line is provably outside every hunk; `record` moves the comment into the body. |
| `rigor-unset` | warning | A comment carries no evidence rung; it records as `said` with `rigor_source: default`. |
| `normalized-fence` | warning | One whole-document code fence was unwrapped. |
| `normalized-extracted` | warning | Prose around exactly one balanced JSON object was dropped. |
| `scalar-to-array` | warning | A scalar where a string array belongs was wrapped in a one-element array. |
| `key-alias` | warning | A known alias (`inBounds`, `in_bounds`, `rung`, `file`) was renamed to its canonical key. |
| `enum-synonym` | warning | A rigor synonym or a case-folded enum value was rewritten to the published value. |
| `normalized-trivia` | warning | A BOM, an explicit null on an optional key, or an exact-duplicate comment was removed. |
| `basis-replaced` | warning | The body's `> Basis:` line disagreed with the computed counts, or was missing from the leading quote block, and the CLI replaced it. |
replaces a disagreeing line, or inserts a missing one under the leading quote
block's confidence lead, with the counts computed from the review and reports
`basis-replaced`. A body with no leading quote block has no lead to insert
under, so it stays byte-identical and reports nothing.

| Code | Level | Meaning |
| ---------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `invalid-json` | error | The file is not one JSON document; the message carries line:col, and the fix is to emit only the object. |
| `wrong-type` | mixed | Error for a wrong type; warning for a non-numeric line/start_line or a fenced suggestion. |
| `missing-field` | mixed | Error for a required field; warning for missing comments or line/start_line. |
| `bad-enum` | mixed | Error for an unknown closed-set value; warning for a malformed KB id or unknown chunk name. |
| `out-of-range` | error | A length or numeric bound is outside its published range (confidence, brief approach/question, caps). |
| `path-not-in-diff` | mixed | Error for a comment path; warning for a brief site outside the diff. |
| `unknown-field` | warning | A key is not in the published shape; the fix names the nearest known key. |
| `risk-stale-head` | warning | `review.json.risk.assessed_head` disagrees with the bundle's current HEAD SHA; the stored claim is marked `stale: true`. |
| `anchor-outside-hunk` | warning | The line is provably outside every hunk; `record` moves the comment into the body. |
| `rigor-unset` | warning | A comment carries no evidence rung; it records as `said` with `rigor_source: default`. |
| `normalized-fence` | warning | One whole-document code fence was unwrapped. |
| `normalized-extracted` | warning | Prose around exactly one balanced JSON object was dropped. |
| `scalar-to-array` | warning | A scalar where a string array belongs was wrapped in a one-element array. |
| `key-alias` | warning | A known alias (`inBounds`, `in_bounds`, `rung`, `file`) was renamed to its canonical key. |
| `enum-synonym` | warning | A rigor synonym or a case-folded enum value was rewritten to the published value. |
| `normalized-trivia` | warning | A BOM, an explicit null on an optional key, or an exact-duplicate comment was removed. |
| `basis-replaced` | warning | The body's `> Basis:` line disagreed with the computed counts and was replaced, or was missing from the leading quote block and was inserted. |

### Sig contract (sig_version 1)

Expand All @@ -277,8 +279,9 @@ previous merge filled a missing rung with `pointed`, and every other recognized
reads as `model`. A review that explicitly wrote `pointed` is indistinguishable from
a default-filled one, so those entries read as `default` either way. Changing the normalization orphans
history — that ships as a new `sig_version`, never silently.
`review.json` inputs stamp `schema_version: "1"` (`post`/`record` refuse unknown
versions, warn-and-proceed when the stamp is missing); every bundle carries
`review.json` inputs stamp `schema_version: "1"` (`post`/`record` refuse a
present value other than `"1"`; an absent key reads as 1 without a warning);
every bundle carries
`.verdict/state/N/store.json` (`{"schema_version":"1"}`) marking the generation of the
whole store — plugins read it instead of sniffing shapes. GitHub bundles keep
`PR.json`/`reviews.json`/`inline.json` as raw API output (GitHub's schema,
Expand Down Expand Up @@ -1075,14 +1078,18 @@ the bundle's own hunk snapshot:
a comma-separated list of 1-based inclusive source-line intervals, and it is
side-agnostic — the evaluator unions the file's `OLD` and `NEW` hunk intervals,
so one interval covers a hunk on either side. Duplicate `file` entries union
rather than count twice. A malformed range warns and makes that entry count as
unattested; it never silently attests the file.
rather than count twice. A malformed range string warns and makes that entry
count as unattested; it never silently attests the file. A `ranges` value that
is neither a string nor `null` — an array, number, or object — refuses
`bad-review-input` naming `$.coverage[N].ranges` on `record`, `post`, and
`next --coverage`, so it can never measure as an unattested 0%.

Coverage is computed from `hunks.tsv`, the snapshot `fetch` already writes —
never from a second diff parse and never from `toc.txt` line counts. Paths the
shared role heuristic classifies as generated or vendored are exempt
automatically; no sentinel is required. `percent` is `null`, never `0`, when no
changed line was measured.
automatically — `*.pb.go`, `*.gen.*`, `zz_generated.*`, lockfiles, minified
assets, and `dist/`, `build/`, or `vendor/` trees — so no sentinel is required.
`percent` is `null`, never `0`, when no changed line was measured.

The default posture is a warning plus one machine-built body line on a partial
attestation. A partial one warns once and adds one machine-built body line;
Expand Down Expand Up @@ -1344,7 +1351,9 @@ esac

Compose it after the expensive step: gate the merge on `--check critical`,
keep `post`'s preflight as the fail-closed backstop — the gate advises,
the human still merges.
the human still merges. A passing gate names that merge in the envelope's
`next` (`merge — gate passes`) and in the human footer, so open findings below
the tier's floor never read as the next step.

`report --category <cat>` filters the ranked findings to one finding kind
(`logic`, `edge-case`, `security`, `performance`, `accidental-commit`,
Expand Down Expand Up @@ -1380,11 +1389,12 @@ a read that stopped before the host's last page, both leave the surface
unread rather than clean: each warns (`live thread recheck unreachable`)
and, under the default `--live auto` posture, leaves the dimension neutral:
the gate reflects local dimensions only, and the run never exits 3 over the
network. Only local bundles skip the lookup with a warn (`live recheck
skipped: ...`), paying zero network; plain `report` without a gate flag
never calls the network. There is no `--force` on report: fetch + triage +
re-check is the recovery, and advisory `report` without `--check` is the
escape hatch.
network. Only local bundles skip the lookup, paying zero network; they say
so with an `info: live recheck skipped: ...` line on the human stream, never
a warning or a `--json` `warnings[]` entry. Plain `report` without a gate
flag never calls the network. There is no `--force` on report: fetch +
triage + re-check is the recovery, and advisory `report` without `--check`
is the escape hatch.

Offline posture (`--live auto|strict|skip`, gate runs only) is always an
explicit operator choice — automation never inherits it silently. `auto`
Expand Down
3 changes: 2 additions & 1 deletion evals/cases/live-offline-contract.sh
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,8 @@ if [ "$EVAL_STATUS" -eq 0 ]; then
else
bad "strict local exits $EVAL_STATUS, want 0"
fi
assert_file_grep "local skip warns structurally" "$EVAL_ERR" "live recheck skipped"
assert_file_grep "local skip notes the structural skip on the human stream" "$EVAL_OUT" "info: live recheck skipped"
assert_stream_nogrep "local skip adds no structural warning" "$EVAL_ERR" "live recheck skipped"
assert_stream_nogrep "local skip prints no fail-closed line" "$EVAL_ERR" "offline posture is strict"
assert_eq "local skip pays zero graphql calls" "0" "$(graphql_since "$m")"

Expand Down
3 changes: 2 additions & 1 deletion evals/cases/live-recheck-report.sh
Original file line number Diff line number Diff line change
Expand Up @@ -273,7 +273,8 @@ if [ "$EVAL_STATUS" -eq 0 ]; then
else
bad "local skip exits $EVAL_STATUS, want 0"
fi
assert_file_grep "local run warn-skips" "$EVAL_ERR" "live recheck skipped"
assert_file_grep "local run notes the skip on the human stream" "$EVAL_OUT" "info: live recheck skipped"
assert_stream_nogrep "local run adds no skip warning" "$EVAL_ERR" "live recheck skipped"
assert_stream_nogrep "local run prints no fetch-local noise" "$EVAL_ERR" "fetch-local" "-i"
assert_stream_nogrep "local run prints no live open line" "$EVAL_ERR" "live open"
assert_eq "local run makes zero recheck calls" "0" "$(graphql_since "$m")"
Expand Down
Loading
Loading