Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/drain-post.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@stainless-code/verdict": minor
---

`drain --post` answers the waiting review threads from a validated `drain.replies.json`. GitHub and GitLab post one marked reply per draft and append `replies.json` under a lock after each write; `withdraw` also resolves its thread. Secret-shaped bodies refuse before any write, stale threads are skipped, and a failed write records `failed[]` and refuses `drain-incomplete` so the same command can finish the remaining replies.
2 changes: 1 addition & 1 deletion apps/docs/content/04-roadmap/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Planned means it is not built yet, and says so on its own line.
| ------------- | ------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Bundle | `fetch`, `fetch-local`, `index` | One diff per review target, from a host PR or your working tree, into `.verdict/state/`, plus the local code index (`index build`) the impact answers read. Chunking and a budget gate are opt-in. |
| Review | `run`, `record`, `panel`, `ingest`, `adopt`, `fix`, `handoff` | Launch your own agent on the bundle (`run`), validate its review and merge it into findings, plan independent disprove attempts (`panel`), ingest SARIF lint findings, carry local findings onto a PR, loop on one finding without escaping scope, and hand the fix brief to someone else. |
| Decide | `report`, `triage`, `threads`, `drain`, `risk`, `next`, `show`, `trend`, `gate` | Render the bundle, gate on severity (`--check`), compose the gate dimensions (`gate`), list review threads waiting on verdict and draft replies with your agent (`drain`, plan and draft only — posting lands later), queue the portfolio risk score (`risk`), and read trend across runs. |
| Decide | `report`, `triage`, `threads`, `drain`, `risk`, `next`, `show`, `trend`, `gate` | Render the bundle, gate on severity (`--check`), compose the gate dimensions (`gate`), list review threads waiting on verdict, draft replies with your agent, and post them with `drain --post` (`drain`), queue the portfolio risk score (`risk`), and read trend across runs. |
| Post | `post`, `pr-wait` | Exactly one review, with anchor validation, stale-SHA and duplicate guards, `--dry-run`, and `--bind` for an approval; `pr-wait` waits on a PR's checks, review, or merge state without writing anything. |
| Trust | `verify`, `check-tree`, `doctor`, `calibrate`, `backtest`, `ready` | Check the bundle, the tree, the host login, past calibration, and readiness in one place, and mine, replay, and score bug-introducing candidates from your own history (`backtest`). |
| Agent surface | `skill`, `mcp` | The embedded skill (`skill`, `skill files`, `skill install`) and the review verbs as MCP tools over stdio. |
Expand Down
32 changes: 17 additions & 15 deletions bench/runner/testdata/tool-free-prompt.golden
Original file line number Diff line number Diff line change
Expand Up @@ -105,10 +105,11 @@ after 3 rounds, stop and report them rather than recording a partial review.
PR fetch for posting. Downstream verbs address the bundle as `local-work`.

Never call `triage` — propose status changes via `triage[]` in `review.json`.
The operator's own reads between runs: `triage` (status changes), `report`
(ranked read), `threads` (refresh the PR conversation feed), `calibrate`
(predictions vs ground truth), and `fix` (one bounded verify-gated pass over a
local bundle — prints the finding plus land instructions).
The operator's own verbs between runs: `triage` (status changes), `report`
(ranked read), `threads` (refresh the PR conversation feed), `drain` (plan and
post replies to waiting threads), `calibrate` (predictions vs ground truth), and
`fix` (one bounded verify-gated pass over a local bundle — prints the finding
plus land instructions).

Delta re-review: `verdict fetch owner/repo 123 --base-sha <prior-reviewed-sha>`,
then load `prompts/incremental-review.md` instead of `prompts/review.md`.
Expand All @@ -117,17 +118,18 @@ and you fall back to the full diff.

## Mode router

| Ask | Load | Terminal write |
| ------------------------------------------ | ------------------------------------------ | -------------------------------------------------------- |
| Review a PR | `prompts/review.md` | one review |
| Review local changes (no PR) | `prompts/review.md` + `source.json` bundle | `record`, then `report` — never `post` |
| New commits landed | `prompts/incremental-review.md` | one review OR no-review note |
| Address review threads | `prompts/address-reviews.md` | commit/push + one reply per thread |
| Fix failing CI | `prompts/fix-ci.md` | commit/push + summary |
| Implement / build | `prompts/build.md` | commit/push + PR link |
| Draft a review report without posting | `prompts/ci.md` | report markdown file, no post |
| Grade closed findings (self-grading probe) | `prompts/grade.md` | grade rows + agent judgments (additive) + agreement rate |
| Anything else | `prompts/system.md` + task text | summary stdout |
| Ask | Load | Terminal write |
| ------------------------------------------ | ------------------------------------------ | --------------------------------------------------------------- |
| Review a PR | `prompts/review.md` | one review |
| Review local changes (no PR) | `prompts/review.md` + `source.json` bundle | `record`, then `report` — never `post` |
| New commits landed | `prompts/incremental-review.md` | one review OR no-review note |
| Address review threads | `prompts/address-reviews.md` | commit/push + one reply per thread; reply-only passes use drain |
| Answer pending review-thread replies | `prompts/address-reviews.md` | `verdict drain owner/repo N --agent <name> --post` |
| Fix failing CI | `prompts/fix-ci.md` | commit/push + summary |
| Implement / build | `prompts/build.md` | commit/push + PR link |
| Draft a review report without posting | `prompts/ci.md` | report markdown file, no post |
| Grade closed findings (self-grading probe) | `prompts/grade.md` | grade rows + agent judgments (additive) + agreement rate |
| Anything else | `prompts/system.md` + task text | summary stdout |

`prompts/system.md` loads on every run. `prompts/reviewer-subagent.md` is the
contract for lens subagents the Review modes dispatch — the orchestrator reads it
Expand Down
1 change: 1 addition & 0 deletions clients/ts/src/refusal-codes.gen.ts
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,7 @@ export type RefusalCode =
| "drafts-invalid"
| "drafts-stale"
| "drain-failed"
| "drain-incomplete"
| "duplicate-payload"
| "fetch-failed"
| "filtered-out"
Expand Down
11 changes: 7 additions & 4 deletions cmd/verdict/usage.go
Original file line number Diff line number Diff line change
Expand Up @@ -277,14 +277,17 @@ func usageText() string {
" makes one host write per thread and writes threads-resolve.json. GitHub and\n" +
" GitLab carry it; Forgejo refuses typed (no documented resolve endpoint).\n" +
" drain <owner/repo> <pr> [--provider <p>] [--max-threads <1-100>] [--json]\n" +
" [--agent <name> [--timeout <secs>] [-- <agent args>...]]\n" +
" [--agent <name> [--timeout <secs>] [-- <agent args>...]] [--post]\n" +
" answer every review thread waiting on verdict: refresh threads, then write\n" +
" .verdict/state/<pr>/drain.json with the pending list, typed skips, and a plan digest.\n" +
" Zero pending is success and launches no agent. --max-threads defaults to 20;\n" +
" older questions are taken first and the rest are deferred. --agent drafts one\n" +
" drain.replies.json through the run launcher; no host write happens in this\n" +
" build (there is no --post yet). A truncated feed or unreadable login refuses\n" +
" drain-failed before the plan is written.\n" +
" drain.replies.json through the run launcher. --post validates those drafts,\n" +
" re-reads threads, posts one marked reply per draft, and appends replies.json\n" +
" after each write; withdraw also resolves its thread. GitHub and GitLab carry\n" +
" replies; other hosts refuse provider-unimplemented. A failed write records\n" +
" failed[] and refuses drain-incomplete so a re-run finishes the rest. A\n" +
" truncated feed or unreadable login refuses drain-failed before any write.\n" +
" trend <owner/repo> [--rules] [--overlap] [--incidents] [--resolution] [--json]\n" +
" portfolio rollup over existing bundle stores (no network, writes nothing):\n" +
" findings opened/addressed/closed over time, broken down by severity tier\n" +
Expand Down
34 changes: 24 additions & 10 deletions docs/manual.md
Original file line number Diff line number Diff line change
Expand Up @@ -316,16 +316,30 @@ write per thread, marks them resolved in `threads.json`, and writes
refuses typed because its API has no documented resolve endpoint.

`verdict drain owner/repo N [--provider p] [--max-threads 1-100] [--agent name]
[--timeout s] [-- args...] [--json]` plans one pass over the threads waiting on
verdict. It refreshes the feed, reads the login once, joins findings, and writes
`drain.json`: `pending[]`, typed `skipped[]` reasons, `other_threads`, and a
`plan_digest` over the pending list. A thread is pending when it is a review
comment verdict authored, unresolved, and its last comment is a human question
with no ledger entry answering it; the oldest questions come first and the rest
are `deferred`. `--max-threads` defaults to 20. Zero pending is success and
launches no agent. `--agent` drafts one `drain.replies.json` through the `run`
launcher; `drafts-invalid` and `drafts-stale` name a bad site or a moved plan.
This build makes no host writes and has no `--post`.
[--timeout s] [-- args...] [--post] [--json]` plans one pass over the threads
waiting on verdict. It refreshes the feed, reads the login once, joins findings,
and writes `drain.json`: `pending[]`, typed `skipped[]` reasons, `other_threads`,
and a `plan_digest` over the pending list. A thread is pending when it is a
review comment verdict authored, unresolved, and its last comment is a human
question with no ledger entry answering it; the oldest questions come first and
the rest are `deferred`. `--max-threads` defaults to 20. Zero pending is success
and launches no agent. `--agent` drafts one `drain.replies.json` through the
`run` launcher; `drafts-invalid` and `drafts-stale` name a bad site or a moved
plan.

`--post` validates the drafts first, checks the `thread_reply` capability, then
refreshes threads and the login. A thread now resolved, answered in the
`replies.json` ledger, or whose waiting comment changed is skipped; a
secret-shaped body refuses `post-secret-shaped` before any write. Each posted
reply carries a hidden `<!-- verdict-reply thread=… to=… -->` marker, so the next
threads refresh reads it as verdict-authored. Bodies have mentions neutralized.
GitHub replies to the root comment and GitLab adds a discussion note; `withdraw`
also resolves through the host's existing resolve call. One ledger entry is
appended under `storelock` after each successful write, so a re-run skips replies
already posted. A failed write is recorded in `failed[]`, the remaining drafts
are attempted, and the run refuses `drain-incomplete`. Forgejo and fetch-only
hosts refuse `provider-unimplemented` before any read. There is no `--dry-run`
for post and no `--allow-secret-values` override.

`post.json` and `record.json` carry `schema_version: "1"`.

Expand Down
10 changes: 8 additions & 2 deletions docs/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -603,6 +603,12 @@ Cause: the bundle, `threads.json`, the login, or the reply ledger could not be r

Fix: refresh threads and confirm host auth, then re-run `drain`.

## `drain-incomplete`

Cause: at least one `drain --post` host write failed; `drain.json` names the threads in `failed[]`. Replies that landed are already in `replies.json`.

Fix: re-run the same `drain --post` command. The ledger skips replies already posted and only the failed threads are retried; check the host's own error in `failed[]` first.

## `duplicate-payload`

Cause: every rendered summary part and comment is present in the live remote lists for this reviewed head.
Expand Down Expand Up @@ -1047,9 +1053,9 @@ Fix: narrow anchor drift or body size (see above), then re-run.

## `post-secret-shaped`

Cause: `review.json` carries a secret-shaped value.
Cause: a review draft carries a secret-shaped value — `review.json` for `post`, `drain.replies.json` for `drain --post`.

Fix: remove the value from `review.json` and re-run, or pass `--allow-secret-values` to publish it anyway.
Fix: remove the value from the draft and re-run. `post` can pass `--allow-secret-values` to publish it anyway; `drain --post` has no override.

## `pr-not-open`

Expand Down
Loading
Loading