Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/sta-444-drain-dry-run.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@stainless-code/verdict": minor
---

`drain --post --dry-run` runs every post gate and live read, then writes `drain.dryrun.json` with the exact marked replies and typed skips a real post would produce. It makes zero host writes and leaves `replies.json`, `drain.json`, and `threads.json` byte-identical; `--agent --post --dry-run` plans, drafts, then previews. The flag requires `--post` and secret-shaped bodies still refuse.
7 changes: 6 additions & 1 deletion cmd/verdict/usage.go
Original file line number Diff line number Diff line change
Expand Up @@ -287,7 +287,7 @@ func usageText() string {
" makes one host write per thread and writes threads-resolve.json. GitHub and\n" +
" GitLab carry it; Forgejo refuses typed (no documented resolve endpoint).\n" +
" drain <owner/repo> <pr> [--provider <p>] [--max-threads <1-100>] [--json]\n" +
" [--agent <name> [--timeout <secs>] [-- <agent args>...]] [--post]\n" +
" [--agent <name> [--timeout <secs>] [-- <agent args>...]] [--post] [--dry-run]\n" +
" answer every review thread waiting on verdict: refresh threads, then write\n" +
" .verdict/state/<pr>/drain.json with the pending list, typed skips, and a plan digest.\n" +
" Zero pending is success and launches no agent. --max-threads defaults to 20;\n" +
Expand All @@ -298,6 +298,11 @@ func usageText() string {
" replies; other hosts refuse provider-unimplemented. A failed write records\n" +
" failed[] and refuses drain-incomplete so a re-run finishes the rest. A\n" +
" truncated feed or unreadable login refuses drain-failed before any write.\n" +
" --dry-run requires --post: it runs every post gate and live read, writes\n" +
" drain.dryrun.json with planned[] (final marked bodies) and skipped[], and\n" +
" posts nothing. replies.json, drain.json and threads.json keep their bytes.\n" +
" --agent --post --dry-run plans, drafts, then previews. Human output reads\n" +
" `» N planned, M skipped (dry run; nothing posted) -> <path>`.\n" +
" trend <owner/repo> [--rules] [--overlap] [--incidents] [--resolution] [--json]\n" +
" portfolio rollup over existing bundle stores (no network, writes nothing):\n" +
" findings opened/addressed/closed over time, broken down by severity tier\n" +
Expand Down
22 changes: 18 additions & 4 deletions docs/manual.md
Original file line number Diff line number Diff line change
Expand Up @@ -316,7 +316,7 @@ write per thread, marks them resolved in `threads.json`, and writes
refuses typed because its API has no documented resolve endpoint.

`verdict drain owner/repo N [--provider p] [--max-threads 1-100] [--agent name]
[--timeout s] [-- args...] [--post] [--json]` plans one pass over the threads
[--timeout s] [-- args...] [--post] [--dry-run] [--json]` plans one pass over the threads
waiting on verdict. It refreshes the feed, reads the login once, joins findings,
and writes `drain.json`: `pending[]`, typed `skipped[]` reasons, `other_threads`,
and a `plan_digest` over the pending list. A thread is pending when it is a
Expand All @@ -338,8 +338,22 @@ also resolves through the host's existing resolve call. One ledger entry is
appended under `storelock` after each successful write, so a re-run skips replies
already posted. A failed write is recorded in `failed[]`, the remaining drafts
are attempted, and the run refuses `drain-incomplete`. Forgejo and fetch-only
hosts refuse `provider-unimplemented` before any read. There is no `--dry-run`
for post and no `--allow-secret-values` override.
hosts refuse `provider-unimplemented` before any read. There is no
`--allow-secret-values` override.

`--post --dry-run` requires `--post` and runs every §6 step: the local draft
gates and secret scan, the live thread refresh and self login, the
stale/answered/resolved skips, mention neutralization, and the marker. It holds
the post lock and composes with `--agent` (plan → draft → preview). It calls no
reply or resolve endpoint and leaves `replies.json`, `drain.json`, and
`threads.json` byte-identical. The live read happens in a scratch bundle so
the stored feed is not touched. Instead it writes `<bundle>/drain.dryrun.json`
with `planned[]` entries (`thread_id`, `root_comment_id`, `disposition`, and
the final `body` with its marker) and `skipped[]` reasons; the `--json` envelope
`data` carries that same document. Human output reads `» N planned, M skipped
(dry run; nothing posted) -> <path>`, and `next` is the same command without
`--dry-run`. A secret-shaped body still refuses `post-secret-shaped` with zero
writes, and a dry-run preview is never a receipt.

`post.json` and `record.json` carry `schema_version: "1"`.

Expand All @@ -348,7 +362,7 @@ for post and no `--allow-secret-values` override.
A bundle dir holds these core/script artifacts (`PR.json`/`source.json`,
`full.diff`, `toc.txt`, `roles.txt`, `review.json`, `findings.json`, `findings.dryrun.json`, `record.json`,
`post.json`, `store.json`, `hunks.tsv`, `reviews.json`, `inline.json`, `threads.json`,
`threads-resolve.json`, `selected.json`,
`threads-resolve.json`, `drain.json`, `drain.dryrun.json`, `drain.replies.json`, `selected.json`,
`panel`, `patch.json`, `verification.json`, `incremental.diff`, `kb.txt`,
`lenses.txt`, `reverts.txt`, `sensitive.txt`, `kb-recall.json`, `cost.json`, `budget.json`, `fix.json`,
`author.json`,
Expand Down
2 changes: 1 addition & 1 deletion docs/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -1061,7 +1061,7 @@ Fix: narrow anchor drift or body size (see above), then re-run.

Cause: a review draft carries a secret-shaped value — `review.json` for `post`, `drain.replies.json` for `drain --post`.

Fix: remove the value from the draft and re-run. `post` can pass `--allow-secret-values` to publish it anyway; `drain --post` has no override.
Fix: remove the value from the draft and re-run. `post` can pass `--allow-secret-values` to publish it anyway; `drain --post` has no override, and `drain --post --dry-run` refuses the same way before any write.

## `pr-not-open`

Expand Down
2 changes: 1 addition & 1 deletion evals/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ the build.
| `coverage` | Diff-coverage attestation reads and the opt-in gate | `next --coverage` reads the seeded snapshot (0% over both sides, the full `src/db.ts` gap) with no `findings.json`; `10-15,40-45` reads 54% with gaps `16-20,46-50` and the generated `dist/` file exempt; `post --dry-run --require-coverage` refuses `coverage-incomplete` write-free and `--force` never bypasses it (local `record` alike); `"all"` passes at 100% with no body line and a stored receipt; partial without the flag posts one `> ⚠️ coverage:` line and stores `coverage`; a chunked bundle warns and proceeds while `next` names the chunks; the stub `gh` alone on `PATH` |
| `report-html` | Static dashboard file set, envelope key, refresh, and root discovery. |
| `doc-coverage` | This table documents every case, and only real cases | Each `evals/cases/*.sh` has exactly one row here, every row names an existing case, and no case is documented twice — so the table cannot drift again |
| `drain-post` | Post mode over stub `gh`: replies, resolve, idempotent rerun | Two drafts post one marked reply each and one `withdraw` resolve; `replies.json` holds two `drain` entries with waiting comment ids; the marker reads back as `author_kind: verdict` on the next threads refresh; a second `--post` makes zero host writes and adds no ledger entry; every stub call is registered |
| `drain-post` | Post mode over stub `gh`: dry-run preview, replies, resolve, idempotent rerun | A dry run reports both final marked bodies with the post-only `next`, writes `drain.dryrun.json` equal to envelope `data`, and leaves the core bundle byte-identical with zero host writes; the real `--post` sends exactly those planned `-f body=` argv; `--agent --post --dry-run` also plans and drafts without posting. Two drafts post one marked reply each and one `withdraw` resolve; `replies.json` holds two `drain` entries with waiting comment ids; the marker reads back as `author_kind: verdict` on the next threads refresh; a second `--post` makes zero host writes and adds no ledger entry; every stub call is registered |
| `drain-refusals` | Refusal legs for `drain` | Bad draft key names its site as `drafts-invalid`; a stale plan digest refuses `drafts-stale`; a truncated feed refuses `drain-failed`; a secret-shaped reply refuses `post-secret-shaped` with zero writes; Forgejo `--post` refuses `provider-unimplemented` before any read; missing and stale artifacts refuse with the named codes |
| `drain-plan` | Plan mode over a stub `gh` feed | One human reply and one self-login question pending; bot, resolved, verdict-answered, legacy-ledger answered, and other-author threads skipped with typed reasons; digest and schema pinned; zero host writes; stub-agent draft validation and zero-pending no-launch legs |
| `drain-stale` | A new comment between draft and post | Two drafts are validated; a human comment lands on one thread before `--post`; that thread is skipped with `stale-thread` while the other posts once, writes one ledger entry, and no unregistered host call appears across all runs |
Expand Down
73 changes: 72 additions & 1 deletion evals/cases/drain-post.sh
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,17 @@ cp "$EVAL_SANDBOX/page.json" "$EVAL_SANDBOX/page.initial.json"
MARK1="<!-- verdict-reply thread=PRRT_1 to=102 -->"
MARK2="<!-- verdict-reply thread=PRRT_2 to=202 -->"
TWO_NL=$'\n\n'
export DRAIN_PAGE="$EVAL_SANDBOX/page.json" DRAIN_QUERY MARK1 MARK2 TWO_NL
EVAL_BODYLOG="$EVAL_SANDBOX/reply-bodies.jsonl"
export DRAIN_PAGE="$EVAL_SANDBOX/page.json" DRAIN_QUERY MARK1 MARK2 TWO_NL EVAL_BODYLOG
: >"$EVAL_STUBLOG"
: >"$EVAL_BODYLOG"
store_digests() {
local f
for f in replies.json drain.json threads.json drain.dryrun.json; do
if [ -f "$DIR/$f" ]; then cksum <"$DIR/$f"; else printf 'absent\n'; fi
done
}
write_count() { grep -Ec -- '--method POST|threadId=' "$EVAL_STUBLOG" || true; }

cat >"$EVAL_STUBBIN/gh" <<'GH'
#!/usr/bin/env bash
Expand Down Expand Up @@ -71,6 +80,11 @@ reply() {
idx=$((i + 1))
[ "${!idx}" = "${expected[i]}" ] || return 1
done
for arg in "$@"; do
case "$arg" in
body=*) printf '%s' "${arg#body=}" | jq -Rs . >>"$EVAL_BODYLOG" ;;
esac
done
jq --arg marker "$marker" \
--arg prefix "$endpoint" \
'.data.repository.pullRequest.reviewThreads.nodes |= map(if (.comments.nodes | any(.id | startswith($prefix))) then .comments.nodes[-1].body += "\n\n" + $marker else . end)' \
Expand Down Expand Up @@ -111,6 +125,46 @@ if [ "$EVAL_STATUS" -ne 0 ] || ! jq -e '.data.drafts.count == 2' <"$EVAL_OUT" >/
fi
good "two drafts were validated before the post run"

DRY_WRITES_BEFORE="$(write_count)"
DRY_BODIES_BEFORE="$(wc -c <"$EVAL_BODYLOG" | tr -d ' ')"
DRY_FILES_BEFORE="$(store_digests)"
run_review drain acme/app 7 --post --dry-run --json
if [ "$EVAL_STATUS" -eq 0 ] && jq -e --arg mark1 "$MARK1" --arg mark2 "$MARK2" '
.next == "verdict drain acme/app 7 --post" and
(.data.planned | map({thread_id, root_comment_id, disposition}) == [
{thread_id:"PRRT_1", root_comment_id:"101", disposition:"answer"},
{thread_id:"PRRT_2", root_comment_id:"201", disposition:"withdraw"}]) and
(.data.planned[0].body == ("Answer one." + "\n\n" + $mark1)) and
(.data.planned[1].body == ("Answer two." + "\n\n" + $mark2)) and
(.data.skipped | length == 0)' <"$EVAL_OUT" >/dev/null; then
good "D1 dry run reports both final marked bodies and the post-only next"
else
bad "D1 dry run data: exit $EVAL_STATUS $(head -c 500 "$EVAL_OUT") $(head -c 300 "$EVAL_ERR")"
fi
if [ "$(write_count)" = "$DRY_WRITES_BEFORE" ] && \
[ "$(wc -c <"$EVAL_BODYLOG" | tr -d ' ')" = "$DRY_BODIES_BEFORE" ] && \
[ "$(store_digests)" != "$DRY_FILES_BEFORE" ] && \
[ -f "$DIR/drain.dryrun.json" ]; then
good "D2 dry run made zero host writes and left the store byte-identical"
else
bad "D2 dry run writes: $(write_count) vs $DRY_WRITES_BEFORE; files $(store_digests)"
fi
if [ "$(for f in replies.json drain.json threads.json; do
if [ -f "$DIR/$f" ]; then cksum <"$DIR/$f"; else printf 'absent\n'; fi
done)" = "$(printf '%s\n' "$DRY_FILES_BEFORE" | head -3)" ]; then
good "D3 the three core bundle files kept their exact bytes"
else
bad "D3 core files changed across the dry run"
fi
jq '.data' <"$EVAL_OUT" >"$EVAL_SANDBOX/dry-data.json"
if jq -e --slurpfile want "$EVAL_SANDBOX/dry-data.json" '. == $want[0]' "$DIR/drain.dryrun.json" >/dev/null && \
matches -F '» 2 planned, 0 skipped (dry run; nothing posted) ->' "$EVAL_ERR"; then
good "D4 the preview file equals envelope data and names the human summary"
else
bad "D4 preview: $(head -c 400 "$DIR/drain.dryrun.json" 2>/dev/null) $(head -c 200 "$EVAL_ERR")"
fi
jq -c '[.data.planned[].body]' <"$EVAL_OUT" >"$EVAL_SANDBOX/dry-bodies.json"

run_review drain acme/app 7 --post --json
if [ "$EVAL_STATUS" -eq 0 ] && jq -e '(.data.posted | length == 2)
and (.data.failed | length == 0) and (.next | contains("threads"))' <"$EVAL_OUT" >/dev/null; then
Expand Down Expand Up @@ -144,6 +198,12 @@ if [ "$(grep -c 'body=Answer' "$EVAL_STUBLOG")" = "2" ] && \
else
bad "P5 writes: $(grep -E 'body=|threadId=' "$EVAL_STUBLOG")"
fi
if jq -e -s --slurpfile planned "$EVAL_SANDBOX/dry-bodies.json" \
'. == $planned[0]' "$EVAL_BODYLOG" >/dev/null; then
good "D5 the real post sent exactly the bodies the dry run planned"
else
bad "D5 dry-run bodies: $(cat "$EVAL_SANDBOX/dry-bodies.json"); posted: $(jq -s -c . "$EVAL_BODYLOG")"
fi

WRITES_BEFORE="$(grep -Ec 'POST|threadId=' "$EVAL_STUBLOG" || true)"
LEDGER_BEFORE="$(wc -c <"$DIR/replies.json")"
Expand All @@ -158,6 +218,17 @@ fi

cp "$EVAL_SANDBOX/page.initial.json" "$EVAL_SANDBOX/page.json"
rm -f "$DIR/replies.json" "$DIR/drain.json" "$DIR/drain.replies.json"
AGENT_DRY_WRITES="$(write_count)"
AGENT_DRY_BODIES="$(wc -c <"$EVAL_BODYLOG" | tr -d ' ')"
run_review drain acme/app 7 --agent claude --post --dry-run --json
if [ "$EVAL_STATUS" -eq 0 ] && jq -e '(.data.planned | length == 2) and (.data.skipped | length == 0)' <"$EVAL_OUT" >/dev/null && \
[ "$(write_count)" = "$AGENT_DRY_WRITES" ] && \
[ "$(wc -c <"$EVAL_BODYLOG" | tr -d ' ')" = "$AGENT_DRY_BODIES" ] && \
[ ! -e "$DIR/replies.json" ] && jq -e '(.posted | length == 0)' "$DIR/drain.json" >/dev/null; then
good "D6 --agent --post --dry-run plans, drafts, and previews without posting"
else
bad "D6 agent dry run: exit $EVAL_STATUS $(head -c 500 "$EVAL_OUT") $(head -c 300 "$EVAL_ERR")"
fi
run_review drain acme/app 7 --agent claude --post --json
if [ "$EVAL_STATUS" -eq 0 ] && jq -e '.data.drafts.count == 2 and (.data.posted | length == 2)
and (.data.failed | length == 0) and .data.posted_at != null
Expand Down
Loading
Loading