From c5a7042d4d72f16f2ae4d1dcbeddd9f4f8080e0b Mon Sep 17 00:00:00 2001 From: Daniel Pittman Date: Wed, 12 Aug 2026 14:48:47 -0600 Subject: [PATCH] ci: stop the review gate passing on a review that never happened MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit When the review agent cannot run — an API rate limit is the usual cause, and the lab's repos share one Anthropic account so concurrent CI across repos trips it — the action still posts its "I'll analyze this and get back to you" placeholder and exits 0. The gate's REVIEW_RESULT check never fires, the extractor truthfully finds no HIGH/CRITICAL in a comment containing nothing, and the gate reports PASS on a pull request nobody reviewed. Observed across four repos on 2026-08-12, with successes and failures interleaved minute by minute, and it put a green gate on four release PRs. Detection is by size, not by matching the placeholder text: matching the text rejects any real review that quotes it. Measured corpus: shortest real review 836 bytes, placeholders 190 and under. The error message names the remedy this failure needs and no other does — `gh run rerun --failed` re-runs only the extractor, which re-reads the same placeholder and fails identically; the whole workflow has to run again. Implementation matches the one already proven in didakt and validated in roadrunner-connect-app (#926), where a 2851-byte review that quotes the placeholder is correctly accepted. --- .github/workflows/claude-code-review.yml | 32 ++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/.github/workflows/claude-code-review.yml b/.github/workflows/claude-code-review.yml index db5c02b..785c3e4 100644 --- a/.github/workflows/claude-code-review.yml +++ b/.github/workflows/claude-code-review.yml @@ -297,6 +297,38 @@ jobs: exit 1 fi + # THE REVIEW JOB CAN REPORT SUCCESS HAVING POSTED ONLY A PROGRESS + # PLACEHOLDER. When the review agent cannot run — an API rate limit is + # the usual cause, and the lab's repos share one account so concurrent + # CI across repos trips it — the action still writes "I'll analyze this + # and get back to you" and exits 0. The gate's REVIEW_RESULT check + # (which treats a non-success review as a transient throttle) never + # fires, the extractor finds no HIGH/CRITICAL in a comment that + # contains nothing, and the gate reports PASS on a pull request nobody + # reviewed. A gate that cannot tell "reviewed and clean" from "never + # reviewed" is worse than a red one, because it looks like evidence. + # + # The REMEDY DIFFERS from every other failure here, which is why it is + # named rather than left to the generic message: `gh run rerun --failed` + # re-runs only the extractor, which re-reads the same placeholder and + # fails identically. The whole workflow has to run again so the REVIEW + # job runs again. + # + # Detected by size, NOT by matching the placeholder text. Matching the + # text rejects any real review that quotes it — which is exactly what + # happened to the review of the PR that introduced this guard. The + # review writes its own heading so there is no fixed marker to require; + # what placeholders have in common is carrying no findings at all. + # Measured corpus: shortest real review 836 bytes, placeholders 190 and + # under. + bytes=$(wc -c < REVIEW.md | tr -d ' ') + if [ "$bytes" -lt 400 ]; then + echo "::error::The review bot's latest comment is ${bytes} bytes, which is a progress placeholder rather than a review. RE-RUN THE WHOLE WORKFLOW (gh run rerun ), not just the failed jobs: --failed re-runs this extractor against the same placeholder and fails again. Comment follows:" >&2 + cat REVIEW.md >&2 + exit 1 + fi + + # SHA-pinned anthropics/claude-code-action v1.0.152. Cheap model (Haiku) — # this is a one-shot classification, not a review. The verdict comes back as # schema-validated STRUCTURED OUTPUT (the `--json-schema` claude_arg), exposed