ci: stop the review gate passing on a review that never happened - #60
Merged
Conversation
When the review agent cannot run — an API rate limit is the usual cause, and the lab's repos share one Anthropic account so concurrent CI across repos trips it — the action still posts its "I'll analyze this and get back to you" placeholder and exits 0. The gate's REVIEW_RESULT check never fires, the extractor truthfully finds no HIGH/CRITICAL in a comment containing nothing, and the gate reports PASS on a pull request nobody reviewed. Observed across four repos on 2026-08-12, with successes and failures interleaved minute by minute, and it put a green gate on four release PRs. Detection is by size, not by matching the placeholder text: matching the text rejects any real review that quotes it. Measured corpus: shortest real review 836 bytes, placeholders 190 and under. The error message names the remedy this failure needs and no other does — `gh run rerun --failed` re-runs only the extractor, which re-reads the same placeholder and fails identically; the whole workflow has to run again. Implementation matches the one already proven in didakt and validated in roadrunner-connect-app (#926), where a 2851-byte review that quotes the placeholder is correctly accepted.
Semgrep (OSS): no findingsClean scan. Posted before the Claude review. |
CI Tests✅ Tests passed: the pytest suite across Python 3.10/3.11/3.12 (and bash syntax checks) is green on Posted before the Claude review so the test outcome is folded into the review. |
Review Gate
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Shared-infrastructure fix. This workflow is byte-identical across the lab's repos, so the bug is too.
When the review agent cannot run (API rate limit — the lab's repos share one Anthropic account, so concurrent CI across repos trips it), the action still posts its placeholder and exits 0. The gate then reports
PASS — Review reported no HIGH/CRITICAL findingson a pull request nobody reviewed.Detection is by size, not text: matching the placeholder string rejects any real review that quotes it. Measured corpus — shortest real review 836 bytes, placeholders ≤190.
Validated in roadrunner-connect-app (#926): a 2851-byte review quoting the placeholder is accepted, a 190-byte stub is rejected.
Note this PR modifies the review workflow, so the review action will refuse to run on it (a deliberate guard against a PR rewriting its own reviewer). It cannot self-review; merged on the strength of the validation above.