Tighten PR review prompt: validate findings before posting#2
Merged
Conversation
Add an explicit draft -> validate -> filter -> post loop to the inline review prompt so findings get a second-pass check against the diff before being posted. The validator drops anything that can't be tied to a specific token in the diff, anything that depends on context outside the diff, anything CI already enforces, and style/opinion calls. Only high-signal survivors (compile/parse failures, definite logic errors, concrete security or data-loss risks, or quotable convention violations) make it into the posted comment. Output adds a 'No high-signal issues found.' fallback when validation drops everything. Goal: cut false-positive noise from the org-wide PR review bot.
Validation was forced to drop any finding that needed context beyond the diff to confirm, which kept too much noise out but also killed real bugs whose proof lives one file over (e.g. an awaited-but-not- async function defined elsewhere, a misnamed import). Grant the validator Read/Grep/Glob and gh-api/git-log/show/blame/diff Bash patterns so it can actually investigate. Update the prompt to encourage purposeful look-outside-the-diff checks (specific question in mind) and to use git blame/log to confirm whether a problem was introduced by the PR vs pre-existing.
Validator now reads outside the diff (grep/read, gh api, git blame), which takes longer on bigger PRs. 5 min was already tight for the single-pass review; with investigation it'll be tighter. 8 gives a comfortable margin without letting a runaway review burn budget.
robert-moore
approved these changes
May 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an explicit draft → validate → filter → post loop to the inline review prompt used by the org-wide PR review bot. The validator forces a second pass on every drafted finding before anything gets posted to a PR.
The validator drops:
Only high-signal survivors make it into the posted comment:
If validation drops everything, the comment says `No high-signal issues found.` instead of forcing a low-quality review.
Why
The current single-pass prompt produces too many low-signal comments — style notes, hypothetical concerns, things the linter already catches. Reviewers start tuning the bot out. This mirrors the validation step from the local `code-review` slash-command pipeline, but keeps the existing single-action CI flow (no timeout bump, no extra tools needed).
Scope
This is a prompt-only change to `claude-code-review-reusable.yml`. Every consumer workflow (`churnkey-api`, `monorepo`, `churnkey-embed`, etc.) references this via `@main`, so the new prompt goes live on every repo's next PR event after merge.
Test plan