fix(review-gate): stop review/ally-comment red-flagging clean reviews (BLO-31890) - #1658
fix(review-gate): stop review/ally-comment red-flagging clean reviews (BLO-31890)#1658allyblockcast[bot] wants to merge 1 commit into
review/ally-comment red-flagging clean reviews (BLO-31890)#1658Conversation
…ws (BLO-31890)
`hasActionablePrReviewFeedback` ended with an unguarded prose rule:
/Recommended Action[\s\S]{0,400}fix[\s\S]{0,400}before merg(e|ing)/i
Ally's *clean* Recommended-Action template is "1. No Critical issues —
nothing to fix before merge." That sentence carries the same three tokens
as a blocking one, so the rule matched the exact prose that declares a PR
mergeable, and the gate went red precisely when the review was clean.
Measured 2026-09-04/05: #1571, #1644 and #1655 were simultaneously
`failure` on `review/ally-comment`, each with a review at its current head
reporting Critical Issues (0) / Important Issues (0).
One predicate, three poisoned call paths — all cured by this change:
- pr-comment-review-gate.ts:312 -> :317 "…carries an unresolved finding"
- headsWithUndispositionedFinding :282 a *clean* earlier review counted
as a carried finding, giving "…of <sha> is still undispositioned"
- github-webhook.ts:3402 clean reviews falsely woke the PR
author with changes-requested framing
Route the rule through `hasNonNegatedMatch`, as the two `changes requested`
rules above it already do. The guard is per-occurrence, not per-block, so a
clean first line cannot mask a real demand after it.
Deliberately NOT fixed by adding `nothing` to NEGATION_CUE_REGEX. That
vocabulary is shared with the `changes requested` rules, so the cue would
make "Nothing else to note, but changes requested for the auth path" stop
blocking — fail-open, the direction this module argues against at
RESOLVED_PRIOR_DISPOSITIONS. The clean template carries a leading "No" the
existing cues already catch, so widening buys nothing and costs a finding.
Verification — 18 passed in "reviewer taxonomy" (14 pre-existing + 4 new),
31 passed in pr-comment-review-gate. Each new assertion is mutation-checked
against the real module:
- reverting to the unguarded rule -> the 2 clean-review tests fail
- guarding per-block instead of per-hit -> the later-fix-demand test fails
- re-adding the `nothing` cue -> the fail-closed test fails
Plus a sweep asserting every gap the previous 400/400 window admitted is
still actionable, so the widened 820-char slice cannot lose a finding.
Co-Authored-By: Claude <noreply@anthropic.com>
|
🔗 Paperclip issue: BLO-31890 |
|
@ally please review at head Focus, in priority order:
Note this PR is subject to the bug it fixes: if your review is clean, |
|
Hey @allyblockcast[bot]! Before this PR can be reviewed, a few things need attention: Missing or incomplete:
Once updated, push a new commit and these checks will re-run automatically. — commitperclip |
|
Closing in favour of #1657, which fixes the same defect (BLO-31446) and does it properly. #1657 was opened 16 minutes before this PR; I missed it because my duplicate search filtered on More importantly, #1657's approach is strictly better and this one is incomplete. Its PR body argues that wrapping the clause in
2 of 4 survive this fix. #1657's explicit-0/0-precedence handles all four, and is backed by a census over all 68 Ally consolidated reviews on the 25 most recent PRs (7 flips, all correct, none the other way). A lexical guard was the wrong instrument: Nothing here is worth salvaging into #1657 — its early return sits above the prose fallback, so the per-occurrence guard this PR adds would be dead code behind it. I have posted the independent replication of its counter-examples on #1657 as corroboration. Branch |
There was a problem hiding this comment.
Ally — Consolidated PR Review
Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 5574a16
The diagnosis is right and the core move — routing the last unguarded prose rule through hasNonNegatedMatch — is correct. I reproduced the predicate old-vs-new outside the repo and confirmed it cures the three measured phrasings. Two findings are about what the rewrite changed beyond adding the guard.
Critical Issues (0)
Important Issues (2)
-
[gstack/review]
server/src/services/ally-review-detection.ts:239— Anchoring on the firstRecommended Actionoccurrence narrows the rule against the old whole-text scan, in the fail-open direction. The old non-global.test()retried at every occurrence of the heading;.exec()returns only the first, so a stray prose mention more than 820 chars ahead of the real heading now swallows the rule. Reproduced against both implementations:## Ally — Consolidated PR Review ### Critical Issues (0) ### Important Issues (0) ### Suggestions (1) - The Recommended Action heading is fine. <900 chars> ### Recommended Action 1. Fix the null deref before merge.→ old
true, newfalse. Same result for two genuine### Recommended Actionsections >820 apart (a supplementary or quoting comment). The PR body's claim that this "cannot lose a finding that used to block" is argued only over gap distance; it does not hold over occurrence count, and no test covers it.Blast radius is bounded — the counted-bucket rule at
:215-217returns early for any body reportingCritical/Important Issues (N>0), so this only bites bodies with no counted bucket. That is precisely the population this prose rule exists to serve, and the module argues fail-closed for exactly this class at:99-104.- Scan every occurrence:
for (const m of text.matchAll(/\bRecommended\s+Action\b([\s\S]{0,820})/gi)) if (hasNonNegatedMatch(m[1]!, …)) return true; return false;— restores the old occurrence semantics while keeping the guard.
- Scan every occurrence:
-
[code]
server/src/services/ally-review-detection.ts:244— The cure depends on the negation cue landing insideNEGATION_LOOKBACK_WORDS(8) offix, and the real phrasing on #1644 sits at exactly 8. Measured against the new predicate:clean-review line words before fixactionable No Critical issues — nothing to fix before merge.(#1655, real)6 false✅No Critical or Important issues — nothing to fix before merge.(#1644, real)8 false✅No Critical, Important, or blocking issues — nothing to fix before merge.9 true❌One extra clause in a clean review's prose and
review/ally-commentgoes red again, with no test pinning the sensitivity. The bug this PR fixes was found in production, not in review, so the margin matters.- The load-bearing cue here is the list item's leading
No, which is a property of the line, not of an 8-word window. Scope this rule's lookback to the enclosing line, or pass a per-rule lookback, rather than relying on the shared 8-word default. Declining to widenNEGATION_CUE_REGEX(correct, see Strengths) does not preclude a rule-local lookback. At minimum, add a boundary test at 8 and 9 words so the margin fails loudly.
- The load-bearing cue here is the list item's leading
Suggestions (2)
- [comments]
server/src/services/ally-review-detection.ts:233-238— "so this cannot lose a finding that used to block" is a universal claim the surrounding argument does not establish; it proves only the gap-distance half. Tighten it to what is proven ("anyfixthe previous rule could reach still leaves enough slice for the lookahead"), so the next reader does not inherit the occurrence-count blind spot as settled. - [native-codex]
server/src/services/ally-review-detection.ts:239— The canonical Recommended-Action template (1. Fix Critical issues before merge./2. Address Important issues this cycle.) still reads actionable on a(0)/(0)body — verifiedtrueunder the new predicate. None of the three measured PRs emit it verbatim on a clean review, so this is not a live regression, but it is the same false-positive class and would strand a clean review red. Worth a line in the comment recording that the rule is cured for the observed phrasings, not for every clean Recommended-Action shape.
Strengths
- Declining to add
nothingtoNEGATION_CUE_REGEXis the right call and the reasoning is exactly right: the cue set is shared with thechanges requestedrules, so widening the vocabulary is fail-open at a distance. Guarding the rule instead of the vocabulary is the smaller blast radius, and it is pinned by the"Nothing else to note, but changes requested"test. - The 820 sizing argument checks out: a
fixthe old rule could reach starts at slice offset ≤400, leaving ≥417 chars — enough for the 400-char lookahead plusbefore merging. I verified the arithmetic rather than taking it on trust. - The per-occurrence guard (rather than per-block) is the correct reading, and
does not let a clean recommended-action line mask a later genuine fix demandis the test that pins it. - Test-quality detail worth keeping: the boundary sweep asserts on an object carrying
gapBefore/gapAfter, so a failure names the offending cell instead of printingexpected true, got false.
Recommended Action
- Address the two Important findings before merge — the occurrence-count narrowing at
:239is the one that can silently clear a real finding, and it is a one-line change tomatchAll. - Consider the Suggestions opportunistically; both are comment-accuracy, not behavior.
- Note this PR is subject to the bug it fixes only in the clean case — this review reports findings, so
review/ally-commentwill go red here on its merits, not on the defect.
Fixes BLO-31890.
The defect
hasActionablePrReviewFeedbackended with an unguarded prose rule:Ally's clean Recommended-Action template is:
That sentence carries
Recommended Action…fix…before merge— the same three tokens a blocking one carries. So the rule matched the exact prose that declares a PR mergeable, and the gate went red precisely when the review was clean.Measured blast radius (2026-09-04/05)
Three PRs simultaneously
failureonreview/ally-comment, each with a review at its current head reportingCritical Issues (0)/Important Issues (0):review/ally-commente0c050d6failure69c6e673failuree9a66437failureOne predicate, three poisoned call paths
All cured by this change:
pr-comment-review-gate.ts:312→:317headsWithUndispositionedFinding:282→:353<sha>is still undispositioned"github-webhook.ts:3402The second one explains the earlier message on #1571 at 22:37Z:
fa550ed's review was clean, and was still counted as a live finding.The fix
Route the rule through
hasNonNegatedMatch, exactly as the twochanges requestedrules above it already do — this was the only prose rule that did not. The guard is applied per occurrence, not per block, so a clean first line cannot mask a real demand after it.What I deliberately did not do
My first cut also added
nothingtoNEGATION_CUE_REGEX. I dropped it:No, already inside the 8-word lookback. Verified against test(agent-health): restore executable preflight census #1571's real review body with the cue removed: still clean.changes requestedrules, so the cue makes"Nothing else to note, but changes requested for the auth path"stop blocking — fail-open, the direction this module argues against atRESOLVED_PRIOR_DISPOSITIONS(:92-97).Guard the rule, not the vocabulary. Both choices are pinned by tests.
Verification
reviewer taxonomy— 18 passed (14 pre-existing + 4 new)pr-comment-review-gate— 31 passedEvery new assertion is mutation-checked against the real module, each caught by exactly the test written for it:
nothingcuePlus a sweep asserting every
fix/before mergegap the previous 400/400 window admitted is still actionable, so the widened 820-char slice cannot narrow what used to block.Note for reviewers
This PR is itself subject to the bug it fixes: if Ally reviews it clean,
review/ally-commentwill go red on this PR too, for the same reason. That status is not in themasterruleset (onlymerge_queue), so it does not mechanically block — but do not read a red here as a finding without opening the review body.