Skip to content

fix(ci): fork first-principles sweep passes the computed verdict - #5966

Merged
NicholasRBowers merged 1 commit into
mainfrom
fix/fork-fp-reference-lane-verdict-5949
Aug 26, 2026
Merged

fix(ci): fork first-principles sweep passes the computed verdict#5966
NicholasRBowers merged 1 commit into
mainfrom
fix/fork-fp-reference-lane-verdict-5949

Conversation

@chenmingwei23

Copy link
Copy Markdown
Contributor

Problem / Motivation

The four ported fork reviewer lanes (design/gpt/opus/ux) complete their stranded check-runs during the finalize sweep by passing the run's own computed verdict ($conclusion / $title). The reference lane, fork-first-principles-review.yml, still passed a hardcoded neutral / "review incomplete (advisory)" at its sweep call site -- the shape #5928 fixed in the four lanes it ported the sweep to, deferred there only because that PR's spec pinned the reference lane as unmodifiable.

Why it matters

A hardcoded conclusion at the sweep site can do one of two bad things on a fork PR:

  • Outvote a genuine green re-run: pr-readiness reads check-runs with fail-precedence, so a swept-to-neutral run sitting beside a real success degrades the lane's reported verdict.
  • Launder a genuine BLOCK: a run whose own finalize PATCH lost both attempts falls through to the sweep, which would publish "review incomplete (advisory)" over a verdict that was actually a premise-concern BLOCK -- the one state the advisory lane exists to surface.

What changed (motivation -> approach -> change)

The sweep's job is to close the stranded-run wedge, not to decide the verdict. The run already computes the correct verdict: the case statement earlier in the same step assigns $conclusion / $title on every path (unconditional *) arm), and with no verdict computed they already hold the neutral/"review incomplete" posture. So the fix is the same one-line shape the ported lanes carry: the sweep call passes "$conclusion" "$title" instead of the hardcoded pair, with the ported lanes' justifying comment mirrored above it. A genuinely-stranded run's advisory behavior is unchanged; only the case where a real verdict exists changes -- it now propagates.

The TestForkLaneStrandedRunSweeps ratchet (added by #5928) is extended: the reference lane joins FORK_SWEEP_LANES (all three existing pins -- PR-scoped external_id at creation, sweep presence, external_id-scoped selection -- hold on it unchanged), and a new test_sweep_completes_with_the_computed_verdict asserts all five lanes pass the computed verdict and the hardcoded-neutral sweep shape cannot come back.

Tests

  • New ratchet assertion is mutation-verified: red on the unfixed workflow (exactly the reference-lane param fails), green with the fix.
  • test/test_ai_review_workflows.py: 168/168 pass; TestForkLaneStrandedRunSweeps 20/20 (4 tests x 5 lanes).
  • isort / flake8 clean; full backend pytest run -- the only failures are environment-owned (byte-identical failure set on pristine main in the same environment, A/B verified via stash).
  • Both pre-push review lanes (GPT, Opus) returned NO BLOCKING FINDINGS; the Opus lane additionally executed the extracted step body under stubbed gh/jq across all VERDICT values (BLOCK/PASS/CONCERNS/SKIPPED/empty/UNKNOWN/WITHHELD) with set -u: no unbound variable on any path, and the negative assertion does not false-positive on the legitimate POST fallback's -f conclusion="neutral".

Manual verification

Verified all five lanes now carry the identical sweep call literal (fork-design:644, fork-first-principles:683, fork-gpt:785, fork-opus:520, fork-ux:836), and that $conclusion / $title are assigned on a flat path with no early exit between the case and the sweep.

Screenshots / video

CI-workflow + backend-test change only; no visual surface.

Related Issues

Closes #5949

Checklist

  • Single conventional commit
  • Tests added/updated and mutation-verified
  • No user-facing strings, no spec-doc surface affected

Contribution License Agreement

By submitting this pull request, I confirm my contribution is made under the terms of the repository's license.

The reference lane's stranded-run sweep completed check-runs with a
hardcoded neutral / 'review incomplete (advisory)', while the four
ported lanes pass the run's computed verdict at their sweep sites. A
hardcoded conclusion either outvotes a genuine green re-run under
pr-readiness's fail-precedence, or launders a genuine BLOCK whose own
PATCH lost both attempts into an un-gated neutral. Pass $conclusion /
$title instead; with no verdict computed they already hold the
incomplete/advisory posture, so a genuinely-stranded run is unchanged.

Extends the TestForkLaneStrandedRunSweeps ratchet: the reference lane
joins FORK_SWEEP_LANES (all three existing pins hold on it), and a new
computed-verdict assertion covers all five lanes.

Closes #5949
@chenmingwei23
chenmingwei23 requested a review from a team as a code owner August 25, 2026 23:44
@chenmingwei23
chenmingwei23 requested a review from dwu96 August 25, 2026 23:44
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Aug 25, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of edc99edf3b4777348b993e8a569672f6d76df0be — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

One-line convergence to the shape the four ported lanes already carry, with a ratchet test that pins all five — sound and minimal.

[DESIGN-REVIEWED] edc99ed

@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of edc99edf3b4777348b993e8a569672f6d76df0be and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] edc99ed

False positive or not applicable? A repository writer can comment:
/ai-review override gpt edc99edf3b4777348b993e8a569672f6d76df0be: <one-sentence reason>

@github-actions

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed edc99edf3b4777348b993e8a569672f6d76df0be — this comment is updated in place on each push.

Review details

The diff is fully contained in one run: shell block. conclusion and title are assigned unconditionally by the case statement (lines 629–637, with a *) default), so they are always in scope and set -u-safe at the sweep site (line 683). Values are properly quoted; both originate from the workflow's own VERDICT output, not attacker text. The sweep is external_id-scoped to this PR, so it never touches a sibling PR's check-run, and completing a same-head stranded run with this run's verdict is consistent because both runs review the same commit. The test changes are test-only (and test/ is exempt regardless). Nothing survives falsification and I found no grounded new defect.

No findings.

[OPUS-REVIEWED] edc99ed

Verdict parsed from the review's SHA-scoped output markers for commit edc99edf3b4777348b993e8a569672f6d76df0be.

False positive or not applicable? A repository writer can comment:
/ai-review override fable edc99edf3b4777348b993e8a569672f6d76df0be: <one-sentence reason>

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of edc99edf3b4777348b993e8a569672f6d76df0be — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

All claims verified. Ready to emit the review.

First-Principles-Verdict: PASS

One-line change deletes the last hardcoded sweep verdict; every item is the fix, its ratchet, or its pin — nothing rides along.

What this change ships

Intent: make the reference fork lane's stranded-run sweep publish the verdict the run actually computed instead of a hardcoded neutral — a FIX (#5949).

  1. Stranded check-runs now complete with the run's real verdict, not "review incomplete" — justified
  2. Reference lane joins the five-lane sweep ratchet (FORK_SWEEP_LANES) — justified, but re-pins two shapes older tests already pin
  3. New test pins the computed-verdict sweep and forbids the hardcoded-neutral shape, all five lanes — justified
  4. Comment above the sweep mirroring the ported lanes' rationale — justified
  5. Ratchet-class docstring updated to name all five lanes — justified

The fix is the deletion this lane would have asked for: it removes the one divergent spelling rather than guarding it. Grepped complete "$id" "neutral" across .github/workflows/ — 0 remaining; all five lanes carry the identical "$conclusion" "$title" literal. $conclusion/$title are assigned by an unconditional case (fork-first-principles-review.yml:629-637) with a flat path to the sweep, and the sweep's external_id scope means the verdict applies only to this PR's own runs on this head. Description and diff match exactly.

Subtractions

  • Delete the two reference-lane-only tests test_fork_finalize_sweeps_stranded_check_runs (test/test_ai_review_workflows.py:659) and test_sweep_only_completes_check_runs_this_pr_created (:671): now that the reference lane is in FORK_SWEEP_LANES, the parametrized tests at :1320/:1330 assert the same literals (completing stranded check-run, the external_id select, the PR guard, the unscoped-sweep negative), and the retry/warning strings are pinned by test_the_helper_matches_the_reference_lane (:1081). Two spellings of one pin will diverge.

[FIRST-PRINCIPLES-REVIEWED] edc99ed

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Aug 26, 2026
@NicholasRBowers
NicholasRBowers enabled auto-merge (squash) August 26, 2026 00:27

@NicholasRBowers NicholasRBowers left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: fix (2 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: fix(ci) with a clear root cause — the fork first-principles finalize sweep passed a hardcoded neutral instead of the run's computed verdict; one-line call-site fix mirroring the four already-ported lanes, plus a mutation-verified ratchet test pinning all five lanes.

@NicholasRBowers
NicholasRBowers merged commit 61c89d4 into main Aug 26, 2026
65 checks passed
@NicholasRBowers
NicholasRBowers deleted the fix/fork-fp-reference-lane-verdict-5949 branch August 26, 2026 00:27
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Aug 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fork-first-principles-review.yml sweep completes stranded runs as hardcoded neutral

2 participants