Skip to content

fix(ci): isolate merge-group control gates from PR saturation (BLO-21953) - #1287

Merged
kkroo merged 1 commit into
masterfrom
kkroo/blo-21953-merge-gate-routing
Aug 11, 2026
Merged

fix(ci): isolate merge-group control gates from PR saturation (BLO-21953)#1287
kkroo merged 1 commit into
masterfrom
kkroo/blo-21953-merge-gate-routing

Conversation

@kkroo

@kkroo kkroo commented Aug 11, 2026

Copy link
Copy Markdown

Thinking Path

  • Paperclip is the control plane used to manage AI-agent work, and every landing change depends on its merge-group CI completing promptly.
  • Heavy merge-group lanes already use the dedicated arc-merge-queue pool, but the initial policy and Helm chart gates and final verify gate still use arc-light.
  • During the post-fix(ci): load-shed stale merge-group generations (BLO-21953) #1260 restage, arc-light was saturated at 24 runners while arc-merge-queue had zero active runners.
  • The sole queue candidate therefore waited for PR and review traffic before it could even admit its dedicated heavy lanes.
  • This pull request routes only merge-group instances of those three control gates to arc-merge-queue; pull-request instances remain on arc-light.
  • The benefit is queue isolation from PR bursts at both admission and completion without increasing runner limits.

Linked Issues or Issue Description

No GitHub issue exists for this infrastructure incident. It is tracked in BLO-21953.

Observed on 2026-08-11 after #1260 merged: merge-group run 31450284773 had both bootstrap jobs queued while arc-light reported 23 running and 1 pending runner and arc-merge-queue reported 0 running or pending runners. The same contention also applies to the final required verify job.

Related work:

What Changed

  • Route merge-group policy, Helm chart, and verify jobs to arc-merge-queue.
  • Preserve arc-light routing for normal pull-request runs.
  • Extend the existing merge-group control regression test to pin all three routing contracts.

Verification

  • node --test scripts/__tests__/merge-group-concurrency.test.mjs - 3/3 pass.
  • node scripts/check-github-runner-labels.mjs - all 20 workflows use allowed ARC labels.
  • node --test scripts/check-github-runner-labels.test.js - 10/10 pass.
  • ruby -ryaml -e "YAML.load_file(%q[.github/workflows/pr.yml])" - workflow parses.
  • git diff --check - clean.

Risks

Low risk. The destination pool already executes the heavier jobs from the same merge-group workflow and has a configured maximum of 12 runners. Pull-request and e2e routing are unchanged. A future pool rename must update the existing fail-closed runner-label policy.

Model Used

OpenAI Codex based on GPT-5, with repository, GitHub CLI, shell, and Kubernetes tool access.

Checklist

  • I have included a thinking path that traces from project context to this change
  • I have specified the model used (with version and capability details)
  • I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
  • I have searched GitHub for duplicate or related PRs and linked them above
  • I have either (a) linked existing issues with Fixes: # / Closes # / Refs # OR (b) described the issue in-PR following the relevant issue template
  • I have run tests locally and they pass
  • I have added or updated tests where applicable
  • If this change affects the UI, I have included before/after screenshots - N/A, CI-only change
  • I have updated relevant documentation to reflect my changes - inline workflow rationale
  • I have considered and documented any risks above
  • All Paperclip CI gates are green
  • Greptile is 5/5 with no open P2s, recommendations, or follow-ups - N/A, not configured
  • I will address all Greptile and reviewer comments before requesting merge

@allyblockcast

allyblockcast Bot commented Aug 11, 2026

Copy link
Copy Markdown

🔗 Paperclip issue: BLO-21953

@cursor

cursor Bot commented Aug 11, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@kkroo

kkroo commented Aug 11, 2026

Copy link
Copy Markdown
Author

@ally please review exact head 2a519c3ff7d4da40c501ff4a6fb17c24318707d8 for merge-group control-gate isolation from arc-light saturation (BLO-21953). The focused routing and runner-label policy tests are green.

@kkroo
kkroo enabled auto-merge August 11, 2026 01:56
Co-Authored-By: Paperclip <noreply@paperclip.ing>
@kkroo
kkroo force-pushed the kkroo/blo-21953-merge-gate-routing branch from 2a519c3 to 6ac8378 Compare August 11, 2026 01:57
@kkroo

kkroo commented Aug 11, 2026

Copy link
Copy Markdown
Author

@ally authoritative review request: please review exact head 6ac8378849f50ef511d0ce499a489c4605da668b. This supersedes the earlier SHA; the code tree is unchanged and the only delta is the required Co-Authored-By: Paperclip <noreply@paperclip.ing> commit trailer.

@kkroo
kkroo requested a review from allyblockcast August 11, 2026 01:59

@allyblockcast allyblockcast Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 6ac8378

Looks good. The merge-group-only runner expressions isolate the initial policy and Helm chart gates plus the final verify aggregation gate, while retaining arc-light for pull-request runs. The regression test pins all three control jobs, and the existing fail-closed runner-label guard validates the expression branches.

Critical Issues (0)

Important Issues (0)

Suggestions (0)

Strengths

  • The scope is limited to the specific control-gate starvation path.
  • The test checks the affected job blocks rather than merely matching a label globally.

Recommended Action

  1. Merge when required checks complete.

@allyblockcast allyblockcast left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 6ac8378

Looks good. The merge-group-only runner expressions isolate the initial policy and Helm chart gates plus the final verify aggregation gate, while retaining arc-light for pull-request runs. The regression test pins all three control jobs, and the existing fail-closed runner-label guard validates the expression branches.

Critical Issues (0)

Important Issues (0)

Suggestions (0)

Strengths

  • The scope is limited to the specific control-gate starvation path.
  • The test checks the affected job blocks rather than merely matching a label globally.

Recommended Action

  1. Merge when required checks complete.

@kkroo
kkroo merged commit 80350da into master Aug 11, 2026
1 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants