Repository navigation
Scheduler pre-review update-branch discards in-flight checks under queue saturation: 76/77 merges since 09-04 had 0/12 required contexts at merge time #1935
Description
Activity
Rate measured after the #1926 merge (
3f88e13a, 13:57:04Z) — peer session's readings at +0/+5/+10 min,.githubonly:t queued bot pushes so far runs created since merge of which bot-triggered +0 (13:57:11Z) 610 (+11) 0 17 0 +5 (14:02:06Z) 614 (+15) 1 44 15 +10 (14:07:10Z) 600 (+1) 6 90 60 Bot pushes arrived at 13:58:46, 14:03:11, 14:03:17, 14:05:18, 14:05:40, 14:06:31Z — roughly one every 100 s, spread evenly, not a burst at the merge. Each push spawns ~10 runs (60/6), so the drip injects ~6 runs/min while the queue's 10-minute net change is +1: cancellations leave at the same rate. The queue is a treadmill, which is why depth has oscillated between ~500 and ~600 all day without moving in either direction.
One scheduler completion ≈ one bot push (victim chosen by the run's scan, not necessarily its own PR: a
codex/originweave-mv3-ownerrun updated #1857) ≈ ~10 new runs + every queued check on that PR cancelled. On #1484 the push also cancelled the PR's own queued scheduler run 6 s later.Consequence for #1926's behavioural evidence (#1730's two CodeQL shards re-running):
codeql-scan-dispatch.ymlhas to survive this treadmill first, so that evidence may not appear until the guard lands. Guard PR follows.독립 검증 + 속도 실측 (host2 세션)
이 이슈의 기전을 peer1 세션이 찾았고, 아래는 제가 별도 검색어와 별도 사례로 확인한 것입니다. 인용이 아니라 실행입니다.
소스 (origin/main
3f88e13a)scripts/ci/pr_review_merge_scheduler_core.py:4774 if behind_by and trigger_reviews: → update 경로 :4740-4790 in_progress|queued|pending|check_run|statusCheckRollup 언급 0건 .github/workflows/pr-review-merge-scheduler.yml:139 UPDATE_BRANCHES = schedule || push || pull_request_target || …현재 head의 check 상태를 보지 않고 update합니다.
pull_request_target에서도 이 경로가 열려 있습니다.사례 1 — #1926 (
7920a70b→ 봇 pushc730ae0b)스케줄러 run 33963177129 pull_request_target created 11:22:53Z job 13:44:14Z~13:44:40Z (2h21m 대기) 봇 push 13:44:37Z ← job 완료 3초 전 이전 head 7920a70b check-runs: total 28 cancelled ±1min 22 success 3 skipped 3사례 2 — #1484 (
13c444fe→ 봇 push777ee3ed, #1926 머지 +102초)이전 head 13c444fe check-runs: total 30 cancelled ±60s 21 cancelled 그 외 0 success 8 skipped 1 → 21/30. success 8건이 superseded head에 고아로 남음. #1484 자기 스케줄러 run 33964505248 (11:52:51Z 생성)은 push 6초 *뒤* 14:58:48Z에 cancelled — 원인이 아니라 희생자.희생 PR은 유발 PR이 아닙니다
33965444839(codex/originweave-mv3-owner의 pull_request_target, 14:03:11Z success)의 scan 로그(##[endgroup]이후):PR #1857: update_branch: current head has no OpenCode approval; branch is outdated before review dispatch; … 스캔한 PR 수: 1 update_branch 결정: 1 (#1857)originweave의 run이 #1857을 밀었습니다. 완주 1건 = push 1건, 희생 PR은 스캔이 고른 것. 그래서 #1484처럼 자기 run이 push에 취소되는 사례가 생깁니다.
속도 (#1926 머지 후 t0/t5/t10,
gh --jq끝까지)[t0 13:57:11Z] queued=610 (+11) | heads 2/100 | runs 17 bot-runs 0 bot-branches 0 [t5 14:02:06Z] queued=614 (+15) | heads 3/100 | runs 44 bot-runs 15 bot-branches 1 [t10 14:07:10Z] queued=600 (+1) | heads 7/100 | runs 90 bot-runs 60 bot-branches 6 봇 push 도착: 13:58:46 · 14:03:11 · 14:03:17 · 14:05:18 · 14:05:40 · 14:06:31Z (~100초 간격)push당 run ~10개, 유입 분당 ~6 run, 10분 순증 +1. 취소가 유입과 같은 속도로 빠집니다. 큐는 트레드밀입니다 — 오늘 큐 깊이가 500~600 사이에서 회전만 하고 어느 방향으로도 안 간 이유입니다. 머지 직후 버스트가 아니라 스케줄러 완주율에 묶인 꾸준한 방울이며, 머지는 P(behind)를 1로 만들 뿐입니다.
한계
- #1484를 실제로 민 run은 못 찾았습니다(자기 run·근접 run·fix(dispatch): parse the trusted-dispatcher allowlist identically in all three consumers (#1929) #1932 push-sweep·fix(codeql): serialise the dispatched scan matrix with toJSON #1926 push-sweep 전부 제외; 후자는 14:06Z 기준 아직 pending). 앱 토큰을 쓰는 다른 워크플로일 수 있습니다. 스케줄러 경로에 가드가 들어가면 그건 별건입니다.
- 위 속도는
.github한 저장소, 10분 창입니다.
수정 방향(현재 head에 대기·진행 중인 필수 check가 있으면
update_branch대신wait)에 동의합니다. 트리거 형태와 무관하게 그 가드가 막습니다.Fix PR: #1937 —
has_in_flight_check_runs+ awaitguard in the pre-review update path, no age cap, contract test with positive/negative/empty cases,CLAUDE.mdnow names both update paths. Gate on the PR head: 2893 passed / 100% coverage / 100% docstrings; the new test fails againstorigin/main's core.- added a commit that references this issue
on Sep 5, 2026 Residual old-code drip removed: cancelled 40 queued scheduler runs created before the #1937 merge.
pr-review-merge-scheduler.ymlhas noactions/checkout; it fetches the central scripts as a tarball atTRUSTED_SOURCE_REF= the run'sworkflow_sha(:313-352), which is fixed when the run is created. So every scheduler run created before7f4c5e3elanded (14:17:16Z) would still execute the guard-lesscore.pywhen it finally got a runner — one victim each, for another 2–3 h. At 14:2xZ all 40 queued scheduler runs in this repository were pre-merge (oldest 10:52:16Z; 37pull_request_target, 1push, 2repository_dispatch); none were in progress. Cancelling a queued run costs no runner time, and each affected PR is rescanned by the hourlyschedulesweep and its next event with the new code, so nothing durable is lost..githubqueue depth went 608 → 568.From here on every scheduler run in this repository carries the guard, so the post-merge predictions (bot pushes → ~0,
cancelled_at_push= 0, queue trending down) are tested cleanly. The same residual exists in every sibling repository where the ruleset injects this workflow; counting it there is in progress before any decision.Cancelled run ids:
33971122915,33971046733,33971024024,33970822494,33970780040,33970761084,33970662388,33970657142,33970601476,33970578666,33970439323,33970363093,33970215310,33970209470,33969870992,33969842287,33969835753,33969784791,33969715681,33969691083,33969631593,33969604212,33969537936,33969389572,33969329111,33969197240,33969142435,33969140347,33969062391,33968270218,33968197172,33967973427,33967682868,33967556296,33967512492,33967260496,33966978881,33966845220,33966317530,33961798794Baseline correction (peer measurement): the evidence discarded per push is ~10–11 checks, not 21–22.
Check-runs for jobs behind
needs:do not exist until their upstream completes; when a push cancels the workflow they are created and cancelled in the same instant. Counting only check-runs withstarted_atbefore the push:replaced head cancelled at push real in-flight phantom (created at cancel) #1926 7920a70b22 10 12 #1484 13c444fe21 11 10 The mechanism and the target (0 real in-flight cancellations per bot push after #1937) are unchanged; the numbers in the issue body, the #1937 description and its CHANGELOG entry ("22/28", "21/30") are the cancelled-at-push totals and overstate the discarded evidence by about 2×. Post-merge readings use the
started_at < pushfilter with a separatephantom=counter.- added a commit that references this issue
on Sep 5, 2026 Org-wide residual removed too, and the first post-merge readings.
The ruleset injects the same scheduler into sibling repositories, with the same creation-time
workflow_shatarball, so their queued pre-merge runs carried the oldcore.pyas well. A peer counted 43 across 19 repositories; my own per-repository query (statusqueued,pathending inpr-review-merge-scheduler.yml,created_at< 14:17:16Z) found 42 (clearfolio 2 vs 3 — one presumably left the queue in between). 38 cancelled; 4 cancel calls were refused — exactly the four zombies queued since 2026-07-13 (naruon, disksage), 08-19 (inkspan) and 09-02 (TEPP), which the API no longer accepts a cancel for and which are unlikely ever to run.One straggler explained. At 14:21:00Z a bot merge landed on #1911 (
ci/main-full-suite-gate), cancelling 12 real in-flight checks (+14 phantom). Its producer was scheduler run33966317530(created 12:32:11Z,workflow_sha19f8ff5f, i.e. old core): it was assigned a runner at 14:20:34Z — 26 s before the.githubcancellation sweep reached it — issued the update-branch, and was cancelled at 14:21:03Z. Thein_progress = 0check preceding the sweep was true when taken; sequential cancels took ~40 s. Not a guard defect. A peer also enumerated every update-branch call site onmain: the scheduler core is the only one that can actually update (opencode-review-dispatch.ymlcalls it with--no-update-branches,agent_mention_router.pyhard-codesupdate_branches: False), so there is no second pusher.Independent post-merge count (surviving-head instrument, 14:25:00Z): of 137 open PRs, exactly 1 head is a bot merge-commit created after 14:17:16Z — #1911 at 14:21:00Z, the straggler above. Under the pre-merge regime the same 8 minutes produced ~5. Predictions ① and ② therefore hold from 14:21:03Z onward; ③ (queue trend) is read against 568.
Also observed live in the same window:
opencode-review-dispatchrun33966200868still rejectedactor=opencode-agent[bot]at 14:20:09Z — the variable behind #1929 is unchanged.Sibling cancellations by repository — ContextualWisdomLab.github.io:
33971329739,33970961103; Orgmetra:33971182649; TEPP:33971132327,33970743395,33648251504; appguardrail:33971327056,33971019552,33968624177; argos:33971093164,33970864336,33967872388,33966466101; bandscope:33971280631,33970256883,33969894325,33968011439; clearfolio:33970280235,33967679022; contextual-orchestrator:33970774990,33970678974,33967327860,33967146342,33967113270; disksage:29257025640; fast-mlsirm:33970736655,33967986067; html4tree:33970255670,33969794800,33969382246; inkspan:33969215494,32214389134; linux-cluster-ops:33970388678; naruon:33969245374,33968577535,29254301754; nonnest2:33970374725; pg-erd-cloud:33970777607,33970152562; scopeweave:33971173311; seedream_evasepic:33968018597; xtrmLLMBatchPython:33970799642⑥ after #1937 (
7f4c5e3e, merged 14:17:16Z) — t5/t10/t15Instrument: per bot-pushed branch since merge, count the previous head's check-runs
cancelledwithin ±60 s of the push withstarted_at < push(real in-flight); check-runs created by the cancellation itself (started_at ≥ push) are reported separately asphantom(see the phantom split in the comment above). Pre-guard references: #1926 = 10 real (+12 phantom), #1484 = 11 real (+10 phantom).read UTC queued bot pushes since merge real in-flight discarded t5 14:22:21 568 1 — ci/main-full-suite-gate(#1911) @14:21:0012 (+14 phantom) t10 14:27:17 580 1 — t15 14:32:23 589 2 — claude/profile-catalog-reconciliation-v2(#1687) @14:27:147 (+12 phantom) Both pushes were made by scheduler runs created before the merge, executing the old core. The scheduler fetches its source at
TRUSTED_SOURCE_REF = workflow_sha, fixed at run creation, so a run queued before 14:17:16Z runs the unguarded core whenever it finally gets a runner.- 14:21:00 (ci(tests): run the full suite on every main push #1911): run
33966317530, created 12:32:11Z, scan job got a runner at 14:20:34Z (10 steps), sent the async update-branch, then was cancelled at 14:21:03Z by the pre-merge-run sweep — 3 s too late. The other cancelled runs sampled hadsteps=0 runner=0. - 14:27:14 (docs: retire removed repository from governance records #1687): run
33966913388, created 12:44:53Z, scan job 14:26:34→14:27:17Zsuccess(push 3 s before completion — the fix(codeql): serialise the dispatched scan matrix with toJSON #1926 signature). Its log:TRUSTED_SOURCE_REF: f250638827f8252b0d9e5cb2601f4d333f96162f(main as of 12:44Z),PR #1687: update_branch: … branch update requested with OpenCode app token,counts: {update_branch: 1}. The new core would have loggedholding the update. This run was inside the sweep's created-range and on its first page (only 72 scheduler runs were created after 12:44:53Z), so the sweep's list or its cancel call missed it.
Second-pusher hypothesis refuted at source. Every
update-branchcaller onmain:pr_review_merge_scheduler_core.py(the scheduler),opencode-review-dispatch.yml:7788which passes--no-update-branches,agent_mention_router.py:371which hard-codes"update_branches": False, and a test script. Only the scheduler core can update a branch, and #1937 guards that path.State now:
created=<2026-09-05T14:17:16Z&status=queued→total_count=0,in_progress→ 0 (server-side filter, no page trap). Every scheduler run from 14:27:17Z forward executes7f4c5e3e. ⑥ restarts from that instant with a fresh +5/+10/+15 series.Queue delta is not bot churn: of the 98 runs created after 14:22Z, 67 were triggered by session pushes (#1717
14d40755, #1936c7115fe3, #1629) and 28 by the bot, across 8 branches. Prediction 3 (depth trending down) must be read net of session pushes.Cross-check on a different artifact (peer session): filtering the 137 open-PR head commits for bot-authored two-parent commits after 14:17:16Z yields exactly the same two events.
- 14:21:00 (ci(tests): run the full suite on every main push #1911): run
Second straggler (#1687, 14:27:14Z) explained — and the sweep's blind spot closed.
The producer
33966913388(created 12:44:53Z, old core,TRUSTED_SOURCE_REF f2506388) was not in the cancellation list (40 listed, 40 cancelled, 0 failed), although it shares the sameworkflow_id299236240andpathand its only job waited until 14:26:34Z. Three listing instruments agree with each other now (workflow-runstotal_count9 = paginated 9 = repo-runs path-filtered 9), so this was not an API omission but a status filter: the scheduler's concurrency group is per repository and per PR (central-pr-review-merge-scheduler-{repo}-pr-{n}), so a PR with two pending events has its second run parked in statuspending, whichstatus=queueddoes not return. Every count so far — mine, the 43 across siblings, and "remaining old runs = 0" — read onlyqueued.Re-measured with
status=pending|waiting|requested+created<14:17:16Z+ path across.githuband the 19 sibling repositories: 5 more old-core scheduler runs —.github33969771830,33969684836,33969659836(13:42–13:44Z), naruon33969517187, appguardrail33968100685. All five cancelled (0 failures) at ~14:39Z. A full re-sweep over bothpendingandqueuednow leaves only the four zombies the API refuses to cancel (naruon/disksage 07-13, inkspan 08-19, TEPP 09-02).So the clean observation window for the guard starts at ~14:39Z, not 14:21Z or 14:27Z. Both stragglers so far (#1911 at 14:21:00Z, #1687 at 14:27:14Z) are old-core runs with the same signature (push seconds before the run completes,
update_branchin the log where the new core printsholding the update); neither is a guard defect.Metric note for the queue-trend prediction:
.githubcurrently has 37 runs inpending(any workflow) on top of thequeueddepth. Cancellingqueuedruns can be masked bypendingruns promoting intoqueued, so readqueued + pending, and exclude session-driven pushes (67 of the 98 runs created 14:22–14:32Z wereseonghobae-actor).Correction to my "state now" paragraph above. The claim "
created=<14:17:16Z&status=queued→ 0 … no page trap" was exact over the page dimension and blind over the state dimension: a scheduler run waiting on its concurrency group (pr-review-merge-scheduler.yml:79-91) sits instatus=pending, notqueued. That is also why the sweep missed33966913388— it waspendingat sweep time, not absent from the API. A peer session re-read withpending|waiting|requestedand found five more pre-merge scheduler runs org-wide:.github×3 (33969771830,33969684836,33969659836, created 13:42–13:44Z), naruon ×1, appguardrail ×1. All three.githubruns are nowcompleted/cancelled; my independent read at 14:41:01Z over all five states (pending,waiting,requested,queued,in_progress) withcreated<14:17:16Zreturns 0 for each.Clean start for ⑥ is therefore 2026-09-05T14:39:13Z (the last of the three cancellations), not 14:27:17Z. Bot pushes between 14:27:17Z and that time are attributable by the pushing run's
created_atexactly as the two above were.Queue-reading rule that falls out of this:
pendingis a separate pool fromqueued(at 14:41:01Z in.github: queued 582, pending 34, in_progress 15). A cancellation that promotes a pending run intoqueuedleaves the queued count flat, so prediction 3 must be read onqueued+pending. The instrument now reports both.General form, for anyone repeating this: a server-side
status=filter is exact only over the states it names — enumeratepending|waiting|requested|queued|in_progressbefore calling a count "remaining".Guard verified in production (16:18Z). Of the 25 scheduler runs created in
.githubafter7f4c5e3elanded, 10 have executed. Their decisions:run PR decision 33971948808#1936 wait— …branch is outdated before review dispatch, but current-head checks are still queued or running; holding the update so their evidence is not discarded33971906521#1934 wait— same reason33971691883#1872 wait— same reason33971548421#1911 review_dispatch— current head has completed Strix evidence; same-head OpenCode dispatched33972657372#1912 security_dispatch33971862168#1687 security_dispatch33972093234#1629 block— 13 unresolved review threads33971711250#1661 block— 1 unresolved review thread33971916754#1717 skip— draft PR33971360998push sweep skip/blockdecisions onlyupdate_branchdecisions: 0 / 10. The three holds are exactly the PRs the old core would have pushed (behind, unreviewed, checks in flight).Queue (
.github,queued+pendingas agreed): 616 at 14:43Z → 509 at 16:18Z (−107 in 95 min), within_progress18 → 24. The treadmill has stopped spinning: cancellations no longer refill the queue, and slots are being spent on runs that finish.Remaining, out of this issue's scope: the post-approval update under
strict=true(bounded, one refresh per approval), the dispatch-actor variable (#1929 — #1911's dispatch above will still be rejected at the gate until it is set), and draft PRs, which none of the gates ever approve (see #1913).⑥ closed — the guard holds on a clean window
Clean start 14:39:13Z (last pre-merge scheduler run cancelled; see the correction above). Same instrument: per bot-pushed branch, the previous head's check-runs cancelled within ±60 s of the push with
started_at < push.window UTC bot pushes real in-flight discarded queued+pending t5b 14:27:17→14:41:16 0 new 0 616 t10b →14:46:22 0 new 0 628 t15b →14:51:28 0 new 0 624 long 14:39:13→16:20:01 (101 min) 2 0 501 The two pushes in the long window:
automation/sbom-inventory@15:28:30Z is the SBOM automation commit, not an update-branch (0 check-runs on the previous head).fix/agent-mention-dispatch-credential-…@15:46:59Z is the real test: the previous head38b300a3carried 37 check-runs and lost none — nothing was in flight when the scheduler updated it, which is precisely the guard's contract. Under the pre-guard rate (~1 push/100 s, ~10 real in-flight each) this window would have shown ~60 pushes.Independent artifact (peer session, scheduler decision logs, 16:18Z): of the 25 scheduler runs created after
7f4c5e3e, 10 executed —wait×3 with the guard's own reason string (#1936, #1934, #1872),review_dispatch1,security_dispatch2,block2,skip1, push sweep 1,update_branch0/10. Check-run observation and decision-log census agree.Predictions 1–3 hold. What the guard does not fix (peer1's scope-out at the top of this issue) is now the visible next layer: #1911 reached
review_dispatchfor the first time today and will be rejected at the dispatch actor gate until the #1929 variable is set; the CodeQL dispatch pipeline sits at 101 queued / 0 completed since #1926.Correction to the last line of my ⑥-closed comment: "the CodeQL dispatch pipeline sits at 101 queued / 0 completed since #1926" was a page artifact — with 110 dispatch runs queued, the 100 most recent by creation contain no completed run. Server-side since #1926 (13:57:04Z): 71 completed, all 71 failures, 0 successes; 55/0 in the hour before. Each fails in
validate-dispatchat the actor gate (codeql-scan-dispatch.yml:142-164,vars.OPENCODE_REPOSITORY_DISPATCH_ACTOR) withauthorization rejected actor=opencode-agent[bot] … must match one configured scheduler identity, and the matrix job carrying #1926's fixed step is skipped. #1926's behavioural proof is therefore not "pending" but blocked by #1929: 71 consecutive dispatches have been turned away at that one variable. Same finding as the direct probe run33969658525(rejected 15:07:07Z) and the dispatch run33966200868(14:20:09Z).Next layer, baselined — "green on the current head AND up to date" (the scope-out at the top of this issue)
Full census of
.github's open PRs at 16:52Z via GraphQLmergeStateStatus+statusCheckRollup(3 pages, 138 rows =totalCount, so complete; a first attempt returned exactly 100 rows from a failed second page and was discarded).mergeStateStatus PRs BEHIND 85 DIRTY 35 BLOCKED 16 UNSTABLE 2 CLEAN 0 Non-draft (105): current-head rollup FAILURE 89 / PENDING 16 / SUCCESS 0. The only
SUCCESSrollups in the repo are on 5 old draft heads, where most gates skip.What fails, by check name, on non-draft heads (each PR counted once per name; 38 PRs sit at the 40-context cap of the query, so these are lower bounds for those):
check PRs noema-review 36 CodeQL compatibility analysis (python) 30 CodeQL compatibility analysis (actions) 28 opencode-review 17 agent-review-runtime-quality 5 Scheduler, contextual-orchestrator, writer, and conflict-scope contracts 5 strix 4 quality / exact-head-path-policy / validate / repair 2 / 2 / 1 / 1 The top four are the structural classes already attributed on this issue and its neighbours: the CodeQL shards fail by design awaiting a dispatch verdict that #1929's variable blocks (71/71 dispatches rejected since #1926);
noema-reviewis dominated by the gateway 502 class (12 of the last 15; CO#1045);opencode-reviewverdicts are gated by the same variable. The tail (~20 PR-checks) may be real and is the review lane's material.The guard's cohort. 16 non-draft PRs currently carry no failing check on their head — 12
BLOCKED, 4BEHIND— because their heads are fresh (today's conflict resolutions) and their checks are still queued: #789 #1382 #1397 #1545 #1623 #1661 #1678 #1687 and eight more (head SHAs recorded). #1937 guarantees those checks will now complete rather than be cancelled. Prediction, written before observation: none of the 16 reaches rollupSUCCESS; they land inFAILUREon the two CodeQL shards (by design, until #1929) and, for a majority, onnoema-review(502 class). Read at ~19:00Z. If that holds, the queue and the guard are no longer the binding constraint on merges — one repository variable is.17 remaining items
First #1949 verdict sample (a peer's read of noema run
34013668803, sidecar created afterfb2ae81d): route preflightcandidate_count 24 / probed 16 / skipped 4 / deferred 2 / ready 6 / rejected 8(rejected =gemma-3404 ×4 + four probes that ran into the recvTimeoutError; ready includesllama-3.2-11bon both NVIDIA keys, which the alphabetical 4-per-key slice never reached). Readiness is back at the pre-#1939 level (5–6/12) → no rollback. Serving then walked 1689 s to502 provider_connection_error:provider_attempt×45, failures 28 (TimeoutError21,HTTPError7),circuit_*×26,unexpected_exception0 — the no-tools shape classifies its transport failures, as expected. Class: (5)-like capacity — routes that answer a 16-token probe and then never answer a real request (#1948 / contextual-orchestrator#1045).Correction to my pin note above: after
414f2297(contextual-orchestrator#1081single_attempt_transport) the no-tools path logsattempt=1/1per candidate too (42 of the 45 attempts here; the remaining 3 were1/3), notn/2. So on the new pin the attempt denominator no longer separates the two request shapes; use the HTTP status (500 internal_error= tools/passthrough leak until contextual-orchestrator#1082 lands;502 provider_connection_error= orchestrated walk) and the presence ofunexpected_exception type=(tools only). Per-candidate cost on a silent route is one recv timeout (~90 s) on either shape; the walk length is what differs.Second before-sample for the tools/500 leak, same shape (#1916 strix run
34008489633, job 05:08–07:24Z, artifactstrix-reports9984863327, pin414f2297): route preflight ready 4/12;provider_attempt×139,provider_attempt_failed error_type=TimeoutError×63 ↔request_failed status=500 code=internal_error×63 (1:1 again),HTTPError×23 →circuit_failure×13 /circuit_cleared×9 (the breaker still reacts only to HTTP errors), strixError code: 500×6 +503×1, fail-closed after 7586 s. With #1930 that is n = 2 for "one 500 per timed-out passthrough attempt, breaker blind to timeouts" on the current pin; contextual-orchestrator#1082 is the fix, and the after-sample is the first Strix run created once the sidecar pin passes it.n = 3 for the tools/500 leak on pin
414f2297, read independently of lane jan's table on #1948: #1946 strix run34008655751(artifact9984885189) — route preflight ready 4/12;provider_attempt×121,provider_attempt_failed error_type=TimeoutError×65 ↔request_failed status=500 code=internal_error×65;HTTPError×21 →circuit_failure×13 /circuit_reset×3; fail-closed after 7569 s. With #1930 (48 ↔ 48) and #1916 (63 ↔ 63) the three post-bump scans agree with jan's ledger (176 timeouts → 176 × 500, breaker blind to all of them; contextual-orchestrator#1081 held —attempt=1/1throughout, no2/3stacking). Owner decision on the preflight-probe deadline (my item above) is still open.- added a commit that references this issue
on Sep 6, 2026 Seventh class — recovered transient (found by a peer, confirmed independently on #1689 strix run
34013778497, artifactstrix-reports9985499801, sidecar bound tomain@fb2ae81d): the scan completed —run.jsonstatuscompleted, 423,895 tokens,strix.log07:06:48→08:09:35Z ending inStrix scan … done, SARIF with 0 results, strix's own report rendered — yet the job failed closed withSTRIX_PROVIDER_UNAVAILABLE: … orchestrator/free exhausted. The only failure signal in the artifact is threeWARNING … strix.core.execution: transient model/provider error for <agent>; replaying turn (attempt 1/5 | 2/5 | 3/5, backoff …): InternalServerError(…)lines — strix-agent 1.5.3's own retry, which succeeded (0ERRORlines, exit 0); exhaustion would log a separate ERROR and a non-zero exit. The quick gate's report check (strix_quick_gate.shhas_strix_report_failure_signal, agrep -Ei 'Warn|Warning|WARNING|Timeout'over the report logs) treats any WARNING as a failure signal, and the narrower Python allow-listknown_internal_warningonly knows theforcing tool continuation (n/m)family. Cost: a clean 63-minute scan on a 60-job-plan slot turned into a failure and a re-dispatch of the same head.Fix being drafted (host 2 → host 1 verification → my merge): allow the recovered retry warning explicitly (attempt
n/mwith the scan's completion still required), with the first sanitizer test for that allow-list. Census rule until it lands: aSTRIX_PROVIDER_UNAVAILABLEverdict whoserun.jsonsayscompletedand whosestrix.loghas WARNINGreplaying turnlines but no ERROR is this class, not a gateway failure.Classes now: (1) gateway walk — tools/500, no-tools/502; (2) verdict schema; (3)
gh pr diff300 files; (4) preflight-fail ready 0; (5) route-pass / gateway-fail ready 1; (6) Strix sandbox bootstrap (#1953, merged43024633); (7) recovered transient mislabelled as provider-unavailable.Class (4) on the post-#1949 launcher (a peer's read of contextual-orchestrator#1078 strix run
34017965035, created 07:00:10Z — afterfb2ae81d, before #1953): route preflightcandidate_count 24 / account_skip_after_429 2, thenpreflight_route_rejected … 429twice on each ofnvidia_nim,nvidia_nim_sub,openrouter→ all three accounts skipped → ready 0 →review sidecar preflight failed, sidecar exited beforehealthz, provisioning step failed 4 minutes in. No sandbox/bootstrap strings in the 1524-line log, so not class (6). The levers behaved exactly as designed (diversified pool, deferral, lazy fill with account skip); the pool was rate-limited end to end — #1948 territory.One measurable benefit is visible in that number: on the pre-#1949 launcher a dead hour cost the full 12 probes (up to 90 s each, ~393 s observed on #1911's old run before the walk even started); with the account-skip rule it costs 6 probes and the job fails closed in ~4 minutes, returning the runner slot instead of holding it. That is the wall-time bound by count and evidence, not by a deadline.
- added a commit that references this issue
on Sep 6, 2026 - added a commit that references this issue
on Sep 7, 2026 Live re-verification of the title defect, 2026-09-13 — proposing closure of this issue (the other failure classes recorded in the thread have their own landed fixes: #1944, #1953, #1960):
- The pre-review update-branch hold shipped in fix(scheduler): hold pre-review branch updates while current-head checks are in flight #1937 (
7f4c5e3e0,scripts/ci/pr_review_merge_scheduler_core.py:4816): an outdated PR is not updated while any newest current-head check is queued or running. - Observed in production today: scheduler run
34738720917for.github#2147decidedwait: current head has no OpenCode approval; branch is outdated before review dispatch, but current-head checks are still queued or running; holding the update so their results are not discarded— the exact scenario of the title ("update-branch discards in-flight checks under queue saturation") now resolves to a hold instead of an update. - Consequently the "0/12 required contexts at merge time" shape cannot be produced by the scheduler any more; the merges landing today (fix(scheduler): permit same-repository commit status reads #2121, fix(pingora): admit bounded HWPX evidence documents without UTF-8 decoding #2144, fix(codeql): grant private consumers pull-request and status reads #2145, fix(review-policy): key OpenRouter ZDR feed routes by model_id #2146) are explicit human bypass merges, not scheduler merges with discarded contexts.
If someone still holds an open class in this thread that is not covered by #1944/#1953/#1960/#1937, please split it into its own issue before closing; otherwise this can close as fixed-and-verified.
- The pre-review update-branch hold shipped in fix(scheduler): hold pre-review branch updates while current-head checks are in flight #1937 (
Triage (2026-09-18): close-as-done candidate
Unique delta appears on protected
mainvia merged #1937 (hold pre-review update-branch while current-head checks are queued/running). Procedure + core cite this issue; seedocs/pr-review-and-merge-procedure.mdandscripts/ci/pr_review_merge_scheduler_core.py.Ranked backlog: #2251. Spot-check one behind PR with in-flight checks before closing.
Closing after spot-check: unique valid delta is on protected
main.Merged fix: #1937 (merged 2026-09-05,
7f4c5e3e0efb7bfe29f33b60d4264858effd2996)Tree evidence on
origin/main:scripts/ci/pr_review_merge_scheduler_core.pyhas_in_flight_check_runsholds pre-review branch updates while current-head checks are queued/running (explicit#1935cite)CLAUDE.mdgovernance paragraph documents the same hold (see#1935)
Verified with
git greponorigin/main(not redundancy alone).- added a commit that references this issue
on Sep 18, 2026
Summary
Under queue saturation, the scheduler's pre-review
update_branchpath discards in-flight checks. A scheduler run waits 2h+ for a runner, executes aftermainhas moved, picks one behind PR from its scan — its own PR (#1926) or another one (acodex/originweave-mv3-ownerrun updated #1857) — and mergesmaininto that head before any review is dispatched. Every completing run therefore has one near-random victim, and the scheduler's completion rate is the PR cancellation rate. The push cancels every queued check on the old head and puts the new head at the back of the queue.mainmoves roughly every 28 minutes (77 merges into this repository in ~36h), so no PR head survives long enough for its 12 required contexts to complete.Measured on
#1926(2026-09-05):33963177129,pull_request_target, created 11:22:53Z, completed 13:44:41Z (success)c730ae0b(Merge branch 'main' into …)PR #1926: update_branch: current head has no OpenCode approval; branch is outdated before review dispatch; branch update requested with OpenCode app token inside GitHub Actions as OpenCode GitHub App7920a70bSame mechanism observed on
#1812(c9244ceb, 13:31:29Z) and, after the#1926merge, on#1484(777ee3ed, 13:58:42Z: 21 of 30 check-runs on the old head cancelled at the push, 8 successes orphaned, new head starts from zero). The pushes are not a burst caused by the merge: they are a steady drip, one per PR scheduler run that finishes its queue wait, and withmainmoving every ~28 minutes almost every run finds its PR behind. The 5–12 bot-pushed branches seen in the 20 minutes after6f8c51d7are the scheduler runs that happened to complete in that window.Consequence on merge governance
Reconstructing the check state at
merged_atfor every PR merged into.githubsince 2026-09-04 (77 PRs, RESTcommits/{sha}/check-runs, only runs completed beforemerged_atcounted,skippedcounted as satisfied):Depth ≥ 2 jobs (
opencode-review,coverage-evidence,trivy-fs,scorecard,osv-scan,dependency-review, bothCodeQL compatibility analysis (...)) had no check-run at all before merge on 75–77 heads: their upstream gate job never got a runner. Every merge since 09-04 was therefore an admin bypass with the required review evidence absent.merged_byis populated on these, so it is not a bypass signal; the check state atmerged_atis.Where
.github/workflows/pr-review-merge-scheduler.yml:139—UPDATE_BRANCHESistrueforpull_request_targetruns, so every PR's own event run performs the update.scripts/ci/pr_review_merge_scheduler_core.py:4774-4785—if behind_by and trigger_reviews:→update_branch; there is no check for required check-runs still queued/in progress on the current head.CLAUDE.mdstates the scheduler "updates a PR branch only when the latest review is approved, no current-head check has failed, and GitHub reports the PR as behind". The pre-review path above is a second, undocumented update path.Proposed fix (smallest root-cause change)
pr_review_merge_scheduler_core.py, before the pre-reviewupdate_branch, decidewaitwhen the current head still has required check-runs inqueued/in_progress— do not discard evidence in flight. One helper, one guard, one contract test.UPDATE_BRANCHEStofalseforpull_request_targetruns and leave the refresh to theschedule/pushsweeps, which are rate-limited byBRANCH_UPDATE_LIMIT.CLAUDE.mdsentence to describe both update paths.Related: #1931 (queue delay invalidates the dispatch side via
base_shadrift — same family, other half), #1929/#1927 (dispatch actor), #1925 (CodeQL dispatch never succeeds).Instrument notes for whoever re-measures: the pulls list endpoint omits
merged_by; a check-run'sstarted_atis null while queued; theclosedevent after merge creates fresh check-runs on the same head, so "latest check-run per name" reads post-merge state unless filtered bymerged_at.