Repository navigation
CodeQL/OpenCode dispatch: single-runner groups held idle by GHAS identity polling (763/165 queued, ~5 days to drain) #2527
Description
Activity
2026-10-01 live queue evidence
The central CodeQL queue remains causal for current exact-head acceptance delays.
- The newest 30 organization workflow runs span
2026-10-01T07:52:05Z–08:15:01Z. - Seven are
codeql-scan-dispatch.ymlruns: three are still queued and four newer same-PR predecessors were cancelled by concurrency after head changes. - Same-repository canary .github#2547 dispatch 36832871913 has been queued since
07:52:25Z. - .github#2548 exact head
8f6a870bca516d34a737bc8bd4abf5d3573573aesuccessfully submitted its two-language dispatch from coordinator job110266646185at07:31:12Z, but still has no authenticatedcodeql-dispatch/*terminal status. Its required CodeQL run 36830753593 therefore remains fail-closed rather than being manually rerun. - Current protected handler concurrency is already keyed by target repository + PR with
cancel-in-progress: true; unrelated PRs do not cancel this canary. This observation is consistent with the documented single-runner queue saturation, not evidence of terminal scan completion.
No policy relaxation, state cycling, manual rerun, or synthetic GREEN was applied.
- The newest 30 organization workflow runs span
- added
on Oct 1, 2026 Current OriginWeave consumer evidence (2026-10-02 04:10 KST), not a rerun or scan verdict:
- PR337 exact head
23e4ca5c78b5930064b91ff28383c7df89843b44, protected base87c4daa1830bac5a5228b6036752ad5633232085: CodeQL dispatch36882212724/ admission job110436543597; OpenCode dispatch36882186317/ admission job110436459313. - PR338 exact head
4f8e2da905d9a0e66b8fab49b38e35ee36072ead, same protected base: CodeQL dispatch36888041684/ admission job110456276323; OpenCode dispatch36887947722/ admission job110455961541. - All four remain queued at first admission,
runner_id=0, trusted central source37b10243cec3d160ecc9c1be75c71428b160a703. Consumer compatibility/review jobs correctly remain failed pending authenticated producer evidence. - Exact workflow source and runner-group membership prove CodeQL admission uses
CWL central CodeQLgroup4, whose sole runner iscwlab-s1-03(1063157, online/busy). OpenCode admission usesCWL central OpenCodegroup5, whose sole runner iscwlab-s1-04(1063158, online/busy). Idle general/control/GPU runners are not members of those groups; idle capacity elsewhere is not a permitted migration path. - Current actual occupying producers are CodeQL
36771361260forgyeot#67(actions job110359625791 in GHAS identity step; prior JS job110359625817 failed that step after passing analysis/SARIF gate) and OpenCode36664208068forclearfolio#646, currently advancing through review preparation after successful sidecar provisioning. This is saturation/queue ownership evidence, not proof the current jobs are stuck or safe to cancel. - Fresh central queued-run endpoint reports324 runs (only first100 read); no unsupported complete-inventory claim. Per-workflow bounded15:00–20:00UTC query returns51 CodeQL /48 OpenCode runs and contains the exact four current consumers above.
- Both OriginWeave native coverage artifacts downloaded and SHA256 matched to API digests; raw JSON verifier passes functions521/521, lines4420/4420, regions5349/5349, branches654/654. This does not substitute for missing central review/security evidence.
No current-head job cancellation, duplicate dispatch, arbitrary runner reassignment, credential change, workflow edit, gate weakening, synthetic success or merge performed. Preserve the valid live producers and address canonical queue throughput/identity polling at the owning central boundary, with separate exact-head proof after any accepted repair.
- PR337 exact head
Material update to the existing single-runner admission evidence, 2026-10-02 06:57 KST. This corrects the earlier statement that both PR337/338 CodeQL runs were still at their first admission job; it does not claim scan completion or queue recovery.
Protected OriginWeave main remains
87c4daa1830bac5a5228b6036752ad5633232085; exact PR337 head23e4ca5c78b5930064b91ff28383c7df89843b44, PR338 head4f8e2da905d9a0e66b8fab49b38e35ee36072ead. Central source remains37b10243cec3d160ecc9c1be75c71428b160a703.- PR337 CodeQL producer
36882212724: admission110436543597actually ran oncwlab-s1-03, completed SUCCESS at2026-10-01T21:15:45Z; recovered raw log validates exact live head/base andcodeql-scan-v2, required run36881970710, source merged22e3b026aa1c2c0cb8310aa2bb848f3c3bbd138. The actual scan jobs are now110588227538actions,110588227604javascript-typescript,110588227728python, all QUEUED. - PR338 CodeQL producer
36888041684: admission110456276323actually ran on the same named runner, completed SUCCESS at2026-10-01T21:33:00Z; raw log validates exact current head/base, required run36887861097, source merge609d3c2428ec4c2e20ffd118ce9161c77518936a. Scan jobs110594707367actions,110594707342javascript-typescript,110594707409python are QUEUED. - PR337 OpenCode
36882186317/ admission110436459313and PR338 OpenCode36887947722/ admission110455961541remain QUEUED at metadata validation. PR330 bounded canonical recovery producer36923574335/110575279136also remains QUEUED. - New PR340 exact head
ad50387bebad48bec1b6b1267cbe55f961d55924has canonical CodeQL36929151093/110593831691and OpenCode36928381456/110591257764, both admission QUEUED. CodeQL title binds head/base/required run36926739858/source merge2e8a949917f23b6047abe38cac73e1662947a7fb; fetching the source merge confirms exact base/head parents. These are real producers, not fabricated or duplicate dispatches.
A live runner-group read confirms group4
CWL central CodeQLand group5CWL central OpenCoderemain restricted to named canonical workflows; each has one online/busy runner. This supports admission/re-entry queue wait, but no current occupying job was re-proven in this update. Do not assume the older cited occupying job is still active or safe to cancel. Control/general/GPU capacity is not a permitted reassignment path without the owner preserving boundaries.All seven exact producer check suites were read in one GraphQL snapshot, with no check-run pagination remaining. Consumer PR330/337/338/339/340 all remain BLOCKED and have no unresolved non-outdated review thread; the PR338 Noema approval does not replace its missing canonical scans/OpenCode receipt. No authenticated current-head terminal scan/review evidence has been verified by this update.
No current-head producer cancellation, duplicate redispatch, consumer manual rerun, arbitrary runner reassignment, credential change, workflow edit, gate weakening, synthetic GREEN or merge. Existing central owner should continue throughput/identity-bound recovery from the actual remaining scan and admission jobs, preserving exact-head acceptance.
- PR337 CodeQL producer
Exact-head consumer specimen — EmbedRelay #4
Fresh consumer evidence at
ContextualWisdomLab/EmbedRelay#4@4ccfe7b921e9e8eeae6a6c60da49e8d42b8ae0b1confirms the central queue/terminal-receipt boundary remains active.- required consumer CodeQL run: 36946858996
- detected matrix:
actions,python - compatibility jobs:
110650942137(actions) and110650942123(python) - coordinator job:
110651090581 - coordinator outcome: success; it obtained OIDC and the repository-scoped app token, then accepted the
codeql-scan-v2repository dispatch - both language jobs intentionally failed closed with
verdict=pendingand the exact log: “CodeQL scan dispatched. The dispatch workflow will rerun this exact failed CodeQL job after publishing its terminal verdict.” - target SHA currently has no authenticated
codeql-dispatch/*status receipt, so re-running the failed jobs now would deterministically fail the attempt>1 guard - sibling exact-head SAST 36946858995 and Security Scan 36946858964 are GREEN
This is not a consumer-code finding and no gate bypass or dummy wake commit is appropriate. Preserve the unchanged exact head and let the canonical owner publish the terminal per-language receipt and perform the bounded same-head rerun.
OriginWeave#337 material queue-stage transition, observed 2026-10-02 10:38–10:44 KST. This updates the prior initial-admission diagnosis; it is not a retry request or a terminal review receipt.
Exact target head
23e4ca5c78b5930064b91ff28383c7df89843b44, base87c4daa1830bac5a5228b6036752ad5633232085, producer OpenCode run36882186317, immutable central source37b10243cec3d160ecc9c1be75c71428b160a703.- Metadata job
110436459313actually ran oncwlab-s1-04/cwl central opencodeat2026-10-02T01:34:50Zand completed SUCCESS. Its raw log binds supplied/live target head/base and publishes source artifact11204581108at01:35:04Z. - Downloaded
opencode-coverage-sourceZIP SHA256b67ace6ec9f4dd52f341f1e9137bcd5b5bf35ec88f86074e2d99288632c8c1d9, matching the API digest. The inner tar SHA256 is4a70050ec3b628248bf72fae0eb50e25ce98900e1c448ff90be4b0ec52eaf878. All214 tracked source files match local immutable PR337 head bytes. No archive Git/config/content was executed; digest/source equality does not establish coverage or model review. - New coverage job
110665637765, created01:35:08Z, remains QUEUED with empty steps, runner_id0, no assigned runner. Thus PR337 OpenCode is no longer waiting for initial metadata admission; it is waiting for downstream coverage admission. No terminal OpenCode verdict or GitHub approval has been produced. - PR338 producer
36887947722still has metadata110455961541QUEUED. Its same-head Noema APP APPROVED is retained but does not clear OpenCode/CodeQL. - PR337 CodeQL admission
110436543597and PR338 admission110456276323remain SUCCESS; each has three unassigned queued scans. PR337 scan IDs:110588227538,110588227604,110588227728. PR338:110594707342,110594707367,110594707409. - Groups4/5 remain singleton online/busy
cwlab-s1-03/cwlab-s1-04. A prior status-filtered list's gyeot#68 IN_PROGRESS summary did not prove current occupancy: exact suite shows coverage completed FAILURE and downstream review QUEUED. Current occupant identity remains unknown. No unrelated job cancellation, group reassignment, ACL/workflow/credential mutation or duplicate dispatch was attempted.
Next recovery evidence must be actual normal assignment and execution of the queued stage, then exact-target terminal publication and normal protected gates. Successful admission or source-archive verification is not review, security acceptance, merge or release. Original dirty46 and baseline11 worktree records remain preserved; protected main and target heads are unchanged.
- Metadata job
PR338 material OpenCode stage transition, observed2026-10-02 10:49–10:51KST. This supersedes its earlier initial-admission snapshot, not its current-head approval or required gates.
Target
ContextualWisdomLab/OriginWeave#338@4f8e2da905d9a0e66b8fab49b38e35ee36072ead, protected base87c4daa1830bac5a5228b6036752ad5633232085, producer36887947722, central execution source37b10243cec3d160ecc9c1be75c71428b160a703.- Metadata
110455961541started2026-10-02T01:39:50Zoncwlab-s1-04/cwl central opencode, completed SUCCESS at01:40:10Z. Raw log verifies supplied/live metadata and finalized source artifact11204522759at01:40:06Z. - New downstream coverage
110666848123, created01:40:10Z, is QUEUED with runner_id0, no assigned runner and empty steps. Both PR337 and PR338 OpenCode now await coverage admission, not initial metadata admission. PR337 coverage is110665637765. - PR338 ZIP SHA256
41c51c74de9863592e65e66683085fbef8d2bb2920d9d3a830dee23bf6f2bf65matches API digest. Inner tar SHA256b8669fe7269519c8b521fa2d2409ddb81ee3ae6a362cc2e6aac3a4f04a740656. All215 tracked file bytes match immutable target4f8e through trusted local Git reads. Archive Git/config/content was not executed. - PR337/338 CodeQL six scans remain QUEUED; metadata/admission success and archive integrity are not terminal scan/model verdicts. PR338 same-head APP APPROVED is preserved; no early base update invalidates it.
This consumer performed no rerun, cancellation, workflow/runner/ACL/credential mutation, synthesized verdict or protection bypass. Material recovery requires actual normal coverage assignment/execution and authenticated exact-target terminal results, then current rules and normal protected merge. Source equality alone is not CI acceptance, counted approval, merge or release.
- Metadata
Korean writing skills — exact-head admission specimen (2026-10-02 KST)
Target
ContextualWisdomLab/korean-writing-skills#4@67b74f359d0ad48226933493ed8307ada9b1f417, baseca94600ed058c0e7448ac69a537b8453241d139b.- Required CodeQL run
36929008593, Python compatibility job110594356535: the actual log recordsDISPATCH_OUTCOME: success,VERDICT_STATE: pending; coordinator110595475033submitted the bound Python matrix successfully. - Existing canonical producer
36930131832, trusted source37b10243cec3d160ecc9c1be75c71428b160a703, admission job110597072787: still QUEUED withrunner_id=0andsteps=[]in the fresh read. No scan/SARIF/publication failure is established. - This current tree contains
tests/skill_structure_test.py; replay of fix(codeql): prove content-only heads before dispatch #2508's actual no-supported-source predicate returns false. The old docs-only repair is not a direct remedy for this Python head.
Please recover normal admission/execution of the existing producer through the canonical queue owner, preserve exact-head terminal receipt and settlement, and identify the operator responsible for the restricted runner group. This is not a request for duplicate dispatch, arbitrary runner reassignment, predecessor-verdict transfer, or manual compatibility-job rerun. The consumer has a separately reproduced base-to-head whitespace repair in progress; if its head changes, this producer must retire normally and its evidence must not transfer.
Separate consumer blocker: Strix job
110593541220has a hosted-runner communication-loss annotation; no report artifact was uploaded. This is not proof of OOM or a source vulnerability. The latest Noema success likewise produced no approval because verdict preparation observed Draft. No merge, deployment or recovery is claimed.- Required CodeQL run
Material OriginWeave PR337 CodeQL scan/publication transition and OpenCode runner availability update, observed October 2, 2026, 17:03–17:15 KST. This updates the prior queued-scan diagnosis; it is not a retry request or merge acceptance.
Exact target PR337 head
23e4ca5c78b5930064b91ff28383c7df89843b44, protected base87c4daa1830bac5a5228b6036752ad5633232085, existing producer36882212724, required consumer36881970710, immutable central source37b10243cec3d160ecc9c1be75c71428b160a703, source merged22e3b026aa1c2c0cb8310aa2bb848f3c3bbd138.- All three scan jobs executed on
cwlab-s1-03and completed SUCCESS: actions110588227538, javascript-typescript110588227604, python110588227728. Actual job steps confirm Medium+ SARIF gate, GHAS base/head configuration identity, SARIF preservation and terminal status publication all succeeded. - Downloaded SARIF artifacts
11215625991,11216030076, and11215274918match their API ZIP digests. Safe data extraction and the unchanged event-pinned SARIF gate independently returnedfiles=1 results=0 medium_plus=0for each. This does not replace the GHAS identity step or the required consumer settlement. - Authenticated
cwl-noema-review[bot]status receipts55432930293,55432854343,55433042470are SUCCESS under the exact base-qualified language contexts. Theircwl1descriptions bind exact head, required run and source merge, and their target points to producer36882212724. - The remaining transition is settlement job
110757728453: QUEUED, runner_id0, steps[]. The actual pinned workflow assigns it to restricted group4CWL central CodeQL. No settlement code, credential exchange, validation or rerun POST has executed. Consumer PR337 is still BLOCKED/REVIEW_REQUIRED; successful scans/statuses alone are not required-workflow green or merge. - Group4 has one online/busy runner
1063157/cwlab-s1-03. Group5's one runner1063158/cwlab-s1-04is now offline, busy=false, rather than the earlier online/busy snapshot. PR337 and PR338 OpenCode downstream coverage jobs110665637765and110666848123remain unassigned QUEUED. This proves runner unavailability, not its host/root cause. The current occupying CodeQL job identity is not established here. - PR338's three CodeQL scans remain QUEUED in this snapshot; its exact-head Noema approval is preserved and does not clear the missing checks.
Please use the existing restricted-runner/central-queue operator to recover the offline OpenCode runner and normal admission of the existing CodeQL settlement. Identify that operator route if not already assigned. Do not duplicate the producers, manually re-run compatibility jobs, arbitrarily reassign runner groups, cancel an unknown occupying job, change credentials or weaken policy.
A local artifact download transiently failed with a TLS handshake timeout; the later bounded read succeeded and digest checks passed. This local transport problem is not the settlement or scanner cause. Evidence is retained in
fleet/originweave-pr337-scan-transition-evidence-20261002/manifest.json, SHA256686208b1c00f5deb45bf181516f9accb3df4a99337450205dab446c209da1653.No new dispatch, producer cancellation, manual rerun, host restart, workflow/ACL/credential mutation, fabricated approval or merge. Original dirty46 and baseline11 worktree records remain preserved; independent consumer regression work continues without treating this item-local dependency as a global stop.
- All three scan jobs executed on
Measurement (2026-09-29 23:14 UTC / 2026-09-30 08:14 KST)
Queue composition, checked with GraphQL against every target PR: CodeQL 539 current heads, 217 superseded heads, 2 closed PRs. OpenCode 121 current, 43 superseded. There is one run per PR, so this isn't duplicate dispatch.
Root cause
scanrunscodeql_ghas_configuration_identity.py, which polls GHAS analyses up toGHAS_CONFIG_IDENTITY_ATTEMPTS=30×20s= 10 min. It waits for Default setup to finish on the head while holding the group's single runner at ~0% CPU. Observed live: s1-03 load 0.00, Worker ~10 min in "Verify GHAS base/head CodeQL configuration identity" for xtrmLLMBatchPython#339. At roughly 10 min per job, 539 current heads take about 90 runner-hours, close to 4 days on one runner.scanonly retires a superseded head aftervalidate-dispatchand the scan job have both been scheduled on the single runner.Done now (operational, no policy change)
Queued dispatch runs whose target PR is closed or whose head has moved were cancelled after re-checking the live PR state immediately before each cancel. The workflow would retire them itself ("retiring this superseded run"), so no verdict is lost. New heads get their own dispatch. Log counts are below.
Proposals (need the CI owner's decision; I have not changed any of them)
cancel-in-progresskeyed on repo#PR), so a new head replaces the older queued one.Related: #2356 (queue-health evidence), #2341 (sidecar hang), #2506/#2511 (Strix false fail-closed).
🤖 Generated with Claude Code