ops: OpenCode Review Dispatch serialized on one runner (CWL central OpenCode = s1-04) — 163 queued, 22/30 recent runs cancelled after ~3h wait

Symptom (2026-09-30 08:30 KST / 2026-09-29 23:30 UTC)

.github queued runs hit the API cap of 1,000. By type: CodeQL Scan Dispatch 650, OpenCode Review Dispatch 163, Required Noema Review 35, others static.
Queued OpenCode dispatches for the same PR at different heads: 0. Cancelling superseded runs would not reduce the queue.
Of the 30 most recently completed OpenCode Review Dispatch runs: 22 cancelled, 8 failed, 0 succeeded.
- The failures are all intended validate-pr-metadata rejections (head changed or PR closed). Example: run 36603201268.
- The cancellations were replaced by a newer head after waiting about 3 hours. Examples: 36620067496 (fast-mlsirm#2273), 36623143994, 36620314334.
Downstream effect: required-review PRs such as fast-mlsirm feat(review): add pinned Graphify code graph #2052, fix(opencode): emit redaction-safe provider failure envelope before approval settlement #2112, fix(runtime): gate Draft admission and sandbox evidence #2109 and [Quarantine runtime] Publish reusable positive-LSM command acceptance workflow #2083 have gone more than a day without an opencode-agent verdict.

Root cause

In opencode-review-dispatch.yml, all three jobs (validate-pr-metadata, coverage-evidence and opencode-review-target) use runs-on: group: CWL central OpenCode.
Runner group CWL central OpenCode (id 5) contains only cwlab-s1-04.
coverage-evidence has timeout-minutes: 300. A single dispatch can hold that one runner for up to 5 hours, so throughput is at most one PR at a time.
Meanwhile, the CWL central control group (id 6: s1-01, s1-05, s2-01, s1-07) has an idle runner, cwlab-s1-07 (busy=false). That group's selected_workflows does not include opencode-review-dispatch.yml, so it can't take the work.

Evidence commands:

gh api orgs/ContextualWisdomLab/actions/runner-groups/5/runners
.../runner-groups/6 selected_workflows
gh api orgs/ContextualWisdomLab/actions/runners
.github/workflows/opencode-review-dispatch.yml L56/L399/L2514 (runs-on), L402 (timeout-minutes: 300)

Options (operator decision: this changes a security-isolation boundary, so I have not changed it myself)

A. Add more runners to the CWL central OpenCode group, for example moving cwlab-s1-07 from group 6. This keeps isolation per group and widens throughput.
B. Add opencode-review-dispatch.yml@refs/heads/main to group 6's selected_workflows. The isolation trade-off needs review.
C. Short term: run scripts/ci/opencode_queue_priority.py (operator runbook) to put maintainer-labelled review-priority PRs first.

Side note

The defect in #2273 (coverage build context missing requirements-noema-document-ci-hashes.txt) appears fixed in the current main opencode-review-dispatch.yml L718, which now stages that file. Please check whether #2273 can be closed.

🤖 Generated with Claude Code

OpenCode queue priority runbook

Metrics: {"queued": 150, "deferred": 118, "stale": 32, "oldest_deferred_hours": 9.4, "priority_positions": {}}

Label review-priority (maintainer-applied) kept: 0. To cancel: 32.

Cancel list (TSV)

run_id	created_at	repository	pr	head_sha	reason
36579768241	2026-09-29T14:03:29+00:00	ContextualWisdomLab/fast-mlsirm	2256	da2c258062bcd77f47d135e542269261bdc36e90	stale
36581847335	2026-09-29T14:19:48+00:00	ContextualWisdomLab/bandscope	1273	afe97ff0e09200d197a45b24d61138a81fadb84e	stale
36582892638	2026-09-29T14:28:10+00:00	ContextualWisdomLab/newsdom-api	955	fc64882805b3c91033b258f8ef2864171a4c75b0	stale
36587389681	2026-09-29T15:04:28+00:00	ContextualWisdomLab/newsdom-api	956	bd8f42f0ba5afbaf33e9e54149e8f45a019c29d6	stale
36587390651	2026-09-29T15:04:28+00:00	ContextualWisdomLab/fast-mlsirm	2189	520b6e7d58b2644b8b37a4a962dada2ecb5cd15d	stale
36587880207	2026-09-29T15:08:17+00:00	ContextualWisdomLab/xtrmLLMBatchPython	354	2702eae22171820a3a6742c59b20d6839a87a490	stale
36589890435	2026-09-29T15:24:01+00:00	ContextualWisdomLab/pg-erd-cloud	1230	36f033ca7f43627584ce1d5c50b95c6efb57875e	stale
36592332476	2026-09-29T15:43:09+00:00	ContextualWisdomLab/mhtml-etl-gateway	122	dbc51fa942eeab10892a698a4734877b52fa41fe	stale
36594350796	2026-09-29T15:59:10+00:00	ContextualWisdomLab/mhtml-etl-gateway	123	4ef72af73fd635dc1838d4aca8a832e1e03f76ed	stale
36612407728	2026-09-29T18:29:23+00:00	ContextualWisdomLab/naruon	1825	5c617ea2e6210179f7d2300fa2039518209b8e5c	stale
36614799882	2026-09-29T18:49:23+00:00	ContextualWisdomLab/argos	705	02a888f6e083757634a0e82e236cf2134bc00aa3	stale
36619796475	2026-09-29T19:31:01+00:00	ContextualWisdomLab/seedream_evasepic	512	14a4660d3e9756031d526b000bfac9938b1bfa96	stale
36620331571	2026-09-29T19:35:33+00:00	ContextualWisdomLab/html4tree	471	c99167cdb9f7037af42730639695466006029371	stale
36620448487	2026-09-29T19:36:33+00:00	ContextualWisdomLab/html4tree	470	73c1dd42bbed2b79bf83b5990c5bfa08a79ffae9	stale
36622756057	2026-09-29T19:56:20+00:00	ContextualWisdomLab/scopeweave	764	2f9a8a400cba38e1196dc53943dfe0de44d82f7f	stale
36623195214	2026-09-29T20:00:03+00:00	ContextualWisdomLab/pg-erd-cloud	1235	b1632b082bbfdbfc799ca14b29cec1497417c1c4	stale
36625349770	2026-09-29T20:17:52+00:00	ContextualWisdomLab/naruon	1807	976eada27855fe207601f0db62672aa737c92e18	stale
36625404371	2026-09-29T20:18:21+00:00	ContextualWisdomLab/seedream_evasepic	304	7c9e865226a9009b2ae7b82748df399bc4c79488	stale
36625450929	2026-09-29T20:18:45+00:00	ContextualWisdomLab/.github	2504	6295bf81938387875d583a39c9d812fc17aa4725	stale
36625942209	2026-09-29T20:22:56+00:00	ContextualWisdomLab/pg-erd-cloud	1229	44fd3e8fb1aba5ff6b7927ea401f11d83babeb5d	stale
36626619020	2026-09-29T20:28:46+00:00	ContextualWisdomLab/seedream_evasepic	469	c7485c5421c62da13281cff78fb78aef184a46ef	stale
36628122580	2026-09-29T20:41:27+00:00	ContextualWisdomLab/appguardrail	1340	bb165a1f2101f2f4908d9b5fac80714d2bfa1bbf	stale
36628531219	2026-09-29T20:44:55+00:00	ContextualWisdomLab/scopeweave	762	9bbab47a6ac83e669553e10017b7bc56f3c63aad	stale
36630253984	2026-09-29T20:59:49+00:00	ContextualWisdomLab/.github	2521	6d5caedad8f22a24c8f1fb2a4a203bfbeb4cab8b	stale
36631568181	2026-09-29T21:11:14+00:00	ContextualWisdomLab/newsdom-api	947	3fffc063bbefcf96e9a993a094e35c6c40e3e38e	stale
36633766105	2026-09-29T21:30:48+00:00	ContextualWisdomLab/seedream_evasepic	519	5b8a86ecea8f09b9ce7fc436fb3c098d80b0fbfe	stale
36634314672	2026-09-29T21:35:44+00:00	ContextualWisdomLab/newsdom-api	967	f7a72cc7e8cea05b1db0870d1a9c1184bc1e0f90	stale
36635657570	2026-09-29T21:48:08+00:00	ContextualWisdomLab/pg-erd-cloud	1236	5832345b8593c9d7d270268cc013738e05cc9060	stale
36640432285	2026-09-29T22:35:07+00:00	ContextualWisdomLab/.github	2525	a8a6ab4b3b348536d802040cc8d6030c6d3046f6	stale
36644307752	2026-09-29T23:16:28+00:00	ContextualWisdomLab/newsdom-api	973	bbb9722a1fa802a9886d50bcc8e285d4e6541816	stale
36644690605	2026-09-29T23:20:40+00:00	ContextualWisdomLab/linux-cluster-ops	353	2fc3872d1194cad52fbbff123d418a3e9666421f	stale
36645305585	2026-09-29T23:27:31+00:00	ContextualWisdomLab/clearfolio	655	8b2b48294b21d059c075897c0bcfca09cc61f4b0	stale

Navigation Menu

Repository navigation

ops: OpenCode Review Dispatch serialized on one runner (CWL central OpenCode = s1-04) — 163 queued, 22/30 recent runs cancelled after ~3h wait #2526

Symptom (2026-09-30 08:30 KST / 2026-09-29 23:30 UTC)

Root cause

Options (operator decision: this changes a security-isolation boundary, so I have not changed it myself)

Side note

OpenCode queue priority runbook

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

ops: OpenCode Review Dispatch serialized on one runner (CWL central OpenCode = s1-04) — 163 queued, 22/30 recent runs cancelled after ~3h wait #2526

Description

Symptom (2026-09-30 08:30 KST / 2026-09-29 23:30 UTC)

Root cause

Options (operator decision: this changes a security-isolation boundary, so I have not changed it myself)

Side note

Activity

seonghobae commented on Sep 29, 2026

OpenCode queue priority runbook

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions