Repository navigation
Org-wide OpenCode model-pool outage: all 3 provider families down since 2026-07-25 — human credential/budget action required #628
Description
Summary
Every opencode-review model-pool run across the org has ended OPENCODE_MODEL_POOL_OUTCOME: exhausted since ~2026-07-25T00:30Z (~34h). The last real model review completed 2026-07-24T23:42:52Z (run 30134503832, 43.5m). Later "success" runs (e.g. 30193838586 at 08:01Z, 30196130362 for #579) passed only via pre-existing same-head approvals — their pools were also exhausted. Approved-but-unreviewed PRs (e.g. #618 at head c4a9023) cannot merge until this is fixed.
Per-provider signature (human action needed)
| Provider family | Failure class | Evidence | Action |
|---|---|---|---|
| github-models (majority of pool) | authentication-or-permission (~5s fatal per candidate) |
runs 30153577309 (att. 21), 30195391225, 30196418659, 30196803994 (naruon#1081: 24× auth class) | STRIX_GITHUB_MODELS_TOKEN secret (PAT, models: read) is rejected — likely expired. Rotate/renew the org/repo secret. Fallback github.token is also failing, so check GitHub Models org entitlement/billing too. |
openai gpt-5.6-luna |
quota-or-budget |
same runs; persists >34h across a daily reset boundary | Hard budget cap reached — raise budget or wait for the billing-cycle reset. |
| openrouter (deepseek-v3.2, qwen3-coder) | credit-exhausted (HTTP 402) |
same runs; root cause documented in merged #621: runaway runs 30120972549 (~3h × 102 pool cycles) and 30122360773 (700+ attempts into 402) burned the entire org credit | Top up OpenRouter credits. Spend guards from #621 now bound future burn. |
Org-wide confirmation (2026-07-26 09:29–09:44Z batch, all identical signature)
kaefa#70, html4tree#178, pg-erd-cloud#596, clearfolio#161, bandscope#645, newsdom-api#401, naruon#1081, fast-mlsirm#259 — all failure, all provider classes as above.
What was already done autonomously
- PR Handle stacked PR retarget and merge-queue trigger gaps #618 adversarial review: no PR-caused defects; every other required check green at head
c4a9023; stale auto-merge (from 07-25) cleared via scheduler dispatch 30195149878. - Same-head OpenCode re-dispatched twice (runs 30195391225, 30196418659) per the "opencode-review failures are infrastructure → re-run only" rule; both exhausted.
- No timing/gate changes were made (fixed rule: review timing untouched; no required-check weakening).
Self-healing after fix
No code change needed. Scheduled scans (:00/:30 repo, :15/:45 org) re-dispatch stale OpenCode reviews automatically once providers work again; approved heads then auto-merge. For an immediate retry: gh api repos/ContextualWisdomLab/.github/dispatches -f event_type=merge-scheduler -F 'client_payload[pr_number]=618' -F 'client_payload[trigger_reviews]=true' -F 'client_payload[enable_auto_merge]=true' (note: omitting the boolean fields makes GitHub expressions coerce them to false — see gotcha below).
Scheduler payload gotcha (worth a follow-up docs/test PR)
pr-review-merge-scheduler.yml maps client_payload.trigger_reviews != false → GitHub expressions coerce absent (null) to false via numeric coercion, so null != false is false. A bare client_payload[pr_number]=N dispatch runs with TRIGGER_REVIEWS/ENABLE_AUTO_MERGE/UPDATE_BRANCHES: false (observed: run 30195265329 → decision block; run 30195382302 with explicit trigger_reviews=true → review_dispatch).
Activity
seonghobae commentedon Aug 8, 2026
RCA/feasibility refresh (2026-08-09 KST): this incident record is no longer an accurate active remediation plan. Its July-26 diagnosis assumed all three then-configured provider families were down and recommended rotating a GitHub Models credential, restoring OpenAI quota, or topping up OpenRouter. GitHub Models is now retired, so credential rotation cannot restore that provider. Fresh protected-main evidence is tracked in #624: central OpenCode run
31265950492reached the model pool on protectedmain6eb06cdd08c79a06f7b390069d4ffa49e2eb7dba; the NVIDIA NIM candidate executed (bounded timeout), while the pool still attempted retiredgithub-models/*entries. That proves the current causal defect is stale provider eligibility/routing plus supported-provider reliability, not the historical three-provider outage described here.Feasibility comparison:
No provider credential, check, review, branch protection, or source gate is changed by this closure. The scheduler payload coercion note remains historical evidence and may be replayed separately if still reproducible. Active provider retirement/removal, bounded fallbacks, and protected-main runtime acceptance remain in #624.