Skip to content

OpenCode model-unavailable fallback reviews satisfied the required review check with no model verdict #2335

Open

Description

@seonghobae

Defect

When the OpenCode model pool is exhausted (model=none), the model-unavailable evidence fallback posts a deterministic CHANGES_REQUESTED "blocker" review ("OpenCode could not approve from deterministic current-head evidence … model-unavailable evidence fallback …"). Two gates counted that fallback review as a model verdict:

  1. The required opencode-review bootstrap (opencode-review.yml, "Fail closed without a current-head OpenCode verdict") accepted any CHANGES_REQUESTED from opencode-agent. The required check went green with no model verdict.
  2. scripts/ci/opencode_review_receipt_gate.py is_formal_receipt filtered fallback markers only for APPROVED. A fallback CHANGES_REQUESTED counted as a formal receipt and suppressed the scheduler wake that would retry a real review.

Impact: this is an observability hole, not an approval hole. Merges stayed blocked by the CHANGES_REQUESTED review decision and the independent-approval requirement. But from the 2026-08-27 gateway switch until the /v1 fix (#2333), OpenCode produced zero model verdicts while required checks showed green. Example: run 34931908846 had its model pool exhausted, model=none, and still ended in success via the fallback.

Owner

dot-github lead (orchestration run_b22de9a1c59d).

Acceptance

  • A review carrying any fallback marker (either state) never satisfies the required opencode-review check. The step fails with an explicit MODEL_OUTPUT_UNAVAILABLE error.
  • A fallback review (either state) is not a formal receipt, so the scheduler keeps waking a real review for that head.
  • A later model-backed APPROVED/CHANGES_REQUESTED review on the same head still passes both gates.
  • Regression tests fail on main and pass with the fix.
  • After fix(review): point OpenCode at the gateway's /v1 routes and split failure classes #2333 lands, at least one hosted run shows a model-backed verdict (not fallback) satisfying the required check.

Activity

  1. seonghobae commented on Sep 21, 2026

    @seonghobae
    ContributorAuthor

    Follow-up, not covered by #2336. #2336 makes a single model-unavailable outcome visible as a red MODEL_OUTPUT_UNAVAILABLE check, but only a count across runs distinguishes transient provider jitter from a multi-week outage like 2026-08-27 → #2333. Proposed acceptance for a separate PR:

    • Count consecutive fallback-only outcomes per repository or org.
    • Raise a distinct alert (issue or notice) once a threshold is crossed.
    • Cover it with a test that fails when the count is not reset by a model-backed verdict.

    Evidence preserved in #2334 (closed as duplicate of this issue): run 34931908846, job opencode-review, head 91be644. It ran attempt 1/1 with exit 1, OPENCODE_MODEL_POOL_OUTCOME exhausted and model=none, yet the job concluded success, while reporting provider secrets 5/5 and gateway preflight ready. The fallback publishes at opencode-review-dispatch.yml publish_blockers_after_model_unavailable() whenever COVERAGE_EVIDENCE_RESULT is success.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions