[NielsRogge]

Why

On 2026-11-03 at 04:31 UTC the scheduled-curation breaker paused dispatch with "Three consecutive scheduled apply or verification stages failed". It had counted:

• Remote runner failed (JSONDecodeError). (resolve-model-paper at 03:58, enrich 131752 at 04:31)
• Remote runner failed (HTTPError). (enrich 131754 at 04:30; its retry completed at 04:39)
• [awaiting_markdown] … (enrich 131750, already uncounted)

nginx logged only 200s for these jobs. Dispatch stayed paused until a manual --enable-dispatch at about 08:40.

Where these errors come from

Remote runner failed (<ExceptionName>). is produced only by the outer except in jobs/run_paper_curation.py. Inside that block, HTTPError and JSONDecodeError can only come from _send, the runner's own callbacks to our API (bootstrap, context, complete). The preflight runs as a subprocess, so its failures surface as CalledProcessError. Applies run inside Codex through jobs/curation_api.py, which turns an HTTP rejection into a RuntimeError for the agent. A rejected apply therefore shows up as an apply/verification error or Codex exited…, never as one of these two names.

For a job whose complete later succeeded, an HTTPError or non-JSON body on these callbacks is almost always the edge (a Cloudflare 5xx or an HTML/empty body), and that matches the clean nginx log. The server-side 4xx cases are a 404 for a disabled feature or a missing paper, and a 409 for a bootstrap bundle that doesn't match the release. A missing paper is a per-paper failure that shouldn't pause the queue. A disabled feature or release mismatch fails every job, and the existing runner_transport_failures condition (more than 50% of the last 25) still pauses for that.

Telling 5xx apart from 4xx at the runner would mean adding the status code to the error string. That edits jobs/run_paper_curation.py and needs a new pinned jobs image, so it's out of scope here.

Change

• Add HTTPError and JSONDecodeError to _TRANSPORT_FAILURE.
• Regression test that replays the four 2026-11-03 rows and checks that dispatch doesn't pause. It fails without the fix with apply_or_verification_failures.
• The "other runner exceptions still count" test now uses CalledProcessError.
• Update docs/operations.md.

Validation

• uv run pytest backend/tests/curation -q: 755 passed, 14 skipped
• ruff check/format clean

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes scheduled-dispatch auto-pause heuristics; misclassification could hide real apply failures or fail to pause on systemic outages, though bulk transport pauses still use the >50% callback-failure rule.

Overview
Fixes scheduled curation dispatch pausing on transient runner→API callback failures that are not apply/verification problems.

HTTPError and JSONDecodeError in Remote runner failed (...) are now classified like other transport blips in _TRANSPORT_FAILURE, so they no longer increment the three-strike consecutive_apply_failures counter or the 25-stage terminal failure rate. Comments clarify these exceptions come from the runner’s bootstrap/context/complete callbacks (often edge 5xx or non-JSON bodies), not from Codex apply rejections.

Adds a regression test replaying the 2026-11-03 production sequence (JSON/HTTP errors plus [awaiting_markdown]) and asserts dispatch stays unpaused; the “non-transport runner errors still count” test now uses CalledProcessError. docs/operations.md documents the expanded transport-failure list.

^{Reviewed by Cursor Bugbot for commit 94428e1. Bugbot is set up for automated code reviews on this repo. Configure here.}