Hand the live loop off explicitly instead of hoping the cron does - #56
Open
dgoodenough wants to merge 2 commits into
Open
dgoodenough wants to merge 2 commits into
dgoodenough wants to merge 2 commits into
Conversation
The loop holds a runner ~5.5h and exits under GitHub's 6h per-job ceiling. Continuation depended on the `*/15` cron having left a queued successor in the shared concurrency group during that window — which the workflow header described as something the coarse cron did "reliably". On 2026-08-27, round 2 of Worlds, it did not. Run #1019 yielded at 20:01Z announcing "the queued successor takes over" and nothing took over. GitHub delivered zero scheduled fires for this workflow between 14:30Z and 23:17Z, when it was restarted by hand — including the 3h11m after the yield when the concurrency slot was completely free, so a congested queue does not explain it. `refresh.yml`'s `0 11 * * *` cron fired that day at 20:57Z, ~10 hours late, which is the corroborating evidence that the scheduler rather than the queue was at fault; that late run is also the only reason the site was 2h15m stale instead of 3h16m. An earlier gap the same day went unnoticed: #1018 ended 08:52Z, #1019 started 14:30Z, 5h38m uncovered. Nothing in the pipeline was broken — the loop, the gate, the publish path and the invariants were all healthy, which is exactly why no run went red and nobody was notified. A liveness failure that produces no failing run produces no alert either. So the yielding run now dispatches its own successor and confirms it exists. `workflow_dispatch` (with `repository_dispatch`) is the documented exception to the rule that GITHUB_TOKEN-triggered events start no workflow run, so this needs no PAT, only `actions: write`. - .github/dispatch-successor.sh dispatches and then POLLS for the run, re-dispatching once and failing non-zero if it cannot confirm one. The verification is the substance: a 204 is not a run, as the 2026-08-26 Actions incident showed, and a hand-off that reports success with nothing enqueued reproduces the outage exactly. - The dispatch happens before the current run exits, so the successor lands in the concurrency group as the pending run and starts as the slot frees. - A failed hand-off fails the run, rather than exiting quietly with nothing lined up — the silence is what made this expensive. - A crash-out gets one automatic retry, marked after_failure=true; a run carrying that flag does not chain again, so transient breaks self-heal while persistent ones cannot spin runners. Still on the cron: starting the loop when an event goes live. That window is more forgiving (the loop exits in seconds when nothing is live, so a late start costs the opening minutes of a round, not hours mid-round), and HARDENING item 10 records the fix if it ever bites. Tests stub `gh` on PATH and drive the SUCCESSOR_* knobs to zero delay. Verified by mutation: removing the poll-for-the-run step fails 4 of the 7. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018qBS4DHn53TNPoQZXRsiPq
Two PDGA rows at Worlds carried impossible cumulative scores on 2026-08-29: MPO Sander Bahnerth cur=+981 at 16:10Z, and two hours later FPO Samantha Zaborowski cur=+849, both far outside the per-round bounds. The publish-gate checks did exactly their job — published the data, then failed the run so the owner was notified. What that costs had not been noticed. `exit 1` ends the run, and the run IS the live loop, so each bad row also stopped the site updating. Worse, the de-dupe meant to keep a persistent violation from re-alerting could never fire: its marker lived in data/cache/, which is gitignored and carried between runs only by actions/cache, whose save step is skipped when the job ends non-zero. The alerting run is always the failing run, so the marker written by the run that alerts was precisely the one guaranteed not to survive. Both runs show "Post Restore results cache: skipped". Every restart re-alerted and died again; round 4 ran on hourly updates (16:10, 17:26, 18:38Z) instead of six-minute ones, and the 17:26 one was the daily cron rather than the loop. The hand-off added in the previous commit did not cover this. It handles the budget yield and the three-crash path via fail_step; the invariant block has its own bare `exit 1`, a third exit with no successor. Found in production, not in review. - .github/invariant-alert.sh makes the alert decision and owns the marker. Exit 3 = new violations, 0 = clean or already alerted, 1 = broken, so the caller can order publish/commit/alert around it. - The marker moves to data/invariant_alerted.txt, tracked alongside the other cross-run state the pipeline reads back, and the decision now runs BEFORE the state commit so it is committed rather than lost with the run. - The invariant exit hands off to a successor, which reads that marker, sees the same violation set and stays quiet — the loop keeps publishing while the run still goes red. - Alerting is per episode: a clean run clears the marker, so a row that breaks, is fixed, and breaks again is reported both times. - The hand-off is withheld if the state push failed. That is the one way this could chain: a successor starting from the stale marker would re-alert. Not addressed: refresh.yml's own invariant step has no de-dupe and will still go red once a day while a violation persists. It is a daily job rather than the liveness path, so that is the intended signal. 8 tests drive the script through the episode semantics. Verified by mutation: disabling the de-dupe fails test_same_violation_stays_quiet, and not clearing on a clean run fails test_recurrence_after_a_clean_run_alerts_again. The unusable-path case uses a file-as-parent rather than chmod, which root ignores. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018qBS4DHn53TNPoQZXRsiPq
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The live loop holds a runner ~5.5h and exits under GitHub's 6h per-job ceiling. Continuation depended on the
*/15cron having left a queued successor in the shared concurrency group — which the workflow header described as something the coarse cron did "reliably".What happened
On 2026-08-27, round 2 of Worlds, it had not. Run #1019 yielded at 20:01Z announcing "the queued successor takes over" and nothing took over.
GitHub delivered zero scheduled fires for this workflow between 14:30Z and 23:17Z, when it was restarted by hand — including the 3h11m after the yield, when the concurrency slot was completely free. A congested queue does not explain that. The corroborating evidence that the scheduler rather than the queue was at fault:
refresh.yml's0 11 * * *cron fired that day at 20:57Z, ~10 hours late. That late run is also the only reason the site was 2h15m stale instead of 3h16m — the once-a-day job happened to land in the hole.An earlier gap the same day went unnoticed entirely: #1018 ended 08:52Z, #1019 started 14:30Z. 5h38m uncovered.
Nothing in the pipeline was broken. The loop, the change-check gate, the publish path and the invariants were all healthy — which is exactly why no run went red and nobody was notified. A liveness failure that produces no failing run produces no alert either.
The fix
The yielding run now dispatches its own successor and confirms it exists.
workflow_dispatch(withrepository_dispatch) is the documented exception to the rule that GITHUB_TOKEN-triggered events start no workflow run, so this needs no PAT — onlyactions: write..github/dispatch-successor.sh(new) dispatches and then polls for the run, re-dispatching once and exiting non-zero if it cannot confirm one. The verification is the substance, not politeness: a 204 is not a run, as the 2026-08-26 Actions incident showed, and a hand-off that reports success with nothing enqueued reproduces this outage exactly.In
live-refresh.yml:after_failure=true; a run carrying that flag does not chain again. Transient breaks self-heal in a minute; persistent ones cannot spin runners.after_failureis astringrather than abooleanon purpose — it is dispatched viagh -f, which sends strings, and the loop compares it as one. No coercion between them for no benefit.Testing
7 new tests stub
ghon PATH and drive polling to zero delay through theSUCCESSOR_*knobs the script exposes for that purpose. 150 pass locally.Checked that they are not vacuous: removing the poll-for-the-run step fails 4 of the 7. The empty-
--repocase is a test rather than a manual check, because an empty array expanded underset -uaborts on bash before 4.4.What this does not fix
Starting the loop when an event goes live still rides on the cron. That window is more forgiving — the loop exits in seconds when nothing is live, so a late start costs the opening minutes of a round rather than hours mid-round — but it is the same dependency, and a multi-hour delivery gap across a Sunday-morning tee time would show. HARDENING item 10 records the fix if it bites: keep looping while a points event starts within the next several hours, instead of exiting on "nothing live right now". Not built here, because it trades a runner held through event-eve nights against a risk that has not materialised.
🤖 Generated with Claude Code
https://claude.ai/code/session_018qBS4DHn53TNPoQZXRsiPq
Generated by Claude Code