fix(scheduler): stop re-provisioning undispatchable zombie jobs#95
Merged
Conversation
A workflow run that GitHub keeps listing as "queued" but never actually dispatches to a runner — classically a run superseded by a newer commit on a workflow without concurrency:cancel-in-progress — would make ephemerd re-provision a full runner/VM for it on every seenTTL, forever. Observed live: 3 stale php-sdk macOS jobs (19h old, 8 commits behind HEAD, runner never assigned) each booted a fresh macOS VM every ~10 min indefinitely, burning ~4GB + boot for nothing and inflating the queue. Add a per-job provisioning-attempt counter. The seen dedup lets a given job past provisioning only ~once per seenTTL, so a live job (which runs to completion and then stops appearing) reaches it once, while a zombie reappears each cycle. After maxProvisionAttempts (~50 min of retries) the job is skipped with a one-time warning instead of re-provisioning. The counter is pruned in cleanSeen once the job stops appearing, so a later legitimate rerun of the same job id starts fresh. Tests: TestHandleQueued_ZombieSkip (skip fires past the cap, never claims) and TestCleanSeen_PrunesAttempts (counter reset on expiry). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This was referenced Jul 8, 2026
This was referenced Jul 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A workflow run GitHub keeps listing as queued but never dispatches to a runner — classically a run superseded by a newer commit on a workflow without
concurrency:cancel-in-progress— made ephemerd re-provision a full runner/VM for it every seenTTL, forever.Observed live: 3 stale php-sdk macOS jobs (19h old, 8 commits behind HEAD,
runner=(never assigned)) each booted a fresh macOS VM every ~10 min indefinitely — ~4GB + boot per cycle for nothing — and inflated the queue so php-sdk looked backed up. The current-HEAD build was green the whole time.Fix
A per-job provisioning-attempt counter. The
seendedup lets a given job past provisioning only ~once perseenTTL, so:maxProvisionAttempts(~50 min of retries) it's skipped with a one-time warning instead of re-provisioning.The counter is pruned in
cleanSeenonce the job stops appearing, so a later legitimate rerun of the same job id starts fresh.Tests
TestHandleQueued_ZombieSkip— skip fires past the cap, job is never claimedTestCleanSeen_PrunesAttempts— counter reset when the seen entry expiresNotes
Generalizes the earlier ghost/zombie churn (deleted-branch runs) into a single guard. Complements a php-sdk-side fix (add
concurrency:cancel-in-progressso superseded runs self-cancel). The 3 live zombies were also cancelled manually.🤖 Generated with Claude Code