ci: supersede in-flight runs per ref — fixes 15-minute queue starvation - #470
Merged
Conversation
Diagnosis (2026-08-06): with five PRs open and pushed in quick succession, jobs were created but NEVER ASSIGNED A RUNNER (runner_name empty), sat queued, and were killed at exactly 15m01s having executed ZERO steps. They reported as 'failure' in gh pr checks — indistinguishable from real test failures. Five PRs read red while the only PR containing source changes was green, which is the dangerous part: a genuine failure could hide among the fake ones. Not slowness. A healthy unit-tests job is ~12 min of real work (3m45s install + 7m43s tests) and passes; lint and aarch64 succeeded in the very runs where unit-tests starved. The workflow had NO concurrency group, so every push left its predecessor's run alive and competing — N pushes to one branch demanded N times the slots, across five branches at once. cancel-in-progress is deliberately FALSE on main: a superseded main run must still complete, since main's CI result is what the release gate and post-merge verification read. Only PR-branch runs are cancellable. Verified: re-running the starved jobs with no code change turned them green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Diagnosis
Jobs across five open PRs reported
failuretoday. None of them ran.runner_nameis empty — no machine was ever assigned. The jobs starved in the queue and were killed at a 15-minute limit, then surfaced asfailureingh pr checks, indistinguishable from real test failures. Five PRs read red while the only PR containing source changes was green. That's the dangerous part: a genuine failure could hide among the fake ones.Not slowness. A healthy
unit-testsjob is ~12 min of real work (3m45s dependency install + 7m43s test run) and passes;lintandaarch64 resolvesucceeded in the very same runs whereunit-testsstarved. Re-running the starved jobs with no code change turned them green.Cause: the workflow had no
concurrencygroup, so every push left its predecessor's run alive and competing for runners — N pushes to one branch demanded N times the slots, across five branches simultaneously.Fix
cancel-in-progressis deliberately false on main — a superseded main run must still complete, because main's CI result is what the release gate and post-merge verification read. Only PR-branch runs are cancellable.Not changed (deliberately)
llama-cpp-python) is a plausible trim, but removing optional deps changes which tests skip vs run — a coverage change that shouldn't ride a CI-infra fix.aarch64 resolveholds a slot 14–45 min and is a secondary contributor to demand; worth path-gating separately if starvation recurs.🤖 Generated with Claude Code