Skip to content

ci: supersede in-flight runs per ref — fixes 15-minute queue starvation - #470

Merged
dennys246 merged 1 commit into
mainfrom
fix/ci-concurrency-starvation
Aug 7, 2026
Merged

ci: supersede in-flight runs per ref — fixes 15-minute queue starvation#470
dennys246 merged 1 commit into
mainfrom
fix/ci-concurrency-starvation

Conversation

@dennys246

Copy link
Copy Markdown
Owner

Diagnosis

Jobs across five open PRs reported failure today. None of them ran.

unit-tests: conclusion=cancelled  runner_name=""  steps=0
created 16:21:10 → completed 16:36:11   (exactly 15m01s)

runner_name is empty — no machine was ever assigned. The jobs starved in the queue and were killed at a 15-minute limit, then surfaced as failure in gh pr checks, indistinguishable from real test failures. Five PRs read red while the only PR containing source changes was green. That's the dangerous part: a genuine failure could hide among the fake ones.

Not slowness. A healthy unit-tests job is ~12 min of real work (3m45s dependency install + 7m43s test run) and passes; lint and aarch64 resolve succeeded in the very same runs where unit-tests starved. Re-running the starved jobs with no code change turned them green.

Cause: the workflow had no concurrency group, so every push left its predecessor's run alive and competing for runners — N pushes to one branch demanded N times the slots, across five branches simultaneously.

Fix

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}

cancel-in-progress is deliberately false on main — a superseded main run must still complete, because main's CI result is what the release gate and post-merge verification read. Only PR-branch runs are cancellable.

Not changed (deliberately)

  • The dependency install (3m45s, includes llama-cpp-python) is a plausible trim, but removing optional deps changes which tests skip vs run — a coverage change that shouldn't ride a CI-infra fix.
  • aarch64 resolve holds a slot 14–45 min and is a secondary contributor to demand; worth path-gating separately if starvation recurs.

🤖 Generated with Claude Code

Diagnosis (2026-08-06): with five PRs open and pushed in quick succession,
jobs were created but NEVER ASSIGNED A RUNNER (runner_name empty), sat
queued, and were killed at exactly 15m01s having executed ZERO steps. They
reported as 'failure' in gh pr checks — indistinguishable from real test
failures. Five PRs read red while the only PR containing source changes was
green, which is the dangerous part: a genuine failure could hide among the
fake ones.

Not slowness. A healthy unit-tests job is ~12 min of real work (3m45s
install + 7m43s tests) and passes; lint and aarch64 succeeded in the very
runs where unit-tests starved. The workflow had NO concurrency group, so
every push left its predecessor's run alive and competing — N pushes to one
branch demanded N times the slots, across five branches at once.

cancel-in-progress is deliberately FALSE on main: a superseded main run must
still complete, since main's CI result is what the release gate and
post-merge verification read. Only PR-branch runs are cancellable.

Verified: re-running the starved jobs with no code change turned them green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@dennys246
dennys246 merged commit 8f569d0 into main Aug 7, 2026
7 checks passed
@dennys246
dennys246 deleted the fix/ci-concurrency-starvation branch August 7, 2026 01:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant