Repository navigation
CI traffic: superseded push runs are never cancelled (21 stale runs, 70 queued macOS jobs on 2026-11-05) #271
Description
On 2026-11-05, with eight merges to main in one day, my push runs piled up: 21 stale runs (mostly Test Python Examples, 4 macOS legs each) were queued behind each other for hours, holding about 70 queued macOS jobs while only 5 macOS jobs run at a time. The release candidate's own macOS legs waited behind them, and the organisation shares its runner cap with ArcadeData/arcadedb, so this is traffic I put on upstream's queue too.
Cause, from the concurrency blocks on main:
| Workflow | Group today | Effect |
|---|---|---|
test-jvm.yml, test-python-examples.yml |
<workflow>-<PR number or run_id>, cancel only on pull_request |
a push, a schedule and a dispatch each get a group of their own, so a superseded push run is never cancelled or replaced (the comments in both files say this is deliberate: a superseded run is the record for its commit) |
test-native.yml |
test-native-<PR number or ref>, cancel unless the ref is a tag |
a newer push to main cancels the older one (good), but a workflow_dispatch on main shares that group, so a manual run can cancel a push run, or be cancelled by the next push |
| other workflows | per PR, or per tag | fine |
Proposal, using GitHub's own behaviour: with cancel-in-progress: false a group holds at most one running and one pending run, and a newer pending run replaces the older pending one.
- Pull requests: unchanged (one group per PR, a new push cancels the old run).
- Push to
main, JVM and examples workflows: one group per workflow (<workflow>-main),cancel-in-progress: false. At most one run is testing and one is waiting, and the waiting one is always the newest commit, so a backlog cannot build. A commit that lands while another is waiting is covered by the newer run; each merge was already gated green on its pull request. - Nightly schedule: its own group per workflow with
cancel-in-progress: true, so a stuck old nightly never blocks a new one. workflow_dispatchandworkflow_call(the release, and runs pinned to an engine tag): unchanged, a group of their own per run, never cancelled. A release must never be cancelled by a push.test-native.yml: giveworkflow_dispatchits own group, so a dispatch and a push cannot cancel each other.- Tests in
tests/test_select_matrix.pystyle: a test reads the three concurrency blocks and checks that a release or a dispatch has a group of its own and that a push tomainshares one.
Out of scope here, to look at afterwards: the examples workflow runs all 16 legs on every push to main (the pull request ran 4); whether that should be nightly and release only is a separate question.
Timing: after the 26.11.1 tag, because a workflow change triggers runs and the macOS queue is full until the release checks finish.
Activity