Skip to content

ci: stop container builds starving the runner fleet - #27

Open
thedancingdeveloper wants to merge 1 commit into
mainfrom
ci/stop-container-build-pileup
Open

ci: stop container builds starving the runner fleet#27
thedancingdeveloper wants to merge 1 commit into
mainfrom
ci/stop-container-build-pileup

Conversation

@thedancingdeveloper

Copy link
Copy Markdown
Collaborator

Three pushes to main between 04:38 and 05:17 left three Container images builds running at once — one on each Node B runner — for 80+ minutes. CI, Dashboard demo and secscan queued behind them, while proxmox-gha-public-rust and wsl2-gha-public-rust sat idle and ineligible.

Nothing was looping. Every run traced to a real push or PR. The builds are genuinely that slow, and nothing stopped them accumulating.

The asymmetry that made it look wedged

Event platforms push Outcome
pull_request linux/amd64 no finished in minutes (#48, #50)
push to main linux/amd64,linux/arm64 yes 80+ min, still running (#47, #49)

arm64 is emulated through QEMU, twice over because there are two images, with caching disabled entirely.

Changes

  1. concurrency groups on CI, Container images and Migration policy — a new push supersedes the in-flight run for that ref. Dashboard demo already had one.
  2. arm64 restricted to tags. Pushes to main build amd64 only; tags still get both.
  3. Registry-backed build cache replacing cache-from/to: ''. Caching had been switched off to work around a BuildKit wedge during GHA cache finalization, so every build has paid full cost ever since. type=registry avoids the path that wedged, and unlike type=local it survives job-completed.sh wiping the workspace after each job. Gated on the same condition as the ghcr login, since writing needs auth.
  4. runs-on: [self-hosted, publish] instead of [..., docker]. All three Node B runners advertise docker, but only the publish pair has a DinD sidecar — run #49 was dispatched to node-b-gha-public-rust, which has no Docker daemon. See gap 4 in the ops repo's docs/github-actions-runner-gaps.md.

Note on required checks

build and policy are required. Job names are unchanged, so branch protection is unaffected. Change 4 narrows build to the 2 DinD runners — correct, since it is the only place it can actually run.

Full context: docs/2026-08-18-runner-fleet-status-and-ci-policy-findings.md.

@thedancingdeveloper
thedancingdeveloper force-pushed the ci/stop-container-build-pileup branch from 1dec52e to 9fcc91e Compare August 18, 2026 06:18
Three pushes to main between 04:38 and 05:17 left three "Container images"
builds running at once, one on each Node B runner, for 80+ minutes. CI,
Dashboard demo and secscan queued behind them. Nothing was looping -- the
builds are genuinely that slow, and nothing stopped them accumulating.

Four changes:

1. concurrency groups on CI, Container images and Migration policy. A new
   push now supersedes the in-flight run for that ref instead of joining it.
   Dashboard demo already had one.

2. arm64 restricted to tags. platforms was amd64+arm64 on every push to main,
   with arm64 emulated through QEMU -- roughly an order of magnitude slower
   than native, twice over because there are two images. PR runs were already
   amd64-only, which is why they finished in minutes while push runs ran for
   over an hour and looked wedged. Tags still get both architectures.

3. Registry-backed build cache, replacing cache-from/cache-to: ''. Caching was
   disabled outright to work around a BuildKit wedge during GHA cache
   finalization, so every build has since paid full cost forever. type=registry
   avoids the GHA cache path that wedged, and unlike type=local it survives
   job-completed.sh wiping the workspace after each job. Both directions are
   gated on the same condition as the ghcr login step, since writing needs
   registry auth; PR builds stay uncached and are fast anyway.

4. runs-on: [self-hosted, publish] instead of [..., docker]. All three Node B
   runners advertise `docker`, but only the publish pair has a DinD sidecar --
   a push-event build was dispatched to node-b-gha-public-rust, which has no
   Docker daemon at all. `publish` is the label that actually means a working
   daemon. See gap 4 in the ops repo's docs/github-actions-runner-gaps.md.
@thedancingdeveloper
thedancingdeveloper force-pushed the ci/stop-container-build-pileup branch from 9fcc91e to 7fee355 Compare August 18, 2026 06:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant