ci: stop container builds starving the runner fleet - #27
Open
thedancingdeveloper wants to merge 1 commit into
Open
ci: stop container builds starving the runner fleet#27thedancingdeveloper wants to merge 1 commit into
thedancingdeveloper wants to merge 1 commit into
Conversation
thedancingdeveloper
force-pushed
the
ci/stop-container-build-pileup
branch
from
August 18, 2026 06:18
1dec52e to
9fcc91e
Compare
Three pushes to main between 04:38 and 05:17 left three "Container images" builds running at once, one on each Node B runner, for 80+ minutes. CI, Dashboard demo and secscan queued behind them. Nothing was looping -- the builds are genuinely that slow, and nothing stopped them accumulating. Four changes: 1. concurrency groups on CI, Container images and Migration policy. A new push now supersedes the in-flight run for that ref instead of joining it. Dashboard demo already had one. 2. arm64 restricted to tags. platforms was amd64+arm64 on every push to main, with arm64 emulated through QEMU -- roughly an order of magnitude slower than native, twice over because there are two images. PR runs were already amd64-only, which is why they finished in minutes while push runs ran for over an hour and looked wedged. Tags still get both architectures. 3. Registry-backed build cache, replacing cache-from/cache-to: ''. Caching was disabled outright to work around a BuildKit wedge during GHA cache finalization, so every build has since paid full cost forever. type=registry avoids the GHA cache path that wedged, and unlike type=local it survives job-completed.sh wiping the workspace after each job. Both directions are gated on the same condition as the ghcr login step, since writing needs registry auth; PR builds stay uncached and are fast anyway. 4. runs-on: [self-hosted, publish] instead of [..., docker]. All three Node B runners advertise `docker`, but only the publish pair has a DinD sidecar -- a push-event build was dispatched to node-b-gha-public-rust, which has no Docker daemon at all. `publish` is the label that actually means a working daemon. See gap 4 in the ops repo's docs/github-actions-runner-gaps.md.
thedancingdeveloper
force-pushed
the
ci/stop-container-build-pileup
branch
from
August 18, 2026 06:24
9fcc91e to
7fee355
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three pushes to
mainbetween 04:38 and 05:17 left three Container images builds running at once — one on each Node B runner — for 80+ minutes. CI, Dashboard demo and secscan queued behind them, whileproxmox-gha-public-rustandwsl2-gha-public-rustsat idle and ineligible.Nothing was looping. Every run traced to a real push or PR. The builds are genuinely that slow, and nothing stopped them accumulating.
The asymmetry that made it look wedged
pull_requestlinux/amd64pushto mainlinux/amd64,linux/arm64arm64 is emulated through QEMU, twice over because there are two images, with caching disabled entirely.
Changes
concurrencygroups on CI, Container images and Migration policy — a new push supersedes the in-flight run for that ref. Dashboard demo already had one.mainbuild amd64 only; tags still get both.cache-from/to: ''. Caching had been switched off to work around a BuildKit wedge during GHA cache finalization, so every build has paid full cost ever since.type=registryavoids the path that wedged, and unliketype=localit survivesjob-completed.shwiping the workspace after each job. Gated on the same condition as the ghcr login, since writing needs auth.runs-on: [self-hosted, publish]instead of[..., docker]. All three Node B runners advertisedocker, but only the publish pair has a DinD sidecar — run #49 was dispatched tonode-b-gha-public-rust, which has no Docker daemon. See gap 4 in the ops repo'sdocs/github-actions-runner-gaps.md.Note on required checks
buildandpolicyare required. Job names are unchanged, so branch protection is unaffected. Change 4 narrowsbuildto the 2 DinD runners — correct, since it is the only place it can actually run.Full context:
docs/2026-08-18-runner-fleet-status-and-ci-policy-findings.md.