Repository navigation
ci: stop rust-cache misses from runner image rollouts - #785
Merged
Merged
Conversation
rust-cache hashed every toolchain preinstalled on the runner. Hosted images are mid-rollout with Rust 1.99.0 next to the pinned 1.98.1, so each job flipped between two keys, missed its cache, and doubled the repo's cache footprint past the 10 GB cap (10.74 GB), which evicted more entries. - Disable the environment hash on every rust-cache step except ebpf-build; rust-toolchain.toml is already part of the lockfile key. ebpf-build keeps it because its nightly and bpf-linker pins live in the workflow. - Save caches from main only. PRs restore main's caches. - Drop the shared atomic-build job: each atomic leg builds inline again, so the Linux atomics no longer wait on the Windows build, and a warm build beats a job hop plus an artifact transfer. Windows stays in two shards.
Karib0u
enabled auto-merge (squash)
October 15, 2026 14:19
Karib0u
added a commit
that referenced
this pull request
Oct 15, 2026
#789) #785 set add-rust-environment-hash-key: false to stop the runner image's preinstalled toolchains splitting each cache key. That option also drops the lockfile hash, so every key became static (v0-rust-<job>-<os>-<arch>): it hits exactly forever, is never re-saved, and a Cargo.lock or rust-toolchain.toml bump never refreshes it. Keep rust-cache's default hashing and remove the split at its source: a step before each cache uninstalls every toolchain except the rust-toolchain.toml channel, so `rustup toolchain list` shows only the pin. A lockfile bump again makes a new key that falls back to the old cache. ebpf-build keeps its own nightly and is left alone.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #783, which made PR CI slower (about 19 min on #784, about 10 min before) because its new atomic caches never hit.
Why the caches miss
rust-cache hashes every toolchain installed on the runner into its key. Hosted images are mid-rollout: some runners ship Rust 1.99.0 next to our pinned 1.98.1, so every job flips between two keys depending on the image it lands on. Run 38743554001 shows it:
Atomic (macOS)listed both 1.98.1 and 1.99.0 and missed, whileClippy (macOS)in the same run had only 1.98.1 and hit.Each job now has two ~500 MB copies, plus per-PR copies, so the repo sits at 10.74 GB, over the 10 GB cap, and LRU eviction drops more entries.
Changes
add-rust-environment-hash-key: falseon every rust-cache step exceptebpf-build.rust-toolchain.tomlis already in the lockfile part of the key, so a toolchain bump still changes it.ebpf-buildkeeps the hash because its nightly and bpf-linker pins live in the workflow; dropping it would let a stale cached bpf-linker survive a pin bump.save-if: github.ref == 'refs/heads/main': PRs restore main's caches but no longer write their own copies.atomic-buildjob from ci: cache macOS atomic build, share one build per OS, shard Windows atomics #783. Each atomic leg builds inline again: the Linux atomics stop waiting on the Windows build, and a warm build is quicker than a job hop plus an artifact transfer. The repo is public, so the extra Windows build costs no minutes. Windows stays split in two shards.sigma-compatibility.ymlandsignpath-dry-run.yml.Expected
The first main run after merge writes caches under the new key format; PRs before that fall back to a prefix match on the old entries. After that, the Windows atomic build should go back to about 110s, and the PR critical path should be eBPF build, then build, then shard: about 6 min.
zizmor --offlineis clean on the three workflows.