Repository navigation
ci: reuse cached builds and packages instead of rebuilding them - #182
Conversation
- Rust jobs: mr-boxington-action's `target` payload pruned the target tree to the packages `cargo metadata` lists for the default features, so every `--all-features` job recompiled the wgpu, CUDA, and Metal dependency trees (and their dependents) on every run. Cache mbx's objects instead, which cover every feature and target directory, with native links (build scripts, proc macros). - fuzz: cargo-fuzz builds in fuzz/target, which the job never cached (it cached the root target/). Cache fuzz/'s own tree. - parity: the uv cache key named no uv release, so after uv moved its cache buckets the exact-hit entry never held the built XGBoost wheel and every run rebuilt XGBoost from source. Key it on the uv release. - lavapipe: cache the .deb files per runner image and install them with dpkg (.github/actions/lavapipe); the Ubuntu mirror stalled jobs for minutes. libclang comes from the runner images' clang, so the CUDA header script no longer installs libclang-dev with apt. - Python wheels, the sdist round trip, and the CUDA runtime build with maturin through mise's cargo, so through mbx and its cache (the CUDA runtime without the release profile's LTO, as the wheel jobs). mise.lock gains windows-x64 entries for the Windows wheel jobs, and mise.ci.toml's RUSTUP_HOME follows mise's data directory there.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7f0da87563
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # resolved on one install as is on every runner of that image. | ||
| - id: key | ||
| shell: bash | ||
| run: echo "key=lavapipe-${ImageOS:?}-${ImageVersion:?}-$(dpkg --print-architecture)" >> "$GITHUB_OUTPUT" |
There was a problem hiding this comment.
Include the installer or a package-set generation in this key. If install-lavapipe.sh later adds or changes a package while ImageVersion remains constant, this restores the previous nonempty directory; the installer then sees an existing *.deb, skips downloading the revised package set, and the exact cache hit prevents replacement. That can leave the Vulkan dependencies incomplete and break the Linux GPU test paths this action is meant to enable.
AGENTS.md reference: AGENTS.md:L103-L106
Useful? React with 👍 / 👎.
Why CI was slow
Measured over the 10 CI runs before this change (median wall 591 s, range 451–1890 s):
install-lavapipe.shandinstall-cuda-headers.sheach ranapt-get update && installin every Linux job. The Azure Ubuntu mirror sometimes served at ~10–30 kB/s: the slowest runs spent 7 min onmesa-vulkan-driversand 19 min onlibclang-devin one job (1880 s test job).targetpayload prunestarget/to the packagescargo metadatalists for the default features. Every--all-featuresjob therefore recompiled wgpu/naga, cuda-core/bindgen, and objc2 (54 of the 72 crates the x86_64 test job compiled).fuzz/target, while the job cached the roottarget/("No reusable Cargo target state was produced").Changes
.github/actions/lavapipe: caches the lavapipe.debs per runner image (ImageOS/ImageVersion) and installs them withdpkg. Only a cache miss contacts the mirror.install-cuda-headers.sh: no more apt. The runner images' clang already shipslibclang, which bindgen loads. The manylinux container mode keepsdnf install clang-devel.github-cache-mode: objects(isolate-objects-cache,cache-links). mbx's objects cover every feature and target directory, plus linked build scripts and proc macros. Still saved only on pushes to main.fuzz/targetitself (MBX_TARGET_VIEWS=0, so it is a real directory). The entry is saved on main without hessboost's and the fuzz targets' outputs, which every crate change rebuilds.UV_CACHE_DIRsits on the checkout's volume, so uv hard-links packages on Windows instead of copying them.zstd-sys's build script erratically slow: 3–6 min instead of <1 min in 5 of 18 builds under mbx, with or withoutMBX_CC=0/MBX_BUILD_SCRIPT_EXECUTION=0, and never in the 25 builds without it (20 baseline, 5 on this branch).mise.lockgainswindows-x64entries, andmise.ci.toml'sRUSTUP_HOMEusesxdg_data_home, so Windows jobs get the pinned toolchain from the mise cache.Timings
Baseline: the 10 runs before this PR. New: the final configuration's run (39901730653). It restored caches built at an earlier commit and rebuilt hessboost, as a typical PR does.
A rerun without source changes (everything cacheable hits) took 293 s wall. The first run on main after merge is cold for the new cache generations, about the cost of the old setup. PRs restore main's entries after that.
Remaining critical path
test (aarch64-darwin)now bounds the wall time. It spends almost all of its time compiling hessboost itself: the library, its unit tests, 33 integration tests, and 30 examples on a 3-core runner. Across runs with identical work, itsnexteststep took 265–508 s, depending on the runner. Building examples in that job costs ~13% of the crate build (measured locally at-j3); clippy--all-targetsalready type-checks them. I left examples in the build, since dropping them would reduce coverage.