Repository navigation
Disk budget: build box cache maxima sum to 839 GiB on a 484 GiB disk #276
Description
Problem
The Linux build box (484 GiB root disk) filled to 100% four times on 2026-09-27/28. Each time ENOSPC killed a gate or release run. Every cache stayed within its own contract, but together the configured maxima added up to 839 GiB, well over the size of the disk.
Measurements (2026-09-29, read-only)
| Owner | On disk | Configured max (before) | Notes |
|---|---|---|---|
cache/target/cargo |
171 GB, rising to 188–208 GB during the release run | 180 GiB | debug/ 153–165 GB: incremental/ 75–83 GB (1,119–1,429 session dirs) and deps/ 73–80 GB. llvm-cov-target 15 GB, not touched in over 7 days. musl/cross targets about 3 GB |
└ debug/deps by kind |
78 GB | test executables 41.7 GB, rlib 27.6 GB, rmeta 4.3 GB, .so 3.5 GB | |
| └ debuginfo share | DWARF is 77% of test executables (8.5 of 11.0 GB sampled), about 41% of rlibs and incremental .o files |
||
| └ duplication | distinct unit hashes per crate: capsem_proto 78–89, capsem_foundation 74–84, capsem_core 61–65, capsem_assets 63–64, capsem_service 42–55. Third-party crates hold about 5% of bytes | ||
| └ age | deps: 45 GB over 7 days old. incremental: 48 GB over 7 days old | ||
| Docker/containerd | 69 GB (/var/lib/containerd: 59 GB snapshots + 9.6 GB content) |
96 GiB (accounted 61.8 GiB) | one generation of each Capsem image is about 20 GB unique+shared. BuildKit 54.5 GB, 48.6 GB reclaimable, mostly last used 3 weeks ago (kernel build layer 2 GB ×3, cargo registry copy 673 MB ×6, cdxgen 1.2 GB ×2) |
cache/objects |
37 GB | 48 GiB, 90 days | ~40 rootfs objects of 845 MB, most link count 1 (unreferenced), from 112 rootfs receipts dated 09-01..09-05 |
cache/tools |
14 GB | pycache 5.4 GB (above its 2 GiB max), sccache 4.9 GB, uv 2.6 GB | |
cache/worktrees |
4.8 GB | 20 GiB | |
/var/tmp/capsem-tests |
7–42 GB | 200 GiB | leak fixed by #271, max now 32 GiB |
~/capsem-nightly (separate clone, own cache) |
51 GB | not governed | cargo 30 GB, coverage 15 GB. Sits outside the shared cache authority |
| Foreign data | ~70 GB | n/a | ~/magika 35 GB, ~/archive 30 GB |
Root causes, ranked by bytes
- Incremental sessions nobody reads back (75–83 GB). A gate prefix is named after its source digest, and workspace units are salted with its path. A session is never reused: changed source means a new prefix, and an exact repeat is already fresh. sccache also refuses incremental output, so workspace crates were never cached.
- Full DWARF in the dev/test profile (roughly 55–60 GB of deps+incremental). It is paid once per checkout because of the path salt.
- Maxima that sum to 1.9× the disk, and retention that only triggers at the maximum. Cargo warm is 150 GiB and prune age is 30 days, so the stage holds a month of dead checkouts before it prunes anything.
- BuildKit age policy is dead code (48.6 GB).
maximum_age_hourswas applied to containers and volumes. The build cache was pruned only once Docker exceeded 96 GiB, which never happened. - Object store kept 90 days of rootfs generations (≈34 GB).
- Cross-checkout unit duplication (60–90 units per workspace crate). Each checkout and each gate prefix builds its own set, and check, clippy, test and build are distinct units. The path salt from
rustc-workspace-wrapperis deliberate: Cargo freshness is mtime-based, so without the salt an older snapshot can take a newer tree's objects as fresh (seeskills/dev-cache). It can only be removed together with content-hash freshness (-Zchecksum-freshness, or its stable successor), and it has to be proven first by the cross-tree correctness test intest_gate_prefix.py. - Enforcement crashed or reported nothing useful. The release
just testdied 5.8 s in because a rustc.rcgu.otemporary disappeared during the enforcement walk (FileNotFoundError). When a held Cargo lock pins every unit, the only output was "cargo remains N bytes above max".
Fixes (branch fix/disk-budget, on release/0.6.4)
| Commit | Fix | Expected saving |
|---|---|---|
perf(build) |
[profile.dev] debug = "line-tables-only". A citadel guard refuses full debuginfo in dev, test or package overrides |
~55–60 GB at today's unit count. A checkout's units shrink ~2–3× |
perf(gate) |
CARGO_INCREMENTAL=0 for gate and bounded builds ([toolchain] cargo_incremental) |
75–83 GB, and workspace crates become sccache-cacheable |
fix(cache) |
the measure walk treats a vanished file as 0 bytes | unblocks enforcement (it crashed the release run) |
fix(cache) |
an over-max violation names the protected bytes, the held lock and the holder's pid/command (/proc/locks) |
diagnosis |
fix(cache) |
the BuildKit age prune runs on every routine prune (docker builder prune --all --filter until=<age>h --reserved-space 0B) |
~48 GB now, and stays bounded |
feat(cache) |
disk-budget invariant plus coherent maxima (below) | caps the steady-state total |
test(cache), docs(agents) |
two pre-existing release-branch breakages (merge-skewed hold_environment call; a citadel contract phrase) |
Disk-budget invariant
[budget] in config/cache.toml: filesystem_fraction = 0.8, headroom_bytes = 32 GiB, minimum_filesystem_bytes = 476 GiB (a 512 GB disk), ephemeral_environment = {RUNNER_ENVIRONMENT = "github-hosted"}.
- At policy load: if Σ(stage + runtime maxima) + headroom > fraction × minimum filesystem, the load raises
ValueError. Every test, CLI and gate command loads the policy, so the check runs everywhere. - At gate start: before an exclusive command waits for the machine lock, it checks Σ + headroom ≤ fraction × the real size of the cache filesystem. An optional runtime that is not installed (Tart on Linux) counts as zero. GitHub-hosted runners are exempt because their caches go away with the runner.
New maxima (GiB, max/warm)
cargo 180/150 → 48/32 (age 30 → 7 days) · docker 96/72 → 40/28 (age 3 → 7 days) · test-temp 200 → 32 (#271) · objects 48/32 → 12/8 (age 90 → 14 days) · sccache 24 → 12 · worktrees 20 → 12 · buildkit-exports 32 → 8 · build-workspaces 24 → 8 · prefix-products 16 → 8 · assets 16 → 12 · install-tests 16 → 8 · release-proofs 12 → 8 · release-staging 8 → 4 · toolchain 8 → 4 · web-parity 4 → 2 · node-pnpm 4 → 2 · generated 4 → 1. Tart stays at 64 (macOS glowup VMs, not measurable here).
Σ maxima: 839 → 348 GiB (284 GiB on Linux without Tart), about 59% of this disk. Warm total 204 GiB.
Still open
- Validate the new maxima with one complete
just testand an exact repeat (skills/dev-cache, "Changing policy"). The cuts to stages that measured ~0 bytes (install-tests, ironbank, release-proofs, build-workspaces, buildkit-exports) are estimates. Recording peak usage per stage during a run would make them measured. - Unit duplication across checkouts: evaluate content-hash freshness so the path salt can be dropped, and cut the per-checkout unit variety (check/clippy/test/build feature sets).
-
python-pycacheis over its maximum (2.8 of 2 GiB). -
~/capsem-nightlyis a standalone clone with a private 50 GB cache, outside the shared authority. Retire it or point it atCAPSEM_CACHE_AUTHORITY. - The flake
test_gate_runlog.py::test_a_run_is_protected_before_its_directory_becomes_visible(PytestUnhandledThreadExceptionWarning) is intermittent onrelease/0.6.4itself, without these changes.
One-time cleanup (not run; the release just test needs these caches)
Run these after the release run finishes, once the branch has landed:
just cache enforce cargo --reason "one-time: line tables + no incremental; new 48 GiB max"
just cache prune objects --apply --reason "one-time: 14-day object retention"
just cache prune docker --apply --reason "one-time: BuildKit records past their age"
just cache enforce python-pycache --reason "over max"
# optional, explicit: rm -rf ~/capsem-nightly (separate clone, 51 GB)The first compile after the profile change rebuilds every workspace unit once, because the profile is part of the unit hash.
Activity
ebursztein commentedon Sep 29, 2026
Lesson from the 0.6.4 release proof (2026-09-29): an early
just cache enforce dockerevicted the currentcapsem-host-builderimage. Rebuilding it depends on snapshot.ubuntu.com, which that day served each index in ~10 s and each .deb in ~1.3 s. The rebuild then failed its 300 s / 600 s outage bounds four times (update, then install of ~489 packages), and blocked the release proof for hours.Requirement for the disk-budget work: Docker enforcement and pruning must never evict the current generation of any builder image the gate needs (host-builder, guest Rust builders, asset tools, base images). Evict only superseded generations and unused BuildKit cache. Add a guard test. Cutting the host builder's apt sources to main/universe over noble, noble-updates and noble-security (12 indexes instead of 32) is already on release/0.6.4; the install phase is still ~489 sequential fetches.
ebursztein commentedon Sep 29, 2026
Round 2: branch
fix/cache-276(onrelease/0.6.4, pushed, not merged)3a02e073ffix(cache)b86df5079fix(cache)python-pycache,python-pytest).f0427737efix(cache)2be5a47a7fix(cache)abf69ae3a,5460e4f441. Current builder images
The runtime journal shows how the images went. At 13:41 and 13:55 today, enforcement treated "current" as the newest image by creation time. It removed
capsem-kernel-dependencies-x86_64:80238a5…, the release proof's current tag, and the proof rebuilt it minutes later. Newest-by-creation is not the current image, for three reasons:The host-builder removal from this morning left no journal record.
Now the gate declares which tags are current (
cache/dockercurrent.py, records instate/docker-current/):reclaim-image --applyrecords its--keeptag for the calling checkout;:lateston every path;A record stays live while its checkout exists and it is younger than 168 h.
scan_runtimesrequirespaths=, so every planner sees these tags as current. Pressure on current images is reported as a violation and never resolved by eviction. Only an explicitcleancan remove a current image. Guard tests:build_system/tests/cache/test_docker_current.py(planner,enforce_runtime, and the CLI end to end) and the host-builder and backend tests.2. Per-checkout Cargo units
Content-hash freshness is not available.
cargo 1.97.1still lists-Z checksum-freshnessas unstable.RUSTC_BOOTSTRAPis not acceptable for release builds, and a privateCARGO_TARGET_DIRper checkout is ruled out by AGENTS.md.The salt stays. Without content-hash freshness there is no correct way to share workspace units across checkouts with different sources.
The duplication is bounded by owner instead. The workspace wrapper links each unit's fingerprint directory to the checkout that compiled it. The link is a symlink, so reading it changes no atime and measuring does not count it. Clippy's wrapper runs through the same script. Retention then works in this order:
--policy-repository, or the bounded command's root) and the authority (main).test_cargo_working_set.pyproves that enforcing over the maximum does not evict the units the calling checkout's next build needs, at the planner,enforce_repositoryand CLI levels.test_cargo_workspace_cache.pyandtest_clippy_checkout_key.pyprove on real cargo that the links are written.Why the order matters: relatime makes shared units look old, so plain LRU evicted third-party crates first. Every candidate run today recompiled 167-231 third-party crates in
prepare.build-binaries(versus 0-3 on 09-04/05), andfast.clippyre-checked 361-537 of them in 4 of 13 runs (1m12s-1m43s against 54-57s warm).Cost: the wrapper is an identity input of the package-builder and install-builder images, so each rebuilds once (cold materialize measured at 90-123 s).
Not covered yet: units built before this change have no link and rank as shared. They age out within the 7-day bound.
3.
python-pycacheover its maximumEvery gate launch and bounded command creates a generation keyed by the checkout's Python sources. Every edit in any worktree therefore adds one (24-207 MB each; 29-31 were on disk). The only routine prune ran in the package and install rails of a complete gate. Now the launcher that creates a generation leases it and enforces both stages; a warm launch pays nothing. On this machine the stage went from 31 generations to 20 (385 MB).
4.
~/capsem-nightlyThe maintainer removed it: a clean tree with no unique commits, 51 GB reclaimed.
Reuse measurements (requirements a and c)
From
ledger.jsonl. Before = 09-04/05 (incremental, full DWARF); after = 09-29.prepare.build-binariesstatic.rust-coveragefast.clippy, warmControlled experiment on
cargo build -p capsem-service(339 units) in throwaway target dirs:sccache caches no workspace crate. The fast-test stats on this branch show 0 hits and 150 non-cacheable calls, 141 of them "multiple input files", because sccache treats the workspace wrapper as the compiler. Filed as #277. The saving above comes from third-party crates only.
CARGO_INCREMENTAL=0tradeoff (requirement c). The gate loses nothing: a new prefix never reads an incremental session back. An agent iterating in one checkout pays for it. Editingcapsem-coreand rebuildingcapsem-service:CARGO_INCREMENTAL=1CARGO_INCREMENTAL=0Incremental also cost 2.7 GB of target against 1.6 GB. So bounded direct cargo is about 3× slower per edit-rebuild in exchange for the ~80 GB of sessions. If that matters, one option is incremental for bounded commands only, since those run in a stable checkout and hold their own lock.
Tests
just fast-test: ok (3m05s).build_system/tests/{cache,gate,image,scripts}: 5370 passed. The 24 failures are all the missing mock-server binary, which is environmental.tests/citadel -n 4: all pass.