Repository navigation
fix(cache): a test-temp run lives exactly as long as its owner - #271
Merged
Merged
Conversation
/var/tmp/capsem-tests/<namespace> reached 42 GB: one ~7 GB run-<pid> per
release precheck plus ~1,900 .run-<pid>.lock files back to early September.
`cache prune test-temp` offered 21 MB; only a cold clean reclaimed it.
Two defects, one on each side of the lease:
- Nothing removed a run when it ended. The launcher leased run-<pid> and let
the lease drop at exit, leaving the directory and its lock file behind on
success and failure alike. gatelaunch now leases the run before creating it
and, at exit, removes the run and its lease through
operations.reclaim_generation -- the same guarded, journaled removal a
prune uses. A killed process leaves a run no lease holds.
- Prune treated a dead run like a retained generation: it aged for 24 hours
under a 200 GiB maximum. For a leased ephemeral stage an unleased
generation now has "no live owner" and is reclaimed whatever its age or
size. Inventory attaches each lease file to its generation, so both leave
together, and reports a lease whose generation is gone as lease_only: it is
collected ("orphaned lease") and never counts toward maximum_count.
plan_clean now removes a generation's members too.
Removal takes each generation's lease exclusively, so a reused pid that
leased after the plan keeps its directory (ApplyResult.busy). retain_path
waits out that exclusive hold instead of failing, and retakes a lease file
that was unlinked under it.
With dead runs reclaimed on sight, test-temp holds only live runs, so its maximum is a statement about how much live scratch is reasonable, not a retention budget. 200 GiB on a 484 GB disk that also carries a 180 GiB Cargo target bounded nothing: the disk filled first. 32 GiB is about four times the largest run observed (a 7.2 GB release precheck), enough for a gate plus several concurrent bounded suites; more than that is a leak that enforcement should report rather than absorb. Warm stays 8 GiB.
pytest's default retention keeps every tmp_path until the next session wipes the basetemp, so a run's scratch peaked at the sum of the whole suite (0.8 GB for build_system/tests alone). With "failed", only failing and in-flight tests hold their directories, and the failing ones remain for inspection until the run ends.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #271 +/- ##
=========================================
- Coverage 65.3% 65.3% -0.1%
=========================================
Files 1452 1452
Lines 127965 128076 +111
Branches 91872 91872
=========================================
+ Hits 83614 83637 +23
- Misses 39361 39446 +85
- Partials 4990 4993 +3
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
ebursztein
added a commit
that referenced
this pull request
Sep 29, 2026
Each cache contract bounded only its own cache. On 2026-09-29 the maxima summed to 839 GiB on a 484 GiB build box (Cargo 180, test-temp 200 before #271, Docker 96, Tart 64, objects 48, and a long tail), every cache honoured its contract, and the disk filled to 100% four times in two days. [budget] in config/cache.toml declares filesystem_fraction (0.8), headroom_bytes (32 GiB) and the smallest retained-cache filesystem (476 GiB, a 512 GB disk). Loading refuses maxima whose sum plus headroom does not fit that share of it, and every exclusive gate command refuses, before waiting for the machine lock, a real filesystem the caches can fill. An optional runtime that is not installed (Tart on Linux) counts zero; a GitHub-hosted runner is exempt because its caches die with it. Maxima now sum to 348 GiB (284 GiB without Tart), measured on this box: cargo 180->48 GiB (warm 32, age 30->7 days), Docker 96->40 GiB (age 3->7 days, BuildKit age pruning now real), objects 48->12 GiB (age 90->14 days; 37 GB held ~40 unreferenced 850 MB rootfs objects), sccache 24->12, worktrees 20->12, buildkit-exports 32->8, build-workspaces 24->8, prefix-products 16->8, assets 16->12, install-tests 16->8, release-proofs 12->8, release-staging 8->4, toolchain 8->4, web-parity 4->2, node-pnpm 4->2, generated 4->1. Stages measured at or near zero bytes take the cuts.
5 tasks
ppcavalcante
pushed a commit
to ppcavalcante/capsem
that referenced
this pull request
Oct 1, 2026
release/0.6.4 merged google#271's probe, written against hold_environment(root), with the bounded-command fix that changed it to hold_environment(environment, root). Both parametrizations failed with 'PosixPath' object is not subscriptable.
ebursztein
added a commit
that referenced
this pull request
Oct 6, 2026
test-temp is namespaced by sha256(authority)[:8], and prune and enforce only inventoried their own authority's namespace. A namespace whose authority was gone -- a deleted worktree that exported CAPSEM_CACHE_AUTHORITY, a test's temporary checkout -- was never looked at again: about 350 of them held 2.4 GB on one machine (#272). Since #271 a run lives exactly as long as a process holds its lease, whichever namespace it is in, so a leased ephemeral external stage is now swept whole (cache/namespaces.py). Every other namespace's runs and leases are entries keyed '<namespace>/<run>', removed under the same exclusive lease guard as the current namespace's, and a namespace left empty is removed with rmdir, never recursively. An owner that loses its just-created namespace to that rmdir recreates it (leases.retain_path). Only a real directory with an 8-hex namespace name is entered. A symlinked namespace, a pre-namespace capsem-test-* directory, and any file that is not a run or its lease are never touched; apply_prune refuses a planned foreign path that is not exactly that key's run, its lease, or its empty namespace. The per-directory entry classification moves to cache/generations.py so both namespaces share it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On 2026-09-28,
/var/tmp/capsem-tests/6e48ba3b/(thetest-tempstage) had grown to 42 GB: onerun-<pid>/of about 7.2 GB per release precheck, plus about 1,900.run-<pid>.lockfiles going back to 2026-09-04.just cache prune test-tempoffered 21 MB. Onlyjust cache clean test-temp --applygot the space back.Root causes
gatelaunchcreatesrun-<pid>(the gate's or the bounded wrapper'sTMPDIRand pytest--basetemp) and leases.run-<pid>.lock. The lease is dropped at process exit, but the directory and the lock file were left behind every time, on success and on failure. Every launcher did this:capsem-gate,capsem-cache,run-bounded-command.py, and every test that starts one.ephemeralstrategy the planner used the same rules asgenerational: a run with no lease was reclaimed only aftermaximum_age_hours(24) or once the stage passed its 200 GiB maximum. Lock files were not matched bymanaged_globs, so they were never collectable at all.contained_environmentcreated the run directory before any lease existed. Tests that build a temporary checkout therefore leftrun-<pid>directories in one new namespace per test. This is where most of the ~350 namespace directories under/var/tmp/capsem-testscame from.Fix
gatelaunch.hold_environmenttakes the lease first, then creates the run. At exit it hands the run tooperations.reclaim_generation, which releases the lease and removes the run and its lease through the same guarded, journaledapply_prunepath. A process that is killed leaves a run that no lease holds.ephemeralstage, an unleased generation is now reported as "no live owner" and is reclaimed whatever its age or size.lease_onlyentry: it is collected as "orphaned lease" and never counts towardmaximum_count.plan_cleannow also removes a generation's members.apply_pruneholds each generation's lease exclusively while removing it. If a reused pid leased the run after the plan was made, the run is kept and reported inApplyResult.busy.retain_pathnow waits for that exclusive hold instead of raising, and re-takes a lease file that was unlinked while it waited.test-tempmax goes from 200 GiB to 32 GiB, with the reasoning inconfig/cache.toml. The stage now holds only live runs, and 32 GiB is about 4x the largest run seen.tmp_path_retention_policy = "failed". On this box, the peak scratch forbuild_system/testsdrops from 0.8 GB to 0.3 GB.Evidence
build_system/testsrun each left theirrun-<pid>behind (the latter 801 MB).node_modulesand assets, the contract suites peak at 0.8 GB and 76 MB. With this change, a run no longer outlives its owner whatever its size.Tests
build_system/tests/cache/test_scratch_runs.py:build_system/tests: the only failures are the 23 environmental ones that also fail onorigin/mainon this machine (test_mock_server_launcherandtest_protocol_fixture_recorderneed thecapsem-mock-serverbinary).tests/citadel -n 4passes: 1255 tests.ruffandcapsem-gate lintare clean.Follow-up (not in this PR)
External namespaces from retired authorities, such as deleted worktrees that exported
CAPSEM_CACHE_AUTHORITY, are never inventoried by any other authority. Their dead runs stay behind until someone cleans them by hand.🤖 Generated with Claude Code