Repository navigation
Stop leaking warm snapshots; sweep the self-hosted Macs' disks - #182
Merged
Merged
Conversation
A snapshot's stamp names the hluk build that saved it (key=), and another build refuses it, but pruning only removed the snapshots of a rebuilt rootfs: every hluk upgrade left each guest's warm snapshots in the cache for good, about 1 GB for the agent and python images. On the self-hosted Macs, whose runners keep their disk, every commit that changed the kernel added a set, until a Mac's disk was full. Prune those of the same rootfs too when saving one, and test what goes and what stays. Signed-off-by: danbugs <danilochiarlone@gmail.com>
There was a problem hiding this comment.
🟡 Changes recommended
Failed cleanup-script downloads can be reported as successful sweeps, making runner coverage inaccurate.
1 open issue
What changed in this PR
This PR limits warm-snapshot cache growth and adds disk cleanup for self-hosted runners to prevent disk exhaustion.
Changes:
- Prunes snapshots from incompatible hluk builds for the same rootfs, with regression coverage.
- Adds disk reporting and cleanup before non-Windows arm64 host setup.
- Adds a Mac runner sweep workflow with coverage reporting.
| File | Description |
|---|---|
| src/cli/rootfs.rs | Prunes incompatible snapshots and tests retention behavior. |
| dev/ci-disk.sh | Reports disk usage and removes caches and build artifacts. |
| CHANGELOG.md | Documents the snapshot cleanup fix. |
| .github/workflows/runner-disk.yml | Adds runner sweeps and reports missed runners. |
| .github/actions/arm64-host/action.yml | Runs cleanup before host setup. |
Review effort: Balanced
| SCRIPT: https://raw.githubusercontent.com/${{ github.repository }}/${{ github.event.pull_request.head.sha || github.sha }}/dev/ci-disk.sh | ||
| run: | | ||
| echo "runner: $RUNNER_NAME ($(whoami))" | ||
| curl -fsSL --retry 3 "$SCRIPT" | bash |
Contributor
Author
There was a problem hiding this comment.
Fixed in bf43566: the step runs with shell: bash, which is bash -eo pipefail, so a failed download fails the job and coverage does not count that runner as swept.
…s, and sweep every runner The Macs' runners keep their disk between jobs, and a Mac's runners share it with hyperlight's jobs; the Linux and Windows arm64 hosts are fresh machines each time. dev/ci-disk.sh shows what fills the disk and removes what hluk and its tests leave there: warm snapshots and pulled images untouched for days, hluk-test-* directories a killed test left, and, under 20 GB free, the checkout's target/ and cargo's download cache. The arm64-host action runs it first, before anything that needs room. runner-disk.yml (dispatched, or on a change to it or the script) sweeps each runner of both Macs: the whole hluk cache, every hluk-* temp directory and this repo's build directories, one job per runner, each holding its runner so they spread, and lists any runner none reached. Other repos' files are shown, not removed. Signed-off-by: danbugs <danilochiarlone@gmail.com>
Point HLUK_CACHE_DIR at the job's temp directory on the self-hosted hosts, which the runner empties when the job ends, cancelled or not, so the images a job pulls and the warm snapshots it saves go with it, instead of staying in the runner user's home for later jobs, and hyperlight's, to find the disk full. Signed-off-by: danbugs <danilochiarlone@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every hluk upgrade left each guest's warm snapshots in the cache, about 1 GB per build for the agent and python images. The self-hosted Macs keep their disk between jobs, so every commit that changed the kernel added a set, until Simons-Mac-mini's disk was full and its jobs, hyperlight's included, started failing with
No space left on device.hluk runnow prunes snapshots of the same rootfs saved by another hluk build, which no run can restore.HLUK_CACHE_DIR=$RUNNER_TEMP/hluk-cache), which the runner empties when the job ends, so a job's snapshots and pulls go with it.dev/ci-disk.sh, which prints what fills the disk and removes what hluk and its tests leave there: snapshots and pulled images untouched for days,hluk-test-*dirs left by killed tests, and, under 20 GB free, the checkout'starget/and cargo's download cache.runner-disk.ymlsweeps every runner of both Macs (all of hluk's cache, temp dirs and this repo's build dirs), one job per runner, and lists any runner it missed so it can be run again. It runs on this PR, and can be dispatched later. Other repos' files are reported, never removed.