Shared-layer lifecycle: accounting, eviction, recovery, materialization

[chruffins]

summary

Capstone stage of the image-storage project — finishes migration, eviction, accounting, recovery, and observability on the shared-layer path.

• Materialization in builds: after a pull, each manifest layer is materialized into images/layers/<digest>/ (best effort; the composed rootfs still comes from blobs, so an artifact failure only degrades sharing). New layer_materialization build phase with cache-hit attribution.
• Reference-protected eviction: image deletion evicts only layer artifacts unreferenced by any remaining manifest model; a 10-minute grace period keeps fresh artifacts safe from racing in-flight builds. Startup also evicts orphans left by an unclean shutdown.
• Interrupted-build recovery: stale .unpack-* / .install-* temp directories are swept at startup (age-gated so live builds are untouched).
• Layer-aware accounting: TotalImageBytes now includes the layer store's physical bytes (TotalLayerBytes exposes them separately), so capacity admission sees the real footprint.
• Observability: hypeman_images_layer_artifacts_evicted_total counter plus slog eviction summaries and per-layer materialization warnings.
• Legacy flattened images remain readable and bootable; they carry no manifest model and no layer references, and are retired only through the existing delete/promotion paths.

validation

• End-to-end lifecycle test: two images sharing a base layer materialize the base once; deleting one image keeps the shared layer; deleting the last reference evicts everything.
• Accounting, grace-period, reference-protection, and stale-temp-dir tests.
• go test ./lib/images ./lib/paths ./lib/builds ./lib/scopes green; cmd/api/api green except Docker Hub pulls (anonymous rate limit — confirmed via direct probe returning TOOMANYREQUESTS) and VM lifecycle tests (need bridge privileges; verified failing identically on unmodified main).
• CI is unavailable during this work; all validation is local.

---

Note

Medium Risk
Changes image deletion, startup cleanup, and disk admission accounting; incorrect eviction could remove data still needed by images, though manifest references and a grace period mitigate that.

Overview
Completes the shared-layer storage path: pulled images materialize per-layer artifacts under images/layers/ (best effort, with a layer_materialization build phase), and reference-aware eviction removes layer dirs only when no manifest model still references them.

Deletion and startup run eviction plus a sweep of stale .unpack-* / .install-* temps, gated by a 10-minute grace period so in-flight builds are not raced. TotalImageBytes now counts ready rootfs metadata plus physical layer-store bytes (TotalLayerBytes exposes layers alone); disk-usage caching tracks the new layerBytes component.

Adds hypeman_images_layer_artifacts_evicted_total and lifecycle tests (shared base layer materialized once, survives partial deletes, full eviction when the last reference goes).

^{Reviewed by Cursor Bugbot for commit 33c8ca3. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 33c8ca3. Configure here.}

[cursor]

Eviction can wipe in-flight layers

Medium Severity

Layer eviction decides freshness from the digest directory ModTime, and it runs concurrently with materialization. Unpack and mkfs.erofs write inside child temp dirs, so the parent mtime goes stale; DeleteImage can then RemoveAll that tree while a build is still using it, especially during the later unlocked ExportRootfs window before the manifest is written.

Additional Locations (2)

• lib/images/manager.go#L534-L569
• lib/images/manager.go#L865-L867

 

^{Reviewed by Cursor Bugbot for commit 33c8ca3. Configure here.}

[cursor]

Layer bytes skipped after failed builds

Medium Severity

Materialized layer artifacts are written to the layer store before conversion, but refreshDiskUsageTotals only runs on successful finalize, delete, or startup. After a failed conversion the artifacts remain on disk while TotalImageBytes keeps the stale cached total, so capacity admission undercounts the real footprint.

Additional Locations (2)

• lib/images/manager.go#L900-L907
• lib/images/disk_usage.go#L110-L153

 

^{Reviewed by Cursor Bugbot for commit 33c8ca3. Configure here.}