Skip to content

sandbox: image import locking, deterministic export and interrupted-load recovery (#175 part 1) - #738

Open
jiangkoumo wants to merge 9 commits into
Ephemeral-AI-Lab:layerfs_v2_0_0from
jiangkoumo:feat/175-image-import-p1
Open

jiangkoumo wants to merge 9 commits into
Ephemeral-AI-Lab:layerfs_v2_0_0from
jiangkoumo:feat/175-image-import-p1

Conversation

@jiangkoumo

Copy link
Copy Markdown
Collaborator

Tier: P1 (control/crates/layerfs-sandbox) ; risks: risk:delete, risk:security ; gate: yes: reclaims this home's leftover export containers and removes stranded unpacked image dirs under the image lock; parses untrusted tar PAX/sparse headers with bounds ; de-escalated: none

First part of Ephemeral-AI-Lab/layerfs-project#175 (Docker image → machine import reliability). This PR does not close #175; the remaining pieces come in follow-ups on top of this branch.

What changes:

  • One copy per image per machine (I01): machine.load takes a cross-process ImageLock per image, so concurrent loads of the same image export once and the rest reuse it; a lock held by a dead process can be taken over. Free space is checked before anything is created.
  • Export (I02): each export runs in its own container (named by home, machine and a random suffix) and removes only its own container and anonymous volumes. Directory mtimes are normalized to epoch 0 (PAX time records stripped, checksum recomputed), so exports taken at different times are byte-identical and a partial download can resume. load reclaims this home's leftover export containers and reports cleanup failures.
  • Recovery and listing (I03): machine.images lists staged partial downloads (size, age, resumable) and stranded unpacked directories without a sidecar; machine.prune only removes what no live lock or Sandbox holds. The sidecar is the publication commit point: probe reports an un-sidecared directory as stranded read-only, and publish removes and re-unpacks it under the lock. Moving a tag strips the stale ref from older sidecars; a ref that matches several cached images is refused (InvalidInput, suggesting machine.prune). Unpack reports dropped FIFOs, devices and init files and sparse expansion bytes; a hardlink to a dropped target is an error. variant/config are recorded, not applied.
  • tar PAX/sparse parsing split into image_tar/{header,pax,sparse,filter}.rs, with extended headers capped at 48 MiB.
  • Docs: docs/architecture/api/sdk/machine.md (incl. two known limitations of ref maintenance), sandbox.md, control/home/README.md, docs/versioned/2.0.0/calls.md.

Tests:

  • new lp176_image_lock, lp176_image_export, lp176_image_recovery, plus image_tar and image_copy updates: 47 pass (real Docker: two exports byte-identical, leftovers reclaimed after kill);
  • on this head (merged with layerfs_v2_0_0 caff58a): fmt/clippy clean, slow 171/171, mount 45/45, PostgreSQL packages pass; compared with a clean layerfs_v2_0_0 caff58a run, no new failures;
  • macOS: 4 concurrent loads of one image exported once, the other 3 reused it, no leftovers.

Known, not from this PR: on caff58a the fast lane does not compile (sqlite_compact_capacity.rs:75:11, E0004, from #732), so CI here will be red until that is fixed upstream; with that test excluded, the remaining fast failures match the base. Docker's classic image store was not exercised (test host uses the containerd store).

…ecked before anything exists

A copy of an image resolved its Docker Id and then raced every other copy of
that image: "another copy is running" meant only that the staged part had
grown in the last 30 s (`image_artifact::RECEIVING`), a waiter went ahead
anyway after 10 minutes (`image_load::WAIT`), and both copies shared the
export container `layerfs-export-<hex>` and the part `.recv-<hex>.part`, so a
second `write` at offset 0 truncated the first one's bytes.

- `ImageLock`: one exclusive `flock` per home, machine and image, at
  `<home>/run/image-lock/<machine>/<hex>.lock`, held for the whole copy, with
  the owner recorded beside it (`pid`, `pid_start`, `pid_boot`, machine,
  image, since). The lock is the exclusion and the record is only what a
  waiter reports: the kernel releases a dead owner's lock, so the next copy
  takes it over and resumes the part, and a record whose pid was reused holds
  nothing. No deadline: a live owner is waited for, with
  `{phase: "lock", waiting: "lock", holder}` each second.
- Capacity runs before the first effect: the check now precedes the export
  container, so a refusal leaves no container, no staged byte and no cache
  folder; the container's `SizeRootFs` only refines the progress estimate.
  `NoSpace {need, free}` becomes `Capacity {needed, free, where}`; `needed`
  counts the bytes a resumed copy already staged once, and `where` names the
  filesystem the probe measured (`probe`'s new `root`).
- The export container is one per image and machine
  (`layerfs-export-<machine>-<hex>`), so two machines copying the same image
  neither share nor remove each other's container.
- `Target` carries the home and the machine whose cache it fills
  (`Target::local`, `Target::worker`), so an in-process Linux load takes the
  same lock as a worker load.
- `image_artifact`: `probe` drops `receiving` (the 30 s mtime test) and adds
  `root`.

Tests: `tests/lp176_image_lock.rs` (a second copy of the same target waits
and then reuses, another machine's copy is not locked out, a dead owner is
taken over and a reused pid holds nothing, a short machine is refused with no
effect, a resumed copy needs only what it has not staged, an in-process copy
takes the same lock) and `tests/image_copy.rs` moved to
`Capacity`/`needed`/`where` and to the probe's `root`.

Production LOC: 169,640 -> 169,779 (delta +139)
  core     39,795 -> 39,795 (0)
  engine   75,102 -> 75,102 (0)
  control  48,397 -> 48,536 (+139)  layerfs-sandbox +139
  api       6,346 ->  6,346 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --detail on 4a8efe6 and the committed tree
…ory mtime

Every image export previously used `layerfs-export-<machine>-<hex>` and swept
older `layerfs-export-*` containers by age (`STALE_SECS = 30 * 60`) with plain
`docker rm -f`: an in-flight failure could leave or collide on a predictable
name across homes or runs, an idle copy >30 minutes old risked having another
caller remove its container, `docker rm -f` leaked anonymous volumes on images
that declare `VOLUME`, and two `docker export` runs >=2 s apart differed on
containerd-snapshotter directory timestamps (`dev/`, `etc/`, `proc/`, `sys/`),
breaking resume across separate exports.

- `image_load`: replace age-based `STALE_SECS` / `remove_stale` with an
  exclusive per-export container named `layerfs-export-<home8>-<machine>-<nonce>`
  and labeled with `layerfs.role=image-export`, `layerfs.home`,
  `layerfs.machine`, `layerfs.image`, and `layerfs.export.nonce`. Guard both
  the container (`ExportContainer`) and streamed child (`ExportChild`) so
  error, cancel, or drop kills the export process and removes only that copy's
  container and its anonymous volumes (`docker rm -v`, with inspect +
  `docker volume rm` fallback when a container or stand-in still remains).
- Ownership enumeration and recovery: add `owned_containers` and
  `cleanup_owned` plus `ImageLock::try_take` so residual containers belonging
  to a home after a hard kill are listed and cleaned under I01's `ImageLock`
  (skipping active locked copies in `active` and leaving other homes
  untouched). Clean any stale container of the same `(home_id, machine, id)`
  under `ImageLock` before creating a new export container or probing sizes.
  Keep `docker.rs` untouched.
- `image_tar::Filtered`: normalize directory (`flag == b'5'`) `mtime` to
  `00000000000\0`, recompute ustar header checksums, strip `mtime`/`atime`/
  `ctime` from PAX `x` extended headers (dropping now-empty `x` blocks), and
  zero extended-header `mtime` (`x`/`L`/`K`) + checksum for directory entries so
  separate real `docker create` + `docker export` runs >=2 s apart yield
  byte-identical filtered tars and valid resume prefixes.

Tests: new `tests/lp176_image_export.rs` (7 tests covering PAX/directory mtime
normalization with real flat rootfs unpack, real Docker >=2 s identical SHA
+ prefix resume, `VOLUME` image anonymous volume cleanup, error + cancel
container cleanup, killed load recovery via `owned_containers` /
`cleanup_owned` under `ImageLock`, cross-home isolation, and pause helper);
I01 `tests/lp176_image_lock.rs` (7), `tests/image_copy.rs` (9), `tests/image_tar.rs`
(4) pass. Deleted tests: none.

Docs: updated `docs/architecture/api/sdk/machine.md` and
`docs/versioned/2.0.0/calls.md` (layerfs-machine / SDK `machine.load`).

Production LOC: 169,779 -> 170,091 (delta +312)
  core     39,795 -> 39,795 (0)
  engine   75,102 -> 75,102 (0)
  control  48,536 -> 48,848 (+312)  layerfs-sandbox +312
  api       6,346 ->  6,346 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --detail on 83843c3 and this tree
…up failures

- Wire cleanup_owned(docker, home, None, None) into load() under ImageLock after clearing the current (machine, image) residuals so killed exports from other images or machines in the same home are reclaimed without waiting on active locks.
- Make remove_container return whether the container and its anonymous volumes were actually removed (treating Missing as gone and checking container inspect + volume rm), add Cleanup.failed, and route failed removals to Cleanup.failed instead of Cleanup.removed.
- Add tests: real Docker M1 test verifying load() on a different image and machine reclaims a SIGKILLed container and its anonymous volume without touching a concurrent active load; S2 fake-docker tests for rm failure and volume rm failure; S4 byte-level image_tar tests for empty PAX blocks, malformed length fields, time-only PAX blocks dropped, and mixed time + SCHILY.xattr.* blocks.
- Update docs/architecture/api/sdk/machine.md and docs/versioned/2.0.0/calls.md to document epoch 0 (1970-01-01) directory mtime normalization, SIGKILL recovery on the next load in the same home, and manual docker rm for legacy layerfs-export-<machine>-<hex> containers.

Tests: cargo +1.85.1 fmt -p layerfs-sandbox -- --check, clippy --locked -p layerfs-sandbox --all-targets -- -D warnings, tests/lp176_image_export.rs (9), tests/lp176_image_lock.rs (7), tests/image_copy.rs (9), tests/image_tar.rs (5), tools/check_product_boundary.py, tools unittest (180) pass. Deleted tests: none.

Docs: updated docs/architecture/api/sdk/machine.md and docs/versioned/2.0.0/calls.md.

Production LOC: 170,091 -> 170,118 (delta +27; cumulative from 83843c3: +339)
  core     39,795 -> 39,795 (0)
  engine   75,102 -> 75,102 (0)
  control  48,848 -> 48,875 (+27)  layerfs-sandbox +27
  api       6,346 ->  6,346 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --detail on e6e05ea and this tree
…arse/hardlink safety (Ephemeral-AI-Lab#176 I03)

Production LOC:
- core:    39795 -> 39795 (+0)
- engine:  75102 -> 75102 (+0)
- control: 48875 -> 49834 (+959, layerfs-sandbox 13322 -> 14281)
- api:      6346 ->  6346 (+0)
- total:  170118 -> 171077 (+959)

Test plan: lp176_image_recovery, image_tar, image_copy, lp176_image_lock, lp176_image_export, fmt, clippy, check_product_boundary.py, tools unittests, lf-ci.sh fast.
…ocs (Ephemeral-AI-Lab#176 I03 R1)

- M1: add ImageLock::try_hold (flock without owner record), isolate per-hex lock probe/guard errors in images::prune/pruning, record skipped reasons, and fail only on targeted bad locks.
- N2: skip corrupt unrelated sidecars in image_artifact::strip_moved_refs instead of aborting publish.
- N5: enforce image_tar::SPARSE_MAX on sparse realsize and cumulative expanded bytes, and count streamed expanded bytes against image_load::capacity.
- N6 & sparse validation: cap oldgnu and PAX 0.x chunk counts by entry size while streaming, require payload_needed == size, and reject malformed PAX records after GNU.sparse.* keys with InvalidData.
- Docs & named(): hint machine.prune on ambiguous ref errors and document N1/N3/N4 known limitations in docs/architecture/api/sdk/machine.md (and skipped in calls.md).

Production LOC (tools/production_loc.py --detail):
- combined: 171,077 -> 171,232 (+155)
- core: 39,795 -> 39,795 (0)
- engine: 75,102 -> 75,102 (0)
- control: 49,834 -> 49,989 (+155; layerfs-sandbox: 14,281 -> 14,436, +155)
- api: 6,346 -> 6,346 (0)
- reference: absent (0)
…ndent chunk cap (Ephemeral-AI-Lab#176 I03 R2)

- R2-M1: add checked padded_size() returning InvalidData on u64 overflow for both ustar/base-256 and PAX sizes; ignore unparseable PAX size on non-sparse entries (matching a5616c8) while rejecting invalid PAX size on sparse entries.
- R2-M2: defer PAX b'x' parsing to PaxMeta::parse_pending after reading the entry header (reusing saved pending blocks), resolving effective entry size first (PAX size or ustar fallback) so push_chunk intercepts excess descriptors regardless of key order or missing PAX size.
- Condensed local InvalidData constructors with invalid() to stay within the 999-line file ceiling (996 lines).
- Tests: add r2_pax_size_overflow_plain_bad_size_and_order_independent_chunk_cap in lp176_image_recovery.rs.

Production LOC (tools/production_loc.py --detail):
- combined: 171,232 -> 171,240 (+8)
- core: 39,795 -> 39,795 (0)
- engine: 75,102 -> 75,102 (0)
- control: 49,989 -> 49,997 (+8; layerfs-sandbox: 14,436 -> 14,444, +8)
- api: 6,346 -> 6,346 (0)
- reference: absent (retired R9c; last counted 65,417 at tree 498dd19)
…extended headers (Ephemeral-AI-Lab#176 I03 R3)

- Split control/crates/layerfs-sandbox/src/image_tar.rs into image_tar/{mod,header,pax,sparse,filter}.rs, keeping public image_tar items intact and adding EXTENDED_TOTAL_MAX (16 MiB).
- Stage 1 (PendingHeaders::admit_header / push, PaxScan::scan_header): enforce 1 MiB per-header (EXTENDED_MAX) and 16 MiB per-entry cumulative (EXTENDED_TOTAL_MAX, accounting 512 + padded per extended header so zero-length header floods are bounded) before allocating payloads; validate size-independent PAX/sparse syntax (when GNU.sparse.* is present), sparse numbers, offset/numbytes pairing, and cross-header SPARSE_CHUNKS_MAX count on arrival without building a descriptor Vec.
- Stage 2 (resolve_entry_size, ensure_chunk_count_within, collect_sparse_chunks, in-place validate_sparse_chunks): resolve effective entry size (preserving plain bad-size fallback and checked 512B alignment), enforce min(size, SPARSE_CHUNKS_MAX) before materializing PAX 0.x chunks, and validate in place without a duplicate Vec.
- Stage 3 (clean EOF & zero end-blocks): bad sparse PAX headers already failed in Stage 1 while trailing plain extended headers retain a5616c8 parity.
- Tests: add consecutive_pax_x_interleaved_g_long_paths_and_dir_times_match_baseline (image_tar.rs) and r3_bad_sparse_eof_cross_header_caps_and_xattr_headroom (lp176_image_recovery.rs); update r2_pax_size_overflow_plain_bad_size_and_order_independent_chunk_cap 3a-3d sentinels to Stage-2 out-of-range chunk (32768, 1) past realsize=16384 while 3e asserts Stage-1 rejection on ,bad,bad and offset=bad.

Production LOC: 171,240 -> 171,386 (delta +146)
  core     39,795 -> 39,795 (0)
  engine   75,102 -> 75,102 (0)
  control  49,997 -> 50,143 (+146)  layerfs-sandbox +146
  api       6,346 ->  6,346 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --detail on parent 5d20e5b and the staged tree
…er cap to 48 MiB (Ephemeral-AI-Lab#176 I03 R4)

- R4-M1: keep Stage-1 immediate sparse syntax, numeric, pairing, and SPARSE_CHUNKS_MAX checks in PaxScan::scan_header while recording pax_size/pax_size_bad without finalizing per header so later x headers can override earlier bad/overflow size; add PendingHeaders::finish_at_end() invoked on both Stage-3 clean EOF and zero end-block paths so trailing sparse bad/overflow size still returns InvalidData.
- R4-M2: raise EXTENDED_TOTAL_MAX from 16 MiB to 48 MiB (covering worst-case decimal encoding of SPARSE_CHUNKS_MAX = 1,048,576 descriptors at 42 bytes/descriptor ~= 44 MB plus per-header framing/xattrs) while preserving 512 + padded accounting.
- Tests: add r4_m1_cross_header_pax_size_override_and_final_validation and r4_m2_long_digit_sparse_map_above_16mib_succeeds in lp176_image_recovery.rs; adjust r3_bad_sparse_eof_cross_header_caps_and_xattr_headroom flood bounds to 48 MiB.

Production LOC: 171,386 -> 171,391 (delta +5)
  core     39,795 -> 39,795 (0)
  engine   75,102 -> 75,102 (0)
  control  50,143 -> 50,148 (+5)  layerfs-sandbox +5
  api       6,346 ->  6,346 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --detail on parent f19ae10 and the staged tree
…175-p1

Merge upstream/layerfs_v2_0_0 (caff58a) into
pr/lp176-175-p1 (e390334, Ephemeral-AI-Lab#175 part 1 I01-I03).

All five overlapping files (control/crates/layerfs-sandbox/src/route.rs,
docs/architecture/api/sdk/machine.md, docs/architecture/api/sdk/sandbox.md,
docs/architecture/control/home/README.md, docs/versioned/2.0.0/calls.md)
auto-merged cleanly without textual conflicts, preserving upstream's 318
commits (Docker::of machine tools/socket lookup, GC/store.update/home/monitoring
updates) alongside I01-I03 (ImageLock, pre-effect Capacity check, exclusive
export container cleanup, deterministic tar normalization, and tar/PAX/sparse
safety).

Docs: updated docs/architecture/api/sdk/machine.md,
docs/architecture/api/sdk/sandbox.md, docs/architecture/control/home/README.md,
and docs/versioned/2.0.0/calls.md.

Production LOC: 171,391 -> 190,283 (delta +18,892)
  core       39,795 ->  40,589 (+794)
  engine     75,102 ->  90,180 (+15,078)
  control    50,148 ->  52,992 (+2,844)
  api         6,346 ->   6,522 (+176)
  reference  absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --detail on e390334 and merged tree
Copilot AI balanced review requested due to automatic review settings November 18, 2026 10:59

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Prune-lock races can delete in-use image artifacts, while unbounded tar-filter bookkeeping can exhaust the importing process’s memory.

6 open issues
What changed in this PR

This first installment improves Docker image imports for Sandbox machines in the control layer, focusing on concurrent loads, repeatable exports, and recovery after interruption.

Changes:

  • Adds per-image locking, pre-copy capacity checks, and independently owned export containers.
  • Splits tar processing into focused modules, normalizes directory timestamps, and adds sparse-file handling and parsing bounds.
  • Extends cache listing, recovery, pruning, and metadata handling, with supporting tests and documentation.
File Description
docs/​versioned/​2.0.0/​calls.md Updates image-loading, listing, pruning, and capacity contracts.
docs/​architecture/​control/​home/​README.md Documents image-lock files and owner records.
docs/​architecture/​api/​sdk/​sandbox.md Updates image-copy capacity errors.
docs/​architecture/​api/​sdk/​machine.md Documents import recovery, metadata, pruning, and limitations.
control/​crates/​layerfs-sandbox/​tests/​lp176_image_lock.rs Adds concurrency, lock takeover, and capacity tests.
control/​crates/​layerfs-sandbox/​tests/​lp176_image_export.rs Adds deterministic-export and container-cleanup tests.
control/​crates/​layerfs-sandbox/​tests/​image_tar.rs Adds PAX and directory-timestamp normalization cases.
control/​crates/​layerfs-sandbox/​tests/​image_copy.rs Adapts copy tests to locking and capacity changes.
control/​crates/​layerfs-sandbox/​src/​route.rs Supplies home and machine context to image loading.
control/​crates/​layerfs-sandbox/​src/​images.rs Lists incomplete artifacts and expands guarded pruning.
control/​crates/​layerfs-sandbox/​src/​image_tar/​sparse.rs Parses sparse maps and streams expanded file data.
control/​crates/​layerfs-sandbox/​src/​image_tar/​pax.rs Handles extended headers, limits, and metadata rewriting.
control/​crates/​layerfs-sandbox/​src/​image_tar/​mod.rs Defines the reorganized tar module’s public interface.
control/​crates/​layerfs-sandbox/​src/​image_tar/​header.rs Extracts tar header, size, and checksum helpers.
control/​crates/​layerfs-sandbox/​src/​image_tar/​filter.rs Implements filtering, sparse expansion, and timestamp normalization.
control/​crates/​layerfs-sandbox/​src/​image_tar.rs Removes the monolithic implementation replaced by submodules.
control/​crates/​layerfs-sandbox/​src/​image_load.rs Adds import locks, export ownership, recovery, and capacity accounting.
control/​crates/​layerfs-sandbox/​src/​image_artifact.rs Adds sidecar-based publication recovery and reference maintenance.

Review effort: Balanced

} else {
self.report.init.push(norm.clone());
}
self.dropped_paths.insert(norm);
Comment on lines +498 to +501
let _local_guard = if target.is_none()
&& matches!(side, Side::Home(_))
&& !machine.is_empty()
&& ImageLock::dir(root, machine).is_dir()
Comment on lines +498 to +502
let _local_guard = if target.is_none()
&& matches!(side, Side::Home(_))
&& !machine.is_empty()
&& ImageLock::dir(root, machine).is_dir()
{
Comment on lines +589 to +592
let variant = args
.get_text("variant")
.map(str::to_owned)
.or_else(|| old.as_ref().and_then(|s| s.variant.clone()));
Comment on lines +231 to +233
if held(&probe) {
let size = kept(&probe).or(Some(inspected.size));
return Ok(reused(id, refs.clone(), size));
// Reclaim any other unheld residual export containers in this home
// (such as killed loads for other images or machines) without waiting
// on another copy's lock.
let _ = cleanup_owned(docker, target.home(), None, None);
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants