Repository navigation
sandbox: image import locking, deterministic export and interrupted-load recovery (#175 part 1) - #738
Open
jiangkoumo wants to merge 9 commits into
Open
jiangkoumo wants to merge 9 commits into
jiangkoumo wants to merge 9 commits into
Conversation
…ecked before anything exists
A copy of an image resolved its Docker Id and then raced every other copy of
that image: "another copy is running" meant only that the staged part had
grown in the last 30 s (`image_artifact::RECEIVING`), a waiter went ahead
anyway after 10 minutes (`image_load::WAIT`), and both copies shared the
export container `layerfs-export-<hex>` and the part `.recv-<hex>.part`, so a
second `write` at offset 0 truncated the first one's bytes.
- `ImageLock`: one exclusive `flock` per home, machine and image, at
`<home>/run/image-lock/<machine>/<hex>.lock`, held for the whole copy, with
the owner recorded beside it (`pid`, `pid_start`, `pid_boot`, machine,
image, since). The lock is the exclusion and the record is only what a
waiter reports: the kernel releases a dead owner's lock, so the next copy
takes it over and resumes the part, and a record whose pid was reused holds
nothing. No deadline: a live owner is waited for, with
`{phase: "lock", waiting: "lock", holder}` each second.
- Capacity runs before the first effect: the check now precedes the export
container, so a refusal leaves no container, no staged byte and no cache
folder; the container's `SizeRootFs` only refines the progress estimate.
`NoSpace {need, free}` becomes `Capacity {needed, free, where}`; `needed`
counts the bytes a resumed copy already staged once, and `where` names the
filesystem the probe measured (`probe`'s new `root`).
- The export container is one per image and machine
(`layerfs-export-<machine>-<hex>`), so two machines copying the same image
neither share nor remove each other's container.
- `Target` carries the home and the machine whose cache it fills
(`Target::local`, `Target::worker`), so an in-process Linux load takes the
same lock as a worker load.
- `image_artifact`: `probe` drops `receiving` (the 30 s mtime test) and adds
`root`.
Tests: `tests/lp176_image_lock.rs` (a second copy of the same target waits
and then reuses, another machine's copy is not locked out, a dead owner is
taken over and a reused pid holds nothing, a short machine is refused with no
effect, a resumed copy needs only what it has not staged, an in-process copy
takes the same lock) and `tests/image_copy.rs` moved to
`Capacity`/`needed`/`where` and to the probe's `root`.
Production LOC: 169,640 -> 169,779 (delta +139)
core 39,795 -> 39,795 (0)
engine 75,102 -> 75,102 (0)
control 48,397 -> 48,536 (+139) layerfs-sandbox +139
api 6,346 -> 6,346 (0)
reference absent (retired R9c; last counted 65,417 at tree 498dd19)
Method: tools/production_loc.py --detail on 4a8efe6 and the committed tree
…ory mtime Every image export previously used `layerfs-export-<machine>-<hex>` and swept older `layerfs-export-*` containers by age (`STALE_SECS = 30 * 60`) with plain `docker rm -f`: an in-flight failure could leave or collide on a predictable name across homes or runs, an idle copy >30 minutes old risked having another caller remove its container, `docker rm -f` leaked anonymous volumes on images that declare `VOLUME`, and two `docker export` runs >=2 s apart differed on containerd-snapshotter directory timestamps (`dev/`, `etc/`, `proc/`, `sys/`), breaking resume across separate exports. - `image_load`: replace age-based `STALE_SECS` / `remove_stale` with an exclusive per-export container named `layerfs-export-<home8>-<machine>-<nonce>` and labeled with `layerfs.role=image-export`, `layerfs.home`, `layerfs.machine`, `layerfs.image`, and `layerfs.export.nonce`. Guard both the container (`ExportContainer`) and streamed child (`ExportChild`) so error, cancel, or drop kills the export process and removes only that copy's container and its anonymous volumes (`docker rm -v`, with inspect + `docker volume rm` fallback when a container or stand-in still remains). - Ownership enumeration and recovery: add `owned_containers` and `cleanup_owned` plus `ImageLock::try_take` so residual containers belonging to a home after a hard kill are listed and cleaned under I01's `ImageLock` (skipping active locked copies in `active` and leaving other homes untouched). Clean any stale container of the same `(home_id, machine, id)` under `ImageLock` before creating a new export container or probing sizes. Keep `docker.rs` untouched. - `image_tar::Filtered`: normalize directory (`flag == b'5'`) `mtime` to `00000000000\0`, recompute ustar header checksums, strip `mtime`/`atime`/ `ctime` from PAX `x` extended headers (dropping now-empty `x` blocks), and zero extended-header `mtime` (`x`/`L`/`K`) + checksum for directory entries so separate real `docker create` + `docker export` runs >=2 s apart yield byte-identical filtered tars and valid resume prefixes. Tests: new `tests/lp176_image_export.rs` (7 tests covering PAX/directory mtime normalization with real flat rootfs unpack, real Docker >=2 s identical SHA + prefix resume, `VOLUME` image anonymous volume cleanup, error + cancel container cleanup, killed load recovery via `owned_containers` / `cleanup_owned` under `ImageLock`, cross-home isolation, and pause helper); I01 `tests/lp176_image_lock.rs` (7), `tests/image_copy.rs` (9), `tests/image_tar.rs` (4) pass. Deleted tests: none. Docs: updated `docs/architecture/api/sdk/machine.md` and `docs/versioned/2.0.0/calls.md` (layerfs-machine / SDK `machine.load`). Production LOC: 169,779 -> 170,091 (delta +312) core 39,795 -> 39,795 (0) engine 75,102 -> 75,102 (0) control 48,536 -> 48,848 (+312) layerfs-sandbox +312 api 6,346 -> 6,346 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd19) Method: tools/production_loc.py --detail on 83843c3 and this tree
…up failures - Wire cleanup_owned(docker, home, None, None) into load() under ImageLock after clearing the current (machine, image) residuals so killed exports from other images or machines in the same home are reclaimed without waiting on active locks. - Make remove_container return whether the container and its anonymous volumes were actually removed (treating Missing as gone and checking container inspect + volume rm), add Cleanup.failed, and route failed removals to Cleanup.failed instead of Cleanup.removed. - Add tests: real Docker M1 test verifying load() on a different image and machine reclaims a SIGKILLed container and its anonymous volume without touching a concurrent active load; S2 fake-docker tests for rm failure and volume rm failure; S4 byte-level image_tar tests for empty PAX blocks, malformed length fields, time-only PAX blocks dropped, and mixed time + SCHILY.xattr.* blocks. - Update docs/architecture/api/sdk/machine.md and docs/versioned/2.0.0/calls.md to document epoch 0 (1970-01-01) directory mtime normalization, SIGKILL recovery on the next load in the same home, and manual docker rm for legacy layerfs-export-<machine>-<hex> containers. Tests: cargo +1.85.1 fmt -p layerfs-sandbox -- --check, clippy --locked -p layerfs-sandbox --all-targets -- -D warnings, tests/lp176_image_export.rs (9), tests/lp176_image_lock.rs (7), tests/image_copy.rs (9), tests/image_tar.rs (5), tools/check_product_boundary.py, tools unittest (180) pass. Deleted tests: none. Docs: updated docs/architecture/api/sdk/machine.md and docs/versioned/2.0.0/calls.md. Production LOC: 170,091 -> 170,118 (delta +27; cumulative from 83843c3: +339) core 39,795 -> 39,795 (0) engine 75,102 -> 75,102 (0) control 48,848 -> 48,875 (+27) layerfs-sandbox +27 api 6,346 -> 6,346 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd19) Method: tools/production_loc.py --detail on e6e05ea and this tree
…arse/hardlink safety (Ephemeral-AI-Lab#176 I03) Production LOC: - core: 39795 -> 39795 (+0) - engine: 75102 -> 75102 (+0) - control: 48875 -> 49834 (+959, layerfs-sandbox 13322 -> 14281) - api: 6346 -> 6346 (+0) - total: 170118 -> 171077 (+959) Test plan: lp176_image_recovery, image_tar, image_copy, lp176_image_lock, lp176_image_export, fmt, clippy, check_product_boundary.py, tools unittests, lf-ci.sh fast.
…ocs (Ephemeral-AI-Lab#176 I03 R1) - M1: add ImageLock::try_hold (flock without owner record), isolate per-hex lock probe/guard errors in images::prune/pruning, record skipped reasons, and fail only on targeted bad locks. - N2: skip corrupt unrelated sidecars in image_artifact::strip_moved_refs instead of aborting publish. - N5: enforce image_tar::SPARSE_MAX on sparse realsize and cumulative expanded bytes, and count streamed expanded bytes against image_load::capacity. - N6 & sparse validation: cap oldgnu and PAX 0.x chunk counts by entry size while streaming, require payload_needed == size, and reject malformed PAX records after GNU.sparse.* keys with InvalidData. - Docs & named(): hint machine.prune on ambiguous ref errors and document N1/N3/N4 known limitations in docs/architecture/api/sdk/machine.md (and skipped in calls.md). Production LOC (tools/production_loc.py --detail): - combined: 171,077 -> 171,232 (+155) - core: 39,795 -> 39,795 (0) - engine: 75,102 -> 75,102 (0) - control: 49,834 -> 49,989 (+155; layerfs-sandbox: 14,281 -> 14,436, +155) - api: 6,346 -> 6,346 (0) - reference: absent (0)
…ndent chunk cap (Ephemeral-AI-Lab#176 I03 R2) - R2-M1: add checked padded_size() returning InvalidData on u64 overflow for both ustar/base-256 and PAX sizes; ignore unparseable PAX size on non-sparse entries (matching a5616c8) while rejecting invalid PAX size on sparse entries. - R2-M2: defer PAX b'x' parsing to PaxMeta::parse_pending after reading the entry header (reusing saved pending blocks), resolving effective entry size first (PAX size or ustar fallback) so push_chunk intercepts excess descriptors regardless of key order or missing PAX size. - Condensed local InvalidData constructors with invalid() to stay within the 999-line file ceiling (996 lines). - Tests: add r2_pax_size_overflow_plain_bad_size_and_order_independent_chunk_cap in lp176_image_recovery.rs. Production LOC (tools/production_loc.py --detail): - combined: 171,232 -> 171,240 (+8) - core: 39,795 -> 39,795 (0) - engine: 75,102 -> 75,102 (0) - control: 49,989 -> 49,997 (+8; layerfs-sandbox: 14,436 -> 14,444, +8) - api: 6,346 -> 6,346 (0) - reference: absent (retired R9c; last counted 65,417 at tree 498dd19)
…extended headers (Ephemeral-AI-Lab#176 I03 R3) - Split control/crates/layerfs-sandbox/src/image_tar.rs into image_tar/{mod,header,pax,sparse,filter}.rs, keeping public image_tar items intact and adding EXTENDED_TOTAL_MAX (16 MiB). - Stage 1 (PendingHeaders::admit_header / push, PaxScan::scan_header): enforce 1 MiB per-header (EXTENDED_MAX) and 16 MiB per-entry cumulative (EXTENDED_TOTAL_MAX, accounting 512 + padded per extended header so zero-length header floods are bounded) before allocating payloads; validate size-independent PAX/sparse syntax (when GNU.sparse.* is present), sparse numbers, offset/numbytes pairing, and cross-header SPARSE_CHUNKS_MAX count on arrival without building a descriptor Vec. - Stage 2 (resolve_entry_size, ensure_chunk_count_within, collect_sparse_chunks, in-place validate_sparse_chunks): resolve effective entry size (preserving plain bad-size fallback and checked 512B alignment), enforce min(size, SPARSE_CHUNKS_MAX) before materializing PAX 0.x chunks, and validate in place without a duplicate Vec. - Stage 3 (clean EOF & zero end-blocks): bad sparse PAX headers already failed in Stage 1 while trailing plain extended headers retain a5616c8 parity. - Tests: add consecutive_pax_x_interleaved_g_long_paths_and_dir_times_match_baseline (image_tar.rs) and r3_bad_sparse_eof_cross_header_caps_and_xattr_headroom (lp176_image_recovery.rs); update r2_pax_size_overflow_plain_bad_size_and_order_independent_chunk_cap 3a-3d sentinels to Stage-2 out-of-range chunk (32768, 1) past realsize=16384 while 3e asserts Stage-1 rejection on ,bad,bad and offset=bad. Production LOC: 171,240 -> 171,386 (delta +146) core 39,795 -> 39,795 (0) engine 75,102 -> 75,102 (0) control 49,997 -> 50,143 (+146) layerfs-sandbox +146 api 6,346 -> 6,346 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd19) Method: tools/production_loc.py --detail on parent 5d20e5b and the staged tree
…er cap to 48 MiB (Ephemeral-AI-Lab#176 I03 R4) - R4-M1: keep Stage-1 immediate sparse syntax, numeric, pairing, and SPARSE_CHUNKS_MAX checks in PaxScan::scan_header while recording pax_size/pax_size_bad without finalizing per header so later x headers can override earlier bad/overflow size; add PendingHeaders::finish_at_end() invoked on both Stage-3 clean EOF and zero end-block paths so trailing sparse bad/overflow size still returns InvalidData. - R4-M2: raise EXTENDED_TOTAL_MAX from 16 MiB to 48 MiB (covering worst-case decimal encoding of SPARSE_CHUNKS_MAX = 1,048,576 descriptors at 42 bytes/descriptor ~= 44 MB plus per-header framing/xattrs) while preserving 512 + padded accounting. - Tests: add r4_m1_cross_header_pax_size_override_and_final_validation and r4_m2_long_digit_sparse_map_above_16mib_succeeds in lp176_image_recovery.rs; adjust r3_bad_sparse_eof_cross_header_caps_and_xattr_headroom flood bounds to 48 MiB. Production LOC: 171,386 -> 171,391 (delta +5) core 39,795 -> 39,795 (0) engine 75,102 -> 75,102 (0) control 50,143 -> 50,148 (+5) layerfs-sandbox +5 api 6,346 -> 6,346 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd19) Method: tools/production_loc.py --detail on parent f19ae10 and the staged tree
…175-p1 Merge upstream/layerfs_v2_0_0 (caff58a) into pr/lp176-175-p1 (e390334, Ephemeral-AI-Lab#175 part 1 I01-I03). All five overlapping files (control/crates/layerfs-sandbox/src/route.rs, docs/architecture/api/sdk/machine.md, docs/architecture/api/sdk/sandbox.md, docs/architecture/control/home/README.md, docs/versioned/2.0.0/calls.md) auto-merged cleanly without textual conflicts, preserving upstream's 318 commits (Docker::of machine tools/socket lookup, GC/store.update/home/monitoring updates) alongside I01-I03 (ImageLock, pre-effect Capacity check, exclusive export container cleanup, deterministic tar normalization, and tar/PAX/sparse safety). Docs: updated docs/architecture/api/sdk/machine.md, docs/architecture/api/sdk/sandbox.md, docs/architecture/control/home/README.md, and docs/versioned/2.0.0/calls.md. Production LOC: 171,391 -> 190,283 (delta +18,892) core 39,795 -> 40,589 (+794) engine 75,102 -> 90,180 (+15,078) control 50,148 -> 52,992 (+2,844) api 6,346 -> 6,522 (+176) reference absent (retired R9c; last counted 65,417 at tree 498dd19) Method: tools/production_loc.py --detail on e390334 and merged tree
There was a problem hiding this comment.
🟡 Changes recommended
Prune-lock races can delete in-use image artifacts, while unbounded tar-filter bookkeeping can exhaust the importing process’s memory.
6 open issues
Export filtering retains unbounded dropped-entry state in memory · New Worker pruning can delete artifacts during an active load · New Prune can skip locking when the image-lock directory is absent · New Normal imports omit Docker's image variant metadata · New Cached-image reuse skips recovery of orphaned export containers · New Load reports success despite container cleanup failures · New
What changed in this PR
This first installment improves Docker image imports for Sandbox machines in the control layer, focusing on concurrent loads, repeatable exports, and recovery after interruption.
Changes:
- Adds per-image locking, pre-copy capacity checks, and independently owned export containers.
- Splits tar processing into focused modules, normalizes directory timestamps, and adds sparse-file handling and parsing bounds.
- Extends cache listing, recovery, pruning, and metadata handling, with supporting tests and documentation.
| File | Description |
|---|---|
docs/versioned/2.0.0/calls.md |
Updates image-loading, listing, pruning, and capacity contracts. |
docs/architecture/control/home/README.md |
Documents image-lock files and owner records. |
docs/architecture/api/sdk/sandbox.md |
Updates image-copy capacity errors. |
docs/architecture/api/sdk/machine.md |
Documents import recovery, metadata, pruning, and limitations. |
control/crates/layerfs-sandbox/tests/lp176_image_lock.rs |
Adds concurrency, lock takeover, and capacity tests. |
control/crates/layerfs-sandbox/tests/lp176_image_export.rs |
Adds deterministic-export and container-cleanup tests. |
control/crates/layerfs-sandbox/tests/image_tar.rs |
Adds PAX and directory-timestamp normalization cases. |
control/crates/layerfs-sandbox/tests/image_copy.rs |
Adapts copy tests to locking and capacity changes. |
control/crates/layerfs-sandbox/src/route.rs |
Supplies home and machine context to image loading. |
control/crates/layerfs-sandbox/src/images.rs |
Lists incomplete artifacts and expands guarded pruning. |
control/crates/layerfs-sandbox/src/image_tar/sparse.rs |
Parses sparse maps and streams expanded file data. |
control/crates/layerfs-sandbox/src/image_tar/pax.rs |
Handles extended headers, limits, and metadata rewriting. |
control/crates/layerfs-sandbox/src/image_tar/mod.rs |
Defines the reorganized tar module’s public interface. |
control/crates/layerfs-sandbox/src/image_tar/header.rs |
Extracts tar header, size, and checksum helpers. |
control/crates/layerfs-sandbox/src/image_tar/filter.rs |
Implements filtering, sparse expansion, and timestamp normalization. |
control/crates/layerfs-sandbox/src/image_tar.rs |
Removes the monolithic implementation replaced by submodules. |
control/crates/layerfs-sandbox/src/image_load.rs |
Adds import locks, export ownership, recovery, and capacity accounting. |
control/crates/layerfs-sandbox/src/image_artifact.rs |
Adds sidecar-based publication recovery and reference maintenance. |
Review effort: Balanced
| } else { | ||
| self.report.init.push(norm.clone()); | ||
| } | ||
| self.dropped_paths.insert(norm); |
Comment on lines
+498
to
+501
| let _local_guard = if target.is_none() | ||
| && matches!(side, Side::Home(_)) | ||
| && !machine.is_empty() | ||
| && ImageLock::dir(root, machine).is_dir() |
Comment on lines
+498
to
+502
| let _local_guard = if target.is_none() | ||
| && matches!(side, Side::Home(_)) | ||
| && !machine.is_empty() | ||
| && ImageLock::dir(root, machine).is_dir() | ||
| { |
Comment on lines
+589
to
+592
| let variant = args | ||
| .get_text("variant") | ||
| .map(str::to_owned) | ||
| .or_else(|| old.as_ref().and_then(|s| s.variant.clone())); |
Comment on lines
+231
to
+233
| if held(&probe) { | ||
| let size = kept(&probe).or(Some(inspected.size)); | ||
| return Ok(reused(id, refs.clone(), size)); |
| // Reclaim any other unheld residual export containers in this home | ||
| // (such as killed loads for other images or machines) without waiting | ||
| // on another copy's lock. | ||
| let _ = cleanup_owned(docker, target.home(), None, None); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Tier: P1 (control/crates/layerfs-sandbox) ; risks: risk:delete, risk:security ; gate: yes: reclaims this home's leftover export containers and removes stranded unpacked image dirs under the image lock; parses untrusted tar PAX/sparse headers with bounds ; de-escalated: none
First part of Ephemeral-AI-Lab/layerfs-project#175 (Docker image → machine import reliability). This PR does not close #175; the remaining pieces come in follow-ups on top of this branch.
What changes:
machine.loadtakes a cross-processImageLockper image, so concurrent loads of the same image export once and the rest reuse it; a lock held by a dead process can be taken over. Free space is checked before anything is created.loadreclaims this home's leftover export containers and reports cleanup failures.machine.imageslists staged partial downloads (size, age,resumable) and stranded unpacked directories without a sidecar;machine.pruneonly removes what no live lock or Sandbox holds. The sidecar is the publication commit point:probereports an un-sidecared directory asstrandedread-only, andpublishremoves and re-unpacks it under the lock. Moving a tag strips the stale ref from older sidecars; a ref that matches several cached images is refused (InvalidInput, suggestingmachine.prune). Unpack reports dropped FIFOs, devices and init files and sparse expansion bytes; a hardlink to a dropped target is an error.variant/configare recorded, not applied.image_tar/{header,pax,sparse,filter}.rs, with extended headers capped at 48 MiB.docs/architecture/api/sdk/machine.md(incl. two known limitations of ref maintenance),sandbox.md,control/home/README.md,docs/versioned/2.0.0/calls.md.Tests:
lp176_image_lock,lp176_image_export,lp176_image_recovery, plusimage_tarandimage_copyupdates: 47 pass (real Docker: two exports byte-identical, leftovers reclaimed after kill);layerfs_v2_0_0caff58a): fmt/clippy clean, slow 171/171, mount 45/45, PostgreSQL packages pass; compared with a cleanlayerfs_v2_0_0caff58a run, no new failures;loads of one image exported once, the other 3 reused it, no leftovers.Known, not from this PR: on caff58a the fast lane does not compile (
sqlite_compact_capacity.rs:75:11, E0004, from #732), so CI here will be red until that is fixed upstream; with that test excluded, the remaining fast failures match the base. Docker's classic image store was not exercised (test host uses the containerd store).