Skip to content

nfs: spill AppleDouble companion bodies to a per-mount scratch area (lp73) - #714

Merged
21 commits merged into
Ephemeral-AI-Lab:layerfs_v2_0_0from
jiangkoumo:lp636-spill
Nov 17, 2026
Merged

21 commits merged into
Ephemeral-AI-Lab:layerfs_v2_0_0from
jiangkoumo:lp636-spill

Conversation

@jiangkoumo

Copy link
Copy Markdown
Collaborator

Tier: P1 (engine/crates/layerfs-nfs, engine/crates/layerfs-daemon) ; risks: risk:kernel, risk:deps, risk:security, risk:delete ; gate: yes: adds a dev-dependency, deletes a per-mount scratch area ; de-escalated: none
Docs: docs/architecture/engine/kernel-routes/nfs.md, docs/architecture/engine/daemon/README.md, docs/general/cleanup.md
Durable: NOT_RUN — disabled by owner until explicit reauthorization

Second of three PRs for #636 (AppleDouble companions beyond 4,096 per export).
It is stacked on the first PR, #713 (lp636-codec, head ff28f6118): the branch
merges that layer, so this diff contains the codec, and it has to be merged
after #713 for the codec. The third PR (the name index) is not part of this one.

Summary

The codec layer keeps an AppleDouble body in memory and refuses an edit that
does not fit the resident budget. This layer gives a body somewhere to go.

  • When the resident budget is full, the least recently used body — decoded
    images included, they share the same RESIDENT_BYTES (32 MiB) pool — is
    written to <mounts>/.appledouble/<namespace>/<serial>. A body that cannot
    fit at all is written out whole and then edited on its file, so growing a
    companion answers success instead of NFS3ERR_NOSPC.
  • Eviction order is Companions::by_use (BTreeSet<(used tick, serial)>): one
    lookup under the lock, not a scan of every companion, and no fixed round
    count. Two commits here are prefixed perf(nfs) because of that shape — an
    order instead of a scan, and moving the adoption rename out of the lock;
    nothing was timed, and they are not claimed as measured speedups.
  • No file I/O under the companion lock: the create_new temporary file and the
    fs::rename run outside it, one write per companion at a time (the edit gate;
    eviction uses try_lock and skips a busy companion). A reader keeps reading
    the previous body in memory until the swap is committed, so it never sees a
    half-written file. If a victim cannot be written out (removed concurrently, or
    its file fails), only that victim is skipped — the error does not reach the
    writer that merely needed room. The writer gets an error only when nothing
    could be freed and its own body cannot be written out either.
  • READ is one pread: the length is read under the lock and the bytes outside
    it, so a concurrent truncation gives a short read instead of a spurious
    NFS3ERR_IO; an interrupted pread is asked once more (nothing had moved).
    A disk edit that fails after the body was written out restores the length the
    companion had, so a refused write does not leave a longer body behind.
  • <mounts>/.appledouble is the per-mount disposable area the owner approved:
    0700 directories, 0600 files, no fsync, no space reserved ahead. Release
    and the daemon's startup reclaim delete it, and reclaim touches nothing else
    in the mount home (a test keeps a neighbouring file and directory alive
    through a reclaim). Both the architecture pages and the cleanup page say so.
  • libc enters as a dev-dependency only (the RLIMIT_NOFILE test), which is
    the risk:deps; tests/write.rs calls getrlimit/setrlimit
    (risk:security); release and startup call
    remove_dir_all(<mounts>/.appledouble) (risk:delete).
  • The branch was brought up to layerfs_v2_0_0 at cb8ced9d3 through the
    first layer's merge (ff28f6118) and then merged here as e3c80ed1c; neither
    merge had a conflict, and neither changed a file of this feature.

Validation

At e3c80ed1c, on Linux, with lf-ci.sh (upstream fast lane, then the nightly
slow and mount lanes) and lf-ci-pg.sh. Cargo commands were queued through the
worktree's build lock; each test invocation had a 120 s wall timeout and none
reached it.

  • fast lane rc=0: 1994 tests run, 1994 passed, 26 skipped.
  • slow lane rc=0: 150 tests run, 150 passed, 11 skipped.
  • mount lane: 45 test binaries passed, 0 failed.
  • PostgreSQL lane: six packages all rc=0; 39 tests passed; NOT_RUN=0.
  • cargo +1.85.1 test --locked -p layerfs-nfs: 51 passed, 0 failed (write 11,
    apple_double_decode 6, apple_double_codec 7, daemon 2, export 5,
    protocol 15, collect 1, helper_deadline 3, refresh 1; the raw
    test result lines sum to 52 because
    spilled_companions_outnumber_the_open_file_limit re-runs itself with
    --exact in a child process and that line is counted once more).
  • cargo +1.85.1 clippy --locked -p layerfs-nfs -p layerfs-daemon --all-targets -- -D warnings: rc=0, no warnings. rustfmt --check on every changed file:
    clean. python3 -B tools/check_product_boundary.py: PASS (1379 files).
  • This PR adds 9 tests over the codec's head:
    an_eviction_victim_that_cannot_be_written_out_does_not_fail_the_writer,
    a_read_racing_a_truncation_is_a_short_read_not_an_io_error,
    eviction_takes_the_decoded_body_before_the_written_ones,
    eviction_follows_the_order_of_use_not_the_shape_of_the_index,
    a_spill_replaces_a_companion_body_only_when_no_call_changed_it,
    a_spilled_companion_keeps_its_length_and_bytes_together,
    a_decoded_companion_is_edited_under_a_full_resident_budget,
    spilled_companions_outnumber_the_open_file_limit,
    startup_reclaim_clears_leftover_companion_spill_directories.

macOS proof

Re-run at this head (e3c80ed1c, release build) after the upstream merge:
macOS 27.2 (arm64, M2) kernel NFSv3, same mount options as the first PR.

600 files carrying a ~100 KiB attribute each (~60 MiB of companion bodies,
~1.9× the 32 MiB resident budget) were copied into one mounted Workspace —
600/600 succeeded with no EPERM/NOSPC and no retry, 326 bodies were spilled
to .appledouble/<namespace> (29,784 KiB on disk) while release reported
held=652 decoded=326 resident_bytes=33465204 (31.91 MiB, under the 32 MiB
budget), and all 600 attributes and data read back byte-identical after 78 s
past the attribute cache. The identical recipe ran three times on three fresh
Sandboxes/mounts, the first on a freshly built, cold-started daemon:
600/600 identical every time, with the same spill count and the same release
line. Every unmount returned Unmounted and the .appledouble area was gone
afterwards. The first PR's 50-file read-back scenario also passed 3/3 here.

On the same Mac, cargo +1.85.1 clippy -p layerfs-nfs -p layerfs-daemon --all-targets -- -D warnings was clean and cargo +1.85.1 test -p layerfs-nfs
passed 54/54.

For the record, an earlier head answered 507 of those 600 files without their
attributes once, while the export still held all 652 companions; later reruns
and controls were correct, including the three above, so it was never
reproduced. The slotted=/unreachable= diagnostics added while chasing it
belong to the third PR and are not in this head's release line.

Risks

risk:kernel (the export's sidecar path), risk:delete (release and startup
delete <mounts>/.appledouble), risk:security (setrlimit in the file-limit
test), risk:deps (libc, dev-dependency only). The scratch area is
export-local and disposable: 0700/0600, no fsync, no space reserved ahead,
deleted at release and at startup, and reclaim removes only that directory.
Bodies that do not fit and cannot be written out still answer NFS3ERR_NOSPC
and keep their bytes. Not covered here: the name index and the 65,536-companion
limit (the third PR), and concurrent-editing pressure beyond what the tests
drive.

Production LOC

Base ff28f6118 (the first PR's head) = 163901 → head e3c80ed1c = 164405,
+504.

Scope: first-party core/engine/control/api production source and shipped
runtime SQL; tests, docs, tools and manifests excluded. Method:
git archive <rev> | tar -x -C <dir>, then
python3 -I tools/production_loc.py --root <dir> (tool SHA-256
d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6).

Related to #636; it does not close it.

….rs (lp73)

Relocation only, no behavior change: `Kernel::release`, `Kernel::held`,
`retained_count` and the `Released` struct move out of `kernel.rs` into
`kernel/released.rs` verbatim, and `pub use released::Released` keeps every
existing path working. This is the first commit of the AppleDouble split
(LP636): `kernel.rs` would otherwise cross the 999-line ceiling once the
companion work lands.

No test or documentation change; the moved functions are the same text in a
new module.

Production LOC: 163505 -> 163511 (delta +6)
Scope: first-party core/engine/control/api production source and shipped
runtime SQL; tests, docs, tools and manifests excluded. Method:
python3 tools/production_loc.py (SHA-256
 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6)
on git archives of the first parent 8cfb6e2 (163505) and this tree (163511).
The +6 is the new module's own two `use` lines, its `impl Kernel` block and the
`mod released;` / `pub use released::Released;` declaration in kernel.rs.
Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
The client lays a `._<name>` companion out as an AppleDouble v2 image: a
header, a FinderInfo entry that runs into an `ATTR` table of extended
attributes, and a resource-fork entry at the end. `apple_double_codec` reads
those fields (`Image::parse`) and, when the fields encode back to the very
bytes the client wrote (`Image::compact`), keeps them as its own record, so no
attribute, value or byte the layout carries is dropped, and one attribute's
alignment padding survives a shorter name replacing a longer one.

`Image::new`/`Image::encode` lay an image out as the client does, which is what
makes the round trip exact and what the macOS evidence compares against.
Malformed, truncated or differently laid-out images, and a resource fork with
data, parse to `None` and are left as the client wrote them.

This is the first layer of the AppleDouble split (LP636): the codec alone, with
its own unit tests; the companion that holds its decoded form comes next.

Tests: new `tests/apple_double_codec.rs`, 6 cases (byte-exact round trip for
every value size, many attributes, alignment padding, whole units, a resource
fork, and malformed images that must not decode), all pass.

Production LOC: 163511 -> 163792 (delta +281)
Scope: first-party core/engine/control/api production source and shipped
runtime SQL; tests, docs, tools and manifests excluded. Method:
python3 tools/production_loc.py (SHA-256
 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6)
on git archives of the first parent 1810b4e (163511) and this tree (163792).
Engine +281 (`apple_double_codec.rs` +280, `lib.rs` +1).
Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
A macOS client writes each companion once, in full, and reads it back
unchanged. A body is therefore classified whenever a write or a resize leaves
it: one that parses back to the very bytes the companion holds
(`Image::compact`) and fits `DECODED_ENTRY_BYTES` is kept as its FinderInfo and
attributes and counted against `DECODED_BYTES` instead of its image, and every
READ synthesizes the same bytes again. Anything else - a resource fork, another
layout, a half-written image - is stored as the client wrote it.

This replaces `TOTAL_BYTES` with two budgets that say what they bound:
`RESIDENT_BYTES` (32 MiB) of written bodies kept in memory, and `DECODED_BYTES`
(512 MiB) of decoded ones counted as `Image::cost`. A write or resize that
neither decodes nor fits the resident budget answers `NFS3ERR_NOSPC` and leaves
the companion exactly as it was; the next layer spills those bodies to disk
instead of refusing them.

`Released` now reports `companions_decoded` and `companion_decoded_bytes`, and
`companions_release`/`companions_held` answer the `Holding` record the counters
live in.

Second layer of the AppleDouble split (LP636), on top of the codec.

Tests: new `tests/apple_double_decode.rs` with the client's own write sequence
(a half-written header, the rest, an edit that breaks the image and one that
completes it again, truncate below and resize back up), read back byte for
byte. Full `layerfs-nfs` suite: 41 passed, 0 failed.

Production LOC: 163792 -> 163886 (delta +94)
Scope: first-party core/engine/control/api production source and shipped
runtime SQL; tests, docs, tools and manifests excluded. Method:
python3 tools/production_loc.py (SHA-256
 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6)
on git archives of the first parent 210d5f8 (163792) and this tree (163886).
Engine +94. Reference absent (retired R9c; last counted 65,417 at tree
498dd19).
The NFS route guide now names `apple_double_codec.rs` and states what each
bound actually holds: `RESIDENT_BYTES` for written bodies and `DECODED_BYTES` /
`DECODED_ENTRY_BYTES` for the decoded form, replacing the `TOTAL_BYTES` row the
code no longer has. The paragraph after the table says which bodies decode
(complete images that encode back to the same bytes) and which stay as written.

Documentation only; it describes the codec layer as it stands.

Production LOC: 163886 -> 163886 (delta 0)
Scope: first-party core/engine/control/api production source and shipped
runtime SQL; tests, docs, tools and manifests excluded. Method:
python3 tools/production_loc.py (SHA-256
 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6)
on git archives of the first parent fa5dd34 (163886) and this tree (163886).
Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
…get (lp73)

A client's companion body no longer has to fit in memory and no longer takes
memory from every other export. Bodies beyond `RESIDENT_BYTES` (32 MiB) are
spilled, least recently touched first, to a scratch directory of the export's
own (`<mounts>/.appledouble/<namespace>/<serial>`), and a write that still does
not fit the budget after bounded eviction rounds is written out itself and
edited in its file, which needs no budget at all.

- `apple_double_spill.rs` owns the scratch files: one per companion serial,
  directory `0700` and files `0600` whatever the umask, written through a
  `<serial>.<n>.tmp` that only the adopting call renames to `<serial>`, no
  fsync and no space reserved ahead. A spilled body is opened by path per
  `pread`/`pwrite`/`set_len` and never held open, so companions far outnumbering
  the process's file limit all work. A failed file operation is answered with
  the NFS status its errno means (`ENOSPC`/`EDQUOT`, `EFBIG`, `EACCES`,
  `EPERM`, `EROFS`, else `NFS3ERR_IO`).
- `apple_double/edit.rs` holds READ, WRITE and SETATTR and the eviction: one
  edit of a companion at a time under that companion's own gate, so a truncate
  and a write can never leave a length its file does not have, and the file
  I/O never holds the companion lock.
- `ExportConfig` gains `companion_spill_dir` and `companion_resident_bytes`.
  The daemon gives each export `<mounts>/.appledouble/<namespace>`, removes it
  at startup before any Attach, at release and when a failed Attach unwinds, and
  touches nothing else under the mount home.
- `Released` reports `companion_bytes`.

Third layer of the AppleDouble split (LP636), on top of the codec; `HELD` and
the name index are the layer after this one.

Production LOC: 163886 -> 164334 (delta +448)
Scope: first-party core/engine/control/api production source and shipped
runtime SQL; tests, docs, tools and manifests excluded. Method:
python3 tools/production_loc.py (SHA-256
 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6)
on git archives of the first parent 584cc3e (163886) and this tree (164334).
Engine +448, the daemon's `export.rs` share included. Tests for the layer
follow in the next commit. Reference absent (retired R9c; last counted 65,417
at tree 498dd19).
The rig can now set the resident budget and an explicit spill directory, and
the cases exercise the layer end to end:

- a body smaller than the budget stays resident and a larger one spills; the
  spilled file carries the bytes and the length the client wrote;
- an edit that leaves a decoded image not whole spills it instead of growing
  memory, while a truncate of a decoded body needs no room at all;
- a spill only replaces a companion no other call changed meanwhile
  (`spill_applies`);
- a truncate and a write of one spilled companion start together, round after
  round, and the length a reply reports is always the length its file holds;
- companions four times the process's own `RLIMIT_NOFILE` all read and write,
  because a spilled body is opened by path per operation: that case lowers a
  process-wide limit and counts the process's descriptors, so it re-runs itself
  in its own process (`run_alone`);
- Attach cleans a leftover `.appledouble` of a daemon that died, release and
  unmount remove it, and startup reclamation removes it while keeping every
  other entry under the mount home.

`libc` is a new dev-dependency of `layerfs-nfs`, only for the file-limit case.

Tests: `cargo test -p layerfs-nfs` 46 passed, 0 failed (the raw sum reads 47
because the file-limit case runs itself once in a child process).

Production LOC: 164334 -> 164334 (delta 0)
Scope: first-party core/engine/control/api production source and shipped
runtime SQL; tests, docs, tools and manifests excluded. Method:
python3 tools/production_loc.py (SHA-256
 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6)
on git archives of the first parent 7b38dad (164334) and this tree (164334).
Tests only. Reference absent (retired R9c; last counted 65,417 at tree
498dd19).
The guide names `edit.rs` and `apple_double_spill.rs`, describes what the
`RESIDENT_BYTES` row does now (spill to `companion_spill_dir`, and which status
each host errno maps to), and records the scratch file's own rules: one file
per companion serial under `<mounts>/.appledouble/<namespace>`, `0700`/`0600`,
written as a temporary and adopted by rename only when no call changed the
body, opened by path per operation and never held open, deleted by startup
reclamation and by release. `ExportConfig` lists the two new fields, and the
`wcc_data` row now says a companion mutation answers from `directory_stats`.

Documentation only; it describes the spill layer as it stands.

Production LOC: 164334 -> 164334 (delta 0)
Scope: first-party core/engine/control/api production source and shipped
runtime SQL; tests, docs, tools and manifests excluded. Method:
python3 tools/production_loc.py (SHA-256
 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6)
on git archives of the first parent 91434f9 (164334) and this tree (164334).
Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
…s for (lp73)

A decoded companion was re-encoded in full under the companion lock on every
READ: up to `FILE_BYTES` (1 MiB) allocated and copied to serve a reply of at
most `TRANSFER_WINDOW` (256 KiB) bytes, and often far less. `Image::encode_range`
lays the image out and fills only `[start, end)`, so a READ pays for the bytes
it answers with; `Image::encode` is that same layout over the whole image, so
there is one implementation of the layout rather than two that can drift.

`edit.rs`'s spill and `edit_resident` still need the whole image and keep
calling `encode()`. Whether a READ is served from a window or from the whole
image is not observable: the reply bytes are the same either way.

Review: finding 7 (full re-encode on every READ).

Tests: new `any_window_of_the_image_encodes_to_that_window` compares every
window of an image (a padding byte the client left behind, values of several
sizes, the empty resource fork) with the same slice of `encode()`, including
empty windows, a window past the image and one that starts after it ends.
`cargo test -p layerfs-nfs --test apple_double_codec --test apple_double_decode`:
7 + 1 passed, 0 failed. Anti-proof: with `encode_range` clamped to start at 0
again the window test fails (`left` starts at image offset 0 while the expected
window starts mid-image).

Production LOC: 163886 -> 163908 (delta +22)
Scope: first-party core/engine/control/api production source and shipped
runtime SQL; tests, docs, tools and manifests excluded. Method:
python3 tools/production_loc.py (SHA-256
 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6)
on git archives of the first parent 584cc3e (163886) and of the staged tree
20b10a3 (163908). Engine +22. Reference absent (retired R9c; last counted
65,417 at tree 498dd19).
…t (lp73)

`DECODED_BYTES` (512 MiB per export) was memory nothing could evict: many
Workspaces can be mounted at once, so a per-export budget that is never paid
back does not fit. Decoded images now take the same `RESIDENT_BYTES` (32 MiB)
as bodies kept as written, a body at its length and a decoded one as
`Image::cost`, so the budget sees every byte the export holds.

`Companion::resident_cost` is that one number; `attach`, `detach` and
`edit_body` count it, `edit_body` decides with it (a body either fits or the
whole edit answers `NFS3ERR_NOSPC` and leaves the companion as it was), and
`classify` no longer consults a second budget. `DECODED_ENTRY_BYTES` stays, now
as the most `Image::cost` one decoded companion may be counted for: a whole
image costing more keeps the client's bytes, so one companion cannot take the
whole pool. This layer has no disk to spill to, so a body the budget does not
take is still refused; the next layer spills it.

`Released` reports the merged pool as `companion_bytes` and keeps
`companions_decoded` as a count, instead of `companion_decoded_bytes` beside a
separate resident total.

Review: owner decision 3 (`DECODED_BYTES` rejected), first half (the budget and
the counts); the spill half is the next layer.

Tests: `the_clients_write_sequence_is_served_byte_for_byte` now asserts the
merged pool: the decoded image's `Image::cost()` after the image arrives, the
written body's length while it does not decode, the cost again once it does, and
zero after the companion is removed. `cargo test -p layerfs-nfs`: 42 passed,
0 failed. Anti-proof: with a decoded body counted as zero again the test fails
at the first assertion (`left: 0, right: 4372`).

Production LOC: 163908 -> 163896 (delta -12)
Scope: first-party core/engine/control/api production source and shipped
runtime SQL; tests, docs, tools and manifests excluded. Method:
python3 tools/production_loc.py (SHA-256
 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6)
on git archives of the first parent b01e805 (163908) and of the staged tree
bfc489b (163896). Engine -12. Reference absent (retired R9c; last counted
65,417 at tree 498dd19).
The layer below changed the read path, the budget and the counters; this layer
had moved the read and the edit into `apple_double/edit.rs` and given the budget
a spill directory. Conflicts resolved by keeping this layer's structure with the
layer below's semantics:

- `Image::encode_range` serves a READ of a decoded body (the read moved to
  `edit.rs`, so the window it now asks for is applied there);
- `DECODED_BYTES` stays deleted: `Holding` and `Released` report one
  `resident_bytes`/`companion_bytes` pool plus the decoded count, and
  `keep_decoded` asks the resident budget, not a second one;
- `classify` keeps this layer's "a body on disk stays as the spill left it" and
  drops the decoded budget check (the per-entry cap stays);
- `adopt_spilled`, `oldest_resident` and this layer's edit path are unchanged
  here.

`a_decoded_companion_is_edited_under_a_full_resident_budget` sets a 5,120-byte
budget: it takes the image's attributes (4,369) but not its 8 KiB of bytes, which
is what the merged budget now means.

Tests: `cargo test -p layerfs-nfs`: 47 passed, 0 failed (raw `test result` sum
48: `spilled_companions_outnumber_the_open_file_limit` runs itself again with
`--exact` in a child process, whose line is counted too).

Production LOC: 164334 -> 164340 (delta +6)
Scope: first-party core/engine/control/api production source and shipped
runtime SQL; tests, docs, tools and manifests excluded. Method:
python3 tools/production_loc.py (SHA-256
 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6)
on git archives of the first parent 8183f89 (164334) and of the staged tree
(first-parent merge comparison, 164340). Engine +6. Reference absent (retired
R9c; last counted 65,417 at tree 498dd19).
With decoded images counted against `RESIDENT_BYTES`, a decoded body is memory
like any other and has to leave memory like any other. `oldest_resident` now
takes the least recently used companion whose `resident_cost` is not zero,
which is a body kept as written or a decoded one, so eviction reaches decoded
bodies. Writing a decoded body out needs nothing new: `spill_one` and
`spill_gated` already encode it and write those bytes, which are exactly the
image the client wrote.

A whole image that does not fit the budget is therefore no longer refused
(`NFS3ERR_NOSPC`) nor kept decoded beyond it: it goes to its spill file and is
edited there, as any body does.

Review: owner decision 3 (`DECODED_BYTES` rejected), second half (the spill).

Tests: new `eviction_takes_the_decoded_body_before_the_written_ones` under a
4,096-byte budget: a decoded image of 213 bytes, then two 2,000-byte bodies.
The decoded body is the least recently used memory, so it is the one spilled
(`companions_decoded` 1 -> 0), the two written bodies stay resident (`4000`
bytes), and the image reads back byte for byte from its file. Anti-proof: with
`oldest_resident` filtered to bodies kept as written again, the test fails
(`companions_decoded` stays 1 and the written body is evicted instead).

Production LOC: 164340 -> 164337 (delta -3)
Scope: first-party core/engine/control/api production source and shipped
runtime SQL; tests, docs, tools and manifests excluded. Method:
python3 tools/production_loc.py (SHA-256
 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6)
on git archives of the first parent 7a9f20c (164340) and of the staged tree
(164337). Engine -3. Reference absent (retired R9c; last counted 65,417 at tree
498dd19).
Eviction walked every companion to find the least recently used one, and
`spill_for` repeated that walk up to `SPILL_ROUNDS` times, so one edit could
scan the index under the companion lock several times.

`Companions::by_use` is now a `BTreeSet<(used tick, serial)>` of the companions
holding memory: `attach` inserts an entry when a body takes budget, `detach`
removes it, and `set_used` (behind `use_serial`, the guest's last touch of the
body) moves it. `oldest_resident` is the first entry, skipping the caller's own
serial, so a round costs one step into the order (O(log n)) and no walk.
`spill_for` no longer bounds its rounds by the number of companions: it takes
the oldest body, and if that body changed under it the other call has already
moved its entry to the newest end, so the next round looks at a different one;
the rounds end when nothing holding memory is left. `SPILL_ROUNDS` goes away.

The companion lock now does O(1) or O(log n) work per step with no I/O:
`attach`/`detach`/`set_used` one set operation, `oldest_resident` one lookup,
and the file written before the lock is taken and renamed after it is dropped.
The kernel table lock is untouched by any of this.

Review: blocking finding 2 (eviction is an O(n) scan under the lock, repeated).

Tests: new `eviction_follows_the_order_of_use_not_the_shape_of_the_index`: 64
companions of 16 bytes under a 1,024-byte budget, three of them read again, then
eight more; the spill directory holds exactly the serials of the eight least
recently used ones, in that order, and the three read again stay resident.
Anti-proof: with `oldest_resident` reversed the test fails, spilling
`SERIALS+2, +64..+70` (the newest) instead of `SERIALS+3..+10`.

Production LOC: 164337 -> 164342 (delta +5)
Scope: first-party core/engine/control/api production source and shipped
runtime SQL; tests, docs, tools and manifests excluded. Method:
python3 tools/production_loc.py (SHA-256
 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6)
on git archives of the first parent e323c0e (164337) and of the staged tree
(164342). Engine +5. Reference absent (retired R9c; last counted 65,417 at tree
498dd19).
…e lock (lp73)

`Companions::adopt_spilled` ran `fs::rename` while holding the companion lock,
so the module documentation's "file I/O never holds the companion lock" was not
true of the step that puts the file in place.

The adoption is now three steps with the lock dropped between them:
`Companions::spill_target` (lock) answers the companion's own file and whether
the body is still the one the file was written from, `fs::rename` runs outside
the lock, and `Companions::commit_spilled` (lock) switches the body to
`Body::Spilled` with its length. A reader keeps reading the body in memory until
the rename is done, so no reader ever sees a half written or missing file.

The exclusion the rename needs is the companion's own edit gate, which already
means "one edit of this companion at a time": a gated edit holds it across its
own adoption, and eviction now takes it with `try_lock`, skipping a companion
another call is editing (it never waits for another call's I/O, so no two spills
of one companion can meet at the file). `spill_for` therefore passes over a
victim it could not spill instead of picking it again, and a companion that is
gone is nothing to spill rather than an error.

Review: finding 5 (disk I/O under the companion lock); the module docs of
`edit.rs` and `apple_double_spill.rs` and `nfs.md` now state what the code does.

Tests: `cargo test -p layerfs-nfs`: 48 passed, 0 failed (raw `test result` sum
49 with the child-process rerun of
`spilled_companions_outnumber_the_open_file_limit`), including the spill, race
and file-limit cases that adopt bodies.

Production LOC: 164342 -> 164372 (delta +30)
Scope: first-party core/engine/control/api production source and shipped
runtime SQL; tests, docs, tools and manifests excluded. Method:
python3 tools/production_loc.py (SHA-256
 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6)
on git archives of the first parent 73f62ef (164342) and of the staged tree
(164372). Engine +30. Reference absent (retired R9c; last counted 65,417 at tree
498dd19).
`companion_read` took the companion's length under the companion lock and then
read the spilled file outside it with `read_exact_at`. A truncation landing
between the two made the file shorter than the window, and `read_exact_at`
turned that into `NFS3ERR_IO` for a read of bytes the client had already been
told were there.

One `read_at` of that window is the whole reply: the bytes the file still holds,
however many, are a short read at the truncation point, and `eof` still follows
the length the lock saw. No retry, no second read.

Review: finding 6 (spurious `NFS3ERR_IO` from READ).

Tests: new `a_read_racing_a_truncation_is_a_short_read_not_an_io_error`, 512
rounds of one client reading a 64 KiB spilled body while another truncates it to
zero: every reply is the body or a prefix of it, and the test client panics on
any error status. Anti-proof: with `read_exact_at` back the test fails at round
0 (`panicked at tests/support/client.rs:248: self.read(...).unwrap()`).

Production LOC: 164372 -> 164374 (delta +2)
Scope: first-party core/engine/control/api production source and shipped
runtime SQL; tests, docs, tools and manifests excluded. Method:
python3 tools/production_loc.py (SHA-256
 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6)
on git archives of the first parent 0cbd34c (164372) and of the staged tree
(164374). Engine +2. Reference absent (retired R9c; last counted 65,417 at tree
498dd19).
…lp73)

Owner decision 2 approved `<mounts>/.appledouble/` on condition that the
architecture and the cleanup docs name it a new disposable area. nfs.md and the
code already described it; the two remaining places did not.

`docs/architecture/engine/daemon/README.md`: the "Starts NFS serving" row of the
startup table now says the same reclaim that force-unmounts leftovers deletes
`<mounts>/.appledouble` (directories `0700`, files `0600`, no fsync, no space
reserved ahead) and touches nothing else under the mount home.

`docs/general/cleanup.md`: the classification table gains the scratch area as a
disposable per-mount resource, with what it is and who deletes it.

The test that says reclaim removes only `.appledouble` is still here:
`startup_reclaim_clears_leftover_companion_spill_directories` puts a `kept/`
file beside a leftover `.appledouble/7/stale`, and asserts the first survives
and the second does not.

Review: owner decision 2, condition "document it in the kernel routes, NFS and
cleanup docs, and keep the reclaim test".

Tests: `cargo test -p layerfs-nfs` at this commit: 49 passed, 0 failed (raw
`test result` sum 50 with the child-process rerun of
`spilled_companions_outnumber_the_open_file_limit`), including that daemon test.

Production LOC: 164374 -> 164374 (delta 0)
Scope: first-party core/engine/control/api production source and shipped
runtime SQL; tests, docs, tools and manifests excluded. Method:
python3 tools/production_loc.py (SHA-256
 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6)
on git archives of the first parent 485dd2f (164374) and of the staged tree
(164374, engine 75,739 in both). Docs and tests only. Reference absent (retired
R9c; last counted 65,417 at tree 498dd19).
…p73)

`cargo fmt --check` reported three files of this layer: the `emit` call rustfmt
wraps in `Image::encode_range`, and two assertions of the new codec and decode
tests. This is that pass and nothing else.

The one remaining fmt report, `api/crates/layerfs-cli/tests/concurrent_reads.rs`,
is not touched by this branch (`git log 8cfb6e2..HEAD -- <file>` is empty) and
is left alone.

Checks: `cargo +1.85.1 fmt --all -- --check` reports only that file;
`cargo +1.85.1 clippy --locked -p layerfs-nfs --all-targets -- -D warnings` is
clean; `cargo test -p layerfs-nfs` passes every binary.

Production LOC: 163896 -> 163901 (delta +5)
Scope: first-party core/engine/control/api production source and shipped
runtime SQL; tests, docs, tools and manifests excluded. Method:
python3 tools/production_loc.py (SHA-256
 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6)
on git archives of the first parent 7ef50a0 (163896) and of the staged tree
(163901). Engine +5: rustfmt splits the `emit` call, which the counter counts.
Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
The layer below made its own files fmt- and clippy-clean. Only
`tests/apple_double_decode.rs` conflicted: this layer inserts a test where the
layer below removes a blank line. The test stays and the file was formatted
again, so the result is the same pass applied to this layer's file.

Checks: `cargo +1.85.1 fmt --all -- --check` reports only
`api/crates/layerfs-cli/tests/concurrent_reads.rs` (not this branch);
`cargo +1.85.1 clippy --locked -p layerfs-nfs --all-targets -- -D warnings` is
clean; `cargo test -p layerfs-nfs` passes every binary.

Production LOC: 164374 -> 164379 (delta +5)
Scope: first-party core/engine/control/api production source and shipped
runtime SQL; tests, docs, tools and manifests excluded. Method:
python3 tools/production_loc.py (SHA-256
 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6)
on git archives of the first parent e3e46ae (164374) and of the staged tree
(164379). Engine +5 from the layer below. Reference absent (retired R9c; last
counted 65,417 at tree 498dd19).
… (lp73)

`spill_for` passed a victim's error straight out with `?`. A victim another
call had just removed answered `NFS3ERR_STALE`, and one whose spill file could
not be written answered `NFS3ERR_NOSPC`/`NFS3ERR_IO`, so an unrelated WRITE
failed on a companion it never named while its own body was still in memory.

A victim that leaves nothing to give back, that another call changed under its
file, that its own edit holds, or that cannot be written out is now passed over
and the round looks at the body after it. The writer only answers its own
error, from spilling its own body, which needs no budget; a writer that finds no
room and cannot write out its own body still reports that.

Review: finding P2-2 (an eviction victim's error reaching an unrelated writer).
The report claimed this was already skipped; the code and the report now agree.

Tests: new `an_eviction_victim_that_cannot_be_written_out_does_not_fail_the_writer`.
Two resident bodies of 1,000 bytes in a 2,000-byte budget, the oldest one's
spill path made a non-empty directory so its spill cannot be adopted, then a
third 1,000-byte write: it lands, the oldest body stays in memory byte-identical,
the second is on disk and reads back, and only that one file is in the spill
directory. Anti-proof: with `?` back the third write answers `NFS3ERR_IO`
(`Err value: 5`) and the test fails on it.

Production LOC: 164379 -> 164379 (delta 0)
Scope: first-party core/engine/control/api production source and shipped
runtime SQL; tests, docs, tools and manifests excluded. Method:
python3 tools/production_loc.py (SHA-256
 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6)
on git archives of the first parent bdb87af (164379) and of the staged tree
(164379). Engine 0: the round's condition keeps one line and the rest is
comment. Reference absent (retired R9c; last counted 65,417 at tree
498dd19).
Three edges of the spill path, none of them a behaviour the client can rely on
today:

- A body written out whole is spilled at the length the edit wanted, and
  `commit_spilled` records that longer length before the edit touches the file.
  A `write_all_at`/`set_len` that then fails (`NFS3ERR_NOSPC`, `NFS3ERR_IO`)
  answered the error but left the companion longer, zero-filled: a refused edit
  that had changed the body's size. `apply_edit` now keeps the length the
  companion had and puts it back when the file operation fails, so a failed
  WRITE/SETATTR leaves the length as it was, as a refused slot does.
- `spill_one` took the victim's edit gate with `try_lock` and read every
  `Err` as "busy", `Poisoned` included: a companion whose edit panicked could
  never be evicted, so it held its body for the export's life. A poisoned gate
  is now taken anyway, as every other holder of it does.
- A spilled READ's one `pread` answered `NFS3ERR_IO` when a signal interrupted
  it, although no byte of it had moved. It is asked again in that case only;
  the read is still one read of one window, and a short read is still the
  reply.

Review: findings P3-2, P3-3 and P3-5.

Tests: none new. The length restore only shows on a filesystem error after a
successful spill (the file is already at the wanted length, so `RLIMIT_FSIZE`
and a directory at the spill path fail before it), the poisoned gate needs a
panicking edit, and `EINTR` needs a signal landing inside one `pread`; all three
are timing- or filesystem-dependent, and the change is a guard each. `cargo test
-p layerfs-nfs`: 51 passed, 0 failed (raw sum 52 with the child-process rerun).

Production LOC: 164379 -> 164405 (delta +26)
Scope: first-party core/engine/control/api production source and shipped
runtime SQL; tests, docs, tools and manifests excluded. Method:
python3 tools/production_loc.py (SHA-256
 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6)
on git archives of the first parent ec17db8 (164379) and of the staged tree
(164405). Engine +26: the length snapshot and restore (8), the poisoned-gate
match (5), the interrupted-read loop and its helper (11), and the two rewritten
error paths. Reference absent (retired R9c; last counted 65,417 at tree
498dd19).
No conflicts. Upstream since `8cfb6e273` brought the standalone fuser Git fork
(Ephemeral-AI-Lab#686, which removes `engine/vendor` and `tools/check_fuser_integrity.py`) with
its pin guard (Ephemeral-AI-Lab#687), and the GC spec items (Ephemeral-AI-Lab#688). None of those files is this
layer's, so the merged tree is upstream's plus this branch's commits.

Production LOC: 163901 -> 163901 (delta 0)
Scope: first-party core/engine/control/api production source and shipped
runtime SQL; tests, docs, tools and manifests excluded (the removed vendored
fuser is third-party code and never counted). Method:
python3 tools/production_loc.py (SHA-256
 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6)
on git archives of the first parent 428ffc3 (163901) and of the merged tree
(163901). Reference absent (retired R9c; last counted 65,417 at tree
498dd19).
… (lp73)

The codec layer now carries `upstream/layerfs_v2_0_0` at `cb8ced9d3`; that
merge had no conflicts, and neither does this one. Upstream's changes since
`8cfb6e273` (the standalone fuser Git fork, which removes `engine/vendor` and
`tools/check_fuser_integrity.py`, its pin guard, and the GC spec items) touch
no file of this layer, so the result is the upstream tree plus this layer's
commits.

Production LOC: 164405 -> 164405 (delta 0)
Scope: first-party core/engine/control/api production source and shipped
runtime SQL; tests, docs, tools and manifests excluded (the removed vendored
fuser is third-party code and never counted). Method:
python3 tools/production_loc.py (SHA-256
 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6)
on git archives of the first parent aec83af (164405) and of the merged tree
(164405). Reference absent (retired R9c; last counted 65,417 at tree
498dd19).
Copilot AI balanced review requested due to automatic review settings November 16, 2026 04:26

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@yifanxuaaa yifanxuaaa closed this pull request by merging all changes into Ephemeral-AI-Lab:layerfs_v2_0_0 in 5742338 Nov 17, 2026
jiangkoumo pushed a commit to jiangkoumo/layerfs that referenced this pull request Nov 17, 2026
…er-mount scratch area (lp73)

Merges the PR head e3c80ed unchanged; review fixes follow in their own commits.

Production LOC: 170247 -> 170751 (delta +504)
  core     39,795 -> 39,795 (0)
  engine   75,670 -> 76,174 (+504)  layerfs-daemon +15, layerfs-nfs +489
  control  48,436 -> 48,436 (0)
  api       6,346 ->  6,346 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --detail on first parent 7c0f937 and the merged tree (merge: first-parent comparison)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jiangkoumo pushed a commit to jiangkoumo/layerfs that referenced this pull request Nov 17, 2026
- A WRITE that grows a spilled body no longer shows its new length before its
  bytes. `apply_edit` wrote the body out pre-extended to the edit's length,
  and `commit_spilled` recorded that length before `edit.disk` wrote anything.
  A GETATTR or READ in between (neither takes the edit gate) saw the larger
  size and read zeros, and a failed edit shrank it back, so a client could see
  a length no WRITE made. The body is now written out at the length it has,
  the edit grows its file itself, and `record_edit` moves the length after the
  bytes are there. `restore_len` and the length it saved are gone, and with
  them the early `Stale` return that skipped `unreserve` after a reservation.
- `spill_target` never replaces a body that is already on disk, whoever asks.
  Before, a spill with no `used` snapshot renamed its file onto the live one
  first and found out afterwards. That was unreachable while the edit gate
  holds, but it would have deleted the live body.
- With no spill directory, `spill_for` stops at once instead of copying or
  encoding every resident victim under the companion lock only to fail on each.
  One victim whose file cannot be written is still passed over for the next
  (`an_eviction_victim_that_cannot_be_written_out_does_not_fail_the_writer`
  keeps that).
- `spill_applies` was public only for a test that fed it integers: both are
  removed, and `spill_target` compares the tick itself. Deleted test:
  `write.rs::a_spill_replaces_a_companion_body_only_when_no_call_changed_it`.
  The behaviour it named holds: every body change runs under the edit gate,
  and a spill takes the gate with `try_lock` and checks the `used` tick.
- `spilled_companions_outnumber_the_open_file_limit` waits for its child with
  a 90 s deadline and kills a child that hangs, instead of `status()`, which
  waits indefinitely.

Docs: nfs.md, the paragraph on one edit per companion.
Tests: no deterministic client-side test reproduces the GETATTR-during-spill
window. The covering suites pass on macOS (apple_double_decode 31, write 15).

Production LOC: 171,523 -> 171,508 (delta -15)
  core     39,795 -> 39,795 (0)
  engine   76,946 -> 76,931 (-15)  layerfs-nfs -15
  control  48,436 -> 48,436 (0)
  api       6,346 ->  6,346 (0)
  reference absent (retired R9c; last counted 65,417 at tree 498dd19)
  Method: tools/production_loc.py --detail on parent 519f8db and the staged tree

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jiangkoumo added a commit to jiangkoumo/layerfs that referenced this pull request Nov 17, 2026
Bring the Ephemeral-AI-Lab#136 native unmount helper reaping and detach-stage custody work
onto upstream/layerfs_v2_0_0 (5742338), including Ephemeral-AI-Lab#712/Ephemeral-AI-Lab#715/Ephemeral-AI-Lab#716/Ephemeral-AI-Lab#718
and the lp73 AppleDouble companion/spill series (Ephemeral-AI-Lab#713/Ephemeral-AI-Lab#714/Ephemeral-AI-Lab#722 via Ephemeral-AI-Lab#727).

Conflicts resolved keeping both sides' behavior intact:
- control/crates/layerfs-sandbox-backend/src/local/os/macos/darwin.rs:
  Keep upstream's Ephemeral-AI-Lab#712 `mount_line` and deadline-bounded `mount_table`
  together with Ephemeral-AI-Lab#644's shared `layerfs_bridge::process::Retained` queue
  (`RETAINED`), dropping the duplicate `Mutex<Vec<Child>>` / `reap_retained`
  / `retain` helpers so both `mount_table` and `force_unmount` reap
  through `RETAINED` (`reap`, `record` on synchronous reap, `retain` for
  background reaping on timeout). In `force::close` (Ephemeral-AI-Lab#716) and
  `machine_force::stop` (Ephemeral-AI-Lab#718), `exec::cancel::settle` cancels running
  workspace commands first, before `Sandbox::stop` -> `os::mounts::detach`
  -> `darwin::force_unmount`, avoiding overlap with `RETAINED`.
- engine/crates/layerfs-daemon/src/control/export.rs:
  Keep both `ExportServing::is_mounted` (Ephemeral-AI-Lab#644) and `companion_spill_dir` /
  `spill_root` (Ephemeral-AI-Lab#714), combine `.appledouble` leftover cleanup with the
  `mount_table`-aware `mounted_under` call in `ExportServing::reclaim`,
  and preserve `companion_spill_dir` in `ExportConfig` and `spill_root`
  rmdir after `ready.directory` removal in `teardown_export` (while
  `server.shutdown()` -> `Kernel::release()` removes the per-namespace
  spill dir first).

Checks: `tools/check_product_boundary.py` PASS; `tools/test_*.py` 180/180;
`lf-ci.sh fast` 2258/2259 (1 pre-existing upstream Ephemeral-AI-Lab#724 failure in
`layerfs-api::machine_files`, reproduced identically at 5742338);
`layerfs-nfs::helper_deadline` 8/8, `daemon` 2/2, `export` 10/10,
`apple_double_codec` 7/7, `apple_double_decode` 32/32; `layerfs-sandbox`
`record_codec` 5/5 & `unmount_record` 9/9; `layerfs-daemon` non-mount
31/31 & selected mount lane 12/12 binaries (36 passed, 1 ignored helper);
Clippy clean on changed crates; `rustfmt --check` clean on all changed
files (`fmt --all --check` fails only on 3 pre-existing upstream files at
5742338). `aarch64-apple-darwin` check blocked on Linux host C toolchain
(`blake3` `-arch arm64`): macOS build and live tests NOT_RUN.

Durable: NOT_RUN — disabled by owner until explicit reauthorization.
Production LOC: 163827 -> 171849 (delta +8022); first parent 89116b1.
  core 39787 -> 39795 (+8), engine 75121 -> 77198 (+2077),
  control 42842 -> 48510 (+5668), api 6077 -> 6346 (+269).
  Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
  Method: python3 -B tools/production_loc.py --detail on git archive of
  first parent 89116b1 and the merged working tree.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants