Repository navigation
nfs: one AppleDouble companion per inode, 65,536 per export, nordirplus on macOS (lp73) - #722
Merged
42 commits merged intoNov 17, 2026
Merged
42 commits merged into
42 commits merged into
Conversation
….rs (lp73) Relocation only, no behavior change: `Kernel::release`, `Kernel::held`, `retained_count` and the `Released` struct move out of `kernel.rs` into `kernel/released.rs` verbatim, and `pub use released::Released` keeps every existing path working. This is the first commit of the AppleDouble split (LP636): `kernel.rs` would otherwise cross the 999-line ceiling once the companion work lands. No test or documentation change; the moved functions are the same text in a new module. Production LOC: 163505 -> 163511 (delta +6) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent 8cfb6e2 (163505) and this tree (163511). The +6 is the new module's own two `use` lines, its `impl Kernel` block and the `mod released;` / `pub use released::Released;` declaration in kernel.rs. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
The client lays a `._<name>` companion out as an AppleDouble v2 image: a header, a FinderInfo entry that runs into an `ATTR` table of extended attributes, and a resource-fork entry at the end. `apple_double_codec` reads those fields (`Image::parse`) and, when the fields encode back to the very bytes the client wrote (`Image::compact`), keeps them as its own record, so no attribute, value or byte the layout carries is dropped, and one attribute's alignment padding survives a shorter name replacing a longer one. `Image::new`/`Image::encode` lay an image out as the client does, which is what makes the round trip exact and what the macOS evidence compares against. Malformed, truncated or differently laid-out images, and a resource fork with data, parse to `None` and are left as the client wrote them. This is the first layer of the AppleDouble split (LP636): the codec alone, with its own unit tests; the companion that holds its decoded form comes next. Tests: new `tests/apple_double_codec.rs`, 6 cases (byte-exact round trip for every value size, many attributes, alignment padding, whole units, a resource fork, and malformed images that must not decode), all pass. Production LOC: 163511 -> 163792 (delta +281) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent 1810b4e (163511) and this tree (163792). Engine +281 (`apple_double_codec.rs` +280, `lib.rs` +1). Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
A macOS client writes each companion once, in full, and reads it back unchanged. A body is therefore classified whenever a write or a resize leaves it: one that parses back to the very bytes the companion holds (`Image::compact`) and fits `DECODED_ENTRY_BYTES` is kept as its FinderInfo and attributes and counted against `DECODED_BYTES` instead of its image, and every READ synthesizes the same bytes again. Anything else - a resource fork, another layout, a half-written image - is stored as the client wrote it. This replaces `TOTAL_BYTES` with two budgets that say what they bound: `RESIDENT_BYTES` (32 MiB) of written bodies kept in memory, and `DECODED_BYTES` (512 MiB) of decoded ones counted as `Image::cost`. A write or resize that neither decodes nor fits the resident budget answers `NFS3ERR_NOSPC` and leaves the companion exactly as it was; the next layer spills those bodies to disk instead of refusing them. `Released` now reports `companions_decoded` and `companion_decoded_bytes`, and `companions_release`/`companions_held` answer the `Holding` record the counters live in. Second layer of the AppleDouble split (LP636), on top of the codec. Tests: new `tests/apple_double_decode.rs` with the client's own write sequence (a half-written header, the rest, an edit that breaks the image and one that completes it again, truncate below and resize back up), read back byte for byte. Full `layerfs-nfs` suite: 41 passed, 0 failed. Production LOC: 163792 -> 163886 (delta +94) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent 210d5f8 (163792) and this tree (163886). Engine +94. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
The NFS route guide now names `apple_double_codec.rs` and states what each bound actually holds: `RESIDENT_BYTES` for written bodies and `DECODED_BYTES` / `DECODED_ENTRY_BYTES` for the decoded form, replacing the `TOTAL_BYTES` row the code no longer has. The paragraph after the table says which bodies decode (complete images that encode back to the same bytes) and which stay as written. Documentation only; it describes the codec layer as it stands. Production LOC: 163886 -> 163886 (delta 0) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent fa5dd34 (163886) and this tree (163886). Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
…get (lp73) A client's companion body no longer has to fit in memory and no longer takes memory from every other export. Bodies beyond `RESIDENT_BYTES` (32 MiB) are spilled, least recently touched first, to a scratch directory of the export's own (`<mounts>/.appledouble/<namespace>/<serial>`), and a write that still does not fit the budget after bounded eviction rounds is written out itself and edited in its file, which needs no budget at all. - `apple_double_spill.rs` owns the scratch files: one per companion serial, directory `0700` and files `0600` whatever the umask, written through a `<serial>.<n>.tmp` that only the adopting call renames to `<serial>`, no fsync and no space reserved ahead. A spilled body is opened by path per `pread`/`pwrite`/`set_len` and never held open, so companions far outnumbering the process's file limit all work. A failed file operation is answered with the NFS status its errno means (`ENOSPC`/`EDQUOT`, `EFBIG`, `EACCES`, `EPERM`, `EROFS`, else `NFS3ERR_IO`). - `apple_double/edit.rs` holds READ, WRITE and SETATTR and the eviction: one edit of a companion at a time under that companion's own gate, so a truncate and a write can never leave a length its file does not have, and the file I/O never holds the companion lock. - `ExportConfig` gains `companion_spill_dir` and `companion_resident_bytes`. The daemon gives each export `<mounts>/.appledouble/<namespace>`, removes it at startup before any Attach, at release and when a failed Attach unwinds, and touches nothing else under the mount home. - `Released` reports `companion_bytes`. Third layer of the AppleDouble split (LP636), on top of the codec; `HELD` and the name index are the layer after this one. Production LOC: 163886 -> 164334 (delta +448) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent 584cc3e (163886) and this tree (164334). Engine +448, the daemon's `export.rs` share included. Tests for the layer follow in the next commit. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
The rig can now set the resident budget and an explicit spill directory, and the cases exercise the layer end to end: - a body smaller than the budget stays resident and a larger one spills; the spilled file carries the bytes and the length the client wrote; - an edit that leaves a decoded image not whole spills it instead of growing memory, while a truncate of a decoded body needs no room at all; - a spill only replaces a companion no other call changed meanwhile (`spill_applies`); - a truncate and a write of one spilled companion start together, round after round, and the length a reply reports is always the length its file holds; - companions four times the process's own `RLIMIT_NOFILE` all read and write, because a spilled body is opened by path per operation: that case lowers a process-wide limit and counts the process's descriptors, so it re-runs itself in its own process (`run_alone`); - Attach cleans a leftover `.appledouble` of a daemon that died, release and unmount remove it, and startup reclamation removes it while keeping every other entry under the mount home. `libc` is a new dev-dependency of `layerfs-nfs`, only for the file-limit case. Tests: `cargo test -p layerfs-nfs` 46 passed, 0 failed (the raw sum reads 47 because the file-limit case runs itself once in a child process). Production LOC: 164334 -> 164334 (delta 0) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent 7b38dad (164334) and this tree (164334). Tests only. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
The guide names `edit.rs` and `apple_double_spill.rs`, describes what the `RESIDENT_BYTES` row does now (spill to `companion_spill_dir`, and which status each host errno maps to), and records the scratch file's own rules: one file per companion serial under `<mounts>/.appledouble/<namespace>`, `0700`/`0600`, written as a temporary and adopted by rename only when no call changed the body, opened by path per operation and never held open, deleted by startup reclamation and by release. `ExportConfig` lists the two new fields, and the `wcc_data` row now says a companion mutation answers from `directory_stats`. Documentation only; it describes the spill layer as it stands. Production LOC: 164334 -> 164334 (delta 0) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent 91434f9 (164334) and this tree (164334). Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
A companion created while the export knows its main file (`<name>` in the same directory) now belongs to that inode: every name of the inode reaches it, so hard links share one, it follows the inode through RENAME and it goes with the inode's last name. Such a companion holding a complete AppleDouble image is `whole`: it takes no slot of the index bound, and `rename`/`silly rename`/ `remove` of the main file drop it through `Kernel::companion_drop_for` and `companion_drop_inode`. `HELD` rises from 4,096 companions to 65,536 slotted entries: at the bound, orphan slotted companions are reclaimed first, and only if 65,536 live ones remain does CREATE answer `NFS3ERR_NOSPC`, count `names_refused` and log at most once per 60 s per export. The name index is the other half of it. `Table::aliases` remembers a hard-linked inode's names beyond `Table::names`, so `entry_taken` (when the engine reports `nlink <= 1`), `renamed`, `finish_unlink` and retirement drop only their own serial's keys from `by_name` instead of scanning it; a name that was moving to another inode is dropped through `Table::unbind_any`. CREATE at the bound releases the companion lock to collect orphans, then re-reads the companion name and main file under the lock it now holds, so two GUARDED creates of one name yield one serial and one `NFS3ERR_EXIST`. `Companions::slots` walks only slotted entries - whole images add nothing - and `classify` decides again under RENAME's destination, so a companion moved onto an unknown main takes its slot back while a spilled body keeps the class the write that spilled it left (a decode there would read its file under the lock). `Kernel::companions_release` reports `Holding`, and `Released` carries `companions_slotted` and `companions` counted over whole images too. This is the last layer of the AppleDouble split (LP636) and reproduces `2e09f0295`'s companion code; the review's open findings are handled in the next task. Tests: `cargo test -p layerfs-nfs` 60 passed, 0 failed (raw sum 61: the file-limit case re-runs itself in a child process): 16 write, 11 decode, 6 codec, 2 daemon cases. Production LOC: 164334 -> 164574 (delta +240) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent 8183f89 (164334) and this tree (164574). Engine +240. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
…lp73) The remaining cases of the layer: a whole image takes no slot and is read back from every name of its inode; a resource fork, an unknown main and a companion moved off its inode keep the stored-as-written path and their slot; `HELD` is 65,536 and a small bound reclaims orphans instead of refusing; two GUARDED creates of one name, and two of them over the last free slot, leave one entry; a reclaimed companion never loses a live main file's attributes, including one that gained a hard link; a renamed hard-link alias keeps the inode's recalled name bound; and companions survive a main-file rename, silly rename and replacing rename exactly as the client expects. `apple_double_companions_spill_preserve_all_bytes_reclaim_orphans_and_clean_up` is the long one: 4,160 companions past the old 4,096 count and a 64 KiB resident budget, every write read back byte for byte, the spill directory's modes, the unlinks a main-file removal or replacing RENAME performs, the `FILE_BYTES` refusal, and the `HELD` bound with orphan reclamation before `NFS3ERR_NOSPC`. The rig can set `companions_held` too. Production LOC: 164574 -> 164574 (delta 0) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent dc2c54c (164574) and this tree (164574). Tests only. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
The AppleDouble section now states what `HELD` (65,536 slotted entries) does at the bound - reclaim orphans, then `NFS3ERR_NOSPC` with one rate-limited log per export - and that a companion whose main inode the export knows belongs to that inode: hard links share it, RENAME keeps it under the new name and its aliases, and its last name or silly rename drops it. It records that a whole image takes no `HELD` slot, that `companion_rename` re-classifies under the destination, and how `Table::names`/`Table::aliases` keep unlink, `entry_taken`, `renamed` and retirement from scanning `by_name`, and how CREATE re-reads the name and main file after collecting orphans. Documentation only; it describes the layer as it stands. Production LOC: 164574 -> 164574 (delta 0) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent 2cc52c813 (164574) and this tree (164574). Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
…s for (lp73) A decoded companion was re-encoded in full under the companion lock on every READ: up to `FILE_BYTES` (1 MiB) allocated and copied to serve a reply of at most `TRANSFER_WINDOW` (256 KiB) bytes, and often far less. `Image::encode_range` lays the image out and fills only `[start, end)`, so a READ pays for the bytes it answers with; `Image::encode` is that same layout over the whole image, so there is one implementation of the layout rather than two that can drift. `edit.rs`'s spill and `edit_resident` still need the whole image and keep calling `encode()`. Whether a READ is served from a window or from the whole image is not observable: the reply bytes are the same either way. Review: finding 7 (full re-encode on every READ). Tests: new `any_window_of_the_image_encodes_to_that_window` compares every window of an image (a padding byte the client left behind, values of several sizes, the empty resource fork) with the same slice of `encode()`, including empty windows, a window past the image and one that starts after it ends. `cargo test -p layerfs-nfs --test apple_double_codec --test apple_double_decode`: 7 + 1 passed, 0 failed. Anti-proof: with `encode_range` clamped to start at 0 again the window test fails (`left` starts at image offset 0 while the expected window starts mid-image). Production LOC: 163886 -> 163908 (delta +22) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent 584cc3e (163886) and of the staged tree 20b10a3 (163908). Engine +22. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
…t (lp73) `DECODED_BYTES` (512 MiB per export) was memory nothing could evict: many Workspaces can be mounted at once, so a per-export budget that is never paid back does not fit. Decoded images now take the same `RESIDENT_BYTES` (32 MiB) as bodies kept as written, a body at its length and a decoded one as `Image::cost`, so the budget sees every byte the export holds. `Companion::resident_cost` is that one number; `attach`, `detach` and `edit_body` count it, `edit_body` decides with it (a body either fits or the whole edit answers `NFS3ERR_NOSPC` and leaves the companion as it was), and `classify` no longer consults a second budget. `DECODED_ENTRY_BYTES` stays, now as the most `Image::cost` one decoded companion may be counted for: a whole image costing more keeps the client's bytes, so one companion cannot take the whole pool. This layer has no disk to spill to, so a body the budget does not take is still refused; the next layer spills it. `Released` reports the merged pool as `companion_bytes` and keeps `companions_decoded` as a count, instead of `companion_decoded_bytes` beside a separate resident total. Review: owner decision 3 (`DECODED_BYTES` rejected), first half (the budget and the counts); the spill half is the next layer. Tests: `the_clients_write_sequence_is_served_byte_for_byte` now asserts the merged pool: the decoded image's `Image::cost()` after the image arrives, the written body's length while it does not decode, the cost again once it does, and zero after the companion is removed. `cargo test -p layerfs-nfs`: 42 passed, 0 failed. Anti-proof: with a decoded body counted as zero again the test fails at the first assertion (`left: 0, right: 4372`). Production LOC: 163908 -> 163896 (delta -12) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent b01e805 (163908) and of the staged tree bfc489b (163896). Engine -12. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
The layer below changed the read path, the budget and the counters; this layer had moved the read and the edit into `apple_double/edit.rs` and given the budget a spill directory. Conflicts resolved by keeping this layer's structure with the layer below's semantics: - `Image::encode_range` serves a READ of a decoded body (the read moved to `edit.rs`, so the window it now asks for is applied there); - `DECODED_BYTES` stays deleted: `Holding` and `Released` report one `resident_bytes`/`companion_bytes` pool plus the decoded count, and `keep_decoded` asks the resident budget, not a second one; - `classify` keeps this layer's "a body on disk stays as the spill left it" and drops the decoded budget check (the per-entry cap stays); - `adopt_spilled`, `oldest_resident` and this layer's edit path are unchanged here. `a_decoded_companion_is_edited_under_a_full_resident_budget` sets a 5,120-byte budget: it takes the image's attributes (4,369) but not its 8 KiB of bytes, which is what the merged budget now means. Tests: `cargo test -p layerfs-nfs`: 47 passed, 0 failed (raw `test result` sum 48: `spilled_companions_outnumber_the_open_file_limit` runs itself again with `--exact` in a child process, whose line is counted too). Production LOC: 164334 -> 164340 (delta +6) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent 8183f89 (164334) and of the staged tree (first-parent merge comparison, 164340). Engine +6. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
With decoded images counted against `RESIDENT_BYTES`, a decoded body is memory like any other and has to leave memory like any other. `oldest_resident` now takes the least recently used companion whose `resident_cost` is not zero, which is a body kept as written or a decoded one, so eviction reaches decoded bodies. Writing a decoded body out needs nothing new: `spill_one` and `spill_gated` already encode it and write those bytes, which are exactly the image the client wrote. A whole image that does not fit the budget is therefore no longer refused (`NFS3ERR_NOSPC`) nor kept decoded beyond it: it goes to its spill file and is edited there, as any body does. Review: owner decision 3 (`DECODED_BYTES` rejected), second half (the spill). Tests: new `eviction_takes_the_decoded_body_before_the_written_ones` under a 4,096-byte budget: a decoded image of 213 bytes, then two 2,000-byte bodies. The decoded body is the least recently used memory, so it is the one spilled (`companions_decoded` 1 -> 0), the two written bodies stay resident (`4000` bytes), and the image reads back byte for byte from its file. Anti-proof: with `oldest_resident` filtered to bodies kept as written again, the test fails (`companions_decoded` stays 1 and the written body is evicted instead). Production LOC: 164340 -> 164337 (delta -3) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent 7a9f20c (164340) and of the staged tree (164337). Engine -3. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
Eviction walked every companion to find the least recently used one, and `spill_for` repeated that walk up to `SPILL_ROUNDS` times, so one edit could scan the index under the companion lock several times. `Companions::by_use` is now a `BTreeSet<(used tick, serial)>` of the companions holding memory: `attach` inserts an entry when a body takes budget, `detach` removes it, and `set_used` (behind `use_serial`, the guest's last touch of the body) moves it. `oldest_resident` is the first entry, skipping the caller's own serial, so a round costs one step into the order (O(log n)) and no walk. `spill_for` no longer bounds its rounds by the number of companions: it takes the oldest body, and if that body changed under it the other call has already moved its entry to the newest end, so the next round looks at a different one; the rounds end when nothing holding memory is left. `SPILL_ROUNDS` goes away. The companion lock now does O(1) or O(log n) work per step with no I/O: `attach`/`detach`/`set_used` one set operation, `oldest_resident` one lookup, and the file written before the lock is taken and renamed after it is dropped. The kernel table lock is untouched by any of this. Review: blocking finding 2 (eviction is an O(n) scan under the lock, repeated). Tests: new `eviction_follows_the_order_of_use_not_the_shape_of_the_index`: 64 companions of 16 bytes under a 1,024-byte budget, three of them read again, then eight more; the spill directory holds exactly the serials of the eight least recently used ones, in that order, and the three read again stay resident. Anti-proof: with `oldest_resident` reversed the test fails, spilling `SERIALS+2, +64..+70` (the newest) instead of `SERIALS+3..+10`. Production LOC: 164337 -> 164342 (delta +5) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent e323c0e (164337) and of the staged tree (164342). Engine +5. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
…e lock (lp73) `Companions::adopt_spilled` ran `fs::rename` while holding the companion lock, so the module documentation's "file I/O never holds the companion lock" was not true of the step that puts the file in place. The adoption is now three steps with the lock dropped between them: `Companions::spill_target` (lock) answers the companion's own file and whether the body is still the one the file was written from, `fs::rename` runs outside the lock, and `Companions::commit_spilled` (lock) switches the body to `Body::Spilled` with its length. A reader keeps reading the body in memory until the rename is done, so no reader ever sees a half written or missing file. The exclusion the rename needs is the companion's own edit gate, which already means "one edit of this companion at a time": a gated edit holds it across its own adoption, and eviction now takes it with `try_lock`, skipping a companion another call is editing (it never waits for another call's I/O, so no two spills of one companion can meet at the file). `spill_for` therefore passes over a victim it could not spill instead of picking it again, and a companion that is gone is nothing to spill rather than an error. Review: finding 5 (disk I/O under the companion lock); the module docs of `edit.rs` and `apple_double_spill.rs` and `nfs.md` now state what the code does. Tests: `cargo test -p layerfs-nfs`: 48 passed, 0 failed (raw `test result` sum 49 with the child-process rerun of `spilled_companions_outnumber_the_open_file_limit`), including the spill, race and file-limit cases that adopt bodies. Production LOC: 164342 -> 164372 (delta +30) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent 73f62ef (164342) and of the staged tree (164372). Engine +30. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
`companion_read` took the companion's length under the companion lock and then read the spilled file outside it with `read_exact_at`. A truncation landing between the two made the file shorter than the window, and `read_exact_at` turned that into `NFS3ERR_IO` for a read of bytes the client had already been told were there. One `read_at` of that window is the whole reply: the bytes the file still holds, however many, are a short read at the truncation point, and `eof` still follows the length the lock saw. No retry, no second read. Review: finding 6 (spurious `NFS3ERR_IO` from READ). Tests: new `a_read_racing_a_truncation_is_a_short_read_not_an_io_error`, 512 rounds of one client reading a 64 KiB spilled body while another truncates it to zero: every reply is the body or a prefix of it, and the test client panics on any error status. Anti-proof: with `read_exact_at` back the test fails at round 0 (`panicked at tests/support/client.rs:248: self.read(...).unwrap()`). Production LOC: 164372 -> 164374 (delta +2) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent 0cbd34c (164372) and of the staged tree (164374). Engine +2. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
…lp73) Owner decision 2 approved `<mounts>/.appledouble/` on condition that the architecture and the cleanup docs name it a new disposable area. nfs.md and the code already described it; the two remaining places did not. `docs/architecture/engine/daemon/README.md`: the "Starts NFS serving" row of the startup table now says the same reclaim that force-unmounts leftovers deletes `<mounts>/.appledouble` (directories `0700`, files `0600`, no fsync, no space reserved ahead) and touches nothing else under the mount home. `docs/general/cleanup.md`: the classification table gains the scratch area as a disposable per-mount resource, with what it is and who deletes it. The test that says reclaim removes only `.appledouble` is still here: `startup_reclaim_clears_leftover_companion_spill_directories` puts a `kept/` file beside a leftover `.appledouble/7/stale`, and asserts the first survives and the second does not. Review: owner decision 2, condition "document it in the kernel routes, NFS and cleanup docs, and keep the reclaim test". Tests: `cargo test -p layerfs-nfs` at this commit: 49 passed, 0 failed (raw `test result` sum 50 with the child-process rerun of `spilled_companions_outnumber_the_open_file_limit`), including that daemon test. Production LOC: 164374 -> 164374 (delta 0) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent 485dd2f (164374) and of the staged tree (164374, engine 75,739 in both). Docs and tests only. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
The layer below replaced the decoded budget with the resident one, added the eviction order, moved the spill rename out of the companion lock and made a raced READ a short read; this layer had added the inode binding, the slot bound and the orphan reclamation. Conflicts resolved by keeping this layer's index (whole images take no slot, unknown mains take one) over the layer below's budget, eviction and spill work: - `DECODED_BYTES` stays deleted here too: `Holding`/`Released` report one `resident_bytes`/`companion_bytes` pool, `keep_decoded` asks the resident budget, and `classify` keeps only the per-entry cap; - `by_use`, `set_used`, `oldest_resident(&BTreeSet)` and `spill_target`/ `commit_spilled` carry over, `commit_spilled` keeping this layer's `file.whole &= len <= file.len` rule; - `record_edit` keeps this layer's "a body changed on disk is no longer known to be a whole image"; - the docs state both: the slot bound with the merged body budget, and the rename outside the lock with the eviction order. `keep_decoded` now answers what it made of the body (`Kept`, `Room(needed)`, `Unchanged`) instead of a bare bool: with decoded images counted against `RESIDENT_BYTES`, an edit that completes a whole image has to ask for the room its *decoded* form needs (`cost - before`), not for the length of its bytes. Evicting one small body is then enough, and an image no longer goes to disk and takes a slot just because its raw bytes exceed the budget. `whole_images_take_no_slot_of_the_index_bound` would otherwise fail: it writes whole images under a 1,024-byte budget, and the fifth one has to evict instead of spilling. Tests: `cargo test -p layerfs-nfs`: 64 passed, 0 failed (raw `test result` sum 65 with the child-process rerun of `spilled_companions_outnumber_the_open_file_limit`). Production LOC: 164574 -> 164626 (delta +52) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent 6af05cd (164574) and of the staged tree (first-parent merge comparison, 164626). Engine +52. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
…itting (lp73) The doc comment and `nfs.md` said the index was bounded by `HELD`, but `attach` inserted every slotted companion without a check: a WRITE that broke a whole image, a SETATTR or a RENAME onto an unknown main could grow `slots` past the bound with no `NFS3ERR_NOSPC`, and every `orphans()` walk then scaled with it. `Companions::admits(serial, slotted)` is now the single adjudication: a body that takes no slot is always admitted, a companion already holding one keeps it, and otherwise the bound decides. It is asked before anything is committed, so a refusal leaves the body, its length and its name exactly as they were: - CREATE asks it for a new companion, as before: reclaim the orphan slots once, re-read the name and its main file, then `NFS3ERR_NOSPC` with the counted and rate-limited log; - a WRITE, SETATTR or RENAME that needs a new slot asks it too. The body is edited to one side (`edit_resident` keeps the previous bytes only in this case) and put back if the slot is refused; a body that would be written out and edited on its file is refused before the file is touched, since such a body is no longer known to be a whole image. `edit_companion` then reclaims the orphans once, outside the companion lock, and applies the same edit once more; a second refusal is the whole edit's `NFS3ERR_NOSPC`. The orphan reclaim moves out of `companion_create` into `Kernel::reclaim_orphans` so both paths use the same one, and `nfs.md` records that `HELD` is the admission limit of this export-local shim (owner decision, option C, a scoped waiver, with option A deferred) and not a Workspace capacity. Review: blocking finding 1 and owner decisions D1/C, including the test the owner asked for on the write path. Tests: new `a_write_that_takes_the_last_slot_is_refused_by_the_bound`, which fills the bound through WRITE and not CREATE: three companions each created as a whole image (no slot), then broken by a write; the first two take the two slots, the third answers `NFS3ERR_NOSPC` and still reads back its image byte for byte, with all three companions and their two slots intact. Anti-proof: with `admits` off by one the new test fails at that assertion (`left: Ok(()), right: Err(28)`) and so do three index-bound tests. Production LOC: 164626 -> 164676 (delta +50) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent 76adb9e (164626) and of the staged tree (164676). Engine +50. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
`Companion::classify` turns a decoded body back into the client's bytes when the companion is renamed onto a name whose main file the export does not know, and `companion_rename` attached it without looking at `RESIDENT_BYTES`: one rename could put a whole image (up to `FILE_BYTES`) of bytes into a budget it does not fit, and nothing evicted it afterwards. `companion_rename` now resolves the serial and validates the name under the lock, then hands over to `rename_companion`, which takes the companion's edit gate and decides before committing: - a decoded body moving onto an unknown main would cost its image; when that does not fit the budget it is written out first (`spill_gated`, with the file written and renamed outside the companion lock under the gate), so it takes no budget at all and the rename only records the new name and class; - a companion that needs a slot of the index bound asks `Companions::admits`, reclaims the orphan slots once and is refused with `NFS3ERR_NOSPC` rather than exceeding the bound, leaving it under its old name. Review: finding 3 (a rename can exceed the resident budget). Tests: new `a_rename_onto_an_unknown_main_keeps_the_resident_budget`: a decoded image (226 bytes counted) and a 700-byte written body under a 1,024-byte budget, then `._a` renamed to `._zzz`, whose main file does not exist. The resident count is 700 afterwards (the image left memory, `companions_decoded` 0), the new name reads the image back byte for byte and `._a` is gone. Anti-proof: without the write-out the test fails with 4,796 bytes resident against a 1,024-byte budget. Production LOC: 164676 -> 164709 (delta +33) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent e9f49fb (164676) and of the staged tree (164709). Engine +33. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
A companion created while its main name was missing from the export's name index got `inode: None`, and `orphans()` read that as "no main file": the name index is lossy (its own bound, a name the export never looked up, a rename on the client side), so a live companion could be reclaimed at the `HELD` bound and lose its attributes. `orphans()` now answers candidates, and `reclaim_orphans` decides on one whose main inode the export does not know with one read-only lookup of its main file in the engine (`Kernel::observe`), run outside the kernel table lock and the companion lock. A file that is there binds the companion back to that inode (`file.inode` and `Companions::inodes`), which keeps its attributes and makes every name of the inode reach it; only `NFS3ERR_NOENT` makes it an orphan; a lookup that cannot answer keeps it as it is. `Companions::slots` is unchanged, so a bound companion keeps the slot it took. Review: finding 4 (live data reclaimed as an orphan). Tests: new `a_companion_of_a_main_file_the_name_index_missed_is_kept`: the main file is part of the tree the export is initialized from and the export never looks its name up, so the companion created for it has no main inode, under a one-slot bound. The next CREATE reclaims, finds the main file, keeps the companion (`companions`/`companions_slotted` 1/1, `names_refused` 1), the image reads back byte for byte, and a hard link of the main file reaches the same companion. Anti-proof: reclaiming on the name index's word again drops it, and the test then fails when the refused CREATE succeeds instead. Production LOC: 164709 -> 164756 (delta +47) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent 812cb84 (164709) and of the staged tree (164756). Engine +47. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
…n (lp73) `cargo fmt --check` reported four files of this branch and `cargo clippy --workspace --all-targets -D warnings` two lint findings in the new tests (`needless_range_loop`) and one in `reclaim_orphans` (`single_match`). This is that pass and nothing else: no behaviour changes. The one remaining fmt report, `api/crates/layerfs-cli/tests/concurrent_reads.rs`, is not touched by this branch (`git log 8cfb6e2..HEAD -- <file>` is empty) and is left alone. Checks: `cargo +1.85.1 fmt --all -- --check` reports only that file; `cargo +1.85.1 clippy --locked --workspace --exclude layerfs-node --all-targets -- -D warnings` is clean; `python3 -B tools/check_product_boundary.py --skip-sdk` PASS, `python3 -B tools/check_fuser_integrity.py` no errors, `python3 -B -m unittest discover -s tools -p 'test_*.py'` 188 OK. `cargo test -p layerfs-nfs`: 67 passed, 0 failed (raw sum 68 with the child-process rerun). Production LOC: 164756 -> 164768 (delta +12) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent e902530 (164756) and of the staged tree (164768). Engine +12: rustfmt splits three long signatures and calls and the `if let` rewrites one match, which the counter counts. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
…p73) `cargo fmt --check` reported three files of this layer: the `emit` call rustfmt wraps in `Image::encode_range`, and two assertions of the new codec and decode tests. This is that pass and nothing else. The one remaining fmt report, `api/crates/layerfs-cli/tests/concurrent_reads.rs`, is not touched by this branch (`git log 8cfb6e2..HEAD -- <file>` is empty) and is left alone. Checks: `cargo +1.85.1 fmt --all -- --check` reports only that file; `cargo +1.85.1 clippy --locked -p layerfs-nfs --all-targets -- -D warnings` is clean; `cargo test -p layerfs-nfs` passes every binary. Production LOC: 163896 -> 163901 (delta +5) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent 7ef50a0 (163896) and of the staged tree (163901). Engine +5: rustfmt splits the `emit` call, which the counter counts. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
The layer below made its own files fmt- and clippy-clean. Only `tests/apple_double_decode.rs` conflicted: this layer inserts a test where the layer below removes a blank line. The test stays and the file was formatted again, so the result is the same pass applied to this layer's file. Checks: `cargo +1.85.1 fmt --all -- --check` reports only `api/crates/layerfs-cli/tests/concurrent_reads.rs` (not this branch); `cargo +1.85.1 clippy --locked -p layerfs-nfs --all-targets -- -D warnings` is clean; `cargo test -p layerfs-nfs` passes every binary. Production LOC: 164374 -> 164379 (delta +5) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent e3e46ae (164374) and of the staged tree (164379). Engine +5 from the layer below. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
The layer below made its own files fmt- and clippy-clean. Only `tests/apple_double_decode.rs` conflicted, again on an inserted test against a blank line removed below: this layer's tests stay and the file was formatted again, so both layers end with the same pass. Checks: `cargo +1.85.1 fmt --all -- --check` reports only `api/crates/layerfs-cli/tests/concurrent_reads.rs` (not this branch); `cargo +1.85.1 clippy --locked --workspace --exclude layerfs-node --all-targets -- -D warnings` is clean; `cargo test -p layerfs-nfs`: 67 passed, 0 failed (raw `test result` sum 68 with the child-process rerun of `spilled_companions_outnumber_the_open_file_limit`). Production LOC: 164768 -> 164768 (delta 0) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent 3c3558f and of the staged tree: both 164,768 production lines, because this branch's own fmt pass had already counted the layer below's formatting. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
… (lp73) `spill_for` passed a victim's error straight out with `?`. A victim another call had just removed answered `NFS3ERR_STALE`, and one whose spill file could not be written answered `NFS3ERR_NOSPC`/`NFS3ERR_IO`, so an unrelated WRITE failed on a companion it never named while its own body was still in memory. A victim that leaves nothing to give back, that another call changed under its file, that its own edit holds, or that cannot be written out is now passed over and the round looks at the body after it. The writer only answers its own error, from spilling its own body, which needs no budget; a writer that finds no room and cannot write out its own body still reports that. Review: finding P2-2 (an eviction victim's error reaching an unrelated writer). The report claimed this was already skipped; the code and the report now agree. Tests: new `an_eviction_victim_that_cannot_be_written_out_does_not_fail_the_writer`. Two resident bodies of 1,000 bytes in a 2,000-byte budget, the oldest one's spill path made a non-empty directory so its spill cannot be adopted, then a third 1,000-byte write: it lands, the oldest body stays in memory byte-identical, the second is on disk and reads back, and only that one file is in the spill directory. Anti-proof: with `?` back the third write answers `NFS3ERR_IO` (`Err value: 5`) and the test fails on it. Production LOC: 164379 -> 164379 (delta 0) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent bdb87af (164379) and of the staged tree (164379). Engine 0: the round's condition keeps one line and the rest is comment. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
Three edges of the spill path, none of them a behaviour the client can rely on today: - A body written out whole is spilled at the length the edit wanted, and `commit_spilled` records that longer length before the edit touches the file. A `write_all_at`/`set_len` that then fails (`NFS3ERR_NOSPC`, `NFS3ERR_IO`) answered the error but left the companion longer, zero-filled: a refused edit that had changed the body's size. `apply_edit` now keeps the length the companion had and puts it back when the file operation fails, so a failed WRITE/SETATTR leaves the length as it was, as a refused slot does. - `spill_one` took the victim's edit gate with `try_lock` and read every `Err` as "busy", `Poisoned` included: a companion whose edit panicked could never be evicted, so it held its body for the export's life. A poisoned gate is now taken anyway, as every other holder of it does. - A spilled READ's one `pread` answered `NFS3ERR_IO` when a signal interrupted it, although no byte of it had moved. It is asked again in that case only; the read is still one read of one window, and a short read is still the reply. Review: findings P3-2, P3-3 and P3-5. Tests: none new. The length restore only shows on a filesystem error after a successful spill (the file is already at the wanted length, so `RLIMIT_FSIZE` and a directory at the spill path fail before it), the poisoned gate needs a panicking edit, and `EINTR` needs a signal landing inside one `pread`; all three are timing- or filesystem-dependent, and the change is a guard each. `cargo test -p layerfs-nfs`: 51 passed, 0 failed (raw sum 52 with the child-process rerun). Production LOC: 164379 -> 164405 (delta +26) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent ec17db8 (164379) and of the staged tree (164405). Engine +26: the length snapshot and restore (8), the poisoned-gate match (5), the interrupted-read loop and its helper (11), and the two rewritten error paths. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
`lp636-spill` grew two commits: an eviction victim's error no longer reaches the writer that needed the room, and a failed disk edit leaves the companion's length where it was (with the poisoned-gate and interrupted-read guards next to them). Both apply unchanged to this layer's `apply_edit`: the slot decision before the file is written stays, and the length snapshot and restore run after it, so a refused write still leaves the body, its file, its name and its slot as they were. Conflicts: `apply_edit` (this layer's `admits` decision before the spill, the lower layer's length restore after it, both kept) and the test file, where the two branches appended a test at the same place (both kept, each with its own trailing teardown). Production LOC: 164768 -> 164794 (delta +26) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent aec83af's tree without the merge (this branch's head 525fcfb, 164768) and of the merged staged tree (164794). The +26 is the lower layer's, carried in by the merge. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
A companion whose main inode the export did not know was, at the `HELD` bound, looked up in the engine and bound back to the inode that lookup found. The binding only set `Companion::inode`: the export's table never learned the name, so `orphans()` still read the companion as an inode holding no name at all, and the *next* reclamation dropped it and lost its attributes. The deciding visit also took one engine lookup reference that the export never recorded and never gave back, so every bound companion leaked one. The successful lookup now goes through `Kernel::entry_taken`, the same call `Kernel::lookup` and every creating mutation make: it remembers the name, takes the one reference the export keeps per serial (forgetting a second) and records the directory. The bound companion is then live for the next reclamation and its reference is given back with the rest in `Kernel::release`. Review: finding P1-1 (the fix for the reclamation lookup was incomplete and leaked a reference). Tests: new `a_companion_bound_back_to_its_main_file_survives_the_next_reclaim`: one slot, a companion for a main file in the initialized tree that this export never looks up, one reclamation that binds it (the export's lookup count grows by exactly one) and a second one with the bound still full, during which the companion keeps its slot, its bytes and its attributes and the export's count does not move; then the main file's last name goes and the companion with it. Anti-proof: without `entry_taken` the test fails at that count (`left: 1, right: 2`), and with the count assertions out of the way it fails on the second reclamation, which now succeeds because it dropped the live companion (`left: Ok(()), right: Err(28)`). Production LOC: 164794 -> 164797 (delta +3) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent c94693a (164794) and of the staged tree (164797). Engine +3: the registration call and the comment block around it. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
… names (lp73) macOS 27.2, build 525fcfb, reproducible 20/20 and 5/5: a file carrying attributes, `ln a b`, then `rm a`, leaves `b` with no attributes at all and `._b` answering `NFS3ERR_NOENT`, while the companion is still in the export's index. The companion was the inode's, so nothing about the inode changed; what the client sent is what removed it. Reproduced in the rig with the client's sequence: `/bin/ln a b` links the main file, then the client drops the `._` name of the name it no longer uses with a REMOVE of `._a`, and REMOVE took the whole companion: `held().companions` 1 to 0, `._b` `NFS3ERR_NOENT`, the attributes gone. A client that carries the attributes to the new name also sends a LINK of the companion name (`._a` to `._b`), which this export refused with `NFS3ERR_ACCES`. - A companion now holds every `._` name of the inode it was linked under (`Companion::also`, indexed and unindexed with its own name, one name promoted when the recorded one goes). - `companion_link` serves LINK of a companion serial: another `._` name of the same companion, `NFS3ERR_EXIST` for a name already held or another companion's, `NFS3ERR_ACCES` for a name of the Workspace. - `companion_remove` drops one name while the companion holds another, or while its inode has another name the export knows (`Table::names` plus `Table::aliases`); only the last name of a one-name inode is the client clearing the attributes, which still takes the companion, as does any name of a companion no inode owns. Review: macOS re-test, finding 1 (hard link then remove the original name). This is the same rule the hard-link contract already stated, which the name scoped REMOVE broke. Tests: two new. `a_hard_link_keeps_the_companion_under_its_other_names`: main file and attributes, LINK, REMOVE `._a`, then REMOVE `a`; `._b` reads the same companion and bytes at every step and the companion goes with `b`. `a_linked_companion_name_is_another_name_of_the_same_companion`: LINK of the companion name succeeds and reads the same bytes, a held name answers `NFS3ERR_EXIST`, a plain name `NFS3ERR_ACCES`, REMOVE of `._a` keeps `._b`, and REMOVE of a one-name inode's only `._` name still removes the companion. Anti-proofs: with the keep rule forced off both tests fail (`left: 0, right: 1` companions, the attributes gone), and with LINK of a companion back to `NFS3ERR_ACCES` the second fails on that LINK. `cargo test -p layerfs-nfs`: 71 passed, 0 failed (raw sum 72 with the child-process rerun). Production LOC: 164797 -> 164884 (delta +87) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent f6cb678 (164797) and of the staged tree (164884). Engine +87: the companion's extra names and the three index operations on them, `companion_link`, the name-count rule in `companion_remove`, and the routed LINK. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
…(lp73) macOS 27.2, build 525fcfb: in one run of 600 files carrying ~100 KiB of attributes each, the first 507 in write order read back with no attributes at all while the export still held all 652 companions, 326 bodies stayed on disk, and two re-runs of the same recipe plus two controls were 100% correct. Nothing in the export's own counts had moved, so the loss was not an eviction: either those companions were still held but no name reached them, or the client never got them back for a reason of its own. The static review found no path that removes a companion's name keys and keeps the companion - every one of them takes the companion with it, or leaves it reachable through its inode - so the run could not be reproduced from the code, and a loss with the index intact has to be told apart from one where the index lost the entry. `Companions::unreachable` counts the companions neither their recorded name nor another name of their inode reaches, and the export's release line now carries it: `held= slotted= decoded= resident_bytes= unreachable=`. It runs once, over the companions held at release, so it costs nothing on the serving path. A non-zero count beside a client that lost attributes says the name, not the body, was lost; a zero says the body itself was gone. Documented with the release line in the nfs route's AppleDouble section. Review: macOS re-test, finding 2 (507 of 600 attributes lost, not reproduced). Tests: none new. The counter can only be non-zero through the very defect it reports, so an ordinary test would assert zero; the release path is covered by every test that stops an export (the line is printed with `held=`). Production LOC: 164884 -> 164900 (delta +16) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent 45d0903 (164884) and of the staged tree (164900). Engine +16: the counting method and the release line's field. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
The `HELD` bound was decided in one place, but on the path that needs I/O between the decision and the commit - a WRITE, SETATTR or RENAME whose body spills and is then edited on its file - `admits` passed before `spill_gated` and the file write, and `record_edit`/`classify` took the slot afterwards without asking again. Two concurrent calls could both be told the last slot was free and both take one, so `slots.len()` could exceed `held_limit` by as much as the number of edits in flight, which contradicts "the bound holds". `Companions::reserve` decides and holds in one step under the companion lock (`Slot::Taken`/`Held`/`Full`); reservations count against the bound in `admits`, and a create or reclaim cannot steal one. The disk-edit path and `rename_companion` reserve before their I/O and give the reservation back if the edit does not land (`unreserve`, `take_serial`, the failure paths of `apply_edit`/`record_edit`/`rename_companion`), so a reservation can neither be held for good nor outlive its companion. CREATE and the in-memory edit decide and take the slot under one lock and keep using `admits`. Review: finding P2-1 (the bound could be exceeded under concurrency), and the route document's claim that the bound holds is now true as written. Tests: new `the_slot_bound_holds_when_writes_claim_the_last_slot_together`. Eight companions holding whole images (no slot), one slot, a body budget that cannot hold the broken body, and one breaking WRITE from each over its own connection after a barrier: exactly one writer takes the slot, the other seven answer `NFS3ERR_NOSPC`, `slotted` is 1, and every companion still reads back as its image or that edit. Anti-proof: with `admits` back in place of `reserve` the test failed 23 of 40 runs with two or three writers succeeding (`[Ok(()), Err(28), Ok(()), ...]`); with the reservation it passed 40 of 40. Production LOC: 164900 -> 164930 (delta +30) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent c96af5b (164900) and of the staged tree (164930). Engine +30: the reservation set, `Slot`, `reserve`/`unreserve`, the two decision sites and the three release paths. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
Reclamation asks the engine about a candidate's main file before dropping it, and treated only "no such file" as dead. A main file whose directory the client had removed cannot answer "no such file": the engine has no such parent, so the lookup answers `NFS3ERR_STALE` (`NFS3ERR_NOTDIR` for a parent that is a file), and the candidate was kept for the export's life. Its slot stayed taken, every later CREATE at the bound repeated the same failing lookup, and the bound could stay full until release. `NFS3ERR_STALE` and `NFS3ERR_NOTDIR` now count as "the main file is not there either" and the candidate is reclaimed, which is what a removed parent means for its entries. Any other answer still leaves the companion alone, and the per-name engine lookup and the slot walk are unchanged. Review: finding P2-3 (a candidate whose main file's directory is gone is never reclaimed). Tests: new `a_companion_of_a_removed_directory_is_reclaimed`: one slot, a companion inside a directory whose main file `x` never existed (so `inode: None` and the slot), the client removes the directory (`rmdir` succeeds - nothing in it is visible to the engine), then a CREATE at the bound: its reclamation's lookup answers `NFS3ERR_STALE`, the companion is dropped and the CREATE lands with one companion and one slot. Anti-proof: with only `NFS3ERR_NOENT` counted the same CREATE answers `NFS3ERR_NOSPC` (`Err value: 28`) and the test fails there, so the lookup really answers a stale parent rather than a missing entry. Production LOC: 164930 -> 164932 (delta +2) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent 386c5b0 (164930) and of the staged tree (164932). Engine +2: the extra answers in the reclaim's match. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
…vershoot (lp73) Two statements about the resident budget were wrong in the direction of claiming more than the code does. A decoded image's `Image::cost` against `RESIDENT_BYTES` is a cap on what one decoded body is charged, not a limit on what a companion may store: an image costing more than `DECODED_ENTRY_BYTES` is kept as the client wrote it and charged at its length. And an edit that leaves a body whole decodes it as it records it rather than asking for room first, so the pool may stand one decoded body over the budget until the next edit needs room - "never memory the budget cannot see" was too strong. The module documentation, the `DECODED_ENTRY_BYTES` item and the `RESIDENT_BYTES` row of the nfs route now say that. The `DECODED_ENTRY_BYTES` value stays 128 KiB: with the images a macOS client lays out it is what keeps a many-attribute image from being charged a whole 4 KiB unit's worth of body for a few dozen bytes of attributes. Review: finding P3-1 (the wording of the un-budgeted decode) and the D3 note on `DECODED_ENTRY_BYTES` (owner confirmation is asked for in the pull request). Tests: none (documentation only). `cargo test -p layerfs-nfs`: 73 passed, 0 failed (raw sum 74 with the child-process rerun). Production LOC: 164932 -> 164932 (delta 0) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent 0b87716 (164932) and of the staged tree (164932). Engine 0: documentation comments only. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
A companion can hold several `._` names of its inode (a LINK of a companion name adds one), but reclamation's candidate test read only the name a companion records. A companion the client never paired with a main inode (`inode: None`) that was then linked under a second name was therefore proposed as an orphan on the strength of the first name alone: the export looked up that name's main file, answered "no such file", and dropped a companion that its other name - whose main file the export does know - still reached. One linked name lost the attributes of the other. `orphans()` now walks every name a companion holds: one whose main file a name of the index holds keeps the companion, and only a companion none of whose names the index knows becomes a candidate, for the lookup of the first such name. The slot walk, the engine lookup and the decision that follows are unchanged. Tests: new `a_companion_reached_by_one_of_its_names_is_not_an_orphan`: one slot, a companion created in a directory for a main file that does not exist (`inode: None`), LINKed under a second `._` name, then the main file of that second name created; a CREATE at the bound reclaims, finds the companion reachable through the linked name and leaves it, so the CREATE is refused and the companion still reads back. Anti-proof: with the recorded name alone the same CREATE lands (`left: Ok(()), right: Err(28)`) because the live companion was reclaimed. `cargo test -p layerfs-nfs`: 74 passed, 0 failed (raw sum 75 with the child-process rerun). Production LOC: 164932 -> 164941 (delta +9) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent d114c13 (164932) and of the staged tree (164941). Engine +9: the loop over a companion's names in `orphans`. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
A macOS client that cannot read a companion's attributes looks the same from
the export whatever happened: a name it never asked for, a name the export's
index could not resolve, and an answer its own caches then dropped all leave no
trace behind. A run of the real client is the only way to tell them apart, so
this adds an opt-in trace of exactly those requests.
`LAYERFS_NFS_TRACE=1` (read once per process, off unless it is exactly `1`)
prints one line per request that names an AppleDouble companion - the `._`
names themselves and the main file names of the inodes that hold one - to the
daemon's standard error:
layerfs-nfs trace: op=LOOKUP dir=6 name="._b" main=Some(9)
route=inode companion=Some(4611686018427387904) status=Ok
`main` is the main inode the export's name index gave the name (or `None`), and
`route` says which part of the companion index answered - the companion's own
name key, the main inode, or neither - so a name no companion answers is
distinguishable from one whose main file the index forgot. A CREATE that finds
the companion already held reports whether it would truncate it, a SETATTR of a
`._` file is traced by handle with its size, `BIND` reports a main name the
kernel table re-attributed to an inode with a companion (`alias` while the
inode keeps several names, `unbound` when the one it held is all it has),
`UNPAIR` reports a main name whose paired `._` name the export dropped, and
`RENAME`/`LINK`/`REMOVE`/`CREATE` report the companion they decided. Nothing
else changes: no state is touched, and every hook costs one `OnceLock` load
while the trace is off.
Tests: no new test - the trace changes no behaviour and the hooks are on paths
`cargo test -p layerfs-nfs` (74 passed, 0 failed, raw sum 75) already drives;
checked by hand against a LINK'd companion's `._` names, where the lines above
are the actual output. `lf-ci.sh` fast: 2017 passed, 0 failed.
clippy -p layerfs-nfs --all-targets: no warning.
Production LOC: 164941 -> 165152 (delta +211)
Scope: first-party core/engine/control/api production source and shipped
runtime SQL; tests, docs, tools and manifests excluded. Method:
python3 tools/production_loc.py (SHA-256
d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6)
on the first parent 54b9c1b (164941) and on the staged tree (165152). Engine
+211: `trace.rs` and its hooks in `apple_double.rs`, `edit.rs`, `kernel.rs` and
`change.rs`. Reference absent (retired R9c; last counted 65,417 at tree
498dd19).
`Kernel::companion_drop_for` dropped the `._<name>` key when a main file name was removed, so `rm a` of a file `b` still named answered `._a` with NFS3ERR_NOENT although the inode and its companion were alive (a macOS 27.2 trace showed the shape three times in one matrix). A companion that belongs to an inode now keeps that `._` name until the inode's last name, which already takes the companion with every `._` name it holds (`companion_drop_inode` -> `take_serial` -> `unindex`); a companion no main inode owns goes as before. A new file under the freed name reaches the old companion nowhere: `Companions::resolve` takes a name key only for the inode its companion belongs to. `Holding` and `Released::companions_unreachable` carry the release count, so a status read reports the same number as the release line. Tests (tests/apple_double_decode.rs): the matrix's g2/g3/g5 orders (`ln`, `rm` before and after one read of the linked name), a new file under the removed name, and the inode's last name reclaiming the companion and every `._` name key, with `._a`/`._b` reading one whole image. The three order tests failed on the parent with `._a` = NFS3ERR_NOENT. Docs: nfs.md states the hard-link rule, `UNPAIR name-kept`, the native route's `env_clear` trace wrapper and the live `unreachable` count. Production LOC: 165152 -> 165155 (delta +3); scope: first-party product source under core|engine|control|api, tools/production_loc.py on the parent tree and the staged tree; reference absent (retired R9c; last counted 65,417 at tree 498dd19).
No conflicts. Upstream since `8cfb6e273` brought the standalone fuser Git fork (Ephemeral-AI-Lab#686, which removes `engine/vendor` and `tools/check_fuser_integrity.py`) with its pin guard (Ephemeral-AI-Lab#687), and the GC spec items (Ephemeral-AI-Lab#688). None of those files is this layer's, so the merged tree is upstream's plus this branch's commits. Production LOC: 163901 -> 163901 (delta 0) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded (the removed vendored fuser is third-party code and never counted). Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent 428ffc3 (163901) and of the merged tree (163901). Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
… (lp73) The codec layer now carries `upstream/layerfs_v2_0_0` at `cb8ced9d3`; that merge had no conflicts, and neither does this one. Upstream's changes since `8cfb6e273` (the standalone fuser Git fork, which removes `engine/vendor` and `tools/check_fuser_integrity.py`, its pin guard, and the GC spec items) touch no file of this layer, so the result is the upstream tree plus this layer's commits. Production LOC: 164405 -> 164405 (delta 0) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded (the removed vendored fuser is third-party code and never counted). Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent aec83af (164405) and of the merged tree (164405). Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
The macOS NFS client answers a `._` name that is absent from a directory's cached READDIRPLUS listing with "not found" without ever asking the server (the skipdu path in nfs_vnop_lookup), and this export serves AppleDouble companions without listing them, so a hard-linked file's extended attributes could read back empty. Turning READDIRPLUS off makes every `._` name arrive as a LOOKUP. Nothing else about the mount or the listing changes. Production LOC: 165155 -> 165155 (delta 0); core/engine/control/api production source only (api/crates/layerfs-*/src, api/python/src/layerfs, api JS packages), counted with tools/production_loc.py at the parent and the committed tree; comments, tests and docs excluded.
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
5742338
jiangkoumo
pushed a commit
to jiangkoumo/layerfs
that referenced
this pull request
Nov 17, 2026
…36 per export (lp73) Merges the PR head 1cdb96a. One conflict, the tail of Kernel::lookup in engine/crates/layerfs-nfs/src/kernel.rs: Ephemeral-AI-Lab#637 returns the job's directory row with the entry (`Answer`), Ephemeral-AI-Lab#722 traces the main file; the resolution keeps both. Kernel::reclaim_orphans adapts to Ephemeral-AI-Lab#637's `observe`, which now answers `Answer` / `Refused` instead of a tuple / `Status`, with no change in what it decides. Review fixes follow in their own commits. Production LOC: 170751 -> 171501 (delta +750) core 39,795 -> 39,795 (0) engine 76,174 -> 76,924 (+750) layerfs-nfs +750 control 48,436 -> 48,436 (0) api 6,346 -> 6,346 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd19) Method: tools/production_loc.py --detail on first parent efa138d and the merged tree (merge: first-parent comparison) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jiangkoumo
pushed a commit
to jiangkoumo/layerfs
that referenced
this pull request
Nov 17, 2026
…meral-AI-Lab#722 review) Ephemeral-AI-Lab#722 left engine/crates/layerfs-nfs/src/apple_double.rs at 1,272 physical lines, and tools/check_product_boundary.py failed on it. This is relocation only, with no behaviour change: - `impl Companions` (names, inodes, slots, resident budget, eviction order) moves to `apple_double/index.rs`; - `Kernel::orphans` and `Kernel::reclaim_orphans` move to `apple_double/reclaim.rs`; - moved methods become `pub(super)`, and each file imports what it uses. The moved lines are otherwise byte-identical (checked by comparing the sorted non-import lines before and after). apple_double.rs is now 809 lines, index.rs 345, reclaim.rs 141. The boundary check passes (1,417 files). Docs: nfs.md's AppleDouble section lists the two new files. Tests: none changed. Production LOC: 171,498 -> 171,516 (delta +18; relocation: the new files' own imports) core 39,795 -> 39,795 (0) engine 76,921 -> 76,939 (+18) layerfs-nfs +18 control 48,436 -> 48,436 (0) api 6,346 -> 6,346 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd19) Method: tools/production_loc.py --detail on parent 9db8f4d and the staged tree Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jiangkoumo
pushed a commit
to jiangkoumo/layerfs
that referenced
this pull request
Nov 17, 2026
…ion (Ephemeral-AI-Lab#722 review) Kernel::reclaim_orphans decides outside both locks: each candidate whose main file is unknown costs one engine lookup, and the commit then took the lock again and re-checked only the companion's inode and slot. A call that made the companion live in that window (a RENAME of `._x` onto a name whose main file exists, a CREATE of the main file followed by a WRITE to `._x`, a READ) did not stop the drop. The binding half could also overwrite another companion's entry for the same main inode. `Orphan` now carries the companion's `used` tick from the decision, and `Companions::unchanged` requires it, the same inode and the slot still taken, both before dropping a companion and before binding one back. A main inode that already has its companion is not rebound; the candidate stays and is looked up again at the next reclaim. Every call that reaches a companion moves its tick. Review finding rejected after testing: the review said a lookup in a directory whose reference the export had collected would answer `Stale` and so drop a live companion. It does not: the engine answers the lookup in a collected directory. The new test collects the directory (a later GETATTR of it resolves it again) and passes both before and after this change, so it is coverage, not a regression test. Test: new `apple_double_decode.rs::a_companion_in_a_collected_directory_is_kept_while_its_main_file_is_there`. The race itself has no deterministic client-side reproduction (the window is between the out-of-lock engine lookup and the commit), so no test reproduces it. Production LOC: 171,516 -> 171,525 (delta +9) core 39,795 -> 39,795 (0) engine 76,939 -> 76,948 (+9) layerfs-nfs +9 control 48,436 -> 48,436 (0) api 6,346 -> 6,346 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd19) Method: tools/production_loc.py --detail on parent fb28450 and the staged tree Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jiangkoumo
pushed a commit
to jiangkoumo/layerfs
that referenced
this pull request
Nov 17, 2026
…phemeral-AI-Lab#722 review) - RENAME of one of a companion's names moves only that name. `rename_companion` never got the source name: it overwrote the companion's primary name and kept every LINK-made name, the renamed one included. With `._a` linked as `._b`, `mv b c` and `._b -> ._c` left `._b` answering and dropped `._a`'s own key. A companion reached only through its inode keeps the names it holds and gains the target. - A slot reservation is never left for a companion that is gone. `Companions::reserve` takes nothing for a serial another call removed (a REMOVE takes no edit gate), and `record_edit` gives the reservation back before it can find the companion gone. Either path used to leave the bound one slot smaller for the life of the export. - macOS mounts no longer add `nordirplus` (owner decision 2026-11-17). This reverts 46550ff: the loss it targeted reproduced 0/5 with and without the option, and every READDIR then costs a LOOKUP per entry. The hard-link loss with READDIRPLUS on stays an open, unconfirmed risk. Review finding not taken: making REMOVE of `._a` drop the whole companion while `a` still exists (to serve `xattr -c` on a hard-linked file). The client's own sequence for `ln a b` removes `._a` while `a` exists and expects the attributes under `b` to stay. That is the macOS loss Ephemeral-AI-Lab#722 fixed, and `a_hard_link_keeps_the_companion_under_its_other_names` pins it. Docs: nfs.md, the companion names paragraph and the macOS client options. Test: new `apple_double_decode.rs::renaming_one_linked_companion_name_moves_only_that_name`. It fails on the parent (3466c3d), where `._b` still answers, and passes here. The reservation race has no deterministic client-side reproduction. Production LOC: 171,508 -> 171,521 (delta +13) core 39,795 -> 39,795 (0) engine 76,931 -> 76,944 (+13) layerfs-nfs +13 control 48,436 -> 48,436 (0) api 6,346 -> 6,346 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd19) Method: tools/production_loc.py --detail on parent 3466c3d and the staged tree Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jiangkoumo
added a commit
to jiangkoumo/layerfs
that referenced
this pull request
Nov 17, 2026
Bring the Ephemeral-AI-Lab#136 native unmount helper reaping and detach-stage custody work onto upstream/layerfs_v2_0_0 (5742338), including Ephemeral-AI-Lab#712/Ephemeral-AI-Lab#715/Ephemeral-AI-Lab#716/Ephemeral-AI-Lab#718 and the lp73 AppleDouble companion/spill series (Ephemeral-AI-Lab#713/Ephemeral-AI-Lab#714/Ephemeral-AI-Lab#722 via Ephemeral-AI-Lab#727). Conflicts resolved keeping both sides' behavior intact: - control/crates/layerfs-sandbox-backend/src/local/os/macos/darwin.rs: Keep upstream's Ephemeral-AI-Lab#712 `mount_line` and deadline-bounded `mount_table` together with Ephemeral-AI-Lab#644's shared `layerfs_bridge::process::Retained` queue (`RETAINED`), dropping the duplicate `Mutex<Vec<Child>>` / `reap_retained` / `retain` helpers so both `mount_table` and `force_unmount` reap through `RETAINED` (`reap`, `record` on synchronous reap, `retain` for background reaping on timeout). In `force::close` (Ephemeral-AI-Lab#716) and `machine_force::stop` (Ephemeral-AI-Lab#718), `exec::cancel::settle` cancels running workspace commands first, before `Sandbox::stop` -> `os::mounts::detach` -> `darwin::force_unmount`, avoiding overlap with `RETAINED`. - engine/crates/layerfs-daemon/src/control/export.rs: Keep both `ExportServing::is_mounted` (Ephemeral-AI-Lab#644) and `companion_spill_dir` / `spill_root` (Ephemeral-AI-Lab#714), combine `.appledouble` leftover cleanup with the `mount_table`-aware `mounted_under` call in `ExportServing::reclaim`, and preserve `companion_spill_dir` in `ExportConfig` and `spill_root` rmdir after `ready.directory` removal in `teardown_export` (while `server.shutdown()` -> `Kernel::release()` removes the per-namespace spill dir first). Checks: `tools/check_product_boundary.py` PASS; `tools/test_*.py` 180/180; `lf-ci.sh fast` 2258/2259 (1 pre-existing upstream Ephemeral-AI-Lab#724 failure in `layerfs-api::machine_files`, reproduced identically at 5742338); `layerfs-nfs::helper_deadline` 8/8, `daemon` 2/2, `export` 10/10, `apple_double_codec` 7/7, `apple_double_decode` 32/32; `layerfs-sandbox` `record_codec` 5/5 & `unmount_record` 9/9; `layerfs-daemon` non-mount 31/31 & selected mount lane 12/12 binaries (36 passed, 1 ignored helper); Clippy clean on changed crates; `rustfmt --check` clean on all changed files (`fmt --all --check` fails only on 3 pre-existing upstream files at 5742338). `aarch64-apple-darwin` check blocked on Linux host C toolchain (`blake3` `-arch arm64`): macOS build and live tests NOT_RUN. Durable: NOT_RUN — disabled by owner until explicit reauthorization. Production LOC: 163827 -> 171849 (delta +8022); first parent 89116b1. core 39787 -> 39795 (+8), engine 75121 -> 77198 (+2077), control 42842 -> 48510 (+5668), api 6077 -> 6346 (+269). Reference absent (retired R9c; last counted 65,417 at tree 498dd19). Method: python3 -B tools/production_loc.py --detail on git archive of first parent 89116b1 and the merged working tree.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tier: P1 (engine/crates/layerfs-nfs, engine/crates/layerfs-daemon) ; risks: risk:kernel, risk:delete ; gate: yes: reclaims orphan companions at the HELD bound, changes the macOS mount options ; de-escalated: none
Docs: docs/architecture/engine/kernel-routes/nfs.md
Durable: NOT_RUN — disabled by owner until explicit reauthorization
Third of three PRs for #636 (AppleDouble companions beyond 4,096 per export).
It is stacked on the second PR, #714 (
lp636-spill, heade3c80ed1c), which is stacked on #713: this diff contains both earlier layers and has to be merged after them.Summary
The first two layers decide how a companion body is stored and where it goes when memory is full. This layer decides whom a companion belongs to and how many there can be.
HELDrises from 4,096 companions to 65,536 slotted entries. At the bound, orphan slotted companions are reclaimed first; a candidate whose main inode the export does not know is confirmed with one read-only engine lookup outside the locks (NFS3ERR_NOENT,NFS3ERR_STALEandNFS3ERR_NOTDIRmean gone; a file that is there binds the companion back throughKernel::entry_taken). Only then does CREATE answerNFS3ERR_NOSPC.Companions::admits) and held across I/O (Companions::reserve/unreserve), so concurrent WRITE/SETATTR/RENAME paths that spill cannot pushslotspast it.._name is served as another name of the same companion; REMOVE of one._name drops only that name while the inode keeps another; when a main name goes, the inode keeps its._<name>key until its last name.LAYERFS_NFS_TRACE=1traces only requests that name a companion (off: oneOnceLockload per hook). The release line addsslotted=andunreachable=.nordirplus. The macOS client answers a._name absent from a directory's cached READDIRPLUS listing with "not found" without asking the server (skipduinnfs_vnop_lookup), and this export never lists companions, so a hard-linked file's attributes could read back empty. With READDIRPLUS off every._name arrives as a LOOKUP. Nothing else about the mount changes.Evidence strength for
nordirplusThe hard-link loss it targets was seen in earlier runs (MAC4/MAC5) but has not reproduced in the last runs at this layer: 0/5 with the default options and 0/5 with
nordirplus. The change rests on the client source (apple-oss-distributions/NFS,kext/nfs_vnops.c,skipdurequiresRDIRPLUS) and on traces showing the server answers every._LOOKUP it receives correctly. It is not claimed as a proven fix. Listing companions in READDIR would remove the cause on the server side (with._names visible tols -a); that is left for the owner to decide.Validation
At
1cdb96a8c, on Linux, withlf-ci.sh(upstream fast lane, then nightly slow and mount lanes) andlf-ci-pg.sh:layerfs-persistence::concurrent_first_installs two_writers_saving_the_same_new_content_both_complete(Integrity("dependency cycle")). Pre-existing and unrelated: run alone 100 times it fails 7 times at this head and 4 times atlayerfs_v2_0_07fb04c7f7; this branch changes nothing undercore/crates/layerfs-storage,layerfs-contentorlayerfs-persistence.rc=0: 150 passed, 11 skipped. Mount lane: 45 binaries passed, 0 failed.rc=0,NOT_RUN=0.cargo +1.85.1 test --locked -p layerfs-nfs: 79 passed, 0 failed.clippy -p layerfs-nfs -p layerfs-daemon --all-targets -D warnings: clean.cargo fmt --check: onlyapi/crates/layerfs-cli/tests/concurrent_reads.rs, identical to the base branch's blob.a_hard_link_keeps_the_companion_under_its_other_names,a_removed_name_still_reaches_the_companion_of_its_inode,the_slot_bound_holds_when_writes_claim_the_last_slot_together,a_companion_of_a_main_file_the_name_index_missed_is_kept,a_companion_of_a_removed_directory_is_reclaimed,two_guarded_creates_of_one_companion_name_leave_one_entry.macOS proof
macOS 27.2 (arm64, M2), kernel NFSv3, release build of
46550ff30(the only change to1cdb96a8cis a conflict-free merge of #714 that touches no file of this layer):nfsstat -mshowsnordirplusin effect. 50-file read-back 50/50, again 50/50 after changing the attributes; no._entry visible in listings.ls -lRwarm 0.65 s.unreachable=0.Risks
risk:kernel(the export's companion paths and the macOS mount options),risk:delete(reclamation drops orphan companions at the bound, only after the engine confirms the main file is gone). Not covered: listing companions in READDIR, and the hard-link loss path while READDIRPLUS is on (no longer the default on macOS).Production LOC
Base
e3c80ed1c(#714's head) = 164405 → head1cdb96a8c= 165155, +750.Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method:
python3 tools/production_loc.py --root <dir>on each tree (tool SHA-256d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6).Related to #636; with #713 and #714 it replaces #636.