Repository navigation
nfs: decode the AppleDouble image a macOS client writes (lp73) - #713
Merged
8 commits merged intoNov 17, 2026
Merged
8 commits merged into
8 commits merged into
Conversation
….rs (lp73) Relocation only, no behavior change: `Kernel::release`, `Kernel::held`, `retained_count` and the `Released` struct move out of `kernel.rs` into `kernel/released.rs` verbatim, and `pub use released::Released` keeps every existing path working. This is the first commit of the AppleDouble split (LP636): `kernel.rs` would otherwise cross the 999-line ceiling once the companion work lands. No test or documentation change; the moved functions are the same text in a new module. Production LOC: 163505 -> 163511 (delta +6) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent 8cfb6e2 (163505) and this tree (163511). The +6 is the new module's own two `use` lines, its `impl Kernel` block and the `mod released;` / `pub use released::Released;` declaration in kernel.rs. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
The client lays a `._<name>` companion out as an AppleDouble v2 image: a header, a FinderInfo entry that runs into an `ATTR` table of extended attributes, and a resource-fork entry at the end. `apple_double_codec` reads those fields (`Image::parse`) and, when the fields encode back to the very bytes the client wrote (`Image::compact`), keeps them as its own record, so no attribute, value or byte the layout carries is dropped, and one attribute's alignment padding survives a shorter name replacing a longer one. `Image::new`/`Image::encode` lay an image out as the client does, which is what makes the round trip exact and what the macOS evidence compares against. Malformed, truncated or differently laid-out images, and a resource fork with data, parse to `None` and are left as the client wrote them. This is the first layer of the AppleDouble split (LP636): the codec alone, with its own unit tests; the companion that holds its decoded form comes next. Tests: new `tests/apple_double_codec.rs`, 6 cases (byte-exact round trip for every value size, many attributes, alignment padding, whole units, a resource fork, and malformed images that must not decode), all pass. Production LOC: 163511 -> 163792 (delta +281) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent 1810b4e (163511) and this tree (163792). Engine +281 (`apple_double_codec.rs` +280, `lib.rs` +1). Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
A macOS client writes each companion once, in full, and reads it back unchanged. A body is therefore classified whenever a write or a resize leaves it: one that parses back to the very bytes the companion holds (`Image::compact`) and fits `DECODED_ENTRY_BYTES` is kept as its FinderInfo and attributes and counted against `DECODED_BYTES` instead of its image, and every READ synthesizes the same bytes again. Anything else - a resource fork, another layout, a half-written image - is stored as the client wrote it. This replaces `TOTAL_BYTES` with two budgets that say what they bound: `RESIDENT_BYTES` (32 MiB) of written bodies kept in memory, and `DECODED_BYTES` (512 MiB) of decoded ones counted as `Image::cost`. A write or resize that neither decodes nor fits the resident budget answers `NFS3ERR_NOSPC` and leaves the companion exactly as it was; the next layer spills those bodies to disk instead of refusing them. `Released` now reports `companions_decoded` and `companion_decoded_bytes`, and `companions_release`/`companions_held` answer the `Holding` record the counters live in. Second layer of the AppleDouble split (LP636), on top of the codec. Tests: new `tests/apple_double_decode.rs` with the client's own write sequence (a half-written header, the rest, an edit that breaks the image and one that completes it again, truncate below and resize back up), read back byte for byte. Full `layerfs-nfs` suite: 41 passed, 0 failed. Production LOC: 163792 -> 163886 (delta +94) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent 210d5f8 (163792) and this tree (163886). Engine +94. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
The NFS route guide now names `apple_double_codec.rs` and states what each bound actually holds: `RESIDENT_BYTES` for written bodies and `DECODED_BYTES` / `DECODED_ENTRY_BYTES` for the decoded form, replacing the `TOTAL_BYTES` row the code no longer has. The paragraph after the table says which bodies decode (complete images that encode back to the same bytes) and which stay as written. Documentation only; it describes the codec layer as it stands. Production LOC: 163886 -> 163886 (delta 0) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent fa5dd34 (163886) and this tree (163886). Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
…s for (lp73) A decoded companion was re-encoded in full under the companion lock on every READ: up to `FILE_BYTES` (1 MiB) allocated and copied to serve a reply of at most `TRANSFER_WINDOW` (256 KiB) bytes, and often far less. `Image::encode_range` lays the image out and fills only `[start, end)`, so a READ pays for the bytes it answers with; `Image::encode` is that same layout over the whole image, so there is one implementation of the layout rather than two that can drift. `edit.rs`'s spill and `edit_resident` still need the whole image and keep calling `encode()`. Whether a READ is served from a window or from the whole image is not observable: the reply bytes are the same either way. Review: finding 7 (full re-encode on every READ). Tests: new `any_window_of_the_image_encodes_to_that_window` compares every window of an image (a padding byte the client left behind, values of several sizes, the empty resource fork) with the same slice of `encode()`, including empty windows, a window past the image and one that starts after it ends. `cargo test -p layerfs-nfs --test apple_double_codec --test apple_double_decode`: 7 + 1 passed, 0 failed. Anti-proof: with `encode_range` clamped to start at 0 again the window test fails (`left` starts at image offset 0 while the expected window starts mid-image). Production LOC: 163886 -> 163908 (delta +22) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent 584cc3e (163886) and of the staged tree 20b10a3 (163908). Engine +22. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
…t (lp73) `DECODED_BYTES` (512 MiB per export) was memory nothing could evict: many Workspaces can be mounted at once, so a per-export budget that is never paid back does not fit. Decoded images now take the same `RESIDENT_BYTES` (32 MiB) as bodies kept as written, a body at its length and a decoded one as `Image::cost`, so the budget sees every byte the export holds. `Companion::resident_cost` is that one number; `attach`, `detach` and `edit_body` count it, `edit_body` decides with it (a body either fits or the whole edit answers `NFS3ERR_NOSPC` and leaves the companion as it was), and `classify` no longer consults a second budget. `DECODED_ENTRY_BYTES` stays, now as the most `Image::cost` one decoded companion may be counted for: a whole image costing more keeps the client's bytes, so one companion cannot take the whole pool. This layer has no disk to spill to, so a body the budget does not take is still refused; the next layer spills it. `Released` reports the merged pool as `companion_bytes` and keeps `companions_decoded` as a count, instead of `companion_decoded_bytes` beside a separate resident total. Review: owner decision 3 (`DECODED_BYTES` rejected), first half (the budget and the counts); the spill half is the next layer. Tests: `the_clients_write_sequence_is_served_byte_for_byte` now asserts the merged pool: the decoded image's `Image::cost()` after the image arrives, the written body's length while it does not decode, the cost again once it does, and zero after the companion is removed. `cargo test -p layerfs-nfs`: 42 passed, 0 failed. Anti-proof: with a decoded body counted as zero again the test fails at the first assertion (`left: 0, right: 4372`). Production LOC: 163908 -> 163896 (delta -12) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent b01e805 (163908) and of the staged tree bfc489b (163896). Engine -12. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
…p73) `cargo fmt --check` reported three files of this layer: the `emit` call rustfmt wraps in `Image::encode_range`, and two assertions of the new codec and decode tests. This is that pass and nothing else. The one remaining fmt report, `api/crates/layerfs-cli/tests/concurrent_reads.rs`, is not touched by this branch (`git log 8cfb6e2..HEAD -- <file>` is empty) and is left alone. Checks: `cargo +1.85.1 fmt --all -- --check` reports only that file; `cargo +1.85.1 clippy --locked -p layerfs-nfs --all-targets -- -D warnings` is clean; `cargo test -p layerfs-nfs` passes every binary. Production LOC: 163896 -> 163901 (delta +5) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded. Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent 7ef50a0 (163896) and of the staged tree (163901). Engine +5: rustfmt splits the `emit` call, which the counter counts. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
No conflicts. Upstream since `8cfb6e273` brought the standalone fuser Git fork (Ephemeral-AI-Lab#686, which removes `engine/vendor` and `tools/check_fuser_integrity.py`) with its pin guard (Ephemeral-AI-Lab#687), and the GC spec items (Ephemeral-AI-Lab#688). None of those files is this layer's, so the merged tree is upstream's plus this branch's commits. Production LOC: 163901 -> 163901 (delta 0) Scope: first-party core/engine/control/api production source and shipped runtime SQL; tests, docs, tools and manifests excluded (the removed vendored fuser is third-party code and never counted). Method: python3 tools/production_loc.py (SHA-256 d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6) on git archives of the first parent 428ffc3 (163901) and of the merged tree (163901). Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
This was referenced Nov 16, 2026
Closed
5742338
jiangkoumo
pushed a commit
to jiangkoumo/layerfs
that referenced
this pull request
Nov 17, 2026
…nt writes (lp73) Merges the PR head ff28f61 unchanged; review fixes follow in their own commits. Production LOC: 169851 -> 170247 (delta +396) core 39,795 -> 39,795 (0) engine 75,274 -> 75,670 (+396) layerfs-nfs +396 control 48,436 -> 48,436 (0) api 6,346 -> 6,346 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd19) Method: tools/production_loc.py --detail on first parent e7349ff and the merged tree (merge: first-parent comparison) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jiangkoumo
pushed a commit
to jiangkoumo/layerfs
that referenced
this pull request
Nov 17, 2026
- SETATTR of a companion runs as one edit. It checked its guard under one lock, resized under a second and set mode/mtime under a third, so a guarded SETATTR could pass its ctime check and resize a body a WRITE had changed in between. It now holds the companion's edit gate from the guard to the mode and time, and a WRITE reads its pre-op attributes under the gate (`edit_gate` + `edit_gated` replace `edit_companion`). A REMOVE takes no gate, so a SETATTR racing one can still answer `Stale` after resizing a body that is gone with its companion; nothing is left behind. - A whole image is decoded only when its decoded form costs less than its bytes (`worth_decoding`). Each attribute is charged its name, value and 56 bytes, so an image of many small attributes cost several times its length decoded, could be refused room its bytes had, and was charged more than it held. - An edit of a decoded body asks for the room its bytes add beyond the decoded charge it gives back, not for its whole length, so overwriting or shrinking one no longer spills more than it needs. - `Image::parse` bounds its first allocation by what the image can hold rather than by the client's entry count (up to 65,535 entries). The resident budget stays at 32 MiB (owner decision 2026-11-17); Ephemeral-AI-Lab#713's description called it upstream's, but the base branch had 64 MiB. The architecture page already says 32 MiB. Not changed: every WRITE to a decoded body still re-encodes and re-parses the whole body under the companion lock; that needs an in-place decoded edit. Docs: nfs.md (the DECODED_ENTRY_BYTES row, and the SETATTR and WRITE gate) and the module docs of apple_double.rs. Test: new `apple_double_decode.rs::an_image_whose_attributes_cost_more_than_its_bytes_is_kept_as_written` (600 two-byte names). It fails on the parent (77e7368), where the image is decoded (decoded count 1), and passes here. The SETATTR race has no deterministic client-side reproduction, so it has no test. Production LOC: 171,525 -> 171,523 (delta -2) core 39,795 -> 39,795 (0) engine 76,948 -> 76,946 (-2) layerfs-nfs -2 control 48,436 -> 48,436 (0) api 6,346 -> 6,346 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd19) Method: tools/production_loc.py --detail on parent 77e7368 and the staged tree Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jiangkoumo
added a commit
to jiangkoumo/layerfs
that referenced
this pull request
Nov 17, 2026
Bring the Ephemeral-AI-Lab#136 native unmount helper reaping and detach-stage custody work onto upstream/layerfs_v2_0_0 (5742338), including Ephemeral-AI-Lab#712/Ephemeral-AI-Lab#715/Ephemeral-AI-Lab#716/Ephemeral-AI-Lab#718 and the lp73 AppleDouble companion/spill series (Ephemeral-AI-Lab#713/Ephemeral-AI-Lab#714/Ephemeral-AI-Lab#722 via Ephemeral-AI-Lab#727). Conflicts resolved keeping both sides' behavior intact: - control/crates/layerfs-sandbox-backend/src/local/os/macos/darwin.rs: Keep upstream's Ephemeral-AI-Lab#712 `mount_line` and deadline-bounded `mount_table` together with Ephemeral-AI-Lab#644's shared `layerfs_bridge::process::Retained` queue (`RETAINED`), dropping the duplicate `Mutex<Vec<Child>>` / `reap_retained` / `retain` helpers so both `mount_table` and `force_unmount` reap through `RETAINED` (`reap`, `record` on synchronous reap, `retain` for background reaping on timeout). In `force::close` (Ephemeral-AI-Lab#716) and `machine_force::stop` (Ephemeral-AI-Lab#718), `exec::cancel::settle` cancels running workspace commands first, before `Sandbox::stop` -> `os::mounts::detach` -> `darwin::force_unmount`, avoiding overlap with `RETAINED`. - engine/crates/layerfs-daemon/src/control/export.rs: Keep both `ExportServing::is_mounted` (Ephemeral-AI-Lab#644) and `companion_spill_dir` / `spill_root` (Ephemeral-AI-Lab#714), combine `.appledouble` leftover cleanup with the `mount_table`-aware `mounted_under` call in `ExportServing::reclaim`, and preserve `companion_spill_dir` in `ExportConfig` and `spill_root` rmdir after `ready.directory` removal in `teardown_export` (while `server.shutdown()` -> `Kernel::release()` removes the per-namespace spill dir first). Checks: `tools/check_product_boundary.py` PASS; `tools/test_*.py` 180/180; `lf-ci.sh fast` 2258/2259 (1 pre-existing upstream Ephemeral-AI-Lab#724 failure in `layerfs-api::machine_files`, reproduced identically at 5742338); `layerfs-nfs::helper_deadline` 8/8, `daemon` 2/2, `export` 10/10, `apple_double_codec` 7/7, `apple_double_decode` 32/32; `layerfs-sandbox` `record_codec` 5/5 & `unmount_record` 9/9; `layerfs-daemon` non-mount 31/31 & selected mount lane 12/12 binaries (36 passed, 1 ignored helper); Clippy clean on changed crates; `rustfmt --check` clean on all changed files (`fmt --all --check` fails only on 3 pre-existing upstream files at 5742338). `aarch64-apple-darwin` check blocked on Linux host C toolchain (`blake3` `-arch arm64`): macOS build and live tests NOT_RUN. Durable: NOT_RUN — disabled by owner until explicit reauthorization. Production LOC: 163827 -> 171849 (delta +8022); first parent 89116b1. core 39787 -> 39795 (+8), engine 75121 -> 77198 (+2077), control 42842 -> 48510 (+5668), api 6077 -> 6346 (+269). Reference absent (retired R9c; last counted 65,417 at tree 498dd19). Method: python3 -B tools/production_loc.py --detail on git archive of first parent 89116b1 and the merged working tree.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tier: P1 (engine/crates/layerfs-nfs) ; risks: risk:kernel ; gate: no ; de-escalated: none
Docs: docs/architecture/engine/kernel-routes/nfs.md
Durable: NOT_RUN — disabled by owner until explicit reauthorization
First of three PRs for #636 (AppleDouble companions beyond 4,096 per export),
split so each layer can be reviewed on its own. This one is the codec: the
export parses the AppleDouble image a macOS client writes into
._<name>,keeps the attributes, and re-encodes the exact bytes on READ. It adds neither
the spill area nor the name index, and does not need them.
Summary
macOS stores extended attributes and FinderInfo on a mounted volume by writing
an AppleDouble image into a
._<name>sidecar next to the file, then readingthat image back. The export is an attribute shim: those bytes never become a
Workspace file. This PR teaches it the format.
apple_double_codec:Image::parsereads the fields, andImage::compactaccepts a body only whenencode()reproduces the client'sbytes, so a body it decodes always has a byte-exact representation. Any other
layout, an image carrying a resource fork, and a truncated image are kept
exactly as the client wrote them, and edits to those bodies work as before.
attributes) instead of the raw bytes; READ synthesises the bytes the client
wrote. READ encodes only the window it is asked for (
Image::encode_range)rather than the whole image — a structural statement, nothing here was timed.
RESIDENT_BYTES(32 MiB, upstream's). A body kept as theclient's bytes is charged by its length; a decoded image is charged
Image::cost. An edit that does not fit is refused withNFS3ERR_NOSPCandleaves the body untouched (the second PR replaces that refusal with a spill).
DECODED_ENTRY_BYTES(128 KiB) is a per-decoded-body charge cap, not acapacity cap: an image that would cost more is kept as the client's bytes and
charged by their length, so it is never refused. This number comes from fix(nfs): spill AppleDouble companion bodies to disk and expand workspace capacity to 65,536 #636
and the owner did not name it; please confirm it stays at 128 KiB. The
architecture page documents it in those words.
Releasednow reportscompanions,companion_bytesandcompanions_decoded.1810b4e3dis relocation only:Kernel::release,Kernel::held,retained_countandReleasedmove fromkernel.rsintokernel/released.rswith no behaviour change, so the later layers change eachfile once.
layerfs_v2_0_0atcb8ced9d3inff28f6118;that merge had no conflicts and changed no file of this feature.
Validation
At
ff28f6118. Commands ran on Linux under this repository's CI runner(
lf-ci.shmirrorsci.ymlfast andnightly.ymlslow/mount). Cargo commandswere queued through the worktree's build lock; each test invocation had a
120 s wall timeout and none reached it.
rc=0: 1985 tests run, 1985 passed, 26 skipped.rc=0: 150 tests run, 150 passed, 11 skipped.layerfs-persistence,layerfs-store-service,layerfs-project,layerfs-api,layerfs-cli,layerfs-control) allrc=0; 39 tests passed;NOT_RUN=0.cargo +1.85.1 test --locked -p layerfs-nfs: 42 passed, 0 failed across 13binaries (
apple_double_codec7,apple_double_decode1,write8,daemon1,export5,protocol15,collect1,helper_deadline3,refresh1).cargo +1.85.1 clippy --locked -p layerfs-nfs --all-targets -- -D warnings:rc=0, no warnings.rustfmt --checkon every changed file: clean.python3 -B tools/check_product_boundary.py: PASS (1377 files scanned).any_window_of_the_image_encodes_to_that_window,values_of_every_size_round_trip,many_attributes_and_finder_info_round_trip,the_clients_alignment_padding_is_kept,a_resource_fork_is_left_as_the_client_wrote_it,malformed_images_are_not_decoded,an_image_the_client_would_write_has_whole_units,the_clients_write_sequence_is_served_byte_for_byte.macOS proof
Re-run on the final head after the upstream merge: macOS 27.2 (arm64, M2)
kernel NFSv3,
vers=3,tcp,soft,rsize=262144,wsize=262144,acregmin=60,acregmax=60,release build of the second PR's head
e3c80ed1c, which merges this head(
ff28f6118) unchanged:On the same Mac at that head,
cargo +1.85.1 clippy -p layerfs-nfs -p layerfs-daemon --all-targets -- -D warningswas clean andcargo +1.85.1 test -p layerfs-nfspassed 54/54.Risks
risk:kernel: this is the NFSv3 export's sidecar path, so a parsing mistakewould surface as wrong attribute bytes.
Image::compactis what bounds that —a body is decoded only if re-encoding it reproduces the client's bytes exactly,
and everything else is kept verbatim. Resident memory stays inside
RESIDENT_BYTES, and a refused edit leaves the body, its length and its name asthey were. Not covered here: the spill area and the name index (the next two
PRs), and macOS at 4,096+ companions.
Production LOC
Base
cb8ced9d3(layerfs_v2_0_0) = 163505 → headff28f6118= 163901,+396.
Scope: first-party core/engine/control/api production source and shipped
runtime SQL; tests, docs, tools and manifests excluded. Method:
git archive <rev> | tar -x -C <dir>, thenpython3 -I tools/production_loc.py --root <dir>(tool SHA-256d3148db88f91b09eb74eb659b2f700209776a24d022f09774dc7bb38ea1eecf6).Related to #636; it does not close it.