Repository navigation
Description
Description
When copying files within a native macOS loopback NFS workspace (e.g. running cp -r or repeated cp commands), once the workspace accumulates 4,096 files carrying extended attributes, all subsequent cp operations fail with:
cp: could not copy extended attributes to <destination>: Operation not permitted
The command terminates with exit code 1.
This behavior:
- Depends on the number of accumulated companion files, not on file size: with the table below its cap, 1 MiB, 8 MiB and 64 MiB sources each copied 50/50 times and 6 concurrent copies ran 300/300, all with 0 failures;
- Affects any file where macOS automatically attaches metadata (including
com.apple.provenance, which the macOS kernel assigns to files on host APFS filesystems); - Is not a regression introduced by feat(nfs): answer mutations from the publishing job and cut per-file create/unlink overhead (layerfs-project#75) #604–feat(nfs): answer RENAME and COMMIT from the publishing job (layerfs-project#78) #607 (independently reproduced on clean baseline commit
aa131e394); - Has two further caps at baseline: each companion body is limited to
FILE_BYTES = 1 MiB(NFS3ERR_FBIG), and all bodies together toTOTAL_BYTES = 64 MiB(NFS3ERR_NOSPC), which a 100 KiB xattr reaches at the 631st copy, before the count cap.
Steps to Reproduce
- Initialize a project and workspace on the native macOS machine (NFS route, SQLite store, Disposable profile):
layerfs store create main layerfs machine update native --daemon "$BIN/layerfs-daemon" layerfs project init main/proj ./src layerfs branch fork main/proj work main layerfs sandbox create main/proj s1 native layerfs workspace mount s1 work - In the mounted directory, sequentially copy a file carrying extended attributes (or any file created on host APFS):
cd /tmp/mnt for i in $(seq 1 4200); do cp /path/to/host/file "file_$i" || { echo "Failed at $i"; break; } done
- Observe:
- Copies
1through4,095succeed without error. - At copy
4,096,cpfails immediately withOperation not permitted(exit code1). - Every subsequent copy (
4,097,4,098, ...) continues to fail (100% failure rate). - If an existing file is removed (
rm file_1), exactly one subsequentcpis permitted before failing again.
- Copies
Root Cause Analysis
- AppleDouble Sidecars on NFS: On NFSv3 the macOS XNU client keeps a file's extended attributes and resource fork in a
._<name>companion file instead of a protocol-level xattr call (engine/crates/layerfs-nfs/src/apple_double.rs:1-12), so copying a file with attributes into the workspace creates._file_1and so on. LayerFS keeps those names out of the Workspace and in the export's own table. - Hardcoded Server Companion Limit: Inside
engine/crates/layerfs-nfs/src/apple_double.rs, the daemon manages companion files in a private per-export table:/// Companion index entries held at once. pub const HELD: usize = 4096; // line 41 at 716197c49 pub const FILE_BYTES: u64 = 1 << 20; // line 43 pub const TOTAL_BYTES: usize = 64 << 20; // line 45 pub(crate) struct Companions { ... held_limit: usize, // initialized from HELD }
- Error Code Translation: When the companion table reaches the cap,
CREATE ._<name>is refused withNFS3ERR_ACCES(13). The macOS kernel handsACCESto user space asEPERM(1)./bin/cpthen fails in its extended-attribute phase and exits 1. At the count cap the file content was already written (a 4 KiB source landed with a matching sha256 and an emptyxattr -l), but a source carryingcom.apple.quarantinewas observed to leave a 0-byte destination when the same refusal happened, so the data outcome is not uniform. - RPC Verification: Using a transparent loopback ONC RPC proxy, we captured the exact RPC traffic at the failure point. The server directly returns status code 13 (
NFS3ERR_ACCES) forCREATE ._<filename>, confirming the failure originates from the server table cap rather than client-side heuristics.
Fix Implementation & Trade-offs
Why In-Memory LRU Eviction Was Rejected
An initial prototype attempted to evict the least-recently-used (LRU) companion entry when reaching the 4,096 cap. Across 4,300 sequential copies it kept every copy succeeding, but 208 of the 4,300 4 KiB files (and 204 of the 4,300 1 MiB files) silently lost their extended attributes — the boundary was exact (a-00000…a-00207) — so the prototype was dropped.
Chosen Solution: Memory Index + Disk Spilling
We implemented a disk-spilling architecture in #636:
- Compact In-Memory Index: In-memory metadata tracks all active companion records (serial, parent inode, name hash, size, file flags).
- Disk Spilling: Companion bodies beyond a 32 MiB resident budget (
RESIDENT_BYTES = 32 << 20) are spilled to one file per companion under the export's own scratch directory (<mounts>/.appledouble/<namespace>, directory 0700, files 0600), written as<serial>.<n>.tmpand renamed on adoption; no descriptor is kept open for a spilled body. - Increased Capacity: Raised the per-export companion cap from 4,096 to 65,536 entries (
HELD = 65_536; index ≈294 B per entry, ≈20 MB per export), and removed the 64 MiB globalTOTAL_BYTEScap in favour of the 32 MiB spill budget.FILE_BYTES = 1 MiBper companion is unchanged. - Lifecycle Linkage & Orphan Reclamation:
- Primary file deletion (REMOVE, RMDIR, overwrite RENAME, silly rename) cascades to immediately release associated companion index slots and delete spilled disk files.
- An orphan garbage collector (
reclaim_orphans) scans and reclaims unreferenced companions when approaching limits. - On daemon startup, stale companion directories from prior abnormal terminations are purged.
- Two Concurrency & Link Boundary Fixes:
- Bug A (Duplicate Serial Race): Two concurrent
CREATErequests for the same companion name briefly released locks, creating two active serials where one orphaned slot became un-reclaimable. Fixed by consolidating name checking, capacity checks, orphan reclamation, and insertion into a single critical section. - Bug B (Hard Link Reclamation): The server previously tracked only the most recent filename per inode in
by_name. When hard links existed, removing one alias caused the companion for remaining aliases to be misclassified as an orphan and discarded. Fixed by retaining all seen directory-entry aliases.
- Bug A (Duplicate Serial Race): Two concurrent
- Error Code & Logging: When reaching the expanded 65,536 limit, the server returns
NFS3ERR_NOSPC(28) instead ofACCES, and logs a rate-limited warning to stderr (throttled to at most once per 60 seconds per export).
Verification & Test Results
Tested on MacBook Air M2 (macOS 27.2) and Linux:
| Metric / Scenario | Baseline (716197c49 / aa131e394) |
LRU Eviction Prototype | Spilling Fix (#636) |
|---|---|---|---|
| Sequential Copies (4 KiB files) | Fails from No. 4,096 on (243/243 attempts after the cap failed; the accumulating run was 208/208) | 4,300/4,300 passed | 4,300/4,300 passed (0 failed) |
| Sequential Copies (1 MiB files) | Fails from No. 4,096 on | 4,300/4,300 passed | 4,300/4,300 passed (0 failed) |
Attribute Retention (full scan after the 60 s acregmin cache expired) |
N/A (copy failed) | 208 (4 KiB) / 204 (1 MiB) files lost xattrs | 4,300/4,300 and 4,300/4,300 kept, 0 lost |
30,000 Continuous Copy Endurance (cp exit codes + non-OK CREATE ._* only) |
N/A | Not tested | 30,000/30,000 passed, 0 non-OK CREATE ._* replies |
| Spilled Payload Volume | 0 B | 0 B | 29,394 files / 1,587,276 KiB (≈1.5 GB) after the 30,000-copy run |
| File Descriptor Hygiene | Normal | Normal | lsof -p constant at 26 lines (12–25 fds) under ulimit -n 256 while spills grew to 4,522 (measured on the previous commit ca153a7f6) |
Linux Unit Tests (ulimit -n 1024) |
— | Not tested | layerfs-nfs 32/32 passed (including the unit test that lowers RLIMIT to 512 and creates 2,048 spilled companions) |
| Linux non-mount daemon tests | — | Not tested | layerfs-daemon 51/51 passed (3 ignored) |
Merge with Upstream 833d04cf4 |
— | N/A | 0 conflicts via git merge-tree; layerfs-nfs 32/32 on the merged tree |
Known Limitations
- Mount-Scoped Ephemeral Metadata: Companion data remains ephemeral to the active mount session. LayerFS Commit continues to persist only POSIX mode and mtime into the global Store; AppleDouble companions are not committed.
- Client-Side Error Mapping at Capacity: When the 65,536 ceiling is reached, the server returns
NFS3ERR_NOSPC(28). However, the macOS XNU client still presents this to user space asEPERMduring/bin/cp. The advantage ofNOSPCis clean protocol semantics and rate-limited server logging. - Abnormal Termination Recovery: If the daemon process is forcefully killed (
kill -9), uncleaned spill files remain on host disk until the workspace sandbox is closed or the mount is re-established. The startup purge only runs when a daemon is started on the same mounts again. - False-Positive EPERM under Concurrent Scanning:
- In a fresh workspace holding ~4,100 companions (6% of the cap), running background
statorgetxattrloops against the destination folder concurrently with/bin/cpproduced 846EPERMfailures in 2,000 copies; the same load without the scanner was 0/2,000. - A user-space RPC proxy confirmed the server answered
NFS3_OKfor all 230,576 RPC calls in that window. Reading the 846 failed destinations back: 821 had both data and extended attributes correctly on disk, and 25 were really missing the 100 KiBcom.example.big(the earliest runs). - A separate pure-
setxattrgroup that failed 422 times did hit the cap (65,158 prefilled companions + 378 successes = 65,536), so that group is the capacity path, not a client race. - The concurrent-scan error is reported by the macOS client; the server side returns success.
- In a fresh workspace holding ~4,100 companions (6% of the cap), running background
- Remaining Count Ceiling: 65,536 companions means roughly 65,000 attributed files per mount. Past that, the copied file still lands but its xattrs fail (the client reports
EPERM). - Per-Companion Size Cap Unchanged: Each companion body is still limited to
FILE_BYTES = 1 MiB(NFS3ERR_FBIG). A file whose AppleDouble data (for example a large resource fork) exceeds 1 MiB still fails to copy its attributes. This PR neither changes nor tests that path.
Open Architectural Question
Should LayerFS natively parse ._* AppleDouble files into real extended attributes and persist them in the Store across Commits?
- Current Architecture:
- The LayerFS Store model (
InodeValue) stores the file kind, the namespace ref count, a content root and a metadata root; the portable metadata carries onlymodeandmtime, and the length lives with the content side. There is no xattr field. - The Linux FUSE route returns
ENOSYSfor the extended-attribute family (getxattr,setxattr,listxattr), sticky per connection. - AppleDouble companions on NFS are kept outside the Workspace: they never reach the inode table,
READDIR, or a Commit.
- The LayerFS Store model (
- Estimated Scope for Native xattrs:
- Requires defining xattr serialization in
layerfs-content/layerfs-storage. - Requires metadata table schema updates in
layerfs-persistence. - Requires implementing xattr handlers across both
layerfs-fuseandlayerfs-nfs. - Estimated implementation size is ~800–1,200+ LOC and introduces a store schema format evolution.
- Requires defining xattr serialization in
- Request: We invite maintainer feedback on whether the current ephemeral spilling approach (expanded to 65,536 files) is sufficient for agent workloads, or if true committed xattrs should be planned on the LayerFS roadmap.
Activity
jiangkoumo commentedon Nov 9, 2026
Follow-up on the long-term direction mentioned above (getting macOS to send real xattr calls instead of
._*AppleDouble files).We looked at switching the native loopback mount from NFSv3 to NFSv4.0 with named attributes (
mount_nfs -o vers=4.0,namedattr), since the macOS client only falls back to AppleDouble when the filesystem reports xattrs as unsupported. Before writing any server code, we tried the client side on its own, using FUSE-T 1.2.9's NFSv4 server as a stand-in.Result: on macOS 27.2 (26B123, M2 MacBook Air),
cpof a file with xattrs onto avers=4.0,namedattrmount panicked the kernel (Kernel data abort; panicked taskcp; backtrace includescom.apple.filesystems.nfs). It happened twice at the same offset inside the NFS kext. FUSE-T's default mount usesnonamedattrand is not affected.We did not symbolicate the panic, so the exact function is unconfirmed, and FUSE-T's server replies may be the trigger. Either way, a kernel panic is a client-side bug, and a LayerFS NFSv4 server could hit the same path. So we are not pursuing NFSv4 + named attributes for now. #636 (the NFSv3 spill approach) remains our proposed fix; SMB loopback and FSKit (macOS 26+) are the remaining long-term options, and we have not tested either yet.