Skip to content

[Bug]: Native macOS NFS mount fails cp with Operation not permitted once 4,096 AppleDouble companions accumulate (companion table cap) #635

Open

Description

@jiangkoumo

Description

When copying files within a native macOS loopback NFS workspace (e.g. running cp -r or repeated cp commands), once the workspace accumulates 4,096 files carrying extended attributes, all subsequent cp operations fail with:

cp: could not copy extended attributes to <destination>: Operation not permitted

The command terminates with exit code 1.

This behavior:


Steps to Reproduce

  1. Initialize a project and workspace on the native macOS machine (NFS route, SQLite store, Disposable profile):
    layerfs store create main
    layerfs machine update native --daemon "$BIN/layerfs-daemon"
    layerfs project init main/proj ./src
    layerfs branch fork main/proj work main
    layerfs sandbox create main/proj s1 native
    layerfs workspace mount s1 work
  2. In the mounted directory, sequentially copy a file carrying extended attributes (or any file created on host APFS):
    cd /tmp/mnt
    for i in $(seq 1 4200); do
        cp /path/to/host/file "file_$i" || { echo "Failed at $i"; break; }
    done
  3. Observe:
    • Copies 1 through 4,095 succeed without error.
    • At copy 4,096, cp fails immediately with Operation not permitted (exit code 1).
    • Every subsequent copy (4,097, 4,098, ...) continues to fail (100% failure rate).
    • If an existing file is removed (rm file_1), exactly one subsequent cp is permitted before failing again.

Root Cause Analysis

  1. AppleDouble Sidecars on NFS: On NFSv3 the macOS XNU client keeps a file's extended attributes and resource fork in a ._<name> companion file instead of a protocol-level xattr call (engine/crates/layerfs-nfs/src/apple_double.rs:1-12), so copying a file with attributes into the workspace creates ._file_1 and so on. LayerFS keeps those names out of the Workspace and in the export's own table.
  2. Hardcoded Server Companion Limit: Inside engine/crates/layerfs-nfs/src/apple_double.rs, the daemon manages companion files in a private per-export table:
    /// Companion index entries held at once.
    pub const HELD: usize = 4096;             // line 41 at 716197c49
    pub const FILE_BYTES: u64 = 1 << 20;      // line 43
    pub const TOTAL_BYTES: usize = 64 << 20;  // line 45
    
    pub(crate) struct Companions {
        ...
        held_limit: usize,   // initialized from HELD
    }
  3. Error Code Translation: When the companion table reaches the cap, CREATE ._<name> is refused with NFS3ERR_ACCES (13). The macOS kernel hands ACCES to user space as EPERM (1). /bin/cp then fails in its extended-attribute phase and exits 1. At the count cap the file content was already written (a 4 KiB source landed with a matching sha256 and an empty xattr -l), but a source carrying com.apple.quarantine was observed to leave a 0-byte destination when the same refusal happened, so the data outcome is not uniform.
  4. RPC Verification: Using a transparent loopback ONC RPC proxy, we captured the exact RPC traffic at the failure point. The server directly returns status code 13 (NFS3ERR_ACCES) for CREATE ._<filename>, confirming the failure originates from the server table cap rather than client-side heuristics.

Fix Implementation & Trade-offs

Why In-Memory LRU Eviction Was Rejected

An initial prototype attempted to evict the least-recently-used (LRU) companion entry when reaching the 4,096 cap. Across 4,300 sequential copies it kept every copy succeeding, but 208 of the 4,300 4 KiB files (and 204 of the 4,300 1 MiB files) silently lost their extended attributes — the boundary was exact (a-00000…a-00207) — so the prototype was dropped.

Chosen Solution: Memory Index + Disk Spilling

We implemented a disk-spilling architecture in #636:

  1. Compact In-Memory Index: In-memory metadata tracks all active companion records (serial, parent inode, name hash, size, file flags).
  2. Disk Spilling: Companion bodies beyond a 32 MiB resident budget (RESIDENT_BYTES = 32 << 20) are spilled to one file per companion under the export's own scratch directory (<mounts>/.appledouble/<namespace>, directory 0700, files 0600), written as <serial>.<n>.tmp and renamed on adoption; no descriptor is kept open for a spilled body.
  3. Increased Capacity: Raised the per-export companion cap from 4,096 to 65,536 entries (HELD = 65_536; index ≈294 B per entry, ≈20 MB per export), and removed the 64 MiB global TOTAL_BYTES cap in favour of the 32 MiB spill budget. FILE_BYTES = 1 MiB per companion is unchanged.
  4. Lifecycle Linkage & Orphan Reclamation:
    • Primary file deletion (REMOVE, RMDIR, overwrite RENAME, silly rename) cascades to immediately release associated companion index slots and delete spilled disk files.
    • An orphan garbage collector (reclaim_orphans) scans and reclaims unreferenced companions when approaching limits.
    • On daemon startup, stale companion directories from prior abnormal terminations are purged.
  5. Two Concurrency & Link Boundary Fixes:
    • Bug A (Duplicate Serial Race): Two concurrent CREATE requests for the same companion name briefly released locks, creating two active serials where one orphaned slot became un-reclaimable. Fixed by consolidating name checking, capacity checks, orphan reclamation, and insertion into a single critical section.
    • Bug B (Hard Link Reclamation): The server previously tracked only the most recent filename per inode in by_name. When hard links existed, removing one alias caused the companion for remaining aliases to be misclassified as an orphan and discarded. Fixed by retaining all seen directory-entry aliases.
  6. Error Code & Logging: When reaching the expanded 65,536 limit, the server returns NFS3ERR_NOSPC (28) instead of ACCES, and logs a rate-limited warning to stderr (throttled to at most once per 60 seconds per export).

Verification & Test Results

Tested on MacBook Air M2 (macOS 27.2) and Linux:

Metric / Scenario Baseline (716197c49 / aa131e394) LRU Eviction Prototype Spilling Fix (#636)
Sequential Copies (4 KiB files) Fails from No. 4,096 on (243/243 attempts after the cap failed; the accumulating run was 208/208) 4,300/4,300 passed 4,300/4,300 passed (0 failed)
Sequential Copies (1 MiB files) Fails from No. 4,096 on 4,300/4,300 passed 4,300/4,300 passed (0 failed)
Attribute Retention (full scan after the 60 s acregmin cache expired) N/A (copy failed) 208 (4 KiB) / 204 (1 MiB) files lost xattrs 4,300/4,300 and 4,300/4,300 kept, 0 lost
30,000 Continuous Copy Endurance (cp exit codes + non-OK CREATE ._* only) N/A Not tested 30,000/30,000 passed, 0 non-OK CREATE ._* replies
Spilled Payload Volume 0 B 0 B 29,394 files / 1,587,276 KiB (≈1.5 GB) after the 30,000-copy run
File Descriptor Hygiene Normal Normal lsof -p constant at 26 lines (12–25 fds) under ulimit -n 256 while spills grew to 4,522 (measured on the previous commit ca153a7f6)
Linux Unit Tests (ulimit -n 1024) — Not tested layerfs-nfs 32/32 passed (including the unit test that lowers RLIMIT to 512 and creates 2,048 spilled companions)
Linux non-mount daemon tests — Not tested layerfs-daemon 51/51 passed (3 ignored)
Merge with Upstream 833d04cf4 — N/A 0 conflicts via git merge-tree; layerfs-nfs 32/32 on the merged tree

Known Limitations

  1. Mount-Scoped Ephemeral Metadata: Companion data remains ephemeral to the active mount session. LayerFS Commit continues to persist only POSIX mode and mtime into the global Store; AppleDouble companions are not committed.
  2. Client-Side Error Mapping at Capacity: When the 65,536 ceiling is reached, the server returns NFS3ERR_NOSPC (28). However, the macOS XNU client still presents this to user space as EPERM during /bin/cp. The advantage of NOSPC is clean protocol semantics and rate-limited server logging.
  3. Abnormal Termination Recovery: If the daemon process is forcefully killed (kill -9), uncleaned spill files remain on host disk until the workspace sandbox is closed or the mount is re-established. The startup purge only runs when a daemon is started on the same mounts again.
  4. False-Positive EPERM under Concurrent Scanning:
    • In a fresh workspace holding ~4,100 companions (6% of the cap), running background stat or getxattr loops against the destination folder concurrently with /bin/cp produced 846 EPERM failures in 2,000 copies; the same load without the scanner was 0/2,000.
    • A user-space RPC proxy confirmed the server answered NFS3_OK for all 230,576 RPC calls in that window. Reading the 846 failed destinations back: 821 had both data and extended attributes correctly on disk, and 25 were really missing the 100 KiB com.example.big (the earliest runs).
    • A separate pure-setxattr group that failed 422 times did hit the cap (65,158 prefilled companions + 378 successes = 65,536), so that group is the capacity path, not a client race.
    • The concurrent-scan error is reported by the macOS client; the server side returns success.
  5. Remaining Count Ceiling: 65,536 companions means roughly 65,000 attributed files per mount. Past that, the copied file still lands but its xattrs fail (the client reports EPERM).
  6. Per-Companion Size Cap Unchanged: Each companion body is still limited to FILE_BYTES = 1 MiB (NFS3ERR_FBIG). A file whose AppleDouble data (for example a large resource fork) exceeds 1 MiB still fails to copy its attributes. This PR neither changes nor tests that path.

Open Architectural Question

Should LayerFS natively parse ._* AppleDouble files into real extended attributes and persist them in the Store across Commits?

  • Current Architecture:
    • The LayerFS Store model (InodeValue) stores the file kind, the namespace ref count, a content root and a metadata root; the portable metadata carries only mode and mtime, and the length lives with the content side. There is no xattr field.
    • The Linux FUSE route returns ENOSYS for the extended-attribute family (getxattr, setxattr, listxattr), sticky per connection.
    • AppleDouble companions on NFS are kept outside the Workspace: they never reach the inode table, READDIR, or a Commit.
  • Estimated Scope for Native xattrs:
    • Requires defining xattr serialization in layerfs-content / layerfs-storage.
    • Requires metadata table schema updates in layerfs-persistence.
    • Requires implementing xattr handlers across both layerfs-fuse and layerfs-nfs.
    • Estimated implementation size is ~800–1,200+ LOC and introduces a store schema format evolution.
  • Request: We invite maintainer feedback on whether the current ephemeral spilling approach (expanded to 65,536 files) is sufficient for agent workloads, or if true committed xattrs should be planned on the LayerFS roadmap.

Activity

  1. jiangkoumo commented on Nov 9, 2026

    @jiangkoumo
    CollaboratorAuthor

    Follow-up on the long-term direction mentioned above (getting macOS to send real xattr calls instead of ._* AppleDouble files).

    We looked at switching the native loopback mount from NFSv3 to NFSv4.0 with named attributes (mount_nfs -o vers=4.0,namedattr), since the macOS client only falls back to AppleDouble when the filesystem reports xattrs as unsupported. Before writing any server code, we tried the client side on its own, using FUSE-T 1.2.9's NFSv4 server as a stand-in.

    Result: on macOS 27.2 (26B123, M2 MacBook Air), cp of a file with xattrs onto a vers=4.0,namedattr mount panicked the kernel (Kernel data abort; panicked task cp; backtrace includes com.apple.filesystems.nfs). It happened twice at the same offset inside the NFS kext. FUSE-T's default mount uses nonamedattr and is not affected.

    We did not symbolicate the panic, so the exact function is unconfirmed, and FUSE-T's server replies may be the trigger. Either way, a kernel panic is a client-side bug, and a LayerFS NFSv4 server could hit the same path. So we are not pursuing NFSv4 + named attributes for now. #636 (the NFSv3 spill approach) remains our proposed fix; SMB loopback and FSKit (macOS 26+) are the remaining long-term options, and we have not tested either yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions