[congwang-mk]

Supersedes #337. Its chroot unix socket fix is the first commit here unchanged; the rest replaces its off-loop dispatch with a smaller model.

Problem

Supervised connects, FFI callbacks and chroot/COW filesystem handlers ran on the notification loop, so one slow peer, callback or copy stalled every trapped syscall in the sandbox. The existing escape hatch, NotifAction::Defer, made a terminal decision: its Continue skipped later handlers, it was refused on fork and exec, and it failed with EAGAIN past a fixed cap.

Design

• Defer resumes where it was returned. A deferred handler's Continue runs the next handler; any other result ends the chain. The outcome then takes the same path as an inline decision (pinned opens, fork accounting, policy event, argv freeze), so deferring on fork or exec is safe. All deferred futures of a notification share one 30 second deadline.
• Synchronous work runs as NotifAction::native(closure) on a process-wide runtime kept apart from the embedder's, whose shutdown would otherwise wait on a closure stuck on a peer. A closure cannot be cancelled, so it has no deadline: failing the syscall early would not stop its side effects. The syscall waits for it as it would for the kernel.
• No admission failures, no per-sandbox slots. Each trapped task has at most one outstanding notification, so deferred work is bounded by the sandbox's own task count. The process-wide thread cap is only a backstop for closures that outlive a dead child.
• Child-memory writes are ordered against exec at the write itself. write_child_mem and write_child_mem_force hold the shared side of a per-sandbox gate for each write made by deferred work; the exec freeze holds the exclusive side until the kernel has copied argv. Exec waits microseconds instead of failing, and no call site has to remember a guard.
• Fork accounting runs once the chain settles on Continue, so no handler can defer a fork after it was counted, and a failed response rolls the slot back.
• Chroot and COW handlers stay on the loop; only the copy-up leaves it. A handler plans under the COW lock as before. When a write needs a lower file in the upper first, one driver copies it as native work and then runs the write under the lock that found every copy in place. The branch handlers no longer copy at all, so there is one copy path and no copier holds the lock across file I/O. This covers COW and chroot open, rename, link, chmod, chown, truncate and utimensat. Fixes cow: a large copy-up stalls the supervisor for its whole duration #242.
• Copies are staged, then installed. A copy streams into a staging file in the branch storage and is installed with RENAME_NOREPLACE, so a half-written upper is never visible. A second caller for the same file waits for the first and plans again; a path deleted while it was copied is not resurrected; a quota resync keeps the reservation of a copy in flight. A directory rename walks the lower tree off the loop and plans the whole tree at once, all or nothing. A write is applied only if its notification is still waiting, so a caller interrupted during the copy (a signal, or an exec freeze without WAIT_KILLABLE_RECV) restarts its syscall onto a write not yet done.
• The listener stays open until the last native closure using its fd number returns.
• Every FFI callback is native work. sandlock_handler_set_deferred is removed (pre-1.0 hard break). A per-handler run guard waits for running callbacks before the FFI run returns, and callbacks still queued fail with EIO instead of touching freed userdata. Python handlers, sync or async, run off the loop with no flag.

Behavior changes

• A deferred or async handler that returns Continue now hands the syscall to the next handler instead of ending the chain.
• FFI and Python callbacks have no deadline: a syscall waits for its callback however long it takes, where a deferred callback used to fail with EIO after 30 seconds.
• An FFI run waits for running callbacks before returning; a callback that never returns prevents the run from completing.
• sandlock_handler_set_deferred is gone from the C ABI and the Python SDK.
• When the branch fails a write it owns (a failed copy, an I/O error), the child gets the errno instead of the syscall continuing to the kernel, which would have changed the lower file. Kernel objects such as devices and FIFOs still pass through.
• A directory rename that fails during the copy keeps the entries already copied, as overlayfs does. Undoing them could discard a write made through them meanwhile. A refusal while planning (quota, merged-view checks) still stages nothing.

Known gaps

• Checkpoint and pause do not wait for native work in flight, so a copy-up or FFI callback can still be running when the sandbox is frozen.

Test plan

• CI run 41945973005 green on all 25 jobs (Rust tests, feature matrix, low-ABI, musl, riscv64 cross-build, cbindgen header, Python, Go)
• New unit tests: chain resume and termination after a deferral, deadline and nesting, native work ignoring the deadline and running to completion after its waiter is dropped, gate exclusion, FFI run guard drain and queued-callback EIO, staged copies invisible until installed, a waiting second copier, no install over a deleted path, a losing copy keeping the winner's file, quota resync of a staged copy
• Python e2e: an async handler's continue_() reaches the next handler on the syscall

🤖 Generated with Claude Code