Repository navigation
Description
Summary
A Sandbox daemon admits DAEMON_CONNECTIONS = 8 control connections (plus the reserved installer slot). A connection worker that fails after a request was decoded leaves its slot as Slot::Retained with no log line, and nothing in the product path ever releases it. Each such failure permanently costs one slot. Once all are retained, the daemon closes every new connection at accept, and every SDK call on the Sandbox (sandbox.get, sandbox.close, workspace.*) fails Busy after 60 s. The Sandbox can't be closed through the SDK, by force or otherwise.
Observed (HEAD 6deb7f6 linked addon, lima1/lima2, PostgreSQL Store)
Sandboxes live-api-lima1 / live-api-lima2 (PostgreSQL Store live-pg-api, project api), about 2 h old:
sandbox.get live-api-lima2→Busy: the Sandbox's daemon admitted no connection for 60s: other processes hold all 8 of its connectionssandbox.get live-api-lima1hangs behind asandbox.close(phasestop) that has been running in the caller's process for many minutes; a second close or a forced close is refusedBusy … runs in process <pid>.
Every other component checks out healthy:
- The VMs are idle (load 0.00, memory and disk free).
- PostgreSQL has no open transactions; every session is idle.
- The host-to-guest SSH forwards are fine.
- Each daemon has exactly one established TCP connection (from the in-VM
layerfs worker) and onelayerfs-control-Nthread. The other threads arefuser-*,layerfs-fence-1,layerfs-mount-1and the accept loop ininet_csk_accept. - A raw connection from inside the guest (
nc 127.0.0.1 <daemon port>) gets closed immediately (FIN-WAIT-2within 1 s, beforehandshake_ms: 2000could expire), i.e. the "no vacant slot" branch. daemon.logholds onlyLAYERFS_DAEMON_LISTENand 6ConnectionFailure { …, cause: Channel(Io(Kind(UnexpectedEof))) }lines. Those are the failures that were released. The retained ones leave no trace.
Cause
engine/crates/layerfs-daemon/src/application/connection.rs (Application::serve):
- Lines 25–29: with no
Slot::Vacant, the accepted socket is dropped. - Lines 51–76: when a worker finishes, its slot becomes
Vacantonly if the result isOk, or if the failure is aChannel/Ioerror withoriginal.is_none() && received.is_none()(logged to stderr first). Any other failure, including a client that disappears or a reply that can't be sent after a request was received, becomesSlot::Retained(cause), without logging. owner.rs:276take_connection_failureis the only path fromRetainedback toVacant. Its doc comment calls it an "explicit operator request", and no SDK or control call makes it.
Callers that die or restart mid-call (an app restart, a killed live-fixtures run, a dropped tunnel, a client read-timeout quarantine in pool.rs) are routine for long-lived Sandboxes, so slots run out over hours of normal use.
Client side, api/crates/layerfs-control/src/daemon/pool.rs:69-104 treats the resulting UnexpectedEof as "other processes hold all 8 of its connections". That's misleading here: no other process holds anything.
Expected
- A retained connection failure should not use up admission capacity for good. One option: move the failure record out of the slot table into a separate store, with a limit, a log line and an operator-visible record, and free the slot. Another: let retained slots be reclaimed when no product operation is still held by them.
- Each retained failure should be logged to stderr /
daemon.log, the same way the released ones are. - The client refusal should say what it actually knows: the daemon refused the connection (no vacant control slot). It should not attribute that to other processes.
- A stuck
sandbox.closeat phasestopshould end with an exact refusal, not hold the operation lock while the daemon refuses connections.
Reproduce (sketch)
- Open a Sandbox. From a client process, start a call that takes a moment (or any call), and kill the client after the request is sent but before the reply is read. Repeat 8–9 times.
- Any new client:
sandbox.get <sandbox>→Busyafter 60 s. In the guest,ss -tnpshows a single established connection to the daemon port.
Recovery today
The only way out is to kill the daemon process in the guest and then close or recreate the Sandbox. Its Workspaces are lost.
Activity
fix(daemon): complete slot-class isolation and capacity notice (Ephem…
fix(control): bound sandbox.get's capacity query to 2s (Ephemeral-AI-…
feat(control): query live daemon capacity in doctor (Ephemeral-AI-Lab…
docs(540): add architecture and recovery diagrams (Ephemeral-AI-Lab#540)
docs(540): use final recovery-sequence diagram (Ephemeral-AI-Lab#540)
mengmengmengqiang commentedon Nov 4, 2026
#540 diagrams
fix(store): keep a refused file-digest write a definite Busy Commit
feat(sandbox): expose the daemon's control capacity in sandbox.get (E…
test(sandbox): cover the capacity notice a full pool's next client re…
fix(sandbox): restore the daemon capacity read in sandbox.get (Epheme…
test(sandbox): cover the capacity read's routes, wall and pool slot (E…
docs(sandbox): cite Ephemeral-AI-Lab#540 plainly in sandbox.get's fie…
fix(daemon): reclaim control slots after retained connection failure (#…
Merge pull request #543 from tokenaissance/fix/540-daemon-slots
merge: upstream sandbox daemon slots fix (Ephemeral-AI-Lab#540/Epheme…
mengmengmengqiang commentedon Nov 12, 2026
Closed: a retained control slot no longer costs admission capacity for good
The leak this issue describes is fixed across two changes, both now on
layerfs_v2_0_0:e9b903eac) — the retained-failure class: a failure after a decoded request is recorded in a bounded, TTL'd store outside the slot table with a visible count and a log line, and the slot is freed.backend.daemon.connection_failuresexposes the record and the capacity read model, including on a full pool (the pre-handshakeLAYERFS-CAPACITYnotice).dispositionclassifier (slice 1–2), a recovery-class reserved slot soclose/status/connection-failures can still get in at a full pool, and idle pool-slot reclamation pastidle_ttl(slice 3–4). Slices 5–6 also landed in the same stack: in-flight request expiry, the one-shotbackend.daemon.reclaim { slot, generation }operator verb, and typed refusal outcomes from the channel error rather than the refusal text.Merged PRs (merge commits, no squash, authorship kept)
fix/547-slice-12(slices 1–2)79fc41f3902fc6c423c069af28bd67601bababbefix/547-slice-34(slices 3–4)cbb7c41edbdd212c6a0e1840fee7fdf38b9dd42efix/547-slice-56(slices 5–6)dbd633facd4959debef999e986e80fafd868e51bMerged head:
layerfs_v2_0_0@dbd633facd4959debef999e986e80fafd868e51b.Test evidence (merged head, Dell
aluminium, x86_64, rustc 1.85.1)Full log:
/home/dell/540-merged-head-tests.logonaluminium. The slot tests run as the non-rootdelluser (the daemon refuses to start when the command uid equals the owner uid); the sandbox/API suite is green in the same configuration here.#547
#547 itself stays open as its tracker; slices 1–4 were the gate for this issue and slices 5–6 are merged too (#652). Nothing about the GC work or other PRs is touched here.