Skip to content

Daemon control slots leak: a connection that fails after a request stays Retained forever, and the daemon then refuses every connection #540

Closed

Description

@yifanxuaaa

Summary

A Sandbox daemon admits DAEMON_CONNECTIONS = 8 control connections (plus the reserved installer slot). A connection worker that fails after a request was decoded leaves its slot as Slot::Retained with no log line, and nothing in the product path ever releases it. Each such failure permanently costs one slot. Once all are retained, the daemon closes every new connection at accept, and every SDK call on the Sandbox (sandbox.get, sandbox.close, workspace.*) fails Busy after 60 s. The Sandbox can't be closed through the SDK, by force or otherwise.

Observed (HEAD 6deb7f6 linked addon, lima1/lima2, PostgreSQL Store)

Sandboxes live-api-lima1 / live-api-lima2 (PostgreSQL Store live-pg-api, project api), about 2 h old:

  • sandbox.get live-api-lima2 → Busy: the Sandbox's daemon admitted no connection for 60s: other processes hold all 8 of its connections
  • sandbox.get live-api-lima1 hangs behind a sandbox.close (phase stop) that has been running in the caller's process for many minutes; a second close or a forced close is refused Busy … runs in process <pid>.

Every other component checks out healthy:

  • The VMs are idle (load 0.00, memory and disk free).
  • PostgreSQL has no open transactions; every session is idle.
  • The host-to-guest SSH forwards are fine.
  • Each daemon has exactly one established TCP connection (from the in-VM layerfs worker) and one layerfs-control-N thread. The other threads are fuser-*, layerfs-fence-1, layerfs-mount-1 and the accept loop in inet_csk_accept.
  • A raw connection from inside the guest (nc 127.0.0.1 <daemon port>) gets closed immediately (FIN-WAIT-2 within 1 s, before handshake_ms: 2000 could expire), i.e. the "no vacant slot" branch.
  • daemon.log holds only LAYERFS_DAEMON_LISTEN and 6 ConnectionFailure { …, cause: Channel(Io(Kind(UnexpectedEof))) } lines. Those are the failures that were released. The retained ones leave no trace.

Cause

engine/crates/layerfs-daemon/src/application/connection.rs (Application::serve):

  • Lines 25–29: with no Slot::Vacant, the accepted socket is dropped.
  • Lines 51–76: when a worker finishes, its slot becomes Vacant only if the result is Ok, or if the failure is a Channel/Io error with original.is_none() && received.is_none() (logged to stderr first). Any other failure, including a client that disappears or a reply that can't be sent after a request was received, becomes Slot::Retained(cause), without logging.
  • owner.rs:276 take_connection_failure is the only path from Retained back to Vacant. Its doc comment calls it an "explicit operator request", and no SDK or control call makes it.

Callers that die or restart mid-call (an app restart, a killed live-fixtures run, a dropped tunnel, a client read-timeout quarantine in pool.rs) are routine for long-lived Sandboxes, so slots run out over hours of normal use.

Client side, api/crates/layerfs-control/src/daemon/pool.rs:69-104 treats the resulting UnexpectedEof as "other processes hold all 8 of its connections". That's misleading here: no other process holds anything.

Expected

  • A retained connection failure should not use up admission capacity for good. One option: move the failure record out of the slot table into a separate store, with a limit, a log line and an operator-visible record, and free the slot. Another: let retained slots be reclaimed when no product operation is still held by them.
  • Each retained failure should be logged to stderr / daemon.log, the same way the released ones are.
  • The client refusal should say what it actually knows: the daemon refused the connection (no vacant control slot). It should not attribute that to other processes.
  • A stuck sandbox.close at phase stop should end with an exact refusal, not hold the operation lock while the daemon refuses connections.

Reproduce (sketch)

  1. Open a Sandbox. From a client process, start a call that takes a moment (or any call), and kill the client after the request is sent but before the reply is read. Repeat 8–9 times.
  2. Any new client: sandbox.get <sandbox> → Busy after 60 s. In the guest, ss -tnp shows a single established connection to the daemon port.

Recovery today

The only way out is to kill the daemon process in the guest and then close or recreate the Sandbox. Its Workspaces are lost.

Activity

  1. added
    bugSomething isn't working
    v2.0.0LayerFS 2.0 scope, roadmap and delivery
    on Nov 4, 2026
  2. mengmengmengqiang commented on Nov 4, 2026

    @mengmengmengqiang
    Contributor

    #540 diagrams

    Control-slot custody

    Recovery sequence

  3. added theissue type on Nov 5, 2026
  4. added this to the LayerFS v2.0.0 milestone on Nov 5, 2026
  5. mengmengmengqiang commented on Nov 12, 2026

    @mengmengmengqiang
    Contributor

    Closed: a retained control slot no longer costs admission capacity for good

    The leak this issue describes is fixed across two changes, both now on layerfs_v2_0_0:

    • fix(daemon): reclaim control slots after retained connection failure (#540) #543 (already merged, e9b903eac) — the retained-failure class: a failure after a decoded request is recorded in a bounded, TTL'd store outside the slot table with a visible count and a log line, and the slot is freed. backend.daemon.connection_failures exposes the record and the capacity read model, including on a full pool (the pre-handshake LAYERFS-CAPACITY notice).
    • daemon: bound an admitted control slot's lifetime and keep recovery admissible when the pool is exhausted (#540 follow-up) #547 slices 1–4 — admitted-slot lifetime and recovery: the slot-custody state machine with a pure disposition classifier (slice 1–2), a recovery-class reserved slot so close/status/connection-failures can still get in at a full pool, and idle pool-slot reclamation past idle_ttl (slice 3–4). Slices 5–6 also landed in the same stack: in-flight request expiry, the one-shot backend.daemon.reclaim { slot, generation } operator verb, and typed refusal outcomes from the channel error rather than the refusal text.

    Merged PRs (merge commits, no squash, authorship kept)

    PR Branch Merge commit
    #650 fix/547-slice-12 (slices 1–2) 79fc41f3902fc6c423c069af28bd67601bababbe
    #651 fix/547-slice-34 (slices 3–4) cbb7c41edbdd212c6a0e1840fee7fdf38b9dd42e
    #652 fix/547-slice-56 (slices 5–6) dbd633facd4959debef999e986e80fafd868e51b

    Merged head: layerfs_v2_0_0 @ dbd633facd4959debef999e986e80fafd868e51b.

    Test evidence (merged head, Dell aluminium, x86_64, rustc 1.85.1)

    cargo +1.85.1 fmt --all -- --check                                          # rc=0
    cargo +1.85.1 test -p layerfs-daemon --test daemon_slots --test slot_state  # rc=0
        daemon_slots.rs: 7 passed, 0 failed
        slot_state.rs: 20 passed, 0 failed
    cargo +1.85.1 test -p layerfs-sandbox -p layerfs-api --no-fail-fast         # rc=0
        every test binary ok; 0 failed
    

    Full log: /home/dell/540-merged-head-tests.log on aluminium. The slot tests run as the non-root dell user (the daemon refuses to start when the command uid equals the owner uid); the sandbox/API suite is green in the same configuration here.

    #547

    #547 itself stays open as its tracker; slices 1–4 were the gate for this issue and slices 5–6 are merged too (#652). Nothing about the GC work or other PRs is touched here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingv2.0.0LayerFS 2.0 scope, roadmap and delivery

Type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions