Skip to content

daemon: bound an admitted control slot's lifetime and keep recovery admissible when the pool is exhausted (#540 follow-up) #547

Closed

Description

@mengmengmengqiang

Summary

#540's retained-slot leak is fixed by #543: Application::serve no longer leaves a failed connection's slot Retained, and every post-decode failure is logged and queryable. Two capacity failure classes remain, and both still end at the same user-visible point — a Sandbox whose SDK calls fail and whose only recovery is sandbox.close --force or killing the daemon, losing uncommitted Workspaces.

  1. An admitted slot has no lifetime. After the first request is decoded the daemon clears the socket read/write timeouts (engine/crates/layerfs-daemon/src/application/connection.rs:305-315), and SlotState::Running carries no deadline (application/owner.rs:71-74). A worker parked in a Store wait, a fence, a diagnostic transfer or answer_call holds 1 of 8 pool slots forever, with no log line and no reclaim verb. Eight of them reproduce Daemon control slots leak: a connection that fails after a request stays Retained forever, and the daemon then refuses every connection #540 exactly.
  2. Capacity shortage still requires destroying the data plane. The one reserved slot is installer-only (owner.rs:28, connection.rs:148-159, owner.rs:455-464); AdmissionClass::Lease is dead code. With the pool full, a non-force sandbox.close fails in 2 s with an exact refusal (sandbox/serve.rs:43) and the only way out is force, which reports undrained Workspaces as unknown and loses their uncommitted changes.

This issue adds the missing lifetime bounds and a recovery admission path. It deliberately does not add per-owner quotas or fairness (see Non-goals: the protocol carries no client identity).

Scope

  • Give every admitted slot a phase with an explicit bound: Handshake { deadline } (exists, handshake_ms), InFlight { started, deadline }, Idle { since }.
  • Reclaim idle slots automatically: a slot whose worker is blocked in receive() past idle_ttl is closed by the daemon. This loses nothing (no operation is in flight); the client pool quarantines the connection and opens another (api/crates/layerfs-control/src/daemon/pool.rs:412-432).
  • Mark in-flight slots that pass request_deadline as expired: record RetainedFailure { cause: Expired, reply_sent: false }, free the slot, and have the caller observe Unavailable { outcome: unknown }. Do not silently kill an operation whose effect is unknown.
  • Add one authenticated operator method, backend.daemon.reclaim { slot, generation }, dispatched like backend.daemon.connection_failures (api/crates/layerfs-control/src/daemon/operator.rs). It closes the named worker's socket, records Reclaimed, and answers Invalid on a generation mismatch. One attempt, no retry, never restarts the daemon and never touches Workspaces.
  • Turn the reserved slot into a recovery class: when the pool has no vacant slot, admit one connection into the reserved slot and, after decoding, allow only the recovery operation set (close, workspace status, backend.daemon.connection_failures, reclaim). Any other request is refused Capacity and the connection is closed — the shape that already exists for the installer (connection.rs:351-358, 419-429).
  • Expose the new counts in the existing read model: admission.idle, admission.inflight, admission.reclaimable in layerfs-bridge::ControlCapacity, defaulted when absent.

Non-goals

Invariants

  1. Every admitted slot ends Vacant exactly once; a stale or duplicate finish changes nothing.
  2. No slot stays Idle past idle_ttl, and no slot stays InFlight past request_deadline without becoming a recorded Expired/Reclaimed disposition.
  3. A failure record never owns a slot; the record store stays bounded (64 / 1 h) with a dropped count.
  4. At most one connection holds the reserved slot; it serves install during startup and recovery afterwards, and it is closed after one request.
  5. Closing, reading status of, or reclaiming a Sandbox never requires destroying its data plane. force remains the last resort, not the only recovery.
  6. Reclaim is authenticated, generation-checked, one-shot, and reports outcome: unknown unless the operation is known idempotent.
  7. A single worker's panic or error never stops the accept loop.

Acceptance

  • With all 8 pool slots held by idle clients, a 9th client is admitted within idle_ttl + handshake_ms; the evicted client's next call either reuses a quarantined pool entry or opens a new connection and succeeds.
  • With the pool 8/8 occupied (idle or in-flight), sandbox.close (no force) succeeds, drains or reports every Workspace by name, and preserves a 64 MiB uncommitted change (cksum unchanged).
  • A worker parked in a fence/Store/diagnostic wait for longer than request_deadline appears as Expired in backend.daemon.connection_failures; backend.daemon.reclaim { slot, generation } frees it without restarting the daemon; the Workspace content and cksum are unchanged; a stale generation answers Invalid.
  • The daemon's slot invariants have in-tree tests that CI runs (today cargo test -p layerfs-daemon --lib runs zero), and the suite fails on the pre-fix(daemon): reclaim control slots after retained connection failure (#540) #543 base commit.
  • No refusal attributes a cause the caller cannot observe: Capacity names resource, pool_vacant, reserved_vacant, and retryable; legacy EOF answers Unknown { observed }, never "no vacant slot".

Tests

# Test Layer Asserts
1 8 post-decode client deaths, then a fresh client in-tree integration (real daemon binary + TCP client) slot freed, new client admitted, one log line per failure
2 retained record via the operator call in-tree integration retained.count > 0, reply_sent, request, cause
3 ring limit/TTL/eviction/dropped entity unit (state machine made pub) 65th record evicts the oldest; TTL prunes; dropped counts
4 stale/duplicate finish entity unit Finished::Stale, state unchanged
5 panic and guard drop entity + integration exactly one finish; slot released
6 16 concurrent failures in-tree integration no deadlock; 8 vacant
7 notice when no pool slot is free in-tree integration 24-byte LAYERFS-CAPACITY frame, reason NoPoolSlot
8 8 idle clients + a 9th in-tree integration idle eviction, then admission
9 sandbox.close with the pool full in-tree control integration closes without force; cksum preserved
10 parked worker + reclaim in-tree integration Expired, reclaim, generation mismatch → Invalid
11 reserved slot refuses a non-recovery request in-tree integration Capacity, connection closed
12 accept loop survives a worker panic unit next connection still admitted
13 version skew: new daemon × old client, old daemon × new client, new control × old worker bundle nightly no hang; each failure is named
14 3 processes × 64 MiB commits nightly bounded waits; close still possible
15 24 h soak: periodic client kills, periodic capacity read nightly pool_vacant / retained time series has no drift
16 close during a #544 holder kill nightly (with #544) bounded; no false Capacity

PR slices

  1. Restore the regression suite (test-only): the four deleted daemon assertions, as package integration tests; make Slots/RetainedFailures pub so the pure state machine is unit-testable. No behavior change. Fails on the base commit.
  2. Structure (no behavior change): move RetainedFailures out of Slots into State; extract disposition(&ConnectionFailure, SlotTicket, &str, Instant) -> Disposition; extract capacity_report(SlotCounts, RetainedSnapshot) -> CapacityModel; pass SlotClass instead of slot: usize to control_operation/hello; delete AdmissionClass::Lease, AdmissionReason::ReservedBusy, and the three aliases of CAPACITY_OPERATION. Guarded by slice 1.
  3. Recovery class: reserved-slot admission when the pool is full; post-decode allow-list of recovery methods; one request then close. Tests 9, 11.
  4. Idle reclamation: Phase::Idle { since } maintained at request boundaries; reap on refusal. Test 8.
  5. In-flight expiry + reclaim: Phase::InFlight { started, deadline }, Expired disposition, backend.daemon.reclaim. Tests 10, 12.
  6. Contract cleanup: typed admission reason, outcome / retry_after, drop the message-substring classification in pool.rs, #[non_exhaustive]/#[must_use], 5-minute runbook, version-skew matrix.

Slices 1–2 are the debt from #543 and can start immediately. 3–4 are one delivery. 5 depends on 3 and 4. 6 lands with the #544 merge.

Relation to #540 / #543 / #545

Activity

  1. mengmengmengqiang commented on Nov 4, 2026

    @mengmengmengqiang
    ContributorAuthor

    Merge / closure conditions

    Tracking: #540 (bug, remains open), #543 (PR, must merge first), #544 (PR, rebases after #543), #545 (B5, cross-reference only).

  2. added theissue type on Nov 5, 2026
  3. added this to the LayerFS v2.0.0 milestone on Nov 5, 2026
  4. mengmengmengqiang commented on Nov 9, 2026

    @mengmengmengqiang
    ContributorAuthor

    Landing order (slices)

    1. Merge fix(daemon): reclaim control slots after retained connection failure (#540) #543 first — now rebased + mergeable/clean (head d788cc0a2). It provides CapacityModel, the capacity notice, Slots::finish always Vacant, and the bounded RetainedFailures store that these slices build on.

    1–2. Restore the deleted daemon regression suite as package integration tests; make Slots/RetainedFailures pub; extract disposition/capacity_report; delete dead AdmissionClass::Lease.

    3–4. (one delivery) recovery-class reserved slot — when the pool is full, the reserved slot admits only recovery ops (close, workspace status, backend.daemon.connection_failures, reclaim); anything else → Capacity + close. Idle reclamation — a worker parked in receive() past idle_ttl is closed by the daemon (no operation in flight, nothing lost).

    1. In-flight expiry — past request_deadline record RetainedFailure { cause: Expired, reply_sent: false }, free the slot, caller observes Unavailable { outcome: unknown } (never silently kill an operation of unknown effect). backend.daemon.reclaim { slot, generation } — authenticated, generation-checked, one-shot, never restarts the daemon, never touches Workspaces.

    2. Typed admission reason (resource / pool_vacant / reserved_vacant / retryable), outcome / retry_after, #[non_exhaustive] / #[must_use], version-skew matrix.

    Discipline:

  5. mengmengmengqiang commented on Nov 12, 2026

    @mengmengmengqiang
    ContributorAuthor

    Closing — all six slices landed

    Slices 1–6 are merged into layerfs_v2_0_0, so the two classes this issue tracked are now both closed:

    Slice PR Merge
    1–2 restore the daemon slot regression suite + structure #650 79fc41f3
    3–4 recovery class (reserved slot) + idle reclamation #651 cbb7c41e
    5–6 in-flight expiry + backend.daemon.reclaim + contract cleanup #652 dbd633fa

    Acceptance mapping

    Acceptance (from the issue) Covered by
    8 idle slots held, 9th admitted within idle_ttl + handshake_ms slice 4 (daemon_slots, slot_state)
    pool 8/8, sandbox.close (no force) succeeds and drains/reports every Workspace slice 3 (recovery-class admission)
    a parked worker past request_deadline → Expired; backend.daemon.reclaim {slot,generation} frees it; stale generation → Invalid slice 5
    slot invariants have in-tree tests CI runs (previously 0) slice 1/2
    no refusal attributes a cause the caller cannot observe (Capacity names resource/retained; legacy EOF → Unknown{observed}) slice 6

    Evidence (Dell aluminium, rustc 1.85.1, merged head dbd633fa)

    cargo fmt --all -- --check                    rc=0
    cargo test -p layerfs-daemon --test daemon_slots --test slot_state
      daemon_slots 7/7 · slot_state 20/20
    cargo test -p layerfs-sandbox -p layerfs-api --no-fail-fast
      164 passed, 0 failed, 0 ignored
    

    Not in this issue (unchanged non-goals)

    Per-owner quota/priority/fair queue (the protocol carries no client identity), persistence for retained records, metrics/alerting/dashboards, and the #538/#544 custody layer. #543 remains the base; #540 is closed and linked.

    Nothing else is open here — closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

new featureA new user-facing capability

Type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions