Repository navigation
Closed
Labels
new featureA new user-facing capabilityA new user-facing capability
Milestone
Description
Summary
#540's retained-slot leak is fixed by #543: Application::serve no longer leaves a failed connection's slot Retained, and every post-decode failure is logged and queryable. Two capacity failure classes remain, and both still end at the same user-visible point — a Sandbox whose SDK calls fail and whose only recovery is sandbox.close --force or killing the daemon, losing uncommitted Workspaces.
- An admitted slot has no lifetime. After the first request is decoded the daemon clears the socket read/write timeouts (
engine/crates/layerfs-daemon/src/application/connection.rs:305-315), andSlotState::Runningcarries no deadline (application/owner.rs:71-74). A worker parked in a Store wait, a fence, a diagnostic transfer oranswer_callholds 1 of 8 pool slots forever, with no log line and no reclaim verb. Eight of them reproduce Daemon control slots leak: a connection that fails after a request stays Retained forever, and the daemon then refuses every connection #540 exactly. - Capacity shortage still requires destroying the data plane. The one reserved slot is installer-only (
owner.rs:28,connection.rs:148-159,owner.rs:455-464);AdmissionClass::Leaseis dead code. With the pool full, a non-forcesandbox.closefails in 2 s with an exact refusal (sandbox/serve.rs:43) and the only way out isforce, which reports undrained Workspaces asunknownand loses their uncommitted changes.
This issue adds the missing lifetime bounds and a recovery admission path. It deliberately does not add per-owner quotas or fairness (see Non-goals: the protocol carries no client identity).
Scope
- Give every admitted slot a phase with an explicit bound:
Handshake { deadline }(exists,handshake_ms),InFlight { started, deadline },Idle { since }. - Reclaim idle slots automatically: a slot whose worker is blocked in
receive()pastidle_ttlis closed by the daemon. This loses nothing (no operation is in flight); the client pool quarantines the connection and opens another (api/crates/layerfs-control/src/daemon/pool.rs:412-432). - Mark in-flight slots that pass
request_deadlineas expired: recordRetainedFailure { cause: Expired, reply_sent: false }, free the slot, and have the caller observeUnavailable { outcome: unknown }. Do not silently kill an operation whose effect is unknown. - Add one authenticated operator method,
backend.daemon.reclaim { slot, generation }, dispatched likebackend.daemon.connection_failures(api/crates/layerfs-control/src/daemon/operator.rs). It closes the named worker's socket, recordsReclaimed, and answersInvalidon a generation mismatch. One attempt, no retry, never restarts the daemon and never touches Workspaces. - Turn the reserved slot into a recovery class: when the pool has no vacant slot, admit one connection into the reserved slot and, after decoding, allow only the recovery operation set (close, workspace status,
backend.daemon.connection_failures, reclaim). Any other request is refusedCapacityand the connection is closed — the shape that already exists for the installer (connection.rs:351-358, 419-429). - Expose the new counts in the existing read model:
admission.idle,admission.inflight,admission.reclaimableinlayerfs-bridge::ControlCapacity, defaulted when absent.
Non-goals
- No per-owner quota, priority or fair queue.
HelloRequesthas no client identity, onlyexpected_instance; the daemon cannot tell which process owns a connection. A real quota needs a new authenticated admission identity and a trust model — that is a separate decision, and the client pool already degrades to a sharedlimit(pool.rs:379-383). - No persistence for retained records.
daemon.logis the durable evidence; the 64-record ring is a query cache (document this; it is currently described as a limitation). - No change to Unmount: force while a foreground exec orphan holds the mount leaves Retained(Detach), and later unmount/force/close cannot recover #538/fix(workspace): force-kill only the target Workspace on v2 (#538) #544's workspace custody (
Retained(Detach),workspace.holders, TERM→KILL). Different layer; see Compatibility. - No metrics framework, alerting or dashboards. Log fields and the read model stay stable so that work can follow.
- No new trait, repository, event bus or async runtime.
Invariants
- Every admitted slot ends
Vacantexactly once; a stale or duplicate finish changes nothing. - No slot stays
Idlepastidle_ttl, and no slot staysInFlightpastrequest_deadlinewithout becoming a recordedExpired/Reclaimeddisposition. - A failure record never owns a slot; the record store stays bounded (64 / 1 h) with a
droppedcount. - At most one connection holds the reserved slot; it serves install during startup and recovery afterwards, and it is closed after one request.
- Closing, reading status of, or reclaiming a Sandbox never requires destroying its data plane.
forceremains the last resort, not the only recovery. - Reclaim is authenticated, generation-checked, one-shot, and reports
outcome: unknownunless the operation is known idempotent. - A single worker's panic or error never stops the accept loop.
Acceptance
- With all 8 pool slots held by idle clients, a 9th client is admitted within
idle_ttl + handshake_ms; the evicted client's next call either reuses a quarantined pool entry or opens a new connection and succeeds. - With the pool 8/8 occupied (idle or in-flight),
sandbox.close(noforce) succeeds, drains or reports every Workspace by name, and preserves a 64 MiB uncommitted change (cksum unchanged). - A worker parked in a fence/Store/diagnostic wait for longer than
request_deadlineappears asExpiredinbackend.daemon.connection_failures;backend.daemon.reclaim { slot, generation }frees it without restarting the daemon; the Workspace content and cksum are unchanged; a stale generation answersInvalid. - The daemon's slot invariants have in-tree tests that CI runs (today
cargo test -p layerfs-daemon --libruns zero), and the suite fails on the pre-fix(daemon): reclaim control slots after retained connection failure (#540) #543 base commit. - No refusal attributes a cause the caller cannot observe:
Capacitynamesresource,pool_vacant,reserved_vacant, andretryable; legacy EOF answersUnknown { observed }, never "no vacant slot".
Tests
| # | Test | Layer | Asserts |
|---|---|---|---|
| 1 | 8 post-decode client deaths, then a fresh client | in-tree integration (real daemon binary + TCP client) | slot freed, new client admitted, one log line per failure |
| 2 | retained record via the operator call | in-tree integration | retained.count > 0, reply_sent, request, cause |
| 3 | ring limit/TTL/eviction/dropped |
entity unit (state machine made pub) |
65th record evicts the oldest; TTL prunes; dropped counts |
| 4 | stale/duplicate finish | entity unit | Finished::Stale, state unchanged |
| 5 | panic and guard drop | entity + integration | exactly one finish; slot released |
| 6 | 16 concurrent failures | in-tree integration | no deadlock; 8 vacant |
| 7 | notice when no pool slot is free | in-tree integration | 24-byte LAYERFS-CAPACITY frame, reason NoPoolSlot |
| 8 | 8 idle clients + a 9th | in-tree integration | idle eviction, then admission |
| 9 | sandbox.close with the pool full |
in-tree control integration | closes without force; cksum preserved |
| 10 | parked worker + reclaim | in-tree integration | Expired, reclaim, generation mismatch → Invalid |
| 11 | reserved slot refuses a non-recovery request | in-tree integration | Capacity, connection closed |
| 12 | accept loop survives a worker panic | unit | next connection still admitted |
| 13 | version skew: new daemon × old client, old daemon × new client, new control × old worker bundle | nightly | no hang; each failure is named |
| 14 | 3 processes × 64 MiB commits | nightly | bounded waits; close still possible |
| 15 | 24 h soak: periodic client kills, periodic capacity read | nightly | pool_vacant / retained time series has no drift |
| 16 | close during a #544 holder kill | nightly (with #544) | bounded; no false Capacity |
PR slices
- Restore the regression suite (test-only): the four deleted daemon assertions, as package integration tests; make
Slots/RetainedFailurespubso the pure state machine is unit-testable. No behavior change. Fails on the base commit. - Structure (no behavior change): move
RetainedFailuresout ofSlotsintoState; extractdisposition(&ConnectionFailure, SlotTicket, &str, Instant) -> Disposition; extractcapacity_report(SlotCounts, RetainedSnapshot) -> CapacityModel; passSlotClassinstead ofslot: usizetocontrol_operation/hello; deleteAdmissionClass::Lease,AdmissionReason::ReservedBusy, and the three aliases ofCAPACITY_OPERATION. Guarded by slice 1. - Recovery class: reserved-slot admission when the pool is full; post-decode allow-list of recovery methods; one request then close. Tests 9, 11.
- Idle reclamation:
Phase::Idle { since }maintained at request boundaries;reapon refusal. Test 8. - In-flight expiry + reclaim:
Phase::InFlight { started, deadline },Expireddisposition,backend.daemon.reclaim. Tests 10, 12. - Contract cleanup: typed admission reason,
outcome/retry_after, drop the message-substring classification inpool.rs,#[non_exhaustive]/#[must_use], 5-minute runbook, version-skew matrix.
Slices 1–2 are the debt from #543 and can start immediately. 3–4 are one delivery. 5 depends on 3 and 4. 6 lands with the #544 merge.
Relation to #540 / #543 / #545
- Daemon control slots leak: a connection that fails after a request stays Retained forever, and the daemon then refuses every connection #540 stays open until the invariants above are enforced and tested. fix(daemon): reclaim control slots after retained connection failure (#540) #543 fixed the retained-slot class; this issue closes the "an admitted slot has no lifetime" class. Keep Daemon control slots leak: a connection that fails after a request stays Retained forever, and the daemon then refuses every connection #540 open, or reopen it as a tracking issue, until slice 4 lands.
- fix(daemon): reclaim control slots after retained connection failure (#540) #543 must merge first: it is the base for slices 1–2 and provides
CapacityModel, the notice and the one-finish protocol this issue extends. Do not definebackend.daemon.reclaimtwice. - Unmount: force while a foreground exec orphan holds the mount leaves Retained(Detach), and later unmount/force/close cannot recover #538 / fix(workspace): force-kill only the target Workspace on v2 (#538) #544 are a different custody layer (Workspace/mount). fix(workspace): force-kill only the target Workspace on v2 (#538) #544 rebases onto fix(daemon): reclaim control slots after retained connection failure (#540) #543 and must not be merged before it: both edit
api/crates/layerfs-control/src/workspace/serve.rs::unmount, and the close budget must becomeholder_kill_budget + margin(today: close probe 2 s, unmount probe 5 s, stop 5 s). - [B] Sandbox compute plane: machines, backends, observability and lifecycle #545 B5 ("abandoned Sandbox reclamation") shares the "bounded lifetime + generation CAS" idea but operates on the Sandbox record, not the daemon's admission slots. Do not merge the two issues; cross-reference them.
- The connect layer: machines, Sandboxes and Stores as typed, extensible units #535 / layerfs-control: one management layer for machines, Stores, Sandboxes and Workspaces #536 own the records and the SDK surface. This issue only adds an operator method and additive
sandbox.getfields — no record schema change, no SDK breaking change. - Version line: 2.0.0 only (
layerfs_v2_0_0).mainhas nodaemon/pool.rs; there is no backport path. If 2.0.0 ships before this lands, it needs a 2.0.1 and a release note, not a silent patch.
Metadata
Metadata
Assignees
Labels
new featureA new user-facing capabilityA new user-facing capability
Activity
mengmengmengqiang commentedon Nov 4, 2026
Merge / closure conditions
CapacityModel, the capacity notice and the one-finish protocol this issue extends. Do not definebackend.daemon.reclaimtwice.api/crates/layerfs-control/src/workspace/serve.rs::unmount). With it, the close budget becomesholder_kill_budget + margin, and test 16 (close during a fix(workspace): force-kill only the target Workspace on v2 (#538) #544 holder kill) lands.Tracking: #540 (bug, remains open), #543 (PR, must merge first), #544 (PR, rebases after #543), #545 (B5, cross-reference only).
test(540): cover the guard-drop release and correct slice 1 claims (E…
docs(547): commit the pre-Ephemeral-AI-Lab#543 repro and correct the …
docs(547): point the base repro at the renamed death test (Ephemeral-…
docs(545): commit the two scenes' editable Excalidraw sources
docs(545): commit the two scenes' editable Excalidraw sources
docs(545): commit the two scenes' editable Excalidraw sources
mengmengmengqiang commentedon Nov 9, 2026
Landing order (slices)
d788cc0a2). It providesCapacityModel, the capacity notice,Slots::finishalwaysVacant, and the boundedRetainedFailuresstore that these slices build on.1–2. Restore the deleted daemon regression suite as package integration tests; make
Slots/RetainedFailurespub; extractdisposition/capacity_report; delete deadAdmissionClass::Lease.3–4. (one delivery) recovery-class reserved slot — when the pool is full, the reserved slot admits only recovery ops (
close, workspace status,backend.daemon.connection_failures,reclaim); anything else →Capacity+ close. Idle reclamation — a worker parked inreceive()pastidle_ttlis closed by the daemon (no operation in flight, nothing lost).In-flight expiry — past
request_deadlinerecordRetainedFailure { cause: Expired, reply_sent: false }, free the slot, caller observesUnavailable { outcome: unknown }(never silently kill an operation of unknown effect).backend.daemon.reclaim { slot, generation }— authenticated, generation-checked, one-shot, never restarts the daemon, never touches Workspaces.Typed admission reason (
resource/pool_vacant/reserved_vacant/retryable),outcome/retry_after,#[non_exhaustive]/#[must_use], version-skew matrix.Discipline:
layerfs_v2_0_0(notmain).backend.daemon.reclaimtwice — build on fix(daemon): reclaim control slots after retained connection failure (#540) #543's capacity model.test(547): restore the deleted daemon slot regression suite and publi…
refactor(547): structure the daemon slot custody for the recovery work (
feat(547): recovery-class reserved slot and idle reclamation (Ephemer…
feat(547): expire in-flight slots and add backend.daemon.reclaim (Eph…
refactor(547): type the admission reason and the refusal outcome (Eph…
test(547): restore the deleted daemon slot regression suite and publi…
refactor(547): structure the daemon slot custody for the recovery work (
feat(547): recovery-class reserved slot and idle reclamation (Ephemer…
feat(547): expire in-flight slots and add backend.daemon.reclaim (Eph…
refactor(547): type the admission reason and the refusal outcome (Eph…
mengmengmengqiang commentedon Nov 12, 2026
Closing — all six slices landed
Slices 1–6 are merged into
layerfs_v2_0_0, so the two classes this issue tracked are now both closed:79fc41f3cbb7c41ebackend.daemon.reclaim+ contract cleanupdbd633faAcceptance mapping
idle_ttl + handshake_msdaemon_slots,slot_state)sandbox.close(noforce) succeeds and drains/reports every Workspacerequest_deadline→Expired;backend.daemon.reclaim {slot,generation}frees it; stale generation →InvalidCapacitynames resource/retained; legacy EOF →Unknown{observed})Evidence (Dell
aluminium, rustc 1.85.1, merged headdbd633fa)Not in this issue (unchanged non-goals)
Per-owner quota/priority/fair queue (the protocol carries no client identity), persistence for retained records, metrics/alerting/dashboards, and the #538/#544 custody layer. #543 remains the base; #540 is closed and linked.
Nothing else is open here — closing.