Repository navigation
Description
Problem
sandbox.get's daemon_capacity field is omitted exactly in the saturated-pool case it exists to explain. At a full daemon pool the call succeeds, but the structured counts are only inside the down_reason prose, and daemon_capacity is absent.
Measured on this branch after #68 (stand-in daemon that writes only the LAYERFS-CAPACITY notice, 8-slot pool with 7 held):
waited = 60.1 s
got: state = running, workspaces = [], next_page_pos = null
down_reason = "Sandbox s1's daemon does not answer at 127.0.0.1:41413:
the Sandbox's daemon has no vacant control slot: 0 pool slot(s) and
1 reserved slot(s) are vacant, with 3 retained failure record(s).
The call was not admitted; retrying is safe; ..."
daemon_capacity = (absent)
Chain (verified)
daemon/pool.rs::open_admitted— after the 60 s admission window returns a preciseCapacityrefusal whosedetailhasresource: daemon_control_slot,pool_vacant,reserved_vacant,retained.direct.rs::unanswered— re-codes every refusal that is notBusy/StaleSandboxtoUnavailable, folds the old message into the new text, and dropsdetail.direct.rs::list— reportsUnavailableasUnreached.get.rs—Unreachedbecomesdown_reason(text only);capacity()returnsNonebecausedownis true, sodaemon_capacityis not attached.
Hard constraint
The capacity read cannot fill the gap either: capacity() uses lend_within(2 s) on the same control pool, so on a full pool it also gets Capacity and returns None. The pre-handshake LAYERFS-CAPACITY notice is the only source of these counts in the saturated case.
Candidate fix (separate change, not in #543)
Expose the notice's counts, without faking daemon_capacity (its CapacityModel::parse fills missing fields with 0, so a partial notice would produce misleading pool_running: 0 / failures: []):
- keep the original
detailindirect::unansweredfor theCapacitycase, and - surface it from
sandbox.getas a new explicit field, e.g.
down: { code, resource, pool_vacant, reserved_vacant, retained },
documented as a partial model, absent on the legacy-EOF path (admission_refusal→retained: null) and when a remote refusal is collapsed to{phase}byvalues::refusal().
Do not let Capacity pass through unanswered: that makes sandbox.get fail outright and loses the record, which is worse.
Notes
- Related: fix(daemon): reclaim control slots after retained connection failure (#540) #543 (the PR that restored
daemon_capacity; this gap is pre-existing, not introduced by it), daemon: bound an admitted control slot's lifetime and keep recovery admissible when the pool is exhausted (#540 follow-up) #547 (daemon-side retained-record tests). - The 60 s admission window is a separate question:
sandbox.getwaits the full window before returningdown_reason. If that is considered too long for a status read, it should be its own issue.
Activity
Merge pull request #655 from Ephemeral-AI-Lab/fix/649-down-capacity
mengmengmengqiang commentedon Nov 12, 2026
Closed: the full-pool capacity notice is now a partial
downfield onsandbox.getPR #655 (merge commit
03883a63cf406ab2329d5e007fa17e1c84c89b29, merged intolayerfs_v2_0_0). No fakedaemon_capacity, andCapacitydoes not pass throughunanswered: the get keeps answering.What changed
direct::unansweredkeeps the originalCapacityrefusaldetail(it already carriesresource,pool_vacant,reserved_vacant,retained,retryable,retry_after) and tags it with itscode, instead of dropping it when re-coding toUnavailable.sandbox.getexposes a new explicit, partial field besidedown_reason:daemon_capacity:CapacityModel::parsewould fill the missing fields with 0, so a notice would read as a falsepool_running: 0/ emptyfailures.The three absence cases
downis absent, not zero-filled, when the refusal has no counts:unansweredkeeps no detail (down_reasononly);daemon_capacityis shown anddownis absent;values::refusal()reduces a worker-forwarded refusal to{phase}, which has no capacity fields.Tests (
layerfs-api/tests/sandbox_capacity.rs, 8/8)a_full_daemon_pool_shows_the_partial_down_counts— reproduces the measured behaviour: the get waits the whole 60 s admission window (asserted>= 55 s; run took 60.34 s) and then answersstate: running,down_reason,down = { code: Capacity, resource: daemon_control_slot, pool_vacant: 0, reserved_vacant: 1, retained: 3 },daemon_capacityabsent.a_legacy_eof_has_no_down_counts—down_reasonpresent,downabsent.a_running_sandbox_s_get_shows_its_daemon_capacity—downabsent,daemon_capacityintact.a_collapsed_remote_refusal_has_no_down_counts—{phase}detail yieldsNone.Evidence
Dell
aluminium(x86_64, 88 cores),rustc 1.85.1, branch on the mergedlayerfs_v2_0_0@dbd633fac:Full log:
/home/dell/649-final.log. Docs updated:sandbox/get.md,sandbox/README.md,daemon-channel.md.The 60 s admission window on a status read is unchanged and remains its own question; this change only makes the counts visible when it ends.
fix(sandbox): carry a full-pool capacity notice into sandbox.get (Eph…
fix(sandbox): read the worker's full pool as down in sandbox.get (#662)