Skip to content

sandbox.get: daemon_capacity is absent exactly at a full daemon pool; counts only in down_reason #649

Closed

Description

@mengmengmengqiang

Problem

sandbox.get's daemon_capacity field is omitted exactly in the saturated-pool case it exists to explain. At a full daemon pool the call succeeds, but the structured counts are only inside the down_reason prose, and daemon_capacity is absent.

Measured on this branch after #68 (stand-in daemon that writes only the LAYERFS-CAPACITY notice, 8-slot pool with 7 held):

waited = 60.1 s
got: state = running, workspaces = [], next_page_pos = null
     down_reason = "Sandbox s1's daemon does not answer at 127.0.0.1:41413:
       the Sandbox's daemon has no vacant control slot: 0 pool slot(s) and
       1 reserved slot(s) are vacant, with 3 retained failure record(s).
       The call was not admitted; retrying is safe; ..."
     daemon_capacity = (absent)

Chain (verified)

  1. daemon/pool.rs::open_admitted — after the 60 s admission window returns a precise Capacity refusal whose detail has resource: daemon_control_slot, pool_vacant, reserved_vacant, retained.
  2. direct.rs::unanswered — re-codes every refusal that is not Busy/StaleSandbox to Unavailable, folds the old message into the new text, and drops detail.
  3. direct.rs::list — reports Unavailable as Unreached.
  4. get.rs — Unreached becomes down_reason (text only); capacity() returns None because down is true, so daemon_capacity is not attached.

Hard constraint

The capacity read cannot fill the gap either: capacity() uses lend_within(2 s) on the same control pool, so on a full pool it also gets Capacity and returns None. The pre-handshake LAYERFS-CAPACITY notice is the only source of these counts in the saturated case.

Candidate fix (separate change, not in #543)

Expose the notice's counts, without faking daemon_capacity (its CapacityModel::parse fills missing fields with 0, so a partial notice would produce misleading pool_running: 0 / failures: []):

  • keep the original detail in direct::unanswered for the Capacity case, and
  • surface it from sandbox.get as a new explicit field, e.g.
    down: { code, resource, pool_vacant, reserved_vacant, retained },
    documented as a partial model, absent on the legacy-EOF path (admission_refusal → retained: null) and when a remote refusal is collapsed to {phase} by values::refusal().

Do not let Capacity pass through unanswered: that makes sandbox.get fail outright and loses the record, which is worse.

Notes

Activity

  1. added a commit that references this issue on Nov 12, 2026
  2. mengmengmengqiang commented on Nov 12, 2026

    @mengmengmengqiang
    ContributorAuthor

    Closed: the full-pool capacity notice is now a partial down field on sandbox.get

    PR #655 (merge commit 03883a63cf406ab2329d5e007fa17e1c84c89b29, merged into layerfs_v2_0_0). No fake daemon_capacity, and Capacity does not pass through unanswered: the get keeps answering.

    What changed

    • direct::unanswered keeps the original Capacity refusal detail (it already carries resource, pool_vacant, reserved_vacant, retained, retryable, retry_after) and tags it with its code, instead of dropping it when re-coding to Unavailable.
    • sandbox.get exposes a new explicit, partial field beside down_reason:
    down: { code: "Capacity", resource: "daemon_control_slot",
            pool_vacant, reserved_vacant, retained }
    
    • It is never written into daemon_capacity: CapacityModel::parse would fill the missing fields with 0, so a notice would read as a false pool_running: 0 / empty failures.

    The three absence cases

    down is absent, not zero-filled, when the refusal has no counts:

    1. legacy EOF — the daemon closes without the notice; unanswered keeps no detail (down_reason only);
    2. normal, non-full get — the daemon answers, so daemon_capacity is shown and down is absent;
    3. remote-collapsed refusal — values::refusal() reduces a worker-forwarded refusal to {phase}, which has no capacity fields.

    Tests (layerfs-api/tests/sandbox_capacity.rs, 8/8)

    • a_full_daemon_pool_shows_the_partial_down_counts — reproduces the measured behaviour: the get waits the whole 60 s admission window (asserted >= 55 s; run took 60.34 s) and then answers state: running, down_reason, down = { code: Capacity, resource: daemon_control_slot, pool_vacant: 0, reserved_vacant: 1, retained: 3 }, daemon_capacity absent.
    • a_legacy_eof_has_no_down_counts — down_reason present, down absent.
    • a_running_sandbox_s_get_shows_its_daemon_capacity — down absent, daemon_capacity intact.
    • a_collapsed_remote_refusal_has_no_down_counts — {phase} detail yields None.

    Evidence

    Dell aluminium (x86_64, 88 cores), rustc 1.85.1, branch on the merged layerfs_v2_0_0 @ dbd633fac:

    cargo +1.85.1 fmt --all -- --check                                              # rc=0
    cargo +1.85.1 clippy --workspace --exclude layerfs-node --all-targets -- -D warnings   # rc=0
    cargo +1.85.1 test -p layerfs-sandbox -p layerfs-api --no-fail-fast             # rc=0
        164 passed, 0 failed; tests/sandbox_capacity.rs 8/8 in 60.34 s
    

    Full log: /home/dell/649-final.log. Docs updated: sandbox/get.md, sandbox/README.md, daemon-channel.md.

    The 60 s admission window on a status read is unchanged and remains its own question; this change only makes the counts visible when it ends.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions