Skip to content

Unmount: force while a foreground exec orphan holds the mount leaves Retained(Detach), and later unmount/force/close cannot recover #538

Open

Description

@mengmengmengqiang

A forced unmount while a foreground workspace.exec orphan still holds the mount returns Retained(TeardownCustody { stage: Detach, detached: false, ... }), and from then on normal unmounts, another force, and sandbox.close all keep failing — even after the orphan command has exited. This came out of verifying #458; the clean path (no force) recovers normally.

Sequence: client SIGKILL during foreground exec → Busy → force → Retained(Detach) → no recovery

Sequence: the client dies during a foreground exec; unmount and discard return Busy; force returns Retained(Detach); after the orphan exits, later unmounts and a second force still return Retained, and close stays MountsActive. The green control has no early force and recovers.

Environment: macOS 26.5.1, OrbStack 7.2.9, kernel 7.2.9-orbstack-00391, Docker 29.8.2; LayerFS 9a03429a (#496/#497 present), real Sandbox.

Steps

  1. sandbox.create(), workspace.mount(sandbox, "lane").
  2. Start a foreground workspace.exec(ws, "sleep 20") (the command's cwd is the mount), then SIGKILL the client process that owns the exec.
  3. workspace.unmount(ws) → Busy (phase: unmount:kernel); unmount(ws, true) (discard) → Busy.
  4. While the orphan is alive, workspace.unmount(ws, true, true) (force) →
Failed: OperationFailure {
  request: ForceUnmount { ..., relinquish_unknown: false },
  cause: Retained(TeardownCustody {
    stage: Detach, detached: false,
    work: Some(MountWork { loops_configured: 2, loops_entered: 2, loops_exited: 2, loops_joined: 2, active: 0, retained: 0, ... }),
    detail: "forced drain stopped at Detach: detach Failed(EBUSY), loops Some(SessionSnapshot { phase: Joined, ... }), work Some(MountWork { active: 0, retained: 0, requests: 5, completed: 5, terminal: true })",
    forced: Some(ForcedFacts { abort: Written, detach: Busy, commit: Absent, fenced: 0 })
  })
}
  1. Wait 30 s (the sleep 20 orphan has exited). Six normal unmounts all return the same Retained(stage: Detach, detached: false); a second force also fails; sandbox.close("1") returns MountsActive with workspaces: ["/workspaces/1"].

Control

Without the early force: unmount retries at 5/10/15 s are Busy; at ~20 s (after sleep 20 exits) the normal unmount returns Unmounted (1033 ms, the #497 grace), and sandbox.close returns Closed (375 ms).

Question

Is the post-force state intended to be unrecoverable? The orphan's cwd/descriptor was still alive when the detach failed (EBUSY), so Retained(Detach) is a reasonable custody record; but once the holder is gone, there is no SDK path to retry the detach from that record. Options:

  • make a later unmount (or a second force) re-attempt the detach once the holder is gone, or
  • refuse force while a cwd/descriptor holder exists and say the caller must kill/wait first, and document that state.

At minimum, the /458 docs answer should mention that force can leave Retained(Detach) and that the workspace then cannot be unmounted or its Sandbox closed. No code change is claimed here; this is a diagnostic report with the exact custody.

Related: #458 (foreground exec client death leaves the command running and the mount Busy), #422 (Workspace/container leak), #497 (join grace, not involved in the failed force detach).

Activity

  1. mengmengmengqiang commented on Nov 3, 2026

    @mengmengmengqiang
    ContributorAuthor

    Uploaded the probe used for this report:

    https://github.com/tokenaissance/layerfs/releases/download/pr-evidence-2026-10-28/layerfs-538-force-retained-probe.mjs

    Usage (it expects to sit beside container.mjs under api/agent/test/verify/):

    cp layerfs-538-force-retained-probe.mjs api/agent/test/verify/
    DOCKER_CONTEXT=orbstack node api/agent/test/verify/layerfs-538-force-retained-probe.mjs outer <bundle-dir>

    The script starts a real Sandbox, mounts, starts workspace.exec("sleep 20"), SIGKILLs the client, then does unmount → discard → force → six normal retries → second force → sandbox.close, printing each result and the full Retained custody.

    I could not add labels (Must have admin rights to Repository). Suggested labels: bug, question, orbstack — a maintainer can apply them.

  2. mengmengmengqiang commented on Nov 3, 2026

    @mengmengmengqiang
    ContributorAuthor

    @yifanxuaaa could you take a look when you have a moment, and add labels? I could not (Must have admin rights to Repository). Suggested: bug, question, orbstack.

    The report has the full Retained(Detach) custody and a reproducer probe linked above; the clean no-force path recovers, while the early force path does not. Related: #458, #422, #497.

  3. mengmengmengqiang commented on Nov 4, 2026

    @mengmengmengqiang
    ContributorAuthor

    @yifanxuaaa The #538 fix is being migrated to layerfs_v2_0_0 (draft PR #544, from #541). Could you add the v2.0.0 label to this issue? This account cannot add labels.

  4. added
    bugSomething isn't working
    questionFurther information is requested
    on Nov 5, 2026
  5. added theissue type on Nov 5, 2026
  6. added this to the LayerFS v2.0.0 milestone on Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

bugSomething isn't workingquestionFurther information is requested

Type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions