Skip to content

feat(daemon): reclaim private-seat scratch and merged-PR worktrees via a leased idle pass - #14776

Open
loom-fleet-dispatch[bot] wants to merge 2 commits into
mainfrom
feature/issue-14678
Open

loom-fleet-dispatch[bot] wants to merge 2 commits into
mainfrom
feature/issue-14678

Conversation

@loom-fleet-dispatch

@loom-fleet-dispatch loom-fleet-dispatch Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add the periodic private-workspace seat pass (remaining Builder increment of #14678; the seat-volume telemetry AC landed earlier via #14693): a leased, idle-only maintenance verb that reclaims inside loom-codex-workspace-<seat> volumes — ended step scratch and clean, published, merged-past-grace linked worktrees — which no host-side pass could reach.

Changes

  • New private-workspace seat-reclaim in-volume verb (private_workspace/seat_reclaim.rs): removes cache/<role>-<n>.* and top-level *-tmp / <role>-<n>.* step dirs past scratchReclaim.maxAgeHours (future-dated mtimes never reclaimable), and reaps a linked worktree only when clean, fully published, unlocked, in-volume, non-detached, and its branch has a confirmed merged PR at that exact tip past gracePeriodSecs. No quarantine, no backup, plain git worktree remove; every other worktree is preserved with a reason.
  • Host half (lifecycle.rs + docker.rs): enumerates seats, takes each account's exclusive Lease::acquire, requires the container running and idle (docker::idle), and execs the verb — mirroring run_job's admission but with no prepare_target, so a dirty seat still gets scratch reclaim and preserved-dirty reporting. Busy/stopped/non-idle seats are skipped with a reason, never failing the pass.
  • New merged-only PR classifier PrAtHead / pr_at_head (worktree_ops/clean_owner.rs): same request and row strictness as finished_pr_at_head, but distinguishing open / closed-unmerged / merged-at-tip (+ merged_at for the grace gate) — the seat pass preserves what preparation-time force-removal (fix: let recover list and clear dirty leftover worktrees that block a private-clone seat #14483) may remove.
  • Scheduled from eager-reclaim tier 1 (eager_reclaim.rs) with a host-wide cooldown; honors the existing scratchReclaim.* surface and gracePeriodSecs — no new flags.
  • Reporting: per-seat reaper.pass spans (class seat, seat account name as the repo label, preserved reasons and bytes); reaper.pass schema row in defaults/docs/telemetry-schema.md gains the seat class. Ownership split vs preparation (fix: let recover list and clear dirty leftover worktrees that block a private-clone seat #14483) recorded in defaults/docs/private-session-dispatch.md ("Who reaps what: preparation vs the periodic seat pass").
  • Host-side ownership skip test-pinned: worktree_reaper keeps an unrelated host issue-N checkout a private job owns.

Acceptance Criteria Verification

Criterion Status Verification
Scratch: ended-step reclaim past maxAgeHours; live steps never touched ✅ seat_reclaim::tests::scratch_reclaim_removes_ended_steps_and_keeps_live_and_future_dated_ones (ended vs live vs future-dated vs non-step vs reserved roots); liveness is structural — the verb runs only under the seat's account lease with the container proven idle
Worktree rule: clean + published + merged-past-grace only; others preserved with reason; no quarantine/backup ✅ reap_merged_worktrees gates in order (locked / outside volume / detached / unpublished / dirty / open / closed-unmerged / grace / unreadable); removal is plain git worktree remove — no RemoveWithQuarantine/RemoveWithBackup route exists in the module
Fixture: eligible beside dirty(published+merged-past-grace)/unpublished/open-PR/closed-unmerged; only eligible disappears ✅ only_the_eligible_checkout_is_removed; negative check: disabling only the cleanliness gate makes it fail (dirty sibling removed), restored to green
Host ownership check preserved; unrelated host checkout untouched ✅ worktree_reaper::tests::test_private_workspace_owned_host_worktree_is_never_reaped; private_route.rs unchanged
Entry point: leased maintenance verb, running+idle, no prepare_target; dirty seat still reclaims scratch ✅ lifecycle.rs::seat_reclaim (acquire → inspect → validate → lease.recover → running+idle); tests::seat_reclaim_exec_drives_the_verb_without_any_preparation asserts argv contains no prepare and no token leak; a_dirty_seat_still_reclaims_its_scratch
Ownership vs #14483 recorded in defaults/docs/private-session-dispatch.md; preparation behavior documented separately ✅ New "Who reaps what" section; re-checked against merged #14483 — preparation paths untouched by this diff
reaper.pass reclaimed bytes + preserved reasons ✅ observability::ops::reaper::tests::a_seat_pass_span_names_the_seat_and_its_class (class seat, seat name label, reason categories, no host path); schema row updated
Seat volume size metric + telemetry-schema doc ✅ Already done via merged #14693 (loom.host.codex_workspace_volume.*); not re-done here
Reuses scratchReclaim.* config; no new flags ✅ run_for reads read_scratch_reclaim_config + worktree_reaper grace; the verb's flags are host→container plumbing params, not operator config

Test Plan

Scoped runs in the issue worktree, all green: cargo test -p loom-daemon --lib -- seat_reclaim eager_reclaim observability::ops::reaper private_workspace_owned_host_worktree (45 passed), CI's exact clippy line for loom-daemon+loom-api --all-targets (clean after one obfuscated_if_else fix), cargo fmt --check, cargo check --workspace, all re-verified after merging current origin/main. Full suite is CI's gate.

TDD: partial — module and tests originate from a prior interrupted attempt in this same worktree (recovered, reviewed, completed: docs, schema row, clippy fix); the key fixture's fail-first was verified here by disabling the cleanliness gate alone and confirming only_the_eligible_checkout_is_removed fails because the dirty sibling is removed.

Live verification NOT performed. The issue Test Plan's manual step (run the pass on a worker with a real seat volume and confirm bytes freed + reaper.pass reasons) requires a live seat container, unavailable in this worktree; no offline fixture claims live behavior. The pass is scheduled from eager-reclaim tier 1 on every fleet worker, so the first below-floor (or 15m-tick) pass post-merge produces the live reaper.pass/seat spans and log lines to confirm.

Closes #14678

loom dashboard

…a a leased idle pass

Add the periodic private-workspace seat pass (#14678): a leased
maintenance verb (account lease, container proven running and idle, no
prepare step) that removes ended step scratch (cache/<role>-<n>.* and
top-level *-tmp / <role>-<n>.* dirs past scratchReclaim.maxAgeHours) and
reaps in-volume linked worktrees only when clean, fully published,
unlocked, and confirmed merged at that exact tip past the reaper grace.
Dirty, unpublished, open-PR, closed-unmerged and unreadable worktrees are
preserved with reasons through reaper.pass spans (new class 'seat');
the host reaper's private-workspace ownership skip is test-pinned.
Scheduled from eager-reclaim tier 1; reuses scratchReclaim.* and
gracePeriodSecs (no new flags). Ownership split vs preparation-time
reaping (#14483) recorded in defaults/docs/private-session-dispatch.md.

Closes #14678
@loom-fleet-dispatch loom-fleet-dispatch Bot added loom:review-requested PR ready for Judge to review. Applied by: Builder when opening PR. loom:important Priority 2 of 3: taken before unlabeled work, not a hold. Applied by: operator, Guide, Champion. labels Nov 20, 2026
@loom-fleet-dispatch

loom-fleet-dispatch Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor Author

PR #14776's loom:reviewing claim is held by the judge of sweep step-judge-14776-1795172696 on host host-a6032f0c. This comment is a PR lease record (#9401): its liveness signal is its own forge-assigned updated_at, renewed while the judge works and released when it is done. Another judge that finds this lease live stands down before touching a worktree. See defaults/docs/lease-record.md § "PR-phase leases".

@loom-fleet-dispatch loom-fleet-dispatch Bot added the loom:reviewing Judge is reviewing this PR. Applied by: Judge only. Stale after LOOM_STALE_REVIEWING_MINUTES (30m). label Nov 20, 2026
@loom-fleet-dispatch

Copy link
Copy Markdown
Contributor Author

Judge: code review of 346bfacf8 complete (diff, linked-issue spec reconciliation, review-feedback gate all done); now waiting in the foreground for the 8 pending Rust/native check-runs to settle before the verdict.

@loom-fleet-dispatch

Copy link
Copy Markdown
Contributor Author

✅ Approved!

Reviewed at 346bfacf8 in the existing builder worktree .loom/worktrees/issue-14678 (resynced to the PR head first — see the worktree note at the end). ~1,500 added lines across 14 files; CI green at this exact head.

Spec reconciliation

Re-read #14678's current body (the 2026-11-19 and 2026-11-20 ## Revision sections, Champion-APPROVED 2026-11-20T03:54:42Z) and its trusted comments rather than this PR's account of them.

What I verified

Destructive-path audit (the thing that matters here). Removal is gated, in order, by: canonical-path identity against the clone root (the main checkout is decremented out of worktrees_scanned and never a candidate), entry.locked, canonical.starts_with(root) (in-volume only), branch != "HEAD" (no detached), ahead_of_origin == 0 (fully published), empty worktree_status (clean), then PrAtHead::Merged(merged_at) only, then check_grace_period. Measured by grep over the new module: the only removal calls are remove_dir_all/remove_file on shape-matched scratch and a plain git worktree remove — no --force, no RemoveWithQuarantine, no RemoveWithBackup anywhere in seat_reclaim.rs. The unforced git worktree remove is a second, independent refusal for dirty/locked state on top of the explicit gates. Every other outcome appends a preserve reason and continues.

PrAtHead parity with finished_pr_at_head. Compared pr_at_head_in_rows line-by-line against finished_pr_in_rows (clean_owner.rs:297): same rows.len() >= 100 truncation refusal, same foreign-base refusal, same missing-head/missing-head-repo refusal, same head-branch-mismatch refusal, same continue on a foreign head repo. Classification differs only as intended — Open returns immediately (so an open PR beats a historical merge, matching the old precedence), ClosedNoMerge preserves instead of counting as finished, Merged requires head.sha == tip, and NoPr/Unknown from classify_pr_row fail closed to Unknown. Checked order-independence of the merged/closed-unmerged pair: Merged assigns unconditionally while ClosedNoMerge only fills an empty slot, so row order cannot change the answer.

Preserve wordings land in real telemetry categories. Traced each new wording through preserve_reason (observability/ops/reaper.rs:53) by hand: uncommitted changes in worktree→uncommitted, unpushed commits on HEAD/PR closed without merging (not merged)/no merged PR for the branch at this tip→not_merged, merged PR grace period not passed→grace (the grace period branch is checked first, so it wins over the merged substring), worktree locked (in use)→in_use, PR still open→pr_open, PR status unknown…→pr_unknown. The three remaining wordings (worktree outside the private volume, detached HEAD (unnamed work), worktree state unreadable) fall to other, which is a legitimate member of the fixed set. telemetry-schema.md's reaper.pass row gains the seat class and documents that loom.repo is the seat account name; the span test asserts no host path leaks into any attribute.

Scratch shape gates. Walked is_step_scratch_name against every name that actually lives at a seat volume root (repo, work, cache, identity.json, plus work/.identity and .cloning-* staging, confirmed by grepping the root-level root.join(...) call sites in repository.rs/bundle.rs): none has a <role>-<digits> shape, so no reserved root is reachable as a candidate at any age. is_reclaimable_age rejects negative ages, so a clock-skewed future mtime is never reclaimable. tree_bytes uses DirEntry::file_type()/DirEntry::metadata(), neither of which follows symlinks, so a symlink in scratch cannot send the walk out of the volume.

Entry point matches the AC. lifecycle::seat_reclaim_one is Lease::acquire → docker::inspect → docker::validate → lease.recover → State.Running == true && docker::idle → exec, with no prepare_target — confirmed both by reading and by seat_reclaim_exec_drives_the_verb_without_any_preparation, which asserts prepare appears nowhere in argv and that the token rides on the child env, never argv. The lease is released by Drop (flock on the held File); no finish() is needed because the pass never calls begin(), so it writes no job.json. Every early return is a per-seat skip string, so one busy/stopped/non-idle seat never fails the pass.

Test quality. only_the_eligible_checkout_is_removed is the AC's fixture built correctly: six linked worktrees, with the dirty sibling pushed and mapped to Merged(30 days ago) so cleanliness is provably the only thing preserving it; worktrees_scanned == 6 confirms the main checkout was excluded rather than skipped silently; the final rev-parse refs/heads/feature/issue-10 assertion confirms the branch and its commits survive the checkout's removal. The scratch test's scratch_scanned == 6 / scratch_removed == 4 arithmetic checks out against the fixture (the tool-cache non-step entry, the live entry and the future-dated entry are all correctly excluded or preserved). main_checkout_and_foreign_worktrees_are_never_candidates covers a host-path worktree the clone still knows about.

TDD claim: the line reads TDD: partial, with the fail-first done by disabling the cleanliness gate alone and watching only_the_eligible_checkout_is_removed fail on the dirty sibling. Per the Judge TDD table a partial/explained claim is advisory, and the stated method is sound — the fixture is constructed so that mutation is exactly what it detects. Recorded as reported by the builder, not re-derived here: I did not re-run the mutated build (CI is the test gate, #11690).

Test Execution

PR test plan (scoped cargo test -p loom-daemon --lib -- seat_reclaim eager_reclaim observability::ops::reaper private_workspace_owned_host_worktree, clippy --all-targets, cargo fmt --check, cargo check --workspace):

  1. Scoped unit runs (45 passed) — ⏭️ Not re-run: CI is the test gate (Host-side test burden to a bare minimum: CI is the test gate, Builders don't run the suite (operator direction 2026-10-13) #11690) and all 24 check-runs are green at 346bfacf8, including Rust Unit Tests (1-5/5) and Native Port Suites. Cited, not re-derived.
  2. Clippy / fmt / cargo check --workspace — ⏭️ Not re-run: same green CI covers these.
  3. Live verification (run the pass on a worker with a real seat volume, confirm bytes freed + reaper.pass reasons) — ⚠️ Not executed; requires a live seat container. See below.

Live verification: unverified, and that is explicit

The PR states plainly that the issue's manual step could not be performed, and no fixture here claims live behavior — the honest path. I checked the two triggers in the Judge "Live Verification and the Circular-Fixture Smell" rule and neither fires: the diff drives no browser, scraper or foreign DOM/HTML parser (trigger A), and #14678's body reports unbounded disk growth, not a silent-omission failure — it contains none of the silent-omission fragments (trigger B). So the blocking evidence requirement does not attach, and I am not stamping loom:ac-verified: nobody has run this against a real seat volume, and I will not certify a step I did not see output from.

No circular fixture: the one place a fabricated payload could hide is the PR-state side, and it is injected through the PrLookup seam with the real row-classification logic tested separately against PrRowRest JSON shapes. The git side of every fixture is produced by real git invocations against a real local remote, not synthesized.

The live gap is real and worth naming: this pass's first production run will be its first contact with a 40 GB, 72-clone seat volume, and the in-container verb only exists in session images built from this code — existing seat containers will answer the protocol error and be skipped (handled: it maps to a per-seat skip reason, not a pass failure), so the pass no-ops on the fleet until containers are recycled. The PR's own plan — the first below-floor pass post-merge produces the reaper.pass/seat spans to confirm — is the right follow-through. Flagging it so it is watched rather than assumed.

Non-blocking: filed #14777

The seat sub-pass runs synchronously inside the dispatch loop's tick (work_finder.rs:1981 → tick_line::sample_pre_span awaits it), walks seats sequentially, and each seat's docker exec is bounded by EXEC_TIMEOUT = DOCKER_RUN_TIMEOUT = 600s. That makes the per-tick worst-case bound seats x 600s — up to ~70 minutes on the 7-seat worker from #14678 — paid at most once per DEFAULT_SCRATCH_MIN_INTERVAL_SECS (1800s), while free space is below the floor, i.e. during the very incident this pass exists for. (Those four numbers are read from the code; the actual wall clock of one seat's pass is not measured — 600s is a ceiling, not an expectation.)

This is not a defect in this PR: #14678's curated "Affected Files" names eager_reclaim.rs - schedule the sub-pass, so the placement is the spec's, and #14560 already established that a whole reclaim pass holds the tick back. It is the first sub-pass to make that window this large, which is worth bounding separately — filed as #14777 with the measurement step first.

Nit (not worth a round-trip)

repository.rs loses the blank line between ahead_of_origin's closing brace and the following doc comment (an incidental deletion in the pub(super) visibility change). cargo fmt --check is green, so it is cosmetic only. Relatedly, observability/ops/reaper.rs's Pass::class field doc still reads /// \issue` / `pr`.whiletelemetry-schema.mdwas correctly updated to includeseat` — a one-line comment drift, fine to pick up in any later touch of that file.

Worktree note

The builder worktree .loom/worktrees/issue-14678 was found at 79199f47d with its upstream mis-pointed at origin/main (the #6095/#6100 defect), so a pull --ff-only fast-forwarded the local branch to main's tip instead of the PR head. Applied the documented recovery — git reset --hard origin/feature/issue-14678 + git branch --set-upstream-to=origin/feature/issue-14678 — and re-verified HEAD == 346bfacf8 with an empty git status before reading a single line of the diff. The worktree was clean throughout, so nothing was at risk, and nothing was pushed from it.

Clean, careful work on a genuinely dangerous surface — the gate ordering, the merged-only classifier's parity with its sibling, and the fixture that makes cleanliness the sole discriminator are all exactly right.

loom dashboard

@loom-fleet-dispatch loom-fleet-dispatch Bot added loom:pr PR approved by Judge, ready for Champion auto-merge. Applied by: Judge. and removed loom:review-requested PR ready for Judge to review. Applied by: Builder when opening PR. loom:reviewing Judge is reviewing this PR. Applied by: Judge only. Stale after LOOM_STALE_REVIEWING_MINUTES (30m). labels Nov 20, 2026
@loom-fleet-dispatch

loom-fleet-dispatch Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor Author

PR #14776's (no label: the lease is the claim) claim is held by the champion of sweep step-champion_merge-14776-1795173439 on host host-e1d4c843. This comment is a PR lease record (#9401): its liveness signal is its own forge-assigned updated_at, renewed while the champion works and released when it is done. Another champion that finds this lease live stands down before touching a worktree. See defaults/docs/lease-record.md § "PR-phase leases".

@loom-fleet-dispatch

Copy link
Copy Markdown
Contributor Author

Champion: Holding for Human Merge

Judge-approved and passing the mechanical safety criteria (CI green, mergeable, fresh verdict), but not auto-merging:

  • Revertability / Blast radius: adds a scheduled pass (seat_reclaim.rs, wired from eager_reclaim.rs tier 1) that runs git worktree remove and deletes scratch dirs inside every loom-codex-workspace-<seat> volume on every fleet worker once rolled. The deletion is one-way, so git revert -m 1 stops future passes but cannot restore a worktree wrongly reaped. It also touches the worktree-deletion and reaper paths (worktree_ops/clean_owner.rs, worktree_reaper.rs, observability/ops/reaper.rs) and the tokens_pool seat lease. These are kept-set entries (worktree and branch deletion, reaper, account binding), so self-release is not available. The PR also states live verification was not performed on a real seat volume.

Next steps — this hold stays in force until one of these happens; Champion re-reading the same diff will not clear it:

  • A human releases it and merges in one audited step: ./.loom/scripts/merge-pr.sh 14776 --release-hold
  • Or applies loom:auto-merge-ok to override the hold; Champion merges on the next tick
  • Or comments here to clear it explicitly, starting the comment with the instruction: clear the hold — <why>
  • Or the situation changes: a new push that narrows the concern, or a deeper Judge re-review

Keeping loom:pr. This PR stays in the queue and is re-checked each tick against the release conditions above.


Automated by Champion role

loom dashboard

@loom-fleet-dispatch loom-fleet-dispatch Bot added the loom:blocked Waiting on something named (another change, a capability); released once it resolves label Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

loom:blocked Waiting on something named (another change, a capability); released once it resolves loom:important Priority 2 of 3: taken before unlabeled work, not a hold. Applied by: operator, Guide, Champion. loom:pr PR approved by Judge, ready for Champion auto-merge. Applied by: Judge.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Private-clone Codex seat volumes grow without bound: a full clone per repo per seat, and nothing reclaims them

0 participants