diff --git a/defaults/docs/private-session-dispatch.md b/defaults/docs/private-session-dispatch.md index c747f33f91..65fa6b9fbe 100644 --- a/defaults/docs/private-session-dispatch.md +++ b/defaults/docs/private-session-dispatch.md @@ -183,6 +183,44 @@ repository. When preparation then refuses (for example `seat_dirty` on a different worktree), the removal is not reported: the refusal names only the worktree that caused it. +### Who reaps what: preparation vs the periodic seat pass + +Two paths reclaim inside a seat volume, with different rules (#14483 vs +#14678): + +- **Job preparation** (per admission, the one target clone only): forgives + untracked caches, reaps linked worktrees whose tip heads a *finished* PR — + merged **or closed-unmerged**, dirty ones included via the forced removal + above — and backs the explicit `accounts session recover` listing and + `--discard`/`--worktree` removal of dirty leftovers. Preparation's reap + runs before its dirty check (`reap_finished_worktrees` before + `clean_worktrees`) and can remove a dirty, unlocked worktree whose tip is + published at a finished PR. That behavior is preparation's own and is not + changed by the seat pass. +- **The periodic seat pass** (idle, every seat and every registered clone): + a leased maintenance verb — the account's exclusive lease, the container + proven running and idle, and **no `prepare` step**, so a dirty seat still + gets its pass — removes step scratch (`cache/-.*`, top-level + `*-tmp` and `-.*` dirs) older than + `autonomous.worktreeReaper.scratchReclaim.maxAgeHours`, and reaps a linked + worktree only when it is clean, fully published, unlocked, in-volume, and + its branch has a confirmed **merged** PR at that exact tip past + `autonomous.worktreeReaper.gracePeriodSecs`. It never quarantines and + never takes a backup; a worktree it cannot prove finished is preserved + with a reason (dirty, unpublished, open PR, closed-unmerged, grace not + passed, state unreadable). The pass honors the existing + `scratchReclaim.*` surface — no new flags — and reports reclaimed bytes + and preserved reasons through `reaper.pass` spans (class `seat`, the span's + repo label the seat's account name) instead of the host reaper's blanket + `preserved.private_workspace` wording for what it now handles. + +The preservation guarantee — a dirty, unpublished, open-PR or +closed-unmerged worktree is never removed — is the **seat pass's** rule. A +dirty worktree whose tip is published at a merged PR past grace is the +dividing case: preparation may force-remove it at the next admission of that +clone, the seat pass preserves and reports it. `.identity` records and the +no-alternates / private-clone invariants are untouched by both paths. + ## Rebinding a registration after a repository transfer A forge transfer changes the owner-qualified URL, and with it the registration diff --git a/defaults/docs/telemetry-schema.md b/defaults/docs/telemetry-schema.md index 49d72fb624..ee089d0b84 100644 --- a/defaults/docs/telemetry-schema.md +++ b/defaults/docs/telemetry-schema.md @@ -2445,7 +2445,7 @@ Guard denies (Issue #11582): |---|---|---|---| | `guard.cargo_target_dir.denied` span | instant; a child of the agent's execution span, delivered through its trace journal (`LOOM_TRACE_CONTEXT_FILE`), so only a session inside a traced execution exports one | `loom.guard.root` (`/tmp`, `$TMPDIR`, `~/.cache`, `~`, `.loom/target-*`), `loom.guard.source` (`inherited`, `inline`, `env`, `export`, `target-dir-flag`, `make-arg`), `loom.role` | one `loom:cargo-target-dir` deny: a cargo build whose target dir was under a root nothing owns ([`guard-hooks.md`](guard-hooks.md#cargo-target-dir-guard-guardscargotargetdir--loom_guard_cargo_target_dir)). Never the path or the command | | `build_gate.run` span | one per host-side run of `buildGate.command`; pre-flight's is a child of the sweep's execution span, delivered through its trace journal (`LOOM_TRACE_CONTEXT_FILE`; an attended session with no `LOOM_TRACEPARENT` emits none), the main-health gate's is its own root through the ops sink | `loom.build_gate.stage` (`preflight` / `main_health`), `loom.build_gate.tier` (`fast` / `full`), `loom.build_gate.outcome` (`pass` / `pass_flaky` / `fail` / `unresolved` / `timeout` / `unevaluated`), `loom.build_gate.duration_ms`, `loom.build_gate.attempt` (1-based within the pre-flight episode; `1` for main-health) | what host-side gating costs and how often it fails, per stage and tier (#11690: CI is the test gate, pre-flight is the fast tier). Status `error` unless `pass` / `pass_flaky`. Never the command, a path or the output | -| `reaper.pass` span | one per worktree-reaper pass over one repo and worktree class, through the ops sink: an instant span, its own root (#12461) | `loom.repo` (workspace directory name, never a path), `loom.reaper.class` (`issue` / `pr`), `loom.reaper.phase` (`reap` removes worktrees; `reclaim` removes build-artifact directories from kept ones), `loom.reaper.scanned` / `removed` / `failed` / `preserved` (worktrees for `reap`; artifact directories removed and worktrees left whole for `reclaim`), `loom.reaper.bytes` (bytes freed; `reap` measures only while an ops sink is registered), `loom.reaper.preserved.` with `` one of `in_use` / `unmanaged` / `editable_install` / `issue_open` / `pr_open` / `pr_unknown` / `not_merged` / `grace` / `uncommitted` / `private_workspace` / `backup_failed` / `other` (the gate's fixed category, never its wording; absent when 0), `loom.reaper.backed_up` (open-item worktrees whose whole state was pushed to a remote backup ref before removal, #12458; always `0` for `reclaim`), `loom.reaper.backup_refs` (those refs, space-separated — `refs/loom/backup///` on `origin`, the restore handle; absent when none) | what the reaper scanned, removed, kept and freed per pass, where #12453 had only host logs and `du`. Status `error` when a removal failed. Carries the daemon version and full SHA like every span | +| `reaper.pass` span | one per worktree-reaper pass over one repo and worktree class, through the ops sink: an instant span, its own root (#12461) | `loom.repo` (workspace directory name, never a path), `loom.reaper.class` (`issue` / `pr` / `seat` — the last the private-workspace seat pass, whose `loom.repo` is the seat's account name, #14678), `loom.reaper.phase` (`reap` removes worktrees; `reclaim` removes build-artifact directories from kept ones), `loom.reaper.scanned` / `removed` / `failed` / `preserved` (worktrees for `reap`; artifact directories removed and worktrees left whole for `reclaim`), `loom.reaper.bytes` (bytes freed; `reap` measures only while an ops sink is registered), `loom.reaper.preserved.` with `` one of `in_use` / `unmanaged` / `editable_install` / `issue_open` / `pr_open` / `pr_unknown` / `not_merged` / `grace` / `uncommitted` / `private_workspace` / `backup_failed` / `other` (the gate's fixed category, never its wording; absent when 0), `loom.reaper.backed_up` (open-item worktrees whose whole state was pushed to a remote backup ref before removal, #12458; always `0` for `reclaim`), `loom.reaper.backup_refs` (those refs, space-separated — `refs/loom/backup///` on `origin`, the restore handle; absent when none) | what the reaper scanned, removed, kept and freed per pass, where #12453 had only host logs and `du`. Status `error` when a removal failed. Carries the daemon version and full SHA like every span | | `main_red.autofile` span | instant, its own root, through the ops sink; one per change the daemon makes to a red `main`'s auto-filed fix issue (Issue #11766) | `loom.repo`, `loom.issue`, `loom.main_red.sha` (the failing or green SHA, `unknown` when not known), `loom.main_red.action` (`created` / `reused` / `closed` / `closed_duplicate`; `reused` once per issue per daemon process), `loom.main_red.source` (`gate` / `ci`) | when a red `main` got its fix issue, and when it was closed again. Status `ok` | | `loom.fleet_dispatch.call` span | its own root, through the ops sink; one per call on each side, `loom.fleet_dispatch.side` says which: `captain`, one per request the fleet-dispatch captain (`loom-daemon fleet-dispatch serve`, #12449) answered; `worker`, one per call a worker makes to it (#12450). Spec `docs/design/fleet-captain-dispatch.md` | `loom.fleet_dispatch.side` (`captain` / `worker`), `loom.fleet_dispatch.host` (on the captain the caller's roster host id, `-` before it is identified; on a worker its own), `loom.fleet_dispatch.endpoint` (`/v1/heartbeat` / `/v1/claim` / `/v1/refuse` / `/v1/release`, `other` for a path the captain does not serve), `loom.fleet_dispatch.latency_ms`, `loom.fleet_dispatch.status` (HTTP status; on a worker `0` with no answer), `loom.fleet_dispatch.result` (`granted` / `refused` / `ok` / `error`, the same set on both sides), `loom.fleet_dispatch.reason` (the refusal reason, `refuse` token, release outcome, or the error class: on a worker `transport` / `status_5xx` / `status_4xx` / `malformed`), `loom.fleet_dispatch.server_epoch`, `loom.assignment.id`, `loom.fleet_dispatch.step` (the step's wire name, when the call names an item), and on a refused claim (#12707) `loom.fleet_dispatch.retry_after_s` (integer seconds), `loom.fleet_dispatch.held_by` (the holding host, only when the captain names one: absent on a no-progress hold), `loom.fleet_dispatch.holder_assignment_id` (#12740: the holder's assignment id, on the captain's span only, only on a `held` refusal from a live row, so absent on a no-progress hold; the wire response does not carry it) and, when the refusal is a no-progress hold (`reason` `no_progress_cooldown` on the captain, `cooldown` on a worker), `loom.fleet_dispatch.hold_until` (RFC 3339 UTC wall clock: the span's time plus `retry_after_s`; no monotonic value is exported). The captain's span also carries that hold's `loom.fleet_dispatch.hold_rung` (consecutive no-progress releases), `loom.fleet_dispatch.hold_set_by_outcome` and `loom.fleet_dispatch.hold_set_by_assignment_id` (the release that set it). On a `champion_merge` claim, release or expiry (#13009) the captain's span also carries `loom.fleet_dispatch.merge_slot_holder`: the repo's `champion_merge` holder (an assignment id) after the call, or `none`, so a holder that outlives its release is visible without joining refusals. Key ETA queries on `reason`, not on `held_by`. Each key is emitted only when known. And `loom.repo` with `loom.issue` or `loom.pr_number` (by whether the step acts on a PR) when the call names an item; a worker's span also carries `loom.role` (the step's role name, as on `loom.dispatch.step`) | which host asked for what and what the captain answered, and when a worker could not reach it (design §12). Both sides use the same keys and values, so one query joins them. Status `error` on a captain's non-200 answer and on a worker's call with no usable answer. `loom.dispatch.step` carries the same `loom.assignment.id` on a granted step, and on a step a captain refused `loom.dispatch.held_by`, `loom.dispatch.retry_after_s` and (for `captain_cooldown`) `loom.dispatch.hold_until` (#12707), and (#12875) on a step the captain granted after an earlier assignment of the same repo, item and step ended: `loom.dispatch.attempt` (how many earlier assignments ended), `loom.dispatch.prev_assignment_id` (join it to the earlier step's `loom.assignment.id`) and `loom.dispatch.prev_end` (`released` / `expired` / `host_lost`, the last when the worker released it because its host was lost to a spot interruption, #12876). A first grant, and a renewed one, carries none of the three. The captain's lineage record is bounded (4096 steps, 24 h) and in memory only: a captain restart resets `attempt`. Each assignment that expires also gets a captain span with `loom.fleet_dispatch.endpoint` `expire`, `loom.fleet_dispatch.result` `expired` and `loom.fleet_dispatch.reason` `lease_expired`, carrying the host, `loom.repo`, `loom.issue` or `loom.pr_number`, `loom.fleet_dispatch.step` and `loom.assignment.id` of the row (a held row's expiry emits none). The captain also hosts the fleet issue-filing lock (#13918): `/v1/filing-lock` (`result` `granted` / `refused`; `reason` `granted`, `refreshed` or `held`, with `held_by` and `retry_after_s` on a refusal) and `/v1/filing-unlock` (`reason` `released`, `not_holder` or `empty`, and `loom.fleet_dispatch.hold_ms`: how long the slot was held, on `released` only), and a hold whose holder never unlocked ends in a captain span with `loom.fleet_dispatch.endpoint` `filing-expire`, `result` `expired`, `reason` `lease_expired` and `hold_ms`. A worker's filing span is one per acquire, whose `latency_ms` is the whole wait: `result` `granted`, `refused` with `reason` `deferred` (the slot stayed held past the bounded wait; the burst did not file), or `error` with `reason` `fallback` or the failed call's token (the burst filed under the host lock alone: the fallback count). The worker side runs in the one-shot `fleet-dispatch filing-lock` subcommand, which registers the standalone ops sink at startup and flushes its queue once at exit (#14129); what a flush could not export within its bound stays queued, retried by the next filing's flush | diff --git a/loom-daemon/src/eager_reclaim.rs b/loom-daemon/src/eager_reclaim.rs index 6c9ab53830..5357845ab9 100644 --- a/loom-daemon/src/eager_reclaim.rs +++ b/loom-daemon/src/eager_reclaim.rs @@ -43,6 +43,12 @@ //! reap has **no cooldown of its own** and makes real REST calls to the forge //! per candidate worktree. //! +//! #14678 adds the private-workspace seat pass +//! ([`crate::tokens_pool::private_workspace::seat_reclaim`]) to tier 1: the +//! scratch and finished-work debris inside seat volumes is exactly the "Loom +//! debris nobody is using" this tier exists for, and no other sub-pass can +//! reach it. +//! //! # Tiers, safest first (#11192) //! //! [`run_pass`] escalates through three tiers and re-probes free space @@ -51,7 +57,8 @@ //! sub-passes' own gates): //! //! 1. **Loom debris nobody is using**: merged-PR worktrees, aborted-fetch -//! `.git` temp files, agent scratch, orphaned cargo target dirs. +//! `.git` temp files, agent scratch, seat-volume scratch and merged-PR +//! worktrees (#14678), orphaned cargo target dirs. //! 2. **Idle build caches of kept worktrees**, across every registered root //! (#11071). Costs a rebuild if the worktree is resumed. //! 3. **The primary checkout's own build cache** (deep clean) and **docker @@ -253,6 +260,10 @@ pub struct EagerReclaimReport { pub docker: Option, /// The `/tmp`-shaped scratch sub-pass's own report (#7512 item 3). pub scratch: Option, + /// The private-workspace seat-volume sub-pass's own report (#14678): + /// step scratch and merged-PR worktrees inside seat volumes the other + /// sub-passes cannot reach. + pub seats: Option, /// The orphaned cargo target dir sub-pass's own report (#8370). pub target_orphans: Option, /// When this pass ran. @@ -278,6 +289,10 @@ impl EagerReclaimReport { self.worktrees_removed > 0 || self.git_tmp.as_ref().is_some_and(|r| r.totals.files > 0) || self.scratch.as_ref().is_some_and(|r| r.removed_count > 0) + || self + .seats + .as_ref() + .is_some_and(|r| r.worktrees_removed() > 0 || r.scratch_removed() > 0) || self .target_orphans .as_ref() @@ -326,6 +341,10 @@ impl EagerReclaimReport { .scratch .as_ref() .map_or_else(|| "n/a".to_string(), ScratchReclaimReport::removed_human); + let seats = self.seats.as_ref().map_or_else( + || "n/a".to_string(), + crate::tokens_pool::private_workspace::seat_reclaim::SeatReclaimReport::summary, + ); let git_tmp = self .git_tmp .as_ref() @@ -351,9 +370,9 @@ impl EagerReclaimReport { format!( "eager_reclaim: {} {} ({} free) — ran an out-of-cycle pass (tier {} of 3) now \ instead of waiting up to 15m for worktree_reaper's own scheduled pass: worktrees \ - {worktrees}, git-tmp {git_tmp}, scratch {scratch}, cargo-target orphans \ - {target_orphans}, idle worktree targets {idle}, deep-clean {deep}, docker {docker} \ - — now {} free vs. floor {}G (#7512, #11192)", + {worktrees}, git-tmp {git_tmp}, scratch {scratch}, seat volumes {seats}, \ + cargo-target orphans {target_orphans}, idle worktree targets {idle}, deep-clean \ + {deep}, docker {docker} — now {} free vs. floor {}G (#7512, #11192)", self.repo_root.display(), TRIGGER_PHRASE, gb_or_unknown(self.free_gb_before), @@ -423,6 +442,10 @@ pub struct SubPasses<'a> { pub docker: &'a dyn Fn(&Path) -> DockerRetentionReport, /// [`crate::scratch_reclaim::run_for`]. pub scratch: &'a dyn Fn(&Path) -> ScratchReclaimReport, + /// [`crate::tokens_pool::private_workspace::seat_reclaim::run_for`] + /// (#14678): the in-volume seat pass (scratch + merged-only worktrees). + pub seats: + &'a dyn Fn(&Path) -> crate::tokens_pool::private_workspace::seat_reclaim::SeatReclaimReport, /// [`crate::target_orphan_reclaim::run_for`] (#8370). pub target_orphans: &'a dyn Fn(&Path) -> TargetOrphanReport, /// Free GB on the worktree-root volume, sampled before, between tiers, @@ -535,6 +558,7 @@ pub fn run_pass( deep_clean: None, docker: None, scratch: None, + seats: None, target_orphans: None, at: inputs.now, git_tmp: None, @@ -576,6 +600,7 @@ pub fn run_pass( // then re-probes free space after it — as in `reap_repo`. report.git_tmp = Some((passes.git_tmp)(repo_root)); report.scratch = Some((passes.scratch)(repo_root)); + report.seats = Some((passes.seats)(repo_root)); report.target_orphans = Some((passes.target_orphans)(repo_root)); // Tier 2 (#11071): kept worktrees' idle build caches, largest first, until @@ -665,6 +690,7 @@ pub fn run_for(repo_root: &Path) -> EagerReclaimReport { let deep_clean = |root: &Path| crate::deep_clean::run_for(root, floor_gb); let docker = crate::docker_image_clean::run_for; let scratch = crate::scratch_reclaim::run_for; + let seats = crate::tokens_pool::private_workspace::seat_reclaim::run_for; let target_orphans = crate::target_orphan_reclaim::run_for; let free_gb = crate::disk_headroom::worktree_root_free_gb; let git_tmp = crate::git_tmp_reclaim::run_for; @@ -684,6 +710,7 @@ pub fn run_for(repo_root: &Path) -> EagerReclaimReport { deep_clean: &deep_clean, docker: &docker, scratch: &scratch, + seats: &seats, target_orphans: &target_orphans, free_gb: &free_gb, git_tmp: &git_tmp, @@ -826,11 +853,19 @@ mod tests { counters.order.borrow_mut().push("idle_targets"); Some(stub_idle_target_report()) }; + let seats = |_: &Path| { + counters.order.borrow_mut().push("seats"); + crate::tokens_pool::private_workspace::seat_reclaim::SeatReclaimReport { + skipped: None, + seats: Vec::new(), + } + }; let passes = SubPasses { reap_worktrees: &reap, deep_clean: &deep, docker: &docker, scratch: &scratch, + seats: &seats, target_orphans: &target_orphans, free_gb: &free_gb, git_tmp: &git_tmp, @@ -861,6 +896,7 @@ mod tests { "worktrees", "git_tmp", "scratch", + "seats", "target_orphans", "idle_targets", "deep", @@ -873,6 +909,7 @@ mod tests { assert!(report.deep_clean.is_some()); assert!(report.docker.is_some()); assert!(report.scratch.is_some()); + assert!(report.seats.is_some()); assert!(report.target_orphans.is_some()); assert_eq!(report.free_gb_before, Some(3)); assert_eq!(report.free_gb_after, Some(3)); @@ -896,7 +933,7 @@ mod tests { inputs.ledger.last_run = Some(t(0)); let report = run_with_counters(&inputs, &counters); assert!(report.skipped.is_none()); - assert_eq!(counters.total(), 7); + assert_eq!(counters.total(), 8); } #[test] @@ -944,6 +981,11 @@ mod tests { let reap = |_: &Path| 0usize; let docker = |_: &Path| stub_docker_report(now); let scratch = |root: &Path| stub_scratch_report(root, now); + let seats = + |_: &Path| crate::tokens_pool::private_workspace::seat_reclaim::SeatReclaimReport { + skipped: None, + seats: Vec::new(), + }; let target_orphans = |root: &Path| stub_target_orphan_report(root); let free_gb = |_: &Path| Some(1u64); let git_tmp = |root: &Path| stub_git_tmp_report(root, now); @@ -953,6 +995,7 @@ mod tests { deep_clean: &deep, docker: &docker, scratch: &scratch, + seats: &seats, target_orphans: &target_orphans, free_gb: &free_gb, git_tmp: &git_tmp, diff --git a/loom-daemon/src/eager_reclaim/level_tests.rs b/loom-daemon/src/eager_reclaim/level_tests.rs index 139701b243..bf52418176 100644 --- a/loom-daemon/src/eager_reclaim/level_tests.rs +++ b/loom-daemon/src/eager_reclaim/level_tests.rs @@ -141,11 +141,17 @@ impl Sim { }; let idle = |root: &Path| run_idle_targets_if_due(root, FLOOR, now, COOLDOWN, &idle_body); + let stub_seats = + |_: &Path| crate::tokens_pool::private_workspace::seat_reclaim::SeatReclaimReport { + skipped: None, + seats: Vec::new(), + }; let passes = SubPasses { reap_worktrees: &reap, deep_clean: &deep, docker: &docker, scratch: &scratch, + seats: &stub_seats, target_orphans: &orphans, free_gb: &free_gb, git_tmp: &git_tmp, @@ -227,6 +233,13 @@ fn run_tiered(probes: &[Option]) -> (EagerReclaimReport, Vec<&'static str>) order.borrow_mut().push("scratch"); scratch_report(root, now, 0) }; + let seats = |_: &Path| { + order.borrow_mut().push("seats"); + crate::tokens_pool::private_workspace::seat_reclaim::SeatReclaimReport { + skipped: None, + seats: Vec::new(), + } + }; let orphans = |root: &Path| { order.borrow_mut().push("target_orphans"); orphan_report(root) @@ -248,6 +261,7 @@ fn run_tiered(probes: &[Option]) -> (EagerReclaimReport, Vec<&'static str>) deep_clean: &deep, docker: &docker, scratch: &scratch, + seats: &seats, target_orphans: &orphans, free_gb: &free_gb, git_tmp: &git_tmp, @@ -265,7 +279,7 @@ fn run_tiered(probes: &[Option]) -> (EagerReclaimReport, Vec<&'static str>) (report, called) } -const TIER1: [&str; 4] = ["worktrees", "git_tmp", "scratch", "target_orphans"]; +const TIER1: [&str; 5] = ["worktrees", "git_tmp", "scratch", "seats", "target_orphans"]; #[test] fn test_tier_one_only_when_already_above_the_floor() { diff --git a/loom-daemon/src/observability/ops/reaper.rs b/loom-daemon/src/observability/ops/reaper.rs index 8ee3b9b5f1..332c10bfdc 100644 --- a/loom-daemon/src/observability/ops/reaper.rs +++ b/loom-daemon/src/observability/ops/reaper.rs @@ -227,12 +227,60 @@ mod tests { "backup_failed", ), ("something nobody categorised", "other"), + // #14678: the in-volume seat pass's preserve wordings, so the + // seat spans land in the same fixed categories. + ("uncommitted changes in worktree", "uncommitted"), + ("unpushed commits on HEAD", "not_merged"), + ("PR closed without merging (not merged)", "not_merged"), + ("no merged PR for the branch at this tip", "not_merged"), + ("merged PR grace period not passed", "grace"), + ("worktree locked (in use)", "in_use"), ] { assert_eq!(preserve_reason(wording), want, "{wording}"); assert!(PRESERVE_REASONS.contains(&preserve_reason(wording))); } } + /// #14678: a seat pass's span names the seat (never a host path) and + /// carries the seat class beside the workspace classes. + #[test] + fn a_seat_pass_span_names_the_seat_and_its_class() { + let p = Pass { + class: "seat", + phase: PHASE_REAP, + scanned: 6, + removed: 1, + failed: 0, + preserved: vec![ + "uncommitted changes in worktree", + "unpushed commits on HEAD", + "PR still open", + "PR closed without merging (not merged)", + "merged PR grace period not passed", + ], + bytes: 2048, + backed_up: 0, + backup_refs: Vec::new(), + }; + let span = pass_span(Path::new("/private-workspace-seats/agent-1"), &p, at()); + assert!(span.validate().is_ok()); + let bounded = span.clone().bounded(); + for (key, want) in [ + ("loom.repo", "agent-1"), + ("loom.reaper.class", "seat"), + ("loom.reaper.preserved.uncommitted", "1"), + ("loom.reaper.preserved.not_merged", "2"), + ("loom.reaper.preserved.pr_open", "1"), + ("loom.reaper.preserved.grace", "1"), + ] { + assert_eq!(bounded.attributes.get(key).map(String::as_str), Some(want), "{key}"); + } + assert!(!span + .attributes + .values() + .any(|v| v.contains("/private-workspace-seats"))); + } + #[test] fn the_span_carries_counts_reasons_bytes_and_provenance() { let preserved = vec!["PR still open", "PR still open", "uncommitted changes"]; diff --git a/loom-daemon/src/tokens_pool/private_workspace/docker.rs b/loom-daemon/src/tokens_pool/private_workspace/docker.rs index 453378f310..a96c6bca1e 100644 --- a/loom-daemon/src/tokens_pool/private_workspace/docker.rs +++ b/loom-daemon/src/tokens_pool/private_workspace/docker.rs @@ -665,6 +665,50 @@ pub(super) fn recover_target( .context("private workspace recover returned an invalid report") } +/// Drive the in-volume `private-workspace seat-reclaim` verb (#14678) under +/// the credential scope of the account's primary repository — the same exec +/// shape as [`recover_target`] with **no preparation step**: the verb must +/// run in a dirty seat too, so nothing here calls [`prepare_target`]. +pub(super) fn seat_reclaim( + config: &Config, + max_age_secs: i64, + grace_secs: i64, + auth: &forge_auth::ExecAuth, +) -> Result { + let scope = format!("LOOM_PRIVATE_REPOSITORY={}", config.repository); + let max_age = max_age_secs.to_string(); + let grace = grace_secs.to_string(); + let mut args = vec!["exec", "--workdir", ROOT, "--env", &scope]; + for name in auth.names() { + args.extend(["--env", name]); + } + args.extend([ + &config.container, + "loom-daemon", + "private-workspace", + "seat-reclaim", + "--account", + &config.account, + "--max-age-secs", + &max_age, + "--grace-secs", + &grace, + ]); + let output = command_env(&args, &auth.child_env(), EXEC_TIMEOUT).context( + "private workspace endpoint failed; use a current session image supporting private-workspace protocol", + )?; + let result: Value = serde_json::from_str(&output) + .context("private workspace endpoint unavailable; update the session image before reuse")?; + if result["protocol"] != PROTOCOL { + bail!("private workspace endpoint protocol mismatch; update session image"); + } + if let Some(error) = result["error"].as_str() { + bail!("private clone seat reclaim refused: {error}"); + } + serde_json::from_value(result["outcome"].clone()) + .context("private workspace seat-reclaim returned an invalid report") +} + /// Drive the in-volume `private-workspace rebind` verb (#14357) under the /// credential scope of the CANONICAL repository. pub(super) fn rebind_target( diff --git a/loom-daemon/src/tokens_pool/private_workspace/lifecycle.rs b/loom-daemon/src/tokens_pool/private_workspace/lifecycle.rs index ec7ba9bd2f..f7d282951c 100644 --- a/loom-daemon/src/tokens_pool/private_workspace/lifecycle.rs +++ b/loom-daemon/src/tokens_pool/private_workspace/lifecycle.rs @@ -868,3 +868,76 @@ pub fn run_job(workspace: &Path, args: JobArgs) -> Result { lease.finish()?; Ok(code) } + +/// The leased seat maintenance pass (#14678): the periodic, all-clones idle +/// counterpart of [`run_job`]'s admission. Same exclusive account lease and +/// the same containment + running-and-idle proof, but **no job and no +/// `docker::prepare_target`** — preparation runs `clean_worktrees`, which +/// refuses a dirty worktree, while this pass must reach a dirty seat's +/// scratch and report its dirty worktrees as preserved instead of stopping. +/// +/// Every seat is independent: a busy, stopped or non-idle seat is skipped +/// with a reason (a retry is the next pass's job, not a failure), never +/// holding up the others. `.identity` records and the no-alternates / +/// private-clone invariants are untouched — the in-container verb only ever +/// removes step scratch and clean, published, merged-past-grace worktrees. +pub fn seat_reclaim( + workspace: &Path, + max_age_secs: i64, + grace_secs: i64, +) -> Vec<(String, Result)> { + use super::super::account_registry::{account_inventory, AccountProvider}; + + let mut out = Vec::new(); + let Ok(accounts) = account_inventory(workspace, AccountProvider::Codex) else { + return out; + }; + for descriptor in accounts { + let name = descriptor.id.name.clone(); + // A tolerant read: an account without a private workspace is not a + // seat, and one whose state cannot be read is skipped below by name. + let Some(config) = registrations(&descriptor.credential_reference) else { + continue; + }; + out.push((name, seat_reclaim_one(workspace, &config, max_age_secs, grace_secs))); + } + out +} + +/// One seat's pass; every early return is a skip reason, not an error worth +/// failing the host pass over. +fn seat_reclaim_one( + workspace: &Path, + config: &Config, + max_age_secs: i64, + grace_secs: i64, +) -> Result { + let dir = state_dir(&config.profile).map_err(|e| e.to_string())?; + let lease = lease::Lease::acquire(&dir).map_err(|error| { + if error.downcast_ref::().is_some() { + "seat busy; a job holds the account lease (retry next pass)".to_owned() + } else { + error.to_string() + } + })?; + let state = docker::inspect(&config.container) + .map_err(|e| e.to_string())? + .ok_or_else(|| "session is stopped; nothing to reclaim this pass".to_owned())?; + docker::validate(config, &state).map_err(|e| e.to_string())?; + lease + .recover(config, Some(&state)) + .map_err(|e| e.to_string())?; + if state["State"]["Running"] != true + || !docker::idle(Some(&state), &config.container).map_err(|e| e.to_string())? + { + return Err("seat is not running and idle; retry next pass".into()); + } + if state["Id"].as_str().is_none() { + return Err("container id unavailable".into()); + } + // The credential scope of the account's primary repository, the same + // context `recover` execs under: the verb's PR probes need a forge + // identity, and the scope never widens past the account's own remotes. + let auth = forge_auth::ExecAuth::resolve(workspace, &config.repository); + docker::seat_reclaim(config, max_age_secs, grace_secs, &auth).map_err(|e| e.to_string()) +} diff --git a/loom-daemon/src/tokens_pool/private_workspace/mod.rs b/loom-daemon/src/tokens_pool/private_workspace/mod.rs index 13cd50d469..124dbb6bd8 100644 --- a/loom-daemon/src/tokens_pool/private_workspace/mod.rs +++ b/loom-daemon/src/tokens_pool/private_workspace/mod.rs @@ -19,6 +19,7 @@ mod lifecycle; #[cfg(test)] mod recover_tests; mod repository; +pub(crate) mod seat_reclaim; #[cfg(test)] mod tests; pub mod transport; diff --git a/loom-daemon/src/tokens_pool/private_workspace/repository.rs b/loom-daemon/src/tokens_pool/private_workspace/repository.rs index 503a8bec1a..e8d2f2e687 100644 --- a/loom-daemon/src/tokens_pool/private_workspace/repository.rs +++ b/loom-daemon/src/tokens_pool/private_workspace/repository.rs @@ -44,6 +44,10 @@ pub enum WorkerCommand { /// location, preserving every branch and file (`accounts session /// rebind`, issue #14357). Rebind(RebindArgs), + /// The periodic seat reclaim (#14678): remove ended step scratch and + /// clean, published, merged-past-grace linked worktrees inside this + /// volume. Host-driven, under the seat lease, while the seat is idle. + SeatReclaim(super::seat_reclaim::SeatReclaimArgs), /// Git credential-helper protocol; output goes only to Git's private pipe. Credential { operation: String, @@ -203,6 +207,22 @@ impl WorkerCommand { } } } + Self::SeatReclaim(args) => { + let lookup = seat_reclaim::github_pr_at_head; + match seat_reclaim::seat_reclaim( + Path::new(ROOT), + &args, + &lookup, + chrono::Utc::now(), + ) { + Ok(outcome) => { + println!("{}", json!({"protocol": PROTOCOL, "outcome": outcome})) + } + Err(error) => { + println!("{}", json!({"protocol": PROTOCOL, "error": error.to_string()})) + } + } + } Self::Credential { operation } => credential(&operation)?, } Ok(()) @@ -221,7 +241,7 @@ fn helper_cause(stderr: &[u8]) -> Option<&'static str> { HELPER_CAUSES.into_iter().find(|cause| text.contains(cause)) } -fn git(root: &Path, args: &[&str]) -> Result { +pub(super) fn git(root: &Path, args: &[&str]) -> Result { let result = Command::new("git") .arg("-C") .arg(root) @@ -553,13 +573,12 @@ pub(super) fn unpublished_branches(repo: &Path) -> Result Ok(unpublished) } -fn ahead_of_origin(repo: &Path, revision: &str) -> Result { +pub(super) fn ahead_of_origin(repo: &Path, revision: &str) -> Result { git(repo, &["rev-list", "--count", revision, "--not", "--remotes=origin"])? .trim() .parse::() .context("workspace Git returned a non-numeric ahead count") } - /// The clone paths a `recover` call acts on. Unlike `additional_paths` this /// never creates anything: a clone that was never materialized has nothing /// to recover. @@ -1118,12 +1137,12 @@ fn worktrees(repo: &Path) -> Result> { .collect()) } -struct WorktreeEntry { - path: PathBuf, - locked: bool, +pub(super) struct WorktreeEntry { + pub(super) path: PathBuf, + pub(super) locked: bool, } -fn worktree_entries(repo: &Path) -> Result> { +pub(super) fn worktree_entries(repo: &Path) -> Result> { Ok(git(repo, &["worktree", "list", "--porcelain", "-z"])? .split("\0\0") .filter_map(|record| { @@ -1143,7 +1162,7 @@ const DIRTY_PATH_LIMIT: usize = 10; /// (#14458). Porcelain v1 C-quotes special path bytes, including newlines, /// keeping each changed path on a single line. Both prefixes are literal in /// a quoted path, so status of any tracked edit is preserved verbatim. -fn worktree_status(path: &Path) -> Result> { +pub(super) fn worktree_status(path: &Path) -> Result> { let status = git( path, &[ @@ -1294,6 +1313,25 @@ fn finished_pr(repo: &Path, repository: &str, branch: &str, tip: &str) -> bool { crate::worktree_ops::clean_owner::finished_pr_at_head(repo, &host, &slug, branch, tip) } +/// `(repository, account)` recorded for one clone by its identity record +/// (the legacy primary's at the volume root, an additional clone's under +/// `work/.identity/`), or `None` when no record exists. A record that exists +/// but is invalid or foreign is an error — the caller preserves the clone. +pub(super) fn recorded_identity(record: &Path) -> Result> { + match std::fs::read(record) { + Ok(bytes) => { + let identity: Identity = serde_json::from_slice(&bytes) + .context("invalid private clone identity; preserve state for operator recovery")?; + if identity.protocol != PROTOCOL { + bail!("private clone identity protocol mismatch; preserve state for operator recovery"); + } + Ok(Some((identity.repository, identity.account))) + } + Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(None), + Err(error) => Err(error.into()), + } +} + /// Reclaim only an in-volume linked checkout whose tip is published and /// whose own repository and branch have a finished PR at that exact tip. /// Keep the branch ref and every commit. A missing, open or unreadable PR diff --git a/loom-daemon/src/tokens_pool/private_workspace/seat_reclaim.rs b/loom-daemon/src/tokens_pool/private_workspace/seat_reclaim.rs new file mode 100644 index 0000000000..17748016a2 --- /dev/null +++ b/loom-daemon/src/tokens_pool/private_workspace/seat_reclaim.rs @@ -0,0 +1,1006 @@ +//! Reclaim inside private-workspace seat volumes (#14678, Asks 1 and 3 — the +//! periodic idle pass; the shared object store is #14680, the per-seat size +//! telemetry is #14693). +//! +//! # What this reclaims, and where it runs +//! +//! A seat volume (`/workspace` inside `loom-codex-workspace-`) grows as +//! seats × registered repos × (clone + worktrees + scratch) and the host-side +//! reaper never sees inside it — its `reaper.pass` spans report +//! `preserved.private_workspace` for everything there. The disk-pressure +//! incident behind this issue (two workers at 100%, 224 GB across 7 seats) +//! was mostly per-step scratch and finished-work checkouts no host pass could +//! reach. This module adds the missing in-volume pass: +//! +//! - **Scratch**: `cache/-.*` entries and top-level step dirs +//! (`-.*`, `*-tmp`) older than the existing +//! `autonomous.worktreeReaper.scratchReclaim.maxAgeHours`. +//! - **Worktrees**: linked checkouts under `repo` and `work/` that are +//! clean, fully published, and whose branch has a confirmed **merged** PR +//! at that exact tip past the existing reaper grace +//! (`autonomous.worktreeReaper.gracePeriodSecs`). +//! +//! The pass runs in two halves. The container half ([`seat_reclaim`]) is a +//! `private-workspace seat-reclaim` verb driven like `prepare`/`recover`; it +//! never mutates `.identity` records, never quarantines and never takes a +//! backup — a worktree it cannot prove finished is preserved with a reason. +//! The host half ([`run_for`], called from the eager-reclaim tier 1 the way +//! [`crate::scratch_reclaim`] is) enumerates the seats, takes each account's +//! exclusive lease, requires the container running and idle — the same +//! admission [`super::lifecycle::run_job`] applies — and execs the verb. It +//! never calls `docker::prepare_target`: preparation runs `clean_worktrees`, +//! which refuses a dirty worktree, and a dirty seat must still get its +//! scratch reclaimed and its size reported. +//! +//! # Ownership vs per-job preparation (#14483) +//! +//! Job preparation (merged #14483) owns the **target clone at admission +//! time**: it forgives untracked caches, force-reaps linked worktrees whose +//! tip has a *finished* PR — merged **or closed-unmerged**, dirty ones +//! included — and offers the explicit `accounts session recover` listing. +//! This pass owns **all seats and all clones, idle**: it is merged-only, +//! clean-only, and publishes what it kept through `reaper.pass`. The dirty +//! sibling in a seat with a published tip at a merged PR is preparation's to +//! remove and this pass's to preserve; see +//! `defaults/docs/private-session-dispatch.md`. +//! +//! # Safety +//! +//! Liveness is structural, not probed: the verb only ever runs while the +//! seat's account lease is held and the container is proven idle, so no step +//! session can be live in the volume; the scratch age floor then bounds the +//! blast radius to entries no recent step wrote. Future-dated mtimes are +//! never reclaimable ([`crate::scratch_reclaim::is_reclaimable_age`]). + +use std::path::{Path, PathBuf}; + +use chrono::{DateTime, Utc}; +use serde::{Deserialize, Serialize}; + +use super::repository; +use crate::worktree_ops::clean::check_grace_period; +use crate::worktree_ops::clean_owner::PrAtHead; + +// ============================================================================ +// The verb (container side) +// ============================================================================ + +#[derive(clap::Args)] +pub struct SeatReclaimArgs { + #[arg(long)] + pub account: String, + /// Scratch age floor in seconds; the host resolves it from + /// `autonomous.worktreeReaper.scratchReclaim.maxAgeHours`. + #[arg(long)] + pub max_age_secs: i64, + /// Post-merge grace in seconds; the host resolves it from + /// `autonomous.worktreeReaper.gracePeriodSecs`. + #[arg(long)] + pub grace_secs: i64, +} + +/// One pass's outcome, reported to the host as the protocol `outcome`. +#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize)] +pub struct SeatReclaimOutcome { + /// Linked worktrees evaluated (the main checkout is never a candidate). + pub worktrees_scanned: usize, + pub worktrees_removed: usize, + pub worktrees_failed: usize, + /// The preserve wording of every worktree left in place. + pub preserved: Vec, + /// Bytes freed by worktree removals plus scratch removals. + pub bytes: u64, + /// Step-scratch entries removed (cache and top-level). + pub scratch_removed: usize, + /// Step-scratch entries evaluated (shape-matched candidates). + pub scratch_scanned: usize, + /// Bytes freed by scratch removals alone. + pub scratch_bytes: u64, +} + +/// The PR-state lookup the worktree half consults, injected so tests never +/// contact a forge: `(clone, repository, branch, tip) -> state`. +pub type PrLookup<'a> = dyn Fn(&Path, &str, &str, &str) -> PrAtHead + 'a; + +/// One in-volume pass. `root` is the seat volume root (`/workspace`). +pub(super) fn seat_reclaim( + root: &Path, + args: &SeatReclaimArgs, + pr_at_head: &PrLookup<'_>, + now: DateTime, +) -> anyhow::Result { + if args.account.is_empty() { + anyhow::bail!("seat reclaim requires the account identity"); + } + let root = root.canonicalize()?; + let mut outcome = SeatReclaimOutcome::default(); + reclaim_scratch(&root, args.max_age_secs, now, &mut outcome); + reap_merged_worktrees(&root, args, pr_at_head, now, &mut outcome)?; + Ok(outcome) +} + +// ============================================================================ +// Scratch selection (pure gates + the sweep) +// ============================================================================ + +/// Whether `name` matches the per-step scratch shape `-` with an +/// optional `.`/`-` remainder: `sta-6421.tar.gz`, `judge-9661`, +/// `builder-6940.x`, `squad-6940-dir`, `loom-7456-check.log`. Reserved roots +/// (`repo`, `work`, `cache`, `identity.json`, dot-entries) match no prefix of +/// this shape. Pure. +#[must_use] +pub fn is_step_scratch_name(name: &str) -> bool { + let Some((role, rest)) = name.split_once('-') else { + return false; + }; + if role.is_empty() || !role.chars().all(|c| c.is_ascii_alphanumeric() || c == '_') { + return false; + } + let digits = rest.chars().take_while(|c| c.is_ascii_digit()).count(); + if digits == 0 { + return false; + } + let remainder = &rest[digits..]; + remainder.is_empty() || remainder.starts_with('.') || remainder.starts_with('-') +} + +/// Whether `name` is a top-level step dir ending in `-tmp`: +/// `doctor-6422-tmp`, `curator-repro-tmp`. Pure. +#[must_use] +pub fn is_tmp_step_name(name: &str) -> bool { + name.len() > 4 && name.ends_with("-tmp") +} + +fn reclaim_scratch( + root: &Path, + max_age_secs: i64, + now: DateTime, + outcome: &mut SeatReclaimOutcome, +) { + let max_age_secs = max_age_secs.max(0); + let mut candidates: Vec = Vec::new(); + if let Ok(entries) = std::fs::read_dir(root.join("cache")) { + for entry in entries.flatten() { + let name = entry.file_name().to_string_lossy().into_owned(); + if is_step_scratch_name(&name) { + candidates.push(entry.path()); + } + } + } + if let Ok(entries) = std::fs::read_dir(root) { + for entry in entries.flatten() { + let name = entry.file_name().to_string_lossy().into_owned(); + if is_step_scratch_name(&name) || is_tmp_step_name(&name) { + candidates.push(entry.path()); + } + } + } + for path in candidates { + outcome.scratch_scanned += 1; + let Ok(metadata) = std::fs::metadata(&path) else { + continue; + }; + let Ok(modified) = metadata.modified() else { + continue; + }; + let modified: DateTime = modified.into(); + let age_secs = (now - modified).num_seconds(); + if !crate::scratch_reclaim::is_reclaimable_age(age_secs, max_age_secs) { + continue; + } + let bytes = if path.is_dir() { + tree_bytes(&path) + } else { + metadata.len() + }; + let removed = if path.is_dir() { + std::fs::remove_dir_all(&path) + } else { + std::fs::remove_file(&path) + }; + if removed.is_ok() { + outcome.scratch_removed += 1; + outcome.scratch_bytes += bytes; + outcome.bytes += bytes; + } else { + log::debug!("seat_reclaim: could not remove {}: {:?}", path.display(), removed); + } + } +} + +fn tree_bytes(path: &Path) -> u64 { + let mut total = 0u64; + if let Ok(entries) = std::fs::read_dir(path) { + for entry in entries.flatten() { + if let Ok(file_type) = entry.file_type() { + if file_type.is_dir() { + total += tree_bytes(&entry.path()); + } else if let Ok(meta) = entry.metadata() { + total += meta.len(); + } + } + } + } + total +} + +// ============================================================================ +// Worktree reap (merged-only, clean-only, in-volume) +// ============================================================================ + +/// Every clone of the volume with its recorded repository: the legacy primary +/// (`repo`, identity at the volume root) and each `work/` carrying a +/// `work/.identity/.json` record. A staging dir (`.cloning-*`) and an +/// unrecorded directory are not ours to touch and are not enumerated. +fn volume_clones(root: &Path) -> Vec<(PathBuf, String)> { + let mut clones = Vec::new(); + let primary_record = root.join("identity.json"); + let primary = root.join("repo"); + if primary.is_dir() { + if let Ok(Some((repository, _))) = repository::recorded_identity(&primary_record) { + clones.push((primary, repository)); + } + } + let work = root.join("work"); + let identities = work.join(".identity"); + if let Ok(entries) = std::fs::read_dir(&work) { + for entry in entries.flatten() { + let name = entry.file_name(); + let Some(name) = name.to_str() else { + continue; + }; + if !entry.path().is_dir() || !super::safe_segment(name) || name.contains("..") { + continue; + } + let record = identities.join(format!("{name}.json")); + if let Ok(Some((repository, _))) = repository::recorded_identity(&record) { + clones.push((entry.path(), repository)); + } + } + } + clones +} + +fn reap_merged_worktrees( + root: &Path, + args: &SeatReclaimArgs, + pr_at_head: &PrLookup<'_>, + now: DateTime, + outcome: &mut SeatReclaimOutcome, +) -> anyhow::Result<()> { + for (repo, repository) in volume_clones(root) { + for entry in repository::worktree_entries(&repo)? { + let path = &entry.path; + outcome.worktrees_scanned += 1; + let Some(canonical) = path.canonicalize().ok() else { + outcome.preserved.push("worktree state unreadable".into()); + continue; + }; + if canonical == repo { + // The main checkout is what the seat IS; never a candidate. + outcome.worktrees_scanned -= 1; + continue; + } + if entry.locked { + outcome.preserved.push("worktree locked (in use)".into()); + continue; + } + if !canonical.starts_with(root) { + outcome + .preserved + .push("worktree outside the private volume".into()); + continue; + } + let branch = repository::git(path, &["rev-parse", "--abbrev-ref", "HEAD"])? + .trim() + .to_owned(); + if branch == "HEAD" { + outcome + .preserved + .push("detached HEAD (unnamed work)".into()); + continue; + } + if repository::ahead_of_origin(path, "HEAD")? > 0 { + outcome.preserved.push("unpushed commits on HEAD".into()); + continue; + } + if !repository::worktree_status(path)?.is_empty() { + outcome + .preserved + .push("uncommitted changes in worktree".into()); + continue; + } + let tip = repository::git(path, &["rev-parse", "HEAD"])? + .trim() + .to_owned(); + match pr_at_head(&repo, &repository, &branch, &tip) { + PrAtHead::Open => outcome.preserved.push("PR still open".into()), + PrAtHead::ClosedNoMerge => outcome + .preserved + .push("PR closed without merging (not merged)".into()), + PrAtHead::NoPr => outcome + .preserved + .push("no merged PR for the branch at this tip".into()), + PrAtHead::Unknown => outcome.preserved.push("PR status unknown".into()), + PrAtHead::Merged(merged_at) => { + let Ok(merged) = DateTime::parse_from_rfc3339(&merged_at) else { + outcome + .preserved + .push("PR status unknown (unparseable merge time)".into()); + continue; + }; + let (past, _) = + check_grace_period(merged.with_timezone(&Utc), args.grace_secs, now); + if !past { + outcome + .preserved + .push("merged PR grace period not passed".into()); + continue; + } + let held = tree_bytes(path); + if repository::git(&repo, &["worktree", "remove", &path.display().to_string()]) + .is_ok() + { + outcome.worktrees_removed += 1; + outcome.bytes += held; + } else { + outcome.worktrees_failed += 1; + } + } + } + } + } + Ok(()) +} + +/// The production lookup the verb binds: resolves the GitHub host/slug of +/// the clone's own repository URL, then asks +/// [`crate::worktree_ops::clean_owner::pr_at_head`] with the clone as the +/// probe's working directory. A repository on any other forge answers +/// [`PrAtHead::Unknown`] — preserved, never removed on a guess. +pub(super) fn github_pr_at_head( + repo: &Path, + repository: &str, + branch: &str, + tip: &str, +) -> PrAtHead { + let host = std::env::var("GH_HOST").unwrap_or_else(|_| "github.com".into()); + let Some(slug) = super::forge_auth::github_slug(repository, &host) else { + return PrAtHead::Unknown; + }; + crate::worktree_ops::clean_owner::pr_at_head(repo, &host, &slug, branch, tip) +} + +// ============================================================================ +// The host-side pass (leased, idle-only, no preparation) +// ============================================================================ + +/// One host-wide pass's outcome over every private-workspace seat. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct SeatReclaimReport { + /// `Some(reason)` when no pass ran at all (disabled, or the cooldown had + /// not elapsed). `None` on a pass that ran. + pub skipped: Option, + /// Per-seat results: the account name and either the verb's outcome or + /// the reason this seat was skipped/refused (busy, stopped, not idle, a + /// containment-validation refusal, an endpoint error). + pub seats: Vec<(String, Result)>, +} + +impl SeatReclaimReport { + /// Worktrees removed across every seat that ran. + #[must_use] + pub fn worktrees_removed(&self) -> usize { + self.seats + .iter() + .filter_map(|(_, result)| result.as_ref().ok()) + .map(|outcome| outcome.worktrees_removed) + .sum() + } + + /// Scratch entries removed across every seat that ran. + #[must_use] + pub fn scratch_removed(&self) -> usize { + self.seats + .iter() + .filter_map(|(_, result)| result.as_ref().ok()) + .map(|outcome| outcome.scratch_removed) + .sum() + } + + /// Bytes freed across every seat that ran. + #[must_use] + pub fn bytes(&self) -> u64 { + self.seats + .iter() + .filter_map(|(_, result)| result.as_ref().ok()) + .map(|outcome| outcome.bytes) + .sum() + } + + /// One human-readable summary for the eager-reclaim log line. + #[must_use] + pub fn summary(&self) -> String { + if let Some(reason) = &self.skipped { + return format!("skipped: {reason}"); + } + format!( + "{} seat(s): {} worktree(s), {} scratch entries ({})", + self.seats.len(), + self.worktrees_removed(), + self.scratch_removed(), + crate::fs_util::human_size(self.bytes()) + ) + } +} + +/// Host-wide cooldown state: one marker key, not per-repo — the pass walks +/// every seat from whatever root the eager trigger names, so a per-root key +/// would re-run it once per registered root per tick. +static LAST_RUN_AT: std::sync::OnceLock>>> = + std::sync::OnceLock::new(); + +fn last_run_slot() -> &'static std::sync::Mutex>> { + LAST_RUN_AT.get_or_init(|| std::sync::Mutex::new(None)) +} + +/// Drop cooldown state. Unused today (the host driver path needs a full +/// account registry fixture); kept as the standard seam for the first test +/// that gets one. +#[cfg(test)] +#[allow(dead_code)] +#[doc(hidden)] +pub fn reset_state_for_test() { + *last_run_slot() + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) = None; +} + +/// Whether the host-wide cooldown suppresses a pass at `now`. Pure: the +/// decision the report's `skipped` reason is built from. +#[must_use] +pub fn cooldown_skip( + last: Option>, + now: DateTime, + min_interval_secs: u64, +) -> Option { + let last = last?; + let since = (now - last).num_seconds(); + (since >= 0 && since < i64::try_from(min_interval_secs).unwrap_or(i64::MAX)) + .then(|| format!("a pass ran {since}s ago (cooldown {min_interval_secs}s)")) +} + +/// One production pass over every private-workspace seat (#14678), honoring +/// the existing `autonomous.worktreeReaper.scratchReclaim.*` surface for its +/// enable gate, scratch age floor and cooldown — no new flags. The worktree +/// half's grace comes from the existing +/// `autonomous.worktreeReaper.gracePeriodSecs`. Scheduled from the eager +/// reclaim tier 1 like [`crate::scratch_reclaim::run_for`]. +pub fn run_for(workspace: &Path) -> SeatReclaimReport { + use crate::observability::ops::reaper; + + let config = crate::scratch_reclaim::read_scratch_reclaim_config(workspace); + let now = Utc::now(); + if !crate::scratch_reclaim::resolve_enabled(&config) { + let report = SeatReclaimReport { + skipped: Some("disabled (scratchReclaim.enabled=false)".into()), + seats: Vec::new(), + }; + log::debug!("private_workspace seat_reclaim: {}", report.summary()); + return report; + } + let min_interval_secs = crate::scratch_reclaim::resolve_min_interval_secs(&config); + let last = *last_run_slot() + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + if let Some(reason) = cooldown_skip(last, now, min_interval_secs) { + let report = SeatReclaimReport { + skipped: Some(reason), + seats: Vec::new(), + }; + log::debug!("private_workspace seat_reclaim: {}", report.summary()); + return report; + } + *last_run_slot() + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) = Some(now); + + let max_age_secs = + i64::try_from(crate::scratch_reclaim::resolve_max_age_hours(&config).saturating_mul(3600)) + .unwrap_or(i64::MAX); + let grace_secs = crate::worktree_reaper::resolve_grace_period( + &crate::worktree_reaper::read_worktree_reaper_config(workspace), + ); + let seats = super::lifecycle::seat_reclaim(workspace, max_age_secs, grace_secs); + + for (account, result) in &seats { + let Ok(outcome) = result else { + log::debug!("private_workspace seat_reclaim: {account} skipped: {result:?}"); + continue; + }; + // The span's repo label is the seat's account name: one path-shaped + // carrier, never a host path (`reaper::pass_span` takes the final + // component only). + let seat = Path::new("/private-workspace-seats").join(account); + let worktree_bytes = outcome.bytes.saturating_sub(outcome.scratch_bytes); + let preserved: Vec<&str> = outcome.preserved.iter().map(String::as_str).collect(); + reaper::publish( + &seat, + &reaper::Pass { + class: "seat", + phase: reaper::PHASE_REAP, + scanned: outcome.worktrees_scanned, + removed: outcome.worktrees_removed, + failed: outcome.worktrees_failed, + preserved: preserved.clone(), + bytes: worktree_bytes, + backed_up: 0, + backup_refs: Vec::new(), + }, + ); + reaper::publish( + &seat, + &reaper::Pass { + class: "seat", + phase: reaper::PHASE_RECLAIM, + scanned: outcome.scratch_scanned, + removed: outcome.scratch_removed, + failed: 0, + preserved: Vec::new(), + bytes: outcome.scratch_bytes, + backed_up: 0, + backup_refs: Vec::new(), + }, + ); + if outcome.worktrees_removed > 0 || outcome.scratch_removed > 0 { + log::info!( + "private_workspace seat_reclaim: seat {account} removed {} worktree(s) and {} \ + scratch entries ({}) — preserved {preserved:?}", + outcome.worktrees_removed, + outcome.scratch_removed, + crate::fs_util::human_size(outcome.bytes), + ); + } else { + log::debug!( + "private_workspace seat_reclaim: seat {account} nothing to reclaim — preserved \ + {preserved:?}" + ); + } + } + + let report = SeatReclaimReport { + skipped: None, + seats, + }; + if report.worktrees_removed() > 0 || report.scratch_removed() > 0 { + log::info!( + "private_workspace seat_reclaim: {} reclaimed across {} seat(s)", + report.summary(), + report.seats.len() + ); + } + report +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::test_git_helpers::{configure, run as raw_git}; + use crate::worktree_ops::clean_owner::pr_at_head_in_rows; + use chrono::Duration; + + /// The volume fixture: a remote, a prepared primary clone (`repo` with + /// its identity record), and nothing else — the same shape + /// [`super::super::tests::RepoFixture`] builds for the prepare tests. + struct Volume { + _temp: tempfile::TempDir, + remote: PathBuf, + root: PathBuf, + } + + impl Volume { + fn new() -> Self { + let temp = tempfile::tempdir().unwrap(); + let remote = temp.path().join("remote"); + let root = temp.path().join("volume"); + std::fs::create_dir(&remote).unwrap(); + std::fs::create_dir(&root).unwrap(); + raw_git(&remote, &["init", "-b", "main"]); + configure(&remote); + std::fs::write(remote.join("fixture"), "base").unwrap(); + raw_git(&remote, &["add", "."]); + raw_git(&remote, &["commit", "-m", "fixture"]); + let args = repository::PrepareArgs { + account: "fixture".into(), + repository: remote.to_string_lossy().into_owned(), + base: "main".into(), + container_id: "test-container".into(), + branch: None, + slug: None, + }; + repository::prepare(&root, &args).unwrap(); + Self { + _temp: temp, + remote, + root, + } + } + + fn repo(&self) -> PathBuf { + self.root.join("repo") + } + + /// A linked worktree under the clone's `.loom/worktrees`, on a new + /// branch with one commit; `push` publishes the branch and fetches it + /// back, so the clone's remote-tracking refs name the tip (the + /// unpublished-branch gate reads those, not the remote). + fn worktree(&self, name: &str, push: bool) -> PathBuf { + let branch = format!("feature/{name}"); + let path = self.repo().join(".loom/worktrees").join(name); + raw_git( + &self.repo(), + &[ + "worktree", + "add", + "-b", + &branch, + path.to_string_lossy().as_ref(), + ], + ); + std::fs::write(path.join(name), "work").unwrap(); + raw_git(&path, &["add", "."]); + raw_git(&path, &["commit", "-m", name]); + if push { + raw_git(&path, &["push", self.remote.to_string_lossy().as_ref(), &branch]); + // Update the clone's remote-tracking refs: `rev-list --not + // --remotes=origin` reads those, not the remote itself. + raw_git(&path, &["fetch", "origin"]); + } + path + } + } + + fn args(max_age_secs: i64, grace_secs: i64) -> SeatReclaimArgs { + SeatReclaimArgs { + account: "fixture".into(), + max_age_secs, + grace_secs, + } + } + + fn merged_days_ago(days: i64) -> String { + (Utc::now() - Duration::days(days)).to_rfc3339() + } + + // =================================================================== + // Pure gates + // =================================================================== + + #[test] + fn step_scratch_shapes_match_only_step_names() { + for name in [ + "sta-6421", + "sta-6421.tar.gz", + "judge-9661.json", + "builder-6940.x", + "squad-6940-dir", + "loom-7456-check.log", + "role_1-2", + ] { + assert!(is_step_scratch_name(name), "{name}"); + } + for reserved in [ + "repo", + "work", + "cache", + "identity.json", + ".identity", + ".cloning-x", + "main", + "not-step", + "steps-late", + "builder-x", + "-6422", + ] { + assert!(!is_step_scratch_name(reserved), "{reserved}"); + } + assert!(is_tmp_step_name("doctor-6422-tmp")); + assert!(is_tmp_step_name("curator-repro-tmp")); + assert!(!is_tmp_step_name("tmp")); + assert!(!is_tmp_step_name("work")); + } + + // =================================================================== + // The periodic-pass fixture: eligible beside dirty / unpublished / + // open-PR / closed-unmerged siblings. Only the eligible checkout + // disappears (#14678). + // =================================================================== + + #[test] + fn only_the_eligible_checkout_is_removed() { + let _fork = super::super::tests::fork_guard(); + let volume = Volume::new(); + // Everything a removal needs — so for the dirty sibling, published + // tip at a merged PR past grace, cleanliness alone preserves it. + let eligible = volume.worktree("issue-10", true); + let dirty = volume.worktree("issue-11", true); + std::fs::write(dirty.join("untracked"), "dirty").unwrap(); + let unpublished = volume.worktree("issue-12", false); + let open = volume.worktree("issue-13", true); + let closed_unmerged = volume.worktree("issue-14", true); + let within_grace = volume.worktree("issue-15", true); + + let lookup = |_: &Path, _: &str, branch: &str, _: &str| -> PrAtHead { + match branch { + "feature/issue-10" | "feature/issue-11" => PrAtHead::Merged(merged_days_ago(30)), + "feature/issue-13" => PrAtHead::Open, + "feature/issue-14" => PrAtHead::ClosedNoMerge, + "feature/issue-15" => PrAtHead::Merged(merged_days_ago(0)), + _ => PrAtHead::NoPr, + } + }; + let outcome = seat_reclaim(&volume.root, &args(0, 3600), &lookup, Utc::now()).unwrap(); + + assert!(!eligible.exists(), "the eligible checkout must disappear"); + assert!(dirty.exists(), "dirty is preserved by cleanliness alone"); + assert!(unpublished.exists(), "unpublished is preserved"); + assert!(open.exists(), "an open PR is preserved"); + assert!(closed_unmerged.exists(), "closed-unmerged is preserved"); + assert!(within_grace.exists(), "merged inside the grace window is preserved"); + + assert_eq!(outcome.worktrees_removed, 1); + assert_eq!(outcome.worktrees_failed, 0); + assert_eq!(outcome.worktrees_scanned, 6); + let reasons = outcome.preserved.join("\n"); + assert!(reasons.contains("uncommitted changes in worktree"), "{reasons}"); + assert!(reasons.contains("unpushed commits on HEAD"), "{reasons}"); + assert!(reasons.contains("PR still open"), "{reasons}"); + assert!(reasons.contains("PR closed without merging"), "{reasons}"); + assert!(reasons.contains("merged PR grace period not passed"), "{reasons}"); + // The branch ref and its commits survive the checkout's removal. + assert!( + raw_git(&volume.repo(), &["rev-parse", "--verify", "refs/heads/feature/issue-10"]) + .len() + == 40 + ); + } + + #[test] + fn main_checkout_and_foreign_worktrees_are_never_candidates() { + let _fork = super::super::tests::fork_guard(); + let volume = Volume::new(); + // A worktree the clone knows about but that lives outside the volume + // (a host path): preserved, never followed. + let outside = volume._temp.path().join("outside-wt"); + raw_git( + &volume.repo(), + &[ + "worktree", + "add", + "-b", + "feature/issue-20", + outside.to_string_lossy().as_ref(), + ], + ); + raw_git( + &outside, + &[ + "push", + volume.remote.to_string_lossy().as_ref(), + "feature/issue-20", + ], + ); + raw_git(&outside, &["fetch", "origin"]); + let lookup = |_: &Path, _: &str, branch: &str, _: &str| -> PrAtHead { + if branch == "feature/issue-20" { + PrAtHead::Merged(merged_days_ago(30)) + } else { + PrAtHead::NoPr + } + }; + let outcome = seat_reclaim(&volume.root, &args(0, 0), &lookup, Utc::now()).unwrap(); + assert_eq!(outcome.worktrees_removed, 0); + assert!(outside.exists()); + assert!(outcome + .preserved + .iter() + .any(|r| r.contains("outside the private volume"))); + } + + // =================================================================== + // Scratch: ended vs live step, future-dated mtime, reserved roots + // =================================================================== + + #[test] + fn scratch_reclaim_removes_ended_steps_and_keeps_live_and_future_dated_ones() { + let _fork = super::super::tests::fork_guard(); + let volume = Volume::new(); + let cache = volume.root.join("cache"); + std::fs::create_dir_all(cache.join("sta-6421.tar")).unwrap(); + std::fs::write(cache.join("sta-6421.tar").join("blob"), "4 GB").unwrap(); + std::fs::write(cache.join("judge-9661.json"), "log").unwrap(); + std::fs::write(cache.join("tool-cache"), "not a step").unwrap(); + std::fs::create_dir_all(volume.root.join("doctor-6422-tmp")).unwrap(); + std::fs::create_dir_all(volume.root.join("builder-6940.x")).unwrap(); + // A live step's scratch and a clock-skewed one: never reclaimable. + std::fs::write(cache.join("live-7777.tar"), "fresh").unwrap(); + std::fs::write(cache.join("skewed-8888.tar"), "future").unwrap(); + + let now = Utc::now(); + // Ended steps: mtime 48h in the past. + for path in [ + cache.join("sta-6421.tar"), + cache.join("judge-9661.json"), + volume.root.join("doctor-6422-tmp"), + volume.root.join("builder-6940.x"), + cache.join("tool-cache"), + ] { + set_mtime(&path, now - Duration::hours(48)); + } + // Clock skew: an mtime in the future. + set_mtime(&cache.join("skewed-8888.tar"), now + Duration::hours(2)); + + let outcome = + seat_reclaim(&volume.root, &args(24 * 3600, 3600), &|_, _, _, _| PrAtHead::NoPr, now) + .unwrap(); + assert_eq!(outcome.scratch_removed, 4, "{:?}", outcome); + assert_eq!(outcome.scratch_scanned, 6); + assert!(!cache.join("sta-6421.tar").exists()); + assert!(!cache.join("judge-9661.json").exists()); + assert!(!volume.root.join("doctor-6422-tmp").exists()); + assert!(!volume.root.join("builder-6940.x").exists()); + assert!(cache.join("tool-cache").exists(), "non-step cache entries stay"); + assert!(cache.join("live-7777.tar").exists(), "a live step's scratch stays"); + assert!(cache.join("skewed-8888.tar").exists(), "a future-dated entry stays"); + assert!(outcome.scratch_bytes > 0); + // The reserved roots are never candidates at any age. + assert!(volume.root.join("repo").exists()); + assert!(volume.root.join("cache").exists()); + assert!(volume.root.join("identity.json").exists()); + } + + /// Stamp an mtime (files and directories) the way the age gates read it. + fn set_mtime(path: &Path, when: DateTime) { + let out = std::process::Command::new("touch") + .arg("-d") + .arg(when.to_rfc3339()) + .arg(path) + .output() + .expect("touch must spawn"); + assert!( + out.status.success(), + "touch {} failed: {}", + path.display(), + String::from_utf8_lossy(&out.stderr) + ); + } + + #[test] + fn a_dirty_seat_still_reclaims_its_scratch() { + let _fork = super::super::tests::fork_guard(); + let volume = Volume::new(); + std::fs::write(volume.repo().join("stray"), "unfinished work").unwrap(); + let cache = volume.root.join("cache"); + std::fs::write(cache.join("sta-6421.tar"), "4 GB").unwrap(); + + let outcome = seat_reclaim( + &volume.root, + &args(0, 3600), + &|_, _, _, _| PrAtHead::Merged(merged_days_ago(30)), + Utc::now(), + ) + .unwrap(); + assert!(!cache.join("sta-6421.tar").exists(), "scratch is reclaimed"); + assert_eq!(outcome.scratch_removed, 1); + assert_eq!(outcome.worktrees_removed, 0); + assert!(volume.repo().join("stray").exists(), "the dirty main checkout is untouched"); + } + + // =================================================================== + // The merged-only row classification the verb's lookup builds on + // (#14678: closed-unmerged is preserved here, finished-at-head is not + // the same question). + // =================================================================== + + fn row( + state: &str, + merged_at: Option<&str>, + closed_at: Option<&str>, + sha: &str, + ) -> crate::worktree_ops::clean_owner::PrRowRest { + serde_json::from_value(serde_json::json!({ + "state": state, + "merged_at": merged_at, + "closed_at": closed_at, + "head": {"ref": "feature/x", "sha": sha, "repo": {"full_name": "Owner/Repo"}}, + "base": {"repo": {"full_name": "Owner/Repo"}}, + })) + .unwrap() + } + + /// The one shape every eligible sibling is built from: closed and merged + /// at `sha`. + fn merged(sha: &str) -> crate::worktree_ops::clean_owner::PrRowRest { + row("closed", Some("2020-01-01T00:00:00Z"), Some("2020-01-01T00:00:00Z"), sha) + } + + #[test] + fn pr_at_head_rows_are_merged_only_and_open_wins() { + assert_eq!( + pr_at_head_in_rows(&[merged("tip")], "Owner/Repo", "feature/x", "tip"), + PrAtHead::Merged("2020-01-01T00:00:00Z".into()) + ); + // A newer open PR on the same branch takes precedence over the merge. + let open = row("open", None, None, "other"); + assert_eq!( + pr_at_head_in_rows(&[merged("tip"), open], "Owner/Repo", "feature/x", "tip"), + PrAtHead::Open + ); + // Closed without merging at the tip: preserved by the seat pass. + let closed = row("closed", None, Some("2020-01-01T00:00:00Z"), "tip"); + assert_eq!( + pr_at_head_in_rows(&[closed], "Owner/Repo", "feature/x", "tip"), + PrAtHead::ClosedNoMerge + ); + // A merge at a different tip leaves no merge at this tip. + assert_eq!( + pr_at_head_in_rows(&[merged("older")], "Owner/Repo", "feature/x", "tip"), + PrAtHead::NoPr + ); + // A truncated or foreign record is never evidence. + assert_eq!( + pr_at_head_in_rows(&[merged("tip")], "Other/Repo", "feature/x", "tip"), + PrAtHead::Unknown + ); + let many: Vec<_> = (0..100).map(|i| merged(&format!("tip{i}"))).collect(); + assert_eq!(pr_at_head_in_rows(&many, "Owner/Repo", "feature/x", "tip"), PrAtHead::Unknown); + } + + // =================================================================== + // The host report folds seat outcomes + // =================================================================== + + #[test] + fn the_host_wide_cooldown_suppresses_only_a_recent_pass() { + let now = Utc::now(); + assert!(cooldown_skip(None, now, 1800).is_none(), "a first pass runs"); + assert!( + cooldown_skip(Some(now - chrono::Duration::seconds(60)), now, 1800).is_some(), + "a pass 60s ago is inside a 30m cooldown" + ); + let held = cooldown_skip(Some(now - chrono::Duration::seconds(1800)), now, 1800); + assert!(held.is_none(), "a pass exactly one cooldown old re-runs: {held:?}"); + // Clock skew backwards never wedges the cooldown shut. + assert!(cooldown_skip(Some(now + chrono::Duration::hours(1)), now, 1800).is_none()); + } + + #[test] + fn report_sums_across_seats() { + let one = SeatReclaimOutcome { + worktrees_scanned: 3, + worktrees_removed: 1, + worktrees_failed: 0, + preserved: vec!["PR still open".into()], + bytes: 100, + scratch_removed: 2, + scratch_scanned: 2, + scratch_bytes: 40, + }; + let report = SeatReclaimReport { + skipped: None, + seats: vec![ + ("agent-1".into(), Ok(one)), + ("agent-2".into(), Err("seat busy".into())), + ], + }; + assert_eq!(report.worktrees_removed(), 1); + assert_eq!(report.scratch_removed(), 2); + assert_eq!(report.bytes(), 100); + assert!(report.summary().contains("worktree")); + let skipped = SeatReclaimReport { + skipped: Some("disabled".into()), + seats: Vec::new(), + }; + assert!(skipped.summary().contains("disabled")); + } +} diff --git a/loom-daemon/src/tokens_pool/private_workspace/tests.rs b/loom-daemon/src/tokens_pool/private_workspace/tests.rs index c990787e88..7c7d4d6e8c 100644 --- a/loom-daemon/src/tokens_pool/private_workspace/tests.rs +++ b/loom-daemon/src/tokens_pool/private_workspace/tests.rs @@ -1222,6 +1222,61 @@ fn repository_bound_job_keeps_the_account_wide_exclusive_lease() { assert_eq!(lease::read(root.path()).unwrap().unwrap().repository, ""); } +/// #14678: the seat-reclaim exec drives the in-volume verb directly — same +/// protocol envelope as `recover`, the account's resolved auth forwarded by +/// name on the child, and **no `prepare` step anywhere in argv**: the pass +/// must reach a dirty seat, which preparation would refuse. +#[test] +fn seat_reclaim_exec_drives_the_verb_without_any_preparation() { + const SECRET: &str = "ghs_seatTESTONLY14678"; + let mut config = config(); + config.repository = "https://github.com/acme/widget".into(); + let argv = std::rc::Rc::new(std::cell::RefCell::new(Vec::::new())); + let seen = argv.clone(); + let _fake = docker::fake::install(Box::new(move |args| { + *seen.borrow_mut() = args.iter().map(|a| (*a).to_owned()).collect(); + Ok(serde_json::json!({ + "protocol": PROTOCOL, + "outcome": { + "worktrees_scanned": 1, + "worktrees_removed": 0, + "worktrees_failed": 0, + "preserved": ["PR still open"], + "bytes": 0, + "scratch_removed": 0, + "scratch_scanned": 0, + "scratch_bytes": 0, + } + }) + .to_string()) + })); + let ws = forge_auth::published_owner_workspace("acme"); + let resolved = + forge_auth::ExecAuth::resolve_with(ws.path(), &config.repository, &|_| None, &|_| { + Some(SECRET.into()) + }); + let outcome = docker::seat_reclaim(&config, 86_400, 3_600, &resolved).unwrap(); + assert_eq!(outcome.worktrees_scanned, 1); + assert_eq!(outcome.preserved, vec!["PR still open".to_string()]); + + let args = argv.borrow().clone(); + let joined = args.join(" "); + assert!(joined.contains("private-workspace seat-reclaim"), "{args:?}"); + assert!(joined.contains("--account account"), "{args:?}"); + assert!(joined.contains("--max-age-secs 86400"), "{args:?}"); + assert!(joined.contains("--grace-secs 3600"), "{args:?}"); + assert!( + !joined.contains("prepare"), + "the seat pass must not prepare the target ({args:?})" + ); + assert!( + args.windows(2) + .any(|w| w[0] == "--env" && w[1] == "GH_TOKEN"), + "{args:?}" + ); + assert!(!args.iter().any(|a| a.contains(SECRET)), "token leaked into argv"); +} + /// #13916: on an App-only host the prepare exec names `GH_TOKEN` as a bare /// `--env` and the resolved value rides on the `docker` child alone — never /// in argv. A token already in the process env adds nothing to the child. diff --git a/loom-daemon/src/work_finder/tick_line.rs b/loom-daemon/src/work_finder/tick_line.rs index a527c0c7e4..d0a5fe1df3 100644 --- a/loom-daemon/src/work_finder/tick_line.rs +++ b/loom-daemon/src/work_finder/tick_line.rs @@ -237,6 +237,7 @@ mod tests { deep_clean: None, docker: None, scratch: None, + seats: None, target_orphans: None, at: Utc::now(), git_tmp: None, diff --git a/loom-daemon/src/worktree_ops/clean_owner.rs b/loom-daemon/src/worktree_ops/clean_owner.rs index a50ee89061..fa2a303bdf 100644 --- a/loom-daemon/src/worktree_ops/clean_owner.rs +++ b/loom-daemon/src/worktree_ops/clean_owner.rs @@ -145,6 +145,104 @@ pub(crate) fn fetch_pr_rows( serde_json::from_slice::>(&out.stdout).map_err(|_| RowsError::Failed) } +/// One private-workspace seat pass's reading of a branch's PR state at an +/// exact tip (#14678): merged-only, and distinct from +/// [`finished_pr_at_head`], which also accepts a closed-unmerged PR because +/// preparation-time force-removal (#14483) treats both as finished. The +/// periodic seat pass preserves a closed-unmerged worktree, so it needs the +/// distinction, plus the merge timestamp the grace gate measures from. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) enum PrAtHead { + /// A PR whose head is exactly this tip merged; carries `merged_at`. + Merged(String), + /// An open PR on this branch (any tip): an open PR always wins over a + /// historical merge, mirroring [`finished_pr_in_rows`'s] precedence. + Open, + /// A PR on this branch closed without merging. + ClosedNoMerge, + /// No PR row matched this branch at all. + NoPr, + /// The record could not be trusted (truncated, foreign, unparseable). + Unknown, +} + +/// Fresh GitHub PR evidence for the periodic private-workspace seat pass +/// (#14678): the PR state of `branch` at `tip`, merged-only semantics. Uses +/// the same request and row-validation strictness as +/// [`finished_pr_at_head`]; only the classification of the matched rows +/// differs. +pub(crate) fn pr_at_head( + repo_root: &Path, + host: &str, + slug: &str, + branch: &str, + tip: &str, +) -> PrAtHead { + finished_pr_request(repo_root, host, slug, branch) + .and_then(super::gh::bounded_via) + .filter(|out| out.status.success()) + .and_then(|out| serde_json::from_slice::>(&out.stdout).ok()) + .map_or(PrAtHead::Unknown, |rows| pr_at_head_in_rows(&rows, slug, branch, tip)) +} + +/// [`pr_at_head`]'s row classification, pure for tests. The same strict shape +/// [`finished_pr_in_rows`] enforces: a truncated listing, a foreign base or a +/// head-branch mismatch makes the whole record unusable ([`PrAtHead::Unknown`]). +pub(crate) fn pr_at_head_in_rows( + rows: &[PrRowRest], + slug: &str, + branch: &str, + tip: &str, +) -> PrAtHead { + if rows.len() >= 100 { + return PrAtHead::Unknown; + } + let mut at_tip = PrAtHead::NoPr; + for row in rows { + if !row + .base_full_name() + .is_some_and(|base| base.eq_ignore_ascii_case(slug)) + { + return PrAtHead::Unknown; + } + let Some(head) = row.head.as_ref() else { + return PrAtHead::Unknown; + }; + let Some(head_repository) = head + .repo + .as_ref() + .and_then(|repo| repo.full_name.as_deref()) + else { + return PrAtHead::Unknown; + }; + if !head_repository.eq_ignore_ascii_case(slug) { + continue; + } + if head.branch.as_deref() != Some(branch) { + return PrAtHead::Unknown; + } + let status = + clean::classify_pr_row(&row.state, row.merged_at.as_deref(), row.closed_at.as_deref()); + match status { + PrStatus::Open => return PrAtHead::Open, + PrStatus::Merged { .. } if head.sha.as_deref() == Some(tip) => { + let PrStatus::Merged { merged_at } = status else { + unreachable!("classify_pr_row returned Merged with merged_at"); + }; + at_tip = PrAtHead::Merged(merged_at); + } + PrStatus::Merged { .. } => {} + PrStatus::ClosedNoMerge { .. } => { + if at_tip == PrAtHead::NoPr { + at_tip = PrAtHead::ClosedNoMerge; + } + } + PrStatus::NoPr | PrStatus::Unknown => return PrAtHead::Unknown, + } + } + at_tip +} + /// Fresh GitHub PR evidence for an independent clone. Use the clone's /// verified origin rather than resolving a base from its Git config. One /// complete page suffices; if it could be truncated, preserve the worktree. diff --git a/loom-daemon/src/worktree_reaper.rs b/loom-daemon/src/worktree_reaper.rs index f7dc77db0d..fd0d977bcb 100644 --- a/loom-daemon/src/worktree_reaper.rs +++ b/loom-daemon/src/worktree_reaper.rs @@ -1584,6 +1584,24 @@ mod tests { assert!(removed.is_empty()); } + /// #14678: the host reaper keeps skipping a host `issue-N` checkout a + /// private-workspace job owns (`private-jobs` record present), even while + /// the in-volume seat pass owns the seat's own clones — the unrelated + /// host checkout is never touched by either side. + #[test] + #[serial] + fn test_private_workspace_owned_host_worktree_is_never_reaped() { + let repo = make_repo(&[(310, true), (311, true)]); + fs::create_dir_all(repo.path().join(".loom/private-jobs")).unwrap(); + fs::write(repo.path().join(".loom/private-jobs/issue-310.json"), "{}").unwrap(); + let (report, removed) = run_pass(repo.path(), &ProbeSpec::default(), &default_opts()); + assert_eq!(removed, vec![311], "only the unrelated checkout is removed"); + assert!(report.skipped.iter().any(|(num, why)| *num == 310 + && why.contains("private workspace ownership") + && why.contains("unrelated"))); + assert!(repo.path().join(".loom/worktrees/issue-310").exists()); + } + // =================================================================== // Safety gates — the reaper must never out-delete `clean --safe` // ===================================================================