From db0af78ed1376da036fc085fb155718c79458360 Mon Sep 17 00:00:00 2001 From: Joe Beda Date: Tue, 15 Sep 2026 10:57:20 -0700 Subject: [PATCH] docs: draft managed delegation fork lifecycle plan Co-Authored-By: mecatl --- docs/acceptance/README.md | 4 + .../managed-delegation-fork-lifecycle.md | 99 ++++++++++++++ .../0283-managed-delegation-fork-lifecycle.md | 122 +++++++++--------- docs/architecture/parallelism.md | 17 +-- 4 files changed, 172 insertions(+), 70 deletions(-) create mode 100644 docs/acceptance/managed-delegation-fork-lifecycle.md diff --git a/docs/acceptance/README.md b/docs/acceptance/README.md index 20e7abec36..238c8bd605 100644 --- a/docs/acceptance/README.md +++ b/docs/acceptance/README.md @@ -146,6 +146,10 @@ PR after verification. There is no cleanup or status-only PR. - [Managed temporary command leases](managed-temporary-command-leases.md) — private, attributable Linux and macOS command/job temporary storage with a permission-visible system escape and deterministic crash-residue reaping. Status: landed. +- [Managed delegation-fork lifecycle](managed-delegation-fork-lifecycle.md) — extends + managed temporary storage to local Subagent, Team, and Parallel environments while + retaining opaque artifacts, validated crash cleanup, and workspace-key-scoped winner + retention. Status: draft; material lifecycle decisions pending. - [Session title generation and token usage](session-title-generation.md) — mecatui `/title`, an opt-in routed model title after up to three genuine prompts, and durable title-model token attribution. Status: draft. - [Per-upstream MCP broker OAuth grants](mcp-broker-multi-upstream-oauth.md) — accept multiple broker OAuth upstreams while keeping grants, callback state, authenticated discovery, and workspace-enrollment progression backend-scoped. Status: draft. - [Broker MCP status](broker-mcp-status.md) — approved owner-scoped broker connector diff --git a/docs/acceptance/managed-delegation-fork-lifecycle.md b/docs/acceptance/managed-delegation-fork-lifecycle.md new file mode 100644 index 0000000000..b3c0f19973 --- /dev/null +++ b/docs/acceptance/managed-delegation-fork-lifecycle.md @@ -0,0 +1,99 @@ +# Managed delegation-fork lifecycle — acceptance plan + +**Contract:** human-reviewed/v2 +**Work classification:** Architectural — changes the exported environment-fork lifecycle seam, durable managed-allocation protocol, operator retention policy, and cross-process cleanup authority. +**Decision record:** [ADR 0283](../adr/0283-managed-delegation-fork-lifecycle.md) +**Phase:** managed temporary-storage fork family +**Status:** draft, 2026-09-15. ADR 0283 corrected for the existing path-free artifact contract; material lifecycle choices await human review. +**Delivery:** Split. The engine API, durable cleanup protocol, configuration, and retention authority require a Plan / Interface review before implementation. +**Expected tasks:** deferred to orchestration after the human decisions below resolve. + +This plan makes locally created delegation environments attributable and recoverable after an abnormal process exit without weakening child isolation or exposing filesystem paths. It extends ADR 0281's private managed namespace with a fork allocation family while preserving immediate cleanup for ordinary children and bounded in-process retention for selected Parallel winners. + +## Human decisions + +- [ ] Approve the replacement of the process-wide eight-entry preserved-fork LRU with independently capped, workspace-key-scoped LRUs (default four); accept that aggregate retained disk is unbounded across workspace keys. — Decision: pending +- [ ] Approve an exported opaque `engine/tool` fork-lifecycle capability returned by `EnvironmentForker`, with cancellation-aware idempotent cleanup and preserved-winner promotion; approve removal/replacement of the current `agent.PreservedForkStore` ownership path after API-compatibility review. — Decision: pending +- [ ] Approve promotion failure behavior: Parallel cleans the selected winner and returns a tool error without publishing an artifact handle; registry shutdown rejects a late promotion and leaves cleanup with the caller. — Decision: pending +- [ ] Approve ADR 0281's workspace key as the sole partition identity for this feature (canonical local path-derived key today, no same-filesystem-move retention promise); correct ADR 0282's incompatible move claim separately before scratch-cache implementation. — Decision: pending +- [ ] Approve durable owner-only worktree cleanup identity as ADR 0281's narrow additional raw-identity metadata exception, and approve crash-only TTL reaping: a live preserved winner is cleaned only by LRU eviction or `Built.Close`; the reaper acts only after its allocation lock is released by process death. — Decision: pending + +## Interface contract + +- **gRPC / protobuf:** None — fork lifecycle, allocation records, and artifact handles remain local implementation details; no wire message carries a path, allocation ID, workspace key, or cleanup identity. +- **Exported Go APIs / interfaces:** Proposed exact replacement, subject to the second Human decision: `engine/tool.EnvironmentForker.Fork` returns a non-nil opaque `ForkLifecycle` instead of `func() error`; `ForkLifecycle` exposes `Cleanup(context.Context) error` and `Preserve() error`. `engine/agent` consumes only that capability for ordinary cleanup and Parallel promotion. A failed `Preserve` produces a Parallel tool error and invokes `Cleanup` without publishing an artifact. Remove `agent.PreservedForkStore` and `agent.LRUForkReaper` after composition moves per-workspace retention behind the lifecycle capability. Update `engine/api/*.txt` and classify the intentional change in `engine/CHANGELOG.md`. +- **Tool schemas:** `Parallel` parameters and result schema remain unchanged. Its result retains only the opaque `ArtifactHandle`; no child root, placement selector, exact `EnvironmentRef`, allocation ID, or cleanup locator is added. +- **CLI / config:** Add strict operator-global `temporary_storage.fork_reap_after` (default `1h`) and `preserved_fork_reap_after` (default `10d`), each constrained to `1m`–`30d` with preserved retention no shorter than ordinary retention, plus positive `preserved_fork_cap` (default `4`). Project-tier values warn-ignore. Only an explicitly supplied `--fork-preserved-cap` wins over the setting; an omitted flag does not overwrite YAML with the historical default. `temporary_storage.mode: system` disables fork registration and fork sweeping but leaves in-process LRU capping active. +- **Events / persistence:** Add a versioned, owner-only managedtemp fork manifest under `workspaces/${workspace-key}/forks/fork-${allocation-id}/`, an allocation lock, and owner-only retry/quarantine breadcrumb. States are `allocating`, `active`, `preserved`, `cleaning`, and retryable/quarantined terminal failure; `allocating` has no `workspace/` child and any unexpected contents quarantine, while every materialized/retryable state carries a closed `copy` or `worktree` strategy before materialization begins. No session snapshot, event, transcript, command, environment, credential, or model content changes. Update ADR 0027 List 1 with the workspace-key → LRU registry, fork allocations/locks, and worker extension; explicitly record that none changes session rehydration fidelity. +- **Security / authority:** Only managed local `internal/adapter/forker.Forker` allocations participate. Registration occurs before materialization under private no-link containment. Fixed cleanup dispatches from validated closed metadata, never child `.git`, artifact handles, agent input, or manifest shell text. A failed/moved/replaced/missing worktree parent is retained with a stable quarantine reason; cleanup never deletes the child workspace alone. Admission rejects managed roots equal to or beneath the source tree. +- **Compatibility / migration:** Existing `mode: system` forks continue using ordinary system temporary directories with current immediate cleanup and LRU behavior. Managed allocations are new, disposable state; unknown/future/malformed records fail closed and remain for inspection. Existing process-wide retained winners are not adopted or migrated. + +## In scope — 3 scenarios, in implementation order + +### Scenario 1 — Registered local fork survives creation and normal cleanup safely + +The local forker allocates each managed copy or worktree under ADR 0281's private workspace-key namespace before materializing it. Its opaque lifecycle capability owns cleanup across Subagent, Team, and Parallel normal paths; a worktree fallback remains within the same allocation. This preserves `EnvironmentForker`'s child-environment isolation contract while making cleanup reconstructible from validated adapter data ([ADR 0283](../adr/0283-managed-delegation-fork-lifecycle.md), [parallelism](../architecture/parallelism.md), [AGENTS.md](../../AGENTS.md)). + +**Acceptance:** +- AC1.1: A managed local copy or worktree fork is registered below its parent workspace key before child materialization and receives an owner-only versioned record and exclusive lifecycle lock; an `allocating` record has no materialized workspace and quarantines unexpected contents. + - verify: `TestManagedDelegationForkLifecycle_Scenario1_RegisterBeforeMaterialization` +- AC1.2: A strategy is recorded before its child materializes, a failed worktree attempt changes the same allocation to copy, and no unregistered fallback directory remains. + - verify: `TestManagedDelegationForkLifecycle_Scenario1_WorktreeFallbackKeepsAllocation` +- AC1.3: Ordinary Subagent, Team, Parallel-loser, and `join=all` cleanup removes the child and allocation record idempotently while holding the allocation lock; partial `cleaning` and cleanup-failure records retain their strategy and are retryable rather than guessed or deleted. + - verify: `TestManagedDelegationForkLifecycle_Scenario1_NormalCleanupSettlement` +- AC1.4: Managed allocation rejects a root at or beneath the source workspace and rejects malformed, symlinked, foreign-owned, or identity-mismatched control-plane entries without recursive deletion. + - verify: `TestInvariant_managed_fork_containment` + +### Scenario 2 — Selected winners retain opaque artifacts within their workspace bound + +A selected Parallel winner transfers its exact opaque lifecycle capability to the composition-owned LRU before its artifact is published. The registry partitions by ADR 0281 workspace key, not a filesystem path or session `EnvironmentRef`; normal system mode retains this memory-only cap. The model-facing result remains path-free and accurately describes artifact ephemerality ([ADR 0283](../adr/0283-managed-delegation-fork-lifecycle.md), [ADR 0291](../adr/0291-server-owned-session-placement.md), [AGENTS.md](../../AGENTS.md)). + +**Acceptance:** +- AC2.1: `join=first` and `join=judge` promote the selected winner before publishing its opaque artifact handle; a pre-promotion crash leaves an ordinary allocation and a post-promotion crash leaves a preserved allocation. A failed or shutdown-raced promotion cleans the selected fork, returns a tool error, and publishes no handle. + - verify: `TestManagedDelegationForkLifecycle_Scenario2_PromotionCrashWindows` +- AC2.2: Four retained winners sharing one workspace key evict the oldest on the fifth, while winners on a different key do not evict them; an empty workspace LRU is removed and `Built.Close` attempts each retained cleanup with a shared deadline, returning without claiming completion for an unfinished allocation. + - verify: `TestManagedDelegationForkLifecycle_Scenario2_WorkspaceScopedLRU` +- AC2.3: `mode: system` creates no managed fork allocation or sweep target but preserves the configured in-process LRU cap. + - verify: `TestManagedDelegationForkLifecycle_Scenario2_SystemModeKeepsLRU` +- AC2.4: Parallel result text and its tool specification expose an opaque artifact handle only and state its ephemeral retention; neither exposes a filesystem path or placement authority. + - verify: `TestManagedDelegationForkLifecycle_Scenario2_OpaqueArtifactGuidance` + +### Scenario 3 — A later process reaps only validated crash residue + +The shared ADR-0281 worker scans fork allocations under its existing root-GC coordination. It reclaims only expired, lock-claimable, validated records using a closed strategy; ordinary and preserved TTLs apply only after a former owner process has died. Ambiguous worktrees are quarantined intact for retry ([ADR 0281](../adr/0281-managed-temporary-command-leases.md), [ADR 0283](../adr/0283-managed-delegation-fork-lifecycle.md), [ADR 0027](../adr/0027-cloud-native.md)). + +**Acceptance:** +- AC3.1: A reaper skips a lock-held active or preserved allocation regardless of age, and reclaims validated crash-orphaned copy/worktree allocations only after the applicable ordinary or preserved retention interval. + - verify: `TestManagedDelegationForkLifecycle_Scenario3_CrashOnlyRetention` +- AC3.2: Reaper worktree cleanup revalidates adapter-recorded parent and Git-administration identities, never trusts the child `.git` pointer, and uses the fixed remove-worktree → remove-directory → prune sequence. + - verify: `TestManagedDelegationForkLifecycle_Scenario3_WorktreeRevalidation` +- AC3.3: A moved, replaced, missing, inaccessible, malformed, or identity-mismatched parent retains the complete allocation and writes an owner-only stable `cleanup-quarantined.json` breadcrumb; a later valid retry can clean it. + - verify: `TestManagedDelegationForkLifecycle_Scenario3_QuarantineAndRetry` +- AC3.4: Project-tier retention settings warn-ignore; valid operator durations are each `1m`–`30d` with preserved retention no shorter than ordinary retention; only an explicit CLI cap overrides YAML. Concurrent servers perform at most one interval-eligible root-GC sweep, and a sweep timeout records no successful completion. + - verify: `TestManagedDelegationForkLifecycle_Scenario3_OperatorConfiguration` +- AC3.5: The living parallelism architecture describes opaque artifact handles and no longer promises winner or merge-conflict filesystem paths. + - verify: inspection — `docs/architecture/parallelism.md` is the living implementation guide and must match the path-free result contract + +## Out of scope + +| Item | Defer-to | Decision | +|---|---|---| +| User-addressable cross-command artifact storage | ADR 0282 | Preserved fork artifacts remain opaque and ephemeral; scratch-cache API is separate | +| Explicit artifact inspect/release API | future ADR | No path or placement authority is introduced by this lifecycle work | +| Aggregate cap across all workspace keys | future capacity ADR | ADR 0283 chooses a per-workspace cap only | +| Managed lifecycle for custom, remote, or third-party forkers | future adapter contract | This plan covers only the in-tree local forker | +| Same-filesystem workspace move retention | ADR 0282 correction | ADR 0281's current path-derived workspace key is used as-is here | + +## Definition of done + +1. `task lint`, `task test`, `task api:check`, and `task docs` pass. +2. `task api:update` updates the intentional engine API snapshot and `engine/CHANGELOG.md` records its compatibility classification. +3. `task ac-trace-strict` resolves every named proof when this plan becomes `landed`. +4. `go run ./cmd/mecademo` remains green. +5. ADR 0027 resource and fidelity inventory is updated, and `/panel-review` has no unwaived ship blocker. + +## Deferred decisions and known risks + +- The five unchecked Human decisions are material. This plan cannot become `proposed` or be dispatched until they are recorded as resolved decisions. +- A workspace-key partition is not a global disk quota. The managed root requires operator capacity planning across distinct workspace keys. +- Fork manifests and locks are durable cleanup control-plane state, not session state; corruption fails closed and may require operator inspection. diff --git a/docs/adr/0283-managed-delegation-fork-lifecycle.md b/docs/adr/0283-managed-delegation-fork-lifecycle.md index 3225cb3bfe..68418274e1 100644 --- a/docs/adr/0283-managed-delegation-fork-lifecycle.md +++ b/docs/adr/0283-managed-delegation-fork-lifecycle.md @@ -2,32 +2,46 @@ - Status: Proposed - Date: 2026-08-30 -- Scope: `internal/adapter/forker`; `engine/agent` Subagent, Parallel, Team, and preserved-winner lifecycle; `internal/app` temporary-storage composition +- Scope: `internal/adapter/forker`; `internal/adapter/managedtemp`; `engine/tool` fork-lifecycle seam; `engine/agent` Parallel winner lifecycle; `internal/app` temporary-storage composition - Depends on: [ADR 0281](./0281-managed-temporary-command-leases.md) - Supersedes: none - Superseded by: none ## Context -`EnvironmentForker` creates `mecatlfork-*` directories for read-only Subagents, Parallel branches, and Team members. It creates either an isolated Git worktree or a force-copy workspace. Ordinary child lifecycle owners already remove their forks; Parallel `join=first` and `join=judge` deliberately retain a winning fork for inspection and manual recovery, subject to the in-process preserved-fork LRU. +`EnvironmentForker` creates temporary filesystem/environment forks for read-only Subagents, Parallel branches, and Team members. It creates either an isolated Git worktree or a force-copy workspace. Ordinary child lifecycle owners already remove their forks; Parallel `join=first` and `join=judge` deliberately retain a winning fork through the in-process preserved-fork LRU. -The existing graceful-shutdown LRU cleanup is necessary but insufficient: a crash, forced termination, or abandoned desktop process bypasses it and leaves a fork under the system temporary directory. These forks are full workspace copies or worktrees and can consume substantially more disk than ordinary command temporary files. +The existing graceful-shutdown LRU cleanup is necessary but insufficient: a crash, forced termination, or abandoned desktop process bypasses it and leaves a full workspace copy or worktree in system temporary storage. These allocations can consume substantially more disk than ordinary command temporary files. -ADR 0281 establishes the prerequisite managed-allocation roots, manifests, per-allocation locks, interval-gated cross-process reaping, operator-global settings, and system-mode rollback. Forks are not command leases, however: they are created by `EnvironmentForker` and may outlive an individual Bash invocation. They require their own lifecycle adapter and retention classes. +ADR 0281 establishes the prerequisite managed-allocation roots, manifests, per-allocation locks, interval-gated cross-process reaping, operator-global settings, and system-mode rollback. Forks are not command leases: they are created by `EnvironmentForker` and may outlive an individual Bash invocation. They need their own allocation family and lifecycle ownership. -**Terminology.** In this ADR, a “fork” is exclusively a filesystem/environment fork: an isolated Git worktree or force-copy workspace made by `EnvironmentForker`. It does not mean a session fork, continuation, or inherited conversation state. In particular, this ADR neither changes nor relies on Subagent's separate `fork: true` history-seeding option; it manages only the child environment's temporary filesystem allocation. +**Terminology.** A “fork” in this ADR is exclusively a filesystem/environment fork: an isolated Git worktree or force-copy workspace made by `EnvironmentForker`. It is not a session fork, continuation, or inherited conversation state. In particular, this ADR neither changes nor relies on Subagent's separate `fork: true` history-seeding option. ## Decision -### 1. Register every managed delegation fork as an owned allocation +### 1. Register every local managed delegation fork as an owned allocation -When `temporary_storage.mode: managed`, `EnvironmentForker` creates every filesystem/environment fork below the parent workspace's managed directory: +When `temporary_storage.mode: managed`, the in-tree local `forker.Forker` creates every filesystem/environment fork below the parent workspace's ADR-0281 workspace key: ```text -/mecatl/workspaces//forks/fork-/workspace/ +/mecatl/workspaces//forks/fork-/workspace/ ``` -It registers that allocation with the ADR-0281 managed-allocation protocol as an **ordinary** fork before materializing a Git worktree or force-copy. The manifest records only lifecycle and cleanup data: allocation ID, creation time, fork strategy (`worktree` or `copy`), owner process identity, the canonical adapter-derived parent-repository locator for a worktree, and physical identities for that parent repository, its Git administration directory, and the managed workspace. It never records model content, command arguments, environment values, credentials, or transcript data. +The forker registers the allocation before materialization. Registration begins in `allocating` state and records no strategy; its `workspace/` child must not exist, so a reaper removes only that empty validated allocation and quarantines unexpected contents. It then records the closed `copy` or `worktree` strategy under the allocation lock **before** materializing that strategy, and transitions to `active` only after materialization succeeds. A failed worktree attempt changes the same allocation to `copy`; it does not mint an untracked fallback directory. A crash in `active`, `preserved`, `cleaning`, or retryable failure is recovered through the recorded strategy. `cleaning` is a resumable idempotent attempt, not authority to guess a strategy. + +The manifest contains only lifecycle and cleanup metadata: allocation ID, timestamps, state, strategy after it is selected, owner process identity, the ADR-0281 workspace key and managed-workspace identity, and the worktree cleanup identity needed to validate the registered parent repository. It never records model content, command arguments, environment values, credentials, or transcript data. The worktree identity is a narrow, owner-only exception to ADR 0281's general no-raw-identity metadata rule: it is written only by the adapter after canonicalization, is never model-visible, and exists solely to revalidate fixed Git cleanup. It must be specified and reviewed with the managedtemp manifest version rather than inferred from the child `.git` file or agent input. + +`EnvironmentForker` remains strategy-agnostic to the agent, but its returned cleanup capability must carry opaque adapter-owned lifecycle state. A Parallel winner promotes that exact capability before publishing its artifact handle; normal owners invoke the same capability's idempotent `Cleanup(context.Context) error`. `Preserve() error` fails closed: Parallel cleans the selected fork and returns a tool error without publishing an artifact handle. A registry that has begun shutdown similarly rejects preservation and leaves cleanup with the caller. The core never parses allocation IDs, workspace keys, or child paths. The exact exported `engine/tool` seam is a material interface decision and is recorded in the acceptance plan before implementation. + +Custom, remote, or third-party `EnvironmentForker` implementations remain unmanaged unless they explicitly implement the new lifecycle capability. This ADR does not claim to redirect arbitrary external forkers beneath the local managed root. + +### 2. Keep opaque artifact handles; do not expose fork paths + +A selected winner continues to return an opaque `ArtifactHandle`, never a child filesystem root, `EnvironmentRef`, placement selector, or cleanup locator. The handle is not a placement capability. Model-visible result text describes the winner as ephemeral: the retained artifact can disappear through LRU eviction, graceful shutdown, or post-crash reaping after the preserved-fork retention interval. It does not promise a usable cross-command path. + +The existing `PreservedForkStore.Preserve(handle, cleanup)` contract cannot select an LRU by workspace or promote the allocation before publication. The replacement lifecycle seam must transfer preservation through the opaque child allocation capability, allowing composition to select the workspace-specific LRU without making paths or allocation metadata visible to `engine/agent`. The obsolete root-based description is not retained. + +### 3. Use per-workspace retained-winner LRUs and crash-only TTL reaping The operator-global settings add: @@ -39,84 +53,68 @@ temporary_storage: preserved_fork_cap: 4 ``` -These follow ADR 0281's shared managed-root layout and strict operator-tier rules. Project settings cannot redirect the managed root or alter retention. `preserved_fork_cap` is a positive **per-workspace-instance** bound, defaulting to four; the existing `--fork-preserved-cap` is an operator-tier CLI override and wins over that setting. Every Parallel tool targeting the same stable workspace identity shares that workspace's LRU, including separate sessions on the same `app.Build`; distinct workspaces do not evict one another's winners. Composition owns a process-scoped workspace-identity → LRU registry, creates an LRU lazily for a workspace, removes an empty one, and drains every remaining LRU on `Built.Close`. This removes the former process-wide total disk bound; total retained winners can reach four times the number of active workspace identities, while the per-workspace cap and ten-day TTL bound each workspace independently. `mode: system` is a lifecycle rollback only: it disables managed allocation registration and TTL reaping, while the per-workspace LRU and its cap remain in force. +These follow ADR 0281's shared managed-root layout and strict operator-tier rules. Both retention durations must be between one minute and thirty days, and `preserved_fork_reap_after` must not be less than `fork_reap_after`. Project settings cannot redirect the managed root or alter retention. `preserved_fork_cap` is a positive per-workspace-key bound, defaulting to four; an explicitly supplied `--fork-preserved-cap` is an operator-tier CLI override and wins over that setting. An omitted flag does not override YAML with its historical default. -### 2. Preserve current normal cleanup owners and strategies +Composition owns a process-scoped workspace-key → LRU registry. It creates an LRU lazily, removes an empty one, and drains every remaining LRU on `Built.Close`. Parallel winners from different sessions on the same ADR-0281 workspace key share one LRU; distinct keys do not evict one another. This intentionally replaces the former process-wide cap with a per-workspace cap. `mode: system` disables managed allocation registration and reaping, while the in-process per-workspace LRU cap remains in force. -This ADR does not move normal cleanup responsibility into the generic reaper: +Active allocations retain their allocation lock through normal cleanup, exactly as ADR 0281 requires. Therefore fork TTLs are **crash-only reaping fallbacks**: a live process's retained winner is removed by LRU eviction or graceful `Built.Close`, not by a concurrent reaper. After a process dies and the operating system releases the lock, an ordinary fork is eligible after `fork_reap_after`; a promoted preserved winner is eligible after `preserved_fork_reap_after`. -- ordinary Subagent forks remain owned by the child-drive lifecycle; -- Team forks remain owned by the supervisor's final cleanup; -- Parallel losers and `join=all` branches remain owned by the Parallel call; -- a preserved winner remains owned by the existing LRU and graceful `Built.Close` cleanup. +### 4. Preserve existing normal owners and make failure retryable -Each normal cleanup path retains its allocation lock while it records terminal state, runs its existing idempotent cleanup protocol, and removes the allocation record; it releases that lock only as the final action. Copy forks use recursive removal. Worktree forks use the existing bounded sequence: remove the Git worktree, remove the directory, then prune Git worktree administration. No new cleanup command is built from a model-controlled path. +Normal cleanup remains with the existing lifecycle owners: -A fork is necessarily ordinary when `EnvironmentForker` creates it: winner selection happens later in the Parallel tool. The composition-owned per-workspace LRU registry wraps `PreservedForkStore` with an adapter-owned opaque allocation handle for the target workspace identity. `Preserve(root, cleanup)` first calls the handle's idempotent `PromotePreserved`, which updates the validated manifest under the allocation lock from ordinary to preserved-winner, then hands the existing cleanup callback to that workspace's LRU. Promotion completes before the winner is published in a Parallel result. A crash before promotion leaves an ordinary one-hour orphan; a crash after promotion leaves a ten-day preserved-winner orphan. The core agent continues to see only its existing root-and-cleanup store contract and never imports the allocation adapter. - -The generic reaper is the crash/forced-termination fallback, not a second active lifecycle owner. - -A selected winner has no dependable "main agent is done inspecting it" signal. The parent may inspect the path in its next turn, a later user turn, or never; parent-run/session completion is therefore not a safe cleanup boundary. In normal operation, a winner is retained until the first of: (1) a fifth winner for the same workspace instance causes that workspace's four-entry LRU to evict the oldest; (2) graceful `Built.Close` drains the workspace LRU; or (3) the managed allocation reaper reaches its ten-day preserved-winner TTL. The path is deliberately an ephemeral inspection opportunity, not a lease the model owns. A future explicit operator release action may shorten that lifetime, but is not needed for automatic cleanup. - -**Future capacity direction.** The per-workspace cap deliberately has no aggregate cap across many workspace identities. A future operator-level global disk or preserved-winner budget may evict the oldest eligible winner across workspace LRUs, but it is not part of this decision; deployments serving many workspaces must size the managed root accordingly. - -### 3. Use two retention classes - -| Class | Members | Default retention | -|---|---|---:| -| ordinary fork | Subagent, Team, Parallel loser, `join=all`, and a failed/abandoned ordinary branch | `fork_reap_after`: 1 hour | -| preserved winner | selected `join=first` / `join=judge` Parallel winner | `preserved_fork_reap_after`: 10 days | - -The per-workspace in-process LRU may clean a winner before its ten-day fallback TTL, and graceful `Built.Close` drains every workspace LRU. The longer TTL only governs a preserved-winner allocation left behind when its owning process dies. +- ordinary Subagent forks remain owned by the child-drive lifecycle; +- Team forks remain owned by the supervisor's final cleanup; +- Parallel losers and `join=all` branches remain owned by the Parallel call; and +- a selected winner is owned by the target workspace's LRU until eviction or `Built.Close`. -As with ADR 0281 command leases, TTL is deliberate cleanup authority for validated managed allocations. The reaper may remove an expired managed fork even if a detached process still uses it; long-lived work must not depend on a disposable fork path. +Each normal cleanup path retains the allocation lock while it records terminal state, runs idempotent cleanup, and removes the allocation record; it releases the lock only as its final action. Copy forks use confined recursive removal. Worktree forks use a fixed, adapter-owned sequence: revalidate the recorded parent-repository and Git-administration identities, remove the Git worktree, remove the managed directory, then prune worktree administration. No cleanup command is built from model-controlled input. -### 4. Reconstruct cleanup only from validated allocation metadata +A cleanup failure never removes the allocation record. It records a stable retryable failure state and leaves the validated allocation for a later sweep. A validation failure, including a moved, replaced, missing, inaccessible, or identity-mismatched parent repository, writes an owner-only `cleanup-quarantined.json` breadcrumb containing only a stable failure reason, timestamp, and retry count, then retains the complete allocation. The reaper never removes a worktree directory alone when parent validation fails. -A reaper may claim a fork allocation only after acquiring its non-blocking exclusive allocation lock and validating root containment, ownership, manifest schema, strategy, and parent/workspace identity. It invokes a fixed cleanup strategy selected from the manifest's closed strategy value; it never executes arbitrary manifest content or derives a shell command from a path supplied by an agent. +`Built.Close` applies the existing ADR-0281 shutdown budget to LRU cleanup through cancellation-aware lifecycle cleanup. It attempts every retained entry before the deadline; an unfinished cleanup retains its allocation record for a later reaper rather than blocking shutdown indefinitely or claiming removal. -For a worktree allocation, cleanup revalidates the manifest's canonical adapter-derived parent-repository locator and its recorded physical Git-administration identity under the allocation lock. It verifies that the managed workspace is that repository's expected registered worktree before applying the existing Git-environment hardening and fixed worktree-remove → directory-remove → prune sequence. It never reads a cleanup locator from the child `.git` pointer or from agent input. If the parent repository was moved, replaced, missing, inaccessible, or otherwise fails validation, the reaper writes an owner-only `cleanup-quarantined.json` breadcrumb in the allocation directory with a stable failure reason, timestamp, and retry count, then retains the complete allocation for operator inspection and later retry; it does not remove the workspace alone. It never widens into a `/tmp/mecatlfork-*` name scan. TTL is therefore a deterministic reclaim target only for allocations whose cleanup identity remains validated, not authority to delete an ambiguous worktree. +### 5. Reap only validated fork allocations -### 5. Keep model-visible paths honest +The shared managed-temp worker scans the `forks/` family under ADR-0281 root-GC coordination. A reaper may claim an allocation only after acquiring its non-blocking exclusive allocation lock and validating root containment, ownership, manifest version/schema, state, workspace identity, and strategy. It dispatches only a fixed cleanup strategy selected from the closed manifest strategy; it neither executes manifest content nor derives a shell command from an agent-controlled path. -Parallel result text continues to expose a selected winner path for inspection, but identifies it as ephemeral. It states that the path can disappear through LRU eviction, graceful shutdown, or the configured preserved-winner TTL. Fork paths remain implementation artifacts, not stable user storage. Cross-command disposable output belongs in the ADR-0282 workspace scratch cache; durable project output belongs in the workspace proper. +For worktrees, cleanup revalidates the adapter-recorded parent repository locator and Git-administration identity. It never reads a cleanup locator from the child `.git` pointer. A missing, malformed, future-version, foreign-owned, symlinked, unrecognised, or identity-mismatched allocation is retained, never deleted. Managed-mode admission must also reject an allocation root equal to or below the source tree, preventing recursive self-copy when an operator configures a managed root inside a workspace. ## Consequences **Benefits:** -- Every mecatl-owned fork is attributable, independently lockable, and reaped after process death rather than accumulating under generic system temp. +- Every locally managed fork is attributable, independently lockable, and reaped after process death rather than accumulating under generic system temp. - Existing immediate cleanup and preserved-winner workflows remain intact. -- Retention is calibrated to intent: one hour for ordinary abandoned forks, ten days for intentionally inspectable winners, and four retained winners per workspace instance. -- Operators can relocate the shared managed root, tune both retention classes and the per-workspace cap, or disable managed fork allocation/reaping with the same rollback switch as command temporary storage. +- Retention is calibrated to intent: one hour for ordinary crash residue, ten days for intentionally retained winners after process death, and four retained winners per workspace key. +- Opaque artifact handles preserve the path-free delegation and placement boundary. **Costs and limits:** -- Worktree cleanup needs a carefully constrained reconstruction path after process death; failures must remain retriable rather than deleting an ambiguous directory. -- An expired managed fork is disposable even if a detached process still references it; users must not use fork paths as long-lived storage. -- `mode: system` restores the existing crash-residue behavior by design, because the harness has no managed ownership record for those forks. -- A per-workspace cap intentionally does not impose a process-wide winner count; deployments that serve many distinct workspaces must size disk capacity against the configured root and the ten-day fallback TTL. -- The managed fork registry extends the outlives-a-call resource inventory and needs conformance tests across copy, worktree, cancellation, LRU, graceful close, crash, and concurrent-server paths. +- The lifecycle capability is an engine exported-API change and requires API-compatibility and changelog review. +- Worktree cleanup needs constrained reconstruction; ambiguous parent identity is quarantined and retried rather than guessed. +- A per-workspace cap intentionally does not impose a process-wide aggregate disk bound; deployments serving many workspaces must size the managed root accordingly. +- `mode: system` restores crash residue behavior by design because no managed ownership record exists. ## Implementation and acceptance outline After ADR 0281 lands: -1. Add `fork_reap_after`, `preserved_fork_reap_after`, and per-workspace `preserved_fork_cap` to the same strict operator-global `temporary_storage` settings schema and reference documentation; use ADR 0281's shared managed-root layout. -2. Extend the ADR-0281 managed-allocation protocol with a closed `fork` family, `copy`/`worktree` strategies, and validated identity metadata. -3. Route `EnvironmentForker` allocation through the managed root and register each fork as ordinary before copy/worktree materialization; persist the validated parent-repository locator and Git-administration identity for worktrees while preserving current fallback and rollback behavior. -4. Add the composition-owned workspace-identity → LRU registry and adapter-owned ordinary→preserved promotion wrapper around `PreservedForkStore`; promote under the allocation lock before the target workspace's LRU retention/result publication, remove empty registries, drain all on `Built.Close`, and pin both crash windows. -5. Implement the fixed, hardened reaper cleanup strategies and retry diagnostics for validated orphaned copy/worktree allocations; a moved, replaced, missing, or invalid parent repository retains the complete allocation for retry. -6. Update model-visible winner-path wording and architecture/operator documentation. -7. Test managed/system modes (including the per-workspace LRU remaining active in system mode), configured TTLs and CLI cap precedence, four-per-workspace isolation across sessions, LRU eviction, empty-registry removal, Build-close draining, copy/worktree cleanup, normal-cleanup versus sweep locking, ordinary/preserved promotion and crash windows, crash-orphan recovery, moved/replaced/missing parent repositories, `cleanup-quarantined.json` breadcrumbs, modified child `.git` pointers, invalid manifests/identity mismatches, reaper concurrency, and retry-on-cleanup-failure behavior. +1. Record the exact lifecycle-capability and preserved-store replacement in the accepted interface contract, then update the engine API snapshot and changelog with implementation. +2. Add fork retention settings to the strict operator-global `temporary_storage` schema, reference documentation, and CLI-precedence tests. +3. Extend managedtemp with the versioned fork allocation state machine, manifest, lock, validated identities, promotion, normal cleanup settlement, and retry/quarantine records. +4. Route local `forker.Forker` allocation through the managed root before copy/worktree materialization, preserving fallback and system-mode behavior. +5. Add the composition-owned workspace-key LRU registry and transfer preserved-winner ownership through the opaque lifecycle capability; remove empty entries and drain all LRUs on `Built.Close` within its cleanup budget. +6. Extend the shared worker with fixed, hardened copy/worktree reaping and quarantine diagnostics. +7. Update living architecture and operator documentation, including ADR 0027's resource inventory. Test managed/system modes, configuration precedence, per-workspace isolation, promotion crash windows, normal-cleanup versus sweep locking, copy/worktree cleanup, retry/quarantine, containment, invalid manifests/identities, reaper concurrency, and model-visible ephemeral-handle guidance. ## See also -- [ADR 0281 — Managed temporary command leases and deterministic reaping](./0281-managed-temporary-command-leases.md) — prerequisite allocation, reaper, configuration, and rollback protocol. -- [ADR 0282 — Managed workspace scratch cache](./0282-managed-workspace-scratch-cache.md) — cross-command artifact storage distinct from forks. -- [ADR 0201 — Background Bash](./0201-background-bash.md) — background command lifecycle. -- [ADR 0027 — Cloud-native arc](./0027-cloud-native.md) — resource inventory convention. -- [Architecture: parallelism](../architecture/parallelism.md) — current Parallel fork and preserved-winner behavior. -- [Architecture: subagents and teams](../architecture/subagents-and-teams.md) — current child and team fork ownership. -- [Architecture: extensibility](../architecture/extensibility.md) — command-runner and environment seams. -- [Production readiness](../design/PRODUCTION-READINESS.md) — shipped/deferred status tracking. +- [ADR 0281 — Managed temporary command leases and deterministic reaping](./0281-managed-temporary-command-leases.md) +- [ADR 0282 — Managed workspace scratch cache](./0282-managed-workspace-scratch-cache.md) +- [ADR 0201 — Background Bash](./0201-background-bash.md) +- [ADR 0027 — Cloud-native arc](./0027-cloud-native.md) +- [Architecture: parallelism](../architecture/parallelism.md) +- [Architecture: subagents and teams](../architecture/subagents-and-teams.md) +- [Architecture: extensibility](../architecture/extensibility.md) +- [Production readiness](../design/PRODUCTION-READINESS.md) diff --git a/docs/architecture/parallelism.md b/docs/architecture/parallelism.md index d6cc075132..19b8e3ea59 100644 --- a/docs/architecture/parallelism.md +++ b/docs/architecture/parallelism.md @@ -29,12 +29,13 @@ the Subagent `agentId:` trailer (issue #30; `InspectSubagent`'s gate admits both **Merging a winner back.** Parallel does NOT auto-merge fan-out: for `join=all` every fork is torn down after the join (the result reports branch ids for transcript pulls, NOT workspace paths — the forks are gone); for `join=first` / -`join=judge` the winner workspace path is reported. The shared process LRU normally -retains winners until eviction or graceful app shutdown, but a path may already be gone -if shutdown began concurrently. Shutdown drains -both currently retained winners and eviction cleanups detached before closure, without -holding the reaper lock during filesystem work; it does not wait for a later `Preserve`. -A crash remains a residual and does not sweep them at startup. A SINGLE-BRANCH `join=first`/`join=judge` winner is auto-merged back +`join=judge` the result reports an opaque artifact handle, never a winner path. +The shared process LRU normally retains winners until eviction or graceful app +shutdown, but an artifact is ephemeral and may already be gone if shutdown began +concurrently. Shutdown drains both currently retained winners and eviction cleanups +detached before closure, without holding the reaper lock during filesystem work; it +does not wait for a later `Preserve`. A crash remains a residual and does not sweep +them at startup. A SINGLE-BRANCH `join=first`/`join=judge` winner is auto-merged back into the parent workspace BY DEFAULT (no flag; see [ADR 0039](../adr/0039-parallel-auto-merge.md)): the winner's diff is applied via `tool.EnvironmentMerger` (the `forker.Merger` adapter — `git diff --no-textconv HEAD` @@ -43,8 +44,8 @@ merge refuses `.gitattributes`-touching patches and runs `--no-textconv` to clos attacker-named `diff.*.textconv`/`filter.*.smudge` RCE from an untrusted fork `.git`). Multi-branch runs and `join=all` NEVER auto-merge (the no-auto-merge boundary stays for fan-out). On a conflict the merge surfaces a tool error with the -winner workspace path, which is ephemeral and may already be gone if graceful shutdown -began; it never forces. `Parallel.ReadOnly()` +opaque artifact handle; the preserved fork is ephemeral and may already be gone if +graceful shutdown began. It never forces. `Parallel.ReadOnly()` stays `true` — the merge is a POST-RUN step, not a dispatch-time mutation, so read-parallel / mutate-serial is unaffected.