diff --git a/skills/delegate-kit/SKILL.md b/skills/delegate-kit/SKILL.md index d65fb4d..99fc280 100644 --- a/skills/delegate-kit/SKILL.md +++ b/skills/delegate-kit/SKILL.md @@ -24,6 +24,8 @@ Clarify the requested result, constraints and acceptance checks. Consider substa Parallel writers need disjoint ownership and stable interfaces. Coupled changes have one owner or proceed sequentially. Choose worker count from ready outcomes and integration capacity; there is no default cap of one. Honor explicit user and host limits. Do independent work while workers run; avoid duplicating their investigation. +Choose worker count for quality and elapsed time within explicit cost and capacity limits, not a default of one. Parallelize distinct ready outcomes with clear ownership; keep coupled work under one owner or staged dependencies. State the count and each worker's distinct value. + ## Prepare and dispatch Use [brief-template.md](references/brief-template.md): outcome, necessary facts, scope, workspace, constraints, checks and authorized finishing actions. Workers should begin from the brief and repository without the parent's full conversation. `when` and coordination instructions belong to you; profile `instructions` and the brief go to the worker. @@ -34,9 +36,11 @@ Prepare with the saved session, stable task ID, selected profile and brief. The `run` starts a CLI supervisor or returns the verified native/Paseo invocation. A bridge invocation is preparation, not an agent: call the actual host tool once and attach its returned ID. If dispatch outcome is uncertain, reconcile with the host before another call. Record correlated completion/permission events. Unique Claude native definitions must be discovered by that host; otherwise use CLI. Never rewrite a shared role when switching presets. +Status reporting is on by default. Read [observability.md](references/observability.md) and announce planned profiles/executors. Combine immediate preparation and dispatch into one runtime-backed launch update; report prepared separately only on delay, failure or user action. Prepared is not running. + ## Await and verify results -Use host completion notifications or runtime `wait`. Read the compact result by default; full private logs are diagnostic artifacts. No log-summarizer model or periodic LLM heartbeat is needed. After each bounded wait, check runtime health or query the saved host agent without sending a prompt. A wait timeout does not stop the worker or authorize a duplicate. An attention alert requires diagnosis of process/turn progress; continue waiting only with a concrete reason, or stop/recover a confirmed stall. See [lifecycle and recovery](references/routing.md#lifecycle-and-recovery). Intervene for a blocker, permission request, user correction, explicit failure, breached limit or data risk. +Use host completion notifications or runtime `watch`; use `wait` for one run. Read compact results; reserve `watch --full` and logs for diagnosis. Report meaningful transitions as one-line rows. On an unchanged 60-second timeout, send at most one aggregate heartbeat if otherwise silent; do not repeat the roster or models. No log-summarizer model is needed. Check runtime health or query the saved host agent without prompting. A timeout neither stops the worker nor authorizes a duplicate. Diagnose attention alerts; keep waiting only for a concrete reason, or recover a confirmed stall. See [lifecycle and recovery](references/routing.md#lifecycle-and-recovery). Intervene for blockers, permission requests, corrections, failures, breached limits or data risk. Transport acknowledgement, terminal turn, valid result and coordinator acceptance are separate. Compare evidence against acceptance checks, run relevant checks the adapter could not perform, and accept only completed work. Explain unverified claims. Runtime `accept` refuses incomplete required review sets; it records your judgment, not proof that tests passed. diff --git a/skills/delegate-kit/references/compatibility.md b/skills/delegate-kit/references/compatibility.md index 28bc09e..2defdae 100644 --- a/skills/delegate-kit/references/compatibility.md +++ b/skills/delegate-kit/references/compatibility.md @@ -2,18 +2,18 @@ | Path | Implemented | Automated evidence | Local evidence | Live model call | |---|---|---|---|---| -| Codex CLI | fresh, result, exact resume, cancellation | legacy + v2 fake CLI tests | 0.153.4 | 2026-09-16: Sol fix + exact resume, Luna independent review + cancellation; isolated fixture, low reasoning | +| Codex CLI | fresh, result, exact resume, cancellation | legacy fake CLI supervisor; v2 preset/lease/observability fixtures | 0.153.4 | 2026-09-16: Sol fix + exact resume, Luna independent review + cancellation; isolated fixture, low reasoning | | Claude CLI | fresh, result, exact resume | adapter + supervisor fixtures | help/version 2.1.268 | not run | | Gemini CLI | fresh, result, exact resume | adapter + supervisor fixtures | help/version 0.36.0 | not run | | OpenCode | permissions, provider/model, exact resume | legacy permission/adapter fixtures | help/version 1.18.23 | not run | -| Pi | official RPC via installed SDK, isolated settings, exact resume | SDK/protocol + fake process tests | not installed | not run | -| OMP | RPC v2 negotiation/chunks, terminal result, resume | protocol + fake CLI tests | 18.1.17 ready/state handshake, no prompt | not run | -| Native Codex/Claude | prepare, exact invocation, unique definitions, attach/events | bridge fixtures | tool/schema-dependent; no agent dispatched | not run | -| Paseo | materialized create/follow-up settings, daemon/workspace lease | bridge fixtures | no daemon/tools available | not run | +| Pi | official RPC via installed SDK, isolated settings, exact resume | adapter/runtime code review only in this repository | not installed | not run | +| OMP | RPC v2 negotiation/chunks, terminal result, resume | adapter/runtime code review only in this repository | 18.1.17 ready/state handshake, no prompt | not run | +| Native Codex/Claude | prepare, exact invocation, unique definitions, attach/events | observability metadata fixture only | tool/schema-dependent; no agent dispatched | not run | +| Paseo | materialized create/follow-up settings, daemon/workspace lease | preset validation and code review only | no daemon/tools available | not run | Native writers require verified host enforcement of the reserved worktree binding. Hosts that cannot establish it must use an explicitly selected CLI route. -Native Pi/OMP and a dedicated T3 bridge are not implemented. A direct CLI does not become a Paseo UI agent. Desktop/cloud chats need actual shell and host tools. [Adapter contracts, official sources and limitations](providers.md). +Native Pi/OMP and a dedicated T3 bridge are not implemented; Pi and OMP use the explicit CLI/RPC transport. A direct CLI does not become a Paseo UI agent. Desktop/cloud chats need actual shell and host tools. [Adapter contracts, official sources and limitations](providers.md). Worktrees coordinate writers, not all filesystem permissions. Read-only tool controls differ by executor. Pi/OMP writers deliberately exclude shell and internal delegation; the coordinator performs command checks and authorized commits. Runtime usage is null when unavailable, not zero. The runtime counts its own reservations/continuations and known legacy work; it cannot account for arbitrary agents launched outside it. No savings or model-quality percentage is promised. diff --git a/skills/delegate-kit/references/hosts.md b/skills/delegate-kit/references/hosts.md index b490e53..3175ef5 100644 --- a/skills/delegate-kit/references/hosts.md +++ b/skills/delegate-kit/references/hosts.md @@ -42,6 +42,6 @@ Use actual `list_providers`, `list_models`, `inspect_provider` results, not Pase Create/select the workspace through Paseo first. Pass `--workspace FILE` containing `owner: "paseo"`, its `id`, `daemon`, and optional `remote: true`; `--cwd` is a daemon path and is not checked locally. The runtime materializes `create_agent` with `provider: harness/model`, workspaceId, settings.modeId and optional thinkingOptionId, and notifyOnFinish. It never requires a Paseo profile. Invoke in the parent agent context on the saved daemon so Paseo owns the parent-child relationship. -Attach with both `--host-agent` and `--workspace-id`. Follow-up uses `send_agent_prompt` with the saved agentId and daemon. Notifications trigger result ingestion. Workspace creation/archive stay with Paseo; preserve changes and references before archiving. A direct Pi/OMP CLI is not automatically visible in Paseo UI. No live Paseo daemon or tool schema was available for this implementation; the bridge is contract-tested against the official documented tool surface and requires current-host verification before use. +Attach with both `--host-agent` and `--workspace-id`. Follow-up uses `send_agent_prompt` with the saved agentId and daemon. Notifications trigger result ingestion. Workspace creation/archive stay with Paseo; preserve changes and references before archiving. A direct Pi/OMP CLI is not automatically visible in Paseo UI. No live Paseo daemon or tool schema was available for this implementation; the bridge was reviewed against the official documented tool surface and requires current-host verification before use. -Sources checked 2026-09-16: [Paseo skill and tool contract](https://github.com/getpaseo/paseo/blob/main/skills/paseo/SKILL.md), [Paseo orchestration](https://paseo.sh/docs/orchestration). Local Desktop/cloud chats without shell and corresponding host tools cannot execute this bridge. +Sources checked 2026-09-16: [Paseo skill and tool contract](https://github.com/getpaseo/paseo/blob/main/skills/paseo/SKILL.md), [Paseo orchestration](https://paseo.sh/docs/orchestration). The bridge was reviewed against that documented surface but is not contract-tested by a complete repository fixture suite. Local Desktop/cloud chats without shell and corresponding host tools cannot execute this bridge. diff --git a/skills/delegate-kit/references/migration.md b/skills/delegate-kit/references/migration.md index 2f8f102..a1ffb78 100644 --- a/skills/delegate-kit/references/migration.md +++ b/skills/delegate-kit/references/migration.md @@ -11,11 +11,12 @@ For assignments that intentionally depended on the parent, supply a decisions fi ```json { "parents": { "gpt": "codex", "claude": "claude" }, + "authors": { "gpt": "codex", "claude": "claude" }, "default_preset": "gpt" } ``` -For auto/native family assignments, a matching explicit parent determines the original harness. Multiple harnesses in one family require that choice rather than selecting the first backend. These mappings are explicit user decisions; team names themselves have no routing meaning. A missing default remains unset. If other ambiguities remain, use conversational setup to construct and validate the intended complete v2 JSON from the dry-run output. Keep the original legacy file unchanged as evidence; the automatic converter deliberately refuses to invent missing models or translate preferences into specialties. +For auto/native family assignments, a matching explicit parent determines the original harness. Legacy reviewer/verifier backend maps also require `authors.PRESET`, because duo routing may deliberately choose a different family from the author. Multiple harnesses in one family require an explicit choice rather than selecting the first backend. These mappings are explicit user decisions; team names themselves have no routing meaning. A missing default remains unset. If other ambiguities remain, use conversational setup to construct and validate the intended complete v2 JSON from the dry-run output. Keep the original legacy file unchanged as evidence; the automatic converter deliberately refuses to invent missing models or translate preferences into specialties. After user authorization, `migrate --apply --decisions FILE` backs up the exact old config, saves complete presets atomically per file, and records `migration-v2.json`. It refuses existing destinations/settings instead of overwriting them. A completed migration is idempotent and preserves later v2 edits. If a process crashes during a multi-file apply before the journal is saved, rerun dry-run and inspect the backup/proposed files; an existing destination is a recovery diagnostic, never permission to overwrite. diff --git a/skills/delegate-kit/references/observability.md b/skills/delegate-kit/references/observability.md new file mode 100644 index 0000000..4f65784 --- /dev/null +++ b/skills/delegate-kit/references/observability.md @@ -0,0 +1,42 @@ +# Team status reporting + +This protocol is active for every delegated task; the user does not enable it with a command or preset field. Runtime state is the source of truth. Use `overview --session HOST:ID --task TASK` to confirm launch state, then retain its cursor and use compact `watch --session HOST:ID --task TASK --after CURSOR --timeout-ms 60000` while agents remain active. Use `watch --full` only for diagnosis. + +## Low-noise policy + +1. Before preparation, state the planned count by role/profile and requested executor. +2. When dispatch immediately follows preparation, omit a separate prepared message and send one launch update after actual dispatch. Surface prepared reservations separately only when dispatch waits, fails or needs user action. +3. Show the full active roster once after launch. After that, show only rows whose meaningful state changed. Repeat the full roster only after a material team change such as adding, replacing or removing an agent. +4. On an unchanged one-minute timeout, send at most one aggregate one-line heartbeat while work remains active, and only if no other meaningful commentary was sent during that interval. Never repeat agent rows, task descriptions or models in an unchanged heartbeat. +5. Report attention immediately. Before final handoff, report completion, acceptance and any failed, blocked or unverified run. + +The timeout performs no model call, but tool output and commentary still occupy coordinator context. Compact watch results and deduplicated messages therefore matter. Do not narrate every poll or streaming-progress update. + +## One-line interface + +Use one line per displayed agent, localized to the user's language: + +```text +🟢 Implementer — implementer-api — работает +🟢 Implementer — implementer-ui — работает +🟡 Researcher — research-auth — завершён, ожидает интеграции +🔴 Reviewer — reviewer-hard — требует внимания +``` + +Use the assigned short outcome instead of the profile when it is already known and fits on one line. Otherwise the stable profile is the task label. Show requested models in the initial plan/launch update and again only when identity changes or is relevant to a problem. Call a model runtime-confirmed only when `actual_model` is present. + +Indicator semantics: + +- 🟢 active work with confirmed execution, or accepted completion; +- 🟡 reserved, starting, cancelling, or completed but not yet accepted/integrated; +- 🔴 permission, orphaned, blocked, failed, timeout, or any health state requiring attention. + +For an unchanged timeout use only the aggregate row, for example: + +```text +🟢 Delegate Kit — 4 агента — работают 3, завершён 1, проблем 0 +``` + +Use lifecycle terms precisely: `prepared` is reserved and `starting` is dispatching. Count an agent as started only when `execution_started` is true; for host routes that requires attach, and for CLI routes it requires a spawned executor process. `running` without that evidence is not yet working. `permission` needs user action, and `finished` is completed but not coordinator acceptance. A continuation replaces its earlier attempt in agent counts; `summary.attempts` in the full overview retains the audit total. The stage follows confirmed active work, while reserved later roles remain visible without advancing it. + +`watch` is event-driven over meaningful lifecycle, health, identity and acceptance changes. It intentionally ignores heartbeat-file churn and streaming token output. Its timeout is a liveness cadence, not evidence of worker progress and not permission to retry. diff --git a/skills/delegate-kit/references/omp-setup.md b/skills/delegate-kit/references/omp-setup.md index 0a88bc8..156ab73 100644 --- a/skills/delegate-kit/references/omp-setup.md +++ b/skills/delegate-kit/references/omp-setup.md @@ -22,6 +22,6 @@ After sign-in, use `omp models openrouter --json` or the model picker to choose Check effective config on both hosts and compare only non-secret managed settings. Launch RPC in an isolated temporary cwd with the same worker overrides; negotiate protocol v2, get_state, select the exact already configured model, disable retry/compaction and get_state again, then close stdin. Do not send prompt. This checks startup and the protocol, not provider authentication or generated results. -Local CLI status checks and supervisor heartbeats call no model. The coordinator does spend tokens when it receives a tool result: prefer a bounded long CLI wait (for example 300000 ms when the host tool supports it) over one-minute status conversations. Internal health checks still return early for completion or required attention. Host tool execution limits may shorten a wait; native/Paseo state checks require the host bridge and may wake the coordinator. This setup does not claim zero total coordination overhead. +Local CLI status checks, `overview`, `watch` and supervisor heartbeats call no model. Follow the default low-noise observability protocol: use compact `watch` with a bounded 60000 ms timeout. If it times out unchanged while work remains active, send at most one aggregate one-line heartbeat and only when no other meaningful commentary was sent during that interval; do not repeat the roster or models. Use `--full` only for diagnosis. A different user-visible cadence requires the user's explicit preference. Internal health checks still return early for completion or required attention. Host tool execution limits may shorten a wait; native/Paseo state checks require the host bridge and may wake the coordinator. This setup does not claim zero total coordination overhead. Sources: [official installation](https://github.com/can1357/oh-my-pi#install), installed 18.1.17 `--help`, `auth-broker list --json`, `config` help, and [RPC contract](https://github.com/can1357/oh-my-pi/blob/main/docs/rpc.md). diff --git a/skills/delegate-kit/references/providers.md b/skills/delegate-kit/references/providers.md index 90ad44a..220ea8b 100644 --- a/skills/delegate-kit/references/providers.md +++ b/skills/delegate-kit/references/providers.md @@ -57,7 +57,7 @@ The adapter requires the documented `createAgentSessionServices`, `createAgentSe The RPC client correlates responses by ID and command. Before prompt it calls exact `set_model`, disables auto-retry and auto-compaction, sets optional reasoning, and verifies state without accepting clamped reasoning. Prompt acceptance is separate from terminal `agent_end`. Assistant message metadata can establish actual provider/model; otherwise identity is unknown. Session ID/file are saved before and after the turn. After the completed turn the idle Pi process is terminated; later continuation reopens its exact transcript. LF alone separates JSONL, preserving U+2028/U+2029 and split UTF-8 sequences. -Sources: [Pi RPC](https://github.com/earendil-works/pi/blob/main/packages/coding-agent/docs/rpc.md), [CLI options and tools](https://github.com/earendil-works/pi/blob/main/packages/coding-agent/README.md), [SDK settings/runtime](https://github.com/earendil-works/pi/blob/main/packages/coding-agent/docs/sdk.md). Covered by deterministic protocol, SDK-bootstrap fixtures and fake-process tests, including unchanged global settings; Pi is not installed for a local handshake/live run. +Sources: [Pi RPC](https://github.com/earendil-works/pi/blob/main/packages/coding-agent/docs/rpc.md), [CLI options and tools](https://github.com/earendil-works/pi/blob/main/packages/coding-agent/README.md), [SDK settings/runtime](https://github.com/earendil-works/pi/blob/main/packages/coding-agent/docs/sdk.md). The implementation was reviewed against these contracts; this repository does not yet contain a deterministic SDK-bootstrap/fake-process suite. Pi is not installed for a local handshake/live run. ### OMP @@ -67,10 +67,10 @@ CLI: `omp --mode rpc --provider PROVIDER --model MODEL --no-extensions --no-skil Display-only extension UI notifications are ignored. A blocking dialog or host-owned tool request cannot be silently approved; it stops the run for explicit resolution. Unexpected subagent/fallback/retry events fail rather than hiding extra work. These controls cover the documented execution surface, not arbitrary host/provider internals or external account activity. -Sources: [OMP RPC framing and lifecycle](https://github.com/can1357/oh-my-pi/blob/main/docs/rpc.md), [OMP settings schema](https://github.com/can1357/oh-my-pi/blob/main/packages/coding-agent/src/config/settings-schema.ts), installed 18.1.17 help. Local startup, v2 negotiation, get_state, exact set_model, disabling retry/compaction and EOF passed with no prompt/model call. Completion, continuation and failures use deterministic fixtures; live provider calls remain unverified. +Sources: [OMP RPC framing and lifecycle](https://github.com/can1357/oh-my-pi/blob/main/docs/rpc.md), [OMP settings schema](https://github.com/can1357/oh-my-pi/blob/main/packages/coding-agent/src/config/settings-schema.ts), installed 18.1.17 help. Local startup, v2 negotiation, get_state, exact set_model, disabling retry/compaction and EOF passed with no prompt/model call. Completion, continuation and failure paths were reviewed against the protocol, but this repository does not yet contain the claimed deterministic RPC fixture suite; live provider calls remain unverified. ### Host bridges -See [hosts.md](hosts.md) for actual host-call boundaries and [compatibility matrix](compatibility.md). Native access must be established by the host, not the profile text. Paseo maps a Delegate Kit profile into create_agent settings without a second team configuration; it preserves daemon and workspace ownership. Both bridge paths are fixture-tested, not live-certified. No cloud shell or native OMP/Pi/T3 integration is implied. +See [hosts.md](hosts.md) for actual host-call boundaries and [compatibility matrix](compatibility.md). Native access must be established by the host, not the profile text. Paseo maps a Delegate Kit profile into create_agent settings without a second team configuration; it preserves daemon and workspace ownership. Bridge paths require current-host verification; this repository does not contain full bridge fixture coverage. No cloud shell or native OMP/Pi/T3 integration is implied. OpenCode variants require a verified `transport: "cli"` capability entry with its harness, observed version, exact combined model ID and supported `reasoning` list. Pass this ephemeral discovery evidence with `prepare --capabilities`; an unknown variant is refused before launch. For auto routing with an incompatible host entry, CLI fallback additionally needs explicit `cli_equivalent: true` evidence for the same account/provider, or an explicit user-authorized `transport: "cli"` selection. Capability evidence cannot authorize another provider. diff --git a/skills/delegate-kit/references/review.md b/skills/delegate-kit/references/review.md index 17afb13..9f3a5ca 100644 --- a/skills/delegate-kit/references/review.md +++ b/skills/delegate-kit/references/review.md @@ -4,7 +4,7 @@ Review substantial delegated implementation and changes whose failure modes just Every initial reviewer gets a fresh read-only context, the same frozen specification and diff, and no other reviewer's findings. Freshness and family diversity are different properties. Two agents of the same model can provide independent analysis; agreement alone is not proof. -A profile's `review.also_run` is a required set, validated and reserved by `prepare`. Dispatch all returned runs and keep their initial results independent. A missing/unavailable mandatory reviewer makes the set incomplete. Optional extra coverage is a coordinator decision within user limits. A review lead can help resolve difficult decomposition; it is never an automatic prelude. +A read-only reviewer profile's `review.also_run` is a required reviewer set over the same already-frozen specification and diff, validated and reserved by `prepare`. Dispatch all returned runs and keep their initial results independent. It does not encode an implementer-to-reviewer dependency: prepare review only after the implementation checkpoint exists. A missing/unavailable mandatory reviewer makes the set incomplete. Optional extra coverage is a coordinator decision within user limits. A review lead can help resolve difficult decomposition; it is never an automatic prelude. ## Lenses diff --git a/skills/delegate-kit/references/routing.md b/skills/delegate-kit/references/routing.md index 6608ddf..2448deb 100644 --- a/skills/delegate-kit/references/routing.md +++ b/skills/delegate-kit/references/routing.md @@ -9,6 +9,8 @@ The coordinator selects a profile semantically from the active catalog. Code val catalog --session codex:CHAT_ID prepare --session codex:CHAT_ID --task feature-a --agent research-general --brief /abs/brief.md --cwd /abs/repository run RUN_ID + overview --session codex:CHAT_ID --task feature-a + watch --session codex:CHAT_ID --task feature-a --after CURSOR --timeout-ms 60000 [--full] wait RUN_ID --timeout-ms 60000 result RUN_ID resume RUN_ID --brief /abs/follow-up.md @@ -38,11 +40,11 @@ A timeout of `wait` returns `wait_timed_out: true` with current state and leaves Each CLI supervisor writes a local heartbeat every five seconds. Status checks verify process identity; a missing supervisor becomes orphaned, and a heartbeat older than 30 seconds requests diagnosis even if the PID still exists. Five minutes without output/progress also sets `health.attention_required`; adjust that diagnostic interval per run with `prepare --stall-ms N` for known long operations. This threshold requests investigation; it does not kill a process or release its lease. Host routes require a real status observation at least once per minute while waiting, and repeated unchanged observations do not reset the progress timer. -`wait` returns early for required attention. On an ordinary wait timeout, inspect returned health and the current host/process state before another bounded wait. On a no-progress alert, inspect bounded logs and the current operation; either document why more time is warranted or interrupt/recover a confirmed stall. Repeating an unchanged wait indefinitely is not a recovery strategy. Status polling and heartbeat files perform no model calls. The coordinator must remain active to perform host probes and decide recovery; an exited parent chat cannot be awakened by this local library. +`watch` aggregates the latest attempt in each continuation lineage for one session/task. Its default result is compact and includes display icons, counts and only the agent fields needed for routine reporting; `--full` returns the diagnostic overview. It returns on a meaningful lifecycle, health, confirmed-model or acceptance change, and otherwise times out with an unchanged snapshot. `wait` returns early for required attention on one run. On an ordinary timeout, inspect returned health and the current host/process state before another bounded wait. On a no-progress alert, inspect bounded logs and the current operation; either document why more time is warranted or interrupt/recover a confirmed stall. Repeating an unchanged wait indefinitely is not a recovery strategy. Status polling, `overview`, `watch` and heartbeat files perform no model calls. The coordinator must remain active to perform host probes, report status and decide recovery; an exited parent chat cannot be awakened by this local library. Native dispatch failures and late IDs are handled through [host reconciliation](hosts.md#dispatch-and-events); uncertain outcomes retain ownership. A confirmed not-started dispatch can release capacity without inventing an agent ID. -`cancel` preserves work and retains ownership until the process group stops. If a supervisor disappears, status becomes orphaned; inspect logs and run `cancel` or `recover` when process identity/termination is established. PID birth checks prevent signalling an unrelated reused PID. Uncertain live descendants retain the lease for manual diagnosis. A stale operation mutex is a visible diagnostic; confirm the owning operation stopped before removing it. +`cancel` preserves work and retains ownership until the process group stops. If a supervisor disappears, status becomes orphaned; inspect logs and run `cancel` or `recover` when process identity/termination is established. PID birth checks prevent signalling an unrelated reused PID. If the supervisor died while child registration was pending, ordinary recovery refuses to release ownership because the child PID is unknown. After explicit process inspection proves no worker started, use `recover ID --confirmed-not-started --evidence FILE`; the evidence is retained with the run. Uncertain live descendants retain the lease for manual diagnosis. A stale operation mutex is a visible diagnostic; confirm the owning operation stopped before removing it. Local writers require a linked worktree and use the existing `delegate-kit.lock`. The v2 runtime owns its lease; `agent-wt release/remove` cannot clear an active v2 lease. A Paseo workspace has a daemon-scoped lease and remains owned by Paseo; local cleanup never removes it. diff --git a/skills/delegate-kit/references/setup.md b/skills/delegate-kit/references/setup.md index 18d6d6e..d24d3d6 100644 --- a/skills/delegate-kit/references/setup.md +++ b/skills/delegate-kit/references/setup.md @@ -4,7 +4,7 @@ Read this for start/help/create/copy/edit, not every delegation. The current cha 1. Run `node /scripts/dk.mjs doctor`. It checks versions without model calls. Describe installed, authorized, supported and live-tested separately. Discover exact models and reasoning through the configured executor or actual host model tools. Do not read secrets into the conversation. 2. Ask for the missing preset ID and authorized execution routes. Offer a small team and discuss specialists' general purposes. Ask a small group of questions at a time. Avoid ranking models or treating an installed CLI as consent to use an account. -3. For each profile capture role, `when`, exact executor and optional persistent instructions. Explain access/tool limitations. Additional specialists use different IDs inside the same JSON. An optional planner is for independent analysis, not the host's plan mode. For a required second reviewer add `review.also_run` references within this preset. +3. For each profile capture role, `when`, exact executor and optional persistent instructions. Explain access/tool limitations. Additional specialists use different IDs inside the same JSON. An optional planner is for independent analysis, not the host's plan mode. For a required second reviewer, add `review.also_run` only to a read-only reviewer profile; every member receives the same already-frozen material. Implementer-to-reviewer sequencing is coordinated after implementation, not encoded as `also_run`. 4. Show the resulting team or a meaningful edit diff. Once confirmed or already explicitly requested, write a temporary complete JSON, validate, then save with the runtime. Ask about default only if its change was not part of the request. Do not require the human to edit JSON. 5. Return the saved ID and a natural next request, e.g. “Use Delegate Kit with Y2 for this task.” Run a live model smoke test only if authorized. diff --git a/skills/delegate-kit/scripts/agent-run b/skills/delegate-kit/scripts/agent-run index 85906aa..8fbc669 100755 --- a/skills/delegate-kit/scripts/agent-run +++ b/skills/delegate-kit/scripts/agent-run @@ -222,7 +222,13 @@ function acquireCapsMutex(timeoutMs = 15_000) { for (;;) { try { fs.writeFileSync(CAPS_MUTEX, String(process.pid), { flag: "wx" }); capsMutexHeld = true; return; } catch (e) { if (e.code !== "EEXIST") throw e; } let holder; try { holder = Number(fs.readFileSync(CAPS_MUTEX, "utf8").trim() || 0); } catch (e) { if (e.code === "ENOENT") continue; throw e; } - if (holder && !alive(holder)) { try { fs.rmSync(CAPS_MUTEX, { force: true }); } catch {} continue; } + // A competing atomic create may be visible just before its PID write. + // Empty ownership is ambiguous until the bounded acquisition window ends. + if (!holder) { + if (Date.now() > deadline) die(`stale or invalid admission lock ${CAPS_MUTEX}; verify no operation is running, then remove it explicitly before retrying`); + Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, 50); continue; + } + if (!alive(holder)) die(`stale or invalid admission lock ${CAPS_MUTEX}; verify no operation is running, then remove it explicitly before retrying`); if (Date.now() > deadline) die(`another agent-run (pid ${holder}) has held ${CAPS_MUTEX} for ${timeoutMs / 1000}s; if it is stuck, kill it or remove the file`); Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, 50); } @@ -242,14 +248,25 @@ function acquireRepoMutex(cwd, timeoutMs = 15_000) { for (;;) { try { fs.mkdirSync(dir); fs.writeFileSync(path.join(dir, "pid"), String(process.pid)); repoMutexHeld = dir; return; } catch (e) { if (e.code !== "EEXIST") throw e; } let holder = 0; try { holder = Number(fs.readFileSync(path.join(dir, "pid"), "utf8").trim() || 0); } catch {} - if (holder && !alive(holder)) { try { fs.rmSync(dir, { recursive: true, force: true }); } catch {} continue; } + if (!holder) { + if (Date.now() > deadline) die(`stale or invalid repository admission lock ${dir}; verify no operation is running, then remove it explicitly before retrying`); + Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, 50); continue; + } + if (!alive(holder)) die(`stale or invalid repository admission lock ${dir}; verify no operation is running, then remove it explicitly before retrying`); if (Date.now() > deadline) die(`another agent-run or agent-wt (pid ${holder}) has held ${dir} for ${timeoutMs / 1000}s; if it is stuck, kill it or remove the directory`); Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, 50); } } function releaseCapsMutex() { - if (repoMutexHeld) { const d = repoMutexHeld; repoMutexHeld = null; try { fs.rmSync(d, { recursive: true, force: true }); } catch {} } - if (!capsMutexHeld) return; capsMutexHeld = false; try { fs.rmSync(CAPS_MUTEX, { force: true }); } catch {} + if (repoMutexHeld) { + const d = repoMutexHeld; repoMutexHeld = null; + let owner = 0; try { owner = Number(fs.readFileSync(path.join(d, 'pid'), 'utf8')); } catch {} + if (owner === process.pid) try { fs.rmSync(d, { recursive: true, force: true }); } catch {} + } + if (!capsMutexHeld) return; + capsMutexHeld = false; + let owner = 0; try { owner = Number(fs.readFileSync(CAPS_MUTEX, 'utf8')); } catch {} + if (owner === process.pid) try { fs.rmSync(CAPS_MUTEX, { force: true }); } catch {} } // Refuse before anything is spawned. Native locks (`agent-wt lock`) of the repository of diff --git a/skills/delegate-kit/scripts/agent-wt b/skills/delegate-kit/scripts/agent-wt index 98cf5b6..7c4a323 100755 --- a/skills/delegate-kit/scripts/agent-wt +++ b/skills/delegate-kit/scripts/agent-wt @@ -23,8 +23,8 @@ STATE_DIR="${DELEGATE_KIT_HOME:-$HOME/.delegate-kit}" CAPS_MUTEX="" GLOBAL_CAPS_MUTEX="" release_caps_mutex() { - [ -n "$CAPS_MUTEX" ] && rm -rf "$CAPS_MUTEX"; CAPS_MUTEX="" - [ -n "$GLOBAL_CAPS_MUTEX" ] && rm -f "$GLOBAL_CAPS_MUTEX"; GLOBAL_CAPS_MUTEX="" + if [ -n "$CAPS_MUTEX" ] && [ "$(cat "$CAPS_MUTEX/pid" 2>/dev/null || true)" = "$$" ]; then rm -rf "$CAPS_MUTEX"; fi; CAPS_MUTEX="" + if [ -n "$GLOBAL_CAPS_MUTEX" ] && [ "$(cat "$GLOBAL_CAPS_MUTEX" 2>/dev/null || true)" = "$$" ]; then rm -f "$GLOBAL_CAPS_MUTEX"; fi; GLOBAL_CAPS_MUTEX="" } trap release_caps_mutex EXIT die() { echo "agent-wt: $*" >&2; exit 1; } @@ -117,9 +117,12 @@ acquire_global_caps_mutex() { for _ in $(seq 1 300); do if (set -o noclobber; echo "$$" > "$file") 2>/dev/null; then GLOBAL_CAPS_MUTEX="$file"; return; fi pid=$(cat "$file" 2>/dev/null || true) - if [ -n "$pid" ] && ! kill -0 "$pid" 2>/dev/null; then rm -f "$file"; continue; fi + # noclobber may expose the new file just before its PID payload is visible. + [ -z "$pid" ] && { sleep 0.05; continue; } + if ! kill -0 "$pid" 2>/dev/null; then die "stale or invalid admission lock $file; verify no operation is running, then remove it explicitly before retrying"; fi sleep 0.05 done + [ -n "${pid:-}" ] || die "stale or invalid admission lock $file; verify no operation is running, then remove it explicitly before retrying" die "another worker has held $file for 15s; verify the owner before removing a stale lock" } known_workers() { @@ -151,9 +154,11 @@ acquire_caps_mutex() { for _ in $(seq 1 300); do if mkdir "$d" 2>/dev/null; then echo "$$" > "$d/pid"; CAPS_MUTEX="$d"; return; fi local pid; pid=$(cat "$d/pid" 2>/dev/null || true) - if [ -n "$pid" ] && ! kill -0 "$pid" 2>/dev/null; then rm -rf "$d"; continue; fi + [ -z "$pid" ] && { sleep 0.05; continue; } + if ! kill -0 "$pid" 2>/dev/null; then die "stale or invalid repository admission lock $d; verify no operation is running, then remove it explicitly before retrying"; fi sleep 0.05 done + [ -n "${pid:-}" ] || die "stale or invalid repository admission lock $d; verify no operation is running, then remove it explicitly before retrying" die "another agent-wt (pid $pid) has held $d for 15s; if it is stuck, kill it or remove the directory" } cmd_lock() { diff --git a/skills/delegate-kit/scripts/dk.mjs b/skills/delegate-kit/scripts/dk.mjs index dbb934c..397ee27 100644 --- a/skills/delegate-kit/scripts/dk.mjs +++ b/skills/delegate-kit/scripts/dk.mjs @@ -5,11 +5,11 @@ import { fileURLToPath } from 'node:url'; import { home, check, readJSON, validatePreset, presetFiles, loadPreset, savePreset, copyPreset, setDefault, context, catalog } from './presets.mjs'; import { discover } from './executors.mjs'; import { migrate } from './migrate.mjs'; -import { prepare, launch, attach, ingest, resume, status, wait, cancel, recover, accept, supervise, getRun, dispatchFailed } from './runtime.mjs'; +import { prepare, launch, attach, ingest, resume, status, overview, watch, wait, cancel, recover, accept, supervise, getRun, dispatchFailed } from './runtime.mjs'; function args(input) { const o = { _: [] }; - const booleans = ['task-only', 'dry-run', 'apply', 'stopped', 'confirmed-not-started']; + const booleans = ['task-only', 'dry-run', 'apply', 'stopped', 'confirmed-not-started', 'full']; for (let i = 0; i < input.length; i++) { if (!input[i].startsWith('--')) { o._.push(input[i]); continue; } const key = input[i].slice(2); check(!Object.hasOwn(o, key), `Duplicate --${key}`); @@ -21,7 +21,7 @@ function args(input) { const optKeys = { context: ['session', 'preset', 'task-only'], catalog: ['session', 'preset', 'task-only', 'role'], presets: ['file', 'revision'], prepare: ['session', 'preset', 'task-only', 'task', 'agent', 'role', 'brief', 'cwd', 'capabilities', 'workspace', 'timeout-ms', 'stall-ms', 'max-workers', 'max-writers', 'max-runs', 'max-retries'], - run: [], attach: ['host-agent', 'workspace-id'], status: [], result: [], wait: ['timeout-ms'], resume: ['brief'], cancel: [], recover: [], accept: [], + run: [], attach: ['host-agent', 'workspace-id'], status: [], result: [], overview: ['session', 'task'], watch: ['session', 'task', 'after', 'timeout-ms', 'full'], wait: ['timeout-ms'], resume: ['brief'], cancel: [], recover: ['confirmed-not-started', 'evidence'], accept: [], 'dispatch-failed': ['dispatch-token', 'confirmed-not-started', 'evidence'], event: ['host-agent', 'event', 'file', 'stopped', 'dispatch-token', 'progress'], doctor: [], migrate: ['decisions', 'dry-run', 'apply'], materialize: ['directory'], _supervise: ['claim'], help: [], start: [], @@ -60,10 +60,12 @@ export async function main(input = process.argv.slice(2)) { case 'dispatch-failed': return dispatchFailed(action, { dispatchToken: o['dispatch-token'], confirmedNotStarted: o['confirmed-not-started'], evidence: o.evidence ? fs.readFileSync(o.evidence, 'utf8') : undefined }); case 'event': return ingest(action, { hostAgent: o['host-agent'], event: o.event, result: o.file ? readJSON(o.file) : undefined, stopped: o.stopped, dispatchToken: o['dispatch-token'], progress: o.progress }); case 'status': case 'result': return status(action); + case 'overview': return overview({ session: o.session, task: o.task }); + case 'watch': return watch({ session: o.session, task: o.task, after: o.after, milliseconds: ms, full: o.full === true }); case 'wait': return wait(action, ms); case 'resume': return resume(action, o.brief); case 'cancel': return cancel(action); - case 'recover': return recover(action); + case 'recover': return recover(action, { confirmedNotStarted: o['confirmed-not-started'], evidence: o.evidence ? fs.readFileSync(o.evidence, 'utf8') : undefined }); case 'accept': return accept(action); case 'doctor': return { executors: discover(), root: home(), note: 'No model calls made. Installed/version is not proof of authorization. See references/providers.md.' }; case 'migrate': check(!(o.apply && o['dry-run']), 'Choose --apply or --dry-run'); return migrate(o.decisions ? readJSON(o.decisions) : {}, o.apply === true); @@ -84,7 +86,8 @@ export async function main(input = process.argv.slice(2)) { 'context open [--session HOST:ID] [--preset X1] [--task-only]', 'catalog --session HOST:ID [--role ROLE]', 'prepare --session HOST:ID --task ID --agent PROFILE --brief FILE [--cwd WORKTREE] [--capabilities FILE] [--workspace FILE]', 'run ID', 'attach ID --host-agent ID [--workspace-id ID]', 'event ID --host-agent ID --dispatch-token TOKEN --event complete --file RESULT --stopped', - 'dispatch-failed ID --dispatch-token TOKEN --confirmed-not-started --evidence FILE', 'status|result|wait|cancel|recover|accept ID', 'resume ID --brief FILE', 'migrate --dry-run [--decisions FILE]'], + 'dispatch-failed ID --dispatch-token TOKEN --confirmed-not-started --evidence FILE', 'overview --session HOST:ID [--task ID]', + 'watch --session HOST:ID [--task ID] [--after CURSOR] [--timeout-ms 60000] [--full]', 'status|result|wait|cancel|accept ID', 'recover ID [--confirmed-not-started --evidence FILE]', 'resume ID --brief FILE', 'migrate --dry-run [--decisions FILE]'], root: home(), note: 'Invoke this script by its installed path. No global dk command is installed. Legacy runs use agent-run.' }; } } diff --git a/skills/delegate-kit/scripts/executors.mjs b/skills/delegate-kit/scripts/executors.mjs index 485fe7a..ef4a9d5 100644 --- a/skills/delegate-kit/scripts/executors.mjs +++ b/skills/delegate-kit/scripts/executors.mjs @@ -41,6 +41,7 @@ export function resolveExecutor(agent, capabilities = [], available = () => true check(host.transport !== 'native' || host.host !== 'claude' || host.dynamic_roles === true, 'Claude native requires verified discovery of per-run role definitions; use CLI if a restart is required'); check(host.launch_provider === undefined || host.launch_provider === e.harness, 'Paseo launch provider cannot replace the selected harness'); check(host.transport !== 'paseo' || host.daemon, 'Paseo capability evidence needs a stable daemon identifier'); + check(host.transport !== 'paseo' || typeof host.mode_ids?.[access] === 'string' && host.mode_ids[access].length > 0, `Paseo capability evidence needs a mode enforcing ${access}`); return { ...e, model: e.inherit_model ? host.current_model : e.model, transport: host.transport, access, capability: host, actual_model: null }; } check(desired === 'auto' || desired === 'cli', `${desired} cannot preserve ${e.harness} model/provider/reasoning/access; supply verified host capabilities or choose CLI explicitly`); diff --git a/skills/delegate-kit/scripts/migrate.mjs b/skills/delegate-kit/scripts/migrate.mjs index 69eecd5..dfd0af5 100644 --- a/skills/delegate-kit/scripts/migrate.mjs +++ b/skills/delegate-kit/scripts/migrate.mjs @@ -1,6 +1,6 @@ import fs from 'node:fs'; import path from 'node:path'; -import { readConfig, backendTable } from './routing.mjs'; +import { readConfig, backendTable, resolve as resolveLegacy } from './routing.mjs'; import { home, hash, check, locked, atomicJSON, readJSON, validatePreset, presetFiles } from './presets.mjs'; // Parent-dependent assignments are resolved only from an explicit migration @@ -17,8 +17,26 @@ export function migrationPlan(decisions = {}, root = home()) { const assignments = { ...legacy.roles, ...profile.roles }; for (const [role, assignment] of Object.entries(assignments)) { const candidateKeys = ['model', 'effort', 'family', 'runner', 'backend', 'efforts']; - const candidates = Array.isArray(assignment) ? assignment : candidateKeys.some(key => Object.hasOwn(assignment, key)) - ? [assignment] : Object.entries(assignment).map(([backend, pair]) => ({ backend, model: pair[0], effort: pair[1] || undefined })); + const backendMap = !Array.isArray(assignment) && !candidateKeys.some(key => Object.hasOwn(assignment, key)); + if (backendMap && !parent) { issues.push(`${id}.${role}: legacy backend map needs an explicit parents.${id} decision before choosing its default`); continue; } + const needsAuthor = ['reviewer', 'verifier'].includes(role); + const author = decisions.authors?.[id]; + if (backendMap && needsAuthor && !author) { issues.push(`${id}.${role}: legacy reviewer routing needs an explicit authors.${id} backend decision`); continue; } + let selectedBackend = parent, selectedRoute = null; + if (backendMap) { + try { + selectedRoute = resolveLegacy({ parent, role, ...(needsAuthor ? { 'author-backend': author } : {}), ...(legacy.profiles?.[id] ? { profile: id } : {}) }, legacy, {}, () => true); + selectedBackend = selectedRoute.backend; + } catch (error) { issues.push(`${id}.${role}: cannot preserve legacy default: ${error.message}`); continue; } + } + let candidates = Array.isArray(assignment) ? assignment : !backendMap ? [assignment] + : Object.entries(assignment).map(([backend, pair]) => ({ backend, model: pair[0], effort: pair[1] || undefined })); + if (backendMap) { + if (!candidates.some(candidate => candidate.backend === selectedBackend)) { + candidates.push({ backend: selectedBackend, model: selectedRoute.model, effort: selectedRoute.effort || undefined }); + } + candidates.sort((a, b) => Number(b.backend === selectedBackend) - Number(a.backend === selectedBackend)); + } candidates.forEach((c, i) => { const at = `${id}.${role}[${i + 1}]`; const backend = c.backend || (c.family ? Object.keys(table).find(b => table[b].family === c.family && (!c.runner || ['auto', 'native'].includes(c.runner) || table[b].adapter === c.runner)) : parent); diff --git a/skills/delegate-kit/scripts/presets.mjs b/skills/delegate-kit/scripts/presets.mjs index 44550d5..865fb2f 100644 --- a/skills/delegate-kit/scripts/presets.mjs +++ b/skills/delegate-kit/scripts/presets.mjs @@ -85,6 +85,7 @@ export function validatePreset(preset) { check(target !== id && Object.hasOwn(preset.agents, target), `${at}.review: invalid reference ${target}`); check(preset.agents[target].role === 'reviewer' && accessOf(preset.agents[target]) === 'read-only', `${at}.review: ${target} must be a read-only reviewer`); } + check(agent.role === 'reviewer' && accessOf(agent) === 'read-only', `${at}.review.also_run is only for a read-only reviewer set over the same frozen material`); } } if (preset.defaults !== undefined) { diff --git a/skills/delegate-kit/scripts/runtime.mjs b/skills/delegate-kit/scripts/runtime.mjs index be1f2f5..78a2deb 100644 --- a/skills/delegate-kit/scripts/runtime.mjs +++ b/skills/delegate-kit/scripts/runtime.mjs @@ -16,6 +16,8 @@ const schema = readJSON(path.join(skill, 'references/result-schema.json')); const active = ['prepared', 'starting', 'running', 'permission', 'cancelling', 'orphaned']; const terminal = ['finished', 'failed', 'cancelled', 'timeout', 'blocked']; const now = () => new Date().toISOString(); +const lockTimeout = () => process.env.NODE_ENV === 'test' && Number.isSafeInteger(Number(process.env.DELEGATE_KIT_TEST_LOCK_TIMEOUT_MS)) && Number(process.env.DELEGATE_KIT_TEST_LOCK_TIMEOUT_MS) > 0 + ? Number(process.env.DELEGATE_KIT_TEST_LOCK_TIMEOUT_MS) : 15000; function diagnostic(message) { let text = String(message); for (const [key, value] of Object.entries(process.env)) { @@ -58,7 +60,7 @@ function admission(fn) { fs.mkdirSync(home(), { recursive: true, mode: 0o700 }); const mutex = path.join(home(), 'caps.lock'), temp = `${mutex}.${randomUUID()}`; fs.writeFileSync(temp, String(process.pid), { mode: 0o600 }); - const deadline = Date.now() + 15000; + const deadline = Date.now() + lockTimeout(); try { for (;;) { try { fs.linkSync(temp, mutex); break; } @@ -66,14 +68,63 @@ function admission(fn) { if (error.code !== 'EEXIST') throw error; let pid; try { pid = Number(fs.readFileSync(mutex, 'utf8')); } catch (e) { if (e.code === 'ENOENT') continue; throw e; } - if (pid && !alive(pid)) { fs.rmSync(mutex, { force: true }); continue; } + if (!pid) { + check(Date.now() < deadline, `Admission lock has no published owner: ${mutex}; verify no operation is running, then remove it explicitly before retrying`); + Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, 25); continue; + } + check(alive(pid), `Admission lock is stale or invalid: ${mutex}; verify no operation is running, then remove it explicitly before retrying`); check(Date.now() < deadline, 'Admission lock held; wait or verify its owner before recovery'); Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, 25); } } - try { return fn(); } finally { fs.rmSync(mutex, { force: true }); } + try { return fn(); } finally { + let owner = null; try { owner = Number(fs.readFileSync(mutex, 'utf8')); } catch {} + if (owner === process.pid) fs.rmSync(mutex, { force: true }); + } } finally { fs.rmSync(temp, { force: true }); } } + +// Writer ownership is repository-global, while DELEGATE_KIT_HOME may differ +// between coordinators. The path, mkdir protocol and lock order are shared with +// agent-run and agent-wt: state mutex first, repository second. Readers tolerate +// the bounded mkdir-to-PID publication window but never steal an unknown owner. +function repositoryAdmission(cwd, fn) { + if (!cwd) return fn(); + const common = git(cwd, ['rev-parse', '--git-common-dir']); + check(common.status === 0, 'Cannot resolve repository admission lock'); + const target = path.join(path.resolve(cwd, common.stdout.trim()), 'delegate-kit.caps.lock'); + const deadline = Date.now() + lockTimeout(); + let created = false, owned = false; + try { + for (;;) { + try { + fs.mkdirSync(target, { mode: 0o700 }); + created = true; + fs.writeFileSync(path.join(target, 'pid'), String(process.pid), { mode: 0o600 }); + owned = true; break; + } + catch (error) { + if (error.code !== 'EEXIST') throw error; + let holder = 0; + try { holder = Number(fs.readFileSync(path.join(target, 'pid'), 'utf8')); } catch (e) { if (e.code !== 'ENOENT') throw e; } + if (!holder) { + check(Date.now() < deadline, `Repository admission lock has no published owner: ${target}; verify no operation is running, then remove it explicitly before retrying`); + } else { + check(alive(holder), `Repository admission lock is stale: ${target}; verify no operation is running, then remove it explicitly before retrying`); + check(Date.now() < deadline, `Repository admission lock held by ${holder}; wait or verify its owner before recovery`); + } + Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, 25); + } + } + try { return fn(); } finally { + let owner = 0; try { owner = Number(fs.readFileSync(path.join(target, 'pid'), 'utf8')); } catch {} + if (owned && owner === process.pid) fs.rmSync(target, { recursive: true, force: true }); + } + } catch (error) { + if (created && !owned) fs.rmSync(target, { recursive: true, force: true }); + throw error; + } +} function workspace(cwd, write, external) { if (external) { check(external.owner === 'paseo' && external.id && external.daemon, 'Paseo workspace requires id and daemon'); @@ -147,41 +198,42 @@ export function prepare(options) { return { schema_version: 2, id: run, parent_session: options.session, task: options.task, budget_task: hash(`${options.session}\0${options.task}`), preset: p.id, preset_hash: selected.revision, profile, agent, executor, workspace: ws, cwd: ws.path, write: accessOf(agent) === 'workspace-write', role: agent.role, - limits: caps, timeout_ms: options.timeoutMs || null, stall_ms: options.stallMs ?? 300000, group, required_profiles: ids, status: 'prepared', created: now(), resume_of: null, - host_attached: false, transport_session_id: null, transport_session_file: null, actual_model: null, usage: null, cost_usd: null, + limits: caps, timeout_ms: options.timeoutMs || null, stall_ms: options.stallMs ?? 300000, group, required_profiles: ids, status: 'prepared', created: now(), resume_of: null, attempt_sequence: 1, + host_attached: false, execution_started_at: null, launch_registration_pending: false, transport_session_id: null, transport_session_file: null, actual_model: null, usage: null, cost_usd: null, result_validated: false, accepted: false, attempt_kind: 'fresh', pid: null }; }); - return admission(() => { + return admission(() => repositoryAdmission(rows.find(m => m.write && m.workspace.owner === 'delegate-kit')?.cwd, () => { enforceCaps(caps, counts(rows[0].workspace.owner === 'delegate-kit' ? rows[0].cwd : null), { workers: rows.length, writers: rows.filter(m => m.write).length }); const before = budget({ stateDir: home(), task: rows[0].budget_task, limits: caps }); check(caps.runs === null || before.runs + rows.length <= caps.runs, `Required review set exceeds max ${caps.runs} runs`); const reserved = []; try { + // Burn attempt budget before any dispatchable metadata is published. A + // crash may conservatively consume an attempt, but can never create an + // uncounted prepared run that launch would accept. + for (const m of rows) m.budget = budget({ stateDir: home(), task: m.budget_task, record: true, limits: caps }); for (const m of rows) { reserveWriter(m); reserved.push(m); atomicJSON(path.join(dir(m.id), 'preset.snapshot.json'), p); fs.writeFileSync(path.join(dir(m.id), 'prompt.md'), promptFor(m, brief), { mode: 0o600 }); save(m); } - // Reservations count once, including failed dispatch. Cancel releases capacity, - // but not the attempt limit; a crash can never create a free retry. - for (const m of rows) { - m.budget = budget({ stateDir: home(), task: m.budget_task, record: true, limits: caps }); save(m); - } } catch (error) { for (const m of reserved) { m.status = 'failed'; m.error = error.message; save(m); releaseWriter(m); } throw error; } return { group, runs: rows.map(compact) }; - }); + })); } export function compact(m) { const { capability, ...executor } = m.executor; - return { id: m.id, parent_session: m.parent_session, task: m.task, preset: m.preset, profile: m.profile, - executor: { ...executor, ...(capability ? { host: capability.host, version: capability.version } : {}) }, status: m.status, attempt_kind: m.attempt_kind, resume_of: m.resume_of, + return { id: m.id, parent_session: m.parent_session, task: m.task, preset: m.preset, profile: m.profile, role: m.role, write: m.write, + executor: { ...executor, ...(capability ? { host: capability.host, version: capability.version } : {}) }, status: m.status, attempt_kind: m.attempt_kind, resume_of: m.resume_of, attempt_sequence: m.attempt_sequence || null, dispatch_token: m.claim || null, transport_session_id: m.transport_session_id, workspace: m.workspace, actual_model: m.actual_model, + launch_registration_pending: Boolean(m.launch_registration_pending), result_validated: m.result_validated, accepted: m.accepted, result: m.result || null, error: m.error || null, - group: m.group, required_profiles: m.required_profiles, logs: dir(m.id), usage: m.usage, cost_usd: m.cost_usd }; + group: m.group, required_profiles: m.required_profiles, created: m.created, started: m.started || null, execution_started: executionStarted(m), execution_started_at: m.execution_started_at || null, finished: m.finished || null, + logs: dir(m.id), usage: m.usage, cost_usd: m.cost_usd }; } function health(m) { if (!active.includes(m.status) || m.status === 'prepared') return { state: 'inactive', attention_required: false }; @@ -206,14 +258,171 @@ function health(m) { ...(idle >= stale ? { action_required: 'Process is alive but output has not advanced. Inspect logs/process activity; choose a justified observation interval or cancel after diagnosis. Do not blindly repeat wait or duplicate the worker.' } : {}) }; } export function status(id) { - return admission(() => { - const m = getRun(id); - if (m.executor.transport === 'cli' && ['running', 'starting', 'cancelling'].includes(m.status) && m.pid && !sameProcess(m.pid, m.pid_fingerprint)) { - m.status = 'orphaned'; m.error = 'Supervisor identity is gone; inspect logs and use recover after verifying process termination'; save(m); + return admission(() => statusUnlocked(id)); +} + +function statusUnlocked(id) { + const m = getRun(id); + if (m.executor.transport === 'cli' && ['running', 'starting', 'cancelling'].includes(m.status) && m.pid && !sameProcess(m.pid, m.pid_fingerprint)) { + m.status = 'orphaned'; m.error = 'Supervisor identity is gone; inspect logs and use recover after verifying process termination'; save(m); + } + return { ...compact(m), health: health(m) }; +} + +function executionStarted(run) { + if (run.execution_started === true) return true; + if (run.execution_started_at) return true; + if (run.executor.transport === 'cli') return Boolean(run.child_pid); + return run.host_attached === true; +} + +function currentAttempts(runs) { + const byId = new Map(runs.map(run => [run.id, run])); + const rootOf = run => { + let current = run, root = run.id; + const seen = new Set([run.id]); + while (current.resume_of && byId.has(current.resume_of) && !seen.has(current.resume_of)) { + root = current.resume_of; seen.add(root); current = byId.get(root); } - return { ...compact(m), health: health(m) }; + return root; + }; + const groups = new Map(); + for (const run of runs) { + const root = rootOf(run); + if (!groups.has(root)) groups.set(root, []); + groups.get(root).push(run); + } + const latest = []; + for (const group of groups.values()) { + const parents = new Set(group.map(run => run.resume_of).filter(id => byId.has(id))); + const leaves = group.filter(run => !parents.has(run.id)); + leaves.sort((a, b) => (Number(a.attempt_sequence) || 0) - (Number(b.attempt_sequence) || 0) || + String(a.created).localeCompare(String(b.created)) || a.id.localeCompare(b.id)); + latest.push(leaves.at(-1)); + } + return latest.sort((a, b) => String(a.created).localeCompare(String(b.created)) || a.id.localeCompare(b.id)); +} + +function stageOf(runs) { + if (!runs.length) return 'idle'; + if (runs.every(run => terminal.includes(run.status))) return 'complete'; + const working = runs.filter(run => executionStarted(run) && ['running', 'permission', 'cancelling', 'orphaned'].includes(run.status)); + if (!working.length) return 'preparation'; + const roles = new Set(working.map(run => run.role)); + if (roles.has('reviewer') || roles.has('verifier') || roles.has('review-lead')) return 'review'; + if (roles.has('implementer')) return 'implementation'; + if (roles.has('planner')) return 'planning'; + if (roles.has('researcher')) return 'research'; + return 'coordination'; +} + +export function overview({ session, task } = {}) { + check(typeof session === 'string' && session.length > 0, 'overview requires --session'); + if (task !== undefined) identifier(task, 'task'); + return admission(() => { + const selected = allRuns().filter(run => run.schema_version === 2 && run.parent_session === session && (task === undefined || run.task === task)); + const attempts = selected.map(run => statusUnlocked(run.id)); + const runs = currentAttempts(attempts); + const statuses = Object.fromEntries([...active, ...terminal].map(name => [name, runs.filter(run => run.status === name).length])); + const activeStates = new Set(active); + const attention = runs.filter(run => run.health?.attention_required || ['permission', 'orphaned', 'blocked', 'failed', 'timeout'].includes(run.status)).length; + const summary = { + agents: runs.length, + attempts: attempts.length, + reserved: statuses.prepared, + started: runs.filter(executionStarted).length, + active: runs.filter(run => activeStates.has(run.status)).length, + working: runs.filter(run => run.status === 'running' && executionStarted(run)).length, + completed: statuses.finished, + accepted: runs.filter(run => run.accepted).length, + attention, + statuses, + }; + const agents = runs.map(run => ({ + id: run.id, + profile: run.profile, + role: run.role, + status: run.status, + health: run.health, + harness: run.executor.harness, + transport: run.executor.transport, + provider: run.executor.provider || null, + requested_model: run.executor.model, + actual_model: run.actual_model || null, + attempt_kind: run.attempt_kind, + attempt_sequence: run.attempt_sequence, + resume_of: run.resume_of, + accepted: run.accepted, + dispatch_started: run.started, + execution_started: executionStarted(run), + execution_started_at: run.execution_started_at, + finished: run.finished, + })); + const cursor = hash(JSON.stringify(agents.map(agent => ({ + id: agent.id, status: agent.status, health: agent.health?.state, attention: agent.health?.attention_required, + actual_model: agent.actual_model, accepted: agent.accepted, execution_started: agent.execution_started, + })))); + const presets = [...new Set(runs.map(run => run.preset))].sort(); + return { session, task: task || null, presets, stage: stageOf(runs), cursor, observed_at: now(), summary, agents }; }); } + +function statusIcon(agent) { + if (agent.health?.attention_required || ['permission', 'orphaned', 'blocked', 'failed', 'timeout'].includes(agent.status)) return '🔴'; + if (agent.status === 'running' && agent.execution_started) return '🟢'; + if (agent.accepted) return '🟢'; + return '🟡'; +} + +function compactSnapshot(snapshot) { + const icon = snapshot.summary.attention > 0 ? '🔴' + : snapshot.summary.working > 0 || (snapshot.summary.agents > 0 && snapshot.summary.accepted === snapshot.summary.agents) ? '🟢' + : '🟡'; + return { + session: snapshot.session, + task: snapshot.task, + stage: snapshot.stage, + cursor: snapshot.cursor, + summary: { + icon, + agents: snapshot.summary.agents, + reserved: snapshot.summary.reserved, + started: snapshot.summary.started, + working: snapshot.summary.working, + completed: snapshot.summary.completed, + accepted: snapshot.summary.accepted, + attention: snapshot.summary.attention, + }, + agents: snapshot.agents.map(agent => ({ + icon: statusIcon(agent), + id: agent.id, + profile: agent.profile, + role: agent.role, + status: agent.status, + health: agent.health?.state || null, + attention: Boolean(agent.health?.attention_required), + actual_model: agent.actual_model, + execution_started: agent.execution_started, + accepted: agent.accepted, + })), + }; +} + +export async function watch({ session, task, after, milliseconds = 60000, full = false } = {}) { + check(Number.isSafeInteger(milliseconds) && milliseconds > 0, 'watch requires a positive timeout'); + const deadline = Date.now() + milliseconds; + for (;;) { + const snapshot = overview({ session, task }); + const result = full ? snapshot : compactSnapshot(snapshot); + if (!after || snapshot.cursor !== after) return { ...result, changed: true }; + if (Date.now() >= deadline) { + if (!full) return { session: result.session, task: result.task, stage: result.stage, cursor: result.cursor, summary: result.summary, changed: false, watch_timed_out: true }; + return { ...result, changed: false, watch_timed_out: true }; + } + await sleep(Math.min(1000, deadline - Date.now())); + } +} + export async function wait(id, milliseconds = 60000) { check(Number.isFinite(milliseconds) && milliseconds > 0, 'wait requires a positive timeout'); const deadline = Date.now() + milliseconds; @@ -253,7 +462,7 @@ export function attach(id, transportId, workspaceId) { check(!m.transport_session_id || m.transport_session_id === transportId, 'Resume must attach the original agent'); if (m.executor.transport === 'paseo') check(workspaceId === m.workspace.id, 'Paseo returned a different workspace; reconcile before proceeding'); check(!allRuns().some(r => r.id !== id && active.includes(r.status) && r.executor?.transport === m.executor.transport && r.executor?.capability?.daemon === m.executor.capability?.daemon && r.transport_session_id === transportId), 'Host agent already belongs to another active attempt'); - m.transport_session_id = transportId; m.host_attached = true; if (m.status !== 'cancelling') m.status = 'running'; m.host_checked_at = now(); m.progress_at ||= m.started; save(m); return compact(m); + m.transport_session_id = transportId; m.host_attached = true; m.execution_started_at ||= now(); if (m.status !== 'cancelling') m.status = 'running'; m.host_checked_at = now(); m.progress_at ||= m.started; save(m); return compact(m); }); } function resultFrom(text) { @@ -315,17 +524,21 @@ export function resume(id, briefFile) { check(!allRuns().some(r => active.includes(r.status) && r.parent_session === prev.parent_session && r.transport_session_id === prev.transport_session_id), 'An attempt already owns this executor session'); if (prev.executor.transport === 'cli' && ['pi', 'omp'].includes(prev.executor.harness)) check(prev.transport_session_file && fs.existsSync(prev.transport_session_file), 'Saved RPC session file is unavailable; do not resume a prefix or last session'); assertWorkspaceBinding(prev.executor, prev.cwd); - enforceCaps(prev.limits, counts(prev.workspace.owner === 'delegate-kit' ? prev.cwd : null), { workers: 1, writers: prev.write ? 1 : 0 }); - const m = { ...prev, id: randomUUID(), status: 'prepared', created: now(), started: null, finished: null, error: null, + const attemptSequence = Math.max(0, ...allRuns().filter(run => run.group === prev.group && run.profile === prev.profile).map(run => Number(run.attempt_sequence) || 0)) + 1; + const m = { ...prev, id: randomUUID(), status: 'prepared', created: now(), started: null, execution_started_at: null, finished: null, error: null, resume_of: id, result: null, result_validated: false, accepted: false, attempt_kind: 'continuation', pid: null, - pid_fingerprint: null, child_pid: null, child_fingerprint: null, usage: null, cost_usd: null, claim: null, invoke: null, host_attached: false, host_checked_at: null, progress_at: null, progress_cursor: null }; - reserveWriter(m); - try { - m.budget = budget({ stateDir: home(), task: m.budget_task, ticket: m.profile, retry: true, record: true, limits: m.limits }); - atomicJSON(path.join(dir(m.id), 'preset.snapshot.json'), readJSON(path.join(dir(id), 'preset.snapshot.json'))); - fs.writeFileSync(path.join(dir(m.id), 'prompt.md'), promptFor(m, brief), { mode: 0o600 }); save(m); - } catch (e) { releaseWriter(m); throw e; } - return compact(m); + attempt_sequence: attemptSequence, pid_fingerprint: null, child_pid: null, child_fingerprint: null, actual_model: null, actual_provider: null, + usage: null, cost_usd: null, claim: null, invoke: null, host_attached: false, host_checked_at: null, progress_at: null, progress_cursor: null, launch_registration_pending: false, launch_recovery_evidence: null }; + return repositoryAdmission(m.write && m.workspace.owner === 'delegate-kit' ? m.cwd : null, () => { + enforceCaps(prev.limits, counts(prev.workspace.owner === 'delegate-kit' ? prev.cwd : null), { workers: 1, writers: prev.write ? 1 : 0 }); + reserveWriter(m); + try { + m.budget = budget({ stateDir: home(), task: m.budget_task, ticket: m.profile, retry: true, record: true, limits: m.limits }); + atomicJSON(path.join(dir(m.id), 'preset.snapshot.json'), readJSON(path.join(dir(id), 'preset.snapshot.json'))); + fs.writeFileSync(path.join(dir(m.id), 'prompt.md'), promptFor(m, brief), { mode: 0o600 }); save(m); + } catch (e) { releaseWriter(m); throw e; } + return compact(m); + }); }); } export async function cancel(id) { @@ -352,10 +565,15 @@ export async function cancel(id) { } return { ...compact(m), action_required: 'Waiting for process termination; ownership is retained' }; } -export function recover(id) { +export function recover(id, { confirmedNotStarted = false, evidence } = {}) { return admission(() => { const m = getRun(id); check(m.executor.transport === 'cli', 'Native recovery requires a correlated host completion event'); + if (m.launch_registration_pending) { + check(confirmedNotStarted === true && typeof evidence === 'string' && evidence.trim(), 'Child registration was interrupted; retain ownership until process inspection confirms no worker started, then provide evidence'); + m.launch_recovery_evidence = diagnostic(evidence); + m.launch_registration_pending = false; + } check(!sameProcess(m.pid, m.pid_fingerprint) && !groupAlive(m.child_pid), 'Supervisor or child group may still be writing; ownership retained'); if (!terminal.includes(m.status)) finish(m, null, 'Recovered stopped run; inspect partial changes before continuation', m.status === 'cancelling' ? 'cancelled' : 'failed'); return compact(m); @@ -366,8 +584,10 @@ export function accept(id) { const m = getRun(id); check(m.status === 'finished' && m.result_validated && m.result.status === 'done', 'Only a validated done result can be accepted'); const group = allRuns().filter(r => r.group === m.group); for (const profile of m.required_profiles) { - const attempts = group.filter(r => r.profile === profile).sort((a, b) => a.created.localeCompare(b.created)); - const latest = attempts.at(-1); + const attempts = group.filter(r => r.profile === profile); + const current = currentAttempts(attempts); + check(current.length === 1, `Required profile ${profile} has ambiguous attempt lineages`); + const latest = current[0]; check(latest?.status === 'finished' && latest.result_validated && latest.result?.status === 'done' && !attempts.some(r => active.includes(r.status)), `Required profile ${profile} has not completed its latest attempt validly`); if (profile === m.profile) check(latest.id === m.id, 'Accept the latest attempt, not an earlier result'); } @@ -386,7 +606,7 @@ export async function supervise(id, claim) { const beat = () => atomicJSON(path.join(dir(id), 'heartbeat.json'), { pid: process.pid, claim, at: now() }); beat(); const heartbeatTimer = setInterval(beat, 5000); heartbeatTimer.unref(); - let child, timer, gracefulTimer, forceTimer, result = null, failure = null, requestedReason = null; + let child, spawned = null, timer, gracefulTimer, forceTimer, result = null, failure = null, requestedReason = null; const e = m.executor, isRPC = ['pi', 'omp'].includes(e.harness); const stdoutFile = path.join(dir(id), 'stdout.log'), stderrFile = path.join(dir(id), 'stderr.log'); const logOut = fs.openSync(stdoutFile, 'a', 0o600), logErr = fs.openSync(stderrFile, 'a', 0o600); @@ -413,9 +633,17 @@ export async function supervise(id, claim) { // Cancellation and child registration share the admission lock. admission(() => { m = getRun(id); check(m.status === 'running', 'Run cancelled before model launch'); + // Persist ambiguity before spawn. If the supervisor is killed before PID + // publication, recovery retains the writer lease until explicit process + // inspection confirms that no unregistered worker remains. + m.launch_registration_pending = true; save(m); child = spawn(built.cmd, built.args, { cwd: m.cwd, detached: true, stdio: ['pipe', 'pipe', 'pipe'], env: { ...process.env, DELEGATE_KIT_DEPTH: '1', ...(built.config ? { OPENCODE_CONFIG_CONTENT: mergeInline(built.config), OPENCODE_AUTO_SHARE: 'false' } : {}) } }); - m.child_pid = child.pid || null; m.child_fingerprint = child.pid ? fingerprint(child.pid) : null; save(m); + spawned = { pid: child.pid || null, fingerprint: child.pid ? fingerprint(child.pid) : null }; + m.child_pid = spawned.pid; m.child_fingerprint = spawned.fingerprint; m.execution_started_at = child.pid ? now() : null; + if (process.env.NODE_ENV === 'test' && process.env.DELEGATE_KIT_TEST_FAIL_CHILD_REGISTRATION === id) throw new Error('Injected child registration failure'); + m.launch_registration_pending = false; + save(m); }); const closed = new Promise((resolve, reject) => { child.once('error', reject); child.once('close', (code, sig) => resolve({ code, sig })); }); closed.catch(() => {}); @@ -454,17 +682,22 @@ export async function supervise(id, claim) { clearInterval(heartbeatTimer); clearTimeout(timer); clearTimeout(gracefulTimer); clearTimeout(forceTimer); process.off('SIGTERM', onSignal); process.off('SIGINT', onSignal); // Keep the lease until all descendants have stopped, even after a leader exits. m = getRun(id); - if (m.child_pid && groupAlive(m.child_pid)) { + const ownedChild = m.child_pid ? { pid: m.child_pid, fingerprint: m.child_fingerprint } : spawned; + if (ownedChild?.pid && groupAlive(ownedChild.pid)) { // This supervisor created and continuously owns this process group. - try { process.kill(-m.child_pid, 'SIGKILL'); } catch (error) { if (error.code !== 'ESRCH') failure ||= error.message; } - for (let i = 0; i < 50 && groupAlive(m.child_pid); i++) await sleep(100); + try { process.kill(-ownedChild.pid, 'SIGKILL'); } catch (error) { if (error.code !== 'ESRCH') failure ||= error.message; } + for (let i = 0; i < 50 && groupAlive(ownedChild.pid); i++) await sleep(100); } child?.stdout?.removeAllListeners('data'); child?.stderr?.removeAllListeners('data'); fs.closeSync(logOut); fs.closeSync(logErr); admission(() => { const value = getRun(id); - if (groupAlive(value.child_pid)) { value.status = 'orphaned'; value.error = failure || 'Child group still active; ownership retained'; save(value); } - else finish(value, result, failure, requestedReason || (value.status === 'cancelling' ? 'cancelled' : null)); + const childPid = value.child_pid || ownedChild?.pid || null; + if (groupAlive(childPid)) { + value.child_pid = childPid; value.child_fingerprint ||= ownedChild?.fingerprint || null; + value.status = 'orphaned'; value.error = failure || 'Child group still active; ownership retained'; save(value); + } + else { value.launch_registration_pending = false; finish(value, result, failure, requestedReason || (value.status === 'cancelling' ? 'cancelled' : null)); } }); } } diff --git a/tests/caps.sh b/tests/caps.sh index d05d360..121523c 100755 --- a/tests/caps.sh +++ b/tests/caps.sh @@ -158,16 +158,47 @@ for w in w1 w2 w3 w4 w5 w6 w7 w8; do "$WT" release "$w" >/dev/null 2>&1; done for i in 1 2 3 4 5 6 7 8; do ("$WT" lock w1 --label "owner-$i" >/dev/null 2>&1 && touch "$BASE/won-$i") & done; wait ok "exactly one successful owner" "$(find "$BASE" -name 'won-*' | wc -l | tr -d ' ')" "1" -echo "── abandoned mutexes with dead PIDs do not block admission" +echo "── abandoned mutexes fail closed until explicitly recovered" for w in w1 w2 w3 w4 w5 w6 w7 w8; do "$WT" release "$w" >/dev/null 2>&1; done mkdir -p "$BASE/repo/.git/delegate-kit.caps.lock"; echo 999999 > "$BASE/repo/.git/delegate-kit.caps.lock/pid" -"$WT" lock w8 --max-writers 8 >/dev/null; ok "agent-wt recovered the stale repository mutex" "$?" "0" +ERR=$("$WT" lock w8 --max-writers 8 2>&1 >/dev/null); RC=$? +ok "agent-wt refuses a stale repository mutex" "$RC" "1" +ok "repository recovery instruction is visible" "$(has "$ERR" "verify no operation is running")" "yes" +ok "repository mutex is retained for diagnosis" "$([ -e "$BASE/repo/.git/delegate-kit.caps.lock" ] && echo held || echo free)" "held" +rm -rf "$BASE/repo/.git/delegate-kit.caps.lock" echo 999999 > "$DELEGATE_KIT_HOME/caps.lock" mkdir -p "$BASE/repo/.git/delegate-kit.caps.lock"; echo 999999 > "$BASE/repo/.git/delegate-kit.caps.lock/pid" run "$WTS/w7" --max-writers 1 -ok "agent-run recovered both stale mutexes and reached the capacity check" "$(has "$ERR" "concurrent writers reached")" "yes" -ok "global mutex released" "$([ -e "$DELEGATE_KIT_HOME/caps.lock" ] && echo held || echo free)" "free" -ok "repository mutex released" "$([ -e "$BASE/repo/.git/delegate-kit.caps.lock" ] && echo held || echo free)" "free" +ok "agent-run refuses a stale global mutex" "$RC" "1" +ok "global recovery instruction is visible" "$(has "$ERR" "verify no operation is running")" "yes" +ok "global mutex is retained for diagnosis" "$([ -e "$DELEGATE_KIT_HOME/caps.lock" ] && echo held || echo free)" "held" +ok "repository mutex is retained for diagnosis" "$([ -e "$BASE/repo/.git/delegate-kit.caps.lock" ] && echo held || echo free)" "held" +rm -f "$DELEGATE_KIT_HOME/caps.lock" +rm -rf "$BASE/repo/.git/delegate-kit.caps.lock" + +echo "── owner publication windows are retried instead of declared stale" +( + : > "$DELEGATE_KIT_HOME/caps.lock" + sleep 0.1 + echo "$$" > "$DELEGATE_KIT_HOME/caps.lock" + sleep 0.2 + rm -f "$DELEGATE_KIT_HOME/caps.lock" +) & +while [ ! -e "$DELEGATE_KIT_HOME/caps.lock" ]; do sleep 0.01; done +"$WT" lock w8 --max-writers 8 >/dev/null +ok "global empty-to-owned publication is tolerated" "$?" "0" +"$WT" release w8 >/dev/null +( + mkdir "$BASE/repo/.git/delegate-kit.caps.lock" + sleep 0.1 + echo "$$" > "$BASE/repo/.git/delegate-kit.caps.lock/pid" + sleep 0.2 + rm -rf "$BASE/repo/.git/delegate-kit.caps.lock" +) & +while [ ! -d "$BASE/repo/.git/delegate-kit.caps.lock" ]; do sleep 0.01; done +"$WT" lock w8 --max-writers 8 >/dev/null +ok "repository empty-to-owned publication is tolerated" "$?" "0" +"$WT" release w8 >/dev/null echo "── agent-wt sees an external writer process lock in any linked worktree" "$WT" release w8 >/dev/null diff --git a/tests/observability.test.mjs b/tests/observability.test.mjs new file mode 100644 index 0000000..114f98b --- /dev/null +++ b/tests/observability.test.mjs @@ -0,0 +1,90 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { spawnSync } from 'node:child_process'; +import { fileURLToPath } from 'node:url'; + +const skill = fileURLToPath(new URL('../skills/delegate-kit', import.meta.url)); +const cli = path.join(skill, 'scripts/dk.mjs'); + +function run(home, args) { + const result = spawnSync(process.execPath, [cli, ...args], { + env: { ...process.env, DELEGATE_KIT_HOME: home }, + encoding: 'utf8', + }); + assert.equal(result.status, 0, result.stderr); + return JSON.parse(result.stdout); +} + +function saveRun(home, run) { + const folder = path.join(home, 'runs', run.id); + fs.mkdirSync(folder, { recursive: true }); + fs.writeFileSync(path.join(folder, 'meta.json'), JSON.stringify(run)); +} + +function fixture(id, status, extra = {}) { + return { + schema_version: 2, + id, + parent_session: 'codex:test', + task: 'status-ui', + preset: 'main-test', + profile: id, + role: id.includes('reviewer') ? 'reviewer' : 'implementer', + write: false, + executor: { harness: 'codex', model: 'gpt-6-astra', transport: 'native', access: 'read-only' }, + workspace: { owner: 'delegate-kit', path: os.tmpdir() }, + status, + attempt_kind: 'fresh', + resume_of: null, + actual_model: null, + result_validated: false, + accepted: false, + required_profiles: [], + created: new Date().toISOString(), + usage: null, + cost_usd: null, + ...extra, + }; +} + +test('watch is compact by default and --full preserves diagnostic detail', () => { + const home = fs.mkdtempSync(path.join(os.tmpdir(), 'dk-observability-')); + try { + saveRun(home, fixture('implementer-api', 'running', { + host_attached: true, + execution_started_at: new Date().toISOString(), + host_checked_at: new Date().toISOString(), + progress_at: new Date().toISOString(), + actual_model: 'gpt-6-astra', + })); + saveRun(home, fixture('reviewer-hard', 'prepared')); + + const compact = run(home, ['watch', '--session', 'codex:test', '--task', 'status-ui', '--timeout-ms', '5']); + assert.equal(compact.summary.icon, '🟢'); + assert.equal(compact.summary.working, 1); + assert.equal(compact.agents.find(agent => agent.id === 'implementer-api').icon, '🟢'); + assert.equal(compact.agents.find(agent => agent.id === 'reviewer-hard').icon, '🟡'); + assert.equal(Object.hasOwn(compact, 'observed_at'), false); + assert.equal(Object.hasOwn(compact, 'presets'), false); + assert.equal(Object.hasOwn(compact.summary, 'statuses'), false); + assert.equal(Object.hasOwn(compact.agents[0], 'requested_model'), false); + + const unchanged = run(home, ['watch', '--session', 'codex:test', '--task', 'status-ui', '--after', compact.cursor, '--timeout-ms', '5']); + assert.equal(unchanged.changed, false); + assert.equal(unchanged.watch_timed_out, true); + assert.equal(unchanged.summary.icon, '🟢'); + assert.equal(Object.hasOwn(unchanged, 'agents'), false); + + const full = run(home, ['watch', '--session', 'codex:test', '--task', 'status-ui', '--timeout-ms', '5', '--full']); + assert.equal(typeof full.observed_at, 'string'); + assert.deepEqual(full.presets, ['main-test']); + assert.equal(full.summary.attempts, 2); + assert.equal(full.agents[0].requested_model, 'gpt-6-astra'); + assert.equal(Object.hasOwn(full.agents[0], 'icon'), false); + } finally { + fs.rmSync(home, { recursive: true, force: true }); + } +}); diff --git a/tests/v2-runtime.test.mjs b/tests/v2-runtime.test.mjs new file mode 100644 index 0000000..258395d --- /dev/null +++ b/tests/v2-runtime.test.mjs @@ -0,0 +1,272 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { spawn, spawnSync } from 'node:child_process'; +import { fileURLToPath } from 'node:url'; +import { migrationPlan } from '../skills/delegate-kit/scripts/migrate.mjs'; + +const skill = fileURLToPath(new URL('../skills/delegate-kit', import.meta.url)); +const cli = path.join(skill, 'scripts/dk.mjs'); + +function command(args, options = {}) { + return spawnSync(args[0], args.slice(1), { encoding: 'utf8', ...options }); +} + +test('v2 example validates through the public CLI', () => { + const result = command([process.execPath, cli, 'presets', 'validate', '--file', path.join(skill, 'examples/main.json')]); + assert.equal(result.status, 0, result.stderr); + assert.deepEqual(JSON.parse(result.stdout), { id: 'main', valid: true }); +}); + +test('migration preserves the parent-selected backend-map default and refuses to guess without it', () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'dk-migrate-v2-')); + try { + fs.writeFileSync(path.join(root, 'config.json'), JSON.stringify({ + roles: { implementer: { codex: ['gpt-model', 'high'], claude: ['claude-model', 'high'] } }, + })); + const decided = migrationPlan({ parents: { legacy: 'claude' } }, root); + assert.deepEqual(decided.issues, []); + const selected = decided.presets[0].agents[decided.presets[0].defaults.implementer].executor; + assert.equal(selected.harness, 'claude'); + assert.equal(selected.model, 'claude-model'); + + const ambiguous = migrationPlan({}, root); + assert.match(ambiguous.issues.join('\n'), /needs an explicit parents\.legacy decision/); + } finally { + fs.rmSync(root, { recursive: true, force: true }); + } +}); + +test('migration preserves cross-family duo reviewer routing and requires author identity', () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'dk-migrate-reviewer-')); + try { + fs.writeFileSync(path.join(root, 'config.json'), JSON.stringify({ + mode: 'duo', + families: ['codex', 'claude'], + roles: { reviewer: { codex: ['gpt-review', 'high'], claude: ['claude-review', 'high'] } }, + })); + const decided = migrationPlan({ parents: { legacy: 'claude' }, authors: { legacy: 'claude' } }, root); + assert.deepEqual(decided.issues, []); + const selected = decided.presets[0].agents[decided.presets[0].defaults.reviewer].executor; + assert.equal(selected.harness, 'codex'); + assert.equal(selected.model, 'gpt-review'); + + const ambiguous = migrationPlan({ parents: { legacy: 'claude' } }, root); + assert.match(ambiguous.issues.join('\n'), /authors\.legacy/); + } finally { + fs.rmSync(root, { recursive: true, force: true }); + } +}); + +test('migration materializes a selected backend missing from a partial role override map', () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'dk-migrate-partial-')); + try { + fs.writeFileSync(path.join(root, 'config.json'), JSON.stringify({ + backends: { codex: { model: 'gpt-default' } }, + roles: { implementer: { claude: ['claude-override', 'high'] } }, + })); + const plan = migrationPlan({ parents: { legacy: 'codex' } }, root); + assert.deepEqual(plan.issues, []); + const selected = plan.presets[0].agents[plan.presets[0].defaults.implementer].executor; + assert.equal(selected.harness, 'codex'); + assert.equal(selected.model, 'gpt-default'); + } finally { + fs.rmSync(root, { recursive: true, force: true }); + } +}); + +test('different state roots cannot reserve the same worktree concurrently', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'dk-v2-lease-')); + try { + const repo = path.join(root, 'repo'); + const worktree = path.join(root, 'writer'); + fs.mkdirSync(repo); + assert.equal(command(['git', 'init', '-q', '-b', 'main'], { cwd: repo }).status, 0); + command(['git', '-c', 'user.name=Test', '-c', 'user.email=test@example.com', 'commit', '-q', '--allow-empty', '-m', 'init'], { cwd: repo }); + assert.equal(command(['git', 'worktree', 'add', '-q', '-b', 'dk/writer', worktree], { cwd: repo }).status, 0); + const brief = path.join(root, 'brief.md'); fs.writeFileSync(brief, 'Implement the bounded fixture.'); + const bin = path.join(root, 'bin'); fs.mkdirSync(bin); + fs.writeFileSync(path.join(bin, 'codex'), '#!/bin/sh\nexit 0\n', { mode: 0o755 }); + const preset = JSON.parse(fs.readFileSync(path.join(skill, 'examples/main.json'), 'utf8')); + const repoMutex = path.join(repo, '.git', 'delegate-kit.caps.lock'); + fs.mkdirSync(repoMutex); + const legacyOwner = spawn(process.execPath, ['-e', `const fs=require('fs');const d=process.argv[1];setTimeout(()=>fs.writeFileSync(d+'/pid',String(process.pid)),100);setTimeout(()=>{fs.rmSync(d,{recursive:true,force:true});process.exit(0)},400)`, repoMutex]); + const legacyOwnerDone = new Promise(resolve => legacyOwner.on('close', resolve)); + const startedAt = Date.now(); + const starts = []; + for (let i = 0; i < 8; i++) { + const home = path.join(root, `state-${i}`); + fs.mkdirSync(path.join(home, 'presets'), { recursive: true }); + fs.writeFileSync(path.join(home, 'presets/main.json'), JSON.stringify(preset)); + starts.push(new Promise(resolve => { + const child = spawn(process.execPath, [cli, 'prepare', '--session', `codex:race-${i}`, '--preset', 'main', '--task', 'race', '--agent', 'implementer', '--brief', brief, '--cwd', worktree], { + env: { ...process.env, PATH: `${bin}${path.delimiter}${process.env.PATH}`, DELEGATE_KIT_HOME: home }, + stdio: ['ignore', 'pipe', 'pipe'], + }); + let stdout = '', stderr = ''; + child.stdout.on('data', data => { stdout += data; }); + child.stderr.on('data', data => { stderr += data; }); + child.on('close', code => resolve({ code, stdout, stderr })); + })); + } + const results = await Promise.all(starts); + await legacyOwnerDone; + assert.ok(Date.now() - startedAt >= 350, 'v2 must wait for the v1 repository owner instead of replacing its empty lock directory'); + assert.equal(results.filter(result => result.code === 0).length, 1, results.map(result => result.stderr).join('\n')); + assert.equal(results.filter(result => /already owned/.test(result.stderr)).length, 7); + const gitDir = command(['git', 'rev-parse', '--absolute-git-dir'], { cwd: worktree }).stdout.trim(); + assert.equal(JSON.parse(fs.readFileSync(path.join(gitDir, 'delegate-kit.lock'), 'utf8')).kind, 'v2'); + + fs.mkdirSync(repoMutex); + const blockedHome = path.join(root, 'state-blocked'); + fs.mkdirSync(path.join(blockedHome, 'presets'), { recursive: true }); + fs.writeFileSync(path.join(blockedHome, 'presets/main.json'), JSON.stringify(preset)); + const blocked = command([process.execPath, cli, 'prepare', '--session', 'codex:empty-lock', '--preset', 'main', '--task', 'empty-lock', '--agent', 'implementer', '--brief', brief, '--cwd', worktree], { + env: { ...process.env, PATH: `${bin}${path.delimiter}${process.env.PATH}`, DELEGATE_KIT_HOME: blockedHome, NODE_ENV: 'test', DELEGATE_KIT_TEST_LOCK_TIMEOUT_MS: '100' }, + timeout: 5000, + }); + assert.notEqual(blocked.status, 0); + assert.match(blocked.stderr, /no published owner/); + assert.equal(fs.existsSync(repoMutex), true, 'unknown repository lock must be retained for diagnosis'); + } finally { + fs.rmSync(root, { recursive: true, force: true }); + } +}); + +test('failed child registration stops the process group before releasing the writer lease', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'dk-v2-child-')); + try { + const repo = path.join(root, 'repo'); + const worktree = path.join(root, 'writer'); + fs.mkdirSync(repo); + command(['git', 'init', '-q', '-b', 'main'], { cwd: repo }); + command(['git', '-c', 'user.name=Test', '-c', 'user.email=test@example.com', 'commit', '-q', '--allow-empty', '-m', 'init'], { cwd: repo }); + command(['git', 'worktree', 'add', '-q', '-b', 'dk/writer', worktree], { cwd: repo }); + const brief = path.join(root, 'brief.md'); fs.writeFileSync(brief, 'Implement the bounded fixture.'); + const childPid = path.join(root, 'child.pid'); + const bin = path.join(root, 'bin'); fs.mkdirSync(bin); + fs.writeFileSync(path.join(bin, 'codex'), '#!/bin/sh\necho $$ > "$CHILD_PID_FILE"\nwhile :; do sleep 1; done\n', { mode: 0o755 }); + const home = path.join(root, 'state'); fs.mkdirSync(path.join(home, 'presets'), { recursive: true }); + fs.writeFileSync(path.join(home, 'presets/main.json'), fs.readFileSync(path.join(skill, 'examples/main.json'))); + const env = { ...process.env, PATH: `${bin}${path.delimiter}${process.env.PATH}`, DELEGATE_KIT_HOME: home, CHILD_PID_FILE: childPid, NODE_ENV: 'test' }; + const prepared = command([process.execPath, cli, 'prepare', '--session', 'codex:child-failure', '--preset', 'main', '--task', 'child-failure', '--agent', 'implementer', '--brief', brief, '--cwd', worktree], { env }); + assert.equal(prepared.status, 0, prepared.stderr); + const id = JSON.parse(prepared.stdout).runs[0].id; + env.DELEGATE_KIT_TEST_FAIL_CHILD_REGISTRATION = id; + const launched = command([process.execPath, cli, 'run', id], { env }); + assert.equal(launched.status, 0, launched.stderr); + + const metaFile = path.join(home, 'runs', id, 'meta.json'); + let meta; + for (let i = 0; i < 100; i++) { + meta = JSON.parse(fs.readFileSync(metaFile, 'utf8')); + if (['failed', 'orphaned'].includes(meta.status)) break; + await new Promise(resolve => setTimeout(resolve, 50)); + } + assert.equal(meta.status, 'failed'); + assert.match(meta.error, /Injected child registration failure/); + const gitDir = command(['git', 'rev-parse', '--absolute-git-dir'], { cwd: worktree }).stdout.trim(); + assert.equal(fs.existsSync(path.join(gitDir, 'delegate-kit.lock')), false); + if (fs.existsSync(childPid)) { + const pid = Number(fs.readFileSync(childPid, 'utf8')); + assert.throws(() => process.kill(pid, 0), error => error.code === 'ESRCH'); + } + } finally { + fs.rmSync(root, { recursive: true, force: true }); + } +}); + +test('interrupted child registration retains ownership until evidenced recovery', () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'dk-v2-recover-')); + try { + const repo = path.join(root, 'repo'); + const worktree = path.join(root, 'writer'); + fs.mkdirSync(repo); + command(['git', 'init', '-q', '-b', 'main'], { cwd: repo }); + command(['git', '-c', 'user.name=Test', '-c', 'user.email=test@example.com', 'commit', '-q', '--allow-empty', '-m', 'init'], { cwd: repo }); + command(['git', 'worktree', 'add', '-q', '-b', 'dk/writer', worktree], { cwd: repo }); + const brief = path.join(root, 'brief.md'); fs.writeFileSync(brief, 'Implement the bounded fixture.'); + const evidence = path.join(root, 'evidence.txt'); fs.writeFileSync(evidence, 'Process table and worktree handles inspected; no child or descendant exists.'); + const bin = path.join(root, 'bin'); fs.mkdirSync(bin); fs.writeFileSync(path.join(bin, 'codex'), '#!/bin/sh\nexit 0\n', { mode: 0o755 }); + const home = path.join(root, 'state'); fs.mkdirSync(path.join(home, 'presets'), { recursive: true }); + fs.writeFileSync(path.join(home, 'presets/main.json'), fs.readFileSync(path.join(skill, 'examples/main.json'))); + const env = { ...process.env, PATH: `${bin}${path.delimiter}${process.env.PATH}`, DELEGATE_KIT_HOME: home }; + const prepared = command([process.execPath, cli, 'prepare', '--session', 'codex:recover', '--preset', 'main', '--task', 'recover', '--agent', 'implementer', '--brief', brief, '--cwd', worktree], { env }); + assert.equal(prepared.status, 0, prepared.stderr); + const id = JSON.parse(prepared.stdout).runs[0].id; + const metaFile = path.join(home, 'runs', id, 'meta.json'); + const meta = JSON.parse(fs.readFileSync(metaFile, 'utf8')); + Object.assign(meta, { status: 'orphaned', pid: 999999, pid_fingerprint: null, child_pid: null, launch_registration_pending: true }); + fs.writeFileSync(metaFile, JSON.stringify(meta)); + const gitDir = command(['git', 'rev-parse', '--absolute-git-dir'], { cwd: worktree }).stdout.trim(); + const lease = path.join(gitDir, 'delegate-kit.lock'); + + const refused = command([process.execPath, cli, 'recover', id], { env }); + assert.notEqual(refused.status, 0); + assert.match(refused.stderr, /registration was interrupted/); + assert.equal(fs.existsSync(lease), true); + + const recovered = command([process.execPath, cli, 'recover', id, '--confirmed-not-started', '--evidence', evidence], { env }); + assert.equal(recovered.status, 0, recovered.stderr); + assert.equal(JSON.parse(recovered.stdout).status, 'failed'); + assert.equal(fs.existsSync(lease), false); + assert.match(JSON.parse(fs.readFileSync(metaFile, 'utf8')).launch_recovery_evidence, /Process table/); + } finally { + fs.rmSync(root, { recursive: true, force: true }); + } +}); + +test('resume uses repository-global ownership across different state roots', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'dk-v2-resume-')); + try { + const repo = path.join(root, 'repo'); + const worktree = path.join(root, 'writer'); + fs.mkdirSync(repo); + command(['git', 'init', '-q', '-b', 'main'], { cwd: repo }); + command(['git', '-c', 'user.name=Test', '-c', 'user.email=test@example.com', 'commit', '-q', '--allow-empty', '-m', 'init'], { cwd: repo }); + command(['git', 'worktree', 'add', '-q', '-b', 'dk/writer', worktree], { cwd: repo }); + const brief = path.join(root, 'brief.md'); fs.writeFileSync(brief, 'Continue the bounded fixture.'); + const bin = path.join(root, 'bin'); fs.mkdirSync(bin); fs.writeFileSync(path.join(bin, 'codex'), '#!/bin/sh\nexit 0\n', { mode: 0o755 }); + const homes = [path.join(root, 'state-a'), path.join(root, 'state-b')]; + for (const home of homes) { fs.mkdirSync(path.join(home, 'presets'), { recursive: true }); fs.writeFileSync(path.join(home, 'presets/main.json'), fs.readFileSync(path.join(skill, 'examples/main.json'))); } + const envA = { ...process.env, PATH: `${bin}${path.delimiter}${process.env.PATH}`, DELEGATE_KIT_HOME: homes[0] }; + const prepared = command([process.execPath, cli, 'prepare', '--session', 'codex:resume-race', '--preset', 'main', '--task', 'resume-race', '--agent', 'implementer', '--brief', brief, '--cwd', worktree], { env: envA }); + assert.equal(prepared.status, 0, prepared.stderr); + const id = JSON.parse(prepared.stdout).runs[0].id; + const runA = path.join(homes[0], 'runs', id); + const metaFile = path.join(runA, 'meta.json'); + const meta = JSON.parse(fs.readFileSync(metaFile, 'utf8')); + Object.assign(meta, { status: 'finished', transport_session_id: 'exact-session', result_validated: true, result: { status: 'done' }, finished: new Date().toISOString() }); + fs.writeFileSync(metaFile, JSON.stringify(meta)); + const gitDir = command(['git', 'rev-parse', '--absolute-git-dir'], { cwd: worktree }).stdout.trim(); + fs.rmSync(path.join(gitDir, 'delegate-kit.lock')); + fs.mkdirSync(path.join(homes[1], 'runs'), { recursive: true }); + fs.cpSync(runA, path.join(homes[1], 'runs', id), { recursive: true }); + + const results = await Promise.all(homes.map(home => new Promise(resolve => { + const child = spawn(process.execPath, [cli, 'resume', id, '--brief', brief], { env: { ...envA, DELEGATE_KIT_HOME: home }, stdio: ['ignore', 'pipe', 'pipe'] }); + let stderr = ''; child.stderr.on('data', data => { stderr += data; }); child.on('close', code => resolve({ code, stderr })); + }))); + assert.equal(results.filter(result => result.code === 0).length, 1, results.map(result => result.stderr).join('\n')); + assert.equal(results.filter(result => /already owned/.test(result.stderr)).length, 1); + } finally { + fs.rmSync(root, { recursive: true, force: true }); + } +}); + +test('v2 waits through v1 empty-owner publication windows', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'dk-v1-v2-lock-')); + try { + const home = path.join(root, 'state'); fs.mkdirSync(home, { recursive: true }); + const mutex = path.join(home, 'caps.lock'); fs.writeFileSync(mutex, ''); + const owner = spawn(process.execPath, ['-e', `const fs=require('fs');const f=process.argv[1];setTimeout(()=>fs.writeFileSync(f,String(process.pid)),100);setTimeout(()=>{fs.rmSync(f,{force:true});process.exit(0)},400)`, mutex]); + const ownerDone = new Promise(resolve => owner.on('close', resolve)); + const overview = command([process.execPath, cli, 'overview', '--session', 'codex:window'], { env: { ...process.env, DELEGATE_KIT_HOME: home }, timeout: 5000 }); + assert.equal(overview.status, 0, overview.stderr); + await ownerDone; + } finally { + fs.rmSync(root, { recursive: true, force: true }); + } +});