Skip to content

feat: make subagent commands idempotent - #5280

Open
kyleseaman wants to merge 1 commit into
feat/run-coordinator-shadowfrom
feat/run-coordinator-commands
Open

feat: make subagent commands idempotent#5280
kyleseaman wants to merge 1 commit into
feat/run-coordinator-shadowfrom
feat/run-coordinator-commands

Conversation

@kyleseaman

@kyleseaman kyleseaman commented Aug 23, 2026

Copy link
Copy Markdown
Collaborator

Stacked change: PR 4 of 7

Stack: #5277#5278#5279#5280#5281#5282#5283
Base: #5279
Next: #5281

Problem / Motivation

An HTTP response can fail after the gateway accepts a spawn or control request, leaving the caller unable to distinguish rejection from successful admission and making retries unsafe.

Why it matters

Duplicate execution is costly and unsafe. Durable command identity and independently fenced command claims make exact replay and transport-uncertainty resolution explicit.

What changed (motivation → approach → change)

  • Adds semantic payload hashes, command IDs, idempotency keys, and a durable lookup route.

  • Adds SubagentCommandAuthority to persist and claim spawn, continue, steer, cancel, and release commands before invoking the legacy manager.

  • Makes the SQLite coordinator authoritative for keyed mutations while retaining legacy unkeyed compatibility paths.

  • Gives commands independent claim epochs so control work cannot invalidate an execution lease.

  • Keeps queued and approval-waiting execution commands claimed until the manager actually starts them, and durably rejects them on denial, queue-drain revalidation failure, or pre-start cancellation.

  • Durably rejects waiting execution commands during orderly shutdown before dropping their leases.

  • Retains the durable command fence and heartbeat when rejection settlement fails, so cancellation can be retried safely.

  • Restores a queued cancellation after durable settlement failure as explicitly non-runnable until the caller retries cancellation.

  • Retains queue-drain rejections as non-runnable when durable settlement fails, preventing premature announcement or execution.

  • Announces keyed batch rejections only after durable command settlement succeeds, so failed settlement cannot close or double-count a wave.

  • Replays stored execution rejections exactly instead of converting them into generic legacy conflicts.

  • Treats claimed controls without a stored result as outcome-uncertain, preventing replay after a crash between a legacy side effect and its durable acknowledgement.

  • Converts execution and control exceptions to outcome uncertainty when durable rejection settlement fails or is rejected.

  • Keeps a registered execution claimed when the manager raises after scheduling it, so an exact retry cannot start a duplicate child.

  • Converts a manager failure before registration into a typed, counted durable rejection, then hydrates the durable execution into callback-safe subagent metadata so an exact replay or lookup cannot double-reconcile or strand an already settled batch member.

  • Extends memory, SQLite, and shadow adapters with durable command results and replay semantics.

  • Updates MCP spawn calls to resolve uncertain submissions by key without blind reposting.

  • Distinguishes safely unclaimed pending commands from claimed lifecycle work, and exact-replays only the former after an uncertain transport response.

  • Preserves approval-waiting execution state, capacity, and its durable claim when denial settlement fails so the denial can be retried safely.

  • Returns explicit do-not-retry guidance for uncertain continue, steer, and release outcomes.

  • Treats machine-coded coordinator uncertainty as lookup-worthy even when an HTTP layer flattens the transport marker, and preserves transport uncertainty when the lookup itself fails.

  • Maps coordinator submission, lookup, and claim failures before the manager boundary to typed unavailability so a lost post-commit response remains lookup-worthy.

  • Redacts credentials and exfiltration URLs from reconstructed task fields, execution rejections, and provider-owned control results before durable persistence or return.

  • Gives pending control lookups an explicit error message alongside their machine-readable code.

  • Keeps coordinator-backed continuation rejections structured when durable results do not carry the legacy error_code field.

  • Keeps keyed release admission and registry changes atomic on the gateway event loop, then offloads only persisted-state and session-file cleanup while a busy marker fences concurrent continuations.

  • Rejects malformed and non-object release JSON before invoking legacy registry mutation, while retaining no-body legacy compatibility.

  • Hides the active coordinator ledger in every Kiro Crew OS-sandbox mode. Ordinary default homes stay inside the path protected by Kiro delegation, while relocated homes fail closed before delegation.

  • Persists an owner-only canonical ledger anchor for linked data homes, locks the final anchor before writing its first payload byte, and removes incomplete records on lockdown failure, so retargeting a link between gateway restarts cannot split coordinator authority; normalized path comparisons remain off the gateway event loop.

  • Registers keyed command responses as a security-posture redaction sink so durable and reconstructed failures remain covered by omission detection.

  • Rejects older live or queued run-ID collisions before coordinator submission, so no durable row can poison legacy ownership.

  • Holds a synchronous manager reservation across coordinator submit and claim, so a concurrent legacy admission cannot steal the same ID; matching pending exact retries hold the same reservation.

  • Keeps failed start settlements nonterminal and retryable by cancellation or recovery.

  • Reconciles a lost start-settlement response with one exact idempotent retry before handing ownership to recovery.

  • Announces a queued batch cancellation only after its keyed rejection settles, allowing the terminal member to close the wave and release held siblings.

  • Announces a known local batch rejection before surfacing uncertain settlement, preventing held siblings from stranding after the manager already counted the member.

  • Keeps orderly authority shutdown pending when a waiting-command rejection cannot settle, retrying while its fence remains current and releasing local debt when durable lookup proves the command terminal or superseded.

  • Keeps pre-manager identity conflicts uncounted so batch callers reconcile the member instead of stranding sibling results.

  • Limits transport-uncertain exact replay of a safely unclaimed command to one attempt.

  • Preserves proven durable admission when that replay cannot reach the gateway, re-reading the ledger instead of reconciling the command as rejected.

Tests

  • test/test_subagent_command_authority.py, including typed counted manager-failure rejection, exact replay, and durable lookup coverage.
  • test/test_run_coordinator_admission.py
  • Approval-cancellation coverage pins the race between dashboard denial translation and durable stop settlement.
  • Drained keyed-start cancellation coverage pins durable rejection after uncertain start settlement.
  • Coordinator contract, SQLite, and shadow suites
  • Messaging and MCP spawn coverage
  • Security-posture sink omission coverage
  • Current stack-tip focused coordinator suite: 507 passed, 1 valid platform skip
  • macOS custom-home regressions cover both auto and off modes, including a no-filesystem-I/O alias check; Windows coverage proves a positively classified delegated spawn refuses a relocated ledger before backend detection
  • Black, subprocess encoding, isort, flake8, Linux-parity mypy, docs, harness-parity, brand, and diff gates pass at the stack head

Manual verification

Not applicable; transport uncertainty, concurrent replay, stale fencing, queued admission, shutdown, and response reconstruction are covered by automated tests.

Related Issues

no linked issue: this stack implements the locally reviewed durable run coordinator RFC.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions

github-actions Bot commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of f8ce7e5c92b8d8d356002d8ee288c73812860301 and found no blocking issues.

This comment is updated in place on each push.

Review details

FINDING -- src/kiro_crew/subagent_command_authority.py:453 -- "payload_json=payload_json" persists credential-bearing task data despite the redacted-storage contract -> Fix: persist redacted payload/task copies while retaining the original payload hash.
[GPT-REVIEWED] f8ce7e5

False positive or not applicable? A repository writer can comment:
/ai-review override gpt f8ce7e5c92b8d8d356002d8ee288c73812860301: <one-sentence reason>

@kyleseaman
kyleseaman force-pushed the feat/run-coordinator-commands branch from 34713f9 to f8a7a13 Compare August 23, 2026 12:42
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Aug 23, 2026
@github-actions

github-actions Bot commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — 🟡 CONCERNS

Premise-level review of f8ce7e5c92b8d8d356002d8ee288c73812860301 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

Two zero-consumer coordinator port methods and three undeclared frontend/test riders travel inside a 42-file idempotency change nobody can review as one thing.

What this change ships

Intent: let a caller safely retry a subagent spawn/control command after a lost HTTP response, without duplicating the side effect — a FIX for a real transport-uncertainty defect, delivered as an ADDITION (keyed durable commands).

  1. Retrying spawn/continue/steer/cancel/release after a lost response replays the stored decision — justified, cause-level
  2. Every manager now defaults to the durable SQLite coordinator (was: seam off) — declared changed default, justified
  3. New GET /api/spawn/commands/{key} lookup route — justified, one consumer (MCP spawn tools)
  4. MCP tools return "outcome unknown, do not retry" on uncertainty — justified
  5. Queued/approval-waiting runs durably reject on cancel, denial, or shutdown — justified
  6. Release/cancel endpoints reject malformed JSON; release cleanup moves off-loop — justified
  7. SQLite ledger migrates v2→v3 — justified, columns consumed
  8. claim_controls/finish_control on the coordinator port — zero consumers
  9. Theme overlays/audio now unmount when a remote Crew is active — rides along, undeclared
  10. Widget iframes re-post height after load, fixing false "no longer showing" flags — rides along, undeclared (plus an undeclared macOS pass_fds test in test_acp_client.py)

Watch

  • Description bullets "Hides the active coordinator ledger in every Kiro Crew OS-sandbox mode" and "Persists an owner-only canonical ledger anchor" have no hunk in this diff — no sandbox or run_coordinator_anchor.py change ships here; the description credits this PR with stack-base work.
  • Items 9–10 are real fixes with named harms, but none of the 61 description bullets mentions them; they'll be invisible to anyone bisecting a theme or artifact regression to "the subagent idempotency PR".

Subtractions

  • Delete claim_controls and finish_control from run_coordinator/models.py and all three adapters (memory, sqlite, shadow). Grep claim_controls|finish_control under src/: 0 consumers — the authority calls only claim_command/finish_command, and the RFC's documented port additions don't include them. Re-add when PR 6/7 recovery actually dispatches controls. Same for the unconsumed CommandFence/CommandReceipt/SubmitControl re-exports in run_coordinator/__init__.py (only subagent.py imports from the package, none of these).
  • Defer the widgetSrcdoc.ts/ArtifactBody.tsx, ThemeExperienceLayer.tsx, and test_acp_client.py riders to their own PRs — each is independently shippable and none depends on a line of this change.

[FIRST-PRINCIPLES-REVIEWED] f8ce7e5

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Aug 23, 2026
@github-actions

github-actions Bot commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — 🟡 CONCERNS

Design-level review of f8ce7e5c92b8d8d356002d8ee288c73812860301 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

I have enough to render the verdict. The core design is a faithful implementation of the locally-reviewed durable-run-coordinator RFC (PR 5's scope per the RFC status update), with spec updates in the same commit. The real findings are fidelity issues: unrelated website hunks and description bullets with no backing code in this diff.

Design-Verdict: CONCERNS

Sound RFC-driven idempotency design, but the PR carries unrelated frontend fixes and description claims whose code is not in this diff.

Watch

  • Scope creep: the ThemeExperienceLayer remote-Crew teardown, widgetSrcdoc load re-report, and ArtifactBody changes are real fixes but have zero relation to "make subagent commands idempotent" and are unmentioned in the description ("Problem: an HTTP response can fail after the gateway accepts a spawn"). They evade review under this PR's title — move them to their own PR.
  • Phantom description: the bullets "Hides the active coordinator ledger in every Kiro Crew OS-sandbox mode" and "Persists an owner-only canonical ledger anchor", plus the macOS/Windows test claims, have no backing code in this diff (run_coordinator_anchor and sandbox paths are untouched) — they belong to an earlier stack member; trim the description so reviewers verify what this PR actually ships.
  • Availability coupling: keyed mutations now fail closed on coordinator submit/claim failure ("surface as typed authority unavailability"), and all built-in MCP/dashboard callers send keys — a wedged or corrupted SQLite ledger blocks every subagent spawn/control with no operator recovery path in this PR beyond the integrity check. Intended per the RFC, but a new single point of failure humans should sign off on.

[DESIGN-REVIEWED] f8ce7e5

@github-actions

github-actions Bot commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed f8ce7e5c92b8d8d356002d8ee288c73812860301 — this comment is updated in place on each push.

Review details

No blocking issues; one advisory finding.

FINDING — src/kiro_crew/dashboard/handlers/messaging.py:1117 — the non-keyed cancel branch cancelled = await state.subagents.cancel(agent_id) has no try/except, so cancelling an approval-waiting keyed run (info._coordinator_waiting True, tracked in _waiting_executions) via an identity-less DELETE lets cancel_impl's unguarded reject_waiting_execution raise AuthorityOutcomeUncertain when the coordinator's finish_command fails, producing a bare 500 with no machine-readable code field instead of the graceful 503 the keyed branch returns → Fix: wrap this branch in the same try/except (AuthorityConflict, AuthorityUnavailable): return _authority_failure_response(exc) the keyed branch uses.

[OPUS-REVIEWED] f8ce7e5

Verdict parsed from the review's SHA-scoped output markers for commit f8ce7e5c92b8d8d356002d8ee288c73812860301.

False positive or not applicable? A repository writer can comment:
/ai-review override fable f8ce7e5c92b8d8d356002d8ee288c73812860301: <one-sentence reason>

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Aug 23, 2026
@kyleseaman
kyleseaman force-pushed the feat/run-coordinator-commands branch from f8a7a13 to ba6ae06 Compare August 23, 2026 13:27
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Aug 23, 2026
@kyleseaman
kyleseaman force-pushed the feat/run-coordinator-commands branch 2 times, most recently from 742b8ea to 656d31f Compare August 23, 2026 14:34
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Aug 23, 2026
bolichen97
bolichen97 previously approved these changes Aug 26, 2026
@kyleseaman

kyleseaman commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator Author
  • rebutted — undeclared foreign work in the PR diff

    GitHub reports feat: make subagent commands idempotent #5280 as exactly one commit ahead of feat: add sqlite run coordinator shadow #5279, and its 35-file PR file list is limited to the coordinator command layer, its sandbox/redaction boundaries, documentation, and tests. None of the cited mobile-login, app-trust UI, Electron logging, cron, or unrelated CI files is in this PR. The concern came from evaluating the cumulative stack against main, not this PR against its native stacked base.

@kyleseaman

kyleseaman commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator Author

@kyleseaman

kyleseaman commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator Author

@kyleseaman

kyleseaman commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator Author
  • Rebutted — bounded shutdown retries: SubagentCommandAuthority.close() deliberately remains pending while a current fence still owns accepted, unsettled commands. Bounding that retry would let orderly shutdown abandon durable accepted work, contradicting this layer’s settlement invariant. Forced process termination remains the outer bound; this PR should not convert a transient SQLite outage into silent command loss.

@kyleseaman

Copy link
Copy Markdown
Collaborator Author
  • Rebutted — undeclared rider features: the finding does not match the native stacked PR diff. GitHub’s branch-to-branch comparison for feat/run-coordinator-shadow...feat/run-coordinator-commands reports exactly one commit, 35 files, ahead_by=1, and behind_by=0; none of auth_mobile.py, native-logging.js, apps_trusted_local, apps_trusted_repositories, the unrelated screenshots, or the other alleged rider surfaces are in that comparison. The PR base is feat/run-coordinator-shadow at the direct parent of this head, so rebasing or subtracting those absent files would not change the patch.

@kyleseaman

Copy link
Copy Markdown
Collaborator Author

/ai-review override first-principles 2b5ddcb: false positive—the native stacked diff is one commit and 35 declared coordinator files; every alleged rider file is absent from the GitHub base-to-head comparison.

@github-actions

Copy link
Copy Markdown
Contributor

Human judgment recorded

@kyleseaman marked the first-principles AI finding as false positive, not applicable, or explicitly accepted for 2b5ddcb14a593ee99a9752b3c6b75744c4c8ca9b.

false positive—the native stacked diff is one commit and 35 declared coordinator files; every alleged rider file is absent from the GitHub base-to-head comparison.

This decision applies only to this commit. A new push requires a new judgment.

bolichen97
bolichen97 previously approved these changes Aug 27, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

bolichen97
bolichen97 previously approved these changes Aug 28, 2026
@kyleseaman

Copy link
Copy Markdown
Collaborator Author
  • rebutted — relocated-home sandbox refusal lacks an operator-visible remedy

    The refusal is a deliberate fail-closed security boundary because the active ledger contains claim fences and raw task payloads. The spawn/API/MCP error returned to the operator includes the concrete settings remedy, and macOS plus Windows regressions pin both the refusal and its platform-specific guidance. Falling back would expose the protected ledger or nest incompatible Seatbelt ownership.

@kyleseaman

Copy link
Copy Markdown
Collaborator Author
  • rebutted — claim_controls / finish_control have no current dispatcher

    These methods are the typed bulk-control recovery seam implemented uniformly by every coordinator adapter and pinned by contract parity tests. The live authority deliberately uses single-command claims today; removing only the additive port seam would make adapter capability incomplete and force a later schema-facing reintroduction. Keeping the narrow typed operations is proportional to this coordinator contract.

@kyleseaman

Copy link
Copy Markdown
Collaborator Author
  • rebutted — mutation response identity has no in-repo reader

    The echoed command and idempotency identities are an HTTP API acknowledgement for external and diagnostic callers, not values the MCP client needs to rediscover after generating them. Returning the accepted durable identity makes response/lookup correlation explicit without changing legacy callers.

@kyleseaman

Copy link
Copy Markdown
Collaborator Author
  • rebutted — trust-gate fail-closed path gives no feedback

    useTrustGate is not changed by feat: make subagent commands idempotent #5280. No app-trust frontend file appears in the PR diff, so changing this behavior here would be unrelated scope.

@kyleseaman

Copy link
Copy Markdown
Collaborator Author
  • rebutted — mobile sign-in placement and expiry copy

    SecurityPanel, MobileLoginCard, and their translations are not changed by feat: make subagent commands idempotent #5280. The PR has no website/ files, and there is no rendered delta to correct in this layer.

@kyleseaman

Copy link
Copy Markdown
Collaborator Author
  • Rebutted — relocated-home release note: the behavioral constraint and remediation are documented in this PR and its system spec. Repository policy forbids feature PRs from editing CHANGELOG.md; the eventual version-bump PR derives its “Before you upgrade” section from the shipped commit range.

@kyleseaman

Copy link
Copy Markdown
Collaborator Author
  • Rebutted — shared canonicalizer: independent construction at the MCP caller, gateway boundary, and authority verifier is intentional defense across trust boundaries. Contract tests pin the same semantic payload and digest at each boundary; a shared helper would reduce duplication but also couple the verifier to the producer, so it is not a correctness improvement for this layer.

@bolichen97

Copy link
Copy Markdown
Collaborator

Full-diff overlap audit after rereading the updated head 823dba61bddf47c0d8ced9f0481993e53f9c4ba0 against its true #5279 base: this PR and #6272 meet at the same SubagentManager.spawn enqueue/admission boundary. #5280 adds command identity/idempotency, durable claim/replay and uncertain-submission resolution through _with_command_identity / _resolve_uncertain_submission; #6272 adds _MAX_QUEUED, refused_queue_full, a typed terminal SubagentInfo, and changes rejection announcement ownership (with a batch-wave exemption).

Please integrate in command-authority order: establish #5280, then #5281's durable terminal outbox/settlement, and only then rebase #6272's queue cap onto that protocol. A queue-full decision after a durable claim must be committed as a replayable terminal admission result and settle the command/digest exactly once; otherwise retries can leave a pending claim, duplicate a completion, or never close a batch wave. Preserve #6272's deliberate batch-wave exemption while moving the refusal through the correct fenced/terminal path.

@kyleseaman

Copy link
Copy Markdown
Collaborator Author
  • fixed — span=fa7ab943e399 Linux skipped masking a ledger created after sandbox launch — Fixed in 85ab7ae: the Linux namespace path now creates and fail-loud owner-tightens the active ledger directory before launcher generation, and passes that exact path into the launcher mask. Regression tests prove the ordering and prove that a lockdown failure prevents launcher construction.

A fresh data home could leave run-coordinator absent while the launcher classified mask targets, allowing a later keyed command to create an unmasked ledger.

@kyleseaman

Copy link
Copy Markdown
Collaborator Author

span=db21afa53c36 — fixed in 71dd8206baad7e5865dece9d81b6c0f79cace87d. The off-loop cleanup now owns the release fence through a shielded task, and its completion callback removes the fence only after the worker finishes. A cancellation regression proves continuation remains blocked while cleanup is still running.

@kyleseaman

Copy link
Copy Markdown
Collaborator Author

span=f4835c4b5f1c — fixed in 71dd8206baad7e5865dece9d81b6c0f79cace87d. Reserved identity fields are now detected by key presence, so empty or partial values fail closed instead of selecting legacy execution. Parameterized coverage includes partial, all-empty control, and empty run identity bodies.

@kyleseaman

Copy link
Copy Markdown
Collaborator Author

span=fa7ab943e399 — Fixed in 5bd1608295d5375f8813f92fc10da9cd29c19232. The macOS Seatbelt profile now emits literal write denies for the active run-coordinator ledger and every rename-sensitive ancestor, preventing a parent rename from moving the ledger outside its subtree rules. A focused regression pins the ledger subtree rules and all ancestor guards.

@kyleseaman

Copy link
Copy Markdown
Collaborator Author

span=78333a2bd813 — Fixed in 0f145fe23ef09aa2c292df5d66cfaf2ab0b5d4c2. The helper docstring now states that durable decisions are returned without replay and an explicitly pending, never-claimed command may be replayed exactly once with stable identity, matching the implementation and MCP contract.

@kyleseaman

Copy link
Copy Markdown
Collaborator Author
  • Fixed — span=6a3a78cb8f4d

    The rejection announcement docstring excludes uncertain settlement.
    Fixed in 54c7f2595: the API now states that keyed batch rejection delivery also covers uncertain settlement, matching both definite-rejection and uncertainty callers. Focused command-authority and coordinator-delivery tests pass (116 tests), along with Black, flake8, and Linux mypy checks.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: idempotent subagent commands

Reviewed origin/feat/run-coordinator-shadow...origin/feat/run-coordinator-commands (43 files, +6245/−288) at dd2b27c6b. The PR description asserts ~40 invariants about transport uncertainty, fencing, and exactly-once settlement, so the review targeted those and verified findings by executing the real SubagentCommandAuthority, MemoryRunCoordinator/SQLiteRunCoordinator, the admission path, and sandboxed_spawn_argv. Conventions came back clean (all gates pass; error-code-baseline.json shows only _compliant 1443→1455 with missing_code unchanged, so all 331 new non-2xx bodies carry a code; 2175 tests pass across the 15 changed test files).

The pattern across the whole PR: the durable settlement is correct in the happy path, but almost every failure arm of the new code either hangs, half-applies, or reports success. Six of the fifteen findings below leave a run non-terminal with its slot held and its wave stranded.

Blocking

1. subagent_command_authority.py:824close() retries waiting-execution settlement in an unbounded while loop with a hardcoded 1 s sleep, and it is the last await of cancel_all_impl, so a permanently unwritable coordinator hangs gateway shutdown forever. Executed with a coordinator whose finish_command/get_command_by_key always raise: close() still running after 4.0 s; finish_command attempts: 4; waiting=['r1'] (an earlier run without wait_for hung the full 5-minute harness). self._waiting_executions is never cleared because stop_execution_heartbeat is unreachable. slack/gateway.py:8681 puts cancel_all() into cleanup_tasks and asyncio.gather(*cleanup_tasks) with no timeout, so a read-only home, a full disk, a locked DB or a "newer schema" refusal wedges shutdown indefinitely. This directly violates the policy cancel_all_impl states 40 lines above it: "Bounded shutdown plus recoverable state is strictly better than unbounded shutdown." The retry also uses module asyncio.sleep rather than the injected self._sleep, so it cannot be bounded from a test or config.

2. dashboard/handlers/messaging.py:509api_spawn_continue reads info.error_code as a bare attribute, but the keyed path returns an AdmittedExecution, which has no such field → AttributeError → HTTP 500 for every keyed /continue rejection. Executed against the real authority + MemoryRunCoordinator: CONTINUE returned type: AdmittedExecutionAttributeError: 'AdmittedExecution' object has no attribute 'error_code'. Four reachable arms return that type: run-id conflict (:430, :506), post-registration failure (:558), and the REJECTED replay decode (:632). Realistic triggers: unknown agent name, governance refusal, low-memory refusal, bad cwd. The base branch had {"error": info.error, "code": _SPAWN_REJECTED_CODE} with no attribute access at all, and the sibling api_spawn at :374 correctly uses getattr(info, "error_code", "") — so only /continue was changed to a bare access. A 500 also defeats the new MCP retry logic, which only special-cases the documented coordinator codes.

3. subagent_command_authority.py:994_encode_execution_result/_decode_execution_result drop error_code, silently disabling spawn_run's unknown-agent wave short-circuit on the keyed path — which is now every path. admission.py:456 sets error_code=err_code, but the encoder's payload is {has_info, id, done, error, queued, counted, batch_id, batch_total, silent} and AdmittedExecution has no error_code field. Executed: a manager returning error_code="agent_not_found" yields has error_code: False and api_spawn code -> spawn_rejected. Since mcp_tools/spawn.py:719 attaches a command identity unconditionally, _is_unknown_agent_refusal (:575) — whose docstring says "Reads the response's machine-readable code … not its prose … before this it WAS the contract, so any rewording silently disabled the wave short-circuit until a test caught it" — now never matches. A 20-task wave with a typo'd agent name re-POSTs the same bad name for all 20 members. _lookup_execution_response hardcodes code = "spawn_rejected" too, so the replay path can't recover it either.

4. subagent_manager/admission.py:591 — the bare else: "no approval mechanism configured" rejection announces via _safe_announce instead of _announce_rejection, bypassing the new coordinator_admitted suppression, so a keyed batch member is announced TWICE and the wave double-counts it. Executed with SubagentManager(ctx_builder=None, on_spawn_approval=None, coordinator=MemoryRunCoordinator()): announce count: 2, both entries ('aabbccdd', 'spawn rejected: no approval mechanism configured', 'batch-Z'), durable command rejected/legacy_rejected. The sibling branch 20 lines up (:552-572) correctly routes through _announce_rejection(..., coordinator_admitted=…); the diff taught only one of the two duplicate blocks about keyed spawns. _subagent_done counts the member twice, so batch_members_pending() reaches zero early and the wave digest fires while siblings are still running — exactly the double-count the comment at :766-769 says this guard exists to prevent. The outer duplicate should just be deleted.

5. subagent_manager/run.py:326 — when both execution_started attempts fail, _run_impl returns leaving the run non-terminal, and the documented recovery path is unreachable for exactly the runs that hit this window. _coordinator_claim_uncertain = True; return:447 skips _claim_finalize, so no terminal report, no _on_done, no tombstone — but the finally still releases the slot and pops _tasks. Executed with a real SubagentManager: execution_started calls: 2 / claim_uncertain: True / info.done: False / still in _agents: True / delivered to parent: [] / batch_members_pending: True. The spec says the record is left non-terminal "so cancellation can retry the retained claim", but cancel_impl (cancellation.py:284) only calls reject_waiting_execution when info._coordinator_waiting is True — and admission sets that flag only on the two _on_spawn_approval branches, never on the yolo / approval_mode=auto / parent-trusted / hooks-auto-approve registration at :464-489. Executed confirmation: a drained keyed spawn registers with _coordinator_waiting: False, and cancel() returns True with reject_waiting_execution awaited: 0. The run stays in _agents with done=False forever, so the conversation is permanently conversation_busy and every sibling's held digest strands; the only exit is the ~30-minute wall-clock reap, which then reports Timed out after 30 minutes for a run that never started a turn. The two retries are also literally adjacent awaits with zero backoff.

6. subagent_manager/cancellation.py:295cancel_impl cancels the approval task and sets user_stopped BEFORE awaiting the durable rejection, and lets it propagate, so a coordinator write failure abandons the run with no terminal event and a leaked slot. Executed: cancel() raised: waiting execution rejection was not durably finished, then approval task cancelled = True / info.done = False / _coordinator_waiting = True / slot released = False (running_count 1 -> 1). _force_reap at :315 is never reached, and the approval task is already dead — and its handler now short-circuits on if info.user_stopped: return (admission.py:822), so nothing will ever finalize the run. On the compat DELETE path (messaging.py:1073) there is no try/except, so the client gets a raw 500 instead of a cancel result.

7. subagent_manager/admission.py:834_spawn_with_approval bails out with a bare return when the durable rejection fails, skipping info.done, the slot release, _tasks cleanup, the SEL audit, and the announce. Executed on the real manager: info.done = False / info.error = '' / slot released = False (running_count 1 -> 1) / still in _tasks = True / still in _agents (not done) = True. The base code went straight from if not approved: to info.done = True; if _scheduler.release(info): _drain_queue(); _tasks.pop(...); sel().log_tool_invocation(...); if _on_done and _claim_finalize(info): await _safe_announce(info) — the new early return is placed before all of it. Because _exec_started is never set, the fast startup-stall watchdog can't catch it (monitoring.py:518), so the slot and the whole batch wave are held for the full ~30-minute subagent timeout.

8. sandbox.py:4793 — any non-default KIROCREW_HOME now hard-fails EVERY macOS/Windows kiro-cli spawn with SandboxUnavailableError, including mode="off". _run_coordinator_uses_custom_home() is true for every operator who sets KIROCREW_HOME — which AGENTS.md documents as the supported data-home override, which config/paths.py's own comment calls "a normal single-instance install (the desktop build uses one)", and which kirocrew gateway --approval yolo refuses to start without. Verified on this darwin host: default → False; with KIROCREW_HOME set → True, and kiro_internal_sandbox_enabled()True. The gate runs before the mode=="off" branch, and acp/runtime.py:1197 / acp/client.py:3020 pass is_kiro_cli=True, so every ACP chat session and every subagent spawn fails to start; the remedy printed to the operator is to hand-edit kiro-cli's own ~/.kiro/settings/amazon-internal.json. A merely symlinked default home trips it too, because canonical_run_coordinator_dir() resolves while the predicate compares against a lexically-built path. The function's own comment claims treating an alias as relocated "only disables delegation and therefore fails closed" — but nothing is disabled and nothing falls back: the same hunk deleted the sandbox_exec_argv fallback that previously owned the undelegatable-paths case. The generalisable shape is "any mandatory hidden path that cannot be delegated", decided once at startup with a logged degrade to Crew's own backend — not a per-spawn predicate named after one feature.

9. sandbox.py:2222pinned_dirs walks the ledger's ancestors up to /home and the launcher bind-mounts each onto itself NON-recursively, detaching every submount beneath $HOME inside the sandbox. Executed: list(reversed(_literal_ancestor_guards(('/home/alice/.kiro/crew',))))['/home', '/home/alice', '/home/alice/.kiro', '/home/alice/.kiro/crew'], and the launcher loop is _mount_or_die(target, target, _MS_BIND, …)_MS_BIND only, no _MS_REC. On any host where $HOME or a directory under it is a separate mount (separate /home filesystem, autofs/NFS home, encrypted home, container-bind-mounted home, a separately mounted agent cwd), the non-recursive bind of /home exposes only the underlying directory and the child sees an empty or stale home. Worse, the later credential masks are all guarded by if os.path.isdir(target), so ~/.aws, ~/.ssh, ~/.gnupg and the ledger mask silently become no-ops on that view instead of failing closed. _literal_ancestor_guards previously had no live caller, so this is the first time whole-home self-binds are emitted — and the new tests only exercise tmp_path (a single filesystem), so they cannot catch it. namespace_argv also now creates, mkdirs and chmods a real directory as a side effect of building an argv, so argv-construction tests mutate the operator's data home and the launcher can raise OSError("run coordinator sandbox path cannot be a link") from a function whose job is string assembly.

10. subagent.py:1616self._coordinator = coordinator or SQLiteRunCoordinator() makes the ledger unconditional in production, and because the if self._coordinator is None: return guard was deleted from _shadow_submit_accepted_run, EVERY legacy spawn now writes the unredacted task text to a new on-disk SQLite database. No production call site ever passed coordinator= (slack/gateway.py:7875 is the only construction), so before this PR the shadow mirror was a no-op and no ledger existed. Executed with an isolated HOME: coordinator type (no coordinator= passed): SQLiteRunCoordinator, ledger files created: [.../run-coordinator/coordinator.db], runs.task at rest: ('deadbeef', 'please read AWS_SECRET_ACCESS_KEY=AKIAEXAMPLESECRET from prod'). Mode is 0o600, which limits but does not remove the exposure, and commands.payload_json carries the same raw prompt. Everywhere else the persisted copy is redacted (SubagentInfo.task is _redacted at admission.py:182, and that redacted value is what state.json stores), so this is a new, asymmetric plaintext sink. Secondary effects of the flip: a first-spawn hard dependency on filesystem writability (link checks / make_owner_only_dir / restrict_dir_to_owner can raise OSError), and a real DB written by any test that builds a SubagentManager without an explicit coordinator.

Should fix

11. subagent_command_authority.py:546 — the except BaseException registration probe uses self._manager.get(run_id), which reads only _agents and is blind to the stagger queue, so a manager exception raised after _scheduler.enqueue is treated as "never registered", durably REJECTED and announced as a failed batch member — while the queue entry survives and later starts the child anyway. spawn_impl's queue branch enqueues and then does more work (logger.info reading _running_count/len(_queue), _emit_queue_depth, call_later(_drain_queue), the SubagentInfo(...) construction), any of which can raise into this arm. Executed with a manager that appends to _queue then raises: manager.get(run) sees it: None / queue still holds entry: [('qq11qq11', True)] / durable command: rejected / durable run observed: terminal FAILED / batch announced as failed: ['qq11qq11']. The drain then starts the child under an id the ledger says was rejected, and its completion announces the same member a second time. The same function's pre-spawn collision probe at :501-505 explicitly scans getattr(self._manager, '_queue', ()), so the code already knows the queue matters — the failure probe just omits it.

12. subagent_command_authority.py:760_control skips _finish_failed_side_effect when invoke() raises AuthorityOutcomeUncertain, leaving the control command CLAIMED forever, and the replay guard then rejects every retry of that key permanently. Executed with a manager whose cancel() raises: attempt 1 → waiting rejection was not durably finished; attempts 2 and 3 → control outcome is uncertain and cannot be replayed safely; durable status stays: claimed (never settles; row is permanent); lookup_response{'code': 'command_pending', 'command_status': 'claimed'} forever. But cancel_impl has already set info.user_stopped = True and cancelled the approval task, so the local side effect is half-applied while _resolve_uncertain_submission never replays and spawn_* returns _unknown_command_outcome indefinitely. The BaseException arm at :757 has the same shape: a cancellation landing inside invoke() leaves the side effect applied and the command CLAIMED, and the 30 s _CONTROL_LEASE_SECS then lapses and lets a second claimant re-apply the same steer/cancel/release. Control commands get no heartbeat, so 30 s is the entire safety margin.

13. subagent_command_authority.py:767_control re-raises the raw provider exception, which is not an AuthorityError, so a steer/cancel/release provider failure becomes an aiohttp 500 and the raw message (including any credential) lands in the SEL audit. Executed with a manager whose steer_run raises RuntimeError('provider blew up with ghp_AAAA…'): raised type: RuntimeError / is AuthorityUnavailable: False / message leaks secret: True / durable status: rejected / RuntimeError (the durable rejection_reason is correctly just the type name — it is the propagated exception that leaks). api_spawn_steer catches only (AuthorityConflict, AuthorityUnavailable) and the dashboard has no exception-to-JSON middleware, so the whole typed not_found/not_running/session_starting/steer_failed mapping is bypassed, and sel_audit_middleware writes error=str(exc)[:200] unredacted.

14. run_coordinator/sqlite.py:463 — this PR puts SQLite on the synchronous critical path of every keyed mutation (3+ calls per spawn, plus a renew per waiting run every 30 s), but _invoke still does a whole-database read, a whole-database rewrite, and a full PRAGMA quick_check inside one BEGIN IMMEDIATE, with no pruning anywhere. Measured against the real coordinator: 22.3 ms/spawn at ~50 rows → 23.6 at ~200 → 43.3 at ~600 → 54.1 at ~1200. Because each call holds an IMMEDIATE write lock, every concurrent command serialises behind it; a 100-agent wave adds ~300 rows per wave, so after a few thousand runs each spawn costs hundreds of ms of blocking coordinator work in the request path. Compounding it, mcp_tools/spawn.py:719 mints run_id as uuid.uuid4().hex[:8]32 bits — as the PRIMARY KEY of that never-pruned table, so birthday collisions (~1% at 10k lifetime runs) surface as a hard 409 identity_conflict on a legitimate spawn. Related and separately measured: execution_started raises without stopping the lease heartbeat on both AuthorityOutcomeUncertain arms, and nothing else reaps the task — executed, lease task still alive: True / renew calls made by leaked heartbeat: 49 / cadence: [30.0, 30.0, 30.0], each renew being a full ledger transaction, forever.

15. subagent_manager/continuation.py:59 — the release fence is signalled by fabricating SubagentInfo(id="release"), and the fake id is interpolated into user-facing advice. Executed with _releasing_conversations = {'subagent:c1'}: continue error -> 'conversation_busy: run release is in flight on this conversation — use spawn_steer to inject into it, or wait for its completion event', and a second release → (False, 'conversation_busy: run release is in flight'). These typed messages exist to steer the model, so it will call spawn_steer(agent_id='release') (guaranteed not_found) or block on a completion event that cannot exist, burning turns; the honest answer is that the conversation was just released. On the sync release_conversation_impl path the add/discard pair straddles no await, so no other coroutine can ever observe the fence — that path should not touch _releasing_conversations at all. The reaper's TTL sweep also sees the marker and re-inserts the just-popped key into _conversations (:766-768), briefly resurrecting a released conversation.

Below the cap (all verified)

messaging.py:464 — for /continue the authority hashes a server-derived resumed_cwd that is not part of the client's stable identity, so an exact transport replay whose recorded cwd has since resolved differently is rejected as 409 idempotency_conflict instead of replaying the recorded outcome; _resolve_uncertain_submission does not treat 409 as uncertain, so spawn_continue returns a hard error while the original run is live · admission.py:674 — a _coordinator_cancel_pending entry is re-enqueued and _drain_queue returns without scheduling any retry when the queue holds only such entries, so it is parked indefinitely: executed, queue after 5 drains: ['aaaa1111'], batch_members_pending('wave-9') = True forever (which also switches off the reaper's stuck-wave backstop, since _sweep_stuck_waves_impl continues when contains_batch() is True), _conversation_busy permanently blocked, _queued_depth permanently overstated, N heartbeats renewing leases for runs that will never start · subagent_command_authority.py:158_coalesce's cleanup only pops _inflight when the task is already done(), but the await is asyncio.shield, so every cancelled request (client disconnect, MCP POST timeout) leaks a dict entry forever and close() neither cancels nor awaits pending _inflight tasks — executed: 3 cancelled requests still ran manager.spawn and left 3 leaked entries; with a 5 s finish_command, close() returned while the abandoned task was pending and the durable command was left CLAIMED with the side effect applied · json.loads of stored result_json is unguarded in _decode_execution_result/_decode_control_result while the sibling payload_json decode is wrapped, so a corrupt row makes the /api/spawn/commands/{key} recovery endpoint — whose entire purpose is resolving a lost mutation — return an unhandled 500 · _start_execution_heartbeat's renew loop is except Exception: continue with no logging and no attempt bound, and on renew() returning False the task exits leaving a stale CommandFence in _waiting_executions (feeding finding 1's loop) while the lapsed claim lets a second claimant invoke the manager again · admit_reserved re-scans the whole stagger queue for a collision reserve_coordinator_run_id already excluded, making 24 lines of the hardest-to-test durable-rejection code effectively unreachable in production · announce_durable_rejection is invoked from four places with three different spellings of the same guard · _validated_command_identity returns a 4-tuple whose last two members are dead at every call site, and five handlers index [0]/[1] positionally at ten places · AdmittedExecution is a hand-mirrored copy of eight SubagentInfo fields that has already drifted · reuse: api_spawn_release hand-rolls body parsing instead of read_bounded_json and loses the 64 KiB pre-decode cap (so an authenticated caller can make the endpoint buffer and parse a 60 MiB body on the event loop for four fixed hex fields), canonical-JSON+sha256 hashing is open-coded in three places across a trust boundary while platform/admission.canonical_signing_bytes exists as "the ONE canonicalization", _redact duplicates security.redact, _redact_result is a third recursive walker, run_coordinator_anchor re-implements sandbox.prime_voice_runtime_sandbox_paths's cached link-checked directory machinery, and submit_control/finish_control duplicate submit/finish_command and then mirror that split across models.py, sqlite.py and shadow.py (eight near-identical bodies).


Execution-verified AI-assisted review (Claude Code). Ten parallel angles ran against a local checkout at dd2b27c6b and each verified its own findings by execution; this summary is my consolidation of their reports, so treat the prioritisation as mine and the evidence as theirs. Nothing was modified in the tree and nothing was posted elsewhere. Verified clean, so please don't re-spend: normalize_target="target" in body cannot persist an unvalidated or stale target (kind is immutable by construction and patch only copies present fields); replace_terminal_id appears nowhere in the base branch, so its 409 is a forward guard for an unimplemented feature rather than a dead frontend path; redact_credentials in normalize_pull_request_target has no false positives across 8 realistic URLs; and the exact-replay guards after PENDING and after CLAIMED both correctly refuse to re-spawn, so I found no exact-replay duplicate-child path — the CLAIMED guard and the reservation fence hold.

@kyleseaman

Copy link
Copy Markdown
Collaborator Author
  • Fixed — span=residual/crash-data-loss-corruption

    Drained keyed runs lose their durable-waiting state.

    Fixed in 153257e7e: a coordinator-admitted spawn re-entering from the queue now retains its durable-waiting state until start settlement. The regression simulates an uncertain drained start and proves cancellation durably rejects the still-claimed command. The focused top-stack verification passed with 408 tests and one valid platform skip, plus Black, subprocess-encoding, flake8, Linux mypy, and diff gates.

@kyleseaman

Copy link
Copy Markdown
Collaborator Author

Bolin review disposition for the current stack

All fifteen numbered items from the September 2 review were rechecked.

  1. Fixed.
  2. Fixed.
  3. Fixed.
  4. Fixed.
  5. Fixed.
  6. Fixed.
  7. Fixed.
  8. Fixed.
  9. Fixed.
  10. Fixed.
  11. Fixed.
  12. Fixed.
  13. Fixed.
  14. Rebutted as a bounded implementation tradeoff: SQLite access is serialized through the dedicated coordinator executor, and the legacy importer was separately changed to batch snapshots rather than producing quadratic persistence.
  15. Fixed: conversation-busy release responses now carry the real conversation ID and an explicit releasing marker; no fabricated release ID is returned.

The fixes cover durable control authority, reservation and collision fencing, exact replay, uncertain transport resolution, approval and queue cancellation settlement, bounded shutdown, redaction, and real conversation identity.

Current submitted head: f8ce7e5.

@bolichen97

Copy link
Copy Markdown
Collaborator

Open PR relationship audit

This is a consolidated, point-in-time code-level audit note. It compares complete merge-base diffs and current/merged code; it does not treat a shared topic as duplication or partial coverage as completion.

Relationship findings

  • PR #5277 is OVERLAPPING relative to this PR. The goals differ or the implementations can complement each other; this is not a duplicate claim. Recommended action for PR #5277: MERGE_DISCUSSION. Later phase of the same stack; unblocks the operations this PR intentionally rejects. Files: src/kiro_crew/run_coordinator/memory.py.

No PR, Issue, label, branch, or review state was changed by the relationship-note portion of this audit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: action required A blocking check or review needs attention

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants