refactor: extract subagent lifecycle boundaries - #5278
Conversation
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of All evidence gathered. Emitting the review. First-Principles-Verdict: CONCERNS A backend "refactor preserving public behavior" ships an undeclared recovery-shutdown behavior change plus three frontend/test riders, two byte-matching commits already on main. What this change shipsIntent: split SubagentManager's scheduling and terminal-lifecycle state into explicit boundaries ahead of the durable-coordinator RFC — a restructuring ADDITION.
Watch
Subtractions
[FIRST-PRINCIPLES-REVIEWED] b14126d |
Opus 4.8 Review — ✅ no blocking findingsReviewed Review detailsI've verified the key seams myself: admission/drain/continuation equivalence, the rejection-path Nothing survives falsification at 80+, and I found no new grounded defect in Step 2. No findings. [OPUS-REVIEWED] b14126d Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
Design Review (Fable 5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS Sound strangler-fig extraction, but the slot-release boundary is only half-moved and one shutdown-race behavior change is undeclared in a "behavior-preserving" refactor. Watch
Note: the website/theme and [DESIGN-REVIEWED] b14126d |
02e3abe to
f1250c0
Compare
f1250c0 to
a118799
Compare
a118799 to
f1250c0
Compare
f1250c0 to
25333f8
Compare
25333f8 to
32928f7
Compare
32928f7 to
768bf5f
Compare
768bf5f to
30eceb6
Compare
30eceb6 to
77be7aa
Compare
77be7aa to
fa0c258
Compare
Human judgment recorded@kyleseaman marked the fable AI finding as false positive, not applicable, or explicitly accepted for
This decision applies only to this commit. A new push requires a new judgment. |
|
/ai-review override design f1d3a83: Exact parent-to-head diff is 13 files (668 additions, 162 deletions) limited to the documented subagent extraction; the cited auth, registry, Electron, ACP, and workflow files are inherited base content and absent from this PR diff. |
Human judgment recorded@kyleseaman marked the design AI finding as false positive, not applicable, or explicitly accepted for
This decision applies only to this commit. A new push requires a new judgment. |
The merge-base changed after approval.
f1d3a83 to
344a417
Compare
The merge-base changed after approval.
344a417 to
8a0ac9b
Compare
bolichen97
left a comment
There was a problem hiding this comment.
Review: extract subagent lifecycle boundaries
Reviewed origin/feat/run-coordinator-types...origin/feat/run-coordinator-boundaries (22 files, +813/−369) at 006d948d6. Because the PR's claim is behaviour preservation, the review was run as a differential: four BEFORE/AFTER harnesses against both checkouts, six mutation tests, and the repo's own gates. 5006 tests pass across all 67 files that touch SubagentManager.
Credit where due, verified by execution rather than assumed: callback ordering and the exactly-once tokens really are byte-identical. Identical output on both trees for plain _force_reap event order (tombstone → subagent_done → on_done), a reap racing _run's finally, a reap after _run already released, queue admission/drain, _unqueue queue-depth emits, cancel-recovery on both terminal and healthy paths, call_later delays across all 9 admission/drain state combinations, queued-conversation lookup, wave batch_members_pending, and cancel_all report-owner bookkeeping. A 12-run × 3-slot seeded contention harness with interleaved reaps produced identical running_count, exactly-once reporting and token state across 12 seeds. admission()/take_ready()/continuation_delay() are arithmetically identical to the deleted gates at every boundary (elapsed == stagger, stagger == 0, last_start in the future, max_concurrent == 0), and the moved guards are still armed (removing the slot-token re-arm from reoccupy fails 3 tests; removing the drain stagger gate fails 2; removing the recovery capacity guard fails 3).
One thing is not preserved, and it is the important one.
Blocking
1. subagent_manager/cancellation.py:91 — folding the terminal/shutdown check into the TOP of the new while True: capacity-wait loop puts _shutting_down ahead of raise RuntimeError("no free slot for recovery respawn"), converting "raise and fully finalize+report" into a silent return. The run loses its terminal record, its cancelled tombstone, its failure stat, and its terminal report to the parent.
Reachable because cancel_all_impl sets _shutting_down = True (:263) and then awaits real work — asyncio.gather(*followup_watchers) (:286) and await _announce_followup_failure(...) (:295) — before it reaches the _tasks cancellation loop (:307). A cancel-recovery parked in await asyncio.sleep(0.25) wakes inside that window. Executed with the real cancel_all(), max_concurrent=1, one slot-holding agent, and one agent carrying pending_followups so the slow branch is taken:
AFTER (this PR): info.done = False info.error = '' _finalized = False tombstone = 0 terminal_report = 0
BEFORE (base): info.done = True info.error = 'cancelled (recovery failed)' _finalized = True tombstone = 1 terminal_report = 1
Recovery's finally pops <id>:recovery from _tasks and the interrupted run's own _run finally already popped _tasks[info.id], so cancel_all()'s cancellation loop finds nothing to finalize either, and _resume_guarded's except CancelledError never runs. The run is left done=False forever: the parent card stays "running", cost is never recorded, and with no tombstone written the only recovery is restart orphan reconciliation. Static corroboration: the or self._manager._shutting_down still sitting in the raise guard at :103 is now provably dead code, since :94 returns first. The only test on this path patches _RECOVERY_SLOT_WAIT_SECS to 0.4s and therefore covers only the deadline branch, which is why 5006 tests stay green — and test_subagent_turn_resilience.py:635-641 asserts info.done is True after cancel_all(), the invariant this path now violates.
2. subagent_manager/admission.py:296 — the spawn admission gate no longer routes through _should_stagger_queue, so that documented override/monkeypatch seam is silently dead — directly contradicting the comment this PR adds at subagent.py:1676 ("Cross-boundary calls still route through this facade, preserving overrides and monkeypatch seams"). Executed on both trees with a forcing override installed:
BEFORE: seam consulted 1× → info.queued = True queue_len = 1
AFTER: seam consulted 0× → info.queued = False queue_len = 0
SubagentManager._should_stagger_queue (subagent.py:2117) and _should_stagger_queue_impl (admission.py:599) now have zero production callers. A subclass or integration that overrides admission policy is bypassed with no error — the spawn starts immediately and exceeds the caller's intended concurrency/stagger policy. The PR concedes it by rewriting five test call sites (test_spawn_reasoning_effort.py:378/386 → mgr._running_count = mgr.max_concurrent; test_subagent_context_group_plumbing.py:96/108/116 → patching mgr._scheduler.admission). Out-of-tree overrides get no such rewrite. The docstring that is the module's only prose spec for the gate (including the dynamic-subagent-sizing.md §5.3 citation) now sits on dead code, while the live policy at subagent_scheduler.py:38-43 has none.
3. subagent_manager/run.py:453 — the same bypass for slot release, and here the retained wrapper is a booby trap. All five production release sites were rewritten to _scheduler.release(info) (run.py:453, terminal.py:418, admission.py:519/542/734), leaving _release_slot with zero callers — but it still consumes the one-shot _slot_released token without decrementing anything. Executed at max_concurrent=1:
_release_slot(info) -> True running_count = 1 (token spent)
_scheduler.release(info) -> False running_count = 1 (slot unrecoverable)
admission().slot_free -> False (spawn queue starved forever)
That is exactly the permanent _running_count inflation the token was introduced to prevent, and the docs still point at the footgun: subagent.py:1419 says _slot_released is "claimed via SubagentManager._release_slot" and terminal.py:66 says the slot "has its own one-shot token (_release_slot)" — both now false. _release_slot_impl's own docstring still asserts "that caller decrements _running_count once and drains the queue." Worse, the new test codifies the divergence as intended (test_subagent_boundary_wiring.py:38-42 asserts _release_slot(info) is True and running_count == 1), so a future maintainer reading either the docstring or the test will pair the wrapper with a manual decrement — or call it alone. Three test files still hand-pair _release_slot with a manual decrement, encoding a production idiom this PR deleted. Fix: delete both, or make the wrapper return self._scheduler.release(info) so there is exactly one behaviour.
4. subagent.py:1731 — three of the eight compatibility views became getter-only, and none work on a __new__-built manager, contradicting the spec text this PR adds (docs/system-specs/modules/subagent.md: "Private manager views for the old counter, queue, report-task, and teardown-gate fields remain as compatibility adapters"). Executed on both trees, all eight assignments:
AFTER: _report_tasks / _report_owners / _teardown_gates -> AttributeError: property has no setter
_queue / _running_count / _max_concurrent / … -> AttributeError: _scheduler (on a __new__ instance)
BEFORE: all eight -> OK
_scheduler/_lifecycle were not added to _COMPONENT_TYPES (:1526), so the class's own documented lazy-composition escape hatch (__getattr__, :1537) does not cover them. Reads misreport too: because the property getter itself raises AttributeError, Python re-dispatches to __getattr__ and names the wrapper (AttributeError: _queue) instead of the real missing _scheduler — and getattr(mgr, '_running_count', default) silently returns the default rather than raising, so a getattr-with-default call site reads a fabricated value. The PR had to delete exactly these assignments from test_stop_kill_cancel.py:670/737; out-of-tree callers (this repo is a fork; the deleted lines carry # Fork adaptation comments) get no such fix. The read-only/read-write split is not a rule — it is a census of what happened to break: the 5 writable names are exactly the names something still assigns, and the 3 read-only names are exactly the ones whose last assignments this PR removed.
Should fix
5. subagent_scheduler.py:94 — remove() selects survivors with entry not in removed, i.e. full dict.__eq__ over the whole spawn kwarg set, where the base filtered on _preassigned_id. "task" is the first key inserted by enqueue({...}) and _preassigned_id the last, so CPython's dict_equal starts each comparison by memcmp'ing multi-KB prompt strings — cost now scales with prompt size, which the previous code was immune to. Measured through the unchanged public API on both checkouts (500-deep queue, 13.5 KB tasks, one distinct str per entry):
_unqueue x1: BEFORE 0.049 ms AFTER 0.293 ms (6.0x)
crew_chat.purge_slot loop x500: BEFORE 7.74 ms AFTER 78.08 ms (10.1x)
same at 64 KB tasks: BEFORE 8.39 ms AFTER 474.0 ms (56.5x)
That is synchronous event-loop blocking in a function whose own comments say building a store inline "froze chats and heartbeats." It is also the wrong identity primitive: two value-equal entries are dropped together even though only one matches run_id (executed with {id:True} / {id:1} / {id:'z'} and remove('True') — BEFORE kept ['1','z'], AFTER kept ['z'], evicting an entry with no id match). Latent today because _preassigned_id is a unique uuid. A single-pass partition on the id is faster than BEFORE (measured 0.0019–0.028 ms) and obviously correct.
6. test/test_mcp_gatewayd_coverage.py:1897 — monkeypatch.setattr(gw.asyncio, "to_thread", …) mutates the stdlib asyncio module process-wide (verified: gw.asyncio is sys.modules['asyncio']), and running the probe inline removes the exact guarantee the production comment says the offload exists to protect. gatewayd.py:3937-3941 states "asyncio.all_tasks() must be read ON the loop… calling it inside the worker thread raises RuntimeError and would kill this watchdog on its first probe." Executed: real to_thread → RuntimeError: no running event loop; inline → returns 1. So a refactor that folds task_count into _snapshot_state — the single named regression — leaves both patched tests green, and the await asyncio.to_thread(_write_diagnostic, …) blocking file append now runs on the loop in a repo that maintains scripts/check_sync_io_in_async.py for exactly that rule. Blast radius: pytestmark = xdist_group('mcp_gateway') pins the module into one worker process, so any leaked background task (this file creates and cancels ~20) that calls asyncio.to_thread during the window silently runs blocking IO on the loop — surfacing as a timeout in an unrelated test. And it is inconsistent: test_healthy_server_only_writes_probe_baselines polls 300× against the real executor without the patch. The seam convention already exists (loop.run_in_executor(subprocess_executor(), …) from kiro_crew.executors, used for this exact kind of probe at cron.py:1526); give _zombie_diagnostic an injectable offload and patch that, scoped to gw exactly like this file's own monkeypatch.setattr(gw, "_write_diagnostic", …) two lines earlier.
7. Scope: 310 of the 355 changed test lines are pure black reformatting, and the one genuinely risky change is buried in it. Isolated by black-normalising each base file and diffing: test_mcp_gatewayd_coverage.py 196 changed / 9 semantic; test_stop_kill_cancel.py 118 / 22; test_subagent_context_group_plumbing.py 41 / 14 — ~26% of the whole 1182-line diff. AGENTS.md:358-361 is explicit: "If a file you touched is listed in the baseline, formatting it is welcome but optional — do it in its own commit", and :242 "One logical change per commit." This PR has one commit with no body. The pruning itself is correct and mandatory (I installed the pinned black==26.3.1 — local was 24.10.0, and setup.cfg:161-163 warns they disagree by ~680 files — and confirmed all three files are clean and the gate exits 0), so the defect is the bundling, which is how the process-wide to_thread patch ships unreviewed at 2% of its hunk's diff. It also makes the PR non-revertible in halves.
8. test/test_subagent_context_group_plumbing.py:131 — _max_concurrent = 4 / _running_count = 0 / _spawn_stagger_secs = 0.0 are now INERT, because the admission MagicMock installed at :121 also hijacks take_ready on the drain path. Executed on both trees with hostile values (_max_concurrent=0, _running_count=99, _spawn_stagger_secs=1e9): AFTER the drain still fires; BEFORE drain fired: False. So these tests can no longer catch a capacity or stagger regression, and the mid-test admission.return_value = … flip at :134 exists only to undo the mock the test installed for the spawn half. That is the clearest evidence the extraction made the seam coarser, not finer: one patch point now governs both _spawn_impl and _drain_queue_impl, so every queue-round-trip test must be choreographed in two phases. The same PR shows the better idiom two files over — test_spawn_reasoning_effort.py:378 uses the coupling-free mgr._running_count = mgr.max_concurrent. The fabricated AdmissionDecision(True, False, 0.0) is also an unreachable triple: the real admission() always returns retry_after=None when slot_free is False.
9. subagent_lifecycle.py:105 — close_teardown pops the gate only when it is still the registered one, replacing the base's unconditional pop and weakening the invariant the deleted comment stated ("Set BEFORE the entry is dropped… one arriving after finds no entry, which now means exactly 'nothing left to wait for'"). Executed: open twice for one run id, close with the first gate → g1.set True, g2.set False, left_behind_unset True; the base leaves nothing behind. If two _run(info) cycles for one id ever overlap, waves.py:303 blocks on asyncio.wait_for(gate.wait(), timeout=_RESET_TIMEOUT + 30) on a gate nobody will set, stalling the wave digest. I could not construct a production interleaving (the recovery handshake serialises on asyncio.wait({orig_task}, …)), so this is ranked low — but the guarantee is now strictly weaker than the comment the PR kept, and _teardown_gates has no other pruner, so the dict grows unboundedly over a long-lived gateway. The identity check is arguably the better semantics; it is the undeclared, untested change that is the problem.
Notes
admission.py:381 — delay = decision.retry_after or 0.0 coalesces on falsiness for an Optional[float] whose None and 0.0 cases are both real; dead today only via an unenforced cross-field invariant, and the same file gets it right 240 lines later (:623, :692 both use is not None). If admission() is ever mocked (the PR's own new test does exactly that) or returns slot_free=True, retry_after=None, this silently becomes call_later(0.0, …) — an immediate burst defeating the stagger gate on the very path that enforces it. · subagent.py:1719 — the _spawn_stagger_secs setter silently clamps with max(0.0, value) where the base stored verbatim, so the view does not round-trip (-1.0 → 0.0, and max(0.0, nan) → 0.0); the clamp is redundant with __init__ and produces the only decision divergence in the whole admission machine (54/2520 cases, all requiring now - last_start < 0). · subagent.py:1598 — self._coordinator = coordinator or MemoryRunCoordinator() is written and never read anywhere in src/; the base commit landed a typed coordinator that already defines admission (ObservedState.QUEUED + SubmitRun.accepted), fenced one-shot claims (RunFence/OwnerLease), and idempotent terminal completion (COMPLETION_REPLAY vs OUTCOME_CONFLICT) — and this PR re-implements all three as raw mutable bools (info.done, _finalized, _slot_released) with no fence, version or epoch. Two vocabularies for one lifecycle, added one commit apart, so PR 4's migration must reconcile a bool-flag model against a versioned/fenced one instead of swapping an implementation. · subagent_manager/terminal.py:489 — the new comment says "Resolve through the session facade so this leaf keeps no ACP edge", but all four helpers still resolve to kiro_crew.acp.client at runtime (verified: [f.__module__ for f in h] is ['kiro_crew.acp.client'] × 4). The gate is diff-scoped, so touching this file would have turned it RED; routing through session.py — already baselined for 12 such edges — keeps current == baseline. cron.py:1503-1573 performs the byte-identical import and is baselined. That is a gate defeat, not a boundary fix, and platform_compat.py already owns the receiving API (process_descendants, get_process_start_id, kill_process_tree_pinned). It also makes session.py's untyped positional 4-tuple a public contract for an unrelated subsystem — this PR already appended the 4th element and had to add a throwaway _ at session_lifecycle.py:379, and a swap would type-check (all four are Callable[..., Any]) and pass the suite while the resulting TypeError is swallowed by except Exception: logger.exception("Reaper: SIGKILL failed"). · The eight compat properties have no ratchet, so the spec's "must not regain independent state" is prose only; this repo already ships the mechanism three times (check_agent_sdk_boundary.py + baseline + test, test_workflows_architecture.py's AST layering scan, check_sync_io_in_async.py). · subagent_scheduler.py and subagent_lifecycle.py land at the top level of a 194-module flat namespace while every production importer and consumer lives inside subagent_manager/, which is already a package for exactly this — and inconsistently with run_coordinator/, the sibling this same RFC created one PR earlier as a package. · SlotOwner declares only _slot_released and no id, so the scheduler structurally cannot own the invariant the spec assigns it — it mutates a foreign private attribute and cannot key its own token; the sibling LifecycleInfo in this same PR does carry id, proving the id-keyed shape was available (executed: an external info._slot_released = True desyncs running_count permanently with no way for the scheduler to notice). · SubagentScheduler(stagger_seconds=0.0) at :1572 is a placeholder overwritten 97 lines later through the compat setter, so during __init__ the boundary reports "no stagger" where the base raised AttributeError — loud → silent. · Five mutators for one counter (occupy, reoccupy, try_reoccupy, claim_release, release) with reoccupy public but called only by try_reoccupy three lines below; max_concurrent unclamped while its sibling stagger_seconds is floored, and both sibling pools clamp. · AdmissionDecision is the third class of that name in src/, and two different AdmissionDecision types are consumed ~90 lines apart in one function body; should_queue is fully derivable from the other two fields, which is why the tests can construct a self-inconsistent triple. · Three byte-identical one-shot latch bodies across the two new files, while the repo already has three claim_*() -> bool helpers (slack/gateway.py:981, slack/outbound.py:136, dashboard/crash_dump_store.py:544). · begin_reap/mark_reaped/await_report are one-line statics over state the boundary does not own, and await_report splits an 8-line docstring from the asyncio.shield it explains. · test_subagent_boundary_wiring.py:22 is tautological — assert manager._queue is manager._scheduler.queue can never fail because the property body is that expression; a test that would bite mutates through one side and observes the other. It also breaks on the first queued cancel, since remove() rebinds self.queue to a fresh list. · docs/request-for-change/rfc-durable-run-coordinator.md:23 advances the prose audit basis to "PR 2 commit 53f365a17" — a SHA that resolves to no object in this repo (the real base is 2c7e842c0) — while last-audited: 2026-08-22, audited-at: c8eda3c6f (also unresolvable) and implementation-prs: [] are left stale, against docs/request-for-change/README.md's explicit same-PR rule. docs-lint.sh does not inspect front matter, so CI catches none of it. · terminal.py:489's in-method import dropped the # circular import marker that made it a permitted AUTOSDE.yaml exception — and there is in fact no cycle: verified that importing kiro_crew.session loads neither subagent_manager nor subagent, and both import orders work at module scope.
Gates, for the record: pinned black==26.3.1 clean on all 18 changed files and check_black_formatting.py green; check_agent_sdk_boundary.py, check_sync_io_in_async.py, check_loop_bound_locks.py, check_testpaths_coverage.py, check_lockdown_before_publish.py, check_brand_name.py, check_harness_parity.py, check_changelog_history.py, docs-lint.sh, scrub-lint.sh all pass; pinned flake8 7.1.0 and isort 6.0.0 clean; pinned mypy 1.14.1 --platform linux → "no issues found in 1248 source files"; new modules at 98%/94% coverage; CHANGELOG correctly untouched; the subagent spec updated in the same commit.
Execution-verified AI-assisted review (Claude Code) across 10 angles plus a gap sweep, run against local checkouts of both this branch and its base at 2c7e842c0; nothing modified in either tree, nothing posted elsewhere. Also cleared so you needn't re-check: no BEFORE usage of any of the 8 moved attributes is orphaned (I enumerated all of them); read-modify-write through every view still works; _load_child_process_helpers's 4-tuple unpack order is correct and no test injects a 3-tuple; __init__ ordering is safe; await_report preserves the cancellation contract exactly; and all *_impl methods have __globals__ rebound by bind_component_globals, so kiro_crew.subagent.<name> patches still reach coordinator internals.
|
Bolin review disposition for the current stack Every numbered item from the September 2 review was rechecked.
The final static-gate regressions found during submission were also fixed in this layer: queue-delay calculation no longer references a removed local, and lazy boundary composition is correctly typed. Current submitted head: b14126d. |
Open PR relationship auditThis is a consolidated, point-in-time code-level audit note. It compares complete merge-base diffs and current/merged code; it does not treat a shared topic as duplication or partial coverage as completion. Relationship findings
No PR, Issue, label, branch, or review state was changed by the relationship-note portion of this audit. |
|
🤖 Kiro Crew [operator: dwu96#8c08bee0]: This PR has been inactive for 7+ days. I reviewed the blockers and they require your input rather than a mechanical fix. CI is not the problem: PR Readiness is green on What needs your decision:
Also noted for whoever picks this up later: the When you've addressed these, the pipeline will re-assess on its next cycle. Add |
Problem / Motivation
SubagentManagerowns scheduling, lifecycle arbitration, slot accounting, terminal delivery, and session cleanup in one large state machine.Why it matters
The existing race discipline is strong, but its boundaries need to be explicit before durable coordination can safely take authority.
What changed (motivation → approach → change)
SubagentSchedulerfor admission queue and slot accounting.SubagentLifecyclearound the existing exactly-once terminal and release tokens.Tests
test/test_subagent_scheduler.pytest/test_subagent_lifecycle.pytest/test_subagent_boundary_wiring.pytest/test_mcp_gatewayd_coverage.pyManual verification
Not applicable; behavior is covered through deterministic backend lifecycle tests.
Related Issues
no linked issue: this stack implements the locally reviewed durable run coordinator RFC.
Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)