fix(autopilot): offload orchestrator config load - #3803
Conversation
00e0407 to
f01a922
Compare
GPT 5.6 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
Opus 4.8 Review (fork) — ✅ no blocking findingsReviewed |
First Principles Review (Fable 5, fork) — 🟡 CONCERNSPremise-level review of All claims verified against the base. I have enough to issue the review. First-Principles-Verdict: CONCERNS The offload is justified and precedented, but the early tracker publication plus new setter re-covers a window What this change shipsIntent: stop the orchestrator's first-entry config read from blocking the gateway's only event loop — a FIX.
Watch
Subtractions
[FIRST-PRINCIPLES-REVIEWED] 1624538 |
Design Review (Fable 5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS The offload follows the repo's established off-loop idiom, and the two cancellation races the new await opens are closed with existing signals, not new machinery. Verified against the base tree: Suggestions
[DESIGN-REVIEWED] 1624538 |
f01a922 to
07d79c6
Compare
07d79c6 to
9cae22c
Compare
ac57f30 to
eb7dd0c
Compare
GPT 5.6 disposition —
|
| run time | |
|---|---|
| before | 14.81s |
| after | 5.11s |
The ~10s delta is exactly the stranded worker's release.wait timeout. All three regions now release and reap in finally, so the assertion failure surfaces immediately and the worker never outlives the test. Full file: 11 passed.
Early tracker publication corrupts stage accounting — NOT ADOPTED. The mechanism is unreachable; recommending disposition rather than the suggested revert.
The arithmetic is correct — a tracker carrying a round makes start_idx non-zero and stage 1 is skipped. What does not hold is that this diff can deliver such a round in time.
start_idx is frozen into a local int before the loop's first suspension point:
286 slot._orch_tracker = tracker <- publication
288 total = slot._plan_stage_count
289 titles = getattr(slot, "_stage_titles", [])
292 start_idx = tracker.current_stage if tracker._stage_rounds else 0 <- frozen here
321 await _load_stage_budget(...) <- FIRST await, already past it
Lines 286–292 are two attribute reads and an assignment. Nothing yields, so on the gateway's single-threaded loop nothing else can execute between the publication and the read. Everything the config await opens up happens strictly after start_idx is already captured.
This is also not a window the diff widened. Before it, the load was a synchronous KiroCrewConfig.load() — which does not yield either — so the base had no suspension point in that region and the diff added none before start_idx.
record_round has two callers repo-wide: this loop (after start_idx), and slack/gateway.py:5873. The gateway one does reach the same object, so the lingering-completion actor named in the finding is real — it simply cannot land early enough to move start_idx.
Pinned rather than argued, in two new tests:
test_a_round_recorded_during_the_config_load_cannot_skip_a_stage— holds the loop suspended in the config load (the widest window this diff opens) and injects the gateway's exact call on the freshly published tracker:stage = tracker.current_stage; tracker.record_round(stage). Both stages still execute.test_a_round_recorded_before_loop_entry_does_skip_a_stage— the positive control. A tracker already carrying a round at entry does skip stage 1, so the first test cannot pass byrecord_roundbeing inert. That is the resume path, where_orch_trackeris already set: nothing is published,_bootstrappingis False, and no config load runs. Untouched by this diff.
Reverting the early-publication hunk would also reopen what it exists to close: with the tracker published only after the load, api_chat_plan_action finds slot._orch_tracker is None for the whole window and a plan Cancel stops nothing while still telling the user the plan was cancelled. The alternative is a second cancellation record beside tracker.stopped, which the First Principles lane on this PR asked to remove.
Happy to be shown a path from the gateway's record_round into lines 286–292 if one exists — that would change the disposition.
Every config.json update in dashboard/handlers/memory.py reads the file, changes one key and writes it all back on the event loop. The read, the JSON parse, and write_config_atomically -- a tmp-file write plus a rename, which can fsync -- all block, stalling every other session while they run. Three sites do it: api_memory_settings, _write_embed_model_config, and _set_migrated, which runs on EVERY boot while migrated is false and so lands the stall exactly when the gateway is bringing sessions up. This is the class the repo has been closing site by site (kirodotdev#4118, kirodotdev#3803, kirodotdev#4550), and kirodotdev#4946's review named this module. The whole transaction crosses over, never just the read. Offloading the read alone would leave the write on the loop and insert a suspension point between the read and the write-back while the file is unguarded on disk: an external editor, a CLI command or another process landing in that gap would be silently overwritten by a write derived from state nobody re-checked. That gap is zero today because the sequence is synchronous, and it stays zero because the worker performs the whole thing without yielding. The existing per-config lock is held across the hop, so two coroutines still cannot interleave. Two of the three sites also hand-rolled a reader this module already imports. read_config_for_update is the documented companion to write_config_atomically with 27 call sites, and api_memory_settings uses it 200 lines above; _set_migrated and _write_embed_model_config instead caught Exception around json.loads. The helper additionally refuses a non-object top level, where the hand-rolled version accepted a list and then raised AttributeError from setdefault -- a crash where a fail-closed refusal was intended. ConfigReadError is not swallowed by the helper: what to tell the user differs per site, and each keeps exactly the behaviour it had -- skip and retry next boot, raise ValueError, or answer 500 config_unreadable. api_memory_settings now validates its body before the transaction. None of that reads the config, and a 400 previously took the lock and abandoned it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
_write_env_updates stats and reads the whole .env, re-parses it line by line, then creates a 0600 temp file, chmods it, writes and renames. All synchronous file I/O, and six async channel config-save handlers call it. Three already reached it through asyncio.to_thread -- telegram, teams, wecom -- and three called it inline on the gateway loop: slack, discord and webex, stalling every other session for the duration of a token save. So this is not a missing convention but an existing one applied to half the call sites. messaging.py already uses asyncio.to_thread 29 times, and with three siblings doing it correctly nothing in the file said which half was right, or stopped the next channel from copying the wrong one. It is the class the repo has been closing site by site (kirodotdev#4118, kirodotdev#3803, kirodotdev#4550), and kirodotdev#4946's review named this module. The WHOLE call is offloaded, never a part of it: the read-modify-write is one transaction, and a suspension point between the read and the rename would let a concurrent writer's keys be dropped by a write derived from lines nobody re-read. Keeping _write_env_updates one synchronous function on one worker preserves that without depending on the caller, which is now said on the function itself so the next channel inherits the reason and not just the shape. The regression pins all six channels rather than the three that moved, since the defect was the split and not any one site. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every config.json update in dashboard/handlers/memory.py reads the file, changes one key and writes it all back on the event loop. The read, the JSON parse, and write_config_atomically -- a tmp-file write plus a rename, which can fsync -- all block, stalling every other session while they run. Three sites do it: api_memory_settings, _write_embed_model_config, and _set_migrated, which runs on EVERY boot while migrated is false and so lands the stall exactly when the gateway is bringing sessions up. This is the class the repo has been closing site by site (kirodotdev#4118, kirodotdev#3803, kirodotdev#4550), and The whole transaction crosses over, never just the read. Offloading the read alone would leave the write on the loop and insert a suspension point between the read and the write-back while the file is unguarded on disk: an external editor, a CLI command or another process landing in that gap would be silently overwritten by a write derived from state nobody re-checked. That gap is zero today because the sequence is synchronous, and it stays zero because the worker performs the whole thing without yielding. The existing per-config lock is held across the hop, so two coroutines still cannot interleave. Two of the three sites also hand-rolled a reader this module already imports. read_config_for_update is the documented companion to write_config_atomically with 27 call sites, and api_memory_settings uses it 200 lines above; _set_migrated and _write_embed_model_config instead caught Exception around json.loads. The helper additionally refuses a non-object top level, where the hand-rolled version accepted a list and then raised AttributeError from setdefault -- a crash where a fail-closed refusal was intended. ConfigReadError is not swallowed by the helper: what to tell the user differs per site, and each keeps exactly the behaviour it had -- skip and retry next boot, raise ValueError, or answer 500 config_unreadable. api_memory_settings now validates its body before the transaction. None of that reads the config, and a 400 previously took the lock and abandoned it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every config.json update in dashboard/handlers/memory.py reads the file, changes one key and writes it all back on the event loop. The read, the JSON parse, and write_config_atomically -- a tmp-file write plus a rename, which can fsync -- all block, stalling every other session while they run. Three sites do it: api_memory_settings, _write_embed_model_config, and _set_migrated, which runs on EVERY boot while migrated is false and so lands the stall exactly when the gateway is bringing sessions up. This is the class the repo has been closing site by site (kirodotdev#4118, kirodotdev#3803, kirodotdev#4550), and The whole transaction crosses over, never just the read. Offloading the read alone would leave the write on the loop and insert a suspension point between the read and the write-back while the file is unguarded on disk: an external editor, a CLI command or another process landing in that gap would be silently overwritten by a write derived from state nobody re-checked. That gap is zero today because the sequence is synchronous, and it stays zero because the worker performs the whole thing without yielding. The existing per-config lock is held across the hop, so two coroutines still cannot interleave. Two of the three sites also hand-rolled a reader this module already imports. read_config_for_update is the documented companion to write_config_atomically with 27 call sites, and api_memory_settings uses it 200 lines above; _set_migrated and _write_embed_model_config instead caught Exception around json.loads. The helper additionally refuses a non-object top level, where the hand-rolled version accepted a list and then raised AttributeError from setdefault -- a crash where a fail-closed refusal was intended. ConfigReadError is not swallowed by the helper: what to tell the user differs per site, and each keeps exactly the behaviour it had -- skip and retry next boot, raise ValueError, or answer 500 config_unreadable. api_memory_settings now validates its body before the transaction. None of that reads the config, and a 400 previously took the lock and abandoned it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every config.json update in dashboard/handlers/memory.py reads the file, changes one key and writes it all back on the event loop. The read, the JSON parse, and write_config_atomically -- a tmp-file write plus a rename, which can fsync -- all block, stalling every other session while they run. Three sites do it: api_memory_settings, _write_embed_model_config, and _set_migrated, which runs on EVERY boot while migrated is false and so lands the stall exactly when the gateway is bringing sessions up. This is the class the repo has been closing site by site (kirodotdev#4118, kirodotdev#3803, kirodotdev#4550), and The whole transaction crosses over, never just the read. Offloading the read alone would leave the write on the loop and insert a suspension point between the read and the write-back while the file is unguarded on disk: an external editor, a CLI command or another process landing in that gap would be silently overwritten by a write derived from state nobody re-checked. That gap is zero today because the sequence is synchronous, and it stays zero because the worker performs the whole thing without yielding. The existing per-config lock is held across the hop, so two coroutines still cannot interleave. Two of the three sites also hand-rolled a reader this module already imports. read_config_for_update is the documented companion to write_config_atomically with 27 call sites, and api_memory_settings uses it 200 lines above; _set_migrated and _write_embed_model_config instead caught Exception around json.loads. The helper additionally refuses a non-object top level, where the hand-rolled version accepted a list and then raised AttributeError from setdefault -- a crash where a fail-closed refusal was intended. ConfigReadError is not swallowed by the helper: what to tell the user differs per site, and each keeps exactly the behaviour it had -- skip and retry next boot, raise ValueError, or answer 500 config_unreadable. api_memory_settings now validates its body before the transaction. None of that reads the config, and a 400 previously took the lock and abandoned it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every config.json update in dashboard/handlers/memory.py reads the file, changes one key and writes it all back on the event loop. The read, the JSON parse, and write_config_atomically -- a tmp-file write plus a rename, which can fsync -- all block, stalling every other session while they run. Three sites do it: api_memory_settings, _write_embed_model_config, and _set_migrated, which runs on EVERY boot while migrated is false and so lands the stall exactly when the gateway is bringing sessions up. This is the class the repo has been closing site by site (kirodotdev#4118, kirodotdev#3803, kirodotdev#4550), and The whole transaction crosses over, never just the read. Offloading the read alone would leave the write on the loop and insert a suspension point between the read and the write-back while the file is unguarded on disk: an external editor, a CLI command or another process landing in that gap would be silently overwritten by a write derived from state nobody re-checked. That gap is zero today because the sequence is synchronous, and it stays zero because the worker performs the whole thing without yielding. The existing per-config lock is held across the hop, so two coroutines still cannot interleave. Two of the three sites also hand-rolled a reader this module already imports. read_config_for_update is the documented companion to write_config_atomically with 27 call sites, and api_memory_settings uses it 200 lines above; _set_migrated and _write_embed_model_config instead caught Exception around json.loads. The helper additionally refuses a non-object top level, where the hand-rolled version accepted a list and then raised AttributeError from setdefault -- a crash where a fail-closed refusal was intended. ConfigReadError is not swallowed by the helper: what to tell the user differs per site, and each keeps exactly the behaviour it had -- skip and retry next boot, raise ValueError, or answer 500 config_unreadable. api_memory_settings now validates its body before the transaction. None of that reads the config, and a 400 previously took the lock and abandoned it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…5059) Every config.json update in dashboard/handlers/memory.py reads the file, changes one key and writes it all back on the event loop. The read, the JSON parse, and write_config_atomically -- a tmp-file write plus a rename, which can fsync -- all block, stalling every other session while they run. Three sites do it: api_memory_settings, _write_embed_model_config, and _set_migrated, which runs on EVERY boot while migrated is false and so lands the stall exactly when the gateway is bringing sessions up. This is the class the repo has been closing site by site (#4118, #3803, #4550), and The whole transaction crosses over, never just the read. Offloading the read alone would leave the write on the loop and insert a suspension point between the read and the write-back while the file is unguarded on disk: an external editor, a CLI command or another process landing in that gap would be silently overwritten by a write derived from state nobody re-checked. That gap is zero today because the sequence is synchronous, and it stays zero because the worker performs the whole thing without yielding. The existing per-config lock is held across the hop, so two coroutines still cannot interleave. Two of the three sites also hand-rolled a reader this module already imports. read_config_for_update is the documented companion to write_config_atomically with 27 call sites, and api_memory_settings uses it 200 lines above; _set_migrated and _write_embed_model_config instead caught Exception around json.loads. The helper additionally refuses a non-object top level, where the hand-rolled version accepted a list and then raised AttributeError from setdefault -- a crash where a fail-closed refusal was intended. ConfigReadError is not swallowed by the helper: what to tell the user differs per site, and each keeps exactly the behaviour it had -- skip and retry next boot, raise ValueError, or answer 500 config_unreadable. api_memory_settings now validates its body before the transaction. None of that reads the config, and a 400 previously took the lock and abandoned it. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
|
This PR materially overlaps #3771 in Please use this config-read/stop-gate shape as the shared baseline, then replay #3771's result-write offload and both regression seams. Verify cancellation/stop gates remain between the offloaded phases and that neither config nor result-file I/O runs synchronously on the event loop. |
The first stage loop for a slot has no OrchestrationTracker yet and builds one from the configured stage timeout. That read goes through KiroCrewConfig.load(), which stats and reads config.json plus any config.local.json overlay, deep-merges them and runs the full jsonschema validation -- synchronously, on the single event loop the gateway shares with every other session. Only the load crosses to a worker; the value is applied and the slot read back on the loop, so no live orchestration state is handed to a thread. That hop is the loop's FIRST suspension point, ahead of every stage gate, so it opens a window on both cancellation channels -- and neither necessarily leaves state a plain re-read can see. Plan Cancel stops slot._orch_tracker. Building the tracker from the load's result leaves that None for the whole window, so the press stops nothing while still telling the user the plan was cancelled. The tracker is therefore published BEFORE the load and the configured budget applied afterwards, which makes tracker.stopped -- the canonical plan-cancel signal every advancement gate already reads through _orchestration_stopped -- cover this window too. A second cancellation record kept beside it would be one more thing to drift. Nothing consults the budget until a stage records its first round, which cannot happen before the load returns, so OrchestrationTracker grows a setter for it; the tracker starts on its own default, the same 1800s this loop fell back to when the load raised. Dashboard Stop has no ACP turn to cancel yet, so stop_turn answers "idle" and the handler releases _stop_state straight back to "idle" -- re-reading _stopping afterwards shows an unstopped slot. slot._stop_generation counts stop INITIATIONS and is never rewound, so it is snapshotted before the wait instead: the same reading, for the same reason, as the poisoned- conversation canary in chat_runner. Abandoning the plan is not abandoning the slot. While the config loads the stage loop owns slot.task, so api_chat queues a user message rather than running it, and _start_next_queued_turn HOLDS it behind _in_stage_execution so it can never run alongside a plan. Only the loop's own finally clears that guard and hands the queue on, so the abort returns from INSIDE the try and leaves through it, exactly as a cancel at stage 0 does. A bare return would strand the message behind a guard with no loop left to clear it, where a later prompt could overtake it and a restart could drop it. That finally then had to stop counting the loop's own task as a live turn. slot.running asks whether a turn a stage STARTED still holds the slot -- _run_chat publishes its own task and clears it when the turn ends -- but where no stage turn ever ran, slot.task is still the stage loop itself, alive by definition inside its finally. The raw read reported a turn that did not exist and skipped both the queue handoff and the idle close on the two paths with nothing else to perform them: an abandoned bootstrap, and a plan with no stages. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
eb7dd0c to
1624538
Compare
bolichen97
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: fix (4 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: offloads the blocking orchestrator config load onto a worker thread with cancel/stop-safe re-check, keeping the default budget on load failure (clear root cause: blocking load on the event loop). CodeQL is not applicable on this fork PR (default-setup emits no check-run); SAST coverage is Semgrep only, latest run success with 0 annotations.
bolichen97
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: fix (4 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: moves the orchestrator's first-stage KiroCrewConfig.load off the gateway event loop (it stats and reads config.json plus the local overlay and runs full schema validation synchronously), publishing the tracker on its default budget first and applying the loaded value after, with both cancellation channels re-checked across the worker hop via a non-rewound _stop_generation snapshot. CodeQL is not applicable on this fork PR (default-setup emits no check-run); SAST coverage is Semgrep only, latest run success with 0 annotations.
Problem / Motivation
_stage_loopinsrc/kiro_crew/dashboard/chat_orchestrator.pyis anasync def,but its first act for a slot that has no tracker yet is a synchronous config read:
KiroCrewConfig.load()is not a cheap accessor. It resolves the config path,exists()/read_text()/json.loads()onconfig.json,is_file()/stat()andmerges any
config.local.jsonoverlay, then runs the fulljsonschemavalidation.None of that yields, so it occupies the event loop for its whole duration.
Being accurate about the cost: the loader keeps a fingerprint cache, so a warm
call short-circuits. The blocking read is the cold path — a freshly started
gateway, or the first plan after either config file is written.
Why it matters
Scheduling only. The gateway runs one event loop, and while this read is in
flight nothing else on it advances — no other slot's turn, no WebSocket
broadcast, no HTTP handler. It lands at plan startup, which is exactly when the
user has just clicked Go and the dashboard is expected to be responsive.
No latency figure is claimed; none was measured.
What changed (motivation → approach → change)
Motivation — take the filesystem and schema work off the shared loop without
altering a single initialisation semantic.
Approach — this is not a new idiom.
await asyncio.to_thread(KiroCrewConfig.load)is already the established form on more than a dozen async paths, four of them in
this same package (
chat_handlers.py,chat_mirror.py,chat_slack.py,chat_summary.py), plusapps/registry.pyandhandlers/agents.py. This callsite is the outlier, not the precedent.
Change — the load, and only the load, moves to a worker:
No slot, tracker,
DashboardState, stage titles, or messages cross the boundary.No generic config executor, no caching layer, no change to reload semantics, and
no change to any default. The existing broad
except Exceptionis kept as-is —an exception raised inside
to_threadstill propagates through theawaitintothe same handler.
One deliberate non-change: cancelling an awaited
to_threaddoes not stop aworker that has already started. That is acceptable here only because
KiroCrewConfig.loadis read-only, so an abandoned load has no effect to undo.No cancellation machinery is added for it.
The suspension point it introduces opens TWO cancellation races, and neither
leaves state a plain re-read can observe. It is the loop's first
await, ahead ofevery
if slot._stopping: break.1. Dashboard Stop. No ACP turn exists yet, so
SessionManager.stop_turnanswers
"idle"andapi_chat_slot_stopreleases the posture it just claimed:2. Plan Cancel — the plan card's own control, a different domain: it stops the
orchestration tracker, not the ACP turn. During this await the tracker does not
exist yet, so the handler stops nothing while still printing "Plan cancelled",
broadcasting
chat_done, and returning{"cancelled": true}:_auto_runcannot serve as the signal:_stage_loopcapturedauto_runas anargument when the task was created, so a Go All keeps its own
Trueand aplain Go was already
False— neither can distinguish a cancel.The fix publishes the tracker BEFORE the await, and reuses the signal
mainalready has. Plan Cancel's entire problem is that
slot._orch_trackerisNonefor the length of the window, so the tracker is built first — on
OrchestrationTracker's own default budget, the same 1800s this loop already fellback to when the load raised — and the configured value applied once the load
returns. The press then reaches a real tracker, and the check afterwards is
_orchestration_stopped(slot, tracker): the helper #4811 introduced, which everyadvancement gate in this loop already reads.
That matters more than it looks. An earlier revision of this PR carried a second
counter,
slot._plan_cancel_generation, alongsidetracker.stopped. Two recordsof one event is two things to keep in step, and First Principles was right to flag
it: a cancel path updating one and not the other would diverge silently, in the
one place the codebase had just finished centralising. That counter is gone, and
state.pynow carries no diff at all.Nothing consults the stage budget until a stage records its first round, which
cannot happen before the load returns, so
OrchestrationTrackergrows a setter forit — documented as safe at exactly that point, and not mid-stage, where it would
move a deadline the current stage is already being measured against.
Dashboard Stop keeps a generation snapshot, because that channel genuinely has
nothing live to re-read: the posture settles all the way back to
"idle"insidethe await.
slot._stop_generationis not this PR's invention — it ismain'sexisting counter of stop INITIATIONS, added for precisely this shape, and
chat_runner's poisoned-conversation canary is the existing consumer reading itthe same way for the same reason.
The stop posture and its card stay owned by the stop handler: nothing here writes
_stop_state,_stop_generation, or the card, and neitherapi_chat_slot_stopnor
SessionManageris touched. No shield, executor, lock, or new cancellationtoken is introduced.
Abandoning the plan is not abandoning the slot. This is the second thing the
earlier revision got wrong: it left the bootstrap with a bare
return, takenbefore the loop's
try.While the config loads, the stage loop owns
slot.task, soapi_chatqueues auser message rather than running it, and
_start_next_queued_turnholds itbehind
_in_stage_executionso it can never run alongside a plan. The stageloop's own
finallyis the single thing that clears that guard and hands thequeue on. Returning before the
tryskipped all of it: the guard stayed set on aslot with no loop left to clear it, and the user's message stayed parked behind
it — where a later prompt could overtake it, and a restart could drop it.
The abort therefore returns from inside the
tryand leaves through that samefinally, exactly as a cancel landing at stage 0 does: guard cleared, queuehanded off, slot closed out.
Which exposed one more thing the
finallyhad to stop doing. Itsnot slot.runningguard is asking whether a turn a stage STARTED still holds theslot —
_run_chatpublishes its own task onslot.taskand clears it when theturn ends. It is not asking about this loop. Yet where no stage turn ever ran,
slot.taskis still the stage loop itself, alive by definition inside its ownfinally, so the raw read reported a live turn that does not exist and skippedboth the queue handoff and the idle close — on precisely the two paths with
nothing else to perform them: an abandoned bootstrap, and a plan with no stages.
It now compares against
asyncio.current_task(). Wherever a stage turn did run,slot.taskis alreadyNoneand the behaviour is unchanged.One existing assertion is rescoped.
test_no_worker_hop_when_no_stage_results_were_capturedrecorded every
asyncio.to_threadcall the stage loop made and required the listto be empty — a valid proxy only while the completion read was the loop's sole
offload. Its subject, per its own docstring, is that an empty result set skips
the excerpt worker, so it now filters the recorded calls to
_completion_excerptsrather than forbidding every hop. It has not been weakened: deleting the
production
if _captured:guard still fails it(
assert [<function _completion_excerpts>] == []).Tests
New
test/test_orchestrator_config_load_off_loop.pydrives the real_stage_loopthrough tracker initialisation. For the offload cases the slot is given an empty
title list, so
_plan_stage_countis zero and the loop initialises the tracker andthen has no stage to run — the branch under test is exercised with no model turn
involved. The Stop-race case gives the slot a real one-stage plan instead, so
reaching stage execution is observable.
The seam is the loader itself:
chat_orchestratorbindsKiroCrewConfigatimport, so the test replaces that module attribute with a wrapper that records
threading.get_ident()and delegates to the real classmethod. The recordedthread is therefore the one that genuinely stats, reads, merges and validates —
not merely one that reached a call site — and the seam behaves identically before
and after the change. The repo's conftest already pins
KIROCREW_HOMEto aper-test tmp dir, so the real loader reads controlled files rather than the
developer's own configuration.
Fail-before / pass-after, each row against its own baseline:
test_config_load_runs_off_the_loop_threadassert 61188 not in [61188]test_stop_during_the_config_load_does_not_start_the_plantest_plan_cancel_during_the_config_load_does_not_start_the_plantest_a_plan_started_after_a_cancel_still_runstest_a_message_queued_during_the_config_load_is_still_handed_offassert [] == ['queued-message-slot']The first row is measured on pristine main at the branch's original base (
90348e1d4); the branch has since been rebased onto current main. The second ismeasured against the offload commit instead: it is a regression the new
awaitmakes reachable, so there is nothing for it to fail on a tree that has no
suspension point there.
That case is ordered with
threading.Events, never sleeps. The loader blocks insidethe worker; the test then presses Stop through the real
api_chat_slot_stop(against a
SessionManagerseam returning"idle", the outcome that creates therace) and asserts the posture has settled all the way back —
_stop_generationincremented by exactly one,
_stop_state == "idle",_auto_runcleared — beforereleasing the loader. Only then does it assert the behaviour that matters: no stage
turn ran, and no tracker was installed.
Mutation-checked, so the guard is doing the work and not the fixture:
if False:a stage turn ran after the user stopped the planif slot._stopping:(the naive fix)The third row is the point: the Stop has already resolved back to
"idle"by thetime the loop looks, so the
_stop_generationhalf is the load-bearing one. Theother five cases stayed green under every mutation.
The two mechanisms this revision changed carry their own negative controls,
each reverting exactly one thing and leaving the rest in place:
Cancel has nothing to stop while the config loads: the tracker must be published before the await, or this window needs its own cancel recordreturnbefore thetry(the earlier shape)the message the user queued during the config load was never handed off; it is stranded on the slot with no loop left to release ittry, but leave thefinally's rawslot.runningreadassert [] == ['queued-message-slot']Two findings from the exact-head GPT review are dispositioned on
eb7dd0cac.Worker cleanup occurs only after assertions — adopted.
release.set()satafter the assertions in all three gated tests, so a failure left the loader thread
parked in
release.wait(timeout=10)past teardown and it then completed the realKiroCrewConfig.load()outside the per-testKIROCREW_HOME. Measured by forcingone assertion inside the parked region to fail: 14.81s before, 5.11s after —
the delta is exactly the stranded worker's timeout. All three now release and reap
in
finally.Early tracker publication corrupts stage accounting — not adopted; the
mechanism is unreachable.
start_idxis frozen into a localintbefore theloop's first
await, with only two attribute reads between it and thepublication, so nothing can record a round in that gap on a single-threaded loop.
The base had no suspension point there either (the load was synchronous), so the
diff widened nothing before
start_idx. Pinned rather than argued, by two tests:test_a_round_recorded_during_the_config_load_cannot_skip_a_stageinjects theother
record_roundcaller's exact call on the freshly published tracker whilethe loop is held in the config load, and both stages still run;
test_a_round_recorded_before_loop_entry_does_skip_a_stageis the positivecontrol, so the first cannot pass by
record_roundbeing inert. Full dispositionin the PR thread.
The last two are separate reverts that fail identically, which is the point: the
handoff needs BOTH the exit path and the corrected guard, so neither is
load-bearing alone and neither is decoration.
The queued-message case is ordered with events, never sleeps. The loader blocks
in the worker, the message is queued while it is parked, and Cancel is pressed
through the real endpoint before the loader is released. It observes the handoff
at the seam the loop actually calls (
_start_next_queued_turn) rather than byrunning a turn, and it asserts the handoff BEFORE the guard state, so a build
that merely clears the flag without draining the queue still fails.
Being precise about the rest: the other four offload cases pass on both trees. They
are preservation coverage, not evidence of a defect, and are reported as such
rather than folded into the fail-before count:
stage_timeout_seconds: 42, written as a realconfig.jsonandread by the real loader, reaches the tracker verbatim;
0survives as0— the stage loop reads it back asif tracker.stage_timeout_seconds:, so collapsing it onto 1800 with anorwould silently re-enable a ceiling the operator turned off;
The shard-1 reproduction quoted here previously was measured on the branch's
ORIGINAL base. This head is a rebase onto current
main(554 commits) plus aredesign, so that number no longer describes it and is not restated; the runs
above are what was actually executed against this head. The rescoped assertion
in
test_completion_result_read_off_loop.pyis still the only pre-existing testthis branch touches, and it passes here.
Local-only failures on this Windows box are unchanged in class and unrelated to
this diff (
WinError 1314symlink-privilege,cp950decode); none occur on theCI runners.
Also green:
flake8,isort --check-only,mypyon both changed modules("Success: no issues found in 2 source files"), and
scripts/check_black_formatting.py(4 files in scope; the new test file is NOTbaselined, so it is held to
blackand formatted).Manual verification
N/A — unit coverage sufficient, and the coverage is not narrow: both cancellation
paths are driven through their REAL endpoints (
api_chat_slot_stop,api_chat_plan_action) against the real_stage_loop, with the config workerparked on an event so the ordering is forced rather than raced. Reproducing this
by hand needs a Cancel or Stop pressed inside a config load that normally
completes in milliseconds, which is not something a person can time.
Two production files:
chat_orchestrator.py(the bootstrap, the abort's exitpath, and the
finally's live-turn test) and a setter onOrchestrationTracker.state.pycarries no diff.Related Issues
N/A — self-discovered while auditing the orchestrator's remaining synchronous I/O
boundaries. #1783 is not cited: its only
stage_timeout_secondsbullet asks for aseparate total-plan duration watchdog and says nothing about this config load, so
a
Refsthere would misattribute the work.Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)_load_stage_budget's docstring and thefinally's comment.Contribution License Agreement