Skip to content

fix(autopilot): offload orchestrator config load - #3803

Merged
bolichen97 merged 1 commit into
kirodotdev:mainfrom
leonlaiyc:fix/orchestrator-config-load-offloop
Sep 1, 2026
Merged

fix(autopilot): offload orchestrator config load#3803
bolichen97 merged 1 commit into
kirodotdev:mainfrom
leonlaiyc:fix/orchestrator-config-load-offloop

Conversation

@leonlaiyc

@leonlaiyc leonlaiyc commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

Problem / Motivation

_stage_loop in src/kiro_crew/dashboard/chat_orchestrator.py is an async def,
but its first act for a slot that has no tracker yet is a synchronous config read:

tracker = slot._orch_tracker
if tracker is None:
    try:
        _timeout = KiroCrewConfig.load().orchestrator.stage_timeout_seconds
    except Exception:
        _timeout = 1800
    tracker = OrchestrationTracker(stage_timeout_seconds=_timeout)

KiroCrewConfig.load() is not a cheap accessor. It resolves the config path,
exists()/read_text()/json.loads() on config.json, is_file()/stat() and
merges any config.local.json overlay, then runs the full jsonschema validation.
None of that yields, so it occupies the event loop for its whole duration.

Being accurate about the cost: the loader keeps a fingerprint cache, so a warm
call short-circuits. The blocking read is the cold path — a freshly started
gateway, or the first plan after either config file is written.

Why it matters

Scheduling only. The gateway runs one event loop, and while this read is in
flight nothing else on it advances — no other slot's turn, no WebSocket
broadcast, no HTTP handler. It lands at plan startup, which is exactly when the
user has just clicked Go and the dashboard is expected to be responsive.

No latency figure is claimed; none was measured.

What changed (motivation → approach → change)

Motivation — take the filesystem and schema work off the shared loop without
altering a single initialisation semantic.

Approach — this is not a new idiom. await asyncio.to_thread(KiroCrewConfig.load)
is already the established form on more than a dozen async paths, four of them in
this same package (chat_handlers.py, chat_mirror.py, chat_slack.py,
chat_summary.py), plus apps/registry.py and handlers/agents.py. This call
site is the outlier, not the precedent.

Change — the load, and only the load, moves to a worker:

loop:       construct OrchestrationTracker (default budget)
            assign slot._orch_tracker        <- published BEFORE the await
worker:     KiroCrewConfig.load()
loop:       tracker.stage_timeout_seconds = cfg.orchestrator....

No slot, tracker, DashboardState, stage titles, or messages cross the boundary.
No generic config executor, no caching layer, no change to reload semantics, and
no change to any default. The existing broad except Exception is kept as-is —
an exception raised inside to_thread still propagates through the await into
the same handler.

One deliberate non-change: cancelling an awaited to_thread does not stop a
worker that has already started. That is acceptable here only because
KiroCrewConfig.load is read-only, so an abandoned load has no effect to undo.
No cancellation machinery is added for it.

The suspension point it introduces opens TWO cancellation races, and neither
leaves state a plain re-read can observe. It is the loop's first await, ahead of
every if slot._stopping: break.

1. Dashboard Stop. No ACP turn exists yet, so SessionManager.stop_turn
answers "idle" and api_chat_slot_stop releases the posture it just claimed:

_stage_loop                     dashboard Stop
  await to_thread(load)   -->
                                _stop_state = "soft_pending"   (generation +1)
                                stop_turn(...) -> "idle"       (no turn exists yet)
                                _stop_state = "idle"           (posture released)
  load returns            <--
  slot._stopping is False -->   stage execution resumes

2. Plan Cancel — the plan card's own control, a different domain: it stops the
orchestration tracker, not the ACP turn. During this await the tracker does not
exist yet, so the handler stops nothing while still printing "Plan cancelled",
broadcasting chat_done, and returning {"cancelled": true}:

Go / Go All -> _stage_loop -> config worker blocks
                                 user clicks Cancel
                                 tracker is None -> tracker.stop() never runs
                                 _auto_run = False
                                 user is told the plan was cancelled
             config returns
             -> tracker installed -> the cancelled stage executes

_auto_run cannot serve as the signal: _stage_loop captured auto_run as an
argument when the task was created, so a Go All keeps its own True and a
plain Go was already False — neither can distinguish a cancel.

The fix publishes the tracker BEFORE the await, and reuses the signal main
already has.
Plan Cancel's entire problem is that slot._orch_tracker is None
for the length of the window, so the tracker is built first — on
OrchestrationTracker's own default budget, the same 1800s this loop already fell
back to when the load raised — and the configured value applied once the load
returns. The press then reaches a real tracker, and the check afterwards is
_orchestration_stopped(slot, tracker): the helper #4811 introduced, which every
advancement gate in this loop already reads.

That matters more than it looks. An earlier revision of this PR carried a second
counter, slot._plan_cancel_generation, alongside tracker.stopped. Two records
of one event is two things to keep in step, and First Principles was right to flag
it: a cancel path updating one and not the other would diverge silently, in the
one place the codebase had just finished centralising. That counter is gone, and
state.py now carries no diff at all.

Nothing consults the stage budget until a stage records its first round, which
cannot happen before the load returns, so OrchestrationTracker grows a setter for
it — documented as safe at exactly that point, and not mid-stage, where it would
move a deadline the current stage is already being measured against.

Dashboard Stop keeps a generation snapshot, because that channel genuinely has
nothing live to re-read: the posture settles all the way back to "idle" inside
the await. slot._stop_generation is not this PR's invention — it is main's
existing counter of stop INITIATIONS, added for precisely this shape, and
chat_runner's poisoned-conversation canary is the existing consumer reading it
the same way for the same reason.

_stop_generation = slot._stop_generation
try:
    cfg = await asyncio.to_thread(KiroCrewConfig.load)
    tracker.stage_timeout_seconds = cfg.orchestrator.stage_timeout_seconds
except Exception:
    ...                        # keep the tracker's default budget
if slot._stop_generation != _stop_generation or _orchestration_stopped(slot, tracker):
    return False               # abandon the plan

The stop posture and its card stay owned by the stop handler: nothing here writes
_stop_state, _stop_generation, or the card, and neither api_chat_slot_stop
nor SessionManager is touched. No shield, executor, lock, or new cancellation
token is introduced.

Abandoning the plan is not abandoning the slot. This is the second thing the
earlier revision got wrong: it left the bootstrap with a bare return, taken
before the loop's try.

While the config loads, the stage loop owns slot.task, so api_chat queues a
user message rather than running it, and _start_next_queued_turn holds it
behind _in_stage_execution so it can never run alongside a plan. The stage
loop's own finally is the single thing that clears that guard and hands the
queue on. Returning before the try skipped all of it: the guard stayed set on a
slot with no loop left to clear it, and the user's message stayed parked behind
it — where a later prompt could overtake it, and a restart could drop it.

The abort therefore returns from inside the try and leaves through that same
finally, exactly as a cancel landing at stage 0 does: guard cleared, queue
handed off, slot closed out.

Which exposed one more thing the finally had to stop doing. Its
not slot.running guard is asking whether a turn a stage STARTED still holds the
slot — _run_chat publishes its own task on slot.task and clears it when the
turn ends. It is not asking about this loop. Yet where no stage turn ever ran,
slot.task is still the stage loop itself, alive by definition inside its own
finally, so the raw read reported a live turn that does not exist and skipped
both the queue handoff and the idle close — on precisely the two paths with
nothing else to perform them: an abandoned bootstrap, and a plan with no stages.
It now compares against asyncio.current_task(). Wherever a stage turn did run,
slot.task is already None and the behaviour is unchanged.

One existing assertion is rescoped. test_no_worker_hop_when_no_stage_results_were_captured
recorded every asyncio.to_thread call the stage loop made and required the list
to be empty — a valid proxy only while the completion read was the loop's sole
offload. Its subject, per its own docstring, is that an empty result set skips
the excerpt worker, so it now filters the recorded calls to _completion_excerpts
rather than forbidding every hop. It has not been weakened: deleting the
production if _captured: guard still fails it
(assert [<function _completion_excerpts>] == []).

Tests

New test/test_orchestrator_config_load_off_loop.py drives the real _stage_loop
through tracker initialisation. For the offload cases the slot is given an empty
title list, so _plan_stage_count is zero and the loop initialises the tracker and
then has no stage to run — the branch under test is exercised with no model turn
involved. The Stop-race case gives the slot a real one-stage plan instead, so
reaching stage execution is observable.

The seam is the loader itself: chat_orchestrator binds KiroCrewConfig at
import, so the test replaces that module attribute with a wrapper that records
threading.get_ident() and delegates to the real classmethod. The recorded
thread is therefore the one that genuinely stats, reads, merges and validates —
not merely one that reached a call site — and the seam behaves identically before
and after the change. The repo's conftest already pins KIROCREW_HOME to a
per-test tmp dir, so the real loader reads controlled files rather than the
developer's own configuration.

Fail-before / pass-after, each row against its own baseline:

before after
test_config_load_runs_off_the_loop_thread FAILassert 61188 not in [61188] PASS
test_stop_during_the_config_load_does_not_start_the_plan FAIL — a stage turn ran after the Stop PASS
test_plan_cancel_during_the_config_load_does_not_start_the_plan FAIL — a stage turn ran after Plan Cancel PASS
test_a_plan_started_after_a_cancel_still_runs FAIL — a cancel disabled the NEXT plan PASS
test_a_message_queued_during_the_config_load_is_still_handed_off FAILassert [] == ['queued-message-slot'] PASS

The first row is measured on pristine main at the branch's original base (90348e1d4); the branch has since been rebased onto current main. The second is
measured against the offload commit instead: it is a regression the new await
makes reachable, so there is nothing for it to fail on a tree that has no
suspension point there.

That case is ordered with threading.Events, never sleeps. The loader blocks inside
the worker; the test then presses Stop through the real api_chat_slot_stop
(against a SessionManager seam returning "idle", the outcome that creates the
race) and asserts the posture has settled all the way back — _stop_generation
incremented by exactly one, _stop_state == "idle", _auto_run cleared — before
releasing the loader. Only then does it assert the behaviour that matters: no stage
turn ran, and no tracker was installed.

Mutation-checked, so the guard is doing the work and not the fixture:

guard new test
as shipped PASS
replaced with if False: FAILa stage turn ran after the user stopped the plan
reduced to if slot._stopping: (the naive fix) FAIL — same assertion

The third row is the point: the Stop has already resolved back to "idle" by the
time the loop looks, so the _stop_generation half is the load-bearing one. The
other five cases stayed green under every mutation.

The two mechanisms this revision changed carry their own negative controls,
each reverting exactly one thing and leaving the rest in place:

revert fails with
publish the tracker AFTER the load (the earlier shape, minus its second counter) Cancel has nothing to stop while the config loads: the tracker must be published before the await, or this window needs its own cancel record
take the abort as a bare return before the try (the earlier shape) the message the user queued during the config load was never handed off; it is stranded on the slot with no loop left to release it
keep the abort inside the try, but leave the finally's raw slot.running read the same assertion — assert [] == ['queued-message-slot']

Two findings from the exact-head GPT review are dispositioned on eb7dd0cac.

Worker cleanup occurs only after assertionsadopted. release.set() sat
after the assertions in all three gated tests, so a failure left the loader thread
parked in release.wait(timeout=10) past teardown and it then completed the real
KiroCrewConfig.load() outside the per-test KIROCREW_HOME. Measured by forcing
one assertion inside the parked region to fail: 14.81s before, 5.11s after
the delta is exactly the stranded worker's timeout. All three now release and reap
in finally.

Early tracker publication corrupts stage accountingnot adopted; the
mechanism is unreachable.
start_idx is frozen into a local int before the
loop's first await, with only two attribute reads between it and the
publication, so nothing can record a round in that gap on a single-threaded loop.
The base had no suspension point there either (the load was synchronous), so the
diff widened nothing before start_idx. Pinned rather than argued, by two tests:
test_a_round_recorded_during_the_config_load_cannot_skip_a_stage injects the
other record_round caller's exact call on the freshly published tracker while
the loop is held in the config load, and both stages still run;
test_a_round_recorded_before_loop_entry_does_skip_a_stage is the positive
control, so the first cannot pass by record_round being inert. Full disposition
in the PR thread.

The last two are separate reverts that fail identically, which is the point: the
handoff needs BOTH the exit path and the corrected guard, so neither is
load-bearing alone and neither is decoration.

The queued-message case is ordered with events, never sleeps. The loader blocks
in the worker, the message is queued while it is parked, and Cancel is pressed
through the real endpoint before the loader is released. It observes the handoff
at the seam the loop actually calls (_start_next_queued_turn) rather than by
running a turn, and it asserts the handoff BEFORE the guard state, so a build
that merely clears the flag without draining the queue still fails.

Being precise about the rest: the other four offload cases pass on both trees. They
are preservation coverage, not evidence of a defect, and are reported as such
rather than folded into the fail-before count:

  • a configured stage_timeout_seconds: 42, written as a real config.json and
    read by the real loader, reaches the tracker verbatim;
  • 0 survives as 0 — the stage loop reads it back as
    if tracker.stage_timeout_seconds:, so collapsing it onto 1800 with an or
    would silently re-enable a ceiling the operator turned off;
  • a raising loader still yields a usable tracker at 1800;
  • a slot that already has a tracker performs no config load at all.
pytest test/test_orchestrator_config_load_off_loop.py -q      11 passed

# every module that drives _stage_loop, plus the cancel-advance gates #4811 added
pytest test/test_orchestrator_config_load_off_loop.py \
       test/test_completion_result_read_off_loop.py \
       test/test_previous_result_read_off_loop.py \
       test/test_orchestrator_cancel_stops_advance.py \
       test/test_merge_queued_messages.py \
       test/test_dashboard_chat.py -q                      683 passed

# the finally-guard change is the widest edit, so every consumer of the
# queue handoff / _in_stage_execution was run too
pytest test/test_active_turn_session_key.py \
       test/test_autonudge_dashboard_fire.py \
       test/test_chat_runner_coverage.py \
       test/test_chat_slot_continue.py \
       test/test_conn_recovery.py \
       test/test_cron_origin_injection.py \
       test/test_dashboard_file_io.py \
       test/test_queue_during_subagents.py \
       test/test_merge_queued_messages.py -q                357 passed

# OrchestrationTracker's new setter
pytest test/test_subagent.py test/test_subagent_result_view.py \
       test/test_mcp_core_spawn_sub_agents.py -q            132 passed

The shard-1 reproduction quoted here previously was measured on the branch's
ORIGINAL base. This head is a rebase onto current main (554 commits) plus a
redesign, so that number no longer describes it and is not restated; the runs
above are what was actually executed against this head. The rescoped assertion
in test_completion_result_read_off_loop.py is still the only pre-existing test
this branch touches, and it passes here.

Local-only failures on this Windows box are unchanged in class and unrelated to
this diff (WinError 1314 symlink-privilege, cp950 decode); none occur on the
CI runners.

Also green: flake8, isort --check-only, mypy on both changed modules
("Success: no issues found in 2 source files"), and
scripts/check_black_formatting.py (4 files in scope; the new test file is NOT
baselined, so it is held to black and formatted).

Manual verification

N/A — unit coverage sufficient, and the coverage is not narrow: both cancellation
paths are driven through their REAL endpoints (api_chat_slot_stop,
api_chat_plan_action) against the real _stage_loop, with the config worker
parked on an event so the ordering is forced rather than raced. Reproducing this
by hand needs a Cancel or Stop pressed inside a config load that normally
completes in milliseconds, which is not something a person can time.

Two production files: chat_orchestrator.py (the bootstrap, the abort's exit
path, and the finally's live-turn test) and a setter on
OrchestrationTracker. state.py carries no diff.

Related Issues

N/A — self-discovered while auditing the orchestrator's remaining synchronous I/O
boundaries. #1783 is not cited: its only stage_timeout_seconds bullet asks for a
separate total-plan duration watchdog and says nothing about this config load, so
a Refs there would misattribute the work.

Checklist

  • Single commit with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable) — N/A: no documented behaviour changes. The scheduling boundary and the two cancellation contracts are described in the code, in _load_stage_budget's docstring and the finally's comment.
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

@leonlaiyc
leonlaiyc requested a review from a team as a code owner August 15, 2026 14:49
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Aug 15, 2026
@leonlaiyc
leonlaiyc force-pushed the fix/orchestrator-config-load-offloop branch from 00e0407 to f01a922 Compare August 15, 2026 16:30
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Aug 15, 2026
@github-actions

github-actions Bot commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed 1624538a93f6f19c1598a1a166af95fc0ec0ab1c via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 1624538

@github-actions

github-actions Bot commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

Opus 4.8 Review (fork) — ✅ no blocking findings

Reviewed 1624538a93f6f19c1598a1a166af95fc0ec0ab1c via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 1624538

@github-actions

github-actions Bot commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — 🟡 CONCERNS

Premise-level review of 1624538a93f6f19c1598a1a166af95fc0ec0ab1c via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

All claims verified against the base. I have enough to issue the review.

First-Principles-Verdict: CONCERNS

The offload is justified and precedented, but the early tracker publication plus new setter re-covers a window slot._plan_cancelled already records unconditionally.

What this change ships

Intent: stop the orchestrator's first-entry config read from blocking the gateway's only event loop — a FIX.

  1. First plan after a cold start no longer stalls the dashboard during config load — justified
  2. Plan Cancel pressed during that load now actually stops the plan — derived (race the offload itself opens)
  3. Dashboard Stop during that load now abandons the plan — derived; reuses main's _stop_generation
  4. A message typed during the load is handed off, not stranded — derived (abort exits via the loop's finally)
  5. A zero-stage plan now emits its done signal and hands off its queue — rides along (pre-existing slot.running self-read), but required by item 4's exit path
  6. Tracker published before the load, on the default budget — duplicate of slot._plan_cancelled coverage (see Watch)
  7. OrchestrationTracker gains a public stage_timeout_seconds setter — one consumer (_load_stage_budget); exists only to serve item 6
  8. A failed config load now logs a debug line — undeclared, trivial
  9. Existing off-loop test scoped to the excerpt hop only — consequence of the fix

Watch

  • The description says Plan Cancel during the await "stops nothing" — but api_chat_plan_action sets slot._plan_cancelled = True unconditionally, "NOT only when a tracker exists" (chat_orchestrator.py:930, Plan cancel racing the first Go can no-op on the tracker: transcript says cancelled, plan advances #6046), cleared only in _reset_auto_run_for_new_plan (chat_title.py:705). I count 2 set sites (chat_orchestrator.py:930, chat_handlers.py:763), both tracker-independent, and no stop-word source can fire mid-load (no turn is running). Re-checking that latch after the await, with the tracker still built from the loaded value, closes the same race with zero new surface — items 6 and 7 are a second spelling of coverage main already has. The PR's own test admits it: "Current main also latches a Cancel that beats tracker creation."
  • Item 5 fixes a real latent bug (empty plan never emitted chat_done) inside a PR framed as scheduling-only; it is load-bearing for item 4, but a reader of the title will not expect it.

Subtractions

  • Drop the stage_timeout_seconds setter (context_management.py) and the pre-await tracker publication; after the await, check slot._plan_cancelled (existing latch, set for exactly this window) alongside the _stop_generation snapshot, and construct the tracker from the loaded value as before — 1 consumer of the setter (_load_stage_budget), 0 elsewhere.

[FIRST-PRINCIPLES-REVIEWED] 1624538

@github-actions

github-actions Bot commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — ✅ PASS

Design-level review of 1624538a93f6f19c1598a1a166af95fc0ec0ab1c via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

The offload follows the repo's established off-loop idiom, and the two cancellation races the new await opens are closed with existing signals, not new machinery.

Verified against the base tree: _stop_generation is main's existing stop-initiation counter with the same snapshot pattern in chat_runner; _orchestration_stopped is the canonical gate; the _plan_cancelled latch already covers cancel-before-loop-entry, so the mid-load window is genuinely the only new one; the budget is read nowhere before the load returns; and routing the abort through the loop's finally (with the slot.running_turn_live correction) is the right ownership call — main's own _note_owner comment already conceded slot.running names the loop's own task in that finally.

Suggestions

  • A Stop-generation abort leaves an unstopped, default-budget tracker on the slot; a later Go on the same plan then skips the config load entirely and runs with 1800s instead of the configured value (a configured 0 regains a ceiling). Clearing slot._orch_tracker on that abort path (the Cancel path is already covered by the _plan_cancelled latch) restores main's guarantee that a bootstrapping plan always gets the configured budget.

[DESIGN-REVIEWED] 1624538

@leonlaiyc
leonlaiyc force-pushed the fix/orchestrator-config-load-offloop branch from f01a922 to 07d79c6 Compare August 16, 2026 01:44
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Aug 16, 2026
@leonlaiyc
leonlaiyc force-pushed the fix/orchestrator-config-load-offloop branch from 07d79c6 to 9cae22c Compare August 16, 2026 13:58
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Aug 16, 2026
@leonlaiyc

Copy link
Copy Markdown
Contributor Author

GPT 5.6 disposition — ac57f303aeb7dd0cac

Worker cleanup occurs only after assertions — ADOPTED.

Confirmed and measured, not just accepted. release.set() sat after the assertions in all three gated tests, so a failure left the loader thread parked in release.wait(timeout=10) past fixture teardown — and it then completed the real KiroCrewConfig.load() outside the per-test KIROCREW_HOME. A failing test silently reading, and caching, the operator's own configuration.

Measured by forcing one assertion inside the parked region to fail:

run time
before 14.81s
after 5.11s

The ~10s delta is exactly the stranded worker's release.wait timeout. All three regions now release and reap in finally, so the assertion failure surfaces immediately and the worker never outlives the test. Full file: 11 passed.


Early tracker publication corrupts stage accounting — NOT ADOPTED. The mechanism is unreachable; recommending disposition rather than the suggested revert.

The arithmetic is correct — a tracker carrying a round makes start_idx non-zero and stage 1 is skipped. What does not hold is that this diff can deliver such a round in time.

start_idx is frozen into a local int before the loop's first suspension point:

286  slot._orch_tracker = tracker          <- publication
288  total = slot._plan_stage_count
289  titles = getattr(slot, "_stage_titles", [])
292  start_idx = tracker.current_stage if tracker._stage_rounds else 0   <- frozen here
321  await _load_stage_budget(...)         <- FIRST await, already past it

Lines 286–292 are two attribute reads and an assignment. Nothing yields, so on the gateway's single-threaded loop nothing else can execute between the publication and the read. Everything the config await opens up happens strictly after start_idx is already captured.

This is also not a window the diff widened. Before it, the load was a synchronous KiroCrewConfig.load() — which does not yield either — so the base had no suspension point in that region and the diff added none before start_idx.

record_round has two callers repo-wide: this loop (after start_idx), and slack/gateway.py:5873. The gateway one does reach the same object, so the lingering-completion actor named in the finding is real — it simply cannot land early enough to move start_idx.

Pinned rather than argued, in two new tests:

  • test_a_round_recorded_during_the_config_load_cannot_skip_a_stage — holds the loop suspended in the config load (the widest window this diff opens) and injects the gateway's exact call on the freshly published tracker: stage = tracker.current_stage; tracker.record_round(stage). Both stages still execute.
  • test_a_round_recorded_before_loop_entry_does_skip_a_stage — the positive control. A tracker already carrying a round at entry does skip stage 1, so the first test cannot pass by record_round being inert. That is the resume path, where _orch_tracker is already set: nothing is published, _bootstrapping is False, and no config load runs. Untouched by this diff.

Reverting the early-publication hunk would also reopen what it exists to close: with the tracker published only after the load, api_chat_plan_action finds slot._orch_tracker is None for the whole window and a plan Cancel stops nothing while still telling the user the plan was cancelled. The alternative is a second cancellation record beside tracker.stopped, which the First Principles lane on this PR asked to remove.

Happy to be shown a path from the gateway's record_round into lines 286–292 if one exists — that would change the disposition.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Aug 23, 2026
leonlaiyc added a commit to leonlaiyc/KiroCrew that referenced this pull request Aug 23, 2026
Every config.json update in dashboard/handlers/memory.py reads the file,
changes one key and writes it all back on the event loop. The read, the
JSON parse, and write_config_atomically -- a tmp-file write plus a rename,
which can fsync -- all block, stalling every other session while they run.
Three sites do it: api_memory_settings, _write_embed_model_config, and
_set_migrated, which runs on EVERY boot while migrated is false and so
lands the stall exactly when the gateway is bringing sessions up. This is
the class the repo has been closing site by site (kirodotdev#4118, kirodotdev#3803, kirodotdev#4550), and
kirodotdev#4946's review named this module.

The whole transaction crosses over, never just the read. Offloading the
read alone would leave the write on the loop and insert a suspension point
between the read and the write-back while the file is unguarded on disk: an
external editor, a CLI command or another process landing in that gap would
be silently overwritten by a write derived from state nobody re-checked.
That gap is zero today because the sequence is synchronous, and it stays
zero because the worker performs the whole thing without yielding. The
existing per-config lock is held across the hop, so two coroutines still
cannot interleave.

Two of the three sites also hand-rolled a reader this module already
imports. read_config_for_update is the documented companion to
write_config_atomically with 27 call sites, and api_memory_settings uses it
200 lines above; _set_migrated and _write_embed_model_config instead caught
Exception around json.loads. The helper additionally refuses a non-object
top level, where the hand-rolled version accepted a list and then raised
AttributeError from setdefault -- a crash where a fail-closed refusal was
intended.

ConfigReadError is not swallowed by the helper: what to tell the user
differs per site, and each keeps exactly the behaviour it had -- skip and
retry next boot, raise ValueError, or answer 500 config_unreadable.

api_memory_settings now validates its body before the transaction. None of
that reads the config, and a 400 previously took the lock and abandoned it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
leonlaiyc added a commit to leonlaiyc/KiroCrew that referenced this pull request Aug 23, 2026
_write_env_updates stats and reads the whole .env, re-parses it line by
line, then creates a 0600 temp file, chmods it, writes and renames. All
synchronous file I/O, and six async channel config-save handlers call it.

Three already reached it through asyncio.to_thread -- telegram, teams,
wecom -- and three called it inline on the gateway loop: slack, discord and
webex, stalling every other session for the duration of a token save.

So this is not a missing convention but an existing one applied to half the
call sites. messaging.py already uses asyncio.to_thread 29 times, and with
three siblings doing it correctly nothing in the file said which half was
right, or stopped the next channel from copying the wrong one. It is the
class the repo has been closing site by site (kirodotdev#4118, kirodotdev#3803, kirodotdev#4550), and
kirodotdev#4946's review named this module.

The WHOLE call is offloaded, never a part of it: the read-modify-write is
one transaction, and a suspension point between the read and the rename
would let a concurrent writer's keys be dropped by a write derived from
lines nobody re-read. Keeping _write_env_updates one synchronous function
on one worker preserves that without depending on the caller, which is now
said on the function itself so the next channel inherits the reason and not
just the shape.

The regression pins all six channels rather than the three that moved,
since the defect was the split and not any one site.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
leonlaiyc added a commit to leonlaiyc/KiroCrew that referenced this pull request Aug 24, 2026
Every config.json update in dashboard/handlers/memory.py reads the file,
changes one key and writes it all back on the event loop. The read, the
JSON parse, and write_config_atomically -- a tmp-file write plus a rename,
which can fsync -- all block, stalling every other session while they run.
Three sites do it: api_memory_settings, _write_embed_model_config, and
_set_migrated, which runs on EVERY boot while migrated is false and so
lands the stall exactly when the gateway is bringing sessions up. This is
the class the repo has been closing site by site (kirodotdev#4118, kirodotdev#3803, kirodotdev#4550), and

The whole transaction crosses over, never just the read. Offloading the
read alone would leave the write on the loop and insert a suspension point
between the read and the write-back while the file is unguarded on disk: an
external editor, a CLI command or another process landing in that gap would
be silently overwritten by a write derived from state nobody re-checked.
That gap is zero today because the sequence is synchronous, and it stays
zero because the worker performs the whole thing without yielding. The
existing per-config lock is held across the hop, so two coroutines still
cannot interleave.

Two of the three sites also hand-rolled a reader this module already
imports. read_config_for_update is the documented companion to
write_config_atomically with 27 call sites, and api_memory_settings uses it
200 lines above; _set_migrated and _write_embed_model_config instead caught
Exception around json.loads. The helper additionally refuses a non-object
top level, where the hand-rolled version accepted a list and then raised
AttributeError from setdefault -- a crash where a fail-closed refusal was
intended.

ConfigReadError is not swallowed by the helper: what to tell the user
differs per site, and each keeps exactly the behaviour it had -- skip and
retry next boot, raise ValueError, or answer 500 config_unreadable.

api_memory_settings now validates its body before the transaction. None of
that reads the config, and a 400 previously took the lock and abandoned it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@bolichen97
bolichen97 enabled auto-merge (squash) August 24, 2026 07:00
leonlaiyc added a commit to leonlaiyc/KiroCrew that referenced this pull request Aug 24, 2026
Every config.json update in dashboard/handlers/memory.py reads the file,
changes one key and writes it all back on the event loop. The read, the
JSON parse, and write_config_atomically -- a tmp-file write plus a rename,
which can fsync -- all block, stalling every other session while they run.
Three sites do it: api_memory_settings, _write_embed_model_config, and
_set_migrated, which runs on EVERY boot while migrated is false and so
lands the stall exactly when the gateway is bringing sessions up. This is
the class the repo has been closing site by site (kirodotdev#4118, kirodotdev#3803, kirodotdev#4550), and

The whole transaction crosses over, never just the read. Offloading the
read alone would leave the write on the loop and insert a suspension point
between the read and the write-back while the file is unguarded on disk: an
external editor, a CLI command or another process landing in that gap would
be silently overwritten by a write derived from state nobody re-checked.
That gap is zero today because the sequence is synchronous, and it stays
zero because the worker performs the whole thing without yielding. The
existing per-config lock is held across the hop, so two coroutines still
cannot interleave.

Two of the three sites also hand-rolled a reader this module already
imports. read_config_for_update is the documented companion to
write_config_atomically with 27 call sites, and api_memory_settings uses it
200 lines above; _set_migrated and _write_embed_model_config instead caught
Exception around json.loads. The helper additionally refuses a non-object
top level, where the hand-rolled version accepted a list and then raised
AttributeError from setdefault -- a crash where a fail-closed refusal was
intended.

ConfigReadError is not swallowed by the helper: what to tell the user
differs per site, and each keeps exactly the behaviour it had -- skip and
retry next boot, raise ValueError, or answer 500 config_unreadable.

api_memory_settings now validates its body before the transaction. None of
that reads the config, and a 400 previously took the lock and abandoned it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
leonlaiyc added a commit to leonlaiyc/KiroCrew that referenced this pull request Aug 24, 2026
Every config.json update in dashboard/handlers/memory.py reads the file,
changes one key and writes it all back on the event loop. The read, the
JSON parse, and write_config_atomically -- a tmp-file write plus a rename,
which can fsync -- all block, stalling every other session while they run.
Three sites do it: api_memory_settings, _write_embed_model_config, and
_set_migrated, which runs on EVERY boot while migrated is false and so
lands the stall exactly when the gateway is bringing sessions up. This is
the class the repo has been closing site by site (kirodotdev#4118, kirodotdev#3803, kirodotdev#4550), and

The whole transaction crosses over, never just the read. Offloading the
read alone would leave the write on the loop and insert a suspension point
between the read and the write-back while the file is unguarded on disk: an
external editor, a CLI command or another process landing in that gap would
be silently overwritten by a write derived from state nobody re-checked.
That gap is zero today because the sequence is synchronous, and it stays
zero because the worker performs the whole thing without yielding. The
existing per-config lock is held across the hop, so two coroutines still
cannot interleave.

Two of the three sites also hand-rolled a reader this module already
imports. read_config_for_update is the documented companion to
write_config_atomically with 27 call sites, and api_memory_settings uses it
200 lines above; _set_migrated and _write_embed_model_config instead caught
Exception around json.loads. The helper additionally refuses a non-object
top level, where the hand-rolled version accepted a list and then raised
AttributeError from setdefault -- a crash where a fail-closed refusal was
intended.

ConfigReadError is not swallowed by the helper: what to tell the user
differs per site, and each keeps exactly the behaviour it had -- skip and
retry next boot, raise ValueError, or answer 500 config_unreadable.

api_memory_settings now validates its body before the transaction. None of
that reads the config, and a 400 previously took the lock and abandoned it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
leonlaiyc added a commit to leonlaiyc/KiroCrew that referenced this pull request Aug 25, 2026
Every config.json update in dashboard/handlers/memory.py reads the file,
changes one key and writes it all back on the event loop. The read, the
JSON parse, and write_config_atomically -- a tmp-file write plus a rename,
which can fsync -- all block, stalling every other session while they run.
Three sites do it: api_memory_settings, _write_embed_model_config, and
_set_migrated, which runs on EVERY boot while migrated is false and so
lands the stall exactly when the gateway is bringing sessions up. This is
the class the repo has been closing site by site (kirodotdev#4118, kirodotdev#3803, kirodotdev#4550), and

The whole transaction crosses over, never just the read. Offloading the
read alone would leave the write on the loop and insert a suspension point
between the read and the write-back while the file is unguarded on disk: an
external editor, a CLI command or another process landing in that gap would
be silently overwritten by a write derived from state nobody re-checked.
That gap is zero today because the sequence is synchronous, and it stays
zero because the worker performs the whole thing without yielding. The
existing per-config lock is held across the hop, so two coroutines still
cannot interleave.

Two of the three sites also hand-rolled a reader this module already
imports. read_config_for_update is the documented companion to
write_config_atomically with 27 call sites, and api_memory_settings uses it
200 lines above; _set_migrated and _write_embed_model_config instead caught
Exception around json.loads. The helper additionally refuses a non-object
top level, where the hand-rolled version accepted a list and then raised
AttributeError from setdefault -- a crash where a fail-closed refusal was
intended.

ConfigReadError is not swallowed by the helper: what to tell the user
differs per site, and each keeps exactly the behaviour it had -- skip and
retry next boot, raise ValueError, or answer 500 config_unreadable.

api_memory_settings now validates its body before the transaction. None of
that reads the config, and a 400 previously took the lock and abandoned it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
iamwhatever pushed a commit that referenced this pull request Aug 25, 2026
…5059)

Every config.json update in dashboard/handlers/memory.py reads the file,
changes one key and writes it all back on the event loop. The read, the
JSON parse, and write_config_atomically -- a tmp-file write plus a rename,
which can fsync -- all block, stalling every other session while they run.
Three sites do it: api_memory_settings, _write_embed_model_config, and
_set_migrated, which runs on EVERY boot while migrated is false and so
lands the stall exactly when the gateway is bringing sessions up. This is
the class the repo has been closing site by site (#4118, #3803, #4550), and

The whole transaction crosses over, never just the read. Offloading the
read alone would leave the write on the loop and insert a suspension point
between the read and the write-back while the file is unguarded on disk: an
external editor, a CLI command or another process landing in that gap would
be silently overwritten by a write derived from state nobody re-checked.
That gap is zero today because the sequence is synchronous, and it stays
zero because the worker performs the whole thing without yielding. The
existing per-config lock is held across the hop, so two coroutines still
cannot interleave.

Two of the three sites also hand-rolled a reader this module already
imports. read_config_for_update is the documented companion to
write_config_atomically with 27 call sites, and api_memory_settings uses it
200 lines above; _set_migrated and _write_embed_model_config instead caught
Exception around json.loads. The helper additionally refuses a non-object
top level, where the hand-rolled version accepted a list and then raised
AttributeError from setdefault -- a crash where a fail-closed refusal was
intended.

ConfigReadError is not swallowed by the helper: what to tell the user
differs per site, and each keeps exactly the behaviour it had -- skip and
retry next boot, raise ValueError, or answer 500 config_unreadable.

api_memory_settings now validates its body before the transaction. None of
that reads the config, and a 400 previously took the lock and abandoned it.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Aug 26, 2026
@bolichen97

Copy link
Copy Markdown
Collaborator

This PR materially overlaps #3771 in dashboard/chat_orchestrator.py::_stage_loop and test_completion_result_read_off_loop.py. The invariants are complementary: this branch offloads orchestrator-config reads and adds stop gates; #3771 offloads stage-result capture/write work.

Please use this config-read/stop-gate shape as the shared baseline, then replay #3771's result-write offload and both regression seams. Verify cancellation/stop gates remain between the offloaded phases and that neither config nor result-file I/O runs synchronously on the event loop.

The first stage loop for a slot has no OrchestrationTracker yet and builds
one from the configured stage timeout. That read goes through
KiroCrewConfig.load(), which stats and reads config.json plus any
config.local.json overlay, deep-merges them and runs the full jsonschema
validation -- synchronously, on the single event loop the gateway shares
with every other session. Only the load crosses to a worker; the value is
applied and the slot read back on the loop, so no live orchestration state
is handed to a thread.

That hop is the loop's FIRST suspension point, ahead of every stage gate,
so it opens a window on both cancellation channels -- and neither
necessarily leaves state a plain re-read can see.

Plan Cancel stops slot._orch_tracker. Building the tracker from the load's
result leaves that None for the whole window, so the press stops nothing
while still telling the user the plan was cancelled. The tracker is
therefore published BEFORE the load and the configured budget applied
afterwards, which makes tracker.stopped -- the canonical plan-cancel signal
every advancement gate already reads through _orchestration_stopped --
cover this window too. A second cancellation record kept beside it would be
one more thing to drift. Nothing consults the budget until a stage records
its first round, which cannot happen before the load returns, so
OrchestrationTracker grows a setter for it; the tracker starts on its own
default, the same 1800s this loop fell back to when the load raised.

Dashboard Stop has no ACP turn to cancel yet, so stop_turn answers "idle"
and the handler releases _stop_state straight back to "idle" -- re-reading
_stopping afterwards shows an unstopped slot. slot._stop_generation counts
stop INITIATIONS and is never rewound, so it is snapshotted before the wait
instead: the same reading, for the same reason, as the poisoned-
conversation canary in chat_runner.

Abandoning the plan is not abandoning the slot. While the config loads the
stage loop owns slot.task, so api_chat queues a user message rather than
running it, and _start_next_queued_turn HOLDS it behind
_in_stage_execution so it can never run alongside a plan. Only the loop's
own finally clears that guard and hands the queue on, so the abort returns
from INSIDE the try and leaves through it, exactly as a cancel at stage 0
does. A bare return would strand the message behind a guard with no loop
left to clear it, where a later prompt could overtake it and a restart
could drop it.

That finally then had to stop counting the loop's own task as a live turn.
slot.running asks whether a turn a stage STARTED still holds the slot --
_run_chat publishes its own task and clears it when the turn ends -- but
where no stage turn ever ran, slot.task is still the stage loop itself,
alive by definition inside its finally. The raw read reported a turn that
did not exist and skipped both the queue handoff and the idle close on the
two paths with nothing else to perform them: an abandoned bootstrap, and a
plan with no stages.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@bolichen97
bolichen97 force-pushed the fix/orchestrator-config-load-offloop branch from eb7dd0c to 1624538 Compare August 30, 2026 05:09
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Aug 30, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: fix (4 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: offloads the blocking orchestrator config load onto a worker thread with cancel/stop-safe re-check, keeping the default budget on load failure (clear root cause: blocking load on the event loop). CodeQL is not applicable on this fork PR (default-setup emits no check-run); SAST coverage is Semgrep only, latest run success with 0 annotations.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: fix (4 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: moves the orchestrator's first-stage KiroCrewConfig.load off the gateway event loop (it stats and reads config.json plus the local overlay and runs full schema validation synchronously), publishing the tracker on its default budget first and applying the loaded value after, with both cancellation channels re-checked across the worker hop via a non-rewound _stop_generation snapshot. CodeQL is not applicable on this fork PR (default-setup emits no check-run); SAST coverage is Semgrep only, latest run success with 0 annotations.

@bolichen97
bolichen97 merged commit 486d0fd into kirodotdev:main Sep 1, 2026
71 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) needs-author-decision PR blocked on author input

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants