fix(dashboard): offload eager spawn's agent-binding config load - #4118
fix(dashboard): offload eager spawn's agent-binding config load#4118leonlaiyc wants to merge 1 commit into
Conversation
5feabb2 to
c58fe84
Compare
|
Hi @leonlaiyc, a maintainer nudge on this one: it is carrying the Current state:
Could you rebase onto the latest git fetch upstream # or: git fetch origin, if this branch lives here
git rebase upstream/main
# resolve any conflicts, run the local gates below, then update the branch with
# a force-with-lease so the rebase lands without clobbering anyone else's workThe local gates to run before updating the branch: black src/kiro_crew test && isort src/kiro_crew test
flake8 src/kiro_crew test && mypy src/kiro_crew
python -m pytest
cd website && npm run build && npm run testA good part of this branch's redness is likely stale rather than a real defect: it predates a lot of what is now on |
Every config.json update in dashboard/handlers/memory.py reads the file, changes one key and writes it all back on the event loop. The read, the JSON parse, and write_config_atomically -- a tmp-file write plus a rename, which can fsync -- all block, stalling every other session while they run. Three sites do it: api_memory_settings, _write_embed_model_config, and _set_migrated, which runs on EVERY boot while migrated is false and so lands the stall exactly when the gateway is bringing sessions up. This is the class the repo has been closing site by site (kirodotdev#4118, kirodotdev#3803, kirodotdev#4550), and kirodotdev#4946's review named this module. The whole transaction crosses over, never just the read. Offloading the read alone would leave the write on the loop and insert a suspension point between the read and the write-back while the file is unguarded on disk: an external editor, a CLI command or another process landing in that gap would be silently overwritten by a write derived from state nobody re-checked. That gap is zero today because the sequence is synchronous, and it stays zero because the worker performs the whole thing without yielding. The existing per-config lock is held across the hop, so two coroutines still cannot interleave. Two of the three sites also hand-rolled a reader this module already imports. read_config_for_update is the documented companion to write_config_atomically with 27 call sites, and api_memory_settings uses it 200 lines above; _set_migrated and _write_embed_model_config instead caught Exception around json.loads. The helper additionally refuses a non-object top level, where the hand-rolled version accepted a list and then raised AttributeError from setdefault -- a crash where a fail-closed refusal was intended. ConfigReadError is not swallowed by the helper: what to tell the user differs per site, and each keeps exactly the behaviour it had -- skip and retry next boot, raise ValueError, or answer 500 config_unreadable. api_memory_settings now validates its body before the transaction. None of that reads the config, and a 400 previously took the lock and abandoned it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
_write_env_updates stats and reads the whole .env, re-parses it line by line, then creates a 0600 temp file, chmods it, writes and renames. All synchronous file I/O, and six async channel config-save handlers call it. Three already reached it through asyncio.to_thread -- telegram, teams, wecom -- and three called it inline on the gateway loop: slack, discord and webex, stalling every other session for the duration of a token save. So this is not a missing convention but an existing one applied to half the call sites. messaging.py already uses asyncio.to_thread 29 times, and with three siblings doing it correctly nothing in the file said which half was right, or stopped the next channel from copying the wrong one. It is the class the repo has been closing site by site (kirodotdev#4118, kirodotdev#3803, kirodotdev#4550), and kirodotdev#4946's review named this module. The WHOLE call is offloaded, never a part of it: the read-modify-write is one transaction, and a suspension point between the read and the rename would let a concurrent writer's keys be dropped by a write derived from lines nobody re-read. Keeping _write_env_updates one synchronous function on one worker preserves that without depending on the caller, which is now said on the function itself so the next channel inherits the reason and not just the shape. The regression pins all six channels rather than the three that moved, since the defect was the split and not any one site. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every config.json update in dashboard/handlers/memory.py reads the file, changes one key and writes it all back on the event loop. The read, the JSON parse, and write_config_atomically -- a tmp-file write plus a rename, which can fsync -- all block, stalling every other session while they run. Three sites do it: api_memory_settings, _write_embed_model_config, and _set_migrated, which runs on EVERY boot while migrated is false and so lands the stall exactly when the gateway is bringing sessions up. This is the class the repo has been closing site by site (kirodotdev#4118, kirodotdev#3803, kirodotdev#4550), and The whole transaction crosses over, never just the read. Offloading the read alone would leave the write on the loop and insert a suspension point between the read and the write-back while the file is unguarded on disk: an external editor, a CLI command or another process landing in that gap would be silently overwritten by a write derived from state nobody re-checked. That gap is zero today because the sequence is synchronous, and it stays zero because the worker performs the whole thing without yielding. The existing per-config lock is held across the hop, so two coroutines still cannot interleave. Two of the three sites also hand-rolled a reader this module already imports. read_config_for_update is the documented companion to write_config_atomically with 27 call sites, and api_memory_settings uses it 200 lines above; _set_migrated and _write_embed_model_config instead caught Exception around json.loads. The helper additionally refuses a non-object top level, where the hand-rolled version accepted a list and then raised AttributeError from setdefault -- a crash where a fail-closed refusal was intended. ConfigReadError is not swallowed by the helper: what to tell the user differs per site, and each keeps exactly the behaviour it had -- skip and retry next boot, raise ValueError, or answer 500 config_unreadable. api_memory_settings now validates its body before the transaction. None of that reads the config, and a 400 previously took the lock and abandoned it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every config.json update in dashboard/handlers/memory.py reads the file, changes one key and writes it all back on the event loop. The read, the JSON parse, and write_config_atomically -- a tmp-file write plus a rename, which can fsync -- all block, stalling every other session while they run. Three sites do it: api_memory_settings, _write_embed_model_config, and _set_migrated, which runs on EVERY boot while migrated is false and so lands the stall exactly when the gateway is bringing sessions up. This is the class the repo has been closing site by site (kirodotdev#4118, kirodotdev#3803, kirodotdev#4550), and The whole transaction crosses over, never just the read. Offloading the read alone would leave the write on the loop and insert a suspension point between the read and the write-back while the file is unguarded on disk: an external editor, a CLI command or another process landing in that gap would be silently overwritten by a write derived from state nobody re-checked. That gap is zero today because the sequence is synchronous, and it stays zero because the worker performs the whole thing without yielding. The existing per-config lock is held across the hop, so two coroutines still cannot interleave. Two of the three sites also hand-rolled a reader this module already imports. read_config_for_update is the documented companion to write_config_atomically with 27 call sites, and api_memory_settings uses it 200 lines above; _set_migrated and _write_embed_model_config instead caught Exception around json.loads. The helper additionally refuses a non-object top level, where the hand-rolled version accepted a list and then raised AttributeError from setdefault -- a crash where a fail-closed refusal was intended. ConfigReadError is not swallowed by the helper: what to tell the user differs per site, and each keeps exactly the behaviour it had -- skip and retry next boot, raise ValueError, or answer 500 config_unreadable. api_memory_settings now validates its body before the transaction. None of that reads the config, and a 400 previously took the lock and abandoned it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every config.json update in dashboard/handlers/memory.py reads the file, changes one key and writes it all back on the event loop. The read, the JSON parse, and write_config_atomically -- a tmp-file write plus a rename, which can fsync -- all block, stalling every other session while they run. Three sites do it: api_memory_settings, _write_embed_model_config, and _set_migrated, which runs on EVERY boot while migrated is false and so lands the stall exactly when the gateway is bringing sessions up. This is the class the repo has been closing site by site (kirodotdev#4118, kirodotdev#3803, kirodotdev#4550), and The whole transaction crosses over, never just the read. Offloading the read alone would leave the write on the loop and insert a suspension point between the read and the write-back while the file is unguarded on disk: an external editor, a CLI command or another process landing in that gap would be silently overwritten by a write derived from state nobody re-checked. That gap is zero today because the sequence is synchronous, and it stays zero because the worker performs the whole thing without yielding. The existing per-config lock is held across the hop, so two coroutines still cannot interleave. Two of the three sites also hand-rolled a reader this module already imports. read_config_for_update is the documented companion to write_config_atomically with 27 call sites, and api_memory_settings uses it 200 lines above; _set_migrated and _write_embed_model_config instead caught Exception around json.loads. The helper additionally refuses a non-object top level, where the hand-rolled version accepted a list and then raised AttributeError from setdefault -- a crash where a fail-closed refusal was intended. ConfigReadError is not swallowed by the helper: what to tell the user differs per site, and each keeps exactly the behaviour it had -- skip and retry next boot, raise ValueError, or answer 500 config_unreadable. api_memory_settings now validates its body before the transaction. None of that reads the config, and a 400 previously took the lock and abandoned it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Round 2 verdicts on First Principles subtraction 1: the false claim in the test module docstringAccepted, and fixed. This one is mine and it is a fair catch. Round 2 corrected the "the load is read-only" claim in the call-site comment, the Fixed by keeping one truth, as the lane asked, and by not duplicating the argument: Restating the reasoning in the docstring is what let the two copies diverge in the First Principles subtraction 2: delete the bare load at
|
0bea21d to
cc34253
Compare
cc34253 to
8161898
Compare
|
GPT 5.6 blocked the rebased head The finding
The write-back migration does exist and it does call Fact 1: the migration is one-shot per process (measured, not read)Probed against the real loader with a real legacy config -- Only the first load whose on-disk config still has a legacy shape writes. Every Fact 2: the offloaded load is never the process's first load on this path
try:
cfg = KiroCrewConfig.load() # 3229, ON THE LOOP
if not cfg.session.eager_spawn:
return None
except Exception:
return None
...
task = asyncio.create_task(_eager_spawn(state, slot, allow_resume=allow_resume)) # 3237So a pending migration is always performed by that on-loop load, on the loop, This is the same structural fact the First Principles lane established when it On the prescribed remedy"Revert the offload" would delete the fix for #4117 while leaving both real items What changed insteadRather than only rebutting, That last sentence matters beyond this round: it is a real constraint on #7734. The Verification on
|
|
GPT 5.6 re-blocked What is proven safeThe ordinary path cannot reach the migration write, and this is measured rather than
What I cannot rule out, stated plainlyMy rebuttal covers "migration already done before the task existed". It does not That requires a legacy-format write landing inside a ~debounce-long window, Two further facts the maintainer should weigh, one each way:
Why I did not fix it forwardThere is no read path that skips the migration.
Options
My read: 1 now with 3 as the follow-up, because 3 fixes the class rather than this one State of the PR at
|
|
Final disposition. This PR is handed to a maintainer for the merge decision, with a Lane state on this head
Every non-review check is green: all Linux and Windows backend shards, backend lint The one blocking finding: real, narrow, and not fixable in this PR's scopeGPT 5.6 blocked twice, on What is proven safe. The ordinary path cannot reach the migration write:
What I could not rule out. GPT's second phrasing names a window the above does not Why it was not fixed forward here. There is no read path that skips the migration. The class is now tracked as #7793 -- the migration
First Principles' remaining advisory: accepted, deliberately not pushedIts PASS carries one subtraction, and it is right: the Not acted on here, for one reason: any push re-rolls the blocking GPT lane, and this What a maintainer is deciding
My read is 1, with #7793 as the follow-up, because #7793 fixes the class for every Provenance@leonlaiyc is the commit author, unchanged through all five rounds. Both Round history, all dispositioned in-thread: the false "load is read-only" claim |
Open PR relationship auditThis is a consolidated, point-in-time code-level audit note. It compares complete merge-base diffs and current/merged code; it does not treat a shared topic as duplication or partial coverage as completion. Relationship findings
No PR, Issue, label, branch, or review state was changed by the relationship-note portion of this audit. |
_eager_spawn resolves the slot's agent bindings before the speculative handshake, and that read is a synchronous KiroCrewConfig.load() on the gateway event loop: path resolution, exists()/read_text()/json.loads() on config.json, is_file()/stat() and a deep merge of any config.local.json overlay, then the full jsonschema validation. None of it yields. The loader's fingerprint cache short-circuits a warm call, so the blocking read is the cold path -- a freshly started gateway, or the first eager spawn after either config file is written. This task is speculative: it exists only to make the slot's first real message faster, nothing depends on its result, and it is already debounced and semaphore-capped. So an optional latency optimisation was doing cold filesystem and schema work on the one loop every live session, WebSocket broadcast and HTTP handler shares. Only the load crosses to a worker. Bindings are resolved, the session acquired and the slot read back on the loop, so no slot or session state is handed to another thread. The new suspension widens an existing window rather than opening a new one. It lands inside the envelope _eager_spawn already maintains: _bound is snapshotted immediately above the load, and the post-handshake guards already discard a session whose creator lost the race, whose slot was deleted or replaced, or whose bindings moved. So a switch or a delete landing during the load is revalidated there, and no new lifecycle state, generation counter or lock is introduced. A worker abandoned by a cancelled eager task is acceptable for a narrower reason than "read-only", which the load is not: it publishes three process-global snapshots and its write-back migration can rewrite config.json. The autocompact publish is ordered on a ticket drawn before the read, so an earlier-starting load cannot overwrite a newer one, and the other two are idempotent rebinds of what was just read. The residue -- those two carry no ordering, so a late worker can republish a superseded value -- is a property of concurrent loads generally, not of this call site; the stop-hook nudge-cap load already runs off-loop. Tests: a thread-identity regression test that fails deterministically on an unfixed tree, plus two preservation tests that hold the loader suspended and fire a slot replacement and an agent switch inside the new window. Refs kirodotdev#4117 Rebased onto current main (the base this branch carried predated the black-baseline gate); the new test file is reformatted with the pinned black==26.3.1 to satisfy it. Production change is byte-identical. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: gh-autofix#2887 <chenmingwei23@users.noreply.github.com>
8161898 to
4a63f79
Compare
|
Rebased onto main Conflicts (1 file):
Gates run locally on changed files: black, isort, flake8 all clean; Please review the resolution. A maintainer push makes the maintainer the last pusher, so under the repo's last-push rule a second approver is needed. Reply if anything looks wrong. |
Problem / Motivation
_eager_spawninsrc/kiro_crew/dashboard/chat_runner.pyresolves the slot'sagent bindings before the speculative handshake, and that read is synchronous:
KiroCrewConfig.load()is not a cheap accessor. It resolves the config path,exists()/read_text()/json.loads()onconfig.json,is_file()/stat()anddeep-merges any
config.local.jsonoverlay, then runs the fulljsonschemavalidation. None of that yields, so the loop is occupied for the whole read.
Being accurate about the cost: the loader keeps a fingerprint cache, so a warm
call short-circuits. The blocking read is the cold path.
Correcting this section against the First Principles review of
a415e3276, whichis right:
_eager_spawnis created only byschedule_eager_spawn, and thatfunction runs its own bare
KiroCrewConfig.load()on the loop atchat_runner.py:3229to testsession.eager_spawnbefore creating the task. So ona cold gateway the loop pays the cold read there, roughly a debounce ahead of the
load this PR offloads, and the offloaded call then usually hits the warm
fingerprint cache. The cold case that reaches this call site is narrower than the
section originally claimed: a config write landing inside the debounce window.
What this PR still buys, stated honestly: the offloaded call is off the loop
whether warm or cold, so the loop stops paying even the warm path's
statpass andstops being exposed to the cold case above. What it does not do is remove the
cold read from the loop, because the
schedule_eager_spawn:3229load is the onethat lands first. That load is a separate defect and is not fixed here -- see
Scope.
Why it matters
_eager_spawnis speculative. It exists only to make a slot's first realmessage faster; nothing depends on its result, and it is already debounced and
capped by
_eager_spawn_sem. So an optional latency optimisation was doing coldfilesystem and schema work on the single loop that every live session, every
WebSocket broadcast and every HTTP handler shares — the one place where the work
has no claim on the loop at all.
It is user-triggered rather than rare:
schedule_eager_spawnre-arms on slotcreate, agent switch, project set and slot focus.
No latency figure is claimed; none was measured.
What changed (motivation → approach → change)
Motivation — take the filesystem and schema work off the shared loop without
altering a single handshake semantic.
Approach — not a new idiom.
await asyncio.to_thread(KiroCrewConfig.load)isalready the established form on more than a dozen async paths, several of them in
this package, and
chat_runner.pyitself already carries eightasyncio.to_threadoffloads — one of them justified in-comment as "
_run_chatshares the singlegateway event loop with every other session". This call site is the outlier.
Change — the load, and only the load, moves to a worker:
No slot, session,
DashboardStateor tracker crosses the boundary. No new configexecutor, no caching layer, no change to reload semantics, no change to any
default. The existing broad
except Exceptionis kept: an exception raised insideto_threadstill propagates through theawaitinto the same handler.Why this is safe: the suspension lands inside an existing envelope
The new
awaitwidens a window that already exists rather than opening a new one._eager_spawnalready maintains a snapshot → handshake → revalidation envelope,and the load sits inside it:
Every action newly able to land during the load — slot deletion or replacement,
an agent/model/project/effort switch, another creator winning the same key — is
therefore revalidated after the handshake by guards that predate this change.
That is why no generation counter, no new slot field and no lock is added here.
Cancellation needs plumbing, but not new plumbing. Correcting this against the GPT
review of
a415e3276, which is right that "the load is read-only" is false:KiroCrewConfig.loadpublishes three process-global snapshots on every call(
mcp.extra_path_dirs, the agent alias table, the autocompact threshold) and itswrite-back migration can rewrite
config.json.An abandoned worker is still acceptable, for a narrower reason than the one
originally given. The autocompact publish is ordered on a ticket drawn before the
read, so a load that began earlier cannot overwrite a newer one -- the loader
designed for exactly this. The other two are idempotent in-memory rebinds of what
was just read, so republishing an unchanged config is a no-op.
The residue, stated rather than hidden: those two publishes carry no ordering of
their own, so a worker that finishes late can republish a value a newer load has
already superseded. That is a property of concurrent loads in general and not of
this call site --
mainalready loads off-loop at the stop-hook nudge-cap site --so it is not introduced here and is not fixed here.
Tests
test/test_eager_spawn_config_load_off_loop.py:test_config_load_runs_off_the_event_loop— the fail-before. Wraps theloader
_eager_spawnactually calls with a delegating wrapper that recordsthreading.get_ident(), invalidates the fingerprint cache so the recordedthread genuinely does the disk work, drives the real
_eager_spawn, andcompares against the loop thread. On an unfixed tree:
No timing assertion, no sleep.
test_slot_replaced_during_the_load_leaves_no_orphan_sessionandtest_binding_switch_during_the_load_removes_the_stale_session—preservation coverage for the new window. A
threading.Event-gated loader isheld suspended, the lifecycle action is fired on the loop while it is
suspended, and the existing guards are shown to still tear the session down.
These are coverage for the new await, so they are not expected to fail on a
pristine tree — only the thread-identity test does that. The event waits are
bounded purely as hang guards; nothing asserts on elapsed time.
Local:
test_eager_spawn.py,test_eager_spawn_config_load_off_loop.py,test_chat_runner_coverage.py,test_active_turn_session_key.py,test_chat_slot_project.py,test_chat_slot_reasoning_effort.py,test_chat_slot_mode.py,test_dashboard_chat.py— 899 collected, all green.The wider set of 74 test files that reference
chat_runneris 4487 passed /76 skipped, with 22 failures that reproduce identically with this commit's
production change stashed: 8 ×
WinError 1314(symlink privilege) and 14 intest_dashboard_file_io.py, both pre-existing on this Windows host.flake8,isort --check-onlyandmypy --platform linuxclean on both changedfiles;
git diff --checkclean.Scope
One call site.
KiroCrewConfig.loadappears at seven sites in this module: one isfixed here, one already runs off-loop (the stop-hook nudge cap), and five bare
on-loop loads remain. The First Principles review of
a415e3276is right that thissection previously named only two of the five, so here they all are:
8161898af)schedule_eager_spawn(3229)_empty_auto_continue_enabled(295)_start_next_queued_turn(4224)_run_chat-> nested_queue_recovery(5150)_run_chat-> nested_queue_recovery(5391)schedule_eager_spawn:3229is the consequential one, because it is this feature'sown first load and it is the one that actually pays the cold read on the loop.
First Principles proposes deleting it and gating
session.eager_spawninside_eager_spawnfrom the already-offloadedcfg, which is the right shape: thefeature would then load config once, off-loop.
It is deliberately not done in this PR. This is an adoption rebase of a
contributor's change onto current
main, and restructuring a second function'scontrol flow -- moving the feature's enablement gate from before task creation to
after it, which changes when a disabled feature stops creating tasks -- is a
different change needing its own argument and its own tests. Folding it in here
would trade this PR's short, checkable argument for a much longer one, which is the
same reason the other four are excluded. It should be its own issue and PR, and the
Pattern harvest gate entry below would surface all five as a bounded list.
Pattern harvest
Rule candidate: extend
scripts/check_sync_io_in_async.pywith a small allowlist ofknown-blocking FIRST-PARTY helpers,
KiroCrewConfig.loadfirst among them.The gate that exists for exactly this defect class did not catch it, and the reason
generalises.
check_sync_io_in_async.pymatches<module-alias>.<attr>(...)for thestdlib and client IO modules (
os,pathlib,shutil,subprocess,requests,httpx,sqlite3), plus DB methods and HTTP verbs. Every blocking call here is real-- path resolution,
exists(),read_text(),json.loads(),is_file(),stat(),a deep merge and full jsonschema validation -- but all of it sits one frame down,
behind a first-party synchronous facade, so the AST scan sees only
KiroCrewConfig.load()and reports nothing. The gate passes on this file both beforeand after the fix.
So the miss is structural, not an oversight in the rule's implementation: any
synchronous project-local function that performs IO internally is invisible to it,
and
KiroCrewConfig.loadis the highest-traffic instance of that shape in this tree.An allowlist keyed on the callee name would have flagged this call site at the moment
it was written.
Two adjacent facts worth recording for whoever picks that up.
mainalready carriesawait asyncio.to_thread(KiroCrewConfig.load)at another call site in this samemodule, so the fixed form is the one the file had already settled on -- the defect was
a lone holdout, which is the signature of a rule that should be mechanical rather than
remembered. And five bare on-loop loads remain in
chat_runner.py, inventoriedunder Scope above and all deliberately out of scope here; a gate entry would surface
them as a bounded, reviewable list instead of leaving them to be rediscovered one
review at a time -- which is how
schedule_eager_spawn:3229surfaced on this PR.Fixes #4117