fix(apps): keep the app-config writes off the event loop - #4550
fix(apps): keep the app-config writes off the event loop#4550leonlaiyc wants to merge 2 commits into
Conversation
da5ba08 to
81ec689
Compare
Every config.json update in dashboard/handlers/memory.py reads the file, changes one key and writes it all back on the event loop. The read, the JSON parse, and write_config_atomically -- a tmp-file write plus a rename, which can fsync -- all block, stalling every other session while they run. Three sites do it: api_memory_settings, _write_embed_model_config, and _set_migrated, which runs on EVERY boot while migrated is false and so lands the stall exactly when the gateway is bringing sessions up. This is the class the repo has been closing site by site (kirodotdev#4118, kirodotdev#3803, kirodotdev#4550), and kirodotdev#4946's review named this module. The whole transaction crosses over, never just the read. Offloading the read alone would leave the write on the loop and insert a suspension point between the read and the write-back while the file is unguarded on disk: an external editor, a CLI command or another process landing in that gap would be silently overwritten by a write derived from state nobody re-checked. That gap is zero today because the sequence is synchronous, and it stays zero because the worker performs the whole thing without yielding. The existing per-config lock is held across the hop, so two coroutines still cannot interleave. Two of the three sites also hand-rolled a reader this module already imports. read_config_for_update is the documented companion to write_config_atomically with 27 call sites, and api_memory_settings uses it 200 lines above; _set_migrated and _write_embed_model_config instead caught Exception around json.loads. The helper additionally refuses a non-object top level, where the hand-rolled version accepted a list and then raised AttributeError from setdefault -- a crash where a fail-closed refusal was intended. ConfigReadError is not swallowed by the helper: what to tell the user differs per site, and each keeps exactly the behaviour it had -- skip and retry next boot, raise ValueError, or answer 500 config_unreadable. api_memory_settings now validates its body before the transaction. None of that reads the config, and a 400 previously took the lock and abandoned it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
_write_env_updates stats and reads the whole .env, re-parses it line by line, then creates a 0600 temp file, chmods it, writes and renames. All synchronous file I/O, and six async channel config-save handlers call it. Three already reached it through asyncio.to_thread -- telegram, teams, wecom -- and three called it inline on the gateway loop: slack, discord and webex, stalling every other session for the duration of a token save. So this is not a missing convention but an existing one applied to half the call sites. messaging.py already uses asyncio.to_thread 29 times, and with three siblings doing it correctly nothing in the file said which half was right, or stopped the next channel from copying the wrong one. It is the class the repo has been closing site by site (kirodotdev#4118, kirodotdev#3803, kirodotdev#4550), and kirodotdev#4946's review named this module. The WHOLE call is offloaded, never a part of it: the read-modify-write is one transaction, and a suspension point between the read and the rename would let a concurrent writer's keys be dropped by a write derived from lines nobody re-read. Keeping _write_env_updates one synchronous function on one worker preserves that without depending on the caller, which is now said on the function itself so the next channel inherits the reason and not just the shape. The regression pins all six channels rather than the three that moved, since the defect was the split and not any one site. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every config.json update in dashboard/handlers/memory.py reads the file, changes one key and writes it all back on the event loop. The read, the JSON parse, and write_config_atomically -- a tmp-file write plus a rename, which can fsync -- all block, stalling every other session while they run. Three sites do it: api_memory_settings, _write_embed_model_config, and _set_migrated, which runs on EVERY boot while migrated is false and so lands the stall exactly when the gateway is bringing sessions up. This is the class the repo has been closing site by site (kirodotdev#4118, kirodotdev#3803, kirodotdev#4550), and The whole transaction crosses over, never just the read. Offloading the read alone would leave the write on the loop and insert a suspension point between the read and the write-back while the file is unguarded on disk: an external editor, a CLI command or another process landing in that gap would be silently overwritten by a write derived from state nobody re-checked. That gap is zero today because the sequence is synchronous, and it stays zero because the worker performs the whole thing without yielding. The existing per-config lock is held across the hop, so two coroutines still cannot interleave. Two of the three sites also hand-rolled a reader this module already imports. read_config_for_update is the documented companion to write_config_atomically with 27 call sites, and api_memory_settings uses it 200 lines above; _set_migrated and _write_embed_model_config instead caught Exception around json.loads. The helper additionally refuses a non-object top level, where the hand-rolled version accepted a list and then raised AttributeError from setdefault -- a crash where a fail-closed refusal was intended. ConfigReadError is not swallowed by the helper: what to tell the user differs per site, and each keeps exactly the behaviour it had -- skip and retry next boot, raise ValueError, or answer 500 config_unreadable. api_memory_settings now validates its body before the transaction. None of that reads the config, and a 400 previously took the lock and abandoned it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every config.json update in dashboard/handlers/memory.py reads the file, changes one key and writes it all back on the event loop. The read, the JSON parse, and write_config_atomically -- a tmp-file write plus a rename, which can fsync -- all block, stalling every other session while they run. Three sites do it: api_memory_settings, _write_embed_model_config, and _set_migrated, which runs on EVERY boot while migrated is false and so lands the stall exactly when the gateway is bringing sessions up. This is the class the repo has been closing site by site (kirodotdev#4118, kirodotdev#3803, kirodotdev#4550), and The whole transaction crosses over, never just the read. Offloading the read alone would leave the write on the loop and insert a suspension point between the read and the write-back while the file is unguarded on disk: an external editor, a CLI command or another process landing in that gap would be silently overwritten by a write derived from state nobody re-checked. That gap is zero today because the sequence is synchronous, and it stays zero because the worker performs the whole thing without yielding. The existing per-config lock is held across the hop, so two coroutines still cannot interleave. Two of the three sites also hand-rolled a reader this module already imports. read_config_for_update is the documented companion to write_config_atomically with 27 call sites, and api_memory_settings uses it 200 lines above; _set_migrated and _write_embed_model_config instead caught Exception around json.loads. The helper additionally refuses a non-object top level, where the hand-rolled version accepted a list and then raised AttributeError from setdefault -- a crash where a fail-closed refusal was intended. ConfigReadError is not swallowed by the helper: what to tell the user differs per site, and each keeps exactly the behaviour it had -- skip and retry next boot, raise ValueError, or answer 500 config_unreadable. api_memory_settings now validates its body before the transaction. None of that reads the config, and a 400 previously took the lock and abandoned it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every config.json update in dashboard/handlers/memory.py reads the file, changes one key and writes it all back on the event loop. The read, the JSON parse, and write_config_atomically -- a tmp-file write plus a rename, which can fsync -- all block, stalling every other session while they run. Three sites do it: api_memory_settings, _write_embed_model_config, and _set_migrated, which runs on EVERY boot while migrated is false and so lands the stall exactly when the gateway is bringing sessions up. This is the class the repo has been closing site by site (kirodotdev#4118, kirodotdev#3803, kirodotdev#4550), and The whole transaction crosses over, never just the read. Offloading the read alone would leave the write on the loop and insert a suspension point between the read and the write-back while the file is unguarded on disk: an external editor, a CLI command or another process landing in that gap would be silently overwritten by a write derived from state nobody re-checked. That gap is zero today because the sequence is synchronous, and it stays zero because the worker performs the whole thing without yielding. The existing per-config lock is held across the hop, so two coroutines still cannot interleave. Two of the three sites also hand-rolled a reader this module already imports. read_config_for_update is the documented companion to write_config_atomically with 27 call sites, and api_memory_settings uses it 200 lines above; _set_migrated and _write_embed_model_config instead caught Exception around json.loads. The helper additionally refuses a non-object top level, where the hand-rolled version accepted a list and then raised AttributeError from setdefault -- a crash where a fail-closed refusal was intended. ConfigReadError is not swallowed by the helper: what to tell the user differs per site, and each keeps exactly the behaviour it had -- skip and retry next boot, raise ValueError, or answer 500 config_unreadable. api_memory_settings now validates its body before the transaction. None of that reads the config, and a 400 previously took the lock and abandoned it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every config.json update in dashboard/handlers/memory.py reads the file, changes one key and writes it all back on the event loop. The read, the JSON parse, and write_config_atomically -- a tmp-file write plus a rename, which can fsync -- all block, stalling every other session while they run. Three sites do it: api_memory_settings, _write_embed_model_config, and _set_migrated, which runs on EVERY boot while migrated is false and so lands the stall exactly when the gateway is bringing sessions up. This is the class the repo has been closing site by site (kirodotdev#4118, kirodotdev#3803, kirodotdev#4550), and The whole transaction crosses over, never just the read. Offloading the read alone would leave the write on the loop and insert a suspension point between the read and the write-back while the file is unguarded on disk: an external editor, a CLI command or another process landing in that gap would be silently overwritten by a write derived from state nobody re-checked. That gap is zero today because the sequence is synchronous, and it stays zero because the worker performs the whole thing without yielding. The existing per-config lock is held across the hop, so two coroutines still cannot interleave. Two of the three sites also hand-rolled a reader this module already imports. read_config_for_update is the documented companion to write_config_atomically with 27 call sites, and api_memory_settings uses it 200 lines above; _set_migrated and _write_embed_model_config instead caught Exception around json.loads. The helper additionally refuses a non-object top level, where the hand-rolled version accepted a list and then raised AttributeError from setdefault -- a crash where a fail-closed refusal was intended. ConfigReadError is not swallowed by the helper: what to tell the user differs per site, and each keeps exactly the behaviour it had -- skip and retry next boot, raise ValueError, or answer 500 config_unreadable. api_memory_settings now validates its body before the transaction. None of that reads the config, and a 400 previously took the lock and abandoned it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…5059) Every config.json update in dashboard/handlers/memory.py reads the file, changes one key and writes it all back on the event loop. The read, the JSON parse, and write_config_atomically -- a tmp-file write plus a rename, which can fsync -- all block, stalling every other session while they run. Three sites do it: api_memory_settings, _write_embed_model_config, and _set_migrated, which runs on EVERY boot while migrated is false and so lands the stall exactly when the gateway is bringing sessions up. This is the class the repo has been closing site by site (#4118, #3803, #4550), and The whole transaction crosses over, never just the read. Offloading the read alone would leave the write on the loop and insert a suspension point between the read and the write-back while the file is unguarded on disk: an external editor, a CLI command or another process landing in that gap would be silently overwritten by a write derived from state nobody re-checked. That gap is zero today because the sequence is synchronous, and it stays zero because the worker performs the whole thing without yielding. The existing per-config lock is held across the hop, so two coroutines still cannot interleave. Two of the three sites also hand-rolled a reader this module already imports. read_config_for_update is the documented companion to write_config_atomically with 27 call sites, and api_memory_settings uses it 200 lines above; _set_migrated and _write_embed_model_config instead caught Exception around json.loads. The helper additionally refuses a non-object top level, where the hand-rolled version accepted a list and then raised AttributeError from setdefault -- a crash where a fail-closed refusal was intended. ConfigReadError is not swallowed by the helper: what to tell the user differs per site, and each keeps exactly the behaviour it had -- skip and retry next boot, raise ValueError, or answer 500 config_unreadable. api_memory_settings now validates its body before the transaction. None of that reads the config, and a 400 previously took the lock and abandoned it. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
|
🤖 Kiro Crew [operator: iamwhatever]: This PR has been inactive for 7+ days with failing CI. I've assessed the blockers and they appear resolvable — I'll push fixes directly to this branch as a co-author. Assessment: Two mechanical blockers, neither in the PR's own logic. (1) The branch is If you'd prefer I don't touch this PR, add the |
81ec689 to
bf0ae7c
Compare
|
Pushed 1. BLOCKING — cancellation persists a trust grant without auditing it: ACCEPTED, fixed. 2. FINDING — function-local
3. FINDING — Worth flagging for whoever owns that commit: the merge conflict is entirely in that rider. Test-merging this head against current Also noted, deliberately not taken: the First Principles review's two |
9141210 to
7885b24
Compare
|
👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated. Missing sections:
Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle. |
|
👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated. Missing sections:
Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle. |
|
👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated. Missing sections:
Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle. |
4 similar comments
|
👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated. Missing sections:
Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle. |
|
👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated. Missing sections:
Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle. |
|
👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated. Missing sections:
Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle. |
|
👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated. Missing sections:
Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle. |
|
👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated. Missing sections:
Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle. |
4 similar comments
|
👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated. Missing sections:
Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle. |
|
👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated. Missing sections:
Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle. |
|
👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated. Missing sections:
Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle. |
|
👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated. Missing sections:
Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle. |
Open PR relationship auditThis is a consolidated, point-in-time code-level audit note. It compares complete merge-base diffs and current/merged code; it does not treat a shared topic as duplication or partial coverage as completion. Relationship findings
No PR, Issue, label, branch, or review state was changed by the relationship-note portion of this audit. |
|
👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated. Missing sections:
Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle. |
Current main already offloads the enable and disable writers from the original change. Complete the remaining registry PUT path by moving its entire read-modify-write off the loop and using the shared locked config updater. Compute newly trusted hosts from the state protected by that lock, preserve unrelated config, deduplicate grants, and emit trust events only after the write commits. config.json has two writer generations that do not exclude each other: update_config_locked takes the sidecar advisory flock, while the legacy dashboard writers (agents endpoint, updates.py, security.py, messaging.py, mcp.py, core.py STT) serialize on the loop-side _get_config_lock alone. Dispatch the registry write through dashboard.chat_utils.run_config_write, the one entry point that holds both, so a concurrent legacy PUT cannot commit a stale snapshot over it. The blocking work still runs in a worker, so the loop never stalls. Add deterministic execution-level and AST coverage for worker-thread dispatch, loop-side lock ownership across the write, locked mutation, preserved data, grant deduplication, and failed-write audit behavior.
run_config_write shields its worker and drains it across cancellation, then re-raises CancelledError and discards the return value. A gateway shutdown or a client disconnect landing while the registry PUT's write is in flight therefore committed the trust grant and never reached the caller-side audit: persisted trust with no registries.host_trust_granted record, which is exactly the reconstruction gap that event exists to close. Emit the per-host grant inside _write_registries_config, immediately after update_config_locked returns. Nothing can interrupt a thread between those two statements, so the event and the commit cannot disagree; the emission is still strictly after the write, so a failed write announces nothing; and SEL adapts to an off-loop caller and swallows filesystem errors on a non-critical event, so this cannot turn a committed write into a 500. The handler keeps the registries.update API-outcome event, which correctly belongs to the request. Three tests: a cancellation delivered while the worker is parked inside the locked write (red-before: config committed, zero grants); exactly-once auditing per newly trusted host; and no grant when the locked write raises. Also corrects the run_config_write import comment, which implied the top-level-imports circular-import exemption. There is no cycle on that edge -- every import order was checked. The real reasons are layering and load cost, both now stated and checkable: no module in apps/ imports dashboard at module scope, and hoisting pulls 117 extra modules into every importer of apps.routes (479 -> 596). The import stays lazy; the justification is now accurate. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
7885b24 to
0eb7742
Compare
|
Rebased onto main Clean rebase — no conflicts. The only delta from your old head is one stray Gates run locally on the rebased head: Please review the rebased head. A maintainer push makes the maintainer the last pusher, so under this repo's last-push rule a second approver is needed. Reply here if anything looks wrong. |
|
👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated. Missing sections:
Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle. |
1 similar comment
|
👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated. Missing sections:
Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle. |
Problem / Motivation
PUT /api/apps/registries performed its config.json read-modify-write synchronously on the asyncio event loop and without holding both config-writer lock generations. A concurrent writer could therefore lose unrelated configuration.
The trust audit also had two opposite failure modes:
Current main already contains the earlier enable/disable offload work. This PR is intentionally limited to the remaining registry PUT path.
What changed
The previous unrelated SessionManager teardown rider was removed in full while resolving the merge conflict. The PR now contains exactly two cohesive commits and changes only:
Concurrency and cancellation contract
config.json currently has two writer generations:
Holding only one permits stale-snapshot overwrites across the two families. run_config_write is the existing bridge that holds both while keeping blocking I/O off the event loop. It shields and drains the worker across cancellation, so the grant audit must live with the worker's committed mutation rather than in the cancelled caller.
Tests
The focused file now has 23 deterministic tests covering:
Validation on the pushed head:
No retries, warning filters, relaxed assertions, longer acceptance timeouts, or scheduler-time sleeps were added.
Related pending PRs
The shared apps/routes.py changes in #6854 and #5488 are in separate semantic regions. #6206 is a broader 27-file App Store refresh PR that is itself conflicting; this smaller config-writer correctness fix is resolved first so #6206 can reconcile against the stable owner.
Checklist
Contribution License Agreement