fix: order the config write-back migration against concurrent writes - #7937
Conversation
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Root-cause fix for a real lost-update race: read moves inside the lock, the write shrinks to a delta, and every failure path defers safely. Alternatives (ticket ordering, lock-around-write-only) are examined and correctly rejected; both new keywords ( [DESIGN-REVIEWED] 0dc8a39 |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of All evidence gathered. The only premise-level finding: the repo already has one non-blocking lock primitive ( First-Principles-Verdict: CONCERNS
What this change shipsIntent: stop a first-load write-back migration from silently overwriting a config change saved while it ran — a FIX.
Watch
Subtractions
[FIRST-PRINCIPLES-REVIEWED] 0dc8a39 |
Opus 4.8 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
b6caed4 to
0f73920
Compare
|
Both GPT 5.6 findings on BLOCKING 1 -- migration can block the gateway event loop. Correct, and the Taken the finding, not the prescribed fix. Skipping persistence when Declining costs nothing, which is why not waiting is the right answer rather than Pinned by BLOCKING 2 -- failure paths leak daemon writer threads. Correct as stated. One note on the first fix's blast radius, since it touches a shared primitive: Verification on |
Audit note — #7830 is being closed in favour of this PRTwo independent adjudicators both ruled this the surviving implementation. What the two sharedSame anchor: origin/main Why this oneYes — #7937 is correct, and it is the survivor on code facts rather than on the nomination. #7830 does not close the case it names: its ticket high-water is bumped only in What #7830 had that this PR does notPlease pick these up (or say they are not wanted): One item, and it is a test shape plus a named residual, not the mechanism. #7830's From a repository-wide duplicate/overlap audit of every pull request open against |
0f73920 to
ce24602
Compare
|
Round 2 on BLOCKING 2 -- migration seeds from a stale default agent. Correct, and it was a Fixed exactly as prescribed: BLOCKING 1 -- migration still races legacy dashboard writers. The mechanism is I am not applying the prescribed fix, for three reasons. It is not introduced here, and this change narrows it. Before this PR the It is class-wide, not specific to this caller. Every one of And the prescribed remedy is a revert. "Skip write-back until it can also A regression of my own, found by CI rather than by review. Cause: the test read Still open, and not yet attributable.
Verification on |
ce24602 to
ed43b48
Compare
|
Round 3 on Round 2 said the seed must come from the freshly re-read base document, because a The fixed point is that the seed has to be the MERGED effective value -- that is
The overlay is read OUTSIDE the lock deliberately: it is user-owned and never Worth stating for the record on the prescribed alternative ("or revert this Two tests pin the new precedence, and the overlay-first half is
Also confirmed on
|
ed43b48 to
0dc8a39
Compare
|
Round 4 on FINDING (loader.py:495) -- symlink containment. Fixed. Correct, and it was my BLOCKING (loader.py:724) -- concurrent local override. Real; escalating. The mechanism is right, and the prescribed fix is implementable -- I checked rather I am escalating instead of applying it, on three grounds. This is round 4 on the same two lines, and each round found a defect in the
Every one of those fixes was correct about its own hazard. That pattern is the The conflict. The seeded I checked the one alternative that would dissolve rounds 2-4 together -- seed Why I am not shipping the two-lock version unilaterally. It would have this What the residual actually is, stated plainly. On the first load of a Recommendation, for whoever decides. Merge as is and fix the class properly by
|
|
Round 5 on BLOCKING (loader.py:768) -- "migration performs synchronous file I/O on the event The migration on _write_migration_backup(path) # shutil.copy2 -- a full file COPY
cfg.save() # to_dict() over ~40 dataclasses,
# config.local.json.is_file() + read_text(),
# _subtract_overlay, write_config_atomicallySo
If synchronous migration I/O on the loop is unacceptable, that is a pre-existing Round 1 raised the loop concern in its accurate form: a POSIX Asking for adjudication, because the rounds are not converging.
Six findings accepted and fixed, each mutation-verified. But rounds 2/3 demanded Where that leaves the PR. Everything except this lane is green on Two open decisions for a maintainer, both mine to implement once someone rules:
|
NicholasRBowers
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: fix (4 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: fix with a clear root cause — orders the config write-back migration as a delta against the on-disk document inside the write lock, so a concurrent config write landing after the load's read survives instead of being clobbered by a re-serialized stale snapshot.
Open PR relationship auditThis is a consolidated, point-in-time code-level audit note. It compares complete merge-base diffs and current/merged code; it does not treat a shared topic as duplication or partial coverage as completion. Relationship findings
No PR, Issue, label, branch, or review state was changed by the relationship-note portion of this audit. |
Problem / Motivation
KiroCrewConfig.load()'s write-back migration calledcfg.save(), whichre-serializes the whole snapshot that load had already parsed. Nothing ordered
that write against any other config writer, so:
load()readsconfig.json, sees a legacy shape, decides tomigrate
kirocrew config set) writes a newer configcfg.save()writes the snapshot from step 1ends with step 2's edit gone. The loss is silent and the lost data is user
configuration.
The migration is one-shot per process, so the window is the first load in a
process whose stored config still has a legacy shape (no
agents, adefault_agentnot present inagents, or -- where jsonschema is unavailable --flat workspace strings).
Why it matters
This is pre-existing on
main, not introduced by any open PR:chat_runner.pyalready runs
await asyncio.to_thread(KiroCrewConfig.load)at the stop-hooknudge-cap site, so a worker-thread migration write is reachable today. It becomes
materially more likely as more latency-sensitive reads move off the loop, which
is the right direction for those reads (#4117 / #4118) -- a fix that is correct
for the event loop should not have to carry an argument about a loader-level
write race. #7734's proposed subtraction would make an offloaded load the
process's first load, and so depends on this.
The tell is a strong one: inside one function,
load(), three publishes and onewrite-back all mutate shared state, and exactly one of the four --
publish_autocompact_pct-- is ordered, with a docstring that spells out thehazard ("a load that began earlier cannot overwrite a newer one"). The heaviest
mutation of the four had no equivalent.
What changed (motivation -> approach -> change)
Root cause: a read-modify-write with no precondition.
cfgis read at the top of_load_resolved;save()writes it back at the bottom, unconditionally and infull. Anything that landed in between is replaced.
The issue proposed two shapes. Both were checked against the code and neither is
sufficient on its own:
publish_autocompact_pct's orders load againstload. The writer being lost here is not a load --
next_config_load_tickethasno caller outside
config/loader.py, so a dashboard PATCH never draws a ticketand the comparison never sees it.
before the lock. The read has to move inside it.
So the migration write now has two properties, separated because only one of them
can be applied everywhere:
_apply_document_migrationsapplies only the keys theload decided on, to the document as read at write time, re-deciding each one
against that document. A concurrent write to any other setting is not merely
ordered but untouchable -- those bytes are never part of our write. Verified: a
migrating load now rewrites 4 top-level keys instead of materializing every
section in the schema.
update_config_locked-- the required path forconfig.jsonmutations, whichre-reads inside an advisory lock and writes through
write_config_atomically. So the migration's own keys are decided from currentstate and cannot interleave with another locked writer.
The load keeps deciding whether to migrate, and now records which migrations
it decided on (
pending), so the write never widens beyond what the mergedbase+overlay view asked for. Re-checking each entry at write time also makes the
write idempotent: a migration another writer already performed returns "nothing
to do" and skips the write entirely, rather than rewriting a file we agree with.
One containment detail.
update_config_lockedtakes its lock on a<path>.locksidecar, and
load()reads whateverconfig_path()resolves to -- which callersredirect at their own temp files. A sidecar beside such a path is exactly the
orphan class that produced 72k stray
.bakfiles on one dev host, andTestMigrationBackupContainmentpins that a migratingload()leaves acaller-owned directory as it found it. So the same containment predicate that
already gated the backup now also gates the lock, extracted as
_inside_data_home: contained (always true in production, whereconfig_path()is
config_dir() / "config.json"), take the lock; redirected, do the delta off animmediate fresh read and leave no sidecar. What the redirected path gives up is
ordering on the migration's own keys against an unlocked writer of those same
keys -- and a redirected config has no gateway writing it, since the dashboard and
CLI paths write
config_path().The lock is taken WITHOUT waiting, which is the second thing to get right:
load()is reached from the event loop all over the tree, so a POSIXflockwaithere would trade a rare data-loss race for a gateway stall lasting as long as the
holder keeps the lock. Giving up costs nothing, because a held lock means another
writer is mid-write and that writer's bytes are precisely what must not be
clobbered -- declining is the correct outcome, not a compromise, and the migration
is already retry-on-next-load by construction. So
platform_compat.file_lockgains
wait=False(POSIXLOCK_NB; Windows already behaves this way on the loopthread, and a zero timeout makes it uniform) raising
BlockingIOError, whichupdate_config_lockedexposes aswait_for_lock. Both default to the existingwaiting behavior, so all 69 existing call sites are unchanged.
BlockingIOErroris an
OSError, so it narrows rather than widens what a caller must handle, andit separates "someone is writing right now" from that function's existing
stuck-holder ceiling. The remaining file I/O on the loop -- one read, one atomic
rename -- is what
cfg.save()did here before, unchanged.The backup now happens immediately before the write (inside the lock hold where
there is one), so it captures the bytes actually being replaced rather than
whatever was there when the load started. A failing copy still propagates and
aborts the write, preserving the existing contract: a config we could not copy
aside is not rewritten, and the migration retries on the next load.
One consequence worth stating plainly, because it is visible on disk: a migrating
load no longer materializes every section of the schema into
config.json. Onlythe keys it migrates are written. That is the delta property, and it is also
strictly better for the separate defect where a materialized key pins a shipped
default forever (#5244) -- fewer keys are pinned. One existing test asserted on
the old side effect rather than on its own subject:
test_put_cannot_add_a_gitlab_ hostreadraw["dashboard"]["gitlab_hosts"]back out of the file, with a commentexplaining the key was present only because "cfg.save() serializes every dataclass
field". Its security property -- a dashboard caller cannot authorize a new GitLab
instance -- is unchanged and still asserted; the raw-file assertion now reads the
stored VALUE (absent or empty are the same thing to the loader, which the sibling
cfg.dashboard.gitlab_hosts == []assertion proves) instead of requiring the keyto exist.
Deliberately out of scope: the issue's optional
load(migrate=False)read-onlypath. It is a new public API surface, not part of the ordering defect.
Also out of scope, and worth naming because a reviewer raised it: the migration's
lock does not serialize against the legacy config writers that call
write_config_atomicallydirectly under the in-process asyncio_get_config_lock(the dashboard agents endpoint,
updates.py,security.py,messaging.py,mcp.py,core.pySTT).update_config_locked's own docstring names them aspending conversion. That gap is shared by every one of its ~69 existing call
sites, it predates this change, and this change strictly narrows it -- before, the
migration took no lock at all and rewrote the whole document; now it holds the
sidecar lock and writes three keys. Closing it properly means converting those
writers, not special-casing this one caller, so it is filed as #8032 with the
sites named.
Accepted residue, recorded so the merge carries it: the seeded agent's kiro agent
is the merged value of
agent.default_agent, which comes from two independentlywritten files.
config.local.jsonis read outside the base lock, so aconfig set --local agent.default_agentlanding between that read and the basewrite leaves the seed one write stale. Closing it needs a two-file critical
section that does not exist today and that no other writer participates in;
#8032 is the path that makes one possible. Seeding empty and resolving live is not
an alternative --
resolve_agent_bindingscomputeskiro_agent = passthrough or agent_cfg.kiro_agentwith no fallback toagent.default_agent, so an empty seed yields an empty binding.Tests
TestMigrationWriteBackOrderingintest/test_config_loader.py:test_a_write_landing_mid_migration_survives-- the deterministic two-writerinterleave. Writer A is a migrating
load()suspended after it has decided tomigrate but before its bytes reach the file; writer B is an ordinary
update_config_lockedwrite (the shape the dashboard PATCH andkirocrew config setboth use) settingsession.autocompact_pct. Both must beobservable afterwards. Threads are started and drained in a
try/finallyso atimeout or assertion cannot leave a writer running against this test's paths.
test_the_migration_only_rewrites_the_keys_it_owns-- parametrized over thedata-home path (which also takes the lock) and a redirected path (which cannot,
without leaving a sidecar). Asserted on the top-level key set, because a
snapshot rewrite materializes every section in the schema.
test_a_held_lock_defers_the_migration_instead_of_waiting-- with the lock heldby another descriptor, the migration returns without writing. Run on a thread
with a bounded join and asserted inside the hold, so a regression to a waiting
acquire FAILS on an assertion instead of hanging the suite.
test_a_migration_another_writer_already_did_writes_nothing-- the re-readturns a stale reason to write into a no-op, with no redundant
.bak.test_the_seeded_agent_takes_its_kiro_agent_from_the_reread-- the seededagent's
kiro_agentcomes from the document read inside the lock, not from theload's older snapshot, so an
agent.default_agentchange landing in between ishonored rather than silently reverted for every default session.
test_the_seed_falls_back_when_the_document_names_no_kiro_agent-- with nothingstored, the fallback is the load's resolved value (which carries the overlay and
the dataclass default) rather than a literal.
RED verified against the base commit with the source reverted and the tests kept:
the interleave test fails on "the config write that landed while the migration was
in flight was lost", and the delta tests on "the migration wrote sections it did
not migrate". Two fixes are mutation-verified individually: flipping
wait_for_lock=FalsetoTruereddens the held-lock test on its assertion, andseeding from the snapshot instead of the re-read reddens the seed test. The
pre-existing
TestMigrationBackupContainmentguards pass unchanged -- the lockcontainment above is there because the first draft broke them.
Run:
test_config_loader.py+test_dashboard_config_gitlab_hosts.py510 passed;test_platform_compat*.py+test_no_blocking_call_on_loop.py+test_loop_lock.py585 passed / 22 skipped; 20 furthertest_config*/test_atomic_write*/test_autocompact_default/ lock-regression files 586passed / 1 skipped. black, flake8 and isort clean on all four changed files; mypy
clean on both source files.
Manual verification
N/A -- the defect is a thread interleave with no user-visible surface, and the
interleave is reproduced deterministically in the test above rather than by hand.
Related Issues
Closes #7793
Follow-up filed from this review: #8032 (legacy config writers bypass the advisory
lock), which carries the accepted residue named above.
Pattern harvest
Rule candidate: review-prompt
Pattern: a function that both READS shared state and WRITES it back must carry
the same ordering guard as its siblings, or say in-comment why it does not need
one. The specific tell is checkable by a reviewer rather than a linter: in one
function, several mutations of the same shared state where exactly one is ordered
and its docstring names the hazard -- an unguarded sibling doing a heavier
mutation is a defect, not a deliberate asymmetry.
Second, narrower rule: when the guard you add is a lock, check two things about
it. Where its lockfile lands -- an advisory-lock sidecar is a file, and a file
beside a path the process does not own is litter, the same containment question
the backup in this very function had already been fixed for. And whether the
acquire can WAIT on a thread that must not block: converting an unordered write
into a correctly ordered one is not a win if the ordering is bought with a
blocking wait on the event loop. For work that is optional and retried, the
answer is a single-shot acquire that declines.