fix(dashboard): preserve concurrent same-key recreate during slot-close teardown (#7191) - #7212
Conversation
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Design Review (Fable 5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS Sound yield-to-replacement design for a real race, honestly scoped; the residual risks are convention-maintained metadata ownership and a retry-ambiguous shared error code. Watch
Suggestions
[DESIGN-REVIEWED] 86a61b8 |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of All checks are complete. The change is a declared fix with every mechanism derived from a named harm; the one first-principles finding is a counted unfixed sibling of the same pop→await→destroy pattern on the history-delete path. First-Principles-Verdict: CONCERNS The teardown-race fix is real and complete for the close paths, but the same pop→await→destroy pattern survives unguarded in history delete. What this change shipsIntent: stop a tab closed mid-recreate from archiving, tearing down, or clobbering the replacement that took its key — a FIX.
Watch
[FIRST-PRINCIPLES-REVIEWED] 86a61b8 |
Opus 4.8 Review — ✅ no blocking findingsReviewed Review detailsI have enough to conclude. Let me verify the one remaining fact for CANDIDATE 1 — that the empty-window branch ( My analysis: CANDIDATE 1 — grounded and survives. A message-less slot can carry a deferred CANDIDATE 2 — dropped. Line 1 punchline note: there is no BLOCKING finding, so no Message-less dirty slots lose an acknowledged metadata edit on the new hand-over drain. FINDING — src/kiro_crew/dashboard/chat_handlers.py:388 — for a message-less slot with a deferred [OPUS-REVIEWED] 86a61b8 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
80a0dc6 to
8964ed5
Compare
8964ed5 to
f3e5e0e
Compare
Round 3: escalating instead of pushing a fourth patch
Three consecutive rounds have landed blocking findings in this one Every exit of both teardown paths
The invariant I would implementNONE ADOPTED — no code changed this round. The invariant I would implement, and am handing over: A teardown may commit durable state only under a predicate evaluated INSIDE the same lock that commits it. Split by where state lives: loop-owned state ( Shape, with an in-repo precedent: thread Why that makes the class unreachable rather than narrower: if the recreate lands after the in-lock predicate, the row was CORRECT when written (the replacement did not exist yet) and the replacement merely inherits it — which I measured to be byte-identical to the ordinary sequential close-then-reopen. That is not a race; it is the designed lifecycle of a reused key, and it is what the resume path's in-lock compare-and-clear exists to compensate. Why I did not implement it here: the finding's actual bite comes from the OTHER half of the same TOCTOU, in
What needs your ruling
Notes
|
f3e5e0e to
2f81851
Compare
2f81851 to
386d513
Compare
|
Resolved: every check is green at
|
Open PR relationship auditThis is a consolidated, point-in-time code-level audit note. It compares complete merge-base diffs and current/merged code; it does not treat a shared topic as duplication or partial coverage as completion. Relationship findings
No PR, Issue, label, branch, or review state was changed by the relationship-note portion of this audit. |
chenmingwei23
left a comment
There was a problem hiding this comment.
Approving: PR Readiness green (the repo's only required check), no failing lanes, MERGEABLE.
Summary
Fixes #7191
This closes the post-pop teardown race in the shared dashboard slot-close sequence.
Both
api_chat_slot_deleteandapi_chat_slots_cleanuppopnamefromstate._slotsand then run several awaits (task cancel,save_slot_off_loop(..., closed=True),sessions.remove(_history_key_for(name))). A concurrent same-key recreate (aPOST /api/chat, or thesession_closeMCP verb) can mint a replacement slot for the same key inside that window. The in-flight original then (a) writes its transcript over the replacement's shared history asclosed=True, and (b) tears down the session the replacement now uses. The failure arms compounded this by blindly restoring the original (state._slots[name] = slot) over whatever now owned the key.Approach
Routing facts on the current base: the single-tab teardown lives in
close_slot, shared byapi_chat_slot_deleteand session-control'sclose_target, whileapi_chat_slots_cleanupcarries its own copy of the same shape; thenote_slot_closedtombstone already fires pre-teardown but the creation path never reads it.Chose Option 2 (ownership re-check at teardown) over Option 1 (consult-tombstone-at-creation): both destructive sites already hold the popped object, so a synchronous, mechanism-free discriminator is available that does not penalize a legitimate fast recreate. Option 1 would require inventing reservation semantics on the hot creation path.
There are two discriminators, because the destructive steps do not all answer to the same owner and one answer cannot gate resources with different owners:
_slot_still_ours(state, name, <popped>)— "does a DIFFERENT object holdname". Governs the key-scoped steps:sessions.remove(_history_key_for(name))(the session an unbound replacement runs on), the failure-arm_slotsrestores, cleanup'sarchivedreport, and the app-dismissal decision coupled to the restore._replacement_shares_transcript(state, name, <popped>)— "does that different object write MY transcript". Governs theclosed=Truesave alone, whose resource is the transcript rather than the key._save_slot_to_historytargetsslot_history_key(slot), so a channel-, cron- or workflow-linked slot writes itslinked_session_keywhile a replacement minted by a plainget_or_create_slot(name)(whatPOST /api/chatand thesession_closeverb do) is unbound and writesdashboard:{name}. Same key, two files. Yielding the archive on such a pair leaves the ORIGINAL's transcript with noclosedflag, andchannel_slots._close_standsreads an absent flag as "the user never dismissed this" — so the reconcile pass resurfaces the tab that was closed. It compares FILE identity (transcript_stemson both sides) rather than key strings, becausehistory._safe_keyfoldsslack:<ts>and theslack_<ts>stem onto one.jsonland a pre-migration thread still resolves to its barethread_tsstem; the two errors are not symmetric, since over-reporting "shared" only declines an archive the next close will make while under-reporting stampsclosedon a file a live slot is writing.The two paths were deliberately not merged further: they diverge past the shared shape (nudge retirement + app-notify in
close_slot;flush_deferred_notes+ batched task cancel + error-row append in cleanup). What they share is three pure helpers — the two predicates above and the_resettle_restricted_key(state, name)postcondition — plus the_persist_handover_taildrain.The guards are applied at three points per path, each with the predicate its resource answers to:
closed=Truesave —_replacement_shares_transcript, so the archive yields only to a replacement that actually holds the file,sessions.remove—_slot_still_ours,_slot_still_ours(now only restores when the key is free or still the popped object).Cleanup additionally skips
archived.appendunder_slot_still_ours, so a key with a live holder is never reported swept —archivednames slot keys, whatever became of the transcript.Yielding to a replacement carries four obligations, because the state a close compensates is not all scoped the same way:
state._restricted_keysholds a session KEY (dashboard:{name}) and_is_restricted_sessionreads it before it looks at the slot, so every teardown exit now routes through one postcondition —_resettle_restricted_key: the key is marked iff the slot currently at it is restricted. A barediscardthe hand-over exits skipped handed a persistent replacement an incognito original's 403; re-deriving rather than discarding keeps a restricted replacement's own marker, since dropping it is the fail-OPEN direction.notify_slot_close_undoneis now conditional on the original getting its key back, not onslot._appalone. With a replacement on the key there is no tab to put back, and resuming the crew would let its watchdog grant auto-approve — then an unbounded nudge clock — to the user-owned slot now holding that key._flush_dirty_slotsiterates exactlystate._slots, so an unreferenced slot has NO retry path:messages[_disk_window_len:], plus any note the bulk path still holds in the in-memory-only_deferred_notes, would simply cease to exist. The pre-save exits need no store failure to reach that — theyreturnbefore the save is attempted, in a window that opens while a turn is in flight, so the rows at risk are typically the reply the user was watching. Every hand-over exit therefore routes through_persist_handover_tail(state, name, slot): it flushes held notes into the window and writes it withclosed=False. Those rows belong on the ORIGINAL's transcript whether or not the replacement shares it; what must not happen is the archive stamp and the session teardown, not the write. The write target isslot_history_key(slot)and never a deriveddashboard:{name}—_save_slot_to_historyresolves its own target the same way and REFUSES a save whoseexpected_history_keynames a different transcript, so the derived form would make the drain a silent no-op for every cron-, channel- or workflow-linked tab and would name a row-less file in the failure log. It is non-destructive against the replacement in both directions —_save_slot_to_history's foreign-append scan carries through every on-disk line the saved window does not represent, so rows a replacement already committed survive. The METADATA line is not this slot's to move, so the write isrows_only:_save_slot_to_historyis otherwise authoritative forSLOT_OWNED_META_KEYSand REBUILDS that line from whichever slot it is handed, so a default save here would revert a folder, pinned title, tag or pin the replacement had already published (POST /api/chat/slotspersists both at birth) — silently undoing an acknowledged edit, and for a tab nobody types in again undoing it for good, so the next restart resurrects the dismissed tab's name and filing.rows_onlykeeps the on-disk value for each of those fields and narrows this write's ownership toROWS_ONLY_OWNED_META_KEYS(the file's identity and accounting, which every writer maintains and which the rebuild reads out of the existing line anyway). The set it defers is named in full asROWS_ONLY_DEFERRED_META_KEYSrather than derived asSLOT_OWNED_META_KEYS - ROWS_ONLY_OWNED_META_KEYS, because that difference under-approximates: the slot save also writes fields that DESCRIBE an owned one without being owned themselves, and a title's provenance and refresh budget (title_origin,title_refresh_mark) travel WITH the title rather than with the writer. Deferring the title while keeping those would commit a line matching NEITHER slot — read back beside another slot's title they either unlock the background title refresh on a name the user typed by hand or lock a generated name out of refresh permanently — so they are deferred with it,created_byandoriginare the same shape with AUTHORIZATION rather than presentation behind them, so they are deferred too:created_byis what session-control's member ownership boundary reads and is meaningless without themodedeferred beside it, andoriginmust round-trip with the deferredappbecause that pair decidesslots:uservisibility and the unattended approval window. The conversation's own MONOTONE once-flags (auto_tagged,human_seen,channel_origin,channel_folder_filed) are set and never cleared, so two writers on one transcript cannot disagree about them in a way that outlives the pair; they stay as written. Deferring to disk is deliberately NOT the same as re-deriving the line from the replacement: a recreate that published nothing has no metadata to protect, and deriving from it would ERASE a real title and filing the shared conversation has — leaving the line alone is what gets both directions right with one write. The deferral is conditional on there BEING another writer to defer to, decided at the save from the line'stab_id— the one per-writer mark it carries, minted per slot object (get_or_create_slotassigns a fresh uuid, a rehydrate adopts the file's) and stamped by every save: fields are held back only on a line ANOTHER slot published, while a line this slot published itself, or no line at all, takes the ordinary rebuild. Unscoped it would cost the original its own uncommitted metadata, because a rename, re-file, tag or pin is acknowledged the instant it lands on the slot (_dirty) and persists on a later_flush_dirty_slots— which iteratesstate._slots, so no flush ever visits a popped slot again, and no failed save is needed to reach that state. Unprovable ownership defers, since the two errors cost differently: a deferred edit was never committed, while a rebuild over a live holder's line reverts what it published and nothing rewrites that for a replacement nobody types in again.tab_idis itself deferred, so a deferring write leaves the other writer's id in place rather than flipping the next drain's answer.closed/closed_atare deferred on that same asymmetry, so the drain is open-shaped without being un-closing. On a line the replacement published, aclosedflag is that holder's own DISMISSAL, and the drain races its close for the transcript lock: erasing it would resurface a tab the user put away, permanently, since both slots are popped by then and nothing rewrites the flag. Leaving a stale one costs nothing durable, because the live holder owns those keys on its next full save. The only path that clears a stale flag from outside the holder is the resume route, and it clears one only when it can prove the close predates its own boundary (clear_closed(..., only_if_closed_before=...), compared inside the store's lock) for exactly this reason. Clearing a stale flag is therefore the job of thetab_idfallback: on a line THIS slot published there is no other holder's dismissal to lose, so the ordinary rebuild runs and the open-shaped write erases it. The failure arms take the same route in place of the restore they skip: a store that rejected theclosed=Truewrite can still accept the next one, and a lock lost to the recreate is exactly that case. When even that write fails the loss is unrecoverable, so it is logged with the exact row count rather than left silent._persist_handover_tailreturns whether rows were owed and reached disk, and every caller honours it, because this frame is the last reference to those rows: nothing retries and nothing else can ever report them. Both PRE-SAVE hand-over exits therefore turn aFalseinto their own path's failure —close_slotraisesSlotCloseError(code="history_save_failed"), the same code an ordinary failed archive raises since from the caller's side it is one thing, and cleanup adds the key tofailed. There is nothing to roll back on either exit (the original is popped, cancelled, and a live replacement holds the key), so the report IS the whole remedy; answering 200 there claims durability the close does not have. The two FAILURE-arm drains need no branch of their own — those arms already end inSlotCloseError/failed.append(name), so a lost tail reaches the caller regardless.What this does NOT close
The guard covers the WIDE window (the app-notify awaits before the pop, the up-to-2.0s task cancel) and not the durable write:
save_slot_off_loopreaches its commit through the process-wide default executor, so a recreate can still land between the last synchronous check and the in-lock write, leavingclosed=Trueon a key a live replacement holds. That residual is what an unguarded close carries too — it is measurably identical onmain— and the row it leaves is the one a plain sequential close-then-reopen of a reused key already produces:closed/closed_atare inSLOT_OWNED_META_KEYS, so the replacement's next full save drops them, andapi_chat_slot_resumecompensates a stale flag with an in-lock compare-and-clear. Closing it AT the commit needs an ownership predicate inside_locked(history_key)on the write AND on the resume's read-then-clear — a durable-metadata contract change across three modules, tracked as follow-up rather than folded in here. Both the code comment andsession.mdstate this scope explicitly. Note the distinction from the row loss above: that residual is a staleclosedFLAG that the replacement's next full save drops, not a lost message — the hand-over exits no longer discard content.Also out of scope:
_remove_slot_for_history_keyinsrc/kiro_crew/dashboard/handlers/sessions.pyhas the same unguarded pop-then-await shape (it pops the slot key, awaitscrew.purge_slot, pin cleanup and a 2.0s task cancel, then destroys the session) with no identity re-check. It is a permanent history DELETE rather than a close, so the recreate it would race is a different intent and the compensation differs; it is named here so the gap is on the record rather than folded into this diff.Files changed
src/kiro_crew/dashboard/chat_handlers.py: the_slot_still_ours(key-scoped) and_replacement_shares_transcript(transcript-scoped) predicates, the_resettle_restricted_keypostcondition, the_persist_handover_taildrain and the honouring of its result, and the guards at all six post-pop sites across both handlers.src/kiro_crew/dashboard/chat_persistence.py: therows_onlysave mode — write the window, leave the metadata line's slot-owned fields as they stand on disk. DefaultFalse, threaded throughsave_slot_off_loop, and used by exactly one caller (the hand-over drain), so no existing save path changes shape.src/kiro_crew/history.py:ROWS_ONLY_OWNED_META_KEYS, the subset ofSLOT_OWNED_META_KEYSa rows-only save still owns, andROWS_ONLY_DEFERRED_META_KEYS, the set such a save must drop so the on-disk values are carried back — wider than the owned difference, because the rebuild also writes unowned fields that describe an owned one. Both defined next to the ownership vocabulary they partition.docs/system-specs/modules/session.md: documents the two post-pop predicates and which step each governs, the four obligations yielding to a replacement carries, and the residual the guards do NOT close.test/test_slot_close_recreation_race.py(new): concurrency tests driving both handlers directly with deterministicasyncio.Eventinterleaving — covering both predicates' polarity on their own, the first pre-save guard (via a parked running turn), the second guard, both failure arms, the failure-arm compensation asymmetry, the inert common path, the divergent-transcript case at both handlers (an unbound recreate over a linked tab: the original's own transcript IS archived with its tail, while the replacement keeps its slot and session), the drain-failure report at both handlers (a 500 withhistory_save_failed, afailedentry), five durability cases that run the REALsave_slot_off_loopagainst a realConversationLogand pin BOTH halves at once (the replacement survives AND every row of the original reaches disk, unclosed — including alinked_session_keyslot whose recreate is bound to the same key, which pins that the drain authorizes the slot's own transcript), and eleven metadata cases that pinrows_onlyfrom both sides (a published replacement keeps its title and folder AND its title's provenance and refresh budget, so the committed line matches one slot rather than half of each; a blank replacement does NOT erase the original's; a transcript with no line yet still gets the slot's own; a dismissal the replacement committed onto its own line survives the drain, while a staleclosedon the ORIGINAL's own line is still erased; both windows' rows survive the write; the bulk path and the delete failure arm inherit the same restraint; the drain's OWN uncommitted rename, re-file and pin reach disk when the line is its own, with the save driven directly so the id on the line is the only variable; and a published replacement still outranks the drain's pending edit, which is the polarity guard a fix that merely stopped deferring would fail). Each destructive assertion fails if its corresponding guard is reverted, and each durability assertion fails if the drain is.Testing
test/test_slot_close_recreation_race.pydrives both handlers directly (not through a client) so the concurrent recreate is scheduled deterministically inside the teardown window, viaasyncio.Events the monkeypatchedsave_slot_off_loopparks on. It covers each predicate's polarity on its own, the first pre-save guard (through a parked running turn), the second guard, both failure arms, the app-dismissal decision on both hand-over exits, the restricted-marker hand-over including a restricted replacement, and the inert ordinary path at both sites. Each destructive assertion fails if its corresponding guard is reverted.Three groups pin what this round changed, and each was proved by reverting the change and watching exactly those tests red:
test_delete_divergent_transcript_still_archives_the_originaland its cleanup sibling give the original alinked_session_keyand let an unbound same-name recreate land in the cancel-await; they assert the linked transcript comes outclosedWITH the original's tail, and that the replacement keeps its slot and its session. Reverting_replacement_shares_transcriptto the key-only answer reds both with "the dismissed linked transcript was left open to resurface".test_shares_transcript_compares_files_not_key_stringspairs a bound channel slot with achannel_originreplacement whose stem never resolved — two key strings, one.jsonl— and asserts the pair reads as shared. Reverting the comparison toslot_history_key(a) == slot_history_key(b)reds it, which is the direction that would stampclosedon a file a live slot is writing.test_delete_handover_write_failure_fails_the_close_and_names_the_rowsasserts the 500 and thehistory_save_failedcode alongside the row-count log (and that the replacement is still untouched — reporting the loss is not a licence to undo the hand-over);test_cleanup_handover_write_failure_is_reported_failedasserts thefailedentry. Dropping the twoif not drained:branches reds both.Three pre-existing tests moved with the behaviour rather than being loosened: the linked-slot drain test now binds its recreate to the same key (which is what a cron re-injecting the same job produces, and is what makes it a hand-over at all — the unbound pairing is the new divergent test), the drain-failure test now expects the 500 it always should have, and the three restricted-marker tests whose
save_slot_off_loopstub raised on EVERY call now record instead, since raising also failed the drain and turned those exits into the drain-failure case. Each still pins what it pinned before: exactly one write,closed=Falseandrows_only.The durability half runs the REAL
save_slot_off_loopagainst a realConversationLograther than a stub, in five cases: the pre-save hand-over persists the tail and keeps the replacement; the same exit on alinked_session_keyslot writes that slot's own transcript (and nothing ontodashboard:{name}); a store that cannot take the tail reports the exact row count; the delete failure arm's hand-over persists the tail; and cleanup's hand-over persists the tail plus the note it was still holding in_deferred_notes, leaving that list empty. Stubbing_persist_handover_tailto return early reds exactly those five plus the three guard assertions that changed from "no save" to "noclosed=Truesave".The close flags' side of the deferral, proved by reverting.
test_delete_handover_keeps_a_dismissal_the_replacement_committedpublishes the replacement's line, commits its dismissal atclosed_at=1234.0while the original's drain is still owed, and asserts both flags survive with the instant intact; puttingclosed/closed_atback intoROWS_ONLY_OWNED_META_KEYSreds it with "the drain erased a dismissal the replacement committed". Its siblingtest_delete_handover_erases_a_stale_closed_flag_on_its_own_linepins the other branch — the replacement publishes nothing, the drain meets the original's own line, and the erase still happens. Both assert_disk_window_len == 2before the close, because the earlier single test published AFTER the tail was appended, which committed the tail and reduced the drain to a no-op that satisfied every line assertion vacuously.Local gate green:
isort,flake8 src/kiro_crew test,mypy --platform linux src/kiro_crew(0 errors, 1,281 files),scripts/check_black_formatting.py,scripts/check_subprocess_encoding.py,scripts/check_brand_name.py,scripts/check_harness_parity.py,scripts/docs-lint.sh, plus every test module that touches a slot save, a metadata carry or the close flags — the 49 modules matchingsave_slot_off_loop/_save_slot_to_history/carry_unowned_metadata/SLOT_OWNED_META_KEYS/clear_closed/update_metadata_if(3,669 passed).scripts/run_scoped_tests.py --surface backendescalates this diff to the full suite, which was run once.Pattern harvest
Rule candidate: semgrep (new rule,
semgrep/popped-key-identity-guard.yaml, alongside the existingfind-sentinel-truthiness.yaml+ itssemgrep-tests/fixtures)Pattern: within one function body,
$D.pop($K, ...)assigned to$V, followed later by an identity guard on$D.get($K) is $V(or$D.get($K) is not $V). The comparison is structurally near-constant: the key was just removed, so on the ordinary path the lookup isNoneand the guard answers the opposite of what the author meant. Both directions are expressible as an orderedpattern: | $V = $D.pop($K, ...) \n ... \n $D.get($K) is $V, and the fix the rule should suggest is the three-way form that classifies the absent state explicitly (current = $D.get($K); current is None or current is $V).Why this class and not a one-off: a post-mutation ownership re-check that fails toward "skip the work" is invisible in every direction a reviewer normally looks. This one type-checked, passed flake8/black/isort, read correctly in prose (the docstring faithfully described the wrong predicate, so prose and code agreed), and returned HTTP 200 on the broken path — the only observable was a leaked kiro-cli session per tab close, an unwritten transcript, and
archived=0from bulk cleanup. It is the same shape as the repo's ownis_kiro_clifail-OPEN lesson in AGENTS.md (§ Harness parity) generalized off the harness axis: a guard whose default answer is the permissive one goes unnoticed until someone pays for it.Secondary rule candidate: AUTOSDE.yaml
recurring-defect-patterns— add a bullet for the reviewer-judgment half that semgrep cannot see. "A diff that answers the SAME ownership/liveness question with two DIFFERENT predicates is self-contradicting; one of them is wrong. Flag the disagreement, do not guess which side is intended." This PR contained both spellings: the shared_slot_still_oursusedget(name) is slot, while the two failure arms it was written to serve hand-rolledcurrent is None or current is slotinline. The correct answer was already present in the diff, four lines away from the defect, in the author's own hand — which is a mechanically detectable signal and stronger evidence than any single site read alone. This slots naturally next to the existing "docstring that CONTRADICTS the code below it" bullet, as the code-vs-code variant of it.Not generalizable (the second half): the accompanying
mypyfailure — areturn web.json_response(...)left insideclose_slot(...) -> None— is not a defect class, it is an unrun gate. The change was authored against a base where that teardown was still inline inapi_chat_slot_delete, and the sandbox had nopytest-asyncio, so the PR shipped on static checks with its own nine new tests never executed; seven of them fail or hang against the code as written. No lint rule retires that. The existingpython3 scripts/check_black_formatting.py && ... && mypy && python -m pytestgate in AGENTS.md already covers it, and CI caught both within minutes.