fix(session): rescue queued follow-ups from a mid-turn reset - #3753
fix(session): rescue queued follow-ups from a mid-turn reset#3753adiarora06 wants to merge 1 commit into
Conversation
|
👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated. Missing sections:
Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle. |
GPT 5.6 Review (fork) — 🔴 changes requested (blocking)Reviewed 3 of 3 blocking finding(s) are security-class and were withheld from adjudication, so the blocking verdict stands. BLOCKING -- src/kiro_crew/session_lifecycle.py:436 -- Stop can miss a queue published after suspension BLOCKING -- src/kiro_crew/session_allocation.py:1449 -- Registration cancellation loses the rescued queue BLOCKING -- src/kiro_crew/session_allocation.py:1449 -- Queue adoption reverses message order FINDING -- src/kiro_crew/session_lifecycle.py:451 -- [BLOCK-MERGE] 92e6dbe Adjudication (Opus 4.8) — is blocking on each finding proportionate?The adjudicable input block is empty (0 findings). Three fenced findings (F1–F3) require annotation. My analysis, from code opened this run: F1 — reset() publishes the rescue inside F2 — the adoption pops F3 — a cold start driven by a new inbound message adopts the older rescued queue into None of the three admits a complete extreme-rarity + recovery record; the asymmetry rule resolves each to UPHOLD-FENCED. |
Design Review (Fable 5, fork) — 🔴 BLOCK (blocking)Design-level review of Design-Verdict: BLOCK The description's fifth and sixth BlockersPhantom description: two documented-as-fixed correctness gaps are not in the patch. Watch
[DESIGN-REVIEWED] 92e6dbe |
First Principles Review (Fable 5, fork) — 🟡 CONCERNSPremise-level review of All claims verified against base. I have what I need to write the review. First-Principles-Verdict: CONCERNS The rescue ships as a third parallel waiting-message store, patched into 8 orphan-aware special cases, where decoupling the queue from Not justified as shipped
What this change shipsInventory (7 items) — 6 justifiedIntent: stop follow-up messages sent mid-turn from silently vanishing when that turn's failure resets the session — a FIX (provenance: #3752, plus added tests that fail on base, where
WatchThe five review rounds each found another session-gated path blind to the shadow store, and one counted sibling stays unfixed by design: of the 4 queue accessors gating on SubtractionsReplace [FIRST-PRINCIPLES-REVIEWED] 92e6dbe |
cd9cff5 to
705d706
Compare
|
👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated. Missing sections:
Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle. |
Opus 4.8 Review (fork) — ✅ no blocking findingsReviewed |
705d706 to
db77b89
Compare
db77b89 to
10c6c54
Compare
d4bddad to
383a689
Compare
383a689 to
f3a39ad
Compare
f3a39ad to
72e9e27
Compare
72e9e27 to
c13b4d7
Compare
|
Responding to GPT 5.6 Review's Rather than the suggested fix (revert the no-session orphan-draining branch in |
|
🤖 Kiro Crew [operator: iamwhatever]: This PR has been inactive for 5+ days. I reviewed the blockers but they require your input:
When you've addressed these, the pipeline will re-assess on its next cycle. |
|
Scope recommendation from triage of #3752: land the narrow Rationale: the narrow fix closes the user-visible data loss (queued follow-ups silently dropped on reset) with a small reviewable diff; the manager-level move changes ownership semantics across every teardown path and deserves its own review rather than riding a bug fix. The branch currently shows CONFLICTING against main, so it needs a rebase either way -- rebasing the narrow version is the fastest route to green. |
bolichen97
left a comment
There was a problem hiding this comment.
Description / code mismatch
The Description asserts a green local lint-and-type gate for exactly the files this PR changes, and the type-check gate is red on the head SHA for errors these lines introduce.
1. "mypy clean on touched files" is contradicted by the diff's own lines and by the red Backend Lint & Type Check run
The Description says —
flake8/isort/mypyclean on touched files (mypy's 3 pre-existinghooks.pyLinux-xattr errors are unrelated and present on unmodifiedmain).
The code does — mypy is not clean on the touched files. Running the repo's pinned mypy under its own pyproject config against the head tree yields five errors that are absent at the merge base: src/kiro_crew/session_lifecycle.py:382, :384, :670, and :672. The cause is that the rescue hunks read and write session.queue through the _SessionEntry Protocol that types the _sessions mapping, and that Protocol declares no queue member (src/kiro_crew/session_lifecycle.py:382; the Protocol itself is at src/kiro_crew/session_lifecycle.py:33-38 in the head blob, and at :43-48 with _sessions at :79 in current main). The flake8 and isort half of the sentence is accurate — all three touched source files are clean at head — so the inaccuracy is confined to the mypy claim.
Risk — mypy src/kiro_crew is a blocking gate in this repo, and Backend Lint & Type Check (3.10) is failure on the head SHA. Because the Description pre-attributes mypy's output to three known unrelated hooks.py xattr errors, a maintainer reading it would take the gate as satisfied and treat the red job as this repo's documented environmental noise. It is not environmental: the errors are on the lines this PR adds, and the fix is one line.
The same paragraph's suite claims (307 passed, 436 passed, 2165 passed) and the ticked Existing tests pass checklist item sit against Backend Tests (3.10, 3), Backend Tests (3.12, 3), and Backend Tests (Windows) (3) all reporting failure on the head SHA while shards 1, 2, and 4 plus the namespace-sandbox shard are green on all three platforms. One shard index failing on three different runners and operating systems is the signature of a deterministic failure rather than a flake. The failing test cannot be named from the cached check-run data, so this half is corroboration for the mismatch rather than an independent finding.
Required change — Either add queue to session_lifecycle._SessionEntry with its concrete type so the two rescue hunks type-check (and address the Optional narrowing at src/kiro_crew/session_lifecycle.py:384), then re-run mypy src/kiro_crew and the failing shard; or remove the mypy word from the claim and the passing-suite counts from the Description, and state what is red on the head SHA.
|
@adiarora06 Thanks for this one. Issue #3752 is still open and the bug is still live on The only neighbouring work that landed is #3776 (for Issue #3768), which unlinks temp files for live queue entries; your branch correctly reuses Two asks before it can land. First, please rebase: at the sha we audited ( One conflict to expect: open #7161 rewrites the same Posted from the 2026-09-08 open-PR relationship audit (read-only, one auditor per PR); reply here if any of this is wrong. |
Problem / Motivation
A message sent while another is being processed on the same key is queued behind it via
sessions.enqueue(), living on that_Session's own.queuedeque untildequeue()picks it up after the current turn finishes.reset(key)— called directly by theAcpPromptBusyhandler inslack/handler.py(fires on the very first occurrence, no threshold), and byrecord_failure's circuit breaker forAcpTimeoutError/AcpProcessDied/AcpError— pops the whole_Sessionobject and tears it down mid-turn without ever looking at its.queue. Any follow-up messages waiting in it go with it.Why it matters
When the failed turn's task finishes,
_on_done/_on_transport_doneinslack/events.pycallsdequeue(key)expecting to find the queued follow-ups, but the session that held them no longer exists (or a different, empty-queue session has taken its place) — the messages are silently and permanently lost, with no notice to whoever sent them.stop_turn()already shows awareness of this class of problem — it explicitly callsclear_queue(key)before a user-initiated hard-kill reset, skippable viapreserve_queue=True. But that's a deliberate user-requested stop; the many failure-recovery paths intoreset()have no equivalent handling, and the user there never asked for their queued follow-ups to be dropped — they just happened to send messages while a turn that was about to fail transparently was still running.What changed (motivation → approach → change)
Symptom: follow-up messages sent while a turn is in flight vanish if that turn later fails. Root cause:
reset()discards the whole_Sessionobject, including its pending-message queue, with nothing preserving it before the object is torn down.Fix:
reset()now rescues a non-empty popped session's.queueinto new manager-level state (self._orphaned_queues: dict[key, deque]), reusing the deque object directly.get_or_create()'s cold-start path (registering a brand-new_Sessionfor a key) pops any rescued queue for that key and seeds the new session's.queuewith it, sodequeue()finds the messages once dispatch next runs for that conversation.destroy()anddiscard_conversation()are unchanged — they're deliberate, user-visible wipes (permanent history deletion, poisoned-conversation escalation), where dropping anything queued behind them is the expected behavior, same asstop_turn's explicitclear_queue()for a user-initiated hard kill.This rescues messages from permanent loss but doesn't force immediate re-dispatch the instant
reset()runs — they're picked up whenever a session next opens for that key (naturally, the very next inbound message for that conversation), the same waydequeue()has always worked. Wiring truly immediate redispatch would meansession.pyreaching into Slack's_dispatch_queued/_on_donemachinery inslack/events.py, which is a layering violation for a much smaller, contained fix.Follow-up fix from review: GPT 5.6 Review correctly flagged that a rescued queue was invisible to
clear_queue(): ifreset()already rescued a queue into_orphaned_queuesand no new session had opened for the key yet,stop_turn()'s own early-return on a missing session (if not session: return "idle") meantclear_queue()never ran at all — so a/stopissued in that window would not clear the rescued messages, and they'd resurface once a session next opened for the key, exactly what the stop was meant to prevent. Verified by tracingstop_turn's early-return andclear_queue's session-gated body directly — confirmed real.clear_queue()now also dropsself._orphaned_queues.pop(key, None)unconditionally, not just a live session's own.queue.Second follow-up fix from review: a later GPT 5.6 Review pass flagged that the fix above still wasn't reachable through the real call path:
stop_turn()'s own early-return (if not session: return "idle") happens before it ever callsself.clear_queue(key)— so in the exact no-live-session scenario the earlier fix targets,clear_queue()(and its_orphaned_queuesdrop) never actually executes. The earlier test calledclear_queue()directly rather than going throughstop_turn(), which is why it didn't catch this.stop_turn()now callsclear_queue(key)on the early-return path too (whenpreserve_queueis false), before returning"idle".Third follow-up fix from review —
[BLOCK-MERGE]: GPT 5.6 Review flagged thatcancel_queued()had the same class of gap as the twoclear_queue/stop_turnfixes above, but for individual-message cancellation rather than a bulk clear: it only ever checked a live session's.queue, so a message rescued into_orphaned_queues(no live session at that key yet) could not be cancelled — a delete/cancel issued in that window silently failed (cancel_queuedreturnedFalsewith no side effect), and the "cancelled" message ran anyway once a later message adopted the rescued queue. Verified by readingcancel_queued,dequeue,enqueue, andis_cancelleddirectly: all four gate onself._sessions.get(key)and return early with no orphaned-queue awareness at all. Fixed the same way as the other two:cancel_queued()now also searches_orphaned_queues[key]when there is no live session, and removes the message there if found.The same review also raised message ordering: a message sent after a reset can cold-start a new session and dispatch immediately, ahead of an earlier, still-orphaned rescued message — since the caller side of
get_or_create()(~8 call sites acrossslack/handler.py,transport_dispatch.py,gateway.py) always dispatches its own message immediately on a cold start and none of them check for or drain an adopted queue first. This is real and I'm disclosing it rather than fixing it here: closing it fully means changing the dispatch contract at everyget_or_create()call site, which is a materially larger, riskier change than this PR's scope (a session.py-internal rescue mechanism). I chose to fix the cancellation-bypass (a correctness bug — an explicitly cancelled message running anyway) and disclose the ordering nuance (rescued messages are never lost or silently dropped, only possibly reordered relative to a message sent after the reset) rather than reverting the whole mechanism, since the original bug (#3752) is total, silent message loss — strictly worse than out-of-order-but-present delivery. Documented inline at the_orphaned_queues[key] = session.queueassignment inreset(), and in the CHANGELOG entry.Fourth follow-up fix from review,
[BLOCK-MERGE]: GPT 5.6 Review found the mirror-image bug of the third fix — this time on the two deliberate, permanent deletion paths,destroy()anddiscard_conversation(). Both already drop a live session's own.queueby construction (they pop and discard the whole_Session, and neither has ever rescued its queue — that's intentional, matchingstop_turn's explicit hard-killclear_queue()). But neither one dropped an already-orphaned queue: ifreset()had rescued a queue into_orphaned_queuesbeforedestroy()/discard_conversation()ran, there was no live session left for either to find (self._sessions.pop(key, None)returnsNone), so the rescued message sat untouched in_orphaned_queuesand survived a caller-intended permanent deletion — the next message to open a session for that key would still adopt and execute it. Verified by reading both methods directly: neither references_orphaned_queuesat all. Fixed by addingself._orphaned_queues.pop(key, None)to both, inside the sameasync with self._lock:block as the session pop, so there's no window between the two.Fifth follow-up fix from review,
[BLOCK-MERGE]: a structural bug underlying all four fixes above. Every one of them keyed_orphaned_queuesoff_fold_key(key)._fold_key()resolves Slack's two historical key forms — barethread_tsand theslack:-namespaced canonical form — onto whichever one currently has a live session (see its docstring: exact match, then canonical, then bare, in that order, checked againstself._sessions). During the orphaned window there is no live session at all for_fold_keyto check against, so it falls through its last branch and returns the caller's own form completely unchanged. Concretely: a session lives under the bare form,reset()rescues its queue under whatever_fold_keyresolved (the bare form, since that's what's live) — but a latercancel_queued()call for the same conversation using the canonical form has nothing live to fold against, so it also gets the canonical form back unchanged, and looks up a different dict key than the one the queue is actually stored under. The cancel silently misses, and the "cancelled" message later executes when a message using the canonical form adopts the queue. Verified directly: reproduced this exact bare-vs-canonical mismatch (cancel_queuedreturningFalsewhere it should returnTrue) against pre-fix code. Fixed with a dedicated_orphan_key()helper —canonical_key(key), not_fold_key(key)— used at all six_orphaned_queuesaccess sites, so the dict is now always keyed by the canonical form regardless of what happens to be live at access time.Rebase note: picking up a fresh
upstream/mainproduced a real conflict insession.pyagainst a just-merged, unrelated PR (#3768/#3776) that teachescancel_queued()/clear_queue()/dequeue()to unlink a discarded queue entry's temp image files, so they don't leak on disk. The code auto-merged cleanly (both changes touch the same functions but different lines); only two docstring paragraphs textually conflicted, resolved by keeping both. But that fix necessarily predates_orphaned_queuesand only covers a live session's own queue — rebasing onto it surfaced the gap directly: this PR's own orphaned-queue branches incancel_queued()/clear_queue(), and the orphan drop indestroy()/discard_conversation(), all discard entries the same way without unlinking their temp files. Since only this PR's own mechanism was affected, and I was already resolving the conflict in this exact code, I fixed all four to unlink too rather than leave a known regression for a follow-up. Verified each by temporarily reverting just that unlink call and confirming the corresponding test fails with the temp file still on disk.Sixth follow-up fix from review,
[BLOCK-MERGE]: the most consequential finding of this whole review cycle. The disclosed ordering nuance in the fifth fix understated the real gap: rescued messages weren't merely reorderable relative to a later inbound message — they were never automatically redispatched at all unless one happened to arrive. The normal end-of-turn drain,_on_transport_done/_on_doneinslack/events.py, callssessions.dequeue(session_key)immediately after a turn's task completes (success or failure) to pick up and redispatch whatever's queued next.reset()already pops the session by the time that callback runs, sodequeue()— which only ever checked a live session — found nothing and silently did nothing, exactly as it's supposed to when there's genuinely nothing queued. A rescued message then sat in_orphaned_queuesindefinitely, with delivery entirely contingent on some unrelated later message for the same conversation arriving to cold-start a new session — not a "delay," a real possibility of never being delivered. Verified by reading_on_transport_donedirectly: it callsdequeue(), falls back to a separate orchestrator-level_pending_queue(unrelated to_orphaned_queues), and does nothing further if both are empty.Fixed by making
dequeue()itself orphan-aware: when there's no live session, it now pops and returns the front of_orphaned_queuesdirectly (no cancelled-set skip logic needed there, sincecancel_queued()'s orphan branch removes an entry outright rather than marking it cancelled). This requires no changes toslack/events.pyor any other caller — the existing drain callback's unmodifieddequeue()call now does the right thing on its own. As a side effect this also substantially narrows the disclosed ordering gap from the fifth fix: since the drain callback fires immediately on task completion, a rescued message is now typically redispatched before a new inbound message could plausibly arrive to race ahead of it. The gap isn't fully eliminated (a message that wins that narrow race still cold-starts viaget_or_create()ahead of the queue, as before), but it's now the exception rather being the default outcome. Verified directly: reproduced the drain doing nothing against pre-fix code, withdequeue()returningNonefor a rescued message immediately afterreset(), exactly matching the review's stated symptom.Tests
New
test/test_session.py::TestResetPreservesQueuedMessages:test_queued_messages_survive_a_reset_and_reach_the_next_session— reproduces the exact bug sequence (turn 1 in flight, messages 2/3 queued,reset()fires, nextget_or_createfor the key) and asserts both queued messages are still dequeue-able in order.test_reset_with_no_queued_messages_leaves_the_next_session_empty— no false-positive rescue when the queue was empty.test_destroy_does_not_rescue_the_queue/test_discard_conversation_does_not_rescue_the_queue— confirms the two intentional-wipe paths are untouched.test_clear_queue_also_drops_an_already_rescued_queue— reproduces the first review-caught gap and is verified failing pre-fix.test_stop_turn_clears_an_already_rescued_queue_even_with_no_live_session— reproduces the second, deeper review-caught gap by callingstop_turn()itself (the real/stopentry point) rather thanclear_queue()directly; verified failing against pre-fixstop_turn()viagit stashon just that hunk, with the rescued message surviving and reappearing indequeue()after the next cold start.test_cancel_queued_removes_an_already_rescued_message— reproduces the third review-caught gap: cancels a message sitting in_orphaned_queues(no live session), then asserts it is genuinely gone once a later message adopts the rescued queue, rather than still executing. Verified failing against pre-fixcancel_queued()withassert False is Trueon the cancel call itself.test_cancel_queued_with_no_orphaned_queue_and_no_session— no false-positive success when there's nothing to cancel at all.test_destroy_also_drops_an_already_rescued_queue/test_discard_conversation_also_drops_an_already_rescued_queue— reproduce the fourth review-caught gap; both verified failing against pre-fix code with the destroyed/discarded message actually reappearing indequeue()after the next cold start.test_orphaned_queue_access_is_stable_across_bare_and_canonical_slack_keys— reproduces the fifth review-caught gap end to end: a session lives under a barethread_ts, gets reset, is cancelled by a caller using the canonicalslack:-namespaced form for the same conversation, and a later cold start under the canonical form must not resurrect the cancelled message. Verified failing against pre-fix code withcancel_queuedreturningFalse(the cancel silently missed) and the message reappearing indequeue().test_cancel_queued_unlinks_temp_files_from_an_orphaned_queue/test_clear_queue_unlinks_temp_files_from_an_orphaned_queue(intest/test_message_queue.py) andtest_destroy_unlinks_temp_files_from_an_already_rescued_queue/test_discard_conversation_unlinks_temp_files_from_an_already_rescued_queue(intest/test_session.py) — cover the rebase-surfaced temp-file-leak gap. Each verified failing by temporarily reverting just its own unlink call and confirming the temp file survives on disk.test_dequeue_drains_an_already_rescued_queue_with_no_live_session— reproduces the sixth review-caught gap: callsdequeue()directly afterreset(), exactly as_on_transport_done's drain callback does, and asserts it returns the rescued messages in order rather thanNone. Verified failing against pre-fix code withassert None == ('ts2', 'second', {}).test_dequeue_with_no_orphaned_queue_and_no_session_returns_none— no false-positive success when there's nothing to drain.Also re-ran the broader Slack event-handling suites (
test_message_queue.py+test_slack_events_coverage.py+test_slack_gateway.py) after this fix — 432 passed — since it changesdequeue()'s contract and those suites exercise_on_transport_done/_on_done's drain callback most directly.Verified the primary test fails against pre-fix code (
git stash) withassert None == ('ts2', 'second', {})— the exact reported symptom.test/test_session.py+test/test_message_queue.py: 307 passed (includes upstream's own new temp-file-unlink tests pulled in by the rebase).test/test_session.py+test/test_messaging_link.py+test/test_message_queue.py+test/test_session_map_conv_state.py+test/test_session_map_mirror.py(the key-resolution-adjacent suites): 436 passed. Earlier broader sweep —test/test_session*.py+ Slack handler + dashboard session-reset/clear tests: 2165 passed (unaffected by these hunks, which are additive).flake8/isortclean on touched files.mypyclean except the 3 pre-existing, unrelatedhooks.pyLinux-xattr errors present on unmodifiedmain.flake8/isort/mypyclean on touched files (mypy's 3 pre-existinghooks.pyLinux-xattr errors are unrelated and present on unmodifiedmain).Manual verification
N/A — unit coverage sufficient. This is an internal queue-lifecycle invariant with no I/O or UI surface; the tests drive the real
SessionManagercode paths (reset,get_or_create,clear_queue,dequeue) exactly as production callers do.Screenshots / video
N/A — no UI change.
Related Issues
Fixes #3752
Checklist