fix(dashboard): stop _bounded_turn crashing when GC closes it directly - #7643
Conversation
An orphaned spawn_guarded_turn dispatch -- a background task nobody awaited or cancelled through a live Task, e.g. a test that never drained state._background_tasks -- gets reclaimed by the garbage collector, which calls close() on the still-suspended _bounded_turn coroutine directly. That throws GeneratorExit at the `await task` suspension point, resumed synchronously by the collector rather than by a loop callback, so there is no guarantee the owning loop is still running. The cleanup path answered any teardown with `task.cancel(); await task` unconditionally. That is correct for a live CancelledError unwind, but suspending on that second await while already unwinding a GeneratorExit raises "coroutine ignored GeneratorExit" as an unraisable exception nobody can catch, and task.cancel() itself raises "RuntimeError: Event loop is closed" once the owning loop is gone -- both surfaced in CI as PytestUnraisableExceptionWarning noise misattributed to whichever unrelated test happened to trigger the GC sweep. _bounded_turn now catches GeneratorExit explicitly and skips the unsafe re-join in that branch, cancelling best-effort only and guarding task.cancel() against a closed loop.
Design Review (Fable 5, fork) — ✅ PASSDesign-level review of The fix targets a real defect in Design-Verdict: PASS Fixes the wrapper's real defect — awaiting in [DESIGN-REVIEWED] bdcd0f5 |
GPT 5.6 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
First Principles Review (Fable 5, fork) — ✅ PASSPremise-level review of All twelve other First-Principles-Verdict: PASS A real interpreter rule — a coroutine may not suspend while unwinding GeneratorExit — is fixed where it was violated, with nothing riding along. What this change shipsIntent: stop
The fix sits at mechanism level: the misbehaving code is the unconditional [FIRST-PRINCIPLES-REVIEWED] bdcd0f5 |
Opus 4.8 Review (fork) — ✅ no blocking findingsReviewed Review detailsThe candidate is the only item to validate. It targets the residual GeneratorExit path at the Reconstructing the post-diff
Checking (a) a concrete input in practice: No concrete input, no production call path — the candidate fails the Step 1 bar and scores well under 80. Step 2: the added No findings. [OPUS-REVIEWED] bdcd0f5 |
bolichen97
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: fix (2 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: fix with a clear root cause -- stop _bounded_turn crashing when the GC closes it directly (GeneratorExit path skips the join and cancels best-effort on a possibly-dead loop). CodeQL is not applicable on this fork PR (default-setup emits no check-run); SAST coverage is Semgrep only, latest run success with 0 annotations.
iamwhatever
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: fix (2 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: fix -- _bounded_turn raised "coroutine ignored GeneratorExit" when the GC close()-d its coroutine directly instead of a live Task cancelling it, so the wrapper now flags that path and has its finally cancel best-effort (swallowing RuntimeError from an already-closed loop) and skip the await-join that would suspend during a GeneratorExit unwind; one source file plus its test file, pure asyncio teardown with no auth/input-parsing/trust-boundary/sandbox/gate/secret surface touched. CodeQL is not applicable on this fork PR (default-setup emits no check-run); SAST coverage is Semgrep only, latest run success with 0 annotations.
PR kirodotdev#7643 fixed _bounded_turn's half of this GeneratorExit race but flagged a second, separate defect in _run_chat's own queue-drain tail as "tracked separately" -- this is that fix. _run_chat's single finally block drains the queue via _start_next_queued_turn (spawn_guarded_turn -> asyncio.create_task) and falls back to _finish_queue_cycle (four more bare asyncio.create_task calls: synthesis, auto-title, title-refresh, session summary). Both assume a running loop. When a turn's coroutine is orphaned as a background task and never joined -- a test/harness bug, since spawn_guarded_turn always registers its task and finish_turn_task always retrieves it in production -- the GC eventually finalizes it via close(), throwing GeneratorExit at whatever await it was suspended on. That unwinds cleanly into the finally block until it hits one of these create_task calls with no loop to schedule onto, which raises RuntimeError and replaces the in-flight GeneratorExit -- surfacing as an unraisable exception blamed on whatever test the GC happened to fire in. Add _loop_is_running() and use it to make _start_next_queued_turn bail out before touching the queue when off-loop, and fold the same check into _finish_queue_cycle's will_synthesize and its other three create_task sites. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Open PR relationship auditThis is a consolidated, point-in-time code-level audit note. It compares complete merge-base diffs and current/merged code; it does not treat a shared topic as duplication or partial coverage as completion. Relationship findings
No PR, Issue, label, branch, or review state was changed by the relationship-note portion of this audit. |
PR kirodotdev#7643 fixed _bounded_turn's half of this GeneratorExit race but flagged a second, separate defect in _run_chat's own queue-drain tail as "tracked separately" -- this is that fix. _run_chat's single finally block drains the queue via _start_next_queued_turn (spawn_guarded_turn -> asyncio.create_task) and falls back to _finish_queue_cycle (four more bare asyncio.create_task calls: synthesis, auto-title, title-refresh, session summary). Both assume a running loop. When a turn's coroutine is orphaned as a background task and never joined -- a test/harness bug, since spawn_guarded_turn always registers its task and finish_turn_task always retrieves it in production -- the GC eventually finalizes it via close(), throwing GeneratorExit at whatever await it was suspended on. That unwinds cleanly into the finally block until it hits one of these create_task calls with no loop to schedule onto, which raises RuntimeError and replaces the in-flight GeneratorExit -- surfacing as an unraisable exception blamed on whatever test the GC happened to fire in. Add _loop_is_running() and use it to make _start_next_queued_turn bail out before touching the queue when off-loop, and fold the same check into _finish_queue_cycle's will_synthesize and its other three create_task sites. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
PR kirodotdev#7643 fixed _bounded_turn's half of this GeneratorExit race but flagged a second, separate defect in _run_chat's own queue-drain tail as "tracked separately" -- this is that fix. _run_chat's single finally block drains the queue via _start_next_queued_turn (spawn_guarded_turn -> asyncio.create_task) and falls back to _finish_queue_cycle (four more bare asyncio.create_task calls: synthesis, auto-title, title-refresh, session summary). Both assume a running loop. When a turn's coroutine is orphaned as a background task and never joined -- a test/harness bug, since spawn_guarded_turn always registers its task and finish_turn_task always retrieves it in production -- the GC eventually finalizes it via close(), throwing GeneratorExit at whatever await it was suspended on. That unwinds cleanly into the finally block until it hits one of these create_task calls with no loop to schedule onto, which raises RuntimeError and replaces the in-flight GeneratorExit -- surfacing as an unraisable exception blamed on whatever test the GC happened to fire in. Add _loop_is_running() and use it to make _start_next_queued_turn bail out before touching the queue when off-loop, and fold the same check into _finish_queue_cycle's will_synthesize and its other three create_task sites. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Summary
PR #5411's CI job "Backend Tests (3.10, 2)" failed on ~264 lines of
PytestUnraisableExceptionWarningnoise, all attributed totest_signed_out_cli_holds_queuedespite that test passing and being unrelated to the change under review. That attribution is misleading:test_signed_out_cli_holds_queueis innocent (verified: zero warnings in isolation) — this is cross-test contamination, an already-documented failure class in this repo (docs/system-specs/common/testing-conventions.md§ "3. Leaked async objects").Root cause:
_bounded_turn(dashboard/turn_dispatch.py) wraps a chat turn under a wall-clock ceiling. When aspawn_guarded_turndispatch is never awaited/cancelled through a live Task — e.g. a test that drives_run_chatand doesn't drainstate._background_tasks— its coroutine is eventually reclaimed by the garbage collector, which callsclose()on it directly. That throwsGeneratorExitat whateverawaitit's suspended on, resumed synchronously by the collector rather than by a loop callback, so there's no guarantee the owning event loop is even still running.The cleanup path (added in #6926) answered every teardown unconditionally with
task.cancel(); await task. That's correct for a liveCancelledErrorunwind, but:awaitwhile already unwinding aGeneratorExitraisesRuntimeError: coroutine ignored GeneratorExit— an unraisable exception nobody can catch.task.cancel()itself raisesRuntimeError: Event loop is closedonce the owning loop is gone, unguarded.Both surface as
PytestUnraisableExceptionWarningat whatever later test happens to trigger the GC sweep — which is exactly what CI observed.Fix
_bounded_turnnow catchesGeneratorExitexplicitly and, in that branch, skips the unsafe re-join — cancelling the inner task best-effort only, and guardingtask.cancel()against a closed loop.Test plan
TestBoundedTurnGeneratorExitintest/test_turn_dispatch.py— two tests that deterministically reproduce the crash (real event loop, closed out from under a suspended_bounded_turncoroutine, then.close()'d directly) and fail against the old code / pass against the fix.test_dashboard_chat.pyunder forced GC +-W error::pytest.PytestUnraisableExceptionWarning: went from 23 unraisable-exception errors to 2 (the 2 remaining are a separate, pre-existing defect in_run_chat's own queue-drain tail, unrelated to this leak — tracked separately).test_dashboard_chat.py+test_turn_dispatch.pysuite passes clean (743 tests, no regressions).Pattern harvest
Not generalizable: the vulnerable shape is
task.cancel(); await taskunconditionally inside afinallyof a coroutine that is sometimes dispatched fire-and-forget (spawn_guarded_turn) rather than always driven to completion by an owning Task — that combination is what makes a directclose()/GeneratorExitteardown reachable at all. A grep acrosssrc/kiro_crewfor the samecancel(); awaitshape inside afinallyturns up ~12 other sites, and every one is an ordinary awaitedclose()/teardown method that is always driven by a live Task, never orphaned — so none share the precondition this bug needed. A semgrep rule for "await insidefinally" would false-positive on all of those legitimate sites without also encoding "and this coroutine may be dispatched without a Task ever guaranteed to join it," which isn't something the pattern-matching layer can see.🤖 Generated with Claude Code