fix(chat): restore the turn deadline by value, not by ContextVar token - #2848
fix(chat): restore the turn deadline by value, not by ContextVar token#2848leonlaiyc wants to merge 1 commit into
Conversation
`_bounded_turn` publishes the turn's deadline on a ContextVar and restores it with a Token, which assumes the finally runs in the context that entered the try. An ABANDONED wrapper breaks that: a shutdown that closes the loop with a turn still pending, or a coroutine never awaited, leaves the object to the garbage collector, which throws GeneratorExit in from whatever context is current when it runs. `Token.reset` refuses that with "was created in a different Context", and because it raises inside __del__ the failure is unraisable — no caller can catch it, it is only printed. Under pytest each becomes a PytestUnraisableExceptionWarning and enough of them crash the xdist worker, which is how it surfaces as an unrelated PR's Windows shard failing. Save and restore the value instead. `ContextVar.set` has no context affinity, so it restores correctly on the normal path and a restore that lands in a foreign context during finalization writes into a context that is about to be discarded. Saving the PREVIOUS value rather than clearing is load-bearing: a bare set(None) would pass a leak test while silently blanking an enclosing turn's remaining budget.
|
👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated. Missing sections:
Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle. |
|
Hi @leonlaiyc, a maintainer nudge on this one: it is carrying the Current state:
Could you rebase onto the latest git fetch upstream # or: git fetch origin, if this branch lives here
git rebase upstream/main
# resolve any conflicts, run the local gates below, then update the branch with
# a force-with-lease so the rebase lands without clobbering anyone else's workThe local gates to run before updating the branch: black src/kiro_crew test && isort src/kiro_crew test
flake8 src/kiro_crew test && mypy src/kiro_crew
python -m pytest
cd website && npm run build && npm run testA good part of this branch's redness is likely stale rather than a real defect: it predates a lot of what is now on |
|
🤖 Kiro Crew [operator: bolichen97#bb3ad1ca]: This PR has been inactive for 7+ days with failing CI. I've assessed the blockers and they appear resolvable — I'll push fixes directly to this branch as a co-author. Assessment: The red is If you'd prefer I don't touch this PR, add the |
|
Full-diff audit result: the production hunk here saves |
Pull request was closed
Fixes #2837.
Problem / Motivation
_bounded_turnpublishes the turn's deadline on aContextVarand restores it with aToken:That assumes the
finallyruns in the context that entered thetry. For an abandoned wrapper it does not. A gateway shutdown that closes the loop with a turn still pending, or a coroutine that is never awaited, leaves the coroutine object to the garbage collector, which throwsGeneratorExitin from whatever context happens to be current when it runs.Token.resetrefuses that:Because it raises inside
__del__the failure is unraisable — no caller can catch it, it is only printed. Under pytest each one becomes aPytestUnraisableExceptionWarning, and enough of them take the xdist worker down with them, which is how this surfaces asBackend Tests (Windows) shard 1failing on unrelated PRs.Why it matters
The unraisable exception is thrown from a GC finalizer, so it cannot be caught by
the code that caused it. Enough of them crash the xdist worker outright, which is
what makes this a production defect rather than noise: any abandoned bounded turn
leaks a broken
ContextVartoken, and the failure surfaces far away from itscause with no stack pointing back.
What changed
Save and restore the value instead of holding a token:
ContextVar.sethas no context affinity, so this is correct in both directions: it restores properly on the normal path, and a restore that lands in a foreign context during finalization writes into a context that is about to be discarded — precisely the no-op wanted.Saving the previous value rather than clearing matters: a bare
set(None)would pass a leak test while silently blanking an enclosing turn's budget. There is a test for exactly that.Why this is a production bug, not only a test-infrastructure one
The unraisable
ValueErroris raised on every abandoned turn wrapper — a shutdown with turns in flight is the ordinary case. It is invisible in production only because nothing is watchingsys.unraisablehookthere; pytest is what makes it loud. The deadline itself is also left wrong in the finalizing context, which is the thing_turn_budget_remainingreads to size in-turn waits.Reproduction
test/test_turn_deadline_contextvar.pymakes the GC shape deterministic: step the wrapper past the publish, then close it from a differentcontextvars.Context— which is what a collector running on an unrelated stack does.1 failed / 6 passed on
24a6f8ee5→ 7 passed. The single fail-before is the exact CIValueError; the other six are the preservation contract and already pass on main:spawn_guarded_turnshape, its own task context) leaves the dispatching context clean;Tests
Windows 11 26200 / py3.10.6.
test_turn_deadline_contextvar.py— 1 failed / 6 passed before, 7 passed after.test_turn_dispatch.py,test_turn_teardown_release.py,test_dashboard_approval.py,test_dashboard_approval_window.py,test_dashboard_chat.py, run on the exact base and on this branch with the identical selection and compared by failing-node set: base 4 failed / 663 passed; branch 3 failed / 664 passed. The one node that disappears is this PR's fail-before; the remaining three aretest_dashboard_approval.pyfailures present on base under the same selection and unrelated to this change.FIX_ONLY = ∅.flake8,isort,git diff --checkclean;mypyreports nothing inturn_dispatch.py.blackwants to reformat one pre-existing line in this file that the change does not touch, so it is left alone.Note on the CI symptom
I triaged this from three job logs before writing any code: the base commit
24a6f8ee5's own push CI fails Windows shard 1 on this same single node (worker 'gw0' crashed), as do PRs #2594 and #2842 (gw1). The crash is not caused by any of them, which is the point of #2837 — and matches the report of it hitting #2823 and #2735 too.Fixing the unraisable removes the crash source. I have not tried to make
test_interactive_rejectitself more robust; if it still proves unstable for an unrelated reason, that is a separate change.