Skip to content

fix(chat): restore the turn deadline by value, not by ContextVar token - #2848

Closed
leonlaiyc wants to merge 1 commit into
kirodotdev:mainfrom
leonlaiyc:fix/turn-deadline-context-reset
Closed

fix(chat): restore the turn deadline by value, not by ContextVar token#2848
leonlaiyc wants to merge 1 commit into
kirodotdev:mainfrom
leonlaiyc:fix/turn-deadline-context-reset

Conversation

@leonlaiyc

@leonlaiyc leonlaiyc commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Fixes #2837.

Problem / Motivation

_bounded_turn publishes the turn's deadline on a ContextVar and restores it with a Token:

deadline_token = _TURN_DEADLINE.set(loop.time() + timeout_secs)
...
finally:
    _TURN_DEADLINE.reset(deadline_token)

That assumes the finally runs in the context that entered the try. For an abandoned wrapper it does not. A gateway shutdown that closes the loop with a turn still pending, or a coroutine that is never awaited, leaves the coroutine object to the garbage collector, which throws GeneratorExit in from whatever context happens to be current when it runs. Token.reset refuses that:

Exception ignored in: <coroutine object _bounded_turn at 0x...>
  File "...\turn_dispatch.py", line 325, in _bounded_turn
    _TURN_DEADLINE.reset(deadline_token)
ValueError: <Token var=<ContextVar name='kirocrew_turn_deadline' ...>> was created in a different Context

Because it raises inside __del__ the failure is unraisable — no caller can catch it, it is only printed. Under pytest each one becomes a PytestUnraisableExceptionWarning, and enough of them take the xdist worker down with them, which is how this surfaces as Backend Tests (Windows) shard 1 failing on unrelated PRs.

Why it matters

The unraisable exception is thrown from a GC finalizer, so it cannot be caught by
the code that caused it. Enough of them crash the xdist worker outright, which is
what makes this a production defect rather than noise: any abandoned bounded turn
leaks a broken ContextVar token, and the failure surfaces far away from its
cause with no stack pointing back.

What changed

Save and restore the value instead of holding a token:

previous_deadline = _TURN_DEADLINE.get()
_TURN_DEADLINE.set(loop.time() + timeout_secs)
...
finally:
    _TURN_DEADLINE.set(previous_deadline)

ContextVar.set has no context affinity, so this is correct in both directions: it restores properly on the normal path, and a restore that lands in a foreign context during finalization writes into a context that is about to be discarded — precisely the no-op wanted.

Saving the previous value rather than clearing matters: a bare set(None) would pass a leak test while silently blanking an enclosing turn's budget. There is a test for exactly that.

Why this is a production bug, not only a test-infrastructure one

The unraisable ValueError is raised on every abandoned turn wrapper — a shutdown with turns in flight is the ordinary case. It is invisible in production only because nothing is watching sys.unraisablehook there; pytest is what makes it loud. The deadline itself is also left wrong in the finalizing context, which is the thing _turn_budget_remaining reads to size in-turn waits.

Reproduction

test/test_turn_deadline_contextvar.py makes the GC shape deterministic: step the wrapper past the publish, then close it from a different contextvars.Context — which is what a collector running on an unrelated stack does.

1 failed / 6 passed on 24a6f8ee5 → 7 passed. The single fail-before is the exact CI ValueError; the other six are the preservation contract and already pass on main:

  • the deadline is published to code running inside the turn;
  • a completed turn does not leak its deadline into the caller's context;
  • a nested wrapper restores the outer turn's deadline rather than blanking it;
  • the ceiling still fires;
  • an abandoned dispatched turn (the spawn_guarded_turn shape, its own task context) leaves the dispatching context clean;
  • closing in the owning context still works.

Tests

Windows 11 26200 / py3.10.6.

  • test_turn_deadline_contextvar.py — 1 failed / 6 passed before, 7 passed after.
  • With test_turn_dispatch.py, test_turn_teardown_release.py, test_dashboard_approval.py, test_dashboard_approval_window.py, test_dashboard_chat.py, run on the exact base and on this branch with the identical selection and compared by failing-node set: base 4 failed / 663 passed; branch 3 failed / 664 passed. The one node that disappears is this PR's fail-before; the remaining three are test_dashboard_approval.py failures present on base under the same selection and unrelated to this change. FIX_ONLY = ∅.
  • flake8, isort, git diff --check clean; mypy reports nothing in turn_dispatch.py. black wants to reformat one pre-existing line in this file that the change does not touch, so it is left alone.

Note on the CI symptom

I triaged this from three job logs before writing any code: the base commit 24a6f8ee5's own push CI fails Windows shard 1 on this same single node (worker 'gw0' crashed), as do PRs #2594 and #2842 (gw1). The crash is not caused by any of them, which is the point of #2837 — and matches the report of it hitting #2823 and #2735 too.

Fixing the unraisable removes the crash source. I have not tried to make test_interactive_reject itself more robust; if it still proves unstable for an unrelated reason, that is a separate change.

`_bounded_turn` publishes the turn's deadline on a ContextVar and restores
it with a Token, which assumes the finally runs in the context that
entered the try. An ABANDONED wrapper breaks that: a shutdown that closes
the loop with a turn still pending, or a coroutine never awaited, leaves
the object to the garbage collector, which throws GeneratorExit in from
whatever context is current when it runs.

`Token.reset` refuses that with "was created in a different Context", and
because it raises inside __del__ the failure is unraisable — no caller can
catch it, it is only printed. Under pytest each becomes a
PytestUnraisableExceptionWarning and enough of them crash the xdist
worker, which is how it surfaces as an unrelated PR's Windows shard
failing.

Save and restore the value instead. `ContextVar.set` has no context
affinity, so it restores correctly on the normal path and a restore that
lands in a foreign context during finalization writes into a context that
is about to be discarded. Saving the PREVIOUS value rather than clearing
is load-bearing: a bare set(None) would pass a leak test while silently
blanking an enclosing turn's remaining budget.
@leonlaiyc
leonlaiyc requested a review from a team as a code owner August 11, 2026 15:29
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Aug 11, 2026
@iamwhatever

Copy link
Copy Markdown
Collaborator

👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated.

Missing sections:

  • ## Tests

Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle.

@bolichen97

Copy link
Copy Markdown
Collaborator

Hi @leonlaiyc, a maintainer nudge on this one: it is carrying the readiness: action required label and has had no activity for about 4 days, so it is not moving toward merge.

Current state:

  • Base: 1091 commits behind main
  • Merge state: mergeable, but the readiness gate is still red
  • Red signals:
    • Backend Tests (Windows) (1)
    • Publish readiness signal
    • PR Readiness

Could you rebase onto the latest main and push a fix?

git fetch upstream           # or: git fetch origin, if this branch lives here
git rebase upstream/main
# resolve any conflicts, run the local gates below, then update the branch with
# a force-with-lease so the rebase lands without clobbering anyone else's work

The local gates to run before updating the branch:

black src/kiro_crew test && isort src/kiro_crew test
flake8 src/kiro_crew test && mypy src/kiro_crew
python -m pytest
cd website && npm run build && npm run test

A good part of this branch's redness is likely stale rather than a real defect: it predates a lot of what is now on main, and several of these gates (the Coverage Gate, the Windows shards, the AI review lanes) have changed since the last run here. A rebase alone often clears them. If something still fails afterwards and you believe it is a false positive, say so in a comment and we will take a look. If the change is no longer needed, feel free to close the PR.

@bolichen97
bolichen97 enabled auto-merge (squash) August 24, 2026 07:01
@iamwhatever iamwhatever added the needs-pr-triage PR scanner: awaiting automated triage label Aug 26, 2026
@bolichen97 bolichen97 added drive-to-green PR claimed by drive-to-green pipeline and removed needs-pr-triage PR scanner: awaiting automated triage labels Aug 26, 2026
@bolichen97

Copy link
Copy Markdown
Collaborator

🤖 Kiro Crew [operator: bolichen97#bb3ad1ca]: This PR has been inactive for 7+ days with failing CI. I've assessed the blockers and they appear resolvable — I'll push fixes directly to this branch as a co-author.

Assessment: The red is test/test_dashboard_approval_window.py::TestArmTimeBudget::test_bounded_turn_publishes_then_clears_the_deadline on the Windows shard — after the bounded turn ends the deadline is not cleared (assert 10472.703 is None), i.e. the restore-by-value path leaves the published deadline set instead of clearing it. That is a concrete defect inside this PR's own mechanism with clear intent; I'll fix the clear-on-restore path, rebase onto current main (base is 9+ days old), and drive CI to green.

If you'd prefer I don't touch this PR, add the pr-no-autofix label.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Aug 27, 2026
@bolichen97

Copy link
Copy Markdown
Collaborator

Full-diff audit result: the production hunk here saves _TURN_DEADLINE.get() and restores that value with _TURN_DEADLINE.set(previous_deadline) instead of resetting a token across context boundaries. Merged #4274 (commit 7bd47050ab301da28ca4b8153dcc424ac18ad06b) contains the same production fix; this branch only adds alternate tests/commentary around behavior already enforced on main. There is no independent production residual, so I am closing this PR as fully superseded.

@bolichen97 bolichen97 closed this Aug 29, 2026
auto-merge was automatically disabled August 29, 2026 18:10

Pull request was closed

@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Aug 29, 2026
@iamwhatever iamwhatever removed the drive-to-green PR claimed by drive-to-green pipeline label Aug 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Backend Tests (Windows) shard 1: test_interactive_reject crashes its xdist worker on a ContextVar token reset

3 participants