Skip to content

Move HTTP/2 stream events cleanup inside _state_lock#1013

Merged
Kludex merged 1 commit into
pydantic:mainfrom
mbeijen:fix/h2-stream-events-race
Jun 2, 2026
Merged

Move HTTP/2 stream events cleanup inside _state_lock#1013
Kludex merged 1 commit into
pydantic:mainfrom
mbeijen:fix/h2-stream-events-race

Conversation

@mbeijen
Copy link
Copy Markdown
Contributor

@mbeijen mbeijen commented Jun 2, 2026

Summary

In AsyncHTTP2Connection._response_closed, the del self._events[stream_id] happens outside the _state_lock, but the not self._events check that decides whether to transition the connection to IDLE happens inside the lock. With multiplexed HTTP/2 requests this creates a race:

  1. Request A is the last active stream. _response_closed runs: releases the semaphore, deletes its events entry, then yields before acquiring the lock.
  2. Request B enters handle_async_request, acquires _state_lock, sets state=ACTIVE, releases the lock.
  3. Request B acquires the semaphore and a new stream ID — but has not yet written self._events[stream_id] = [].
  4. Request A resumes, acquires _state_lock, observes not self._events is True (B's entry isn't there yet), and marks the connection IDLE with an expiry timer — while Request B is actively using it.

The pool may then see the connection as expired and close it mid-request, or evict it as an excess idle connection.

Moving del self._events[stream_id] inside _state_lock makes the events-dict mutation and the IDLE-transition check atomic.

Ports encode/httpcore#1062 (bysiber, currently open upstream), via codeberg.org/httpxyz/httpcorexyz@e88f30b. The fork bundled this with encode/httpcore#1061; I'm splitting them into two httpx2 PRs to mirror the upstream split. Sibling PR for #1061 is #1012.

Notes

  • Only _async/http2.py was edited by hand; _sync/http2.py was regenerated by scripts/unasync.py.
  • No new test: reproducing this race deterministically needs careful coroutine scheduling and isn't currently feasible against the public AsyncConnectionPool API. The upstream PR doesn't ship a test either. 100% coverage preserved; existing HTTP/2 tests continue to pass.

Note: this change was prepared with AI assistance (Claude Code).

`_response_closed` deletes `self._events[stream_id]` *outside* the
`_state_lock`, but the `not self._events` check that decides whether
to transition the connection to IDLE happens *inside* it. With
multiplexed HTTP/2 requests this creates a race:

1. Request A is the last active stream; `_response_closed` releases
   the semaphore, deletes its events entry, then yields before
   acquiring `_state_lock`.
2. Request B enters `handle_async_request`, acquires `_state_lock`,
   sets `state=ACTIVE`, releases the lock.
3. Request B acquires the semaphore and stream ID but has not yet
   added `self._events[stream_id] = []`.
4. Request A resumes, acquires `_state_lock`, sees `not self._events`
   is True, and marks the connection IDLE with an expiry — while
   Request B is actively using it.

Moving `del self._events[stream_id]` inside the lock makes the events
mutation and the IDLE-transition check atomic.

Ports encode/httpcore#1062 (bysiber), via
codeberg.org/httpxyz/httpcorexyz@e88f30b (second of two fixes bundled
in that commit; the first landed as pydantic#1012).

Co-Authored-By: Kadir Can Ozden <101993364+bysiber@users.noreply.github.com>
@codspeed-hq
Copy link
Copy Markdown

codspeed-hq Bot commented Jun 2, 2026

Merging this PR will not alter performance

✅ 15 untouched benchmarks
⏩ 7 skipped benchmarks1


Comparing mbeijen:fix/h2-stream-events-race (62d8310) with main (fe3da36)

Open in CodSpeed

Footnotes

  1. 7 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

mbeijen added a commit to mbeijen/httpx2 that referenced this pull request Jun 2, 2026
Copy link
Copy Markdown

@cubic-dev-ai cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 2 files

Re-trigger cubic

@Kludex Kludex merged commit 2b0daf5 into pydantic:main Jun 2, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants