Repository navigation
gh-161792: Fix crashes in the callable iterator - #161793
Merged
Merged
Conversation
calliter_iternext() used borrowed references to it_callable and it_sentinel, which could be freed by __setstate__() during the call or comparison (pythongh-158031), or by another thread in the free-threaded build. Take strong references when reading the fields, and update them with an atomic exchange plus a delayed DECREF.
nascheme
marked this pull request as ready for review
November 11, 2026 05:16
ViviBeri
reviewed
Nov 13, 2026
ViviBeri
left a comment
There was a problem hiding this comment.
The new tests cover concurrent exhaustion and re-entrant __setstate__() from the sentinel comparison, but they do not exercise the other concurrency case called out in the PR description: __setstate__() replacing it_sentinel / it_stop_exc while another thread is inside calliter_iternext() reading or comparing them.
Could you add a free-threaded regression test that synchronizes a next(it) call with a concurrent it.__setstate__(...) while the callable or sentinel comparison is paused? That would directly exercise the atomic load/exchange plus delayed-DECREF path for these two fields, rather than relying on the re-entrant test (which does not establish concurrent access).
kumaraditya303
approved these changes
Nov 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
calliter_iternext()calledit_callableand compared the result againstit_sentinelthrough borrowed pointers.__eq__can call__setstate__()and free the sentinel while the comparison is using it (uaf of it_sentinel in callable_iterator.__next__ #158031).__setstate__(), and two threads exhausting the iterator can both DECREF the callable (Crash when a callable iterator is shared between threads #161792).Sharing a callable iterator between threads, or calling
__setstate__()on one that is in use, is not supported: the values produced are unspecified. It must not crash, though. That lets the fix avoid a lock:__setstate__()) swap the field with an atomic exchange, so only one thread gets the old value, and release it with_PyObject_XDecRefDelayed(). In the free-threaded build that defers the DECREF until no thread can still be between its load and its incref; in the default build it is a plainPy_XDECREF().This also covers gh-158031, so it overlaps with #158046, which takes a strong reference to the sentinel only.
Performance
Release builds (Clang 20, no PGO/LTO), single thread pinned to one CPU, ns per item including consumption overhead (
deque(islice(it, N), maxlen=0)), best of 7, mean of two runs.Free-threaded build:
iter(int, 1)1object()sentinel__next__ofitertools.repeat, sentinel1Default (GIL) build:
iter(int, 1)1object()sentinel__next__ofitertools.repeat, sentinel1The cost is about 1 ns per item in the free-threaded build and 0-3 ns in the default build, from the strong references to the callable and sentinel. It is only visible when the callable is trivially cheap.