Repository navigation
Description
Bug report
Bug description:
Extensions compiled against 3.11 or older headers implement Py_DECREF as an inline --ob_refcnt. That includes abi3 wheels built on 3.11, which are common. Since 3.12 the interpreter hands out references to immortal objects without incrementing their refcount, so each such reference the extension releases is a net decrement. 3.14 starts immortal objects at 3 << 30 and treats (int32_t)ob_refcnt < 0 as immortal, which leaves a margin of 2**30 (see gh-125174).
That margin runs out in real long-running processes. Home Assistant on 3.14.6 crashes after hours to days because PyAV's cp311-abi3 wheels drain True by a few counts per av.open() and per FFmpeg log line (PyAV-Org/PyAV#2420, home-assistant/core#178218). A core dump from an affected instance had _Py_TrueStruct.ob_refcnt == 0x7ffb2327, which is 2**30 + 318,681 below 0xC0000000.
On 3.12 and 3.13, losing immortality is fairly benign: the object is then refcounted like any other, and results stay correct. On 3.14 it is a silent correctness failure. The reason is in the default build's stackrefs:
PyStackRef_IsTrue/IsFalse/IsNonecompare the tagged bits exactly._PyStackRef_FromPyObjectNewandPyStackRef_MakeHeapSafedecide whether to tag with_Py_IsImmortal(), which reads the refcount.PyStackRef_FromPyObjectStealdecides withob_flags.
Once True's refcount is below 2**31, a True loaded by a specialized attribute load (LOAD_ATTR_INSTANCE_VALUE and others) or passed through RETURN_VALUE is untagged, and POP_JUMP_IF_TRUE treats it as false. True from constants, comparisons and C calls still tests true, so the failures look random. In Home Assistant, Thread.name asserted Thread.__init__() not called, Condition.notify raised cannot notify on un-acquired lock, traceback printed None as the exception type, and so on, across threads, until the process exited.
Reproducer. ctypes stands in for the extension's unbalanced decrefs:
import ctypes, sys
print(sys.version)
class C:
def __init__(self):
self.flag = True
def check_attr(o):
if not o.flag: # LOAD_ATTR_INSTANCE_VALUE, POP_JUMP_IF_TRUE
return "WRONG: o.flag is True but tested false"
return "ok"
def get_true():
return True
def check_return():
if not get_true(): # CALL_PY_EXACT_ARGS, RETURN_VALUE, POP_JUMP_IF_TRUE
return "WRONG: get_true() returned True but tested false"
return "ok"
o = C()
for _ in range(1000): # warm up so both functions get specialized
check_attr(o), check_return()
refcnt = ctypes.c_uint32.from_address(id(True)) # low 32 bits of ob_refcnt (64-bit little-endian)
print("before: ob_refcnt = %#x" % refcnt.value, check_attr(o), check_return())
refcnt.value = 2**31 - 10**6 # 3.14: 2**30 + 10**6 raw decrements from 0xc0000000
print("after: ob_refcnt = %#x" % refcnt.value, check_attr(o), check_return())
print("o.flag is True:", o.flag is True, "| bool(o.flag):", bool(o.flag))Output with the official Docker images (python:3.x, Linux aarch64, default build):
3.14.7 (main, Sep 19 2026, 03:34:22) [GCC 14.2.0]
before: ob_refcnt = 0xc0000000 ok ok
after: ob_refcnt = 0x7ff0bdc0 WRONG: o.flag is True but tested false WRONG: get_true() returned True but tested false
o.flag is True: True | bool(o.flag): True
3.15.0rc2 (main, Sep 19 2026, 03:32:36) [GCC 14.2.0]
before: ob_refcnt = 0xc0000000 ok ok
after: ob_refcnt = 0x7ff0bdc0 WRONG: o.flag is True but tested false WRONG: get_true() returned True but tested false
o.flag is True: True | bool(o.flag): True
3.13.15 (main, Sep 19 2026, 03:38:46) [GCC 14.2.0]
before: ob_refcnt = 0xffffffff ok ok
after: ob_refcnt = 0x7ff0bdc0 ok ok
o.flag is True: True | bool(o.flag): True
3.12.14 (main, Sep 19 2026, 03:37:20) [GCC 14.2.0]
before: ob_refcnt = 0xffffffff ok ok
after: ob_refcnt = 0x7ff0bdc0 ok ok
o.flag is True: True | bool(o.flag): True
The value has to be far enough below 2**31. At exactly 0x7fffffff, the first mortal-path incref brings True back to 0x80000000, and it is immortal again.
The drift itself is the extension's bug, and PyAV is fixing its build. But 3.14 turns it from a lost optimization into wrong branches, and nothing in the process points at the cause. Some options, in no particular order:
- Tag consistently.
_PyStackRef_FromPyObjectNewandPyStackRef_MakeHeapSafecould useob_flags & _Py_IMMORTAL_FLAGSasPyStackRef_FromPyObjectStealdoes. Alternatively,PyStackRef_IsTrue/IsFalse/IsNonecould ignore the tag bit. Either would fix the truth tests, though not the drift. - Treat statically allocated objects (
_Py_STATICALLY_ALLOCATED_FLAG) as immortal whatever their refcount says. - Periodically reset the refcount of static immortal singletons, for example during GC, as suggested in "Immortal" objects aren't immortal and that breaks things. #125174.
CPython versions tested on:
3.12, 3.13, 3.14, 3.15
Operating systems tested on:
Linux
Activity
picnixz commentedon Sep 29, 2026
should I understand that the problem is extensions not properly managing refcounts or is there something else? I don't understand the issue exactly here. AFAIU:
So that's the extension problem? or is it ours?
clairernovotny commentedon Oct 1, 2026
Both, in different ways.
The drift itself is the extension's bug: PyAV's abi3 wheels were compiled against 3.11 headers, so their inline
Py_DECREFis a raw decrement, applied to references (e.g.Py_TruefromPyObject_RichCompare) that 3.12+ hands out without an incref. PyAV 19.0.0 builds against 3.12 headers and no longer drifts (measured: 0 vs. −12k to −34k per 4,000av.opencalls on 17.x/18.x).The part that may be CPython's is what happens once such drift takes a singleton below the immortality threshold. On 3.12/3.13 that is harmless, and the reproducer above prints
ok ok. On 3.14/3.15, specialized code starts treatingTrueas false (WRONG: o.flag is True but tested false), and in a real process that showed up as "impossible" assertion failures across threads, then a crash. Since the stable ABI still lets extensions built with 3.11 headers load on 3.14, the question is whether 3.14 should tolerate a de-immortalized singleton rather than silently miscompute, for example by not deriving the stackref tag from_Py_IsImmortal, or by checkingob_flagsconsistently.picnixz commentedon Oct 1, 2026
Ah I see, ok now it's clearer. What we never want is true becoming false at some point if extensions correctly manage their references. If this is the case, then this means something is wrong.
The problem is that 3.11 is security-only and this can't really be considered a security issue. The best we can do is possibly detecting on 3.14+ whether we are using old stable ABI or not to determine what to do with the refcount properly.
I will let @markshannon decide whwat to do because I don't have enough insight on whether checking immortality properly could actually slow down things (I belieev it can, but by how much I don't know). In some sense, may I read this issue as "3.14 broke Stable ABI because of immortality assumptions"?
clairernovotny commentedon Oct 1, 2026
Yes, I think that's a fair reading, with one refinement: the extension is managing its references correctly by the rules it was compiled for.
Under 3.11 semantics,
PyObject_RichComparereturns a new reference toPy_Trueand the extensionPy_DECREFs it, which is balanced. On 3.12+, the runtime skips the incref for immortal objects, but the 3.11-header inlinePy_DECREFin the extension still does a raw--ob_refcnt. So code that is correct under the limited API it targets produces a net decrement per call, and in a long-running process that adds up. In the Home Assistant case it crossed the threshold in hours to days, depending on how much FFmpeg logged.PEP 683 anticipated old extensions touching immortal refcounts and relied on the margin being too large to matter. That held on 3.12/3.13 because losing immortality there is benign. On 3.14 it isn't, because the stackref fast paths derive the tag from
_Py_IsImmortal()while other paths use the static flags, soTruecan compare unequal to itself.So I agree the fix belongs in 3.14+, not 3.11. A cheap option might be to make
PyStackRef_IsTrue/IsFalse/IsNone(and the tagging of these static singletons) independent of the current refcount, for example by comparing the untagged pointer or by tagging statically allocated immortals fromob_flags. That avoids an immortality check on the hot incref/decref paths. Whether that is acceptable performance-wise is Mark's call.v0.9.549 Python 3.14 で数日動かすと動かなくなるのを防いだ(None・True・False の参照カウントのずれを戻す)
nthanhcongaasc commentedon Nov 4, 2026
Another affected extension: PyQt6 (6.11.0 and the latest 6.12.0,
cp310-abi3wheels), CPython 3.14.7, Windows x64.Drift source. Every PyQt6 method that returns
Noneadds exactly +1 toNone's raw refcount, and that +1 is never taken back. Methods returning other values add nothing.True/Falsewere unaffected in my measurements.Noneraw refcount per callQPoint.setX(),QObject.setObjectName(),QColor.setRed(),QApplication.processEvents()QWidget.isVisible()(returns bool)Time to failure is hours, not days. The pattern
while w.isVisible(): app.processEvents()is very common in PyQt apps, and every iteration is +1. In a real app this was about 160k/s, soNonelost immortality after roughly 1h50m. In the live process I read_Py_NoneStructaslow32=0x0edd14b7, ob_overflow=1, whileTrue,False,...and()were all still0xC0000000.The
Nonecase is more severe than the truthiness case. OnceNoneis no longer immortal, the following break process-wide and stay broken until restart:__init__:TypeError: __init__() should return None, not 'NoneType'yield fromover a builtin iterator:AttributeError: 'range_iterator' object has no attribute 'send'(seen in pandas'RangeIndex.__iter__)warnings.catch_warnings():TypeError: _py_warnings.simplefilter() argument after * must be an iterable, not NoneTypeAs a result, logging, traceback and pandas all fail, so the application cannot even log the error.
In the broken process I checked the
co_constsof every live function, about 20k of which containNone. All of them point to the same_Py_NoneStruct. It is the same object; only the stackref tagging differs.Minimal repro (no memory writes; takes about 5 minutes):
Because a well-behaved, widely used abi3 extension hits this within hours of normal use, +1 for making
PyStackRef_IsNone/IsTrue/IsFalseindependent of the current refcount, for example by tagging static immortals fromob_flags.picnixz commentedon Nov 4, 2026
cc @encukou @ngoldbaum for more visibility
vstinner commentedon Nov 7, 2026
and:
If you build PyAV and PyQt6 with Python 3.12 or newer, do you still hit the issue?
I changed Py_INCREF/DECREF() implementation to make them an opaque function call in Python 3.12 to fix this exact problem: https://discuss.python.org/t/limited-c-api-implement-py-incref-and-py-decref-as-function-calls/27592. Sadly, it means that you need a different binary for Python 3.12+ than the binary for Python 3.11 and older.
This refcount issue is just a timebomb awaiting to explode :-( Building C extensions with Python 3.12 or newer fix the root issue.
In the past, I added
_Py_IsImmortalLoose()(commit b826e45) to try to reduce the risk of hitting this refcnt/stable ABI issue (issues gh-121528 and gh-118997). Later, Mark made commit c901437 to "GH-125174: Make immortal objects more robust, following design from PEP 683".nthanhcongaasc commentedon Nov 8, 2026
In my case,the PyQt maintainer will build future PyQt6 releases and snapshots against the 3.14 headers (mailing list). I'll rerun the repro above on the first such build and report back.
da-woods commentedon Nov 9, 2026
So I think this is mainly a Python bug and should be fixed there. But at least the PyAV code comes from Cython (PyQt6 doesn't though).
One thing we discussed at the time was using
Py_IncRef/Py_DecRefis the Limited API rather than using the macros? We dismissed it at the time as a lot of effort for no real benefit. Would it be worth reconsidering this?vstinner commentedon Nov 9, 2026
I only changed Py_INCREF/DECREF() implementation to become a function call in the limited C API version 3.12, because I was worried about the impact on performance. Py_INCREF() implements as
ob->ob_refcnt++is more efficient! You can also see that in 2023, the discussion on that change was quite long: https://discuss.python.org/t/limited-c-api-implement-py-incref-and-py-decref-as-function-calls/27592. So it wasn't an easy decision to take.But well, since 2023 a few things changed. For example, Free Threading is now around, required to design its own ABI (abi3t), and also requires to implement Py_INCREF/DECREF() as function calls.
Technically, Py_DecRef() and Py_IncRef() exist since the stable ABI version 3.2 (oldest stable ABI), whereas _Py_DecRef() and _Py_IncRef() require Python 3.10 or newer. So we can use function calls in all limited C versions.
@da-woods: Which limited C API version should start using function calls?
da-woods commentedon Nov 10, 2026
I was proposing to apply this in Cython rather than in the Python headers. So if I did it, then it'd be for 3.9-3.11 (i.e. any version that Cython supports that doesn't do this automatically)
Obviously you could update the Python headers but that'd only apply when people are compiling old limited API against new Python headers, which is probably rare (I don't think cibuildwheel works this way by default for example)
Use IncRef/DecRef functions on old Limited API (#8125)
Use IncRef/DecRef functions on old Limited API (#8125)
Use IncRef/DecRef functions on old Limited API (#8125)
da-woods commentedon Nov 10, 2026
One thing I spot (ignoring the possibility of a bigger change):
cpython/Include/refcount.h
Line 263 in 9d05428
I think that should be
# if Py_LIMITED_API+0 >= 0x030a0000(and the same inDECREF) sincePy_LIMITED_APIis almost always set to end with0000and so most people defining it for 3.10 won't get the version that uses the function. (The only other exception to this in the Python source code looks to be a single function in sliceobject.h but that's explicitly documented so shouldn't change).vstinner commentedon Nov 10, 2026
Oh, you're right! I wrote a way larger change to use _Py_PACK_VERSION() #161622 which replaces
0x030a00A7with_Py_PACK_VERSION(3, 10):-) It solves that issue.nthanhcongaasc commentedon Nov 16, 2026
Following up on my earlier comment: I reran the repro on PyQt6 6.12.1, the first release built against the 3.14 headers, and also checked the other abi3 wheels installed in the same environment (CPython 3.14.7, Windows x64). Numbers are the change in
None's refcount per call:cp37-abi3, anycv2function returningNone), psutil 7.2.2 (cp37-abi3,Process(),Process.is_running(),Process.ppid()), curl_cffi 0.17.0 (cp310-abi3,Curl.reset()).So 6 of the 11 abi3 packages I could measure drifted
Nonein the versions installed (True/Falsenever moved). Old-header abi3 builds still seem common.vstinner commentedon Nov 16, 2026
I understand that usually, when the limited C API version
VERSIONis used, the old Python versionVERSIONis used.I wrote #162120 to implemented Py_INCREF()/Py_DECREF() as function calls on all limited C API versions. So if you use Python 3.16 and target older limited C API version, you should work around this "objects de-immortalized" issue.
This change doesn't cover all cases, but it should fix the issue if you use Python 3.16.