Skip to content

3.14: objects de-immortalized by old stable-ABI extensions break truthiness checks in specialized bytecode #158401

Open

Description

@clairernovotny

Bug report

Bug description:

Extensions compiled against 3.11 or older headers implement Py_DECREF as an inline --ob_refcnt. That includes abi3 wheels built on 3.11, which are common. Since 3.12 the interpreter hands out references to immortal objects without incrementing their refcount, so each such reference the extension releases is a net decrement. 3.14 starts immortal objects at 3 << 30 and treats (int32_t)ob_refcnt < 0 as immortal, which leaves a margin of 2**30 (see gh-125174).

That margin runs out in real long-running processes. Home Assistant on 3.14.6 crashes after hours to days because PyAV's cp311-abi3 wheels drain True by a few counts per av.open() and per FFmpeg log line (PyAV-Org/PyAV#2420, home-assistant/core#178218). A core dump from an affected instance had _Py_TrueStruct.ob_refcnt == 0x7ffb2327, which is 2**30 + 318,681 below 0xC0000000.

On 3.12 and 3.13, losing immortality is fairly benign: the object is then refcounted like any other, and results stay correct. On 3.14 it is a silent correctness failure. The reason is in the default build's stackrefs:

  • PyStackRef_IsTrue/IsFalse/IsNone compare the tagged bits exactly.
  • _PyStackRef_FromPyObjectNew and PyStackRef_MakeHeapSafe decide whether to tag with _Py_IsImmortal(), which reads the refcount.
  • PyStackRef_FromPyObjectSteal decides with ob_flags.

Once True's refcount is below 2**31, a True loaded by a specialized attribute load (LOAD_ATTR_INSTANCE_VALUE and others) or passed through RETURN_VALUE is untagged, and POP_JUMP_IF_TRUE treats it as false. True from constants, comparisons and C calls still tests true, so the failures look random. In Home Assistant, Thread.name asserted Thread.__init__() not called, Condition.notify raised cannot notify on un-acquired lock, traceback printed None as the exception type, and so on, across threads, until the process exited.

Reproducer. ctypes stands in for the extension's unbalanced decrefs:

import ctypes, sys

print(sys.version)

class C:
    def __init__(self):
        self.flag = True

def check_attr(o):
    if not o.flag:                      # LOAD_ATTR_INSTANCE_VALUE, POP_JUMP_IF_TRUE
        return "WRONG: o.flag is True but tested false"
    return "ok"

def get_true():
    return True

def check_return():
    if not get_true():                  # CALL_PY_EXACT_ARGS, RETURN_VALUE, POP_JUMP_IF_TRUE
        return "WRONG: get_true() returned True but tested false"
    return "ok"

o = C()
for _ in range(1000):                   # warm up so both functions get specialized
    check_attr(o), check_return()

refcnt = ctypes.c_uint32.from_address(id(True))   # low 32 bits of ob_refcnt (64-bit little-endian)
print("before: ob_refcnt = %#x" % refcnt.value, check_attr(o), check_return())
refcnt.value = 2**31 - 10**6            # 3.14: 2**30 + 10**6 raw decrements from 0xc0000000
print("after:  ob_refcnt = %#x" % refcnt.value, check_attr(o), check_return())
print("o.flag is True:", o.flag is True, "| bool(o.flag):", bool(o.flag))

Output with the official Docker images (python:3.x, Linux aarch64, default build):

3.14.7 (main, Sep 19 2026, 03:34:22) [GCC 14.2.0]
before: ob_refcnt = 0xc0000000 ok ok
after:  ob_refcnt = 0x7ff0bdc0 WRONG: o.flag is True but tested false WRONG: get_true() returned True but tested false
o.flag is True: True | bool(o.flag): True

3.15.0rc2 (main, Sep 19 2026, 03:32:36) [GCC 14.2.0]
before: ob_refcnt = 0xc0000000 ok ok
after:  ob_refcnt = 0x7ff0bdc0 WRONG: o.flag is True but tested false WRONG: get_true() returned True but tested false
o.flag is True: True | bool(o.flag): True

3.13.15 (main, Sep 19 2026, 03:38:46) [GCC 14.2.0]
before: ob_refcnt = 0xffffffff ok ok
after:  ob_refcnt = 0x7ff0bdc0 ok ok
o.flag is True: True | bool(o.flag): True

3.12.14 (main, Sep 19 2026, 03:37:20) [GCC 14.2.0]
before: ob_refcnt = 0xffffffff ok ok
after:  ob_refcnt = 0x7ff0bdc0 ok ok
o.flag is True: True | bool(o.flag): True

The value has to be far enough below 2**31. At exactly 0x7fffffff, the first mortal-path incref brings True back to 0x80000000, and it is immortal again.

The drift itself is the extension's bug, and PyAV is fixing its build. But 3.14 turns it from a lost optimization into wrong branches, and nothing in the process points at the cause. Some options, in no particular order:

  • Tag consistently. _PyStackRef_FromPyObjectNew and PyStackRef_MakeHeapSafe could use ob_flags & _Py_IMMORTAL_FLAGS as PyStackRef_FromPyObjectSteal does. Alternatively, PyStackRef_IsTrue/IsFalse/IsNone could ignore the tag bit. Either would fix the truth tests, though not the drift.
  • Treat statically allocated objects (_Py_STATICALLY_ALLOCATED_FLAG) as immortal whatever their refcount says.
  • Periodically reset the refcount of static immortal singletons, for example during GC, as suggested in "Immortal" objects aren't immortal and that breaks things. #125174.

CPython versions tested on:

3.12, 3.13, 3.14, 3.15

Operating systems tested on:

Linux

Linked PRs

Activity

  1. picnixz commented on Sep 29, 2026

    @picnixz
    Member

    should I understand that the problem is extensions not properly managing refcounts or is there something else? I don't understand the issue exactly here. AFAIU:

    The drift itself is the extension's bug, and PyAV is fixing its build

    So that's the extension problem? or is it ours?

  2. added
    type-bugAn unexpected behavior, bug, or error
    interpreter-core(Objects, Python, Grammar, and Parser dirs)
    pendingThe issue will be closed if no feedback is provided
    on Sep 29, 2026
  3. clairernovotny commented on Oct 1, 2026

    @clairernovotny
    Author

    Both, in different ways.

    The drift itself is the extension's bug: PyAV's abi3 wheels were compiled against 3.11 headers, so their inline Py_DECREF is a raw decrement, applied to references (e.g. Py_True from PyObject_RichCompare) that 3.12+ hands out without an incref. PyAV 19.0.0 builds against 3.12 headers and no longer drifts (measured: 0 vs. −12k to −34k per 4,000 av.open calls on 17.x/18.x).

    The part that may be CPython's is what happens once such drift takes a singleton below the immortality threshold. On 3.12/3.13 that is harmless, and the reproducer above prints ok ok. On 3.14/3.15, specialized code starts treating True as false (WRONG: o.flag is True but tested false), and in a real process that showed up as "impossible" assertion failures across threads, then a crash. Since the stable ABI still lets extensions built with 3.11 headers load on 3.14, the question is whether 3.14 should tolerate a de-immortalized singleton rather than silently miscompute, for example by not deriving the stackref tag from _Py_IsImmortal, or by checking ob_flags consistently.

  4. picnixz commented on Oct 1, 2026

    @picnixz
    Member

    Ah I see, ok now it's clearer. What we never want is true becoming false at some point if extensions correctly manage their references. If this is the case, then this means something is wrong.

    The problem is that 3.11 is security-only and this can't really be considered a security issue. The best we can do is possibly detecting on 3.14+ whether we are using old stable ABI or not to determine what to do with the refcount properly.

    I will let @markshannon decide whwat to do because I don't have enough insight on whether checking immortality properly could actually slow down things (I belieev it can, but by how much I don't know). In some sense, may I read this issue as "3.14 broke Stable ABI because of immortality assumptions"?

  5. removed
    pendingThe issue will be closed if no feedback is provided
    on Oct 1, 2026
  6. clairernovotny commented on Oct 1, 2026

    @clairernovotny
    Author

    Yes, I think that's a fair reading, with one refinement: the extension is managing its references correctly by the rules it was compiled for.

    Under 3.11 semantics, PyObject_RichCompare returns a new reference to Py_True and the extension Py_DECREFs it, which is balanced. On 3.12+, the runtime skips the incref for immortal objects, but the 3.11-header inline Py_DECREF in the extension still does a raw --ob_refcnt. So code that is correct under the limited API it targets produces a net decrement per call, and in a long-running process that adds up. In the Home Assistant case it crossed the threshold in hours to days, depending on how much FFmpeg logged.

    PEP 683 anticipated old extensions touching immortal refcounts and relied on the margin being too large to matter. That held on 3.12/3.13 because losing immortality there is benign. On 3.14 it isn't, because the stackref fast paths derive the tag from _Py_IsImmortal() while other paths use the static flags, so True can compare unequal to itself.

    So I agree the fix belongs in 3.14+, not 3.11. A cheap option might be to make PyStackRef_IsTrue/IsFalse/IsNone (and the tagging of these static singletons) independent of the current refcount, for example by comparing the untagged pointer or by tagging statically allocated immortals from ob_flags. That avoids an immortality check on the hot incref/decref paths. Whether that is acceptable performance-wise is Mark's call.

  7. added
    3.14bugs and security fixes
    3.15bugs and security fixes
    3.16new features, bugs and security fixes
    on Oct 1, 2026
  8. nthanhcongaasc commented on Nov 4, 2026

    @nthanhcongaasc

    Another affected extension: PyQt6 (6.11.0 and the latest 6.12.0, cp310-abi3 wheels), CPython 3.14.7, Windows x64.

    Drift source. Every PyQt6 method that returns None adds exactly +1 to None's raw refcount, and that +1 is never taken back. Methods returning other values add nothing. True/False were unaffected in my measurements.

    call (100k times) None raw refcount per call
    QPoint.setX(), QObject.setObjectName(), QColor.setRed(), QApplication.processEvents() +1.00
    QWidget.isVisible() (returns bool) 0
    pure-Python function / 3.14 builtin returning None 0

    Time to failure is hours, not days. The pattern while w.isVisible(): app.processEvents() is very common in PyQt apps, and every iteration is +1. In a real app this was about 160k/s, so None lost immortality after roughly 1h50m. In the live process I read _Py_NoneStruct as low32=0x0edd14b7, ob_overflow=1, while True, False, ... and () were all still 0xC0000000.

    The None case is more severe than the truthiness case. Once None is no longer immortal, the following break process-wide and stay broken until restart:

    • every Python-level __init__: TypeError: __init__() should return None, not 'NoneType'
    • yield from over a builtin iterator: AttributeError: 'range_iterator' object has no attribute 'send' (seen in pandas' RangeIndex.__iter__)
    • warnings.catch_warnings(): TypeError: _py_warnings.simplefilter() argument after * must be an iterable, not NoneType

    As a result, logging, traceback and pandas all fail, so the application cannot even log the error.

    In the broken process I checked the co_consts of every live function, about 20k of which contain None. All of them point to the same _Py_NoneStruct. It is the same object; only the stackref tagging differs.

    Minimal repro (no memory writes; takes about 5 minutes):

    import ctypes
    from PyQt6.QtCore import QPoint
    
    low32 = lambda: ctypes.c_uint32.from_address(id(None)).value
    overflow = lambda: ctypes.c_uint16.from_address(id(None) + 4).value
    
    class A:
        def __init__(self):
            pass
    
    f = QPoint().setX
    before = low32()
    for _ in range(100_000):
        f(5)
    print("None refcount delta per call:", (low32() - before) / 100_000)   # 1.0
    
    while overflow() == 0:            # about 1.07e9 calls
        for _ in range(5_000_000):
            f(5)
    A()   # TypeError: __init__() should return None, not 'NoneType'

    Because a well-behaved, widely used abi3 extension hits this within hours of normal use, +1 for making PyStackRef_IsNone/IsTrue/IsFalse independent of the current refcount, for example by tagging static immortals from ob_flags.

  9. picnixz commented on Nov 4, 2026

    @picnixz
    Member

    cc @encukou @ngoldbaum for more visibility

  10. vstinner commented on Nov 7, 2026

    @vstinner
    Member

    Extensions compiled against 3.11 or older headers implement Py_DECREF as an inline --ob_refcnt. (...) PyAV's cp311-abi3 wheels

    and:

    PyQt6 (6.11.0 and the latest 6.12.0, cp310-abi3 wheels)

    If you build PyAV and PyQt6 with Python 3.12 or newer, do you still hit the issue?

    I changed Py_INCREF/DECREF() implementation to make them an opaque function call in Python 3.12 to fix this exact problem: https://discuss.python.org/t/limited-c-api-implement-py-incref-and-py-decref-as-function-calls/27592. Sadly, it means that you need a different binary for Python 3.12+ than the binary for Python 3.11 and older.

    This refcount issue is just a timebomb awaiting to explode :-( Building C extensions with Python 3.12 or newer fix the root issue.


    In the past, I added _Py_IsImmortalLoose() (commit b826e45) to try to reduce the risk of hitting this refcnt/stable ABI issue (issues gh-121528 and gh-118997). Later, Mark made commit c901437 to "GH-125174: Make immortal objects more robust, following design from PEP 683".

  11. nthanhcongaasc commented on Nov 8, 2026

    @nthanhcongaasc

    If you build PyAV and PyQt6 with Python 3.12 or newer, do you still hit the issue?

    In my case,the PyQt maintainer will build future PyQt6 releases and snapshots against the 3.14 headers (mailing list). I'll rerun the repro above on the first such build and report back.

  12. da-woods commented on Nov 9, 2026

    @da-woods
    Contributor

    So I think this is mainly a Python bug and should be fixed there. But at least the PyAV code comes from Cython (PyQt6 doesn't though).

    One thing we discussed at the time was using Py_IncRef/Py_DecRef is the Limited API rather than using the macros? We dismissed it at the time as a lot of effort for no real benefit. Would it be worth reconsidering this?

  13. vstinner commented on Nov 9, 2026

    @vstinner
    Member

    One thing we discussed at the time was using Py_IncRef/Py_DecRef is the Limited API rather than using the macros? We dismissed it at the time as a lot of effort for no real benefit. Would it be worth reconsidering this?

    I only changed Py_INCREF/DECREF() implementation to become a function call in the limited C API version 3.12, because I was worried about the impact on performance. Py_INCREF() implements as ob->ob_refcnt++ is more efficient! You can also see that in 2023, the discussion on that change was quite long: https://discuss.python.org/t/limited-c-api-implement-py-incref-and-py-decref-as-function-calls/27592. So it wasn't an easy decision to take.

    But well, since 2023 a few things changed. For example, Free Threading is now around, required to design its own ABI (abi3t), and also requires to implement Py_INCREF/DECREF() as function calls.

    Technically, Py_DecRef() and Py_IncRef() exist since the stable ABI version 3.2 (oldest stable ABI), whereas _Py_DecRef() and _Py_IncRef() require Python 3.10 or newer. So we can use function calls in all limited C versions.

    @da-woods: Which limited C API version should start using function calls?

  14. da-woods commented on Nov 10, 2026

    @da-woods
    Contributor

    @da-woods: Which limited C API version should start using function calls?

    I was proposing to apply this in Cython rather than in the Python headers. So if I did it, then it'd be for 3.9-3.11 (i.e. any version that Cython supports that doesn't do this automatically)

    Obviously you could update the Python headers but that'd only apply when people are compiling old limited API against new Python headers, which is probably rare (I don't think cibuildwheel works this way by default for example)

  15. da-woods commented on Nov 10, 2026

    @da-woods
    Contributor

    One thing I spot (ignoring the possibility of a bigger change):

    # if Py_LIMITED_API+0 >= 0x030a00A7

    I think that should be # if Py_LIMITED_API+0 >= 0x030a0000 (and the same in DECREF) since Py_LIMITED_API is almost always set to end with 0000 and so most people defining it for 3.10 won't get the version that uses the function. (The only other exception to this in the Python source code looks to be a single function in sliceobject.h but that's explicitly documented so shouldn't change).

  16. vstinner commented on Nov 10, 2026

    @vstinner
    Member

    I think that should be # if Py_LIMITED_API+0 >= 0x030a0000 (and the same in DECREF) since Py_LIMITED_API is almost always set to end with 0000 and so most people defining it for 3.10 won't get the version that uses the function.

    Oh, you're right! I wrote a way larger change to use _Py_PACK_VERSION() #161622 which replaces 0x030a00A7 with _Py_PACK_VERSION(3, 10) :-) It solves that issue.

  17. nthanhcongaasc commented on Nov 16, 2026

    @nthanhcongaasc

    Following up on my earlier comment: I reran the repro on PyQt6 6.12.1, the first release built against the 3.14 headers, and also checked the other abi3 wheels installed in the same environment (CPython 3.14.7, Windows x64). Numbers are the change in None's refcount per call:

    • PyQt6: fixed. PyQt6 6.12.1 and PyQt6-WebEngine 6.140.1 (now on PyPI) stay at 0; 6.11.0 and 6.12.0 were +1 per void call. So building with newer headers does fix it.
    • Still +1 per call in the latest release: opencv-python 5.0.0.93 (cp37-abi3, any cv2 function returning None), psutil 7.2.2 (cp37-abi3, Process(), Process.is_running(), Process.ppid()), curl_cffi 0.17.0 (cp310-abi3, Curl.reset()).
    • protobuf 7.36.2 / 7.37.0rc1: +1, already fixed on main (Build limited API Python wheels against Python 3.14 headers. protocolbuffers/protobuf#31089).
    • Clean: pymupdf 1.28.2, tokenizers 0.23.2, safetensors 0.8.0, cryptography 50.0.1, pycryptodome 3.23.0.

    So 6 of the 11 abi3 packages I could measure drifted None in the versions installed (True/False never moved). Old-header abi3 builds still seem common.

  18. vstinner commented on Nov 16, 2026

    @vstinner
    Member

    I understand that usually, when the limited C API version VERSION is used, the old Python version VERSION is used.

    I wrote #162120 to implemented Py_INCREF()/Py_DECREF() as function calls on all limited C API versions. So if you use Python 3.16 and target older limited C API version, you should work around this "objects de-immortalized" issue.

    This change doesn't cover all cases, but it should fix the issue if you use Python 3.16.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.14bugs and security fixes3.15bugs and security fixes3.16new features, bugs and security fixesinterpreter-core(Objects, Python, Grammar, and Parser dirs)topic-C-APItype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions