Skip to content

test_capi.test_unicode.test_check_interned fails on a default build that reads a .pyc written by a free-threaded build #162092

Closed

Description

@maikschulze

Bug report

I was exploring peak RSS gaps of the free-threaded build with mimalloc v3 (#159718) with overnight stress tests and had a default build and a free-threaded build configured out of tree from one checkout, as a reference. The default build then fails one test every time, also when it runs alone:

$ ./python -m test test_capi -m test_check_interned
  File "Lib/test/test_capi/test_unicode.py", line 1896, in test_check_interned
    self.assertEqual(check_interned(s), SSTATE_INTERNED_MORTAL)
AssertionError: 2 != 1

It looks like an assumption of the test (added in #157798, also in 3.14 and 3.15) rather than a runtime bug. Main at e8c0584 (2026-11-10) on Linux arm64, and at 7c59816 (2026-11-16) on macOS arm64. An AI assistant helped with the analysis.

What happens

Whether it fails depends on which build wrote Lib/test/test_capi/__pycache__/test_unicode.cpython-316.pyc:

the .pyc was written by test run by the default build test run by the free-threaded build
the default build passes passes
the free-threaded build fails passes

Both builds use the cache tag cpython-316, so they read each other's .pyc files. The free-threaded build interns every string constant of a code object (_PyCode_ShouldInternString()), and marshal writes an interned string as TYPE_INTERNED. When the default build loads such a file, marshal interns every string marked that way as immortal. The test's literal 'long string unlikely to be used by Python' is a constant of the test's own code, so it is already interned and immortal by then, and sys.intern() of an equal fresh string returns that one.

With PYTHONPYCACHEPREFIX pointing to a directory of its own, the default build passes, and so does its full test suite.

Without the test suite

# interned_mod.py
import sys, _testcapi

def fresh_string(text):
    return text.encode().decode()

s = sys.intern(fresh_string('long string unlikely to be used by Python'))
print("free-threaded" if "t" in sys.abiflags else "default (GIL)", "build: PyUnicode_CHECK_INTERNED() of an interned copy is", _testcapi.unicode_check_interned(s), "(1 = mortal, 2 = immortal)")
$ rm -rf __pycache__; default/python -c "import interned_mod"
default (GIL) build: PyUnicode_CHECK_INTERNED() of an interned copy is 1 (1 = mortal, 2 = immortal)
$ rm -rf __pycache__; free-threaded/python -c "import interned_mod"
free-threaded build: PyUnicode_CHECK_INTERNED() of an interned copy is 2 (1 = mortal, 2 = immortal)
$ default/python -c "import interned_mod"
default (GIL) build: PyUnicode_CHECK_INTERNED() of an interned copy is 2 (1 = mortal, 2 = immortal)

A possible change

The test could intern a text that is put together at run time, so that it can't be a constant of its own code. Other tests build such strings at run time too, to get one that isn't interned yet, for example test_capi/test_object.py (''.join(['decref', '_delayed', '_immortal'])) and test_sys.test_intern ("never interned before" + str(random.randrange(0, 10**9))). In Lib/test/test_capi/test_unicode.py:

-        s = fresh_string('long string unlikely to be used by Python')
+        # Built at run time: a constant of this module may already be
+        # interned (and immortal) when the .pyc file was written by a
+        # free-threaded build.
+        s = fresh_string(''.join(['long string ', 'unlikely to be used by Python']))

With that change the test passes on the default build with a .pyc written by the free-threaded build, and the other way round, and test_capi.test_unicode passes on both builds. I'll open a PR with this change.

One thing beyond the test, in case it is of interest: a default build that loads .pyc files written by a free-threaded build keeps every string constant of those modules immortal. That only happens where both builds import the same source files, for example two builds from one checkout, or a project's own modules run with both python3.16 and python3.16t (the installed standard libraries are separate). I haven't measured what it costs, and whether the two builds should share .pyc files at all is a separate question; the test change doesn't depend on it.

cc @vstinner

CPython versions tested on:

3.14, 3.15, CPython main branch

Operating systems tested on:

macOS, Linux

Linked PRs

Activity

  1. maikschulze commented on Nov 16, 2026

    @maikschulze
    ContributorAuthor

    I've reached the limit of three open pull requests for this repository, so the PR has to wait until one of mine is merged or closed. The change is ready on a branch, one commit with the test change and a NEWS entry: main...maikschulze:cpython:gh-162092-test-check-interned

    Anyone is welcome to take it in the meantime.

  2. vstinner commented on Nov 16, 2026

    @vstinner
    Member

    This bug is complicated to reproduce. You need a GIL-enabled build and a Free Threaded build, side by side, and run tests in a specific order.

    cd ~/python/main
    # WARNING: remove all untracked files!
    git clean -fdx
    
    mkdir gil-build && cd gil-build
    ../configure --with-pydebug && make 
    cd ..
    
    mkdir freethreading-build && cd freethreading-build
    ../configure --with-pydebug --disable-gil && make 
    cd ..
    
    rm -rf Lib/test/test_capi/__pycache__/
    
    # Create the PYC file
    ./freethreading-build/python -m test -v test_capi.test_unicode
    
    # Load the PYC file created by Free Threaded build
    gil-build/python -m test -v test_capi.test_unicode
    

    Using Hachoir, I looked at the bytecode. PYC created by Free Threading build, the string is stored as "Short ASCII interned" (marshal format Z):

           - 232) item[2]= "long string unlikely to be used by Pytho(...)": Short ASCII interned (43 bytes)
                0.0) bytecode= 'Z': Bytecode (7 bits)
                0.7) flag_ref= False: Is a reference? (1 bits)
                1) len= 41: Number of ASCII characters (1 bytes)
                2) text= "long string unlikely to be used by Pytho(...)": String content (41 bytes)
    

    Using GIL build, the string is stored as "Short ASCII" (not interned) (marshal format z):

           - 232) item[2]= "long string unlikely to be used by Pytho(...)": Short ASCII (43 bytes)
                0.0) bytecode= 'z': Bytecode (7 bits)
                0.7) flag_ref= False: Is a reference? (1 bits)
                1) len= 41: Number of ASCII characters (1 bytes)
                2) text= "long string unlikely to be used by Pytho(...)": String content (41 bytes)
    
  3. vstinner commented on Nov 16, 2026

    @vstinner
    Member

    I've reached the limit of three open pull requests for this repository, so the PR has to wait until one of mine is merged or closed. The change is ready on a branch, one commit with the test change and a NEWS entry: main...maikschulze:cpython:gh-162092-test-check-interned

    This change only works around the problem. I prefer to fix the fresh_string() function instead. I created PR gh-162099. I added you as a co-author.

  4. maikschulze commented on Nov 16, 2026

    @maikschulze
    ContributorAuthor

    Thank you, @vstinner ! I figured it's worth to report so others don't stumble on this, e.g., when asking AI assistants create A/B test environments. It took this route a couple of times for me.

  5. added 3 commits that reference this issue on Nov 16, 2026
  6. vstinner commented on Nov 16, 2026

    @vstinner
    Member

    Fixed. Thanks for bug report and your fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    testsTests in the Lib/test dirtype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions