Repository navigation
Description
Bug report
I was exploring peak RSS gaps of the free-threaded build with mimalloc v3 (#159718) with overnight stress tests and had a default build and a free-threaded build configured out of tree from one checkout, as a reference. The default build then fails one test every time, also when it runs alone:
$ ./python -m test test_capi -m test_check_interned
File "Lib/test/test_capi/test_unicode.py", line 1896, in test_check_interned
self.assertEqual(check_interned(s), SSTATE_INTERNED_MORTAL)
AssertionError: 2 != 1
It looks like an assumption of the test (added in #157798, also in 3.14 and 3.15) rather than a runtime bug. Main at e8c0584 (2026-11-10) on Linux arm64, and at 7c59816 (2026-11-16) on macOS arm64. An AI assistant helped with the analysis.
What happens
Whether it fails depends on which build wrote Lib/test/test_capi/__pycache__/test_unicode.cpython-316.pyc:
the .pyc was written by |
test run by the default build | test run by the free-threaded build |
|---|---|---|
| the default build | passes | passes |
| the free-threaded build | fails | passes |
Both builds use the cache tag cpython-316, so they read each other's .pyc files. The free-threaded build interns every string constant of a code object (_PyCode_ShouldInternString()), and marshal writes an interned string as TYPE_INTERNED. When the default build loads such a file, marshal interns every string marked that way as immortal. The test's literal 'long string unlikely to be used by Python' is a constant of the test's own code, so it is already interned and immortal by then, and sys.intern() of an equal fresh string returns that one.
With PYTHONPYCACHEPREFIX pointing to a directory of its own, the default build passes, and so does its full test suite.
Without the test suite
# interned_mod.py
import sys, _testcapi
def fresh_string(text):
return text.encode().decode()
s = sys.intern(fresh_string('long string unlikely to be used by Python'))
print("free-threaded" if "t" in sys.abiflags else "default (GIL)", "build: PyUnicode_CHECK_INTERNED() of an interned copy is", _testcapi.unicode_check_interned(s), "(1 = mortal, 2 = immortal)")$ rm -rf __pycache__; default/python -c "import interned_mod"
default (GIL) build: PyUnicode_CHECK_INTERNED() of an interned copy is 1 (1 = mortal, 2 = immortal)
$ rm -rf __pycache__; free-threaded/python -c "import interned_mod"
free-threaded build: PyUnicode_CHECK_INTERNED() of an interned copy is 2 (1 = mortal, 2 = immortal)
$ default/python -c "import interned_mod"
default (GIL) build: PyUnicode_CHECK_INTERNED() of an interned copy is 2 (1 = mortal, 2 = immortal)
A possible change
The test could intern a text that is put together at run time, so that it can't be a constant of its own code. Other tests build such strings at run time too, to get one that isn't interned yet, for example test_capi/test_object.py (''.join(['decref', '_delayed', '_immortal'])) and test_sys.test_intern ("never interned before" + str(random.randrange(0, 10**9))). In Lib/test/test_capi/test_unicode.py:
- s = fresh_string('long string unlikely to be used by Python')
+ # Built at run time: a constant of this module may already be
+ # interned (and immortal) when the .pyc file was written by a
+ # free-threaded build.
+ s = fresh_string(''.join(['long string ', 'unlikely to be used by Python']))With that change the test passes on the default build with a .pyc written by the free-threaded build, and the other way round, and test_capi.test_unicode passes on both builds. I'll open a PR with this change.
One thing beyond the test, in case it is of interest: a default build that loads .pyc files written by a free-threaded build keeps every string constant of those modules immortal. That only happens where both builds import the same source files, for example two builds from one checkout, or a project's own modules run with both python3.16 and python3.16t (the installed standard libraries are separate). I haven't measured what it costs, and whether the two builds should share .pyc files at all is a separate question; the test change doesn't depend on it.
cc @vstinner
CPython versions tested on:
3.14, 3.15, CPython main branch
Operating systems tested on:
macOS, Linux
Activity
maikschulze commentedon Nov 16, 2026
I've reached the limit of three open pull requests for this repository, so the PR has to wait until one of mine is merged or closed. The change is ready on a branch, one commit with the test change and a NEWS entry: main...maikschulze:cpython:gh-162092-test-check-interned
Anyone is welcome to take it in the meantime.
vstinner commentedon Nov 16, 2026
This bug is complicated to reproduce. You need a GIL-enabled build and a Free Threaded build, side by side, and run tests in a specific order.
Using Hachoir, I looked at the bytecode. PYC created by Free Threading build, the string is stored as "Short ASCII interned" (marshal format
Z):Using GIL build, the string is stored as "Short ASCII" (not interned) (marshal format
z):vstinner commentedon Nov 16, 2026
This change only works around the problem. I prefer to fix the fresh_string() function instead. I created PR gh-162099. I added you as a co-author.
maikschulze commentedon Nov 16, 2026
Thank you, @vstinner ! I figured it's worth to report so others don't stumble on this, e.g., when asking AI assistants create A/B test environments. It took this route a couple of times for me.
gh-162092: Fix test_check_interned() of test_capi (#162099)
[3.15] gh-162092: Fix test_check_interned() of test_capi (GH-162099) (#…
[3.14] gh-162092: Fix test_check_interned() of test_capi (GH-162099) (#…
vstinner commentedon Nov 16, 2026
Fixed. Thanks for bug report and your fix.