R07. The stable ABI

README.md

@@ -165,6 +165,7 @@ A lesson can also end with a boss fight, which is a problem the text does not so
 | R04 | [Frozen modules](lessons/r04-frozen-modules/r04.ipynb) | The import system is written in Python, so it cannot be imported, and the way out of that is to compile a handful of modules during CPython's own build and write the bytecode into the binary as C arrays. The module body of _frozen_importlib contains zero IMPORT_NAME opcodes, which is what makes it loadable with no import system running, and init_importlib in C hands sys and _imp to it as arguments. Thirty three names are frozen in a stock 3.15 build, in three groups that the flag treats differently: three for the import system that can never be switched off, nineteen for what a bare startup needs, and eleven hello world modules for the test suite. A frozen module still knows where it came from, because the loader puts the original path in loader_state and copies it onto __file__, which is why inspect.getsource and tracebacks still work while co_filename says <frozen os>. Switching the flag off in process with _imp._override_frozen_modules_for_tests moves os from FrozenImporter to SourceFileLoader and leaves the same constants behind. What freezing actually buys is the finder search and the file read, not the unmarshal, which is identical on both paths, and two recordings put it at about a tenth of a startup on a release build and a tenth on a debug build, where it is off by default | M8 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tamnd/cpython-internals/blob/main/lessons/r04-frozen-modules/r04.ipynb) |
 | R05 | [Lazy imports](lessons/r05-lazy-imports/r05.ipynb) | PEP 810 adds one word to the import statement in 3.15, and lazy import json binds the name now and does the finding, reading and running of the module the first time something reads that name back. It compiles to the same IMPORT_NAME opcode as a plain import, with the name index shifted up by two bits and the two bits underneath saying lazy, forced eager or ordinary, so dis prints json, json + lazy or json + eager. What gets bound is not a module but a five field placeholder that is not in sys.modules, that you can look at without setting it off, and whose name sits in sys.lazy_modules until it resolves. Only two opcodes resolve one, reading it as a bare name and reading it as an attribute of a module, so dict lookups, membership tests and reprs all leave it alone, and a placeholder copied into another variable resolves when that variable is read rather than when it was copied. The scope rules come from the symbol table rather than the code generator, which is why a lazy import inside a try block is refused with its own message. A failure inside a deferred module arrives with a second exception attached as its cause, pointing back at the lazy import line from information the placeholder was carrying. Three recordings put the startup saving at about three quarters of a run, and show that resolution takes the interpreter wide import lock rather than a lock per module name, so on a free threaded build four threads waking four different deferred imports keep 0.98 cores busy while four ordinary imports keep 3.60 | M8 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tamnd/cpython-internals/blob/main/lessons/r05-lazy-imports/r05.ipynb) |
 | R06 | [The C API tiers](lessons/r06-the-c-api-tiers/r06.ipynb) | The C API is three directories and two macros. Include is open to any extension, Include/cpython needs Py_LIMITED_API to be undefined, and Include/internal starts nearly every file with three lines that stop the compiler unless you define Py_BUILD_CORE. More than half the header lines are in that third directory. Defining Py_LIMITED_API hides 186 of the 766 functions the public headers declare and all 974 in the other two, and the part that costs is not the functions but the struct layouts, because with no fields to read Py_TYPE becomes a call and Py_DECREF becomes a call to _Py_DecRef. The naming convention nearly matches the directories and the exceptions have a reason: a private name has to be exported when a public macro expands to it, which is what the 17 underscore names in the public tier are. None of it survives the build. Every tier resolves through ctypes.pythonapi, and calling _PyDict_SizeOf by hand gives the same number dict.__sizeof__ does, 16 bytes short of sys.getsizeof because that adds the collector header. Two recordings show the split is deliberate: inside the internal headers 93 percent of the names spelled PyAPI_FUNC resolve against 0.4 percent of the ones spelled plain extern, with 168 comments naming which bundled extension needs each export | M8 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tamnd/cpython-internals/blob/main/lessons/r06-the-c-api-tiers/r06.ipynb) |
+| R07 | [The stable ABI](lessons/r07-the-stable-abi/r07.ipynb) | An extension has to get past two gates before it is a module, and both of them can be driven from Python. The first is the file name, read before anything is opened: the tag has to be one of the handful in a C array compiled into the interpreter, and a name that is wrong is not rejected but simply never looked at. Put empty files in a directory and ask the finder what it sees and the whole rule falls out, including that a free threaded build takes abi3t and refuses abi3, an ordinary 3.15 build takes both, and 3.14 has never heard of abi3t. The 3 in abi3 is not a Python version, it is PYTHON_ABI_VERSION, a counter that stopped moving in 2010, and sys.api_version has been 1013 since 2006. The second gate is PyABIInfo, twelve bytes new in 3.15, and because PyABIInfo_Check is a plain exported function the struct can be built in ctypes and handed to the real check, which refuses four of eight sample extensions and gives a different reason for each. The stable ABI itself is dated by the preprocessor gates around the declarations, 173 functions added since 3.2 with the busiest releases being the recent ones. Two recordings run both gates on a release build and on a build made with --disable-gil, where six tags become four and four refusals become six | M8 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tamnd/cpython-internals/blob/main/lessons/r07-the-stable-abi/r07.ipynb) |
 
 More are landing in order. [lessons/README.md](lessons/README.md) explains how one is put together and how to run them locally.
 

citations.lock.json

@@ -1,5 +1,10 @@
 {
   "citations": {
+    "Doc/c-api/stable.rst:100-105@v3.15.0rc1": {
+      "digest": "1376594fb65566a6",
+      "first_line": "On some platforms, Python will look for and load shared library files named",
+      "lines": 6
+    },
     "Doc/c-api/stable.rst:21-35@v3.15.0rc1": {
       "digest": "9299cb98031d1463",
       "first_line": "There are two tiers of C API with different stability expectations:",
@@ -150,6 +155,11 @@
       "first_line": "typedef struct _PyLongValue {",
       "lines": 9
     },
+    "Include/cpython/modsupport.h:41-45@v3.15.0rc1": {
+      "digest": "e20f48a944439662",
+      "first_line": "#ifdef Py_BUILD_CORE",
+      "lines": 5
+    },
     "Include/cpython/object.h:147-151@v3.15.0rc1": {
       "digest": "5c8d29cfecbeeaa4",
       "first_line": "// as well.",
@@ -960,11 +970,21 @@
       "first_line": "#define PyList_Check(op) \\",
       "lines": 3
     },
+    "Include/modsupport.h:135-143@v3.15.0rc1": {
+      "digest": "53f1e39873eb54fa",
+      "first_line": "#define _PyABIInfo_DEFAULT {                                \\",
+      "lines": 9
+    },
     "Include/modsupport.h:85-101@v3.15.0rc1": {
       "digest": "88ce91d467657676",
       "first_line": "/* ABI info & checking (new in 3.15) */",
       "lines": 17
     },
+    "Include/modsupport.h:86-101@v3.15.0rc1": {
+      "digest": "2f4ed0e32c8ed1fb",
+      "first_line": "#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= _Py_PACK_VERSION(3, 15)",
+      "lines": 16
+    },
     "Include/moduleobject.h:78-89@v3.15.0rc1": {
       "digest": "391087226de90dc4",
       "first_line": "/* for Py_mod_multiple_interpreters: */",
@@ -1070,6 +1090,11 @@
       "first_line": "#define ANNOTATIONS_PLACEHOLDER                256",
       "lines": 11
     },
+    "Include/patchlevel.h:53-62@v3.15.0rc1": {
+      "digest": "3dae732b1e88bab7",
+      "first_line": "/* The API and ABI versions are left for backwards compatibility.",
+      "lines": 10
+    },
     "Include/refcount.h:105-117@v3.15.0rc1": {
       "digest": "38a06fa5fe8a79cd",
       "first_line": "static inline Py_ssize_t _Py_REFCNT(PyObject *ob) {",
@@ -2165,6 +2190,11 @@
       "first_line": "case Py_mod_gil:",
       "lines": 6
     },
+    "Objects/moduleobject.c:471-481@v3.15.0rc1": {
+      "digest": "051ece003855e3a8",
+      "first_line": "case Py_mod_gil:",
+      "lines": 11
+    },
     "Objects/object.c:1573-1595@v3.15.0rc1": {
       "digest": "c469a683fb938006",
       "first_line": "_PyObject_ComputedDictPointer(PyObject *obj)",
@@ -4125,6 +4155,11 @@
       "first_line": "// None",
       "lines": 34
     },
+    "Python/dynload_shlib.c:39-60@v3.15.0rc1": {
+      "digest": "a07dedafcbd988d4",
+      "first_line": "const char *_PyImport_DynLoadFiletab[] = {",
+      "lines": 22
+    },
     "Python/executor_cases.c.h:2729-2744@v3.15.0rc1": {
       "digest": "4430981ca716fced",
       "first_line": "case _UNARY_NOT_r01: {",
@@ -4625,6 +4660,11 @@
       "first_line": "// Check if the filter disables the lazy import.",
       "lines": 25
     },
+    "Python/import.c:5175-5204@v3.15.0rc1": {
+      "digest": "655604314e26eb8e",
+      "first_line": "static PyObject *",
+      "lines": 30
+    },
     "Python/initconfig.c:1193-1201@v3.15.0rc1": {
       "digest": "0cd6d8c3fdac2d2a",
       "first_line": "#ifdef MS_WINDOWS",
@@ -4720,6 +4760,16 @@
       "first_line": "#define TYPE_NULL               '0'",
       "lines": 39
     },
+    "Python/modsupport.c:731-753@v3.15.0rc1": {
+      "digest": "e64c5435101ed32f",
+      "first_line": "if (info->flags & PyABIInfo_STABLE) {",
+      "lines": 23
+    },
+    "Python/modsupport.c:768-782@v3.15.0rc1": {
+      "digest": "209293eecfaefc64",
+      "first_line": "/* Free-threading/GIL */",
+      "lines": 15
+    },
     "Python/opcode_targets.h:1-12@v3.15.0rc1": {
       "digest": "1e469cc6f9bf653e",
       "first_line": "#if !_Py_TAIL_CALL_INTERP",

experiments/README.md

@@ -49,6 +49,8 @@ So those programs run somewhere else. They run in the images this project publis
 | [r05-how-much-of-a-wake-up-is-parallel-without-the-lock](tier1/r05-how-much-of-a-wake-up-is-parallel-without-the-lock.md) | R05 | freethreaded | With the global interpreter lock gone, does waking up a deferred import scale? |
 | [r06-what-leaves-the-binary](tier1/r06-what-leaves-the-binary.md) | R06 | release | How much of the C API that the headers call private is callable anyway? |
 | [r06-what-leaves-the-binary-on-a-free-threaded-build](tier1/r06-what-leaves-the-binary-on-a-free-threaded-build.md) | R06 | freethreaded | Does dropping the global interpreter lock change what the C API exports? |
+| [r07-what-a-build-will-load](tier1/r07-what-a-build-will-load.md) | R07 | release | What does the interpreter check before it agrees to load an extension? |
+| [r07-what-a-build-will-load-without-the-lock](tier1/r07-what-a-build-will-load-without-the-lock.md) | R07 | freethreaded | How much of the stable ABI does dropping the global interpreter lock rule out? |
 
 ## The commands
 

experiments/tier1/r07-what-a-build-will-load-without-the-lock.md

@@ -0,0 +1,230 @@
+# The same two gates on the build that abi3t was invented for
+
+Generated by `just build-tier1`. Do not edit by hand, the change will be overwritten.
+
+How much of the stable ABI does dropping the global interpreter lock rule out?
+
+- Lesson: R07
+- Build: freethreaded
+- Image: ghcr.io/tamnd/cpython-internals/cpython:freethreaded@sha256:db72284e3a49f43c38b96bec2baed1380b8348e27ea6f54f6e8d0810b59c3144
+- Interpreter: 3.15.0rc1 free-threading build (37e98da:37e98da, Aug 29 2026, 09:25:00) [GCC 14.2.0]
+- Recorded: 2026-09-06
+
+Why this needs the freethreaded build: it needs a build configured with --disable-gil, because the whole question is which file names and which ABI flags that build is willing to accept.
+
+## The program
+
+```python
+"""What a build agrees to load, and the two gates that decide it.
+
+An extension arrives as a file with a long name. The interpreter decides whether to consider
+it at all from that name alone, before it opens it. If it does open it, and the extension was
+built against 3.15 or later, there is a second gate: a five field `PyABIInfo` struct the
+extension carries, which `PyABIInfo_Check` compares against the running interpreter.
+
+This program runs both gates by hand. It asks the finder which file names it is willing to
+look at, by putting empty files in a temporary directory and asking for the module. Then it
+builds `PyABIInfo` structs in ctypes for a row of hypothetical extensions and calls the real
+check function on each one.
+
+The point of running it on two builds is that both gates move. A free threaded build will not
+look at a name with `abi3` in it, and it refuses a struct that says the extension wants the
+global interpreter lock.
+
+One thing is printed differently from how the interpreter spells it. The last part of a file
+name tag is the machine the build was made for, which is `x86_64-linux-gnu` on one runner and
+`aarch64-linux-gnu` on another. That part says nothing about the ABI question, so it is
+replaced with the word `PLATFORM` everywhere below and the recording is the same either way.
+"""
+
+import ctypes
+import importlib.machinery
+import pathlib
+import re
+import sys
+import sysconfig
+import tempfile
+
+VERSIONED = re.compile(r"(cpython-)(\d+)(t?)")
+
+STABLE = 0x0001
+GIL = 0x0002
+FREETHREADED = 0x0004
+INTERNAL = 0x0008
+
+
+class PyABIInfo(ctypes.Structure):
+    """The struct an extension built against 3.15 or later carries about itself."""
+
+    _fields_ = [
+        ("abiinfo_major_version", ctypes.c_uint8),
+        ("abiinfo_minor_version", ctypes.c_uint8),
+        ("flags", ctypes.c_uint16),
+        ("build_version", ctypes.c_uint32),
+        ("abi_version", ctypes.c_uint32),
+    ]
+
+
+def pack(major, minor):
+    """The version format the C API uses, which is the two numbers in the top two bytes."""
+    return (major << 24) | (minor << 16)
+
+
+def considered(name):
+    """Put one empty file in a directory of its own and ask whether the finder sees it."""
+    room = pathlib.Path(tempfile.mkdtemp())
+    (room / name).write_bytes(b"")
+    return importlib.machinery.PathFinder.find_spec("demo", [str(room)]) is not None
+
+
+TAGS = importlib.machinery.EXTENSION_SUFFIXES
+here = sysconfig.get_config_var("EXT_SUFFIX")
+flipped = VERSIONED.sub(lambda m: m.group(1) + m.group(2) + ("" if m.group(3) else "t"), here)
+future = VERSIONED.sub(lambda m: m.group(1) + "999" + m.group(3), here)
+machine = re.sub(r"^\.cpython-\d+t?-?", "", here).removesuffix(".so")
+
+
+def tidy(text):
+    """Take the machine out of a file name tag, since it says nothing about the ABI."""
+    return text.replace(machine, "PLATFORM") if machine else text
+
+
+print("version:", sys.version.split()[0])
+print("abiflags:", repr(sys.abiflags))
+print("gil disabled:", sysconfig.get_config_var("Py_GIL_DISABLED"))
+print("soabi:", tidy(sysconfig.get_config_var("SOABI")))
+print("api version, unchanged since 2006:", sys.api_version)
+print()
+
+print("file name tags this build will load:")
+for suffix in TAGS:
+    print("   ", tidy(suffix))
+print()
+
+print("what the finder does with one file of each kind")
+for name in ("demo" + here, "demo" + flipped, "demo" + future, "demo.abi3.so", "demo.abi3t.so"):
+    verdict = "considered" if considered(name) else "invisible, the name is wrong"
+    print(f"  {tidy(name):34} {verdict}")
+print()
+
+check = ctypes.pythonapi.PyABIInfo_Check
+check.argtypes = [ctypes.POINTER(PyABIInfo), ctypes.c_char_p]
+check.restype = ctypes.c_int
+
+CASES = (
+    ("built for this exact version", GIL, pack(*sys.version_info[:2])),
+    ("built for 3.13 and nothing else", GIL, pack(3, 13)),
+    ("stable abi since 3.2", STABLE | GIL, pack(3, 2)),
+    ("stable abi since 3.14", STABLE | GIL, pack(3, 14)),
+    ("stable abi from the future", STABLE | GIL, pack(3, 99)),
+    ("free threaded only", STABLE | FREETHREADED, pack(3, 14)),
+    ("happy either way", STABLE | GIL | FREETHREADED, pack(3, 14)),
+    ("claims internal and stable", STABLE | INTERNAL | GIL, sys.hexversion),
+)
+
+print("what PyABIInfo_Check makes of each kind of extension")
+refused = 0
+for label, flags, version in CASES:
+    info = PyABIInfo(1, 0, flags, sys.hexversion, version)
+    try:
+        check(ctypes.byref(info), label.encode())
+    except ImportError as problem:
+        refused += 1
+        print(f"  {label:32} refused: {str(problem).split(': ', 1)[1]}")
+    else:
+        print(f"  {label:32} loads")
+print()
+
+DECLARED = re.compile(r"^PyAPI_FUNC\([^)]*\)\s*\**\s*(\w+)", re.M)
+OPENS = re.compile(r"^\s*#\s*if")
+CLOSES = re.compile(r"^\s*#\s*endif")
+OLD = re.compile(r"Py_LIMITED_API\+0\s*>=\s*0x03([0-9a-fA-F]{2})0000")
+NEW = re.compile(r"Py_LIMITED_API\+0\s*>=\s*_Py_PACK_VERSION\((\d+),\s*(\d+)\)")
+
+
+def gate(line):
+    """The stable ABI version a preprocessor line gates on, when it gates on one."""
+    found = NEW.search(line)
+    if found:
+        return int(found.group(2))
+    found = OLD.search(line)
+    return int(found.group(1), 16) if found else None
+
+
+include = pathlib.Path(sysconfig.get_paths()["include"])
+arrived = {}
+for path in sorted(include.glob("*.h")):
+    stack = []
+    for line in path.read_text(errors="replace").splitlines():
+        if OPENS.match(line):
+            stack.append(gate(line))
+        elif CLOSES.match(line) and stack:
+            stack.pop()
+        found = DECLARED.match(line)
+        if found and any(v for v in stack):
+            arrived[found.group(1)] = max(v for v in stack if v)
+
+print("when the public headers say each gated function arrived")
+counted = {}
+for version in arrived.values():
+    counted[version] = counted.get(version, 0) + 1
+for version in sorted(counted):
+    print(f"  3.{version:<4} {counted[version]:4}")
+print()
+
+print(f"~ file name tags this build will load: {len(TAGS)}")
+print(f"~ hypothetical extensions refused: {refused} of {len(CASES)}")
+print(f"~ public functions behind a stable abi version gate: {len(arrived)}")
+```
+
+## What it printed
+
+```text
+version: 3.15.0rc1
+abiflags: 't'
+gil disabled: 1
+soabi: cpython-315t-PLATFORM
+api version, unchanged since 2006: 1013
+
+file name tags this build will load:
+    .cpython-315t-PLATFORM.so
+    .abi3t-PLATFORM.so
+    .abi3t.so
+    .so
+
+what the finder does with one file of each kind
+  demo.cpython-315t-PLATFORM.so      considered
+  demo.cpython-315-PLATFORM.so       invisible, the name is wrong
+  demo.cpython-999t-PLATFORM.so      invisible, the name is wrong
+  demo.abi3.so                       invisible, the name is wrong
+  demo.abi3t.so                      considered
+
+what PyABIInfo_Check makes of each kind of extension
+  built for this exact version     refused: incompatible with free-threaded CPython
+  built for 3.13 and nothing else  refused: incompatible ABI version (3.13)
+  stable abi since 3.2             refused: incompatible with free-threaded CPython
+  stable abi since 3.14            refused: incompatible with free-threaded CPython
+  stable abi from the future       refused: incompatible future stable ABI version (3.99)
+  free threaded only               loads
+  happy either way                 loads
+  claims internal and stable       refused: cannot use both internal and stable ABI
+
+when the public headers say each gated function arrived
+  3.3      26
+  3.4       3
+  3.5      13
+  3.6       4
+  3.7      12
+  3.8       1
+  3.9      10
+  3.10      5
+  3.11     16
+  3.12      6
+  3.13     27
+  3.14     18
+  3.15     32
+
+~ file name tags this build will load: 4
+~ hypothetical extensions refused: 6 of 8
+~ public functions behind a stable abi version gate: 173
+```