Skip to content

Free-threaded build: a two-object critical section that begins during a stop-the-world can leave its first mutex locked #162087

Open

Description

@maikschulze

Bug report

I was exploring peak RSS gaps of the free-threaded build with mimalloc v3 (#159718) with overnight stress tests and, while looking into lock handoffs, read the fix for #144513 (#144524). One path in it looks like it leaves a mutex locked. An AI assistant helped with the analysis.

What happens

_PyCriticalSection2_BeginSlow() returns early when the interpreter's world is stopped and records no mutexes, so _PyCriticalSection2_End() unlocks nothing. But the inline fast path, _PyCriticalSection2_BeginMutex(), has already locked the first mutex when it calls the slow path for the second one (is_m1_locked == 1). That first mutex then stays locked after the section has ended, and the next critical section on that object, in any thread, waits forever.

It needs the second mutex to be busy during a stop-the-world, which is the case #144513 is about (a mutex handed off to a thread that is already suspended). I couldn't find code that gets there today: of the stop-the-world regions on main, the ones I traced take a single-object critical section at most (for example object_set_class_world_stopped()), and those are fine. So this looks like a trap for the first two-object critical section that runs while the world is stopped, not like something that can be hit now.

A test that shows it

For Modules/_testinternalcapi/test_critical_sections.c (free-threaded build only); the locked second mutex stands for one that was handed off:

static PyObject *
test_critical_section2_stw_first_lock(PyObject *self, PyObject *Py_UNUSED(args))
{
    // gh-144513 made the slow paths skip locking while the world is stopped.
    // The inline fast path of a two-object critical section has already
    // locked the first mutex when it calls the slow path for the second one.
    PyObject *d1 = PyDict_New();
    PyObject *d2 = PyDict_New();
    if (d1 == NULL || d2 == NULL) {
        Py_XDECREF(d1);
        Py_XDECREF(d2);
        return NULL;
    }
    PyObject *lo = (uintptr_t)d1 < (uintptr_t)d2 ? d1 : d2;   // locked first
    PyObject *hi = (lo == d1) ? d2 : d1;

    // Stands for a mutex that was handed off to a thread which is suspended.
    PyMutex_Lock(&hi->ob_mutex);

    PyInterpreterState *interp = PyInterpreterState_Get();
    _PyEval_StopTheWorld(interp);
    Py_BEGIN_CRITICAL_SECTION2(lo, hi);
    Py_END_CRITICAL_SECTION2();
    _PyEval_StartTheWorld(interp);

    // The section has ended: its first mutex must not be locked any more.
    assert(!PyMutex_IsLocked(&lo->ob_mutex));

    PyMutex_Unlock(&hi->ob_mutex);
    Py_DECREF(d1);
    Py_DECREF(d2);
    Py_RETURN_NONE;
}

python -m test test_capi -m test_critical_section2_stw_first_lock, main at e8c0584 (2026-11-10), free-threaded:

Linux arm64 macOS arm64
as it is aborts on the assertion aborts on the assertion
with the change below passes passes

A possible change

diff --git a/Python/critical_section.c b/Python/critical_section.c
index dbee6f236a..15cf315224 100644
--- a/Python/critical_section.c
+++ b/Python/critical_section.c
@@ -67,6 +67,10 @@ _PyCriticalSection2_BeginSlow(PyThreadState *tstate, PyCriticalSection2 *c, PyMu
 {
 #ifdef Py_GIL_DISABLED
     if (tstate->interp->stoptheworld.world_stopped) {
+        if (is_m1_locked) {
+            // The inline fast path locked m1 for this section, which we skip.
+            PyMutex_Unlock(m1);
+        }
         c->_cs_base._cs_mutex = NULL;
         c->_cs_mutex2 = NULL;
         c->_cs_base._cs_prev = 0;

This is the only place that needs it: _PyCriticalSection_BeginSlow() is never called with a mutex already locked, and the recursion shortcut in _PyCriticalSection2_BeginSlow() can't coincide with is_m1_locked, since the fast path's PyMutex_LockFast(m1) fails on a mutex the thread already holds.

With the change the full test suite passes on macOS (free-threaded, today's main), and test_capi, test_free_threading, test_dict, test_set, test_list, test_gc and test_threading pass on Linux.

I ran main only. The same lines are in 3.15 and in 3.14 since 3.14.4, where the fix was backported.

I'll open a PR with the test and the change.

cc @colesbury

CPython versions tested on:

3.14, 3.15, CPython main branch

Operating systems tested on:

macOS, Linux

Linked PRs

Activity

  1. colesbury commented on Nov 16, 2026

    @colesbury
    Contributor

    I don't see how this scenario can happen. I think the "stand-in" may make it seem possible when it isn't. How does a lock get handed off when all other threads are paused?

  2. maikschulze commented on Nov 16, 2026

    @maikschulze
    ContributorAuthor

    Thanks for looking. The handoff happens just before the stop: thread B parks waiting for a mutex (detached), thread A unlocks it after more than 1 ms and mutex_unpark() hands it to B, and thread C stops the world before B has run. B is suspended without being waited for and owns the mutex during the whole stop-the-world. The test's locked mutex only stands in for this, since the real sequence is a race.

    One place where this shows is os.fork(): the stop-the-world in PyOS_BeforeFork() doesn't wait for B, so the child inherits the mutex locked, with no owner. I ran the script below as python fork_lock.py 15, 8 copies at once (main at e8c0584, free-threaded release build, Linux). In every copy, a child hung. Then I switched the handoff off (int should_be_fair = 0; in mutex_unpark()), and no copy hung.

    fork_lock.py
    import os, sys, threading, time
    warnings = __import__("warnings"); warnings.simplefilter("ignore")
    s = set()
    stop = time.monotonic() + float(sys.argv[1])
    
    def hammer():
        while time.monotonic() < stop:
            s.add(1)
            s.discard(1)
    
    threads = [threading.Thread(target=hammer) for _ in range(8)]
    for t in threads:
        t.start()
    forks = 0
    while time.monotonic() < stop:
        pid = os.fork()
        if pid == 0:
            s.copy()
            os._exit(0)
        deadline = time.monotonic() + 20
        while os.waitpid(pid, os.WNOHANG)[0] == 0:
            if time.monotonic() > deadline:
                print(f"child {pid} is still running after {forks} forks", flush=True)
                os._exit(3)
            time.sleep(0.001)
        forks += 1
    for t in threads:
        t.join()
    print("no hang in", forks, "forks")
  3. colesbury commented on Nov 16, 2026

    @colesbury
    Contributor

    Ok, I think I understand the original issue now. Regarding the patch, I'd prefer we fill in the critical section struct to indicate the m1 is locked rather than unlock it there.

    The fork thing is a different bug - yes, lock handoff can cause locks to be held across fork. This is an issue even without critical sections. PyMutex_Lock(&mutex) can be given the lock via hand-off while detached, allowing another thread to fork() -- effectively killing the lock owner. I don't think the patch would fix it. The "unlocking" before fork has generally been best-effort.

  4. added a commit that references this issue on Nov 16, 2026
  5. maikschulze commented on Nov 16, 2026

    @maikschulze
    ContributorAuthor

    Thanks, done in b041c7d on the PR (#162088): when the world is stopped and the fast path has already locked the first mutex, _PyCriticalSection2_BeginSlow() now records it in the critical section, so _PyCriticalSection2_End() unlocks it. The new test and the full test suite pass (macOS, free-threaded build).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions