Repository navigation
Description
Bug report
I was exploring peak RSS gaps of the free-threaded build with mimalloc v3 (#159718) with overnight stress tests and, while looking into lock handoffs, read the fix for #144513 (#144524). One path in it looks like it leaves a mutex locked. An AI assistant helped with the analysis.
What happens
_PyCriticalSection2_BeginSlow() returns early when the interpreter's world is stopped and records no mutexes, so _PyCriticalSection2_End() unlocks nothing. But the inline fast path, _PyCriticalSection2_BeginMutex(), has already locked the first mutex when it calls the slow path for the second one (is_m1_locked == 1). That first mutex then stays locked after the section has ended, and the next critical section on that object, in any thread, waits forever.
It needs the second mutex to be busy during a stop-the-world, which is the case #144513 is about (a mutex handed off to a thread that is already suspended). I couldn't find code that gets there today: of the stop-the-world regions on main, the ones I traced take a single-object critical section at most (for example object_set_class_world_stopped()), and those are fine. So this looks like a trap for the first two-object critical section that runs while the world is stopped, not like something that can be hit now.
A test that shows it
For Modules/_testinternalcapi/test_critical_sections.c (free-threaded build only); the locked second mutex stands for one that was handed off:
static PyObject *
test_critical_section2_stw_first_lock(PyObject *self, PyObject *Py_UNUSED(args))
{
// gh-144513 made the slow paths skip locking while the world is stopped.
// The inline fast path of a two-object critical section has already
// locked the first mutex when it calls the slow path for the second one.
PyObject *d1 = PyDict_New();
PyObject *d2 = PyDict_New();
if (d1 == NULL || d2 == NULL) {
Py_XDECREF(d1);
Py_XDECREF(d2);
return NULL;
}
PyObject *lo = (uintptr_t)d1 < (uintptr_t)d2 ? d1 : d2; // locked first
PyObject *hi = (lo == d1) ? d2 : d1;
// Stands for a mutex that was handed off to a thread which is suspended.
PyMutex_Lock(&hi->ob_mutex);
PyInterpreterState *interp = PyInterpreterState_Get();
_PyEval_StopTheWorld(interp);
Py_BEGIN_CRITICAL_SECTION2(lo, hi);
Py_END_CRITICAL_SECTION2();
_PyEval_StartTheWorld(interp);
// The section has ended: its first mutex must not be locked any more.
assert(!PyMutex_IsLocked(&lo->ob_mutex));
PyMutex_Unlock(&hi->ob_mutex);
Py_DECREF(d1);
Py_DECREF(d2);
Py_RETURN_NONE;
}
python -m test test_capi -m test_critical_section2_stw_first_lock, main at e8c0584 (2026-11-10), free-threaded:
| Linux arm64 | macOS arm64 | |
|---|---|---|
| as it is | aborts on the assertion | aborts on the assertion |
| with the change below | passes | passes |
A possible change
diff --git a/Python/critical_section.c b/Python/critical_section.c
index dbee6f236a..15cf315224 100644
--- a/Python/critical_section.c
+++ b/Python/critical_section.c
@@ -67,6 +67,10 @@ _PyCriticalSection2_BeginSlow(PyThreadState *tstate, PyCriticalSection2 *c, PyMu
{
#ifdef Py_GIL_DISABLED
if (tstate->interp->stoptheworld.world_stopped) {
+ if (is_m1_locked) {
+ // The inline fast path locked m1 for this section, which we skip.
+ PyMutex_Unlock(m1);
+ }
c->_cs_base._cs_mutex = NULL;
c->_cs_mutex2 = NULL;
c->_cs_base._cs_prev = 0;
This is the only place that needs it: _PyCriticalSection_BeginSlow() is never called with a mutex already locked, and the recursion shortcut in _PyCriticalSection2_BeginSlow() can't coincide with is_m1_locked, since the fast path's PyMutex_LockFast(m1) fails on a mutex the thread already holds.
With the change the full test suite passes on macOS (free-threaded, today's main), and test_capi, test_free_threading, test_dict, test_set, test_list, test_gc and test_threading pass on Linux.
I ran main only. The same lines are in 3.15 and in 3.14 since 3.14.4, where the fix was backported.
I'll open a PR with the test and the change.
cc @colesbury
CPython versions tested on:
3.14, 3.15, CPython main branch
Operating systems tested on:
macOS, Linux
Activity
colesbury commentedon Nov 16, 2026
I don't see how this scenario can happen. I think the "stand-in" may make it seem possible when it isn't. How does a lock get handed off when all other threads are paused?
maikschulze commentedon Nov 16, 2026
Thanks for looking. The handoff happens just before the stop: thread B parks waiting for a mutex (detached), thread A unlocks it after more than 1 ms and
mutex_unpark()hands it to B, and thread C stops the world before B has run. B is suspended without being waited for and owns the mutex during the whole stop-the-world. The test's locked mutex only stands in for this, since the real sequence is a race.One place where this shows is
os.fork(): the stop-the-world inPyOS_BeforeFork()doesn't wait for B, so the child inherits the mutex locked, with no owner. I ran the script below aspython fork_lock.py 15, 8 copies at once (main at e8c0584, free-threaded release build, Linux). In every copy, a child hung. Then I switched the handoff off (int should_be_fair = 0;inmutex_unpark()), and no copy hung.fork_lock.py
colesbury commentedon Nov 16, 2026
Ok, I think I understand the original issue now. Regarding the patch, I'd prefer we fill in the critical section struct to indicate the m1 is locked rather than unlock it there.
The fork thing is a different bug - yes, lock handoff can cause locks to be held across fork. This is an issue even without critical sections.
PyMutex_Lock(&mutex)can be given the lock via hand-off while detached, allowing another thread to fork() -- effectively killing the lock owner. I don't think the patch would fix it. The "unlocking" before fork has generally been best-effort.pythongh-162087: Record the first mutex instead of unlocking it early
maikschulze commentedon Nov 16, 2026
Thanks, done in b041c7d on the PR (#162088): when the world is stopped and the fast path has already locked the first mutex,
_PyCriticalSection2_BeginSlow()now records it in the critical section, so_PyCriticalSection2_End()unlocks it. The new test and the full test suite pass (macOS, free-threaded build).