Skip to content

gh-143620: Fix race condition in test_traceback_when_child_process_terminates_abruptly - #143621

Open
a12k wants to merge 1 commit into
python:mainfrom
a12k:flaky-process-pool-test
Open

a12k wants to merge 1 commit into
python:mainfrom
a12k:flaky-process-pool-test

Conversation

@a12k

@a12k a12k commented Jan 9, 2026

Copy link
Copy Markdown
Contributor

Fixes gh-143620

There is a race condition in test_concurrent_futures.test_process_pool. The test asserts that a BrokenProcessPool exception has a __cause__ immediately after future.result() raises.

In concurrent.futures.process, the _terminate_broken method sets the exception on the future before it attaches the _RemoteTraceback to the __cause__ attr. Sometimes the test thread wakes up and inspects the exception while __cause__ is still None, which leads to a test failure. I was able to repro (see issue for more details) by injecting a delay in _terminate_broken between the set_exception() call and the __cause__ assignment. This causes a 100% failure rate with AssertionError: None is not an instance of _RemoteTraceback.

My proposed fix is to update the test to use support.sleeping_retry to wait for the __cause__ to be populated. This seems to be the pattern elsewhere in the test suite.

@WYSIATI

WYSIATI commented Apr 2, 2026

Copy link
Copy Markdown
Contributor

I think the race condition analysis here may be slightly off. Looking at _terminate_broken(), bpe.__cause__ is assigned on line 493 before set_exception() on line 498, so the test thread shouldn't be able to see __cause__ as None due to ordering.

The actual issue seems to be that when cause is None (the os._exit path), p.exitcode can still return None at line 487 because waitpid() hasn't collected the exit status yet — even though the process sentinel has already fired. This means errors stays empty, cause_str stays None, and bpe.__cause__ is never set at all.

A sleeping_retry in the test won't help here because nobody goes back to set __cause__ after the fact — once _terminate_broken() finishes, the exception is final.

A fix in _terminate_broken() itself might work better — calling p.join(timeout=3) before reading p.exitcode would ensure waitpid() has collected the exit status:

for p in self.processes.values():
    p.join(timeout=3)
    if p.exitcode is not None and p.exitcode != 0:
        errors.append(...)

The sentinel has already fired at this point so join() should return almost immediately. This way p.exitcode is reliably available when we read it.

@github-actions

github-actions Bot commented May 4, 2026

Copy link
Copy Markdown

This PR is stale because it has been open for 30 days with no activity.

@github-actions github-actions Bot added the stale Stale PR or inactive for long period of time. label May 4, 2026
@maikschulze

Copy link
Copy Markdown
Contributor

I was exploring peak RSS gaps of the free-threaded build with mimalloc v3 (#159718) with overnight stress tests, and this test failed in one of my full test runs on macOS. I looked into it, and my measurements support the explanation given above in this thread: the worker's exit code isn't available yet when its sentinel fires. A retry in the test can't help then, so I'd like to suggest a small change in process.py instead, plus one in the test. An AI assistant helped with the analysis.

The exit code can lag behind the sentinel

With the fork and spawn start methods, p.exitcode comes from os.waitpid(pid, os.WNOHANG), and the operating system reports the sentinel (the child's end of a pipe is closed) a little before the child can be waited for. A child calls os._exit(99), the parent waits for its sentinel and asks for the exit code once (sentinel_race.py, in the fold). Children, of 2,400, whose exit code was still None (main at e8c0584 (2026-11-10), eight copies at once):

start method Linux, free-threaded build Linux, GIL build macOS, free-threaded build macOS, GIL build
fork 49 33 8 11
spawn 6 6 24 27
forkserver 0 0 0 0

The exit code arrived up to 8 ms later.

The pool then has no cause to attach

_terminate_broken() reads p.exitcode right after the manager thread saw the worker's sentinel. If it is None, no cause is attached to the BrokenProcessPool, and nothing adds one later. The test's scenario in a loop (pool_cause.py, in the fold); pools, of 480, whose BrokenProcessPool had no cause:

start method Linux, free-threaded build Linux, GIL build macOS, free-threaded build
fork 49 60 2
spawn 2 3 13
forkserver 0 0 0
any of the three, with the change below 0 0 0
diff --git a/Lib/concurrent/futures/process.py b/Lib/concurrent/futures/process.py
index 5d26748104..1a9e9b51ec 100644
--- a/Lib/concurrent/futures/process.py
+++ b/Lib/concurrent/futures/process.py
@@ -582,6 +582,12 @@ def _terminate_broken(self, cause, bpe_message=None):
             # so include all of them in the traceback.
             errors = []
             for p in self.processes.values():
+                if p.exitcode is None and mp.connection.wait([p.sentinel], 0):
+                    # The sentinel is ready, so the process has ended, but
+                    # its exit status may not be available yet (the
+                    # operating system closes the child's end of the pipe
+                    # before the child can be waited for).
+                    p.join(timeout=1)
                 if p.exitcode is not None and p.exitcode != 0:
                     errors.append(f"Process {p.pid} terminated abruptly "
                                   f"with exit code {p.exitcode}")

The one-second limit is arbitrary. With the change test_concurrent_futures passes on macOS; I haven't tried Windows. It would also make the cause reliable for users of the pool, not only for the test.

The test runs every variant with the default start method

The test builds its executor without mp_context, so its Fork, Forkserver and Spawn variants all use the platform default: spawn on macOS, forkserver on Linux. That fits where it fails: on macOS, in all three variants, and not on Linux. The test alone on macOS, 128 runs (16 at once):

failed runs Fork variant Forkserver variant Spawn variant
as it is 28 16 7 7
with mp_context=self.get_context() 11 0 0 11
the same with the change in process.py 0 0 0 0
diff --git a/Lib/test/test_concurrent_futures/test_process_pool.py b/Lib/test/test_concurrent_futures/test_process_pool.py
index dafbda862c..db8365074a 100644
--- a/Lib/test/test_concurrent_futures/test_process_pool.py
+++ b/Lib/test/test_concurrent_futures/test_process_pool.py
@@ -145,7 +145,8 @@ def test_traceback_when_child_process_terminates_abruptly(self):
         # gh-139462 enhancement - BrokenProcessPool exceptions
         # should describe which process terminated.
         exit_code = 99
-        with self.executor_type(max_workers=1) as executor:
+        with self.executor_type(max_workers=1,
+                                mp_context=self.get_context()) as executor:
             future = executor.submit(os._exit, exit_code)
             with self.assertRaises(BrokenProcessPool) as bpe:
                 future.result()

The two changes belong together: with the test change alone, the Fork variant would start to fail on Linux (the fork row of the second table).

If you'd like to take these into this PR, please do; I can also open a separate one.

sentinel_race.py and pool_cause.py
import multiprocessing as mp, os, sys, time
from multiprocessing.connection import wait

def main():
    ctx = mp.get_context(sys.argv[1])
    n = int(sys.argv[2])
    none = other = 0
    late = []
    for i in range(n):
        p = ctx.Process(target=os._exit, args=(99,))
        p.start()
        wait([p.sentinel])
        t0 = time.perf_counter()
        code = p.exitcode              # one waitpid(WNOHANG) for fork and spawn
        if code is None:
            none += 1
            while p.exitcode is None:
                pass
            late.append((time.perf_counter() - t0) * 1e6)
        elif code != 99:
            other += 1
        p.join()
    worst = f", exit code arrived up to {max(late):.0f} us later" if late else ""
    print(f"{sys.argv[1]}: {n} children | sentinel ready but exit code still None: {none}{worst} | other exit codes: {other}")

if __name__ == "__main__":
    main()
import multiprocessing as mp, os, sys
from concurrent.futures import ProcessPoolExecutor
from concurrent.futures.process import BrokenProcessPool, _RemoteTraceback

def main():
    ctx = mp.get_context(sys.argv[1])
    n = int(sys.argv[2])
    none = wrong = 0
    for i in range(n):
        with ProcessPoolExecutor(max_workers=1, mp_context=ctx) as ex:
            f = ex.submit(os._exit, 99)
            try:
                f.result()
            except BrokenProcessPool as e:
                cause = e.__cause__
                if cause is None:
                    none += 1
                elif not (isinstance(cause, _RemoteTraceback) and "terminated abruptly with exit code 99" in cause.tb):
                    wrong += 1
    print(f"{sys.argv[1]}: {n} pools | BrokenProcessPool without a cause: {none} | with another cause: {wrong}")

if __name__ == "__main__":
    main()

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

awaiting review stale Stale PR or inactive for long period of time. tests Tests in the Lib/test dir

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Race condition in test_traceback_when_child_process_terminates_abruptly

3 participants