Repository navigation
Conversation
|
I think the race condition analysis here may be slightly off. Looking at The actual issue seems to be that when A A fix in for p in self.processes.values():
p.join(timeout=3)
if p.exitcode is not None and p.exitcode != 0:
errors.append(...)The sentinel has already fired at this point so |
|
This PR is stale because it has been open for 30 days with no activity. |
|
I was exploring peak RSS gaps of the free-threaded build with mimalloc v3 (#159718) with overnight stress tests, and this test failed in one of my full test runs on macOS. I looked into it, and my measurements support the explanation given above in this thread: the worker's exit code isn't available yet when its sentinel fires. A retry in the test can't help then, so I'd like to suggest a small change in The exit code can lag behind the sentinelWith the fork and spawn start methods,
The exit code arrived up to 8 ms later. The pool then has no cause to attach
diff --git a/Lib/concurrent/futures/process.py b/Lib/concurrent/futures/process.py
index 5d26748104..1a9e9b51ec 100644
--- a/Lib/concurrent/futures/process.py
+++ b/Lib/concurrent/futures/process.py
@@ -582,6 +582,12 @@ def _terminate_broken(self, cause, bpe_message=None):
# so include all of them in the traceback.
errors = []
for p in self.processes.values():
+ if p.exitcode is None and mp.connection.wait([p.sentinel], 0):
+ # The sentinel is ready, so the process has ended, but
+ # its exit status may not be available yet (the
+ # operating system closes the child's end of the pipe
+ # before the child can be waited for).
+ p.join(timeout=1)
if p.exitcode is not None and p.exitcode != 0:
errors.append(f"Process {p.pid} terminated abruptly "
f"with exit code {p.exitcode}")The one-second limit is arbitrary. With the change The test runs every variant with the default start methodThe test builds its executor without
diff --git a/Lib/test/test_concurrent_futures/test_process_pool.py b/Lib/test/test_concurrent_futures/test_process_pool.py
index dafbda862c..db8365074a 100644
--- a/Lib/test/test_concurrent_futures/test_process_pool.py
+++ b/Lib/test/test_concurrent_futures/test_process_pool.py
@@ -145,7 +145,8 @@ def test_traceback_when_child_process_terminates_abruptly(self):
# gh-139462 enhancement - BrokenProcessPool exceptions
# should describe which process terminated.
exit_code = 99
- with self.executor_type(max_workers=1) as executor:
+ with self.executor_type(max_workers=1,
+ mp_context=self.get_context()) as executor:
future = executor.submit(os._exit, exit_code)
with self.assertRaises(BrokenProcessPool) as bpe:
future.result()The two changes belong together: with the test change alone, the Fork variant would start to fail on Linux (the fork row of the second table). If you'd like to take these into this PR, please do; I can also open a separate one. sentinel_race.py and pool_cause.pyimport multiprocessing as mp, os, sys, time
from multiprocessing.connection import wait
def main():
ctx = mp.get_context(sys.argv[1])
n = int(sys.argv[2])
none = other = 0
late = []
for i in range(n):
p = ctx.Process(target=os._exit, args=(99,))
p.start()
wait([p.sentinel])
t0 = time.perf_counter()
code = p.exitcode # one waitpid(WNOHANG) for fork and spawn
if code is None:
none += 1
while p.exitcode is None:
pass
late.append((time.perf_counter() - t0) * 1e6)
elif code != 99:
other += 1
p.join()
worst = f", exit code arrived up to {max(late):.0f} us later" if late else ""
print(f"{sys.argv[1]}: {n} children | sentinel ready but exit code still None: {none}{worst} | other exit codes: {other}")
if __name__ == "__main__":
main()import multiprocessing as mp, os, sys
from concurrent.futures import ProcessPoolExecutor
from concurrent.futures.process import BrokenProcessPool, _RemoteTraceback
def main():
ctx = mp.get_context(sys.argv[1])
n = int(sys.argv[2])
none = wrong = 0
for i in range(n):
with ProcessPoolExecutor(max_workers=1, mp_context=ctx) as ex:
f = ex.submit(os._exit, 99)
try:
f.result()
except BrokenProcessPool as e:
cause = e.__cause__
if cause is None:
none += 1
elif not (isinstance(cause, _RemoteTraceback) and "terminated abruptly with exit code 99" in cause.tb):
wrong += 1
print(f"{sys.argv[1]}: {n} pools | BrokenProcessPool without a cause: {none} | with another cause: {wrong}")
if __name__ == "__main__":
main() |
Fixes gh-143620
There is a race condition in
test_concurrent_futures.test_process_pool. The test asserts that aBrokenProcessPoolexception has a__cause__immediately afterfuture.result()raises.In
concurrent.futures.process, the_terminate_brokenmethod sets the exception on the future before it attaches the_RemoteTracebackto the__cause__attr. Sometimes the test thread wakes up and inspects the exception while__cause__is stillNone, which leads to a test failure. I was able to repro (see issue for more details) by injecting a delay in_terminate_brokenbetween theset_exception()call and the__cause__assignment. This causes a 100% failure rate withAssertionError: None is not an instance of _RemoteTraceback.My proposed fix is to update the test to use
support.sleeping_retryto wait for the__cause__to be populated. This seems to be the pattern elsewhere in the test suite.