Skip to content

[3.15] gh-119592: gh-152967: Fix ProcessPoolExecutor stranding submitted work when a max_tasks_per_child worker exits (GH-152978)#153363

Merged
gpshead merged 1 commit into
python:3.15from
miss-islington:backport-0c6422f-3.15
Jul 8, 2026
Merged

[3.15] gh-119592: gh-152967: Fix ProcessPoolExecutor stranding submitted work when a max_tasks_per_child worker exits (GH-152978)#153363
gpshead merged 1 commit into
python:3.15from
miss-islington:backport-0c6422f-3.15

Conversation

@miss-islington

Copy link
Copy Markdown
Contributor

gh-119592: Fix ProcessPoolExecutor stranding submitted work when a max_tasks_per_child worker exits

Worker replacement went through the executor object: the manager thread
read executor attributes that shutdown(wait=False) clears concurrently,
and could not replace workers at all once the executor was garbage
collected. A worker exiting at its max_tasks_per_child limit in those
states left the remaining submitted work permanently unexecuted and hung
interpreter exit; the racing case could crash the manager thread.

Replace workers from the executor manager thread using its own state
plus configuration read through the live executor weakref, which
shutdown() never clears:

Co-authored-by: Gregory P. Smith 68491+gpshead@users.noreply.github.com
Co-authored-by: Claude Fable 5 noreply@anthropic.com
Reviewed-multiple-times-by: Gregory P. Smith

…ubmitted work when a max_tasks_per_child worker exits (pythonGH-152978)

pythongh-119592: Fix ProcessPoolExecutor stranding submitted work when a max_tasks_per_child worker exits

Worker replacement went through the executor object: the manager thread
read executor attributes that shutdown(wait=False) clears concurrently,
and could not replace workers at all once the executor was garbage
collected. A worker exiting at its max_tasks_per_child limit in those
states left the remaining submitted work permanently unexecuted and hung
interpreter exit; the racing case could crash the manager thread.

Replace workers from the executor manager thread using its own state
plus configuration read through the live executor weakref, which
shutdown() never clears:

- After shutdown(wait=False) with the executor still referenced, a
  replacement is spawned and the remaining work is executed as
  documented.
- Once the executor has been garbage collected (pythongh-152967), or a
  replacement worker cannot be started and no workers remain, the
  remaining futures now fail with BrokenProcessPool instead of never
  resolving.
- A new _force_shutting_down flag stops both spawn paths from starting
  workers that would escape terminate_workers()/kill_workers().
(cherry picked from commit 0c6422f)

Co-authored-by: Gregory P. Smith <68491+gpshead@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Reviewed-multiple-times-by: Gregory P. Smith
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants