Repository navigation
Prevent ProcessPrefetchDatasetIterator exit hang after set_state() is called. - #1403
Merged
Merged
Conversation
copybara-service
Bot
force-pushed
the
test_987025332
branch
3 times, most recently
from
September 30, 2026 02:00
5c77a90 to
bd9eff4
Compare
… called. Why: When `ProcessPrefetchDatasetIterator.set_state()` is called (e.g. during checkpoint restoration), `self._set_state_queue.put()` starts the `multiprocessing.Queue` background feeder thread, which registers `Queue._finalize_close` at `exitpriority=10`. Because `ProcessPrefetchDatasetIterator` previously registered `_close_if_alive` via `multiprocessing.util.Finalize` at `exitpriority=1`, Python interpreter shutdown ran `Queue._finalize_close` before `_close_if_alive`. When `_close_if_alive` subsequently called `_stop_prefetch()` -> `_clear_set_state_queue()`, `q.get_nowait()` raised `OSError: handle is closed` on the already-closed queue. Because `_clear_queue_and_maybe_unlink_shm` only caught `queue.Empty` and `_stop_prefetch()` did not guard process termination with `try...finally`, the `OSError` aborted `_stop_prefetch()` before reaching `self._prefetch_process.join(timeout=_PROCESS_KILL_TIMEOUT_S)` and `self._prefetch_process.kill()`. Python's `multiprocessing.util._exit_function()` then fell back to sending `SIGTERM` and calling `p.join()` with no timeout on `active_children()`, causing the parent process to hang indefinitely at exit if the worker process did not exit on `SIGTERM`. What: 1. Increase the `exitpriority` of `_close_if_alive` in `ProcessPrefetchDatasetIterator.__init__()` from `1` to `20` so it runs before `multiprocessing.Queue`'s `exitpriority=10` finalizer closes `_set_state_queue` and `_buffer`. 2. Catch `(queue.Empty, ValueError, OSError, EOFError)` around `q.get_nowait()` in `_clear_queue_and_maybe_unlink_shm()` so already-closed or broken queues do not raise during shutdown cleanup. 3. Wrap `_unregister_fn()`, `_prefetch_should_stop.set()`, and queue draining in `_stop_prefetch()` in a `try...finally` block so `self._prefetch_process.join(timeout=_PROCESS_KILL_TIMEOUT_S)` and `self._prefetch_process.kill()` are guaranteed to run even if cleanup raises an exception, and call `self._prefetch_process.join(timeout=_PROCESS_KILL_TIMEOUT_S)` after `.kill()` to reap the child process and remove it from `multiprocessing.active_children()`. 4. Add `test_worker_process_killed_during_queue_put_after_set_state` in `process_prefetch_test.py` to cover interpreter shutdown after `set_state()` when the worker is blocked and ignores `SIGTERM`. PiperOrigin-RevId: 990673775
copybara-service
Bot
force-pushed
the
test_987025332
branch
from
September 30, 2026 02:26
bd9eff4 to
4420a1b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prevent ProcessPrefetchDatasetIterator exit hang after set_state() is called.
Why:
When
ProcessPrefetchDatasetIterator.set_state()is called (e.g. during checkpoint restoration),self._set_state_queue.put()starts themultiprocessing.Queuebackground feeder thread, which registersQueue._finalize_closeatexitpriority=10. BecauseProcessPrefetchDatasetIteratorpreviously registered_close_if_aliveviamultiprocessing.util.Finalizeatexitpriority=1, Python interpreter shutdown ranQueue._finalize_closebefore_close_if_alive.When
_close_if_alivesubsequently called_stop_prefetch()->_clear_set_state_queue(),q.get_nowait()raisedOSError: handle is closedon the already-closed queue. Because_clear_queue_and_maybe_unlink_shmonly caughtqueue.Emptyand_stop_prefetch()did not guard process termination withtry...finally, theOSErroraborted_stop_prefetch()before reachingself._prefetch_process.join(timeout=_PROCESS_KILL_TIMEOUT_S)andself._prefetch_process.kill(). Python'smultiprocessing.util._exit_function()then fell back to sendingSIGTERMand callingp.join()with no timeout onactive_children(), causing the parent process to hang indefinitely at exit if the worker process did not exit onSIGTERM.What:
exitpriorityof_close_if_aliveinProcessPrefetchDatasetIterator.__init__()from1to20so it runs beforemultiprocessing.Queue'sexitpriority=10finalizer closes_set_state_queueand_buffer.(queue.Empty, ValueError, OSError, EOFError)aroundq.get_nowait()in_clear_queue_and_maybe_unlink_shm()so already-closed or broken queues do not raise during shutdown cleanup._unregister_fn(),_prefetch_should_stop.set(), and queue draining in_stop_prefetch()in atry...finallyblock soself._prefetch_process.join(timeout=_PROCESS_KILL_TIMEOUT_S)andself._prefetch_process.kill()are guaranteed to run even if cleanup raises an exception, and callself._prefetch_process.join(timeout=_PROCESS_KILL_TIMEOUT_S)after.kill()to reap the child process and remove it frommultiprocessing.active_children().test_worker_process_killed_during_queue_put_after_set_stateinprocess_prefetch_test.pyto cover interpreter shutdown afterset_state()when the worker is blocked and ignoresSIGTERM.