Repository navigation
[Windows ARM64] Tie EngineCore lifetime to its parent process (parent sentinel watchdog + Windows job object) - #5
Draft
arif-ahmed-nv wants to merge 2 commits into
Conversation
arif-ahmed-nv
force-pushed
the
contrib/v029-enginecore-parent-lifetime
branch
from
September 16, 2026 19:56
b385085 to
0baa5c0
Compare
vortex-captain
force-pushed
the
v029_win_arm64
branch
from
September 17, 2026 06:56
0b423e9 to
ec73180
Compare
… sentinel watchdog + Windows job object) When the API server / launcher process is killed hard (TerminateProcess, closed console, taskkill without /T, OOM-kill), the EngineCore child keeps running: the only cleanup path is the parent's weakref.finalize -> kill_process_tree, which never runs in that case. The orphan keeps its GPU memory, ZMQ sockets and, on Windows, cannot be found by port. Observed repeatedly on an RTX Spark N1X (Windows 11 ARM64) where an orphaned EngineCore held 20 GiB and made the next server fail its startup memory check. Two mechanisms, both without new dependencies: 1. watch_parent_process(): a daemon thread in the EngineCore waits on multiprocessing.parent_process().sentinel and, when the parent exits, requests the normal shutdown (same path as SIGTERM) and hard-exits after shutdown_timeout + 10 s if the busy loop does not return. Cross-platform; covers spawn/forkserver where PR_SET_PDEATHSIG is unavailable or unset. 2. On Windows, CoreEngineProcManager assigns each EngineCore to a Job Object with JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE right after proc.start(); the job handle lives as long as the parent, so the OS terminates the child and its descendants when the parent dies for any reason. Failure to create/assign the job (nested-job restrictions) is logged and ignored. Verified on Windows 11 ARM64 / CPython 3.13 with a standalone reproduction: without the watcher the child survives the parent's TerminateProcess; with it the child exits within 100 ms; a job-bound child terminates the moment the job handle closes. Ported from the upstream-main patch to v029_win_arm64: import-context hunks re-applied by hand; logic hunks unchanged. Signed-off-by: Arif Ahmed <arahmed@nvidia.com> (cherry picked from commit 0baa5c05e72e86d9a3e69972860e3b7ecb6d19a2)
arif-ahmed-nv
force-pushed
the
contrib/v029-enginecore-parent-lifetime
branch
from
September 17, 2026 22:08
0baa5c0 to
651fbe4
Compare
CreateJobObjectW and OpenProcess return a HANDLE, which is pointer-sized; with the ctypes default return type (c_int) a 64-bit handle would be truncated and the AssignProcessToJobObject call would silently fail. Load kernel32 with use_last_error=True so the logged error codes are this call's GetLastError rather than a stale value, and declare restype/argtypes for the five calls. Signed-off-by: Arif Ahmed <arahmed@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
When the API server or launcher process is killed hard (TerminateProcess, closed console window,
taskkillwithout/T, OOM kill), the EngineCore child keeps running. The only cleanup path today is the parent'sweakref.finalize -> shutdown()inCoreEngineProcManager, which never runs in that case. The orphan keeps its GPU memory and ZMQ sockets and, on Windows, cannot be found by port.Observed repeatedly while serving Cosmos3 on an RTX Spark N1X (Windows 11 ARM64): an orphaned EngineCore held ~20 GiB and made the next server fail vLLM's startup memory check; a later orphan kept port-less GPU state that had to be found by command line.
This closes a gap one level above an existing mechanism:
multiproc_executor.pyalready gives every worker a death pipe so it notices when the EngineCore exits, and the EngineCore already watches worker sentinels withmultiprocessing.connection.wait. The EngineCore itself has no equivalent for its own parent.Two layers, no new dependencies:
watch_parent_process()invllm/utils/system_utils.py). A daemon thread in the EngineCore waits onmultiprocessing.parent_process().sentineland, when the parent exits, requests the normal shutdown (same path as SIGTERM:shutdown_state = REQUESTED, wake the busy loop), so the engine drains or aborts pershutdown_timeoutexactly as it would on SIGTERM. If the busy loop has not returned aftershutdown_timeout + 10 sit callsos._exit(1); this skips Python finalizers by design, since the parent is gone and the alternative is a process holding the GPU indefinitely. Standard library only; works for spawn, forkserver and fork children on every platform, so it also covers Linux/macOS spawn children wherePR_SET_PDEATHSIGis unavailable or unset. ReturnsNoneand does nothing when the caller is not a multiprocessing child.assign_to_kill_on_close_job()). On Windows, right afterproc.start(),CoreEngineProcManagerputs the EngineCore into a Job Object flaggedJOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE. The job handle lives as long as the parent; when the parent dies for any reason the OS closes it and terminates the child and its descendants (the multiproc executor workers inherit job membership). Failure to create or assign the job (nested-job restrictions on some hosts) is logged and ignored, so the caller keeps working without the guarantee. The second commit declares the Win32 signatures (HANDLE-sized return types,use_last_error=True) so a 64-bit handle is never truncated and the logged error codes are real.Both layers are opt-out-free; layer 1 adds one daemon thread per EngineCore. If you would rather have layer 2 behind an env flag, happy to add one. The
time.sleepin the watchdog callback is intentional (no event loop on that thread).Follows #3 (native shutdown primitives), which covers the cooperative shutdown path; this PR covers the non-cooperative one.
Scope and known limits
run_busy_loop(). A parent killed during weight loading (30–140 s on the N1X) still orphans the engine on Linux/macOS; on Windows the job object covers that window because it is assigned immediately afterproc.start(). A follow-up could register a minimal watchdog before construction whose only action isos._exit.Test Plan
verify_fixes.py, CPython 3.13 ARM64, Windows 11 ARM64, RTX Spark N1X): spawn a child that waits on the parent sentinel, TerminateProcess the parent, measure how long the child survives; same with the job object.v029_win_arm64(ec73180); the three files compile;ruff checkandruff format --checkintroduce no findings beyond those already present on the base branch for these files;mypy(3.10 and 3.13 hooks) passes. Re-run after the signature commit.vllm serveon a v0.29 Windows ARM64 build with the patch (our validated wheel is on the v0.26 line). Will add once the v0.29 wheel is built here.Test Result
Server logs showing the orphan and the resulting startup-memory failure on the v0.26 wheel are in the week-1/2 reports referenced in the first commit message.