Skip to content

[Windows ARM64] Tie EngineCore lifetime to its parent process (parent sentinel watchdog + Windows job object) - #5

Draft
arif-ahmed-nv wants to merge 2 commits into
vortex-captain:v029_win_arm64from
arif-ahmed-nv:contrib/v029-enginecore-parent-lifetime
Draft

arif-ahmed-nv wants to merge 2 commits into
vortex-captain:v029_win_arm64from
arif-ahmed-nv:contrib/v029-enginecore-parent-lifetime

Conversation

@arif-ahmed-nv

@arif-ahmed-nv arif-ahmed-nv commented Sep 16, 2026 •

Copy link
Copy Markdown

Purpose

When the API server or launcher process is killed hard (TerminateProcess, closed console window, taskkill without /T, OOM kill), the EngineCore child keeps running. The only cleanup path today is the parent's weakref.finalize -> shutdown() in CoreEngineProcManager, which never runs in that case. The orphan keeps its GPU memory and ZMQ sockets and, on Windows, cannot be found by port.

Observed repeatedly while serving Cosmos3 on an RTX Spark N1X (Windows 11 ARM64): an orphaned EngineCore held ~20 GiB and made the next server fail vLLM's startup memory check; a later orphan kept port-less GPU state that had to be found by command line.

This closes a gap one level above an existing mechanism: multiproc_executor.py already gives every worker a death pipe so it notices when the EngineCore exits, and the EngineCore already watches worker sentinels with multiprocessing.connection.wait. The EngineCore itself has no equivalent for its own parent.

Two layers, no new dependencies:

  1. Parent-sentinel watchdog (watch_parent_process() in vllm/utils/system_utils.py). A daemon thread in the EngineCore waits on multiprocessing.parent_process().sentinel and, when the parent exits, requests the normal shutdown (same path as SIGTERM: shutdown_state = REQUESTED, wake the busy loop), so the engine drains or aborts per shutdown_timeout exactly as it would on SIGTERM. If the busy loop has not returned after shutdown_timeout + 10 s it calls os._exit(1); this skips Python finalizers by design, since the parent is gone and the alternative is a process holding the GPU indefinitely. Standard library only; works for spawn, forkserver and fork children on every platform, so it also covers Linux/macOS spawn children where PR_SET_PDEATHSIG is unavailable or unset. Returns None and does nothing when the caller is not a multiprocessing child.
  2. Windows Job Object (assign_to_kill_on_close_job()). On Windows, right after proc.start(), CoreEngineProcManager puts the EngineCore into a Job Object flagged JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE. The job handle lives as long as the parent; when the parent dies for any reason the OS closes it and terminates the child and its descendants (the multiproc executor workers inherit job membership). Failure to create or assign the job (nested-job restrictions on some hosts) is logged and ignored, so the caller keeps working without the guarantee. The second commit declares the Win32 signatures (HANDLE-sized return types, use_last_error=True) so a 64-bit handle is never truncated and the logged error codes are real.

Both layers are opt-out-free; layer 1 adds one daemon thread per EngineCore. If you would rather have layer 2 behind an env flag, happy to add one. The time.sleep in the watchdog callback is intentional (no event loop on that thread).

Follows #3 (native shutdown primitives), which covers the cooperative shutdown path; this PR covers the non-cooperative one.

Scope and known limits

  • Load-time window. The watchdog is registered after the model has loaded, right before run_busy_loop(). A parent killed during weight loading (30–140 s on the N1X) still orphans the engine on Linux/macOS; on Windows the job object covers that window because it is assigned immediately after proc.start(). A follow-up could register a minimal watchdog before construction whose only action is os._exit.
  • Assumption to confirm. The watchdog treats "parent exited" as "shut down." I found no launch mode where the launcher process is meant to exit while its engines keep serving (headless and data-parallel launchers keep their launcher alive as the parent), but if such a mode exists the watchdog should be gated on it. Reviewer input welcome.

Test Plan

  • Standalone reproduction without importing vLLM (verify_fixes.py, CPython 3.13 ARM64, Windows 11 ARM64, RTX Spark N1X): spawn a child that waits on the parent sentinel, TerminateProcess the parent, measure how long the child survives; same with the job object.
  • Patch rebased onto v029_win_arm64 (ec73180); the three files compile; ruff check and ruff format --check introduce no findings beyond those already present on the base branch for these files; mypy (3.10 and 3.13 hooks) passes. Re-run after the signature commit.
  • Not yet run: an end-to-end vllm serve on a v0.29 Windows ARM64 build with the patch (our validated wheel is on the v0.26 line). Will add once the v0.29 wheel is built here.

Test Result

1. parent sentinel, no watcher  : child 14840 SURVIVED 5 s after the parent was killed (orphan) - cleaned up
1. parent sentinel, with watcher: child 8576 exited 0.0s after the parent was killed
2. job object                   : child 3444 terminated 0.0s after the job handle closed

Server logs showing the orphan and the resulting startup-memory failure on the v0.26 wheel are in the week-1/2 reports referenced in the first commit message.

@arif-ahmed-nv
arif-ahmed-nv force-pushed the contrib/v029-enginecore-parent-lifetime branch from b385085 to 0baa5c0 Compare September 16, 2026 19:56
… sentinel watchdog + Windows job object)

When the API server / launcher process is killed hard (TerminateProcess, closed console, taskkill without /T,
OOM-kill), the EngineCore child keeps running: the only cleanup path is the parent's weakref.finalize ->
kill_process_tree, which never runs in that case. The orphan keeps its GPU memory, ZMQ sockets and, on
Windows, cannot be found by port. Observed repeatedly on an RTX Spark N1X (Windows 11 ARM64) where an orphaned
EngineCore held 20 GiB and made the next server fail its startup memory check.

Two mechanisms, both without new dependencies:
1. watch_parent_process(): a daemon thread in the EngineCore waits on
   multiprocessing.parent_process().sentinel and, when the parent exits, requests the normal shutdown
   (same path as SIGTERM) and hard-exits after shutdown_timeout + 10 s if the busy loop does not return.
   Cross-platform; covers spawn/forkserver where PR_SET_PDEATHSIG is unavailable or unset.
2. On Windows, CoreEngineProcManager assigns each EngineCore to a Job Object with
   JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE right after proc.start(); the job handle lives as long as the parent,
   so the OS terminates the child and its descendants when the parent dies for any reason. Failure to
   create/assign the job (nested-job restrictions) is logged and ignored.

Verified on Windows 11 ARM64 / CPython 3.13 with a standalone reproduction: without the watcher the child
survives the parent's TerminateProcess; with it the child exits within 100 ms; a job-bound child terminates
the moment the job handle closes.

Ported from the upstream-main patch to v029_win_arm64: import-context hunks re-applied by hand; logic hunks unchanged.

Signed-off-by: Arif Ahmed <arahmed@nvidia.com>
(cherry picked from commit 0baa5c05e72e86d9a3e69972860e3b7ecb6d19a2)
@arif-ahmed-nv
arif-ahmed-nv force-pushed the contrib/v029-enginecore-parent-lifetime branch from 0baa5c0 to 651fbe4 Compare September 17, 2026 22:08
CreateJobObjectW and OpenProcess return a HANDLE, which is pointer-sized; with the ctypes default return type (c_int) a 64-bit handle would be truncated and the AssignProcessToJobObject call would silently fail. Load kernel32 with use_last_error=True so the logged error codes are this call's GetLastError rather than a stale value, and declare restype/argtypes for the five calls.

Signed-off-by: Arif Ahmed <arahmed@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant