Skip to content

[Core][runtime_env] Avoid zombie subprocesses in check_output_cmd - #65611

Open
d3c3mber wants to merge 3 commits into
ray-project:masterfrom
d3c3mber:fix/runtime-env-zombie-subprocess
Open

[Core][runtime_env] Avoid zombie subprocesses in check_output_cmd#65611
d3c3mber wants to merge 3 commits into
ray-project:masterfrom
d3c3mber:fix/runtime-env-zombie-subprocess

Conversation

@d3c3mber

@d3c3mber d3c3mber commented Aug 20, 2026

Copy link
Copy Markdown
Clipboard_Screenshot_1787210906

When PipPlugin.delete_uri cancels the outer task, the pending CancelledError caused 'await proc.wait()' to return immediately without reaping the child, leaving a process that got re-parented to init.

Wrap the wait in asyncio.shield + a bounded timeout, and fall back to transport.close() so the asyncio child watcher can reap the child in the background.

Thank you for contributing to Ray! 🚀
Please review the Ray Contribution Guide before opening a pull request.

⚠️ Remove these instructions before submitting your PR.

💡 Tip: Mark as draft if you want early feedback, or ready for review when it's complete.

Description

Briefly describe what this PR accomplishes and why it's needed.

Related issues

Link related issues: "Fixes #1234", "Closes #1234", or "Related to #1234".

Additional information

Optional: Add implementation details, API changes, usage examples, screenshots, etc.

@d3c3mber
d3c3mber requested a review from a team as a code owner August 20, 2026 07:19

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request addresses a zombie process leak in check_output_cmd by wrapping the process wait call in asyncio.shield and asyncio.wait_for during cleanup, and adds corresponding unit tests. The review feedback suggests two important improvements: first, ensuring that any caught asyncio.CancelledError is re-raised to prevent swallowing task cancellation; second, checking the specific spawned process PID in the zombie test instead of using a wildcard os.waitpid(-1, ...) to avoid flakiness in concurrent test environments.

Comment thread python/ray/_private/runtime_env/utils.py Outdated
Comment thread python/ray/tests/unit/test_runtime_env.py
When PipPlugin.delete_uri cancels the outer task, the pending
CancelledError caused 'await proc.wait()' to return immediately
without reaping the child, leaving a <defunct> process that got
re-parented to init.

Wrap the wait in asyncio.shield + a bounded timeout, and fall back
to transport.close() so the asyncio child watcher can reap the
child in the background.

Signed-off-by: botanggl <d3c3mber@users.noreply.github.com>
@d3c3mber
d3c3mber force-pushed the fix/runtime-env-zombie-subprocess branch from 7927d50 to 1cd664d Compare August 20, 2026 07:25
Signed-off-by: botanggl <d3c3mber@users.noreply.github.com>
@d3c3mber
d3c3mber force-pushed the fix/runtime-env-zombie-subprocess branch from 2cde793 to f26f1f0 Compare August 20, 2026 07:42
@ray-gardener ray-gardener Bot added core Issues that should be addressed in Ray Core community-contribution Contributed by the community labels Aug 20, 2026
Signed-off-by: d3c3mber <d3c3mber@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution Contributed by the community core Issues that should be addressed in Ray Core

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ray fails to serialize self-reference objects

1 participant