This repository was archived by the owner on Jul 17, 2026. It is now read-only.
fix(pd-lm): SIGPIPE race in node-setup driver check kills 16-node jobs#984
Open
claude-spd1 wants to merge 1 commit into
Open
fix(pd-lm): SIGPIPE race in node-setup driver check kills 16-node jobs#984claude-spd1 wants to merge 1 commit into
claude-spd1 wants to merge 1 commit into
Conversation
The per-node workspace setup runs under `set -euo pipefail` and gated the CUDA extra on `nvidia-smi --query-gpu=driver_version ... | head -n 1`. On an 8-GPU node nvidia-smi prints 8 lines; head exits after the first, and when nvidia-smi loses the race it dies of SIGPIPE (exit 141), pipefail fails the pipeline, and set -e kills the whole srun task — cancelling all 16 nodes seconds into the job. Observed on jobs 172348, 172349 (Jul 8) and 174502, 174513 (Jul 13); roughly a coin flip per 16-node launch. Query only GPU 0 instead: single output line, nothing truncates the pipe. Verified on job 174520 (16 nodes, all through setup, run completed). Crew-Address: task/profile-llama-32b-cw-east-artifacts Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Collaborator
Author
|
Independent confirmation from task Crew-Address: task/workspace-fetch-sigpipe-flake |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The per-node workspace setup runs under
set -euo pipefailand gated the CUDA extra onnvidia-smi --query-gpu=driver_version ... | head -n 1 | cut .... On an 8-GPU node nvidia-smi prints 8 lines;head -n 1exits after the first, and when nvidia-smi loses the race it dies of SIGPIPE (exit 141) → pipefail fails the pipeline →set -ekills the srun task → SLURM cancels all 16 nodes seconds into the job.Observed failures with this exact signature: jobs 172348 and 172349 (dan.braun, Jul 8) and 174502 and 174513 (pd-user, Jul 13) — roughly a coin flip per 16-node launch, always written off as "node flakiness, retry".
Fix: query only GPU 0 (
nvidia-smi --id=0) — single output line, nothing truncates the pipe, no behavior change otherwise.Verified: job 174520 (16 nodes, memprobe for run p-077fe120) launched with the patched setup script; all 16 nodes passed the driver check and the run completed.
Context: task:profile-llama-32b-cw-east-artifacts
Crew-Address: task/profile-llama-32b-cw-east-artifacts
🤖 Generated with Claude Code