Skip to content
This repository was archived by the owner on Jul 17, 2026. It is now read-only.

fix(pd-lm): SIGPIPE race in node-setup driver check kills 16-node jobs#984

Open
claude-spd1 wants to merge 1 commit into
feature/jaxfrom
bridge/task-profile-llama-32b-cw-east-artifacts
Open

fix(pd-lm): SIGPIPE race in node-setup driver check kills 16-node jobs#984
claude-spd1 wants to merge 1 commit into
feature/jaxfrom
bridge/task-profile-llama-32b-cw-east-artifacts

Conversation

@claude-spd1

Copy link
Copy Markdown
Collaborator

The per-node workspace setup runs under set -euo pipefail and gated the CUDA extra on nvidia-smi --query-gpu=driver_version ... | head -n 1 | cut .... On an 8-GPU node nvidia-smi prints 8 lines; head -n 1 exits after the first, and when nvidia-smi loses the race it dies of SIGPIPE (exit 141) → pipefail fails the pipeline → set -e kills the srun task → SLURM cancels all 16 nodes seconds into the job.

Observed failures with this exact signature: jobs 172348 and 172349 (dan.braun, Jul 8) and 174502 and 174513 (pd-user, Jul 13) — roughly a coin flip per 16-node launch, always written off as "node flakiness, retry".

Fix: query only GPU 0 (nvidia-smi --id=0) — single output line, nothing truncates the pipe, no behavior change otherwise.

Verified: job 174520 (16 nodes, memprobe for run p-077fe120) launched with the patched setup script; all 16 nodes passed the driver check and the run completed.

Context: task:profile-llama-32b-cw-east-artifacts

Crew-Address: task/profile-llama-32b-cw-east-artifacts

🤖 Generated with Claude Code

The per-node workspace setup runs under `set -euo pipefail` and gated the
CUDA extra on `nvidia-smi --query-gpu=driver_version ... | head -n 1`. On an
8-GPU node nvidia-smi prints 8 lines; head exits after the first, and when
nvidia-smi loses the race it dies of SIGPIPE (exit 141), pipefail fails the
pipeline, and set -e kills the whole srun task — cancelling all 16 nodes
seconds into the job. Observed on jobs 172348, 172349 (Jul 8) and 174502,
174513 (Jul 13); roughly a coin flip per 16-node launch.

Query only GPU 0 instead: single output line, nothing truncates the pipe.
Verified on job 174520 (16 nodes, all through setup, run completed).

Crew-Address: task/profile-llama-32b-cw-east-artifacts

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@claude-spd1

Copy link
Copy Markdown
Collaborator Author

Independent confirmation from task workspace-fetch-sigpipe-flake (the same flake reported as ~1-in-6 on pd-lm launches): on an and-gmi 8×H200 node (driver 570.133.20) I ran the exact unpatched pipeline from _node_workspace_setup 300 times under set -euo pipefail: 57/300 (19%) exited 141. The patched variant from this PR (--id=0): 0/300. That per-invocation rate compounds per node, which explains both the ~coin-flip 16-node failures and the ~1-in-6 smaller launches.

Crew-Address: task/workspace-fetch-sigpipe-flake

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant