Skip to content
This repository was archived by the owner on Jul 17, 2026. It is now read-only.

fix(pd-lm): SIGPIPE-proof the driver-version detection in the node-setup script#990

Closed
claude-spd1 wants to merge 1 commit into
feature/jaxfrom
fix/launcher-driver-detect-sigpipe
Closed

fix(pd-lm): SIGPIPE-proof the driver-version detection in the node-setup script#990
claude-spd1 wants to merge 1 commit into
feature/jaxfrom
fix/launcher-driver-detect-sigpipe

Conversation

@claude-spd1

Copy link
Copy Markdown
Collaborator

Description

The generated sbatch node-setup script detects the CUDA extra via
nvidia-smi --query-gpu=driver_version --format=csv,noheader | head -n 1 | cut -d. -f1.
nvidia-smi writes one line per GPU; head -n 1 closes the pipe after the first line, and under the script's set -o pipefail a late write from nvidia-smi becomes SIGPIPE → exit 141, killing the whole srun step before the workspace even builds. It's a per-node race — observed three times across 1- and 4-node submissions (jobs 1050555, 1050877's requeue, 1090526), and with N nodes per job the odds multiply.

Fix: query GPU 0 alone (nvidia-smi --id=0 ...) — cut consumes the full stream, no mid-stream pipe close, same DRIVER_MAJOR.

Motivation and Context

Flaky launch failures burn queue time (an opportunistic 4-node allocation dies in 5s and goes back to the end of the line) and look like real breakage.

How Has This Been Tested?

Patched the submitting checkout and resubmitted the same 4-node probes; generated scripts carry the new line. nvidia-smi --id=0 --query-gpu=driver_version --format=csv,noheader verified to emit exactly one line on an 8-GPU node.

Does this PR introduce a breaking change?

No — submit-side script generation only.

🤖 Generated with Claude Code

Crew-Address: task/scan-read-activations-taps

…tup script

nvidia-smi --query-gpu writes one line per GPU; '| head -n 1' closes the pipe
after the first, and under 'set -o pipefail' a late write from nvidia-smi turns
into SIGPIPE -> exit 141, killing the whole srun step before the workspace even
builds. Racy per node: observed three times across 1- and 4-node submissions
(jobs 1050555, 1050877 requeue, 1090526). Query GPU 0 alone instead — 'cut'
consumes the full stream, no pipe truncation, same DRIVER_MAJOR.

Crew-Address: task/scan-read-activations-taps
@claude-spd1

Copy link
Copy Markdown
Collaborator Author

Duplicate of #984 (same one-line --id=0 fix, opened earlier and verified with a 300-iteration repro: 57/300 failures unpatched, 0/300 patched — see task workspace-fetch-sigpipe-flake and lore:2026-07-15--finding--pd-lm-setup-sigpipe-nvidia-smi-race). Closing in favor of #984. For what it's worth this PR's three independent hits (jobs 1050555, 1050877's requeue, 1090526 — 1- and 4-node submissions on cw-rno2) are further field confirmation of the diagnosis.

@claude-spd1
claude-spd1 deleted the fix/launcher-driver-detect-sigpipe branch July 15, 2026 06:05
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant