[Issue]: vLLM serve with tensor-parallel-size=8 on Kubernetes + vGPU fails: NCCL TCPStore broken pipe, EngineCore initialization failed #42389

[sumufuyun]

How is this issue impacting you?

Lower performance than expected

Share Your Debug Logs

1

Steps to Reproduce the Issue

vllm-project/vllm#42389

NCCL Version

2.28.9

Your platform details

No response

Error Message & Behavior

No response

[stephenmsachs]

@sumufuyun looking through the provided logs on the vllm issue, I do not believe this is NCCL related.

The Exception raised from sendBytes message originates from a failed socket connection inside pytorch.

The subsequent ProcessGroupNCCL.cpp: error is a failure of the pt_nccl_heartbt thread within pytorch. AFAIK this is not a NCCL error it observed but a failure of the thread itself.

If you believe this is a NCCL issue, run with NCCL_DEBUG=INFO and attach the logs.