[Question]: Does NCCL have any plan to provide QP counter metrics of GDAKI 's QPs?

[baymaxhuang]

Question

AFAIK, the QPs of GDAKI are finally created by DEVX, and we could not see any counters of DEVX's QP in /sys/class/infiniband/mlx5_xxx/ports/1/hw_counters/ such as out_of_sequence, req_cqe_error and local_ack_timeout_err. If we want to get the counter of DEVX's QPs, we should bind a qp counter set id to the QPs created by DEVX. Does NCCL have any plan to provide the QP counter metrics of GDAKI 's QPs so that we could easily observe the network issues when using GDAKI?

[xiaofanl-nvidia]

This is an interesting debuggability & observability idea. We will consider this.

@baymaxhuang could you help describe how you could potentially consume the counters from your software stack and infra?
E.g. would it be helpful to stream the counters via something like profiler/inspector?

[baymaxhuang]

@xiaofanl-nvidia Thanks for your reply. I think the QP counter could be shown like RAS subsystem and it could provide the following ability:

1. The user could observe the overall QP counter of each rank
2. The user could watch the specific rank's QP counter every second
3. The GDAKI subsystem could dump all the QP counters when the NCCL aborts so that we could find the relative network issue when the NCCL runs failure.

[xiaofanl-nvidia]

@baymaxhuang Thanks for sharing the ideas. It is aligned with another debuggability feature we had in mind related to the net_ib NIC counters, which have been helpful in large scale debugging use cases.

I'm adding @pakmarkthub and @armratner to further review & discuss. Pak/Armen - please file an internal RFE to track the idea if we have actionable & new features from this discussion. Thanks!

[armratner]

Hey @baymaxhuang, yes this is a good Idea, thank you, it does align well with another effort we have running to enable debuggability. we'll track internally and I'll make sure to keep you updated.