[Issue] RDMA PushRequest protocol mismatch causes ibv_post_send failure and data_num != 3 assertion on non-GDR (Ascend NPU) environment

[Dong-Jiahuan]

Summary

While porting StepMesh to a non-CUDA, Ascend NPU (ARM64) environment, Push operations crash consistently.
Initially the run failed with:

ibv_post_send failed.

But after further instrumentation, the crash was traced to the following protocol-layer assertion:

rdma_transport.h: PS_CHECK_EQ(buffer_ctx->data_num, 3)

This confirms the failure is not caused by RDMA verbs, but by an incorrect PushRequest data layout on the sender side.

---

Environment

• Hardware: Ascend NPU (ARM64)

• Architecture: aarch64

• StepMesh version: master (2025-01)

• RDMA: roce

• No CUDA / no GDR support

---

How I Identified the Real Root Cause

The initial crash happened inside ibv_post_send(). To rule out Inline SEND issues (EINVAL is common if inline size > device limit), I modified RDMAWriteWithImm:

if (inline_write) {
    wr.send_flags |= IBV_SEND_INLINE;
}
// Force-disable inline (important for ruling out EINVAL)

wr.send_flags &= ~IBV_SEND_INLINE;
if (prev_wr == nullptr) {

PS_CHECK_EQ(ibv_post_send(qp, &wr, &bad_wr), 0);

} else {

prev_wr->next = ≀

PS_CHECK_EQ(ibv_post_send(qp, prev_wr, &bad_wr), 0);

}

After disabling inline writes, the failure moved deterministically to:

rdma_transport.h: PS_CHECK_EQ(buffer_ctx->data_num, 3)

This shows:

1. RDMA verbs and QP are functional.

2. Inline was not the real cause.

3. The true bug is that the PushRequest sender is constructing an incorrect number of data segments.

Thus the crash is purely a protocol mismatch, not a hardware issue.

---

Expected Behavior

Per StepMesh internal RDMA protocol:

PushRequest (worker → server)
data_num = 3
segment[0] = keys
segment[1] = vals
segment[2] = lens

RecvPushRequest enforces this strictly:

PS_CHECK_EQ(buffer_ctx->data_num, 3);

rdma_transport

---

Actual Behavior on NPU Port

Sender constructs 1 or 2 segments, not 3.
As a result:

1. msg_buf->data.size() is wrong

2. SendRendezvousBegin sends a wrong data_num to peer

3. Receiver fails the assertion (data_num != 3)

4. Before disabling inline, this propagated as ibv_post_send errors (invalid WR state)

---

Root Cause

StepMesh currently relies on implicit, scattered assumptions for how many data segments each message type should contain.

The real contract is:

Message Type | Expected data_num | Required Segments -- | -- | -- PushRequest | 3 | keys / vals / lens PullRequest | 2 | keys / empty vals PullResponse | 3 | keys / vals / lens PushResponse | 0 | none

For PushRequest (non-GDR), the NPU sender must construct exactly 3 segments.
Because this convention is not centralized or validated, it is easy for non-GPU backends to violate it and cause fatal RDMA failures.

Attachments

Relevant source locations:

rdma_transport.h – receives PushRequest and asserts data_num == 3

rdma_van.h – converts msg.data into MessageBuffer

rdma_utils.h – Rendezvous structures

van.cc – SendMsg() code path
Closing

This issue is not RDMA-hardware related.
After disabling inline writes, the crash clearly originates from:

PushRequest sender not constructing 3 data segments.

A centralized protocol definition or normalization step would fix the issue and make StepMesh portable beyond CUDA/GDR environments.

[niehao100]

Hi,
Did you test build with no STEPMESH_USE_GDR flag in cmakelist ?

The original bytePS support the NO GDR mode and we start with it. The GDR support only put data on gpu and meta data on cpu, few modification can fix it.

We don't has Arm + 910 testbed. We also discuss with Ascend teams, as its roce function is implemented on NPU card, and verbs/fast sync is not supported, we believe the current version of stepmesh can not work on it without refactor, and it seems their teams are working on it.

If you are testing on a superpod, mayby ub interface is better choice.

[Dong-Jiahuan]

Hi, Did you test build with no STEPMESH_USE_GDR flag in cmakelist ?

The original bytePS support the NO GDR mode and we start with it. The GDR support only put data on gpu and meta data on cpu, few modification can fix it.

We don't has Arm + 910 testbed. We also discuss with Ascend teams, as its roce function is implemented on NPU card, and verbs/fast sync is not supported, we believe the current version of stepmesh can not work on it without refactor, and it seems their teams are working on it.

If you are testing on a superpod, mayby ub interface is better choice.

Yes, I am indeed building without the STEPMESH_USE_GDR flag enabled, so the error occurs outside the STEPMESH_USE_GDR-related code sections. Furthermore, I have implemented using the Ascend ACL interface according to your documentation (StepMesh/docs/backend.md) and registered the backend. I have also modified other parts of the code to adapt to the ARM architecture.

I am not sure if the current error is caused by my code modifications or if it is a general issue that arises when building without the STEPMESH_USE_GDR flag. I plan to test this in a GPU environment.

[Dong-Jiahuan]

Hi,
Did you test build with no STEPMESH_USE_GDR flag in cmakelist ?

The original bytePS support the NO GDR mode and we start with it. The GDR support only put data on gpu and meta data on cpu, few modification can fix it.

We don't has Arm + 910 testbed. We also discuss with Ascend teams, as its roce function is implemented on NPU card, and verbs/fast sync is not supported, we believe the current version of stepmesh can not work on it without refactor, and it seems their teams are working on it.

If you are testing on a superpod, mayby ub interface is better choice.

Hi,

I have fixed the PushRequest protocol mismatch
(PS_CHECK_EQ(buffer_ctx->data_num, 3)), and the communication proceeds further.

However, during communication tests I still hit a timeout:

PS_LOG(FATAL) << "request timeout " << timeout_ms << "ms, handler ..."

This happens in both cases:

• RDMA enabled (export DMLC_ENABLE_RDMA=1)
• RDMA disabled (no DMLC_ENABLE_RDMA)

Based on Postoffice::InitEnvironment, when RDMA is disabled the system falls back to the ZMQ Van by default. I would like to confirm if this understanding is correct.

Given that Ascend NPU does not support verbs / fast sync, I am wondering:

1. Is the ZMQ Van expected to support full data-plane communication (Push/Pull request-response) in StepMesh?

2. Is the observed timeout expected when running StepMesh on Ascend, even with ZMQ, due to platform or backend limitations?

3. Does Ascend support UCX-based communication?

I want to confirm whether this timeout indicates a misconfiguration on my side, or a current limitation of StepMesh on Ascend platforms.

Thanks.

[niehao100]

Hi @Dong-Jiahuan,

Thanks for you test. We didn't test ZMQ VAN for stepmesh.

I think maybe run cpu only mode with RDMA nic, and then DtoH/HtoD is a better choice.

[Dong-Jiahuan]

Hi @Dong-Jiahuan,

Thanks for you test. We didn't test ZMQ VAN for stepmesh.

I think maybe run cpu only mode with RDMA nic, and then DtoH/HtoD is a better choice.

Thanks for the suggestion. That is also what I was planning to try.
The server is currently unstable, so I will try it again later.
Thanks for the advice~

[Dong-Jiahuan]

Hi @Dong-Jiahuan,

Thanks for you test. We didn't test ZMQ VAN for stepmesh.

I think maybe run cpu only mode with RDMA nic, and then DtoH/HtoD is a better choice.

Hi @niehao100 , I have tried the method you recommended. But the program is stuck at:

[02:34:38] scheduler 0 /root/djh/StepMesh/src/van.cc:312: AddNode (1/2): [role=worker, ip=10.88.206.100, port=42203, is_recovery=0, aux_id=0, num_ports=1]

Here's the script:

import torch, os
import time
import fserver_lib as f
is_worker = os.environ.get('DMLC_ROLE') == 'worker'
is_server = os.environ.get('DMLC_ROLE') == 'server'

f.init()

if is_worker:
    npu = os.environ.get('STEPMESH_GPU')
    push_tensors = [
        torch.ones([1, 8192], dtype=torch.float32, device=f'npu:{npu}').cpu(),
        torch.ones([1, 8192], dtype=torch.float32, device=f'npu:{npu}').cpu(),
        torch.ones([1, 8192], dtype=torch.float32, device=f'npu:{npu}').cpu(),
    ]
    pull_tensors = [
        torch.rand([1, 8192], dtype=torch.float32, device=f'npu:{npu}').cpu()
    ]
    handler = f.push_pull(
        push_tensors, 
        [i for i in range(len(push_tensors))], 
        pull_tensors, 
        [i for i in range(len(pull_tensors))]
    )
    f.wait(handler)
    # assert torch.allclose(sum(push_tensors), pull_tensors[0])
    print(f'sum(push_tensors):{sum(push_tensors)}')
    print(f'pull_tensors[0]:{pull_tensors[0]}')
    print("worker test done")

elif is_server:
    # npu = os.environ.get('STEPMESH_GPU')
    torch.set_default_device('cpu')
    res = []
    while len(res) == 0:
        time.sleep(1)
        res = f.get_batch()
    print(f'res:{res}')
    for r in res:
        print(f'r:{r}')
        comm_id, batch, _ = r
        f.respond([sum(batch)], comm_id)

f.stop()

Here is the sh command I used:

THIS_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
function cleanup() {
    echo "kill all testing process of ps lite for user $USER"
    # pkill -9 -f test_bench
    pkill -9 -f test_remote_moe
    pkill -9 -f test_fserver
    sleep 1
}
trap cleanup EXIT
cleanup

# common setup
export BIN=${BIN:-test_fserver}

# Use InfiniBand interface for RDMA communication
# Check if InfiniBand interfaces are available
ENP_INTERFACE=$(ip -o -4 addr | grep "enp" | head -1 | awk '{print $2}' | cut -d'@' -f1)
if [ -z "$ENP_INTERFACE" ]; then
    echo "Warning: No ENP interface found, falling back to bond0"
    export DMLC_INTERFACE=${RNIC:-bond0}
else
    echo "Using InfiniBand interface: $ENP_INTERFACE"
    export DMLC_INTERFACE=$ENP_INTERFACE
fi
export DMLC_INTERFACE="enp125s0f0"
export SCHEDULER_IP=$(ip -o -4 addr | grep ${DMLC_INTERFACE} | awk '{print $4}' | cut -d'/' -f1)
export DMLC_NUM_WORKER=1
export DMLC_NUM_SERVER=1
export DMLC_PS_ROOT_URI=$SCHEDULER_IP  # scheduler's RDMA interface IP 
export DMLC_PS_ROOT_PORT=8124     # scheduler's port (can random choose)
export DMLC_ENABLE_RDMA=1
export DMLC_GROUP_SIZE=1

# Bind all roles to the same RDMA IP
export DMLC_NODE_HOST=${SCHEDULER_IP}

# Disable QP LAG to avoid mlx5dv LAG errors on InfiniBand
export STEPMESH_SPLIT_QP_LAG=0
export STEPMESH_GPU=0
export PS_VERBOSE=3

echo "Using scheduler IP: $SCHEDULER_IP"
echo "Using interface: $DMLC_INTERFACE"


STEPMESH_BACKEND=CPU DMLC_NODE_RANK=0 DMLC_ROLE=scheduler python3 $THIS_DIR/$BIN.py &
STEPMESH_BACKEND=CPU DMLC_NODE_RANK=0 DMLC_ROLE=server python3 $THIS_DIR/$BIN.py $@ &
STEPMESH_BACKEND=CPU DMLC_NODE_RANK=0 DMLC_ROLE=worker python3 $THIS_DIR/$BIN.py $@


wait

Here are the error details:

kill all testing process of ps lite for user 
Using InfiniBand interface: enp125s0f0
Using scheduler IP: 10.88.206.100
Using interface: enp125s0f0
[02:34:37] server 0 /root/xxxxx/StepMesh/src/postoffice.cc:75: Creating Van: 1. group_size=1
[02:34:37] server 0 /root/xxxxx/StepMesh/src/van.cc:95: Creating RDMAVan.
[02:34:37] server 0 /root/xxxxx/StepMesh/src/van.cc:96: DMLC_ENABLE_RDMA=1 will be deprecated. Please use DMLC_ENABLE_RDMA=ibverbs instead.
[02:34:37] server 0 /root/xxxxx/StepMesh/src/van.cc:562:  interface and ip from env: enp125s0f0 (10.88.206.100)
[02:34:37] server 0 /root/xxxxx/StepMesh/src/./rdma_van.h:179: bind to DMLC_NODE_HOST: 10.88.206.100
[02:34:37] server 0 /root/xxxxx/StepMesh/src/van.cc:601: Bind to [role=server, ip=10.88.206.100, port=59689, is_recovery=0, aux_id=-1, num_ports=1]
[02:34:37] server 0 /root/xxxxx/StepMesh/src/./rdma_van.h:204: Connecting to Node 1, My_Node=32767
[02:34:37] server 0 /root/xxxxx/StepMesh/src/././rdma_transport.h:190: qp created: pd=0xfffe0c000d50 , cq=0xfffe0c0010f0, qp=522
[02:34:38] server 0 /root/xxxxx/StepMesh/src/././rdma_transport.h:190: qp created: pd=0xfffe0c000d50 , cq=0xfffe0c0010f0, qp=523
[02:34:38] server 0 /root/xxxxx/StepMesh/src/./rdma_van.h:1004: 32767 to 1 connection rejected, retrying...
[02:34:38] server 0 /root/xxxxx/StepMesh/src/./rdma_van.h:1004: 32767 to 1 connection rejected, retrying...
[02:34:38] scheduler 0 /root/xxxxx/StepMesh/src/postoffice.cc:75: Creating Van: 1. group_size=1
[02:34:38] scheduler 0 /root/xxxxx/StepMesh/src/van.cc:95: Creating RDMAVan.
[02:34:38] scheduler 0 /root/xxxxx/StepMesh/src/van.cc:96: DMLC_ENABLE_RDMA=1 will be deprecated. Please use DMLC_ENABLE_RDMA=ibverbs instead.
[02:34:38] scheduler 0 /root/xxxxx/StepMesh/src/./rdma_van.h:179: bind to DMLC_NODE_HOST: 10.88.206.100
[02:34:38] scheduler 0 /root/xxxxx/StepMesh/src/van.cc:601: Bind to [role=scheduler, id=1, ip=10.88.206.100, port=8124, is_recovery=0, aux_id=-1, num_ports=1]
[02:34:38] scheduler 0 /root/xxxxx/StepMesh/src/./rdma_van.h:204: Connecting to Node 1, My_Node=1
[02:34:38] scheduler 0 /root/xxxxx/StepMesh/src/././rdma_transport.h:190: qp created: pd=0xfffdf8000d50 , cq=0xfffdf80010f0, qp=492
[02:34:38] scheduler 0 /root/xxxxx/StepMesh/src/././rdma_transport.h:190: qp created: pd=0xfffdf8000d50 , cq=0xfffdf80010f0, qp=493
[02:34:38] scheduler 0 /root/xxxxx/StepMesh/src/././rdma_transport.h:190: qp created: pd=0xfffdf8000d50 , cq=0xfffdf80010f0, qp=500
[02:34:38] scheduler 0 /root/xxxxx/StepMesh/src/./rdma_van.h:1058: 1 OnConnect to 1 with Transport=RDMA QP_NUM 500
[02:34:38] scheduler 0 /root/xxxxx/StepMesh/src/././rdma_transport.h:190: qp created: pd=0xfffdf8000d50 , cq=0xfffdf80010f0, qp=501
[02:34:38] scheduler 0 /root/xxxxx/StepMesh/src/./rdma_van.h:1058: 1 OnConnect to 1 with Transport=RDMA QP_NUM 501
[02:34:38] worker 0 /root/xxxxx/StepMesh/src/postoffice.cc:75: Creating Van: 1. group_size=1
[02:34:38] worker 0 /root/xxxxx/StepMesh/src/van.cc:95: Creating RDMAVan.
[02:34:38] worker 0 /root/xxxxx/StepMesh/src/van.cc:96: DMLC_ENABLE_RDMA=1 will be deprecated. Please use DMLC_ENABLE_RDMA=ibverbs instead.
[02:34:38] worker 0 /root/xxxxx/StepMesh/src/van.cc:562:  interface and ip from env: enp125s0f0 (10.88.206.100)
[02:34:38] worker 0 /root/xxxxx/StepMesh/src/./rdma_van.h:179: bind to DMLC_NODE_HOST: 10.88.206.100
[02:34:38] worker 0 /root/xxxxx/StepMesh/src/van.cc:601: Bind to [role=worker, ip=10.88.206.100, port=42203, is_recovery=0, aux_id=-1, num_ports=1]
[02:34:38] worker 0 /root/xxxxx/StepMesh/src/./rdma_van.h:204: Connecting to Node 1, My_Node=32767
[02:34:38] worker 0 /root/xxxxx/StepMesh/src/././rdma_transport.h:190: qp created: pd=0xfffdf8000d50 , cq=0xfffdf80010f0, qp=486
[02:34:38] worker 0 /root/xxxxx/StepMesh/src/././rdma_transport.h:190: qp created: pd=0xfffdf8000d50 , cq=0xfffdf80010f0, qp=487
[02:34:38] scheduler 0 /root/xxxxx/StepMesh/src/././rdma_transport.h:190: qp created: pd=0xfffdf8000d50 , cq=0xfffdf80010f0, qp=508
[02:34:38] scheduler 0 /root/xxxxx/StepMesh/src/./rdma_van.h:1058: 1 OnConnect to 32767 with Transport=RDMA QP_NUM 508
[02:34:38] scheduler 0 /root/xxxxx/StepMesh/src/././rdma_transport.h:190: qp created: pd=0xfffdf8000d50 , cq=0xfffdf80010f0, qp=509
[02:34:38] scheduler 0 /root/xxxxx/StepMesh/src/./rdma_van.h:1058: 1 OnConnect to 32767 with Transport=RDMA QP_NUM 509
[02:34:38] worker 0 /root/xxxxx/StepMesh/src/./rdma_van.h:1141[: 3276702:34:38 OnConnected to ] 1scheduler
 0 /root/xxxxx/StepMesh/src/./rdma_van.h:1141: 1 OnConnected to 32767
[02:34:38] worker 0 /root/xxxxx/StepMesh/src/./rdma_van.h:1141: 32767 OnConnected to 1
[02:34:38] scheduler 0 /root/xxxxx/StepMesh/src/./rdma_van.h:1141: 1 OnConnected to 32767
[02:34:38] worker 0 /root/xxxxx/StepMesh/src/././rdma_transport.h:405: SendRendezvousBegin 
[02:34:38] worker 0 /root/xxxxx/StepMesh/src/van.cc:701: rdma 32767       sent: ? => 1. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ [role=worker, ip=10.88.206.100, port=42203, is_recovery=0, aux_id=0, num_ports=1] } }. NOT DATA MSG!, Slave QP Count: 0
[02:34:38] scheduler 0 /root/xxxxx/StepMesh/src/./rdma_van.h:474: 3. 1 RecvMsg: ? => 1. Meta: request=0, app_id=32767, customer_id=32767, simple_app=0, push=0, sid=0, key=0, Slave QP Count: 0
[02:34:38] scheduler 0 /root/xxxxx/StepMesh/src/van.cc:731: rdma 1        received: ? => 1. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ [role=worker, ip=10.88.206.100, port=42203, is_recovery=0, aux_id=0, num_ports=1] } }. NOT DATA MSG!, Slave QP Count: 0
[02:34:38] scheduler 0 /root/xxxxx/StepMesh/src/van.cc:312: AddNode (1/2): [role=worker, ip=10.88.206.100, port=42203, is_recovery=0, aux_id=0, num_ports=1]