#17 - Add RTX PRO 6000 benchmark YAMLs and baseline

[chloecrozier]

Addresses #17

RTX PRO 6000 raw/RoCE/socket benchmarks, now validated over a real wire closed-loop (two ConnectX-7 400G ports cabled together) with a multi-queue core-scaling sweep and plots.

What this PR adds / fixes

Bug fixes (found via wire testing):

• Cross-GPU illegal memory access in the GPU workload path — cuFFT/cuBLAS handles and reorder buffers were created on the default CUDA device while wire RX buffers live on GPU 1 (affinity: 1), faulting the context and stalling RX. rx_count_worker now binds to the RX buffer's device before workload init (fixes both dpdk and dpdk-hds fft/gemm on the wire).
• RDMA YAML schema — replaced the removed protocol:/remote_ip keys with socket_config.local_addr: roce://… on one connected subnet.
• Runner — auto RoCE IP/sysctl setup + teardown, parse_rdma_drops integer bug, and scoped the "RX too low" stall guard to raw packet backends (RDMA/socket move large messages).

New tooling:

• Multi-queue (TX,RX) core-scaling base YAML + gen_rtx_pro_mq_config.py + run_rtx_pro_mq_bench.sh (sweeps payload × core cells).
• scripts/plot_rtx_pro_bench.py — pure-stdlib SVG plots (white background), no matplotlib dependency.
• run_rtx_pro_push_400g.sh — batch / host_pinned / dual-loop experiments toward line rate.

Results (wire closed-loop, 15 s/cell)

Backend	none	fft	gemm	Notes
dpdk (GPUDirect)	216.0	215.3	215.2	0 drops
dpdk-hds	218.9	218.2	218.3	0 drops
ibverbs (MPRQ)	130.2	128.8	129.8	paired DPDK TX
rdma (RoCE, 8 MB msg)	378.6	137.9	5.2	gemm compute-bound

Memory kind (same cable, 8 KB): device GPUDirect 216 Gbps vs host_pinned 384 Gbps (~96% of 400G) — the GPUDirect RX DMA path, not the NIC/PCIe (Gen5 x16), is the current ceiling.

Core scaling × payload (GPUDirect RX Gb/s):

Cell	64 B	256 B	1 KB	8 KB
1 TX / 1 RX	31	76	188	216
1 TX / 2 RX	55	130	213	215
2 TX / 2 RX	56	129	210	215

Extra RX cores ~2× small-packet throughput; at jumbo all cells converge to link-limited ~215 Gbps.

How to run

cmake -S . -B build -DCMAKE_CUDA_ARCHITECTURES=120 && cmake --build build -j
source scripts/discover_rtx_pro_topology.sh
# Backend matrix
sudo -E env ETH_DST_ADDR="$ETH_DST_ADDR" ./examples/run_rtx_pro_bench.sh dpdk issue17 --seconds 15
# Multi-queue core/payload sweep + plots
sudo -E env ETH_DST_ADDR="$ETH_DST_ADDR" ./examples/run_rtx_pro_mq_bench.sh

[greptile-apps]

Greptile Summary

This PR completes the RTX PRO 6000 benchmark suite originally started in the earlier partial commit, adding nine new examples/ YAML configs, a full wire-tested benchmark runner, multi-queue core-scaling tooling, and three bug fixes validated over a real ConnectX-7 400G closed loop.

• Engine fixes: ibverbs catch-all matcher priority corrected (1,000,000 → 0) to prevent uint16_t truncation that caused ENOMEM on mlx5dv rule creation; socket set_packet_lengths now validates negative lengths and oversized payloads before assigning to pkt_lens. The DrMatchBuf struct (64 B) is retired in favour of the existing DrMatchParam (512 B) for all flow rules including the catch-all.
• New configs: eight RTX PRO 6000 YAML files registered in examples/CMakeLists.txt, spanning sw-loopback, dual-port NIC, HDS, ibverbs MPRQ, multi-queue base, and RoCE closed-loop; accompanied by run_rtx_pro_bench.sh (auto RoCE IP/sysctl setup/teardown, stall-guard scoped to raw backends) and run_rtx_pro_mq_bench.sh sweeping payload × core cells with pure-stdlib SVG plotting.
• Doc sync: AGENTS.md benchmark table, docs/benchmarks/raw_benchmarking.md tip block, and docs/tutorials/configuration-walkthrough.md decision tree are all updated to cover every new config.

Confidence Score: 4/5

Safe to merge once the missing sign-off on commit ef9bcc6 is added.

The engine bug-fixes are narrowly scoped and well-motivated by wire-test evidence. One merge commit is missing a DCO sign-off trailer while every other commit carries one correctly. Resolving that single item unblocks merge.

Only AGENTS.md is flagged for the DCO issue; all source and YAML files are clean.

Important Files Changed

Filename	Overview
src/engines/ibverbs/daqiri_ibverbs_engine.cpp	Priority constant corrected from 1,000,000 (silently truncates to 16960 in uint16_t) to 0, and the catch-all path upgraded from the removed 64-byte DrMatchBuf to the full DrMatchParam so the rule no longer fails with ENOMEM.
src/engines/socket/daqiri_socket_engine.cpp	set_packet_lengths now guards against negative lengths and validates against the endpoint's max_packet_size, preventing a silent negative-to-uint32_t wrap and catching oversized payloads before they corrupt the buffer.
examples/CMakeLists.txt	Registers eight new RTX PRO 6000 YAML configs for cmake install and adds ibverbs to the static-link engine loop — correct since daqiri_ibverbs is now a distinct target from daqiri_rdma.
examples/daqiri_bench_rdma_tx_rx_rtx_pro_6000.yaml	RoCE closed-loop config using socket_config.local_addr: roce:// endpoints; fixed YAML schema (no removed protocol:/remote_ip keys). Memory regions use host_pinned with correct numa affinity split.
examples/run_rtx_pro_bench.sh	New 855-line runner covering dpdk/dpdk-hds/rdma/ibverbs/socket-udp/socket-tcp backends; includes RoCE IP auto-setup/teardown with a trap on EXIT. Logic looks complete; references bench_capture_environment.sh which must already exist in examples/.
AGENTS.md	Benchmark table updated with all new RTX PRO 6000 YAML configs and the ibverbs RX / tx-only pair.
docs/benchmarks/raw_benchmarking.md	Adds RTX PRO 6000 tip block with config links, run commands, tx_phy_packets/rx_phy_packets verification note — mirrors the DGX Spark block in structure.
docs/tutorials/configuration-walkthrough.md	Decision tree extended with all eight new RTX PRO 6000 configs including ibverbs MPRQ, HDS, MQ, and RoCE entries.

_{Reviews (8): Last reviewed commit: "#17 - Fix ibverbs catch-all flow steerin..." | Re-trigger Greptile}