#15 - Clean up of RoCE bench issues#223
Merged
Merged
Conversation
Pin the GEMM dimension (n=1024, --workload-gemm-n) so the RoCE-vs-raw comparison isolates transport from problem size. The prior batch sweep confounded the two (n scaled with batch), which produced the misleading "gap is pipelining depth" reading. The window-matched fixed-n data shows the gap is real and receive-path/threading bound, not the GPU; adds the GEMMs/s + SM% table, flags the batch sweep as superseded, and documents the reproduce commands. Assisted-by: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ramya Gurunathan <rgurunathan@nvidia.com>
Pin the square GEMM dimension (--workload-gemm-n) so the FLOP count per call is fixed while the I/O unit is swept, isolating transport from problem size; init logs the chosen GEMM/FFT shape and FLOP count, and rejects a pinned n whose A operand exceeds the received working set. Add --workload-sync-interval to vary GPU stream drain depth for characterizing the single-threaded receive+compute ceiling. Thread both through all bench mains and expose them via run_spark_bench.sh (GEMM_N, SYNC_INTERVAL envs; post_process_gemm_n, post_process_sync CSV columns). Assisted-by: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ramya Gurunathan <rgurunathan@nvidia.com>
The --workload-sync-interval sweep (2->32, both precisions) shows no measurable change in RoCE throughput, and fully-synchronous matches deep-async. Sync cadence is not the ceiling; the RoCE path already drains once per held-receive batch, masking the knob, so the fix is structural (dedicated GPU-worker thread) rather than sync tuning. Updates the fixed-size GEMM section accordingly. Assisted-by: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ramya Gurunathan <rgurunathan@nvidia.com>
…de flags The receive-path GPU workload sized its GEMM from a --workload-batch-bytes knob (deriving n = sqrt(batch/4)) and sub-divided each RoCE message into chunk-sized slices, which conflated problem size with pipelining depth and made the fixed-n RoCE-vs-raw comparison confusing to run (juggling WORKLOAD_BATCH + PAYLOADS_OVERRIDE). Pin the square GEMM side length directly instead: - Rename --workload-gemm-n -> --workload-gemm-dim (default 1024); the compute working set is exactly n*n*elem_size, read from the front of each received I/O unit. GpuWorkload rejects a unit too small to hold the operand. - Delete --workload-batch-bytes / parse_workload_batch_bytes and the derive-n-from-bytes branch. Each backend keeps its default reorder window (raw 1024 packets, socket 8 MB, RoCE whole message); RoCE now runs one fixed-dimension GEMM per received message. - run_spark_bench.sh: GEMM_N -> GEMM_DIM (default 1024); drop WORKLOAD_BATCH and PAYLOADS_OVERRIDE. When a workload is active the payload sweep is restricted to the headline size (the small cells cannot hold the operand). CSV: post_process_gemm_n -> post_process_gemm_dim; drop post_process_batch. - Docs: update AGENTS.md bench-flag summary and the DGX Spark perf report methodology/reproduce; remove the superseded batch-size-sweep section and mark the fixed-GEMM table pending refresh. Example-only change; no library rebuild required. Signed-off-by: Ramya Gurunathan <rgurunathan@nvidia.com> Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
…drain The RoCE workload bench read each recv buffer zero-copy and, to know when the buffer was safe to free/repost, drained the whole GPU stream (cudaStreamSynchronize) once per held-receive batch -- and run() also drained every sync_interval GEMMs. On the single receive thread that drain parks the thread on the GPU, so it stops reaping completions and reposting receives while the GEMM runs. That serialization (not the GPU, buffers, or transport) is why the fixed-n RoCE point ran ~2x slower than raw at the identical workload. Replace the blocking drain with non-blocking, event-based recycling: - GpuWorkload gains a lazily-created pool of cudaEvents (cudaEventDisableTiming) and record_event / event_done / wait_event / release_event, plus run_async (enqueue with no maybe_sync). Events complete in stream order, so an event stamped after a message's GEMM signals that the read of that recv buffer is done. The pool costs nothing for the run()+sync() callers that never record. - rdma_bench holds each completion with its event in a FIFO, frees buffers as soon as event_done() reports the GEMM finished (reclaim_completed), and keeps reaping completions meanwhile. Outstanding GPU work is bounded by max_inflight (backpressure: block on the oldest event only at the cap), which replaces the periodic sync_interval drain as the measurement-honesty bound. Shutdown / connection loss drains all events fully before teardown. DPDK/socket are unchanged (they copy into the pipeline's shared output buffer, so overlapping GEMMs would need that buffer ring-buffered first); a comment in raw_bench_common notes the prerequisite. --workload-sync-interval still governs the DPDK/socket run()+sync() path; the RoCE path bounds work with events and ignores it. Example-only change; no library rebuild required. Signed-off-by: Ramya Gurunathan <rgurunathan@nvidia.com> Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
…iling knobs The RoCE GPU-workload receive thread parked on the GPU every message -- a blocking drain of held recv buffers on the idle path, plus an eager cap wait -- which starved the GPU and throttled the sender. Reclaim held recv buffers non-blocking instead: drop the idle-path drain, reclaim before the last-resort cap block, and only block when the event pool would otherwise starve. nsys confirmed the per-message cudaEventSynchronize -- not the GPU, memory, or transport -- was the cap. Add the bench knobs used to root-cause and re-measure this: - run_spark_bench.sh: UNIDIR (one-way RoCE, matching the DPDK and socket benches which are already one-way), NSYS/NSYS_BIN (wrap the rdma server in nsys), RUN_SECONDS override, MAX_INFLIGHT, and RDMA_BUDGET_GIB/RX_NB/TX_NB for the buffer-depth experiment. - --workload-max-inflight bench flag and a larger CUDA event pool. The residual RoCE<DPDK gap turned out to be the bidirectional benchmark, not the pipeline: one-way RoCE holds line rate (FP32 ~103 Gb/s), matching DPDK. Assisted-by: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: rgurunathan <rgurunathan@nvidia.com>
…on results Replace the stale, pre-fix bidirectional numbers with a single same-session table comparing DPDK and one-way RoCE (none / FFT / GEMM FP32); both hold ~line rate under the GPU workload. Delete the obsolete "single-threaded RoCE stall" narrative and the GEMMs/s + duty-cycle-SM analysis -- the gap was the bidirectional benchmark, now removed. Drop the FP16 GEMM result, clarify both workloads are FP32, and remove the misleading "native shape" wording (8 KB is a chosen operating point, not the throughput peak). Fix the Reproduce commands to set UNIDIR=1 for the RoCE runs. Assisted-by: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: rgurunathan <rgurunathan@nvidia.com>
The RoCE netns bench was the only transport running bidirectionally (both roles send + receive), which understated its one-way rate and made it inconsistent with the DPDK bench (inherently TX->RX one-way) and the socket bench (explicitly one-way). Flip the combined base config so the server is receive-only (runs the GEMM) and the client is send-only, mirroring the socket netns base; gen_spark_netns_config.py splits each role's section verbatim, so the split configs inherit the one-way flags. Remove the now-redundant UNIDIR env knob and its awk send:/receive: rewrite from run_spark_bench.sh -- one-way is the default. Drop UNIDIR=1 from the perf-doc Reproduce commands and reword the methodology note accordingly. The RoCE result numbers are re-measured in a follow-up commit. Assisted-by: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: rgurunathan <rgurunathan@nvidia.com>
The setup script hardcoded CLIENT_IF/SERVER_IF (enp1s0f0np0 / enP2p1s0f1np1),
which no longer exist after PCIe re-enumeration on the DGX Spark. With dead
names, detect_macs aborts, the dq_wire_{client,server} namespaces are never
created, and the netns-based socket/RoCE benches (launched via `ip netns exec`)
silently have nowhere to run.
Make the cabled ports auto-detected and robust to enumeration drift:
- CLIENT_IF/SERVER_IF/CLIENT_RDMA/SERVER_RDMA default to blank, env-overridable.
- autodetect_ports() picks the two carrier-up RoCE-backed netdevs in init_net
and derives the RDMA devices from sysfs (rdma_for_netdev); errors with
guidance if not exactly two are found.
- down() now moves every non-lo netdev in each namespace back to init_net, so
teardown no longer depends on a hardcoded interface name.
- resolve_ports() lets verify()/monitor() read the in-namespace netdev names
after up() has moved the ports, falling back to auto-detect otherwise.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: rgurunathan <rgurunathan@nvidia.com>
autodetect_ports() required exactly 2 carrier-up RoCE netdevs, but Spark's single
CX-7 exposes each of its 2 physical ports over both PCIe segments (socket-direct),
so a cabled loopback lights carrier on up to 4 PFs at once. The count check then
aborts ("found 3/4"), and it could also pick two PCIe reps of the SAME physical
port -- the on-chip eswitch shortcut instead of an over-the-wire path.
Group carrier-up RoCE netdevs by phys_port_name (the hardware port label, stable
across PCIe re-enumeration and interface renames), keep one rep per physical port
(lowest BDF, matching the doc's canonical enp1s0f{0,1}np{0,1} names), and require
exactly 2 distinct ports. CLIENT_IF/SERVER_IF therefore always straddle p0 and p1
(over-the-wire). Falls back to per-netdev grouping when phys_port_name is
unavailable so the 2-port guard still holds. Refresh the header comments and the
manual-override example accordingly.
See docs/tutorials/system_configuration.md "Port topology: 4 PFs, 2 ports".
Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: rgurunathan <rgurunathan@nvidia.com>
The FFT workload's 1-D C2C transform length was a hardcoded constant (1024), while GEMM already had --workload-gemm-dim. Add a symmetric --workload-fft-len so the FFT compute size is a benchmark knob, independent of the GEMM dimension: - parse_workload_fft_len() mirrors parse_workload_gemm_dim; GpuWorkload::init() takes an fft_len arg (default 1024) and threads it into cufftPlan1d, replacing the kFftLen constant. The working set still fans out across as many batched length-N transforms as fit. - Thread fft_len through every bench main (rdma, socket, raw gpudirect/hds) and rx_count_worker, exactly as gemm_dim is. - run_spark_bench.sh gains an FFT_LEN env (default 1024) passed as --workload-fft-len for all backends. Verified in-container: FFT_LEN=512 logs "fft shape = 1953 x C2C length-512". Assisted-by: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: rgurunathan <rgurunathan@nvidia.com>
Now that the RoCE bench is one-way, re-measure and refresh the perf report so all three transports are apples-to-apples: - Results Summary RoCE row and message-size sweep updated with one-way numbers (best case 106.0 Gb/s at 64 KB; 8 MB 105.0). Large messages rose vs the old bidirectional run (reverse-direction contention removed); the op-rate-bound small cells are the honest single-direction rate. - CPU-utilization table: fix the receiver core. For rdma, cpu_rx measured the idle client RX core (18); the one-way receiver is the SERVER, whose RX poller + bench worker share core 19. Point CPU_RX at 19 and re-measure -- the receiver core is ~1.1% at the bare-loopback 8 MB cell, confirming the RoCE RC "HCA DMAs straight to memory" signature with the correct core. - Reproduce: collapse the two redundant GPU-workload blocks into one (a workload-active sweep auto-pins to the summary-table cell, so it equals smoke); add a general smoke-vs-sweep explanation; document the FFT default and the new FFT_LEN / --workload-fft-len knob; drop the redundant explicit GEMM_DIM=1024. - Rename "headline cell" -> "summary-table cell" throughout. Assisted-by: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: rgurunathan <rgurunathan@nvidia.com>
RamyaGuru
marked this pull request as ready for review
July 10, 2026 15:13
Contributor
|
| Filename | Overview |
|---|---|
| examples/rdma_bench.cpp | Replaces the per-batch flush approach with CUDA-event recycling: each received buffer is stamped with an event and freed only once event_done() returns true, eliminating the per-message blocking stall. All BurstParams free paths are correctly handled — the ev==nullptr fallback syncs the stream and frees immediately; the normal path defers to reclaim_completed() / drain_held_recv(). max_inflight cap is correctly clamped to kMaxWorkloadInflight (< the event pool) so the pool never starves in steady state. |
| scripts/setup_spark_wire_loopback_netns.sh | New autodetect_ports() groups carrier-up RoCE netdevs by phys_port_name, picks the lowest-BDF representative per physical port, and requires exactly 2 ports to produce a valid client/server pair. Robust to PCIe re-enumeration and interface renames. |
| examples/bench_workload.cu | Adds parse_workload_gemm_dim/fft_len/sync_interval/max_inflight parsers; pins the GEMM side length so FLOP count per call is fixed; adds record_event/event_done/wait_event/release_event using a lazily-created 64-event pool. |
| examples/run_spark_bench.sh | Adds GEMM_DIM, FFT_LEN, SYNC_INTERVAL, MAX_INFLIGHT, NSYS, RDMA_BUDGET_GIB, RDMA_RX/TX_NB env vars. Fixes socket port-collision, adds separate send/receive core pinning, RoCE one-way mode and wire-transit phy check. |
| examples/daqiri_bench_rdma_tx_rx_spark_netns.yaml | Adds send: false / receive: true on the server and send: true / receive: false on the client, making RoCE one-way by default to match DPDK and socket benches. |
| docs/benchmarks/performance-dgx-spark.md | Comprehensive refresh to phy-verified 100 GbE numbers: corrects RoCE/DPDK peaks, rewrites multi-queue section, updates TCP/UDP tables, replaces workload batch-size sweep with fixed-n GEMM comparison table. |
Reviews (4): Last reviewed commit: "#15 - Pin socket send/receive to separat..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Address Greptile doc-sync review on PR #223: - Bench usage strings listed only --workload-gemm-dim. Add the other workload flags each main actually parses: --workload-fft-len and --workload-sync-interval to all four, plus --workload-max-inflight to the rdma bench (RoCE event path). - raw_benchmarking.md: note that --workload-gemm-dim / --workload-fft-len pin the GEMM side length / FFT transform length so the per-call FLOP count is fixed across the I/O sweep. - run_spark_bench.sh: GEMM_DIM / FFT_LEN validation accepted 0 (^[0-9]+$) despite the "positive integer" message; tighten to ^[1-9][0-9]*$. (The review's claim that the docs still document --workload-batch-bytes is stale -- that flag was already removed; no references remain in the tree.) Assisted-by: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: rgurunathan <rgurunathan@nvidia.com>
| | Raw Ethernet / GPUDirect | 4 KB packet | **105.5 ±0.9 Gb/s** | 0 | 98.5 Gb/s single-queue at the 8 KB native shape | | ||
| | Socket / RoCE (SEND) | 8 MB message | **102.2 ±0.3 Gb/s** | 0 | Single QP, batch 1 | | ||
| | Raw Ethernet / GPUDirect | 4 KB packet | **105.5 ±0.9 Gb/s** | 0 | 98.5 Gb/s single-queue at 8 KB | | ||
| | Socket / RoCE (SEND) | 64 KB message | **106.0 ±0.1 Gb/s** | 0 | Single QP, batch 1 | |
Collaborator
There was a problem hiding this comment.
I think we should just artificially say this is 100Gbps since it could be confusing to people without a footnote. The Spark networking is very strange so 106Gbps could look bad in some cases if people think it's a 200G link.
Collaborator
Author
There was a problem hiding this comment.
No, this was it :/ I had grabbed a second QSFP from the office that turned out to be 200 instead of 100 Gps, and think I must have been running with that cable for some time. I'm so sorry for the confusion! I refreshed all numbers in this doc with a vetted 100G link. It reproduced most of the old numbers, but this time saturated at wire rate properly instead of giving > 100 Gbps throughput values.
I think the old report was capping ~108 Gbps on the 200G link because I was measuring unidirectional traffic only p0 --> p1?
| | Core | Busy% | Note | | ||
| | -------------------- | ----: | ----------------------------------------------- | | ||
| | Master (CPU 8) | 0.7% | Orchestration only | | ||
| | Client TX (CPU 17) | 74.8% | Post-and-poll spin; rate-independent | |
Collaborator
There was a problem hiding this comment.
What does rate-independent mean here?
Collaborator
Author
There was a problem hiding this comment.
I think meaning that the CPU % for polling CPUs is not dependent on the data rate on the wire? But I removed that phrase because I don't believe it was something I actually tested for.
|
|
||
| // Parse "--workload-fft-len N" from argv: the 1-D C2C transform length. The burst | ||
| // working set is fanned out across as many batched length-N transforms as fit, so | ||
| // N sets the per-transform cost (~5*N*log2(N) flops) while the batch count tracks |
Collaborator
There was a problem hiding this comment.
That equation only works for powers of two, so we may want to restrict it or remove that comment
Collaborator
Author
There was a problem hiding this comment.
Thanks. Removed comment for now.
|
|
||
| | Workload | DPDK (Raw / GPUDirect) | RoCE (RC) | | ||
| | -------- | ---------------------: | --------: | | ||
| | none (baseline) | 98.4 ±0.2 | 108.0 ±0.1 | |
Collaborator
There was a problem hiding this comment.
This one is surprising to me. Do we know why the packet reception went down by 5% when a workload was enabled? I would think given the memory bandwidth it would be unaffected.
Collaborator
Author
There was a problem hiding this comment.
I don't have a clear explanation yet. I removed speculative explainer text in the report. I think another data point on IGX or RTX Pro server would help me see if it has something to do with unified memory on the Spark that I'm not handling right?
cliffburdick
approved these changes
Jul 10, 2026
- examples/bench_workload.h: the --workload-fft-len comment gave the FFT cost as ~5*N*log2(N) flops -- the radix-2 estimate, valid only for power-of-two N; cuFFT uses different algorithms (and costs) for other lengths. Drop the equation so the comment holds for any N (r3560531760). - performance-dgx-spark.md: "rate-independent" was undefined jargon (r3560468420). Reword the RoCE TX and DPDK poller CPU notes to say the busy% comes from the post-and-poll / poll-mode spin loop rather than the throughput, and add a sentence spelling out that the core stays near this level whether the link runs at 10 or 100 Gb/s. Assisted-by: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: rgurunathan <rgurunathan@nvidia.com>
The DPDK bench path already asserted per-cell wire transit via *_phy SerDes
counters; the RoCE path did not, so an on-chip eswitch loopback could inflate a
RoCE number above the 100GbE line rate undetected. Add the same check to the RoCE
path: a netns-aware phy_counter reads the server rx_packets_phy and client
tx_packets_phy inside the dq_wire_{server,client} loopback namespaces around each
cell.
Also harden the threshold on all three checks (RoCE, DPDK, multi-queue). A bare
">0" let a stray background/control packet pass an on-chip run as "wire OK" -- a
+22 phy delta slipped through on a 23M-message run. Require the phy delta to reach
~half the message/packet count before certifying wire transit.
Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: rgurunathan <rgurunathan@nvidia.com>
Three bench-harness fixes surfaced while re-measuring the DGX Spark sockets and multi-queue cells over the wire: - run_spark_bench.sh: generate_socket_yaml's local_addr/remote_addr port rewrite required a quoted "tcp://..." URI, but gen_spark_netns_config.py emits it unquoted (PyYAML strips the quotes), so the substitution silently missed. Every multi-pair TCP/UDP cell then reused one port -- both servers bound the same port -> 'Address already in use', a dead pair, and a wedged sweep. Make the quote optional in all three URI sed patterns. (Overlaps the fix/socket-netns-rx work.) - run_spark_mq_bench.sh: LD_LIBRARY_PATH pointed at $BUILD_DIR, but the per-engine sub-libs (libdaqiri_dpdk.so.0 etc.) live in $BUILD_DIR/src, so the bench failed with 'cannot open libdaqiri_dpdk.so.0'. Put $BUILD_DIR/src first. - plot_mq_payload_sweep.py: predated REPEATS>1 and plotted every rep as its own point; average reps per (cell, payload) so the line has one point per x. Assisted-by: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: rgurunathan <rgurunathan@nvidia.com>
The loopback cable is 100 GbE, so the report's >100 Gb/s figures could not have crossed the wire -- they were on-chip eswitch hairpin (and/or a since-swapped 200 Gbps cable). Re-measured every cell over the phy-verified p0<->p1 wire loop (rx_packets_phy confirmed to advance) and corrected the numbers: - RoCE SEND: 106.0 -> 97.6 peak (8MB/1MB/64KB all ~97; small cells unchanged). - DPDK sweep: 105.5@4KB -> ~98.8 peak; 1KB 92 -> 97 (stale low); best case now ~98.8 @ 4-8KB. - Multi-queue: the '2nd TX core scales to 108.8' story was pure on-chip -- over the wire one queue already saturates (~97-99), and cores only help small, packet-rate-bound payloads via RX (64B 20->27, 256B 50->66). Section rewritten, payload-sweep plot re-rendered. - GPU workload: RoCE none 108.0 -> 96.6 (now ~= DPDK 98.7); table + prose redone. - Sockets (re-run after the port-collision fix): TCP 4-pair 97.2 -> 90.6, UDP 4-pair 29.8 -> 28.5. - SUT table now states the 100 GbE link rate. Assisted-by: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: rgurunathan <rgurunathan@nvidia.com>
…eams The socket sweep co-pinned a pair's send (client) and receive (server) processes on one core. A shared send/receive core ping-pongs a single stream and can wedge it at half rate for an entire run -- the 8 KB (~1 MSS) cells were bimodal (~15 vs ~30 Gb/s, std +-8.6). Pin the two sides to SEPARATE isolated cores, kept in the SAME CPU cluster (pairs 0-1 in 15-19, pairs 2-3 in 5-9): straddling GB10's two big-core clusters makes every payload pay cross-cluster cache-coherence latency and regresses large messages, so intra-cluster placement is required. Single-stream variance drops sharply; the only cost is the 4-pair cells, whose eight cores must span both clusters (the 5-9 pairs sit farther from the NIC), so they scale slightly sub-linearly. Add SOCKET_NOPIN=1 (cpu_core -1, unpinned) for the pinned-vs-scheduler comparison; not used in the report. Refresh the TCP/UDP tables, summary rows, methodology prose, and the SUT core-pinning row in performance-dgx-spark.md to the re-measured split numbers. Assisted-by: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: rgurunathan <rgurunathan@nvidia.com>
| | Socket / RoCE (SEND) | 64 KB message | **97.6 ±0.1 Gb/s** | 0 | Single QP, batch 1 | | ||
| | Socket / TCP | 1 MiB × 4 pairs | **90.6 ±1.5 Gb/s** | ~0 | Flow-controlled (App TX = App RX) | | ||
| | Socket / UDP | 8 KB × 4 pairs | **28.5 ±0.6 Gb/s** | ~54% loss | Receiver goodput; unpaced sender | | ||
| | Socket / TCP | 8 KB × 4 pairs | **87.3 ±2.2 Gb/s** | ~0 | Flow-controlled (App TX = App RX) | |
Collaborator
There was a problem hiding this comment.
These numbers are still backwards to me. We probably should investigate at some point
chloecrozier
added a commit
that referenced
this pull request
Jul 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Clean up of various RoCE bench issues on DGX Spark:
bidirectionally; make it one-way by default (matching DPDK and sockets) so the
numbers are apples-to-apples, and re-measure the perf report.
GPU every message; recycle recv buffers with CUDA events so the receiver keeps
reaping completions instead.
matching FFT-length knob) so the FLOP count per call is constant across the
message-size sweep.
by
phys_port_name, robust to PCIe re-enumeration.