diff --git a/docs/api-reference/configuration.md b/docs/api-reference/configuration.md
index 3e69b69..c49323b 100644
--- a/docs/api-reference/configuration.md
+++ b/docs/api-reference/configuration.md
@@ -54,7 +54,7 @@ These settings apply globally to both TX and RX:
## Memory Regions
-`memory_regions:` — List of regions where packet buffers are stored. The number of regions
+`memory_regions:` List of regions where packet buffers are stored. The number of regions
and their `kind` determines the receive mode (CPU-only, header-data split, or batched GPU).
- **`name`**: Memory region name. Referenced by queue configurations.
@@ -62,13 +62,13 @@ and their `kind` determines the receive mode (CPU-only, header-data split, or ba
- **`kind`**: Memory type.
- type: `string`
- values:
- - `huge` — CPU hugepages (recommended for CPU buffers)
- - `device` — GPU VRAM (discrete GPUs only; requires GPUDirect via peermem or DMA-BUF)
- - `host_pinned` — Pinned CPU pages allocated via `cudaHostAlloc`. **Recommended on
+ - `huge`: CPU hugepages (recommended for CPU buffers)
+ - `device`: GPU VRAM (discrete GPUs only, requires GPUDirect via peermem or DMA-BUF)
+ - `host_pinned`: Pinned CPU pages allocated via `cudaHostAlloc`. **Recommended on
integrated GPUs (e.g. NVIDIA GB10 / DGX Spark)**, where the NIC cannot peer-DMA
into device memory and CUDA reports DMA-BUF unsupported. On discrete-GPU systems,
prefer `device` for high-throughput RX/TX paths.
- - `host` — Regular CPU memory (not recommended)
+ - `host`: Regular CPU memory (not recommended)
- **`affinity`**: GPU ID for `device` memory, or NUMA node ID for CPU memory.
- type: `integer`
- **`access`**: Memory access permissions.
@@ -87,7 +87,7 @@ and their `kind` determines the receive mode (CPU-only, header-data split, or ba
### Example: Header-Data Split
-Two regions — a small CPU region for headers and a GPU region for payload:
+Two regions, a small CPU region for headers and a GPU region for payload:
```yaml
memory_regions:
@@ -109,7 +109,7 @@ memory_regions:
## Interfaces
-`interfaces:` — List of NIC interfaces to configure.
+`interfaces:` List of NIC interfaces to configure.
- **`name`**: Interface name. Used to look up port IDs at runtime via `get_port_id()`.
- type: `string`
@@ -158,7 +158,7 @@ engine.
### Queues
-`rx.queues:` — List of receive queues on the interface.
+`rx.queues:` List of receive queues on the interface.
- **`name`**: Queue name.
- type: `string`
@@ -168,11 +168,11 @@ engine.
performance.
- type: `string`
- **`batch_size`**: Number of packets per batch passed from the NIC to the application. Larger
- values increase throughput; smaller values reduce latency.
+ values increase throughput, and smaller values reduce latency.
- type: `integer`
- **`memory_regions`**: List of memory region names (defined in [Memory Regions](#memory-regions)).
The order determines segment mapping: first region = segment 0, second = segment 1, etc.
- A single region means all packet data lands in one place; two regions enables header-data
+ A single region means all packet data lands in one place, while two regions enables header-data
split.
- type: `list`
- **`timeout_us`**: Timeout in microseconds. A partial batch is delivered if this time elapses
@@ -182,11 +182,11 @@ engine.
### Flex Items
-`rx.flex_items:` — Flexible parser items for custom flow matching beyond standard UDP fields.
+`rx.flex_items:` Flexible parser items for custom flow matching beyond standard UDP fields.
- **`name`**: Name of the flex item.
- type: `string`
-- **`id`**: ID of the flex item. Scoped per interface; the same numeric ID on two
+- **`id`**: ID of the flex item. Scoped per interface. The same numeric ID on two
interfaces may refer to different parser settings.
- type: `integer`
- **`offset`**: Byte offset after the UDP header where matching begins. Must be a multiple
@@ -197,8 +197,8 @@ engine.
### Flows
-`rx.flows:` — Static startup flow rules that steer packets to specific queues based on
-match criteria. This sequence may be omitted; a queues-only RX config can add DPDK RX
+`rx.flows:` Static startup flow rules that steer packets to specific queues based on
+match criteria. This sequence may be omitted, and a queues-only RX config can add DPDK RX
flows later with the dynamic flow API. For Raw Ethernet on the DPDK and ibverbs engines,
RX flows can also perform hardware VLAN pop or tunnel decapsulation before queue delivery.
@@ -236,7 +236,7 @@ RX flows can also perform hardware VLAN pop or tunnel decapsulation before queue
- **`mask`**: 32-bit mask applied before matching (with flex items).
- type: `integer`
- **`ecpri`**: eCPRI-over-Ethernet match (EtherType `0xAEFE`). Presence of this map selects
- the eCPRI flow class; the EtherType is matched implicitly. Cannot be combined with UDP/IP
+ the eCPRI flow class. The EtherType is matched implicitly. Cannot be combined with UDP/IP
or flex-item matching. A flow with an empty `ecpri: {}` map matches all eCPRI frames.
- **`msg_type`**: eCPRI common-header message type (e.g. `0` = IQ data, `2` = real-time
control). Optional.
@@ -257,14 +257,14 @@ firmware steering, so any interface with eCPRI flows is automatically switched t
`dv_flow_en=1` (logged as a warning); a side effect is that the async/template dynamic-RX-flow
API is unavailable on that interface. The `ibverbs` engine has no such restriction.
-A single RX interface must use exactly one flow class — standard UDP/IP, flex-item, or eCPRI.
+A single RX interface must use exactly one flow class: standard UDP/IP, flex-item, or eCPRI.
Each class installs its own DPDK group-0 jump rule, and these conflict when mixed, so only one
class is reachable per interface. `daqiri_init` rejects mixed configs with a clear error.
Flex-item flows cannot be combined with VLAN/tunnel transform actions in v1.
### Flow Isolation
-`rx.flow_isolation:` — When `true`, only packets matching an explicit flow rule are delivered
+`rx.flow_isolation:` When `true`, only packets matching an explicit flow rule are delivered
to the application. Static startup flows install send-to-kernel fallback rules per flow class
(standard, flex-item, or eCPRI), so unmatched traffic in those classes is steered back to the Linux
kernel. Queues-only configs can set `flow_isolation: true` and then install dynamic RX flows
@@ -280,9 +280,9 @@ batch.
### Dynamic Flow Capacity
-`rx.dynamic_flow_capacity:` — DPDK template-table capacity reserved for dynamic RX flow
+`rx.dynamic_flow_capacity:` DPDK template-table capacity reserved for dynamic RX flow
rules on this interface. `0` disables DPDK template/async setup on startup. Set a positive
-value to opt in to the template fast path when it is available; legacy fallback paths still
+value to opt in to the template fast path when it is available. Legacy fallback paths still
accept dynamic RX flow operations but do not use a template table.
- type: `integer`
@@ -290,7 +290,7 @@ accept dynamic RX flow operations but do not use a template table.
### Hardware Timestamps
-`rx.hardware_timestamps:` — Enable per-packet hardware RX timestamps for Raw Ethernet
+`rx.hardware_timestamps:` Enable per-packet hardware RX timestamps for Raw Ethernet
(`stream_type: "raw"`).
When enabled, DAQIRI requires `RTE_ETH_RX_OFFLOAD_TIMESTAMP` support from the NIC/PMD and
initialization fails if DAQIRI cannot provide nanosecond timestamps for the selected PMD.
@@ -302,7 +302,7 @@ clock domain, not wall-clock time.
### RX Reorder Configs
-`rx.reorder_configs:` — Optional automatic packet reordering/aggregation plans. Implemented
+`rx.reorder_configs:` Optional automatic packet reordering/aggregation plans. Implemented
for Raw Ethernet (`stream_type: "raw"`) only in v1. GPU reorder requires CUDA-addressable
packet buffers (`device` or `host_pinned` memory regions). CPU reorder requires CPU-addressable
packet buffers (`host`, `host_pinned`, or `huge` memory regions).
@@ -330,7 +330,7 @@ v1 batch-size requirement:
- values: `gpu`, `cpu`
- **`memory_region`**: Output memory region where reordered payload is written.
- type: `string`
- - requirements: for `gpu`, must reference a `device` or `host_pinned` memory region; for
+ - requirements: for `gpu`, must reference a `device` or `host_pinned` memory region. For
`cpu`, must reference a `host`, `host_pinned`, or `huge` memory region
- **`payload_byte_offset`**: Byte offset in each packet where copied payload starts. Bytes before
this offset are skipped.
@@ -380,7 +380,7 @@ daqiri::set_reorder_cuda_stream("rx_port", "rx_reorder_0", stream);
### Queues
-`tx.queues:` — List of transmit queues on the interface.
+`tx.queues:` List of transmit queues on the interface.
- **`name`**: Queue name.
- type: `string`
@@ -390,7 +390,7 @@ daqiri::set_reorder_cuda_stream("rx_port", "rx_reorder_0", stream);
performance.
- type: `string`
- **`batch_size`**: Number of packets per batch sent to the NIC. Larger values increase
- throughput; smaller values reduce latency.
+ throughput, and smaller values reduce latency.
- type: `integer`
- **`memory_regions`**: List of memory region names. Same segment mapping rules as RX.
- type: `list`
@@ -400,7 +400,7 @@ daqiri::set_reorder_cuda_stream("rx_port", "rx_reorder_0", stream);
- values: `tx_eth_src` (auto-fill source MAC address)
- **`pacing_mbps`**: Packet-pacing rate cap for this queue, in megabits per second of L2 frame
bytes (the data the application transmits, excluding preamble/IFG/FCS). The NIC meters the queue
- out so its long-run average TX rate stays at or below this value; the limit is enforced on an
+ out so its long-run average TX rate stays at or below this value. The limit is enforced on an
average basis and idle gaps do not accumulate burst credit. `0` (the default) disables pacing
and sends at line rate. Supported only by the default `dpdk` raw engine on a NIC with hardware
send scheduling (ConnectX-7 or later); on devices without it, `pacing_mbps` is ignored with a
@@ -411,9 +411,9 @@ daqiri::set_reorder_cuda_stream("rx_port", "rx_reorder_0", stream);
### Transmit Flows
-`tx.flows:` — Raw Ethernet hardware transform rules for outgoing packets. Supported on
+`tx.flows:` Raw Ethernet hardware transform rules for outgoing packets. Supported on
the DPDK and ibverbs raw engines only. TX flows match the packet as supplied by the
-application, then push or encapsulate headers in hardware; the application buffer remains
+application, then push or encapsulate headers in hardware. The application buffer remains
the pre-encap packet.
- **`name`** / **`id`**: Flow label and ID.
@@ -435,12 +435,12 @@ the pre-encap packet.
DAQIRI validates transform overhead against the configured packet buffer size and
the supported jumbo-frame bound. For RX decap/pop and TX encap/push, MTU sizing
-accounts for the outer wire frame; packet buffers hold the post-decap (RX) /
+accounts for the outer wire frame. Packet buffers hold the post-decap (RX) /
pre-encap (TX) frame.
### Accurate Send
-`tx.accurate_send:` — Enable hardware-timed packet transmission using PTP timestamps. When
+`tx.accurate_send:` Enable hardware-timed packet transmission using PTP timestamps. When
enabled, use `set_packet_tx_time()` to schedule packets. Requires ConnectX-7 or later.
- type: `boolean`
diff --git a/docs/api-reference/cpp.md b/docs/api-reference/cpp.md
index e11f73a..ad43987 100644
--- a/docs/api-reference/cpp.md
+++ b/docs/api-reference/cpp.md
@@ -71,7 +71,7 @@ if (st != daqiri::Status::SUCCESS) {
## Receiving Packets
-### RX Step 1 — Get a burst
+### RX Step 1: Get a burst
```cpp
daqiri::BurstParams *burst;
@@ -92,7 +92,7 @@ daqiri::get_rx_burst(&burst, 0);
daqiri::get_rx_burst(&burst);
```
-### RX Step 2 — Access packet data
+### RX Step 2: Access packet data
For a single-segment configuration (CPU-only or batched GPU):
@@ -129,7 +129,7 @@ for (int i = 0; i < daqiri::get_num_packets(burst); i++) {
}
```
-### RX Step 3 — Free buffers
+### RX Step 3: Free buffers
When you are done processing, free the burst to return buffers to the pool:
@@ -232,7 +232,7 @@ Packets matching a dynamic rule are marked with the same `FlowId` returned by
the add completion, so `get_packet_flow_id()` gives the handle to pass to
`delete_flow_async()`. `poll_flow_op()` returns `Status::NOT_READY` when no flow
operation has completed yet. A dynamic flow is deletable only after its add
-completion has been polled successfully; deleting a flow that is still pending returns
+completion has been polled successfully. Deleting a flow that is still pending returns
`Status::INVALID_PARAMETER`.
Multiple RX flows can be added as one operation. On DPDK this maps to a single
@@ -312,7 +312,7 @@ if ((burst->hdr.hdr.burst_flags & daqiri::DAQIRI_BURST_FLAG_REORDERED) != 0U) {
### GPU packet processing on reordered bursts
-When using batched GPU mode, packets arrive in CUDA-addressable buffers — each at an
+When using batched GPU mode, packets arrive in CUDA-addressable buffers, each at an
arbitrary GPU address. Launch your own CUDA work directly on the packet pointers. Packet
reordering and aggregation should be configured through `rx.reorder_configs`; see
`raw_reorder_seq_bench.cpp` and `raw_reorder_quantize_bench.cpp` for complete examples
@@ -334,7 +334,7 @@ daqiri::free_all_packets_and_burst_rx(burst);
## Transmitting Packets
-### TX Step 1 — Allocate a burst
+### TX Step 1: Allocate a burst
```cpp
auto burst = daqiri::create_tx_burst_params();
@@ -342,7 +342,7 @@ daqiri::set_header(burst, port_id, queue_id, batch_size, num_segments);
auto status = daqiri::get_tx_packet_burst(burst);
if (status != daqiri::Status::SUCCESS) {
- // No buffers available — retry later
+ // No buffers available, retry later
}
```
@@ -363,7 +363,7 @@ daqiri::set_connection_id(burst, conn_id);
auto rx_conn_id = daqiri::get_connection_id(rx_burst);
```
-### TX Step 2 — Fill packets
+### TX Step 2: Fill packets
Use the header helper functions for standard UDP packets:
@@ -380,7 +380,7 @@ for (int i = 0; i < daqiri::get_num_packets(burst); i++) {
Or construct raw packets by writing directly into the packet buffer returned by
`get_packet_ptr()`.
-### TX Step 3 — Send
+### TX Step 3: Send
```cpp
daqiri::send_tx_burst(burst);
@@ -535,7 +535,7 @@ copies each packet's post-offset logical bytes into owned host staging memory
before submission, so the burst may be released after
`daqiri_write_raw_to_s3_objects_async()` succeeds. Header-data split and other
multi-segment packets are concatenated into one object. The first S3 version
-uses one single-part `PutObject` per packet; objects larger than 5 GiB return
+uses one single-part `PutObject` per packet. Objects larger than 5 GiB return
`NOT_SUPPORTED`, and multipart/burst aggregation is future work.
The Python bindings expose the same C++ writer when both
@@ -721,7 +721,7 @@ workflow sections above show the common call order and ownership rules.
| Function | Purpose |
| --- | --- |
| `get_mac_addr(port, mac)` | Copy a port MAC address into a six-byte buffer. |
-| `format_eth_addr(dst, addr)` | Convert a `xx:xx:xx:xx:xx:xx` MAC string into a six-byte buffer; invalid input zeroes the buffer. |
+| `format_eth_addr(dst, addr)` | Convert a `xx:xx:xx:xx:xx:xx` MAC string into a six-byte buffer. Invalid input zeroes the buffer. |
| `get_port_id(key)` | Resolve an interface name or PCIe address to a port ID. |
| `get_num_rx_queues(port_id)` | Return the configured or engine-reported RX queue count. |
| `drop_all_traffic(port)` | Install a high-priority drop rule on a port. |
diff --git a/docs/api-reference/index.md b/docs/api-reference/index.md
index f6b411c..b71726e 100644
--- a/docs/api-reference/index.md
+++ b/docs/api-reference/index.md
@@ -58,20 +58,20 @@ The typical DAQIRI application lifecycle has six steps:
Each step maps directly to a section of the
[C++ API Usage](cpp.md) and [Python API Usage](python.md) pages. The
-buffer-release step in particular is load-bearing — see
+buffer-release step in particular is load-bearing. See
[Zero-Copy Ownership](../concepts.md#zero-copy-ownership) on the
Concepts page for why missed frees cause queue drops.
## See also
-- [Concepts](../concepts.md) — terminology and background for everything
+- [Concepts](../concepts.md): terminology and background for everything
this page references.
-- [Configuration YAML Reference](configuration.md) — every YAML key, its
+- [Configuration YAML Reference](configuration.md): every YAML key, its
type, and its valid values.
-- [C++ API Usage](cpp.md) — initialization, RX/TX workflows, buffer
+- [C++ API Usage](cpp.md): initialization, RX/TX workflows, buffer
lifecycle, file writing, utilities, and the full C++ function reference.
-- [Python API Usage](python.md) — the same workflow through the pybind11
+- [Python API Usage](python.md): the same workflow through the pybind11
bindings, including GIL behavior, tuple return shapes, and the full
Python function reference.
-- [Configuration walkthrough tutorial](../tutorials/configuration-walkthrough.md)
- — annotated YAML walkthrough with a use-case decision tree.
+- [Configuration walkthrough tutorial](../tutorials/configuration-walkthrough.md):
+ annotated YAML walkthrough with a use-case decision tree.
diff --git a/docs/api-reference/python.md b/docs/api-reference/python.md
index 4e9c1c5..f7dcb7c 100644
--- a/docs/api-reference/python.md
+++ b/docs/api-reference/python.md
@@ -19,7 +19,7 @@ region*, *zero-copy ownership*, *RX reorder*), keep the
Functions that can fail return a `daqiri.Status` value, or a tuple whose first
element is a `Status` and whose remaining elements are output values. Packet
storage is owned by DAQIRI, so received and transmitted bursts must be explicitly
-freed — see
+freed. See
[Concepts → Zero-Copy Ownership](../concepts.md#zero-copy-ownership) for why a
missed free causes RX drops.
@@ -28,9 +28,9 @@ For a complete executable Python benchmark, see
## Building and Importing
-Follow the build flow in [Getting Started](../getting-started.md) — container
-or bare-metal — and add `-DDAQIRI_BUILD_PYTHON=ON` to the CMake configure step
-(the default is `OFF`). The container path is recommended; it already provides
+Follow the build flow in [Getting Started](../getting-started.md), container
+or bare-metal, and add `-DDAQIRI_BUILD_PYTHON=ON` to the CMake configure step
+(the default is `OFF`). The container path is recommended, and it already provides
`pybind11` and the rest of the build dependencies. For example, inside the
container:
@@ -154,7 +154,7 @@ if status == daqiri.Status.SUCCESS:
If GPU RX `reorder_configs` are configured for the DPDK engine, set one CUDA
stream per GPU reorder plan before pulling reordered bursts. Pass the CUDA
-stream as an integer address; pass `0` to use the default stream. See the
+stream as an integer address. Pass `0` to use the default stream. See the
[Configuration YAML Reference](configuration.md#rx-reorder-configs)
for reorder configuration constraints.
@@ -166,10 +166,10 @@ if status != daqiri.Status.SUCCESS:
## Receiving Packets
-### RX Step 1 — Get a burst
+### RX Step 1: Get a burst
`get_rx_burst()` is non-blocking. It returns `(Status, BurstParams | None)`.
-`Status.SUCCESS` means a burst is ready; other statuses (including `NULL_PTR`)
+`Status.SUCCESS` means a burst is ready. Other statuses (including `NULL_PTR`)
mean no burst is ready yet or an error occurred.
```python
@@ -186,7 +186,7 @@ status, burst = daqiri.get_rx_burst() # any queue on any port
status, burst = daqiri.get_rx_burst_for_connection(conn_id, server=True)
```
-### RX Step 2 — Access packet data
+### RX Step 2: Access packet data
For a single-segment burst (CPU-only or batched GPU), copy data out with
`get_packet_bytes`:
@@ -219,9 +219,9 @@ RX hardware timestamps are available only when the DPDK engine is configured
with `rx.hardware_timestamps: true` and the NIC supports
`RTE_ETH_RX_OFFLOAD_TIMESTAMP`. See
[C++ API Usage → Receiving Packets](cpp.md#receiving-packets) for the clock
-semantics; the Python wrapper exposes the same timestamps in nanoseconds.
+semantics. The Python wrapper exposes the same timestamps in nanoseconds.
-### RX Step 3 — Free buffers
+### RX Step 3: Free buffers
When you are done with a burst, free it:
@@ -246,7 +246,7 @@ transfer ownership to Python.
Prefer the copy helpers unless the application deliberately needs raw addresses
for CUDA or foreign-function interop. The copy helpers handle both CPU pointers
-and CUDA device pointers; device copies use `cudaMemcpy` internally.
+and CUDA device pointers. Device copies use `cudaMemcpy` internally.
```python
status = daqiri.copy_buffer_to_segment_packet(
@@ -276,8 +276,8 @@ This section covers how to consume reordered bursts from Python.
Reordered RX bursts are identified by flags on `burst.hdr.hdr.burst_flags`:
-- `DAQIRI_BURST_FLAG_REORDERED` — burst contains one aggregated reorder buffer.
-- `DAQIRI_BURST_FLAG_REORDER_TIMEOUT` — the aggregate was emitted by the
+- `DAQIRI_BURST_FLAG_REORDERED`: burst contains one aggregated reorder buffer.
+- `DAQIRI_BURST_FLAG_REORDER_TIMEOUT`: the aggregate was emitted by the
timeout path rather than by filling the configured `packets_per_batch`.
For reordered bursts, `burst.hdr.hdr.max_pkt` is the logical number of source
@@ -302,7 +302,7 @@ any.
## Transmitting Packets
-### TX Step 1 — Allocate a burst
+### TX Step 1: Allocate a burst
```python
port_id = daqiri.get_port_id("tx_port")
@@ -321,11 +321,11 @@ else:
# retry later
```
-### TX Step 2 — Fill packets
+### TX Step 2: Fill packets
Header helpers are available when DAQIRI is responsible for filling packet
headers. `set_ipv4_header()` takes the source and destination hosts as
-**integers**, not dotted-quad strings — convert with `ipaddress.IPv4Address` (or
+**integers**, not dotted-quad strings. Convert with `ipaddress.IPv4Address` (or
your own packer) before calling:
```python
@@ -349,11 +349,11 @@ for idx in range(daqiri.get_num_packets(burst)):
status = daqiri.set_packet_lengths(burst, idx, [len(packet)])
```
-### TX Step 3 — Send
+### TX Step 3: Send
```python
status = daqiri.send_tx_burst(burst)
-# Do not free `burst` after a SUCCESS or NO_SPACE_AVAILABLE return — see the ownership note below.
+# Do not free `burst` after a SUCCESS or NO_SPACE_AVAILABLE return. See the ownership note below.
```
`send_tx_burst()` takes ownership of the burst on success and on a full-ring
@@ -390,7 +390,7 @@ status = daqiri.daqiri_write_pcap_to_file(
CUDA device-backed packet segments require `DAQIRI_ENABLE_GDS=ON` and working
NVIDIA cuFile support. See
[C++ API Usage → Writing Bursts to Storage](cpp.md#writing-bursts-to-storage)
-for the full GDS constraints; the asynchronous file-write API is C++-only at
+for the full GDS constraints. The asynchronous file-write API is C++-only at
this time.
## Socket and RDMA
@@ -479,16 +479,16 @@ The bindings release the Python GIL around blocking or long-running DAQIRI
calls where the pybind11 wrapper declares `py::gil_scoped_release`. This
currently includes:
-- `daqiri_init` — all input forms (YAML path/string, dict, `NetworkConfig`,
+- `daqiri_init`: all input forms (YAML path/string, dict, `NetworkConfig`,
`value`/`as_dict` config-like objects). The GIL is released around the
- underlying DAQIRI call; any Python-side conversion (PyYAML `dump`, `as_dict()`
+ underlying DAQIRI call. Any Python-side conversion (PyYAML `dump`, `as_dict()`
invocation) still runs with the GIL held.
- `daqiri_init_from_yaml_string` and `daqiri_init_from_yaml_file`
- `get_rx_burst` (all overloads) and `get_rx_burst_for_connection`
- `is_tx_burst_available`, `get_tx_packet_burst`, `send_tx_burst`
-- `copy_buffer_to_packet`, `copy_buffer_to_segment_packet` — released around
+- `copy_buffer_to_packet`, `copy_buffer_to_segment_packet`: released around
the underlying memory copy (host or `cudaMemcpy`).
-- `get_packet_bytes`, `get_segment_packet_bytes` — released around the copy
+- `get_packet_bytes`, `get_segment_packet_bytes`: released around the copy
out of the DAQIRI buffer.
- `daqiri_write_raw_to_file`, `daqiri_write_pcap_to_file`
- `synchronize_burst_event`
@@ -604,7 +604,7 @@ The workflow sections above show the common call order and ownership rules.
| Function | Purpose |
| --- | --- |
| `get_mac_addr(port)` | Return `(Status, "aa:bb:cc:dd:ee:ff")`. |
-| `format_eth_addr(addr)` | Return six MAC-address bytes from a `xx:xx:xx:xx:xx:xx` MAC string; invalid input returns zero bytes. |
+| `format_eth_addr(addr)` | Return six MAC-address bytes from a `xx:xx:xx:xx:xx:xx` MAC string. Invalid input returns zero bytes. |
| `get_port_id(key)` | Resolve an interface name or PCIe address to a port ID. |
| `get_num_rx_queues(port_id)` | Return configured RX queue count for a port. |
| `drop_all_traffic(port)` | Install a high-priority drop rule on a port. |
@@ -623,7 +623,7 @@ The workflow sections above show the common call order and ownership rules.
| `rdma_get_server_conn_id(server_addr, server_port)` | Return `(Status, conn_id)`. |
Dynamic RX flows are RX-only in v1. The `action` attribute remains the single queue-action
-shorthand; use ordered `actions` when a raw DPDK or raw ibverbs dynamic rule needs hardware
+shorthand. Use ordered `actions` when a raw DPDK or raw ibverbs dynamic rule needs hardware
VLAN pop or VXLAN/GRE/NVGRE decapsulation before the final queue action. Static TX
encapsulation/push rules are configured in YAML under `tx.flows`.
@@ -676,7 +676,7 @@ encapsulation/push rules are configured in YAML under `tx.flows`.
All exposed classes have a default constructor and read/write attributes.
Config classes mirror the C++ configuration structs, with Python attribute
names that mostly omit the trailing underscore from the C++ member name (e.g.
-`name_` → `name`). A few fields are renamed for clarity — for example `mrs_`
+`name_` → `name`). A few fields are renamed for clarity, for example `mrs_`
→ `memory_regions` and `ifs_` → `interfaces`.
| Class | Purpose |
diff --git a/docs/benchmarks/index.md b/docs/benchmarks/index.md
index bc9c8fd..a2bd1f2 100644
--- a/docs/benchmarks/index.md
+++ b/docs/benchmarks/index.md
@@ -17,7 +17,7 @@ DAQIRI ships with several stream types to handle different types of incoming and
| Use case | DAQIRI config | Benchmark | Start here |
|---|---|---|---|
-| Ingest from or egress to a programmable PCIe sensor, such as an FPGA on the PCIe bus. | `stream_type: "pcie"` | — | Under development (see note below). |
+| Ingest from or egress to a programmable PCIe sensor, such as an FPGA on the PCIe bus. | `stream_type: "pcie"` | n/a | Under development (see note below). |
| Compare against normal Linux networking, run on a non-NVIDIA NIC, or test a peer that speaks TCP/UDP sockets. | `stream_type: "socket"` with `tcp://` or `udp://` endpoints | `daqiri_bench_socket` | [Socket and RDMA Benchmarking](socket_benchmarking.md) |
| Test a peer that already implements RDMA verbs over RoCE. | `stream_type: "socket"` and `roce://` endpoints | `daqiri_bench_rdma` | [Socket and RDMA Benchmarking](socket_benchmarking.md#run-the-rdma-roce-benchmark) |
| Drive raw Ethernet packets directly from an NVIDIA NIC under DAQIRI control. | `stream_type: "raw"` | `daqiri_bench_raw_gpudirect` and the other `raw_*` benches | [Raw Ethernet Benchmarking](raw_benchmarking.md) |
@@ -52,7 +52,3 @@ DAQIRI ships with several stream types to handle different types of incoming and
- [Socket and RDMA Benchmarking](socket_benchmarking.md) covers Linux TCP/UDP and RoCE/RDMA runs with matching client/server namespace setup.
- [Raw Ethernet Benchmarking](raw_benchmarking.md) covers the DPDK/raw Ethernet examples, hugepage sizing, physical loopback configuration, and raw benchmark troubleshooting.
- [Understanding the Configuration File](../tutorials/configuration-walkthrough.md) explains the YAML fields once you have selected the stream type and example config.
-
----
-**Previous:** [System Configuration](../tutorials/system_configuration.md)
-**Next:** [Socket and RDMA Benchmarking](socket_benchmarking.md)
diff --git a/docs/benchmarks/performance-dgx-spark.md b/docs/benchmarks/performance-dgx-spark.md
index fdc0777..53d15b9 100644
--- a/docs/benchmarks/performance-dgx-spark.md
+++ b/docs/benchmarks/performance-dgx-spark.md
@@ -23,19 +23,19 @@ loopback). The exact commands are collected under [Reproduce](#reproduce) below.
| Component | Detail |
| --------- | ------ |
-| Platform | DGX Spark (GB10), 20 cores, isolcpus `16-19` (the multi-queue sweep expands this; see [Multi-queue core scaling](#multi-queue-core-scaling)) |
+| Platform | DGX Spark (GB10), 20 cores, isolcpus `16-19` (the multi-queue sweep expands this, see [Multi-queue core scaling](#multi-queue-core-scaling)) |
| NIC | ConnectX-7, ports p0 ↔ p1 cross-cabled with a **100 GbE QSFP28** loopback cable (single-host loopback), MTU 9000 |
| Build | Release (`-DCMAKE_BUILD_TYPE=Release`), `DAQIRI_ENGINE="dpdk ibverbs"` |
-| Loopback | Raw/DPDK uses the two physical ports directly; socket/RoCE use the `dq_wire_*` network-namespace wire loopback |
+| Loopback | Raw/DPDK uses the two physical ports directly, while socket/RoCE use the `dq_wire_*` network-namespace wire loopback |
| Core pinning | Each direction has a busy-spin queue poller and an app worker on separate isolated X925 cores (PR #149). Single-queue: DPDK pollers 17/18, workers 16/19. Multi-queue: TX pollers 16/19, RX pollers 18/9, each with its own worker core, master 8. Sockets pin each pair's send and receive to separate cores in the same CPU cluster (all with `isolcpus=5-9,15-19`). |
## Results Summary (C++ loopback)
Each transport at its best-case **operation size**. Raw/RoCE are
-single-stream; socket TCP/UDP scale with the number of client/server pairs, so the
+single-stream. Socket TCP/UDP scale with the number of client/server pairs, so the
four-pair aggregate is shown.
-The **100 GbE QSFP28 loopback cable sets the maximum data rate** here — not the
+The **100 GbE QSFP28 loopback cable sets the maximum data rate** here, not the
ConnectX-7 (which is rated for higher line rates) or the software path. A 100 GbE
link tops out near ~99.6 Gb/s of payload, so every large-transfer result saturates
just under that ceiling.
@@ -45,7 +45,7 @@ just under that ceiling.
| Raw Ethernet / GPUDirect | 8 KB packet | **98.8 ±0.1 Gb/s** | 0 | Flat ~98.7 across 4–8 KB, all batch sizes |
| Socket / RoCE (SEND) | 64 KB message | **97.6 ±0.1 Gb/s** | 0 | Single QP, batch 1 |
| Socket / TCP | 8 KB × 4 pairs | **87.3 ±2.2 Gb/s** | ~0 | Flow-controlled (App TX = App RX) |
-| Socket / UDP | 8 KB × 4 pairs | **34.5 ±0.6 Gb/s** | ~48% loss | Receiver goodput; unpaced sender |
+| Socket / UDP | 8 KB × 4 pairs | **34.5 ±0.6 Gb/s** | ~48% loss | Receiver goodput, unpaced sender |
Each transport is best read at its own best-case operation size (see the per-transport
tables below); a single cross-transport unit of work isn't meaningful here, since
@@ -61,7 +61,7 @@ Throughput is flat across batch size and stable run-to-run (3 reps per cell,
≤1% spread).
Achieved Gb/s measured at App RX (equal to App TX, since every cell is
-drop-free), unpaced, mean of 3 reps; run-to-run spread ≤0.5 Gb/s (<1%):
+drop-free), unpaced, mean of 3 reps. Run-to-run spread ≤0.5 Gb/s (<1%):
@@ -84,7 +84,7 @@ drop-free), unpaced, mean of 3 reps; run-to-run spread ≤0.5 Gb/s (<1%):
At ≥1 KB the link saturates near line rate (~97–99 Gb/s) regardless of batch.
Below that the path is packet-rate-bound: 256 B ~50 Gb/s (19.5 M pps), 64 B
-~20 Gb/s (20 M pps) — a ~20 M pps single-queue ceiling (the multi-queue
+~20 Gb/s (20 M pps), a ~20 M pps single-queue ceiling (the multi-queue
section lifts it). Gb/s here is the L2 frame rate including the 64 B header, so
pps ≈ Gb/s ÷ ((payload + 64) × 8). These small-payload cells are flat across batch
size and stable run-to-run. Because every cell is drop-free, the achieved rate is
@@ -94,14 +94,14 @@ also the no-drop rate: pacing the sender below it hits the target with zero drop
| Core | Busy% | Note |
| ------------------------ | ----: | ------------------------------------- |
-| Master (CPU 8) | 3.7% | Orchestration only; mostly idle |
+| Master (CPU 8) | 3.7% | Orchestration only, mostly idle |
| TX queue poller (CPU 17) | ~92% | Poll-mode busy-spin |
| RX queue poller (CPU 18) | ~92% | Poll-mode busy-spin |
The benchmark app workers run on their own cores (TX 16, RX 19) alongside these
-pollers; this run sampled only the poller cores.
-The pollers stay near 92% across every drop-curve step from 1 Gb/s to line rate —
-DPDK's poll-mode driver spins regardless of offered load. The GPU stays idle (SM
+pollers. This run sampled only the poller cores.
+The pollers stay near 92% across every drop-curve step from 1 Gb/s to line rate,
+because DPDK's poll-mode driver spins regardless of offered load. The GPU stays idle (SM
and memory-controller utilization both ~0%): it is a DMA target for the payload,
not a compute engine.
@@ -109,7 +109,7 @@ not a compute engine.
Each packet-handling core spins in poll-mode. At large payloads (≥1 KB) a single
queue already saturates the 100 GbE line (~97–99 Gb/s), so extra cores add
-nothing there — the multi-queue win is confined to the small,
+nothing there. The multi-queue win is confined to the small,
packet-rate-bound payloads, where **RX cores** are the lever. The matrix sweeps
(TX cores, RX cores) over `(1,1)`, `(1,2)`, `(2,1)`, `(2,2)`.
@@ -132,8 +132,8 @@ count matters); at ≥1 KB every cell converges at the wire ceiling regardless:
| (2,1) | 16,19 | 18 | 49.0 |
| (2,2) | 16,19 | 18,9 | 64.7 |
-A second **RX** core lifts 256 B from 50.0 to 66.4 Gb/s; a second **TX** core does
-nothing (49.0 ≈ 50.0). The full payload sweep shows why — the bottleneck depends
+A second **RX** core lifts 256 B from 50.0 to 66.4 Gb/s, while a second **TX** core does
+nothing (49.0 ≈ 50.0). The full payload sweep shows why, since the bottleneck depends
on payload size:
@@ -151,11 +151,11 @@ Generated by
RoCE SEND over the netns wire loopback, single queue-pair, batch 1. Throughput
is App RX goodput, equal to App TX with 0 drops. Large messages up to 64 KB
-saturate the wire; the smallest messages are bound by per-operation software
+saturate the wire, while the smallest messages are bound by per-operation software
overhead.
**Message-size sweep (single QP, batch 1, 0 drops).** Mean ± sample std over 3
-reps; run-to-run spread <1% in every cell.
+reps. Run-to-run spread <1% in every cell.
| Message size | Gb/s |
| ------------ | ---: |
@@ -180,12 +180,12 @@ sized to the message so the in-flight window stays full.
| -------------------- | ----: | ----------------------------------------------- |
| Master (CPU 8) | 0.7% | Orchestration only |
| Client TX (CPU 17) | 74.8% | Busy-spins posting sends and polling completions |
-| Server RX (CPU 19) | 1.1% | HCA DMAs straight to memory; worker only reaps completions |
+| Server RX (CPU 19) | 1.1% | HCA DMAs straight to memory, worker only reaps completions |
The TX core busy-spins in a post-and-poll loop, so its ~75% busy time is set by
that spin, not by the throughput: it stays near this level whether the link runs
at 10 or 100 Gb/s (the same reason the DPDK pollers sit near 92% regardless of
-offered load). The near-idle RX core is the expected RoCE RC signature — the HCA
+offered load). The near-idle RX core is the expected RoCE RC signature. The HCA
places incoming data directly into registered memory, so the receive worker only
reaps completions and reposts (~1% at this message rate). The GPU stays idle here
too (SM and memory-controller ~0%; DMA target, not a compute engine).
@@ -220,7 +220,7 @@ Throughput in Gb/s (App TX = App RX), mean ± std over 3 reps:
-Throughput scales with the pair count; retransmits stay negligible over the run.
+Throughput scales with the pair count, and retransmits stay negligible over the run.
At four pairs the eight cores span both clusters, and the pairs in 5–9 sit farther
from the NIC, so the 4-pair cells scale slightly sub-linearly.
@@ -228,7 +228,7 @@ from the NIC, so the 4-pair cells scale slightly sub-linearly.
Four one-way UDP client/server pairs, same per-side pinning (send and receive on
separate cores). UDP has no flow control, so each sender runs flat-out and the
-receiver drops whatever it cannot drain — the loss column is an inherent property
+receiver drops whatever it cannot drain, and the loss column is an inherent property
of unpaced UDP. App RX is the delivered goodput; App-level loss is
`(App TX − App RX) / App TX`.
@@ -252,9 +252,9 @@ Each cell shows **receiver goodput in Gb/s** (mean ± std over 3 reps) with the
The sweep stops at 8000 B (single Ethernet frame). Larger UDP datagrams
-fragment above the ~8972 B MTU payload; reassembly is all-or-nothing out of a
+fragment above the ~8972 B MTU payload. Reassembly is all-or-nothing out of a
shared per-namespace pool, so under multi-pair unpaced load delivery collapses
-(≈100% loss at 65507 B / 4 pairs). The wire itself is loss-free here; the loss
+(≈100% loss at 65507 B / 4 pairs). The wire itself is loss-free here. The loss
is host-side socket-buffer and reassembly pressure.
## GPU workloads in the receive path
@@ -264,25 +264,25 @@ the GPU also crunches the incoming data. The benchmarks accept
`--workload none|fft|gemm|gemm_fp16`, exposed by `run_spark_bench.sh` as the
`WORKLOAD` env var (recorded in the CSV `post_process` column); more workload kinds
can be added to the same reusable component over time. The workload runs on the
-received packet data — every backend first assembles the burst's
+received packet data. Every backend first assembles the burst's
payloads into one contiguous GPU buffer (a sequence-number **reorder** on the
out-of-order transports, an arrival-order **gather** on the in-order ones) and the
compute consumes that buffer.
-**What the two workloads compute** — both in **FP32** (single precision) — from the
+**What the two workloads compute**, both in **FP32** (single precision), from the
reusable component `examples/bench_workload.{h,cu}`:
-- **FFT** — a batched 1-D **complex-to-complex forward FFT** via cuFFT
+- **FFT**: a batched 1-D **complex-to-complex forward FFT** via cuFFT
(`cufftExecC2C`). The reordered buffer is treated as an array of single-precision
complex samples and transformed as many independent length-1024 FFTs, batched so
the transforms cover the whole reorder window. This models a streaming
- signal-processing receiver — channelization or spectral analysis that FFTs every
+ signal-processing receiver, such as channelization or spectral analysis that FFTs every
frame as it arrives.
-- **GEMM** — a dense **matrix multiply** `C = A·B` via cuBLAS on square *n×n*
+- **GEMM**: a dense **matrix multiply** `C = A·B` via cuBLAS on square *n×n*
matrices, with the reordered buffer supplying the *A* operand. The side length is
**pinned at n=1024** (`--workload-gemm-dim`, env `GEMM_DIM`), so every call is an
identical **2.15 GFLOP** matmul reading the first **4 MB** (n²·4 B, FP32) of each
- received unit — the compute is fixed regardless of message size, which is what
+ received unit. The compute is fixed regardless of message size, which is what
makes it comparable across transports. The matmul is FP32 (`cublasSgemm`). This models a
receiver feeding incoming data into a dense linear-algebra or neural-network
stage (beamforming, correlation, an inference layer).
@@ -304,12 +304,12 @@ compute overlaps ingest. The reorder window is sized so the contiguous buffer is
!!! note "Where the data lives, per backend"
On the integrated GB10 the GPU shares memory with the CPU, so the raw and RoCE
- receive buffers (`host_pinned`) are GPU-accessible with **no copy** — the
+ receive buffers (`host_pinned`) are GPU-accessible with **no copy**, and the
reorder/gather kernel reads them in place. Sockets are different: the kernel
hands received bytes to the application in pageable host memory, so the socket
- path must **stage each payload host→device** before the GPU can touch it — a
+ path must **stage each payload host→device** before the GPU can touch it, a
copy on the measured path that the raw/RoCE paths avoid. Lost packets (raw/UDP)
- leave their reorder slots zero-filled; the FLOP/copy volume is unchanged.
+ leave their reorder slots zero-filled, and the FLOP/copy volume is unchanged.
Fixed **n=1024**, one GEMM (or a length-1024 batched FFT) per received unit. DPDK runs
at an **8 KB payload** (~8 MB reorder window, 1024 packets × 8000 B), matched to RoCE's
@@ -342,7 +342,7 @@ export LD_LIBRARY_PATH=/opt/daqiri/lib:${LD_LIBRARY_PATH:-}
```
The base container does not ship the network tools the setup scripts and RoCE
-baseline depend on; install them first, or
+baseline depend on. Install them first, or
`scripts/setup_spark_wire_loopback_netns.sh` fails with `ip: command not found`:
```bash
@@ -355,11 +355,11 @@ These provide `ip`/`nstat` (`iproute2`), `ethtool`, and `ib_send_bw` (`perftest`
Each `run_spark_bench.sh ` invocation takes a **mode** that sets
which cells run: `sweep` runs the full payload × batch × pairs matrix (the
per-transport message-size tables above), while `smoke` runs just the single
-summary-table cell — one payload/batch/pairs operating point. `REPEATS=N` repeats
+summary-table cell, one payload/batch/pairs operating point. `REPEATS=N` repeats
every cell N times for error bars.
**Raw Ethernet / GPUDirect (DPDK)** drives the two physical ports directly, so
-the `dq_wire_*` namespaces must **not** be up — they capture the ports and
+the `dq_wire_*` namespaces must **not** be up, since they capture the ports and
hide them from DPDK. Tear them down first (no-op if they were never created).
`` below is the RX physical port (p1 in the p0→p1 loopback):
@@ -384,7 +384,7 @@ export ETH_DST_ADDR=$(cat /sys/class/net//address)
**Socket / RoCE and sockets** cross the cable through the `dq_wire_client` →
`dq_wire_server` namespaces. Bring the loopback up and confirm PHY counters move
-before running; tear it down when finished:
+before running, and tear it down when finished:
```bash
./scripts/setup_spark_wire_loopback_netns.sh up # create the namespaces
@@ -399,11 +399,11 @@ before running; tear it down when finished:
in the receive path by exporting `WORKLOAD` (`none` | `fft` | `gemm` |
`gemm_fp16`), run once per received I/O unit on the real payload. Each call is a
fixed **1024³ GEMM** (override with `GEMM_DIM` / `--workload-gemm-dim`) or a batched
-**length-1024 FFT** (override with `FFT_LEN` / `--workload-fft-len`) — both compute
+**length-1024 FFT** (override with `FFT_LEN` / `--workload-fft-len`). Both compute
sizes are held constant while the message size varies, so the FLOP count per call
is fixed. It composes with the same netns setup as above (dpdk in the default
-namespace, rdma in the `dq_wire_*` namespaces). Use `smoke` — the single
-summary-table cell that the fixed-n table reports — and run all three workloads
+namespace, rdma in the `dq_wire_*` namespaces). Use `smoke`, the single
+summary-table cell that the fixed-n table reports, and run all three workloads
with error bars:
```bash
@@ -423,6 +423,3 @@ Each run writes `bench-results/--/runs.csv`. See
[Socket and RDMA Benchmarking](socket_benchmarking.md) and
[Raw Ethernet Benchmarking](raw_benchmarking.md) for the namespace setup and
per-transport details.
-
----
-**Previous:** [Raw Ethernet Benchmarking](raw_benchmarking.md)
diff --git a/docs/benchmarks/raw_benchmarking.md b/docs/benchmarks/raw_benchmarking.md
index 6f63b3c..b9faabe 100644
--- a/docs/benchmarks/raw_benchmarking.md
+++ b/docs/benchmarks/raw_benchmarking.md
@@ -52,18 +52,18 @@ docker run --rm -it --privileged \
For systems configured per the [DGX Spark profile](../tutorials/system_configuration.md#dgx-spark-profile), use these configs to skip the PCIe/IP/CPU-core edits below:
- - [`daqiri_bench_raw_tx_rx_spark.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_tx_rx_spark.yaml) for `daqiri_bench_raw_gpudirect` — still set `eth_dst_addr` to the RX MAC. The rx_port is `0002:01:00.1` (physical port p1), so read its MAC: `cat /sys/class/net/enP2p1s0f1np1/address`. This p0-to-p1 pairing is intentional for an over-the-wire single-machine loopback; using two PFs that map to the same physical port exercises the on-chip eswitch path instead.
- - [`daqiri_bench_rdma_tx_rx_spark.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_rdma_tx_rx_spark.yaml) for `daqiri_bench_rdma` — no further edits needed.
+ - [`daqiri_bench_raw_tx_rx_spark.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_tx_rx_spark.yaml) for `daqiri_bench_raw_gpudirect`. Still set `eth_dst_addr` to the RX MAC. The rx_port is `0002:01:00.1` (physical port p1), so read its MAC: `cat /sys/class/net/enP2p1s0f1np1/address`. This p0-to-p1 pairing is intentional for an over-the-wire single-machine loopback. Using two PFs that map to the same physical port exercises the on-chip eswitch path instead.
+ - [`daqiri_bench_rdma_tx_rx_spark.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_rdma_tx_rx_spark.yaml) for `daqiri_bench_rdma`. No further edits needed.
- For the multi-queue core-scaling matrix, use the single base config [`daqiri_bench_raw_tx_rx_spark_mq.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_tx_rx_spark_mq.yaml) (the balanced TX=2/RX=2 superset) with `daqiri_bench_raw_gpudirect`, driven by [`run_spark_mq_bench.sh`](https://github.com/nvidia/daqiri/blob/main/examples/run_spark_mq_bench.sh) — it derives the four `(TX, RX)` cells from the base (via `scripts/gen_spark_mq_config.py`) and sweeps the payload.
+ For the multi-queue core-scaling matrix, use the single base config [`daqiri_bench_raw_tx_rx_spark_mq.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_tx_rx_spark_mq.yaml) (the balanced TX=2/RX=2 superset) with `daqiri_bench_raw_gpudirect`, driven by [`run_spark_mq_bench.sh`](https://github.com/nvidia/daqiri/blob/main/examples/run_spark_mq_bench.sh). It derives the four `(TX, RX)` cells from the base (via `scripts/gen_spark_mq_config.py`) and sweeps the payload.
The Spark configs also pin the benchmark application's `bench_tx.cpu_core` / `bench_rx.cpu_core` fields to the high-frequency Cortex-X925 cores. Keep both the DAQIRI queue cores and the application worker cores on cores 16-19 unless you intentionally want a lower-power core in the measurement.
#### Cross-host two-DGX-Spark loopback
-If you have two DGX Sparks cross-cabled p0↔p0 instead of a chassis QSFP loop on one machine, use the `_xhost` configs. Each host runs only its own role, so the YAML on each side configures one port instead of two. Both hosts must already be set up per the [DGX Spark profile](../tutorials/system_configuration.md#dgx-spark-profile), with one adjustment: the `daqiri-tx` (`1.1.1.1/24`) and `daqiri-rx` (`2.2.2.2/24`) nmcli profiles are *split across* the two hosts — bring up `daqiri-tx` on the TX host's p0 and `daqiri-rx` on the RX host's p0, instead of both on one box.
+If you have two DGX Sparks cross-cabled p0↔p0 instead of a chassis QSFP loop on one machine, use the `_xhost` configs. Each host runs only its own role, so the YAML on each side configures one port instead of two. Both hosts must already be set up per the [DGX Spark profile](../tutorials/system_configuration.md#dgx-spark-profile), with one adjustment: the `daqiri-tx` (`1.1.1.1/24`) and `daqiri-rx` (`2.2.2.2/24`) nmcli profiles are *split across* the two hosts. Bring up `daqiri-tx` on the TX host's p0 and `daqiri-rx` on the RX host's p0, instead of both on one box.
-**Network prerequisite (required for RDMA; recommended for raw).** Assigning `/24` addresses on each host is not enough for the kernel to reach the peer over a direct cable. RDMA-CM uses the kernel stack, so you need a host route and a static neighbor on the cabled port before ping or RoCE will work. Run [`scripts/setup_spark_xhost_net.sh`](https://github.com/nvidia/daqiri/blob/main/scripts/setup_spark_xhost_net.sh) on **both** hosts after bringing up the nmcli profile — see the [cross-host variant](../tutorials/system_configuration.md#cross-host-variant-two-sparks) in System Configuration for the full steps.
+**Network prerequisite (required for RDMA, recommended for raw).** Assigning `/24` addresses on each host is not enough for the kernel to reach the peer over a direct cable. RDMA-CM uses the kernel stack, so you need a host route and a static neighbor on the cabled port before ping or RoCE will work. Run [`scripts/setup_spark_xhost_net.sh`](https://github.com/nvidia/daqiri/blob/main/scripts/setup_spark_xhost_net.sh) on **both** hosts after bringing up the nmcli profile. See the [cross-host variant](../tutorials/system_configuration.md#cross-host-variant-two-sparks) in System Configuration for the full steps.
```bash
# TX host (peer MAC from RX: cat /sys/class/net/enp1s0f0np0/address)
@@ -212,7 +212,7 @@ bench_tx:
??? info "Show explanation"
- `eth_dst_addr` - the destination ethernet MAC address - will be embedded in the packet headers by the application. This is required here because the Rx interface above has `flow_isolation: true` (explained in more details below). In that configuration, only the packets listing the adequate destination MAC address will be accepted by the Rx interface.
- - `cpu_core` - the benchmark application's own TX worker thread affinity. Set the matching `bench_rx.cpu_core` for RX workers too; these app-thread fields are distinct from the DAQIRI queue `cpu_core` values that poll the NIC.
+ - `cpu_core` - the benchmark application's own TX worker thread affinity. Set the matching `bench_rx.cpu_core` for RX workers too. These app-thread fields are distinct from the DAQIRI queue `cpu_core` values that poll the NIC.
- We ignore the IP fields (`ip_src_addr`, `ip_dst_addr`) for now, as we are testing on a layer 2 network by just connecting a cable between the two interfaces on our system, therefore having mock values has no impact.
- You might have noted the lack of a `eth_src_addr` field in this `bench_tx` section. This is because the source Ethernet MAC address can be inferred automatically by the DAQIRI library from the PCIe address of the Tx interface referenced above.
@@ -243,26 +243,26 @@ By default the application runs for 10 seconds and then exits. You can change th
## Flow programming smoke test
Raw Ethernet flow rules are programmed into the NIC during `daqiri_init()`. Software loopback
-(`loopback: "sw"`) skips NIC init entirely, so it is a build/runtime smoke test only — not a
+(`loopback: "sw"`) skips NIC init entirely, so it is a build/runtime smoke test only, not a
flow programming test.
| Step | Command / action | Expected |
|------|------------------|----------|
-| Build smoke | `daqiri_bench_raw_sw_loopback.yaml --seconds 5` | Init succeeds; no NIC flows created |
+| Build smoke | `daqiri_bench_raw_sw_loopback.yaml --seconds 5` | Init succeeds, no NIC flows created |
| Good NIC config | `daqiri_bench_raw_tx_rx.yaml` (filled placeholders, cabled NIC) | Init succeeds; RX and `tx_eth_src` flows programmed |
| Dynamic RX flow config | `daqiri_example_dynamic_rx_flow.yaml` with `daqiri_example_dynamic_rx_flow` | Starts with `flow_isolation: true` and no `rx.flows`, drops unmatched traffic, then adds/deletes runtime UDP steering rules |
| Bad queue ID | Copy `daqiri_bench_raw_tx_rx.yaml`, set `flows[0].action.id: 99` | Fails in `validate_config()` before EAL/NIC init with `references unknown RX queue` |
-| Mixed flows (optional) | On one interface, add two of the three flow classes — standard UDP/IP, flex-item (see `rx.flex_items`), or eCPRI (`match.ecpri`) — in the [configuration reference](../api-reference/configuration.md) | Fails in `validate_config()` with `mixes standard (UDP/IP), flex-item and/or eCPRI` |
-| eCPRI flow (optional) | On a cabled NIC, add a flow with `match: { ecpri: { msg_type: 0, pc_id: 1 } }` (see the [configuration reference](../api-reference/configuration.md)) | Init succeeds; eCPRI-over-Ethernet (EtherType 0xAEFE) frames matching the message type and pc_id steer to the flow's queue |
+| Mixed flows (optional) | On one interface, add two of the three flow classes (standard UDP/IP, flex-item (see `rx.flex_items`), or eCPRI (`match.ecpri`)) in the [configuration reference](../api-reference/configuration.md) | Fails in `validate_config()` with `mixes standard (UDP/IP), flex-item and/or eCPRI` |
+| eCPRI flow (optional) | On a cabled NIC, add a flow with `match: { ecpri: { msg_type: 0, pc_id: 1 } }` (see the [configuration reference](../api-reference/configuration.md)) | Init succeeds. eCPRI-over-Ethernet (EtherType 0xAEFE) frames matching the message type and pc_id steer to the flow's queue |
## Cap the transmit rate with packet pacing
To meter the transmit side at a fixed rate in hardware, set a per-queue `pacing_mbps` cap
on the TX queue. [`daqiri_bench_raw_tx_rx_pacing.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_tx_rx_pacing.yaml)
is the loopback config above with `pacing_mbps: 10000` (10 Gbps) on the TX queue. Pacing is
-supported only on the default DPDK engine; the `ibverbs` engine does not support `pacing_mbps`
+supported only on the default DPDK engine. The `ibverbs` engine does not support `pacing_mbps`
(init fails if it is set on an `ibverbs` queue). The NIC meters the queue out so its average
-TX rate stays at or below the configured value; the limit is enforced on an average basis and
+TX rate stays at or below the configured value. The limit is enforced on an average basis and
idle gaps do not accumulate burst credit.
```bash
@@ -323,7 +323,7 @@ interface and queue.
??? abstract "See an example output"
- This is an illustrative excerpt; source line numbers and pointer values vary by build.
+ This is an illustrative excerpt. Source line numbers and pointer values vary by build.
```log
[INFO] /workspace/daqiri/src/../include/daqiri/common.h:1045: Finished reading DAQIRI configuration
@@ -578,7 +578,3 @@ The `*_packets_phy` and `*_bytes_phy` counters are physical-link counters. They
```
You might need to kill some of the listed processes to free up GPU VRAM.
-
----
-**Previous:** [Benchmarking](index.md)
-**Next:** [Understanding the Configuration File](../tutorials/configuration-walkthrough.md) — deep dive into the YAML parameters
diff --git a/docs/benchmarks/socket_benchmarking.md b/docs/benchmarks/socket_benchmarking.md
index 425ce8d..636f8ab 100644
--- a/docs/benchmarks/socket_benchmarking.md
+++ b/docs/benchmarks/socket_benchmarking.md
@@ -7,7 +7,7 @@ hide:
Use this page when the peer transport is TCP, UDP, or RoCE/RDMA. These benchmarks use the Linux networking stack for TCP/UDP and RDMA verbs for RoCE, so the same client/server namespace shape is useful for proving that traffic leaves the host through the expected NIC path.
-For **two-host Spark cross-cable** tests (not netns), RoCE/RDMA still needs kernel reachability to the peer — apply the host route and static neighbor steps in [System Configuration → Cross-host variant](../tutorials/system_configuration.md#cross-host-variant-two-sparks) before running `daqiri_bench_rdma` with the `_xhost` configs.
+For **two-host Spark cross-cable** tests (not netns), RoCE/RDMA still needs kernel reachability to the peer, so apply the host route and static neighbor steps in [System Configuration → Cross-host variant](../tutorials/system_configuration.md#cross-host-variant-two-sparks) before running `daqiri_bench_rdma` with the `_xhost` configs.
Make sure to [build DAQIRI](../getting-started.md#build-the-daqiri-library) with the `ibverbs` engine first (for the RoCE/RDMA benchmark); Linux UDP/TCP sockets are always available.
@@ -56,7 +56,7 @@ apt-get install -y iproute2 iputils-ping ethtool iperf3 rdma-core ibverbs-utils
## Create isolated namespaces
-Choose one transmit-facing interface and one receive-facing interface. The example below uses the Spark pair that was verified to increment physical counters on the tested system; adjust names, IPs, and MAC addresses on other machines.
+Choose one transmit-facing interface and one receive-facing interface. The example below uses the Spark pair that was verified to increment physical counters on the tested system. Adjust names, IPs, and MAC addresses on other machines.
```bash
CLIENT_NS=dq_wire_client
@@ -383,7 +383,3 @@ mlxconfig -d set FORCE_LOOPBACK_DISABLE=1
```
Treat firmware settings as maintenance-window changes: query first, set only with the proper Mellanox tooling available, then reset or reboot as required and rerun the same `rx_packets_phy` proof.
-
----
-**Previous:** [Benchmarking](index.md)
-**Next:** [Raw Ethernet Benchmarking](raw_benchmarking.md)
diff --git a/docs/concepts.md b/docs/concepts.md
index 62b5874..a6fd534 100644
--- a/docs/concepts.md
+++ b/docs/concepts.md
@@ -18,8 +18,8 @@ endpoint URI schemes**, **GPUDirect**, **packet / burst / segment**,
DAQIRI exposes a single C++ API on top of several packet-I/O stacks. The
choice is configured per-application in YAML with:
-- `stream_type` — the I/O stack family.
-- endpoint URI schemes — `tcp://`, `udp://`, or `roce://` in
+- `stream_type`: the I/O stack family.
+- endpoint URI schemes: `tcp://`, `udp://`, or `roce://` in
`socket_config.local_addr` and `socket_config.remote_addr`.
The shipped Ethernet stream types use NICs as their hardware endpoint.
@@ -33,7 +33,7 @@ An **engine** is the library that *implements* a stream type. It is a
separate concept from `stream_type`: the stream type says *what* kind of
stream you want, the engine is the *implementation* behind it.
-You normally never set an engine — DAQIRI picks a sensible default from
+You normally never set an engine, since DAQIRI picks a sensible default from
the `stream_type` and the endpoint URI scheme:
| Stream | Default engine | Notes |
@@ -47,7 +47,7 @@ The engine concept exists so the implementation can be swapped without
changing the stream type. For example, raw Ethernet is served by the
`dpdk` engine by default but can instead use the `ibverbs` engine (a
Multi-Packet/striding Receive Queue implementation) by setting
-`engine: "ibverbs"` on the stream; RoCE is served by the `ibverbs` engine,
+`engine: "ibverbs"` on the stream. RoCE is served by the `ibverbs` engine,
and a future release could add a DOCA RDMA engine as an alternative for the
same `roce://` stream.
@@ -63,7 +63,7 @@ Kernel-bypass raw Ethernet. The application talks directly to NIC ring
buffers in user space, skipping the Linux network stack entirely. This
is the highest-performance path and the only one with hardware flow
steering (see [Flows](#flow) below). Implemented on top of
-[DPDK](https://www.dpdk.org/) by default; the DPDK dependency is an
+[DPDK](https://www.dpdk.org/) by default. The DPDK dependency is an
implementation detail, not a user-facing concept. Setting `engine: "ibverbs"`
on the stream instead uses a pure-libibverbs/DevX Multi-Packet (striding)
Receive Queue engine on Mellanox/mlx5 NICs, which packs many packets into one
@@ -76,11 +76,11 @@ Requires an NVIDIA SmartNIC (ConnectX-6 Dx or later).
*YAML:* `stream_type: "socket"`. The specific transport is chosen by endpoint
URI schemes:
-- **`udp://`** / **`tcp://`** — Linux kernel UDP and TCP sockets. No NIC
+- **`udp://`** / **`tcp://`**: Linux kernel UDP and TCP sockets. No NIC
privileges required, no special hardware. Useful
as a comparison baseline against the kernel-bypass paths and as a way
to get first results on a system without an NVIDIA NIC.
-- **`roce://` endpoints** — RDMA over Converged Ethernet, using the
+- **`roce://` endpoints**: RDMA over Converged Ethernet, using the
open-source [`rdma-core`](https://github.com/linux-rdma/rdma-core)
library. A server/client connection model, NIC-level reliable
transport (RC), and in-order delivery. Primarily intended for
@@ -120,11 +120,11 @@ in the configuration walkthrough.
- **Raw Ethernet** (`stream_type: "raw"`) is supported, distributed
with the DAQIRI library, and is the only stream type actively
tested at this time.
- - **Socket — UDP / TCP** (`stream_type: "socket"` with `udp://` /
- `tcp://` endpoints) is supported and distributed; integration testing is
+ - **Socket: UDP / TCP** (`stream_type: "socket"` with `udp://` /
+ `tcp://` endpoints) is supported and distributed. Integration testing is
under development.
- - **Socket — RoCE** (`stream_type: "socket"` and
- `roce://` endpoints) is supported and distributed; integration
+ - **Socket: RoCE** (`stream_type: "socket"` and
+ `roce://` endpoints) is supported and distributed. Integration
testing is under development.
- The **PCIe programmable-sensor** path is under development.
@@ -182,7 +182,7 @@ For step-by-step system setup, see the
DAQIRI is a batch processing library. Packets are received from DAQIRI
and sent to DAQIRI in batches called **bursts**. Larger bursts can
-increase throughput at the expense of latency; smaller bursts decrease
+increase throughput at the expense of latency, while smaller bursts decrease
latency but cap total throughput because of the per-burst processing
overhead. The terms below appear throughout the API, configuration, and
code paths.
@@ -199,7 +199,7 @@ reassembles them on the wire transparently.
A **burst** is the metadata container DAQIRI uses to describe a batch
of packets being transmitted or received. The C++ type for a burst is
-`BurstParams`. It is intentionally opaque — applications use helper
+`BurstParams`. It is intentionally opaque, so applications use helper
functions (`get_packet_ptr`, `get_packet_length`, `get_num_packets`,
...) to inspect or modify it rather than touching its fields directly.
@@ -269,7 +269,7 @@ dynamic flows are not part of v1.
Raw DPDK and raw ibverbs flows can also use ordered `actions:` for hardware VLAN
pop/push and VXLAN, GRE, or NVGRE decap/encap. RX decap/pop actions deliver
-post-decap packets to application buffers; TX encap/push actions leave
+post-decap packets to application buffers, while TX encap/push actions leave
application buffers as pre-encap packets and change only the wire frame.
Dynamic RX flows use the same ordered action model for runtime decap/pop rules,
while TX transform flows remain static startup configuration.
@@ -301,7 +301,7 @@ The kind of a memory region determines whether packet data ends up on
the CPU or the GPU:
- `huge`: CPU hugepages (recommended for CPU buffers).
-- `device`: GPU VRAM (discrete GPUs; requires GPUDirect via peermem or
+- `device`: GPU VRAM (discrete GPUs, requires GPUDirect via peermem or
DMA-BUF).
- `host_pinned`: pinned CPU pages allocated via `cudaHostAlloc`.
Recommended on integrated GPUs (NVIDIA GB10 / DGX Spark), where the
@@ -324,7 +324,7 @@ pool (for headers, segment 0); its second region is a `device` GPU pool
DAQIRI is designed around zero-copy packet delivery. When a receive API
returns packet data, the application is reading the buffers the NIC
-DMA'd into; the API passes pointers and metadata, not copies.
+DMA'd into. The API passes pointers and metadata, not copies.
That zero-copy model makes **buffer release part of the API contract**.
Applications must free RX bursts after processing and free or send TX
diff --git a/docs/getting-started.md b/docs/getting-started.md
index 2079ea8..eb78eeb 100644
--- a/docs/getting-started.md
+++ b/docs/getting-started.md
@@ -13,11 +13,11 @@ DAQIRI's baseline requirements depend on which [stream type](concepts.md#stream-
|-----------|-------------|
| **OS** | Linux (kernel 5.4+), Ubuntu 22.04 recommended |
| **CUDA** | CUDA Toolkit 12.2+ (the container ships CUDA 13.1) |
-| **NIC** *(Raw Ethernet / GPUDirect / RoCE only)* | NVIDIA ConnectX-6 Dx or later. Default Ubuntu kernel drivers (inbox) are sufficient; we recommend also installing `doca-ofed` for the diagnostic utilities (`ibstat`, `ibv_devinfo`, `ibdev2netdev`, `mlnx_perf`, `mlxconfig`, …). |
+| **NIC** *(Raw Ethernet / GPUDirect / RoCE only)* | NVIDIA ConnectX-6 Dx or later. Default Ubuntu kernel drivers (inbox) are sufficient. We recommend also installing `doca-ofed` for the diagnostic utilities (`ibstat`, `ibv_devinfo`, `ibdev2netdev`, `mlnx_perf`, `mlxconfig`, and so on). |
| **GPU** *(GPUDirect only)* | RTX or Data Center GPU. GeForce is not supported. |
| **DPDK** | Included in the DAQIRI container (patched for dma-buf, so `nvidia-peermem` is **not required** inside the container); see [bare-metal dependencies](#bare-metal-dependencies) below for the host build. |
| **RoCE** | `libibverbs` and `librdmacm` (for `stream_type: "socket"` and `roce://` endpoints). |
-| **GDS** | Optional `cufile.h` and `libcufile` for file writes from CUDA device memory. Runtime device-memory writes require a working cuFile installation; for regular `nvidia-fs` mode, the `nvidia-fs` kernel module must be loaded and the destination storage stack must be supported. |
+| **GDS** | Optional `cufile.h` and `libcufile` for file writes from CUDA device memory. Runtime device-memory writes require a working cuFile installation. For regular `nvidia-fs` mode, the `nvidia-fs` kernel module must be loaded and the destination storage stack must be supported. |
| **S3** | Optional AWS SDK for C++ with the `s3` component for raw packet uploads to Amazon S3 or S3-compatible object stores. The DAQIRI container builds this SDK from source. |
Supported platforms include [NVIDIA Data Center](https://www.nvidia.com/en-us/data-center/) systems, edge systems like [NVIDIA IGX](https://www.nvidia.com/en-us/edge-computing/products/igx/) and [NVIDIA DGX Spark](https://www.nvidia.com/en-us/products/workstations/dgx-spark/), and `x86_64` systems with the above components.
@@ -96,7 +96,7 @@ Then build the DAQIRI library:
BASE_TARGET=dpdk DAQIRI_ENGINE="dpdk ibverbs" scripts/build-container.sh
```
- Set `BASE_IMAGE=torch` to build on top of NGC PyTorch instead of the default CUDA base — useful for Torch / TensorRT inference workflows that ingest packets directly into GPU memory:
+ Set `BASE_IMAGE=torch` to build on top of NGC PyTorch instead of the default CUDA base. This is useful for Torch / TensorRT inference workflows that ingest packets directly into GPU memory:
```bash
BASE_IMAGE=torch BASE_TARGET=dpdk DAQIRI_ENGINE="dpdk ibverbs" scripts/build-container.sh
@@ -212,11 +212,11 @@ DAQIRI's shared-library ABI version is tracked separately through
| `DAQIRI_ENABLE_OTEL_METRICS` | `OFF` | Enable OpenTelemetry C++ metrics instrumentation. When enabled, OpenTelemetry C++ API package metadata must be available to CMake. |
| `DAQIRI_ENABLE_S3` | `OFF` | Enable AWS SDK-backed asynchronous raw packet writes to S3. |
| `DAQIRI_PREFER_SYSTEM_YAML_CPP` | `OFF` | Prefer a system-installed `yaml-cpp` over the vendored `third_party/yaml-cpp` submodule. Keep `OFF` if a conda/miniforge env is on `PATH`. |
-| `BUILD_SHARED_LIBS` | — | Build as shared library. |
+| `BUILD_SHARED_LIBS` | n/a | Build as shared library. |
Linux UDP/TCP sockets are always available. Applications that need kernel socket
tuning can call `socket_setsockopt()` after resolving a TCP/UDP connection ID,
-passing the numeric `level` and option constants from system headers; DAQIRI does
+passing the numeric `level` and option constants from system headers. DAQIRI does
not maintain symbolic socket-option mappings in YAML.
For Raw Ethernet (`stream_type: "raw"`), `daqiri_init()` validates that each `rx.flows`
@@ -225,8 +225,8 @@ entry's legacy `action.id` or final ordered `actions:` queue action matches an
Initialization fails if any RX flow rule, TX transform flow, send-to-kernel fallback
(when `flow_isolation: true`), or `tx_eth_src` offload rule cannot be installed.
Raw DPDK and raw ibverbs can offload VLAN push/pop and VXLAN, GRE, or NVGRE
-encap/decap through flow actions; socket/RDMA streams reject those actions.
-`rx.flows` may also be omitted for queues-only startup; applications can then add and delete
+encap/decap through flow actions. Socket/RDMA streams reject those actions.
+`rx.flows` may also be omitted for queues-only startup, and applications can then add and delete
RX flow rules at runtime with `add_rx_flow_async()` / `delete_flow_async()`. Dynamic
RX flows can use the same decap/pop action ordering as static RX flows. The DPDK template
fast path is enabled and sized by `rx.dynamic_flow_capacity` (default `0`, set a positive
@@ -254,7 +254,7 @@ writes. Host-backed burst writes continue to use POSIX APIs and do not require G
OpenTelemetry metrics builds register observable counters for received packets,
transmitted packets, received bytes, transmitted bytes, and dropped packets. DAQIRI
-does not configure an SDK reader or exporter; applications that want exported data
+does not configure an SDK reader or exporter. Applications that want exported data
must configure the OpenTelemetry C++ SDK before or during DAQIRI initialization.
When using `DAQIRI_ENABLE_S3=ON`, the container build installs AWS SDK for C++
@@ -275,8 +275,8 @@ rules, not both. Mixed configs are rejected at `daqiri_init`. See
Once DAQIRI is built, follow the tutorials to configure your system and run your first benchmark:
-1. [**Concepts**](concepts.md) — terminology (stream types, engines, endpoint URI schemes, packet, burst, segment, flow, queue, memory region), GPUDirect, and zero-copy ownership. Keep this open in a second tab.
-2. [**API Guide**](api-reference/index.md) — the six-step DAQIRI application lifecycle and configuration-first model
-3. [**System Configuration**](tutorials/system_configuration.md) — NIC drivers, link layers, GPUDirect, hugepages, CPU isolation, GPU clocks, and more
-4. [**Benchmarking**](benchmarks/index.md) — choose an engine, then run socket/RDMA or raw Ethernet benchmarks
-5. [**Understanding the Configuration File**](tutorials/configuration-walkthrough.md) — annotated YAML walkthrough
+1. [**Concepts**](concepts.md): terminology (stream types, engines, endpoint URI schemes, packet, burst, segment, flow, queue, memory region), GPUDirect, and zero-copy ownership. Keep this open in a second tab.
+2. [**API Guide**](api-reference/index.md): the six-step DAQIRI application lifecycle and configuration-first model
+3. [**System Configuration**](tutorials/system_configuration.md): NIC drivers, link layers, GPUDirect, hugepages, CPU isolation, GPU clocks, and more
+4. [**Benchmarking**](benchmarks/index.md): choose an engine, then run socket/RDMA or raw Ethernet benchmarks
+5. [**Understanding the Configuration File**](tutorials/configuration-walkthrough.md): annotated YAML walkthrough
diff --git a/docs/index.md b/docs/index.md
index 1211aab..56e04da 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -1,5 +1,5 @@
---
-title: DAQIRI — Data Acquisition for Integrated Real-time Instruments
+title: "DAQIRI: Data Acquisition for Integrated Real-time Instruments"
template: home.html
hide:
- navigation
diff --git a/docs/landing/examples.html b/docs/landing/examples.html
index b6e1abf..585c57c 100644
--- a/docs/landing/examples.html
+++ b/docs/landing/examples.html
@@ -94,12 +94,12 @@
Extends sequence reorder with an in-kernel type conversion step (e.g., int4 → fp32), so the GPU buffer is both reordered and in the format your compute pipeline expects — all before your application code runs.
+
Extends sequence reorder with an in-kernel type conversion step (e.g., int4 → fp32), so the GPU buffer is both reordered and in the format your compute pipeline expects, all before your application code runs.
Captures live network traffic to a standard .pcap file you can open in Wireshark or tcpdump. Packets are received via GPUDirect and staged through pinned host memory to disk; capture continues until you press Ctrl+C.
+
Captures live network traffic to a standard .pcap file you can open in Wireshark or tcpdump. Packets are received via GPUDirect and staged through pinned host memory to disk. Capture continues until you press Ctrl+C.
Measures TCP and UDP throughput over standard Linux sockets — no ConnectX NIC or special privileges required. A good comparison baseline before moving to kernel-bypass, or for connecting to a peer that only speaks standard sockets.
+
Measures TCP and UDP throughput over standard Linux sockets, with no ConnectX NIC or special privileges required. A good comparison baseline before moving to kernel-bypass, or for connecting to a peer that only speaks standard sockets.
Scientific and industrial instruments generate data that is richest at the source — before it is filtered, decimated, or summarized. DAQIRI places NVIDIA GPU hardware directly in that data path, forging a tight bond between upstream sensors, their data converters, and the NVIDIA compute ecosystem. The result is a new foundation for developers: the ability to work with instrument data in its rawest form, at wire speed, and to build a new class of autonomous experiments where AI can observe phenomena directly at the source, augment human analysis, and steer experiments in real time. Stream data into and out of GPUs efficiently while leveraging common tensor-compute libraries.
+
Scientific and industrial instruments generate data that is richest at the source, before it is filtered, decimated, or summarized. DAQIRI puts NVIDIA GPU hardware directly in that data path, connecting sensors and their data converters to the NVIDIA compute ecosystem. That lets developers work with instrument data in its rawest form, at wire speed, and build experiments where AI can read the data as it arrives, assist with analysis, and adjust the experiment in real time. Stream data into and out of GPUs efficiently using common tensor-compute libraries.
NVIDIA — Initial public release under Apache 2.0, featuring Raw Ethernet and RoCE stream types with GPUDirect support for ConnectX-6 Dx and later NICs.
+
NVIDIA. Initial public release under Apache 2.0, featuring Raw Ethernet and RoCE stream types with GPUDirect support for ConnectX-6 Dx and later NICs.
NVIDIA — DAQIRI is undergoing large improvements to become a fully supported NVIDIA product. API breakages may occur before v1.0. Contributions welcome.
+
NVIDIA. DAQIRI is undergoing large improvements to become a fully supported NVIDIA product. API breakages may occur before v1.0. Contributions welcome.
For the Raw Ethernet / GPUDirect / RoCE path, you also need an NVIDIA ConnectX-6 Dx (or newer) NIC. The default Ubuntu kernel drivers are sufficient; we recommend additionally installing doca-ofed for the diagnostic utilities (ibstat, ibv_devinfo, mlxconfig, mlnx_perf, …).
+
For the Raw Ethernet / GPUDirect / RoCE path, you also need an NVIDIA ConnectX-6 Dx (or newer) NIC. The default Ubuntu kernel drivers are sufficient. We also recommend installing doca-ofed for the diagnostic utilities (ibstat, ibv_devinfo, mlxconfig, mlnx_perf, and so on).
@@ -37,7 +37,7 @@
Build from Source
3
Or Build the Container
-
The Dockerfile builds DPDK from source with dmabuf patches — no peermem needed inside the container. Set BASE_IMAGE=torch to build on top of NGC PyTorch for Torch / TensorRT inference workflows.
+
The Dockerfile builds DPDK from source with dmabuf patches, so no peermem is needed inside the container. Set BASE_IMAGE=torch to build on top of NGC PyTorch for Torch / TensorRT inference workflows.
uint32_t plen =
daqiri::get_segment_packet_length(
burst, 1, i);
- // pay is already on GPU — no copy
+ // pay is already on GPU, no copy
}
diff --git a/docs/stylesheets/landing.css b/docs/stylesheets/landing.css
index 4a2a835..63c17d3 100644
--- a/docs/stylesheets/landing.css
+++ b/docs/stylesheets/landing.css
@@ -120,6 +120,24 @@ body:has(.daqiri-landing) .md-header__title {
font-size: 0.9rem;
}
+/* On the landing page the site title and the page title are identical, so
+ Material's scroll swap just slides the same text sideways when you scroll
+ from the very top. Keep a single, static title instead of animating it.
+ Material toggles .md-header__title--active after scrolling past the H1,
+ which would otherwise hide the first topic (opacity/transform) and set it
+ to pointer-events: none / z-index: -1; reset all four so the title stays
+ fully visible and interactive in both states. */
+body:has(.daqiri-landing) .md-header__title .md-header__topic + .md-header__topic {
+ display: none;
+}
+body:has(.daqiri-landing) .md-header__title .md-header__topic:first-child {
+ opacity: 1;
+ transform: none;
+ transition: none;
+ pointer-events: auto;
+ z-index: auto;
+}
+
.daqiri-landing .md-typeset :is(h1, h2, h3, h4, p, pre, ul, ol, dl) {
margin-top: 0;
margin-bottom: 0;
diff --git a/docs/tutorials/bare-metal-cmake-build.md b/docs/tutorials/bare-metal-cmake-build.md
index 9fac5c0..6f34621 100644
--- a/docs/tutorials/bare-metal-cmake-build.md
+++ b/docs/tutorials/bare-metal-cmake-build.md
@@ -25,7 +25,7 @@ Before installing anything, run the checks below and resolve any failures. They
| Check | Command | Expected |
|---|---|---|
-| NVIDIA driver + GPU | `nvidia-smi` | Driver version printed; one or more GPUs listed |
+| NVIDIA driver + GPU | `nvidia-smi` | Driver version printed, one or more GPUs listed |
| CUDA Toolkit (11.7+) | `nvcc --version` | `Cuda compilation tools, release 11.7` or later |
| NIC kernel drivers | `lsmod \| grep ib_core` | `ib_core` listed |
| NIC user-space tools | `ibv_devinfo` | At least one `mlx5_*` HCA in `PORT_ACTIVE` state |
@@ -34,7 +34,7 @@ Before installing anything, run the checks below and resolve any failures. They
??? failure "`nvidia-smi: command not found`"
- Install the NVIDIA driver. On Ubuntu the simplest path is `sudo ubuntu-drivers install`. The driver must be a recent branch with GPUDirect support; see the [system requirements table](../getting-started.md#system-requirements).
+ Install the NVIDIA driver. On Ubuntu the simplest path is `sudo ubuntu-drivers install`. The driver must be a recent branch with GPUDirect support. See the [system requirements table](../getting-started.md#system-requirements).
??? failure "`nvcc: command not found` (but `nvidia-smi` works)"
@@ -58,7 +58,7 @@ DAQIRI's RDMA/ibverbs dependencies (`libibverbs-dev`, `librdmacm-dev`, `libmlx5-
!!! note
- The container build uses DOCA `3.2.1` (see `DOCA_VERSION` in the [Dockerfile](https://github.com/NVIDIA/daqiri/blob/main/Dockerfile)). Earlier DOCA releases also work; the version pinning is for build reproducibility, not a strict requirement.
+ The container build uses DOCA `3.2.1` (see `DOCA_VERSION` in the [Dockerfile](https://github.com/NVIDIA/daqiri/blob/main/Dockerfile)). Earlier DOCA releases also work. The version pinning is for build reproducibility, not a strict requirement.
## Step 2: Install build tooling and RDMA libraries
@@ -89,7 +89,7 @@ sudo apt install -y --no-install-recommends \
On many host configurations (always on Ubuntu 24.04 with DOCA 3.2.1, where `libmlx5-1` has no DOCA candidate) apt resolves `libmlx5-1` to the `ibverbs-providers` virtual package (the inbox bundle of mlx5/mlx4 user-space providers). This is fine: `libmlx5.so.1` ends up installed either way and DAQIRI links against it through `libibverbs`. If you want the DOCA-provided build specifically, install `libmlx5-1` and `libmlx5-dev` after the DOCA repo from [Step 1](#step-1-configure-the-doca-apt-repository) is configured.
-DAQIRI's top-level CMake requires version 3.20 or newer (see [CMakeLists.txt](https://github.com/NVIDIA/daqiri/blob/main/CMakeLists.txt)). On Ubuntu 22.04 the distribution package is too old; install a current build from the Kitware APT repository:
+DAQIRI's top-level CMake requires version 3.20 or newer (see [CMakeLists.txt](https://github.com/NVIDIA/daqiri/blob/main/CMakeLists.txt)). On Ubuntu 22.04 the distribution package is too old. Install a current build from the Kitware APT repository:
```bash
OS_CODENAME=$(. /etc/os-release && echo "$VERSION_CODENAME")
@@ -221,9 +221,9 @@ The sections below explain each option you can flip from the default, with expli
| Recipe | What you get | When to use |
|---|---|---|
-| `-DDAQIRI_ENGINE="dpdk"` | Raw Ethernet (DPDK) + built-in UDP/TCP sockets | No RoCE/RDMA; smallest NIC build |
-| `-DDAQIRI_ENGINE="ibverbs"` | RDMA/RoCE (ibverbs) + built-in UDP/TCP sockets | RDMA-only; no raw Ethernet |
-| `-DDAQIRI_ENGINE="dpdk ibverbs"` | Raw Ethernet + RDMA/RoCE + built-in UDP/TCP sockets | Recommended default; matches the container |
+| `-DDAQIRI_ENGINE="dpdk"` | Raw Ethernet (DPDK) + built-in UDP/TCP sockets | No RoCE/RDMA, smallest NIC build |
+| `-DDAQIRI_ENGINE="ibverbs"` | RDMA/RoCE (ibverbs) + built-in UDP/TCP sockets | RDMA-only, no raw Ethernet |
+| `-DDAQIRI_ENGINE="dpdk ibverbs"` | Raw Ethernet + RDMA/RoCE + built-in UDP/TCP sockets | Recommended default, matches the container |
!!! note "Sockets are always available; `ibverbs` backs RoCE"
@@ -235,7 +235,7 @@ The sections below explain each option you can flip from the default, with expli
### `DAQIRI_ENABLE_GDS`: cuFile burst writes
-`-DDAQIRI_ENABLE_GDS=ON` enables the `cuFile`-backed file-write path for bursts whose payload lives in CUDA device memory. The build requires `cufile.h` and `libcufile` (both shipped with the CUDA Toolkit). At runtime you also need the `nvidia-fs` kernel module loaded and a GDS-supported destination filesystem; verify with the snippet in [Getting Started](../getting-started.md#cmake-options). Without this flag, device-memory burst writes return `NOT_SUPPORTED`; host-memory writes are unaffected.
+`-DDAQIRI_ENABLE_GDS=ON` enables the `cuFile`-backed file-write path for bursts whose payload lives in CUDA device memory. The build requires `cufile.h` and `libcufile` (both shipped with the CUDA Toolkit). At runtime you also need the `nvidia-fs` kernel module loaded and a GDS-supported destination filesystem. Verify with the snippet in [Getting Started](../getting-started.md#cmake-options). Without this flag, device-memory burst writes return `NOT_SUPPORTED`, and host-memory writes are unaffected.
### `CMAKE_CUDA_ARCHITECTURES`: GPU compute capability
@@ -254,7 +254,7 @@ cmake -S . -B build ... -DCMAKE_CUDA_ARCHITECTURES="80;89;90"
cmake -S . -B build ... -DCMAKE_CUDA_ARCHITECTURES=120
```
-The override is honored because `src/CMakeLists.txt` only sets `CMAKE_CUDA_ARCHITECTURES` when it is not already defined; a user-supplied `-D` value takes precedence. Common values:
+The override is honored because `src/CMakeLists.txt` only sets `CMAKE_CUDA_ARCHITECTURES` when it is not already defined, so a user-supplied `-D` value takes precedence. Common values:
| GPU family | Architecture |
|---|---|
@@ -278,7 +278,7 @@ The override is honored because `src/CMakeLists.txt` only sets `CMAKE_CUDA_ARCHI
- `-DBUILD_SHARED_LIBS=ON`: produces `libdaqiri.so` (recommended). With `OFF`, you get a static library.
- `-DDAQIRI_BUILD_EXAMPLES=ON`: builds the `daqiri_bench_*` executables under `build/examples/`. Required for the smoke test in [Step 5.3](#53-smoke-test). On by default.
-- `-DDAQIRI_REORDER_GPU_PROFILE=ON`: instruments the CUDA reorder kernels with CUDA event timing. Off by default; turn on only when profiling.
+- `-DDAQIRI_REORDER_GPU_PROFILE=ON`: instruments the CUDA reorder kernels with CUDA event timing. Off by default. Turn on only when profiling.
## Step 5: Build, install, and verify
@@ -288,7 +288,7 @@ The override is honored because `src/CMakeLists.txt` only sets `CMAKE_CUDA_ARCHI
cmake --build build -j"$(nproc)"
```
-A clean build on a modern workstation takes a few minutes; the slowest single translation unit is `src/kernels.cu`.
+A clean build on a modern workstation takes a few minutes. The slowest single translation unit is `src/kernels.cu`.
### 5.2 Install
@@ -324,7 +324,7 @@ A successful run prints a stream of `[INFO]` lines followed by an RX/TX rate sum
!!! tip "DGX Spark"
- On DGX Spark, use the prefilled `daqiri_bench_raw_tx_rx_spark.yaml` instead; only `eth_dst_addr` needs an edit. See the [DGX Spark profile callout](../benchmarks/raw_benchmarking.md#update-the-loopback-configuration) for the exact MAC-lookup command.
+ On DGX Spark, use the prefilled `daqiri_bench_raw_tx_rx_spark.yaml` instead. Only `eth_dst_addr` needs an edit. See the [DGX Spark profile callout](../benchmarks/raw_benchmarking.md#update-the-loopback-configuration) for the exact MAC-lookup command.
!!! note "No NIC available?"
@@ -383,7 +383,7 @@ A successful run prints a stream of `[INFO]` lines followed by an RX/TX rate sum
/usr/local/cuda/gds/tools/gdscheck.py -p
```
- Full setup notes are in [Getting Started → CMake Options](../getting-started.md#cmake-options). If `nvidia-fs` cannot be loaded on this host, drop `-DDAQIRI_ENABLE_GDS=ON` from the CMake configure step; host-memory burst writes do not require it.
+ Full setup notes are in [Getting Started → CMake Options](../getting-started.md#cmake-options). If `nvidia-fs` cannot be loaded on this host, drop `-DDAQIRI_ENABLE_GDS=ON` from the CMake configure step. Host-memory burst writes do not require it.
??? failure "`pkg-config: command not found`"
@@ -409,7 +409,7 @@ The build recipe above is the same on every supported host. The notes below cove
=== "DGX Spark (GB10)"
- - The integrated **ConnectX-7** appears in `ibv_devinfo` as one or two `mlx5_*` HCAs depending on link configuration; no separate driver install beyond the [DOCA repository setup](#step-1-configure-the-doca-apt-repository) is needed.
+ - The integrated **ConnectX-7** appears in `ibv_devinfo` as one or two `mlx5_*` HCAs depending on link configuration. No separate driver install beyond the [DOCA repository setup](#step-1-configure-the-doca-apt-repository) is needed.
- GB10 is **compute capability 12.1** (`sm_121`). DAQIRI's default arch list adds `121` automatically when configuring with **CUDA Toolkit 13.0 or newer**; on those toolkits no override is needed. On older toolkits, GB10 is not supported.
- DGX Spark uses **NVLink-C2C unified memory** and has no separate GPU BAR1, so data buffers in YAML configs use `kind: host_pinned` rather than `kind: device`. The DGX-Spark-prefilled YAMLs in `examples/*_spark.yaml` already encode this.
- `nvidia-peermem` is not used; GPUDirect goes through the dma-buf path enabled by the DPDK patches in [Step 3](#step-3-build-dpdk-with-daqiri-patches).
@@ -424,7 +424,7 @@ The build recipe above is the same on every supported host. The notes below cove
```
Use a different value (or extra entries) if you have a different dGPU installed.
- - On IGX Orin, several system settings (BAR1 size, MRRS, NIC link layer) need to be applied **before** DAQIRI will run end-to-end. Follow the [IGX Orin tab in System Configuration](system_configuration.md) for the full sequence; the build itself does not require those changes.
+ - On IGX Orin, several system settings (BAR1 size, MRRS, NIC link layer) need to be applied **before** DAQIRI will run end-to-end. Follow the [IGX Orin tab in System Configuration](system_configuration.md) for the full sequence. The build itself does not require those changes.
- GPUDirect can use either the patched DPDK's dma-buf path (recommended) or the legacy `nvidia-peermem` module. The patched DPDK built in [Step 3](#step-3-build-dpdk-with-daqiri-patches) removes the peermem dependency.
- For ARM64 (`aarch64`) hosts, set `PKG_CONFIG_PATH` to the aarch64 directory:
@@ -436,14 +436,14 @@ The build recipe above is the same on every supported host. The notes below cove
=== "x86_64 RTX Pro Server"
- - "RTX Pro Server" covers any `x86_64` workstation or server with a ConnectX-6 Dx (or later) NIC and an RTX Pro / Workstation GPU. Confirm `nvidia-smi` reports a GPUDirect-capable GPU (any RTX Pro / Quadro / Data Center class card; **not** GeForce; see the warning in [Concepts → GPUDirect](../concepts.md#gpudirect)).
+ - "RTX Pro Server" covers any `x86_64` workstation or server with a ConnectX-6 Dx (or later) NIC and an RTX Pro / Workstation GPU. Confirm `nvidia-smi` reports a GPUDirect-capable GPU (any RTX Pro / Quadro / Data Center class card, **not** GeForce, and see the warning in [Concepts → GPUDirect](../concepts.md#gpudirect)).
- Set `-DCMAKE_CUDA_ARCHITECTURES` to match the installed card. RTX PRO 6000 Blackwell Server/Workstation Edition is `120` (workstation Blackwell, `sm_120`); RTX 6000 Ada is `89`; RTX A6000 is `86`. Confirm with `nvidia-smi --query-gpu=compute_cap --format=csv,noheader` (e.g. `12.0` → `120`):
```bash
cmake -S . -B build ... -DCMAKE_CUDA_ARCHITECTURES=120
```
- - x86_64 hosts use `/usr/local/lib/x86_64-linux-gnu/pkgconfig` for the DPDK `.pc` file; that's the default `PKG_CONFIG_PATH` entry shown in [Step 3.4](#34-verify).
+ - x86_64 hosts use `/usr/local/lib/x86_64-linux-gnu/pkgconfig` for the DPDK `.pc` file. That's the default `PKG_CONFIG_PATH` entry shown in [Step 3.4](#34-verify).
- All other steps are identical to the generic recipe above.
## Next steps
@@ -466,6 +466,6 @@ scripts/cleanup.sh cmake --dry-run # show what would be removed
scripts/cleanup.sh cmake --yes # non-interactive
```
-The script reads `build/install_manifest.txt` (written by [Step 5.2](#52-install)) for the canonical list of installed paths, then refuses to remove anything unless every manifest entry is under `DAQIRI_PREFIX`. Manifest entries that are already absent are reported and skipped so cleanup can be rerun after a partial removal; verification still runs and decides the final exit status. Override the install prefix with `DAQIRI_PREFIX=...` if you installed somewhere other than `/opt/daqiri`, or `BUILD_DIR=...` if your build tree is named something other than `build`. When the manifest is missing, the script falls back to a name-scoped scan that auto-removes only DAQIRI-owned artifacts and flags vendored `spdlog/`, `yaml-cpp/`, and `libyaml-cpp.so*` for manual review. The final step runs verification (`ls /opt/daqiri`, `pkg-config --modversion daqiri`, `ldconfig -p | grep daqiri`) and exits non-zero if any DAQIRI artifact is still present.
+The script reads `build/install_manifest.txt` (written by [Step 5.2](#52-install)) for the canonical list of installed paths, then refuses to remove anything unless every manifest entry is under `DAQIRI_PREFIX`. Manifest entries that are already absent are reported and skipped so cleanup can be rerun after a partial removal. Verification still runs and decides the final exit status. Override the install prefix with `DAQIRI_PREFIX=...` if you installed somewhere other than `/opt/daqiri`, or `BUILD_DIR=...` if your build tree is named something other than `build`. When the manifest is missing, the script falls back to a name-scoped scan that auto-removes only DAQIRI-owned artifacts and flags vendored `spdlog/`, `yaml-cpp/`, and `libyaml-cpp.so*` for manual review. The final step runs verification (`ls /opt/daqiri`, `pkg-config --modversion daqiri`, `ldconfig -p | grep daqiri`) and exits non-zero if any DAQIRI artifact is still present.
Pass `all` instead of `cmake` to also remove the container image (`docker image rm "$IMAGE_TAG"`) in the same run.
diff --git a/docs/tutorials/configuration-walkthrough.md b/docs/tutorials/configuration-walkthrough.md
index b1aef38..54c99ac 100644
--- a/docs/tutorials/configuration-walkthrough.md
+++ b/docs/tutorials/configuration-walkthrough.md
@@ -11,75 +11,75 @@ hide:
DAQIRI exposes a single API on top of multiple packet I/O stacks, selected at runtime with `stream_type` and endpoint URI schemes such as `udp://`, `tcp://`, and `roce://`. Pick the row that matches your hardware and the role of the other endpoint:
-- **Raw Ethernet** — `stream_type: "raw"`. Kernel-bypass with GPUDirect zero-copy. Highest performance. Requires an [NVIDIA ConnectX-class NIC](https://www.nvidia.com/en-us/networking/ethernet-adapters/); `tx_port` and `rx_port` can share one physical NIC for a single-host closed-loop bench, or be split across two hosts.
-- **Socket — UDP / TCP** — `stream_type: "socket"` with `udp://` or `tcp://` endpoints. Plain Linux kernel sockets. No NIC, no privileges, no special CMake flags. Useful as a comparison baseline and as a path to first results on a system without an NVIDIA NIC. Socket options are runtime API calls: resolve a connection ID, then pass native Linux constants to `socket_setsockopt()` rather than adding option names to YAML.
-- **Socket — RoCE (RDMA)** — `stream_type: "socket"` and `roce://` endpoints. RDMA verbs over Ethernet, with a server/client connection model and a NIC-level reliable transport. Primarily intended for setups where **one** endpoint is a third-party RoCE implementation (FPGA, instrument, customer black box). When both peers run DAQIRI, prefer an upper-layer library such as MPI / NCCL / UCX instead.
+- **Raw Ethernet**: `stream_type: "raw"`. Kernel-bypass with GPUDirect zero-copy. Highest performance. Requires an [NVIDIA ConnectX-class NIC](https://www.nvidia.com/en-us/networking/ethernet-adapters/). `tx_port` and `rx_port` can share one physical NIC for a single-host closed-loop bench, or be split across two hosts.
+- **Socket (UDP / TCP)**: `stream_type: "socket"` with `udp://` or `tcp://` endpoints. Plain Linux kernel sockets. No NIC, no privileges, no special CMake flags. Useful as a comparison baseline and as a path to first results on a system without an NVIDIA NIC. Socket options are runtime API calls: resolve a connection ID, then pass native Linux constants to `socket_setsockopt()` rather than adding option names to YAML.
+- **Socket (RoCE / RDMA)**: `stream_type: "socket"` and `roce://` endpoints. RDMA verbs over Ethernet, with a server/client connection model and a NIC-level reliable transport. Primarily intended for setups where **one** endpoint is a third-party RoCE implementation (FPGA, instrument, customer black box). When both peers run DAQIRI, prefer an upper-layer library such as MPI / NCCL / UCX instead.
-If you don't have any NIC at all, the `*_sw_loopback*` variants of the Raw Ethernet configs need no hardware — useful for first-time build verification.
+If you don't have any NIC at all, the `*_sw_loopback*` variants of the Raw Ethernet configs need no hardware, which is useful for first-time build verification.
-(`DAQIRI_ENGINE` at the CMake layer selects which optional engine implementations to compile in — `dpdk` enables `stream_type: "raw"`, and `ibverbs` enables `roce://` endpoints. Linux UDP/TCP sockets are always built in. The default build is `dpdk ibverbs`.)
+(`DAQIRI_ENGINE` at the CMake layer selects which optional engine implementations to compile in. `dpdk` enables `stream_type: "raw"`, and `ibverbs` enables `roce://` endpoints. Linux UDP/TCP sockets are always built in. The default build is `dpdk ibverbs`.)
For a shorter selection guide, start with the [Benchmarking overview](../benchmarks/index.md). With a stream type in mind, read down the questions below and stop at the first one that matches what you're trying to do. Each section names the YAML, the binary that consumes it, and any platform-specific notes.
??? question "1. I want to measure baseline throughput"
Pick the stream type that matches your stack (see the [overview](#choosing-the-appropriate-daqiri-stream-type-for-your-setup) above), then the hardware or transport variant.
- **Raw Ethernet** (`stream_type: "raw"`) — runs on `daqiri_bench_raw_gpudirect`.
+ **Raw Ethernet** (`stream_type: "raw"`) runs on `daqiri_bench_raw_gpudirect`.
- - **Generic discrete GPU** (template — replace ``) — [`daqiri_bench_raw_tx_rx.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_tx_rx.yaml). This is the file annotated line-by-line in the [walkthrough below](#annotated-walkthrough).
- - **Four queue closed-loop TX+RX** (template — replace ``) — [`daqiri_bench_raw_tx_rx_4q.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_tx_rx_4q.yaml). Uses one application worker per TX/RX queue, with each `bench_tx` entry sending a different UDP flow.
- - **DGX Spark / GB10** (prefilled) — [`daqiri_bench_raw_tx_rx_spark.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_tx_rx_spark.yaml). `kind: host_pinned` for the integrated GPU; cores, PCIe addresses, and IPs are prefilled. See the [Spark profile callout](../benchmarks/raw_benchmarking.md#update-the-loopback-configuration) for run details.
- - **DGX Spark multi-queue core-scaling matrix** (prefilled) — one base config [`daqiri_bench_raw_tx_rx_spark_mq.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_tx_rx_spark_mq.yaml) (the balanced TX=2/RX=2 superset; cores TX → 16,17, RX → 18,19) from which `examples/run_spark_mq_bench.sh` (via `scripts/gen_spark_mq_config.py`) derives the four `(TX, RX)` cells — (1,1), (1,2) (RX scaling), (2,1) (TX scaling), (2,2) (balanced) — by pruning queues/flows. All run on `daqiri_bench_raw_gpudirect` at the native 8 KB shape.
- - **DGX Spark cross-host** (prefilled, runs on two Sparks) — [`daqiri_bench_raw_tx_spark_xhost.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_tx_spark_xhost.yaml) on the TX host and [`daqiri_bench_raw_rx_spark_xhost.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_rx_spark_xhost.yaml) on the RX host. Each host runs `daqiri_bench_raw_gpudirect` against its own half; cables connect p0↔p0 between the two boxes. Apply the [cross-host network setup](../tutorials/system_configuration.md#cross-host-variant-two-sparks) before running. See the [Cross-host two-DGX-Spark loopback](../benchmarks/raw_benchmarking.md#cross-host-two-dgx-spark-loopback) section for run details.
- - **No physical NIC available** — [`daqiri_bench_raw_sw_loopback.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_sw_loopback.yaml). `loopback: "sw"`, no NIC required. Useful for first-time build verification, not representative of production performance.
+ - **Generic discrete GPU** (template, replace ``): [`daqiri_bench_raw_tx_rx.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_tx_rx.yaml). This is the file annotated line-by-line in the [walkthrough below](#annotated-walkthrough).
+ - **Four queue closed-loop TX+RX** (template, replace ``): [`daqiri_bench_raw_tx_rx_4q.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_tx_rx_4q.yaml). Uses one application worker per TX/RX queue, with each `bench_tx` entry sending a different UDP flow.
+ - **DGX Spark / GB10** (prefilled): [`daqiri_bench_raw_tx_rx_spark.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_tx_rx_spark.yaml). `kind: host_pinned` for the integrated GPU, and cores, PCIe addresses, and IPs are prefilled. See the [Spark profile callout](../benchmarks/raw_benchmarking.md#update-the-loopback-configuration) for run details.
+ - **DGX Spark multi-queue core-scaling matrix** (prefilled): one base config [`daqiri_bench_raw_tx_rx_spark_mq.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_tx_rx_spark_mq.yaml) (the balanced TX=2/RX=2 superset, cores TX → 16,17, RX → 18,19) from which `examples/run_spark_mq_bench.sh` (via `scripts/gen_spark_mq_config.py`) derives the four `(TX, RX)` cells (1,1), (1,2) (RX scaling), (2,1) (TX scaling), and (2,2) (balanced) by pruning queues/flows. All run on `daqiri_bench_raw_gpudirect` at the native 8 KB shape.
+ - **DGX Spark cross-host** (prefilled, runs on two Sparks): [`daqiri_bench_raw_tx_spark_xhost.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_tx_spark_xhost.yaml) on the TX host and [`daqiri_bench_raw_rx_spark_xhost.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_rx_spark_xhost.yaml) on the RX host. Each host runs `daqiri_bench_raw_gpudirect` against its own half, and cables connect p0↔p0 between the two boxes. Apply the [cross-host network setup](../tutorials/system_configuration.md#cross-host-variant-two-sparks) before running. See the [Cross-host two-DGX-Spark loopback](../benchmarks/raw_benchmarking.md#cross-host-two-dgx-spark-loopback) section for run details.
+ - **No physical NIC available**: [`daqiri_bench_raw_sw_loopback.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_sw_loopback.yaml). `loopback: "sw"`, no NIC required. Useful for first-time build verification, not representative of production performance.
- **Raw Ethernet hardware tunnel transforms** — run on `daqiri_bench_raw_gpudirect`; replace placeholders and use a raw DPDK or raw ibverbs build.
+ **Raw Ethernet hardware tunnel transforms** run on `daqiri_bench_raw_gpudirect`. Replace placeholders and use a raw DPDK or raw ibverbs build.
- - **VXLAN encap + decap** — [`daqiri_bench_raw_tx_rx_vxlan.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_tx_rx_vxlan.yaml).
- - **VLAN push + pop** — [`daqiri_bench_raw_tx_rx_vlan.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_tx_rx_vlan.yaml).
- - **GRE encap + decap** — [`daqiri_bench_raw_tx_rx_gre.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_tx_rx_gre.yaml).
- - **NVGRE encap + decap** — [`daqiri_bench_raw_tx_rx_nvgre.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_tx_rx_nvgre.yaml).
+ - **VXLAN encap + decap**: [`daqiri_bench_raw_tx_rx_vxlan.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_tx_rx_vxlan.yaml).
+ - **VLAN push + pop**: [`daqiri_bench_raw_tx_rx_vlan.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_tx_rx_vlan.yaml).
+ - **GRE encap + decap**: [`daqiri_bench_raw_tx_rx_gre.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_tx_rx_gre.yaml).
+ - **NVGRE encap + decap**: [`daqiri_bench_raw_tx_rx_nvgre.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_tx_rx_nvgre.yaml).
To watch the same raw loopback benchmark with live Prometheus and Grafana
counters, use the Grafana compose stack described in
[Watch live OpenTelemetry metrics in Grafana](../benchmarks/raw_benchmarking.md#watch-live-opentelemetry-metrics-in-grafana).
- **Socket — RoCE (RDMA)** (`stream_type: "socket"`, `roce://` endpoints) — runs on `daqiri_bench_rdma` (use `--mode {tx,rx,both}`). Configs use `kind: host_pinned` regardless of platform.
+ **Socket (RoCE / RDMA)** (`stream_type: "socket"`, `roce://` endpoints) runs on `daqiri_bench_rdma` (use `--mode {tx,rx,both}`). Configs use `kind: host_pinned` regardless of platform.
- - **Generic** (template — replace IPs) — [`daqiri_bench_rdma_tx_rx.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_rdma_tx_rx.yaml).
- - **DGX Spark** (prefilled) — [`daqiri_bench_rdma_tx_rx_spark.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_rdma_tx_rx_spark.yaml). See [Socket and RDMA Benchmarking](../benchmarks/socket_benchmarking.md#run-the-rdma-roce-benchmark) for namespace and wire-counter run details.
- - **DGX Spark netns wire loopback** (prefilled, combined base) — [`daqiri_bench_rdma_tx_rx_spark_netns.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_rdma_tx_rx_spark_netns.yaml). Carries both roles; `examples/run_spark_bench.sh` (via `scripts/gen_spark_netns_config.py`) splits it per role and runs each in its own network namespace (`--mode server` / `--mode client`) so RDMA-CM resolves over the wire; see [Socket and RDMA Benchmarking](../benchmarks/socket_benchmarking.md#run-the-rdma-roce-benchmark).
- - **DGX Spark cross-host** (prefilled, runs on two Sparks) — [`daqiri_bench_rdma_tx_rx_spark_xhost.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_rdma_tx_rx_spark_xhost.yaml). Run with `--mode server` on the RX host and `--mode client` on the TX host. Apply the [cross-host network setup](../tutorials/system_configuration.md#cross-host-variant-two-sparks) before running. See the [Cross-host two-DGX-Spark loopback](../benchmarks/raw_benchmarking.md#cross-host-two-dgx-spark-loopback) section for run details.
+ - **Generic** (template, replace IPs): [`daqiri_bench_rdma_tx_rx.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_rdma_tx_rx.yaml).
+ - **DGX Spark** (prefilled): [`daqiri_bench_rdma_tx_rx_spark.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_rdma_tx_rx_spark.yaml). See [Socket and RDMA Benchmarking](../benchmarks/socket_benchmarking.md#run-the-rdma-roce-benchmark) for namespace and wire-counter run details.
+ - **DGX Spark netns wire loopback** (prefilled, combined base): [`daqiri_bench_rdma_tx_rx_spark_netns.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_rdma_tx_rx_spark_netns.yaml). Carries both roles. `examples/run_spark_bench.sh` (via `scripts/gen_spark_netns_config.py`) splits it per role and runs each in its own network namespace (`--mode server` / `--mode client`) so RDMA-CM resolves over the wire. See [Socket and RDMA Benchmarking](../benchmarks/socket_benchmarking.md#run-the-rdma-roce-benchmark).
+ - **DGX Spark cross-host** (prefilled, runs on two Sparks): [`daqiri_bench_rdma_tx_rx_spark_xhost.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_rdma_tx_rx_spark_xhost.yaml). Run with `--mode server` on the RX host and `--mode client` on the TX host. Apply the [cross-host network setup](../tutorials/system_configuration.md#cross-host-variant-two-sparks) before running. See the [Cross-host two-DGX-Spark loopback](../benchmarks/raw_benchmarking.md#cross-host-two-dgx-spark-loopback) section for run details.
- **Socket — UDP / TCP** (`stream_type: "socket"` with `udp://` or `tcp://` endpoints) — runs on `daqiri_bench_socket`. The shipped smoke-test configs bind to `127.0.0.1`; see [Socket and RDMA Benchmarking](../benchmarks/socket_benchmarking.md) for namespace-based wire tests.
+ **Socket (UDP / TCP)** (`stream_type: "socket"` with `udp://` or `tcp://` endpoints) runs on `daqiri_bench_socket`. The shipped smoke-test configs bind to `127.0.0.1`. See [Socket and RDMA Benchmarking](../benchmarks/socket_benchmarking.md) for namespace-based wire tests.
- - **UDP** — [`daqiri_bench_socket_udp_tx_rx.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_socket_udp_tx_rx.yaml).
- - **UDP — DGX Spark netns wire loopback** (combined base) — [`daqiri_bench_socket_udp_tx_rx_spark_netns.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_socket_udp_tx_rx_spark_netns.yaml). Carries both roles; `examples/run_spark_bench.sh` (via `scripts/gen_spark_netns_config.py`) splits it per role and runs each in its own network namespace (`--mode server` / `--mode client`) so same-host IPs cross the wire instead of looping through `lo`; see [Socket and RDMA Benchmarking](../benchmarks/socket_benchmarking.md#run-the-linux-socket-benchmark).
- - **TCP** — [`daqiri_bench_socket_tcp_tx_rx.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_socket_tcp_tx_rx.yaml).
- - **TCP — DGX Spark netns wire loopback** (combined base) — [`daqiri_bench_socket_tcp_tx_rx_spark_netns.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_socket_tcp_tx_rx_spark_netns.yaml). Carries both roles; `examples/run_spark_bench.sh` (via `scripts/gen_spark_netns_config.py`) splits it per role and runs each in its own network namespace (`--mode server` / `--mode client`); see [Socket and RDMA Benchmarking](../benchmarks/socket_benchmarking.md#run-the-linux-socket-benchmark).
+ - **UDP**: [`daqiri_bench_socket_udp_tx_rx.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_socket_udp_tx_rx.yaml).
+ - **UDP, DGX Spark netns wire loopback** (combined base): [`daqiri_bench_socket_udp_tx_rx_spark_netns.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_socket_udp_tx_rx_spark_netns.yaml). Carries both roles. `examples/run_spark_bench.sh` (via `scripts/gen_spark_netns_config.py`) splits it per role and runs each in its own network namespace (`--mode server` / `--mode client`) so same-host IPs cross the wire instead of looping through `lo`. See [Socket and RDMA Benchmarking](../benchmarks/socket_benchmarking.md#run-the-linux-socket-benchmark).
+ - **TCP**: [`daqiri_bench_socket_tcp_tx_rx.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_socket_tcp_tx_rx.yaml).
+ - **TCP, DGX Spark netns wire loopback** (combined base): [`daqiri_bench_socket_tcp_tx_rx_spark_netns.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_socket_tcp_tx_rx_spark_netns.yaml). Carries both roles. `examples/run_spark_bench.sh` (via `scripts/gen_spark_netns_config.py`) splits it per role and runs each in its own network namespace (`--mode server` / `--mode client`). See [Socket and RDMA Benchmarking](../benchmarks/socket_benchmarking.md#run-the-linux-socket-benchmark).
??? question "2. I have out-of-order UDP packets that need to be reordered on the GPU"
DAQIRI's flagship pipeline: a CUDA kernel reads a sequence number from each packet's header and places packets at the correct offset in a GPU buffer, so a downstream consumer sees a fully ordered stream without a CPU touch. Configs run on `daqiri_bench_raw_reorder_seq` unless 2.4 applies. Sub-questions:
**2.1 Which algorithm matches how your packets encode batches?**
- - *"My wire format sends a fixed N packets per logical batch; the seqno identifies position within the batch"* — `seq_packets_per_batch`.
- - *"My wire format identifies the batch index in the seqno; packets-per-batch is fixed for the stream"* — `seq_batch_number`.
+ - *"My wire format sends a fixed N packets per logical batch, and the seqno identifies position within the batch"*: `seq_packets_per_batch`.
+ - *"My wire format identifies the batch index in the seqno, and packets-per-batch is fixed for the stream"*: `seq_batch_number`.
**2.2 Where should the reorder run?**
- - GPU kernel (default, recommended) — `reorder_type: "gpu"`.
- - CPU (throughput-bounded; comparison/baseline path) — `reorder_type: "cpu"`.
+ - GPU kernel (default, recommended): `reorder_type: "gpu"`.
+ - CPU (throughput-bounded, comparison/baseline path): `reorder_type: "cpu"`.
**2.3 Self-contained, or do you have a TX peer?**
- - TX+RX — closed-loop in one process.
- - RX-only — you'll generate traffic separately. **A standalone run of any `raw_rx_*` config exits cleanly with `0` packets if no traffic arrives — that is not a bug; you need a TX peer.**
+ - TX+RX: closed-loop in one process.
+ - RX-only: you'll generate traffic separately. **A standalone run of any `raw_rx_*` config exits cleanly with `0` packets if no traffic arrives, which is expected. You need a TX peer.**
**2.4 Do you also need an in-kernel payload type conversion?**
- - No — pick a leaf from the table below.
- - Yes — [`daqiri_bench_raw_tx_rx_reorder_quantize_seq_batch.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_tx_rx_reorder_quantize_seq_batch.yaml) (runs on `daqiri_bench_raw_reorder_quantize`, not `daqiri_bench_raw_reorder_seq`). Combines `seq_batch_number` reorder with an in-kernel payload type conversion; the `data_types` block sets the input and output types (the example uses int4 → fp32). Pick this when wire format and compute format differ.
+ - No: pick a leaf from the table below.
+ - Yes: [`daqiri_bench_raw_tx_rx_reorder_quantize_seq_batch.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_tx_rx_reorder_quantize_seq_batch.yaml) (runs on `daqiri_bench_raw_reorder_quantize`, not `daqiri_bench_raw_reorder_seq`). Combines `seq_batch_number` reorder with an in-kernel payload type conversion. The `data_types` block sets the input and output types (the example uses int4 → fp32). Pick this when wire format and compute format differ.
Concrete leaves (without conversion):
@@ -105,9 +105,9 @@ For a shorter selection guide, start with the [Benchmarking overview](../benchma
A [diff-style walkthrough](#header-data-split-hds) of this config appears below.
??? question "4. I need flow-based load balancing across multiple RX queues"
- - **Closed-loop TX+RX with four queues** — [`daqiri_bench_raw_tx_rx_4q.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_tx_rx_4q.yaml) (runs on `daqiri_bench_raw_gpudirect`).
+ - **Closed-loop TX+RX with four queues**: [`daqiri_bench_raw_tx_rx_4q.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_tx_rx_4q.yaml) (runs on `daqiri_bench_raw_gpudirect`).
- [`daqiri_bench_raw_rx_multi_q.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_rx_multi_q.yaml) (runs on `daqiri_bench_raw_gpudirect`).
- - **Dynamic RX flow lifecycle** — [`daqiri_example_dynamic_rx_flow.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_example_dynamic_rx_flow.yaml) (runs on `daqiri_example_dynamic_rx_flow`). Starts with `flow_isolation: true` and no configured flows, then dynamically routes one UDP flow to RX queue 0 and queue 1 in sequence.
+ - **Dynamic RX flow lifecycle**: [`daqiri_example_dynamic_rx_flow.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_example_dynamic_rx_flow.yaml) (runs on `daqiri_example_dynamic_rx_flow`). Starts with `flow_isolation: true` and no configured flows, then dynamically routes one UDP flow to RX queue 0 and queue 1 in sequence.
The four-queue TX+RX config is self-contained and maps each `bench_tx`/`bench_rx` list entry to the matching DAQIRI queue. The RX-only config is for an external traffic source. The dynamic-flow example demonstrates queues-only startup and runtime flow insertion/deletion. All three demonstrate flow-rule-based routing across multiple RX queues, with explicit CPU cores for both DAQIRI queue workers and benchmark application workers.
@@ -118,24 +118,24 @@ For a shorter selection guide, start with the [Benchmarking overview](../benchma
??? question "5. I need to record packet data to disk"
Sub-question: **which output format?**
- **5.1 Wireshark- / tcpdump-compatible PCAP** — runs on `daqiri_example_pcap_writer`. Default; works on any filesystem. Run shape: `daqiri_example_pcap_writer [--tx]` (omit `--tx` for an RX-only tcpdump-style capture).
+ **5.1 Wireshark- / tcpdump-compatible PCAP** runs on `daqiri_example_pcap_writer`. This is the default and works on any filesystem. Run shape: `daqiri_example_pcap_writer [--tx]` (omit `--tx` for an RX-only tcpdump-style capture).
- - **Hardware loopback** — [`daqiri_example_pcap_writer_tx_rx.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_example_pcap_writer_tx_rx.yaml).
- - **No physical NIC available** — [`daqiri_example_pcap_writer_sw_loopback.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_example_pcap_writer_sw_loopback.yaml).
+ - **Hardware loopback**: [`daqiri_example_pcap_writer_tx_rx.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_example_pcap_writer_tx_rx.yaml).
+ - **No physical NIC available**: [`daqiri_example_pcap_writer_sw_loopback.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_example_pcap_writer_sw_loopback.yaml).
*Requires: Raw Ethernet build (`DAQIRI_ENGINE` includes `dpdk`). No special CMake flag.*
- **5.2 Zero-copy GPU → NVMe writes** (advanced) — runs on `daqiri_example_gds_write`. Pick this *only* if the GPU-to-disk zero-copy path is the specific subject of investigation; otherwise pick PCAP (5.1).
+ **5.2 Zero-copy GPU → NVMe writes** (advanced) runs on `daqiri_example_gds_write`. Pick this *only* if the GPU-to-disk zero-copy path is the specific subject of investigation. Otherwise pick PCAP (5.1).
- - **Hardware loopback** — [`daqiri_example_gds_write_tx_rx.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_example_gds_write_tx_rx.yaml).
- - **No physical NIC available** — [`daqiri_example_gds_write_sw_loopback.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_example_gds_write_sw_loopback.yaml).
+ - **Hardware loopback**: [`daqiri_example_gds_write_tx_rx.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_example_gds_write_tx_rx.yaml).
+ - **No physical NIC available**: [`daqiri_example_gds_write_sw_loopback.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_example_gds_write_sw_loopback.yaml).
*Requires: built with `-DDAQIRI_ENABLE_GDS=ON`, NVMe-backed storage, working cuFile / `nvidia_fs` stack, `gdscheck.py -p` reports `NVMe : Supported`.*
??? question "6. I need to cap (pace) the transmit rate in hardware"
- [`daqiri_bench_raw_tx_rx_pacing.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_tx_rx_pacing.yaml) (runs on `daqiri_bench_raw_gpudirect`).
- This is the base TX+RX template with a per-queue `pacing_mbps` cap added to the TX queue (it uses the default DPDK engine; pacing is supported only on the DPDK engine — the `ibverbs` engine does not support `pacing_mbps` and `daqiri_init()` fails if it is set on an `ibverbs` queue). The NIC meters the queue out so its average TX rate stays at or below the configured Mbps; the limit is enforced on an average basis and idle gaps do not accumulate burst credit. Set `pacing_mbps: 0` (or remove it) to send at line rate. Validate by computing the achieved rate from the benchmark's RX line (`Gbps = bytes * 8 / seconds / 1e9`) and confirming it tracks the configured cap. See the `pacing_mbps` key in the [TX queue configuration](../api-reference/configuration.md#transmit-configuration-tx).
+ This is the base TX+RX template with a per-queue `pacing_mbps` cap added to the TX queue (it uses the default DPDK engine. Pacing is supported only on the DPDK engine. The `ibverbs` engine does not support `pacing_mbps`, and `daqiri_init()` fails if it is set on an `ibverbs` queue). The NIC meters the queue out so its average TX rate stays at or below the configured Mbps. The limit is enforced on an average basis and idle gaps do not accumulate burst credit. Set `pacing_mbps: 0` (or remove it) to send at line rate. Validate by computing the achieved rate from the benchmark's RX line (`Gbps = bytes * 8 / seconds / 1e9`) and confirming it tracks the configured cap. See the `pacing_mbps` key in the [TX queue configuration](../api-reference/configuration.md#transmit-configuration-tx).
*Requires: a Mellanox/mlx5 NIC with hardware send scheduling (ConnectX-7 or later). On devices without it, `pacing_mbps` is ignored with a warning and TX runs at line rate.*
@@ -145,10 +145,10 @@ This section walks through four YAML topics: the base TX+RX template, flow steer
Annotations are prefixed with a category icon when applicable:
-- :material-wrench:{ title="System-specific" } **System-specific** — must be changed for your hardware
-- :material-package-variant:{ title="Payload-dependent" } **Payload-dependent** — adjust based on your application's packet format and throughput needs
+- :material-wrench:{ title="System-specific" } **System-specific**: must be changed for your hardware
+- :material-package-variant:{ title="Payload-dependent" } **Payload-dependent**: adjust based on your application's packet format and throughput needs
-In each code block, the lines you're most likely to tune are highlighted: system-specific addresses, cores, and MAC/IPs in the base walkthrough; feature-defining values (split boundaries, batch sizes, sequence-number positions) in the HDS and reorder diff snippets below.
+In each code block, the lines you're most likely to tune are highlighted: system-specific addresses, cores, and MAC/IPs in the base walkthrough, and feature-defining values (split boundaries, batch sizes, sequence-number positions) in the HDS and reorder diff snippets below.
### Base TX+RX config
@@ -227,35 +227,35 @@ bench_tx: # (25)!
udp_dst_port: 4096
```
-1. The `daqiri` section configures the DAQIRI library, which is responsible for setting up the NIC. It is passed to `daqiri_init(...)` during application startup. Within this section, `name:` fields on interfaces, queues, flows, and memory regions are used only for logging — pick any descriptive string.
-2. **`stream_type`** · `string` · *required* — High-level transport family selected for this config. **Supported:** `"raw"` (Raw Ethernet via kernel bypass, used here), `"socket"` (kernel sockets and RDMA). Use endpoint URI schemes such as `tcp://`, `udp://`, and `roce://` for socket-style transports.
-3. :material-wrench: **`master_core`** · `integer (CPU core ID)` · *required* — Core used for DAQIRI setup. Does not need to be isolated; recommended to differ from the `cpu_core` fields below that poll the NIC.
-4. **`loopback`** · `string` · *default: `""`* — Loopback mode. **Supported:** `""` (no loopback; use the physical NIC), `"sw"` (software loopback — no NIC required, used by the `*_sw_loopback*` configs for first-time build verification).
+1. The `daqiri` section configures the DAQIRI library, which is responsible for setting up the NIC. It is passed to `daqiri_init(...)` during application startup. Within this section, `name:` fields on interfaces, queues, flows, and memory regions are used only for logging. Pick any descriptive string.
+2. **`stream_type`** · `string` · *required*: High-level transport family selected for this config. **Supported:** `"raw"` (Raw Ethernet via kernel bypass, used here), `"socket"` (kernel sockets and RDMA). Use endpoint URI schemes such as `tcp://`, `udp://`, and `roce://` for socket-style transports.
+3. :material-wrench: **`master_core`** · `integer (CPU core ID)` · *required*: Core used for DAQIRI setup. Does not need to be isolated, and it is recommended to differ from the `cpu_core` fields below that poll the NIC.
+4. **`loopback`** · `string` · *default: `""`*: Loopback mode. **Supported:** `""` (no loopback, use the physical NIC), `"sw"` (software loopback, no NIC required, used by the `*_sw_loopback*` configs for first-time build verification).
5. The `memory_regions` section lists where the NIC will write/read data from/to when bypassing the OS kernel. Tip: when using GPU buffer regions, keeping the sum of their buffer sizes below 80% of your BAR1 size is generally a good rule of thumb.
-6. :material-package-variant: **`kind`** · `string` · *required* — Type of memory backing the region. **Supported:** `device` (GPU VRAM via GPUDirect — preferred on discrete GPUs), `host_pinned` (CPU pinned memory — required on integrated GPUs like NVIDIA GB10/DGX Spark where peer-DMA isn't available), `huge` (hugepages, CPU), `host` (CPU unpinned). See the [memory regions reference](../api-reference/configuration.md#memory-regions). Choose based on whether packets are processed on the GPU or CPU and on the GPU class.
-7. :material-wrench: **`affinity`** · `integer (GPU ID / NUMA node)` · *required* — GPU device ID when `kind: device` or `kind: host_pinned`; NUMA node ID for CPU memory regions (`huge`, `host`).
-8. :material-package-variant: **`num_bufs`** · `integer` · *required* — Number of buffers in the region. Higher gives more time to process packets but uses more BAR1 space; too low risks NIC drops (RX) or buffering latency (TX). A good starting point is 3×–5× the queue `batch_size`. For Raw Ethernet (`stream_type: "raw"`), `num_bufs` below 1.5× the NIC ring size deadlocks the worker; `daqiri_init` auto-bumps such regions to 3× the ring (24576 with the default 8192) and logs a `WARN`.
-9. :material-package-variant: **`buf_size`** · `integer (bytes)` · *required* — Size of each buffer in the region. Should equal your maximum packet size, or smaller when chaining regions per packet (e.g. header-data split — see the [HDS walkthrough](#header-data-split-hds) below).
+6. :material-package-variant: **`kind`** · `string` · *required*: Type of memory backing the region. **Supported:** `device` (GPU VRAM via GPUDirect, preferred on discrete GPUs), `host_pinned` (CPU pinned memory, required on integrated GPUs like NVIDIA GB10/DGX Spark where peer-DMA isn't available), `huge` (hugepages, CPU), `host` (CPU unpinned). See the [memory regions reference](../api-reference/configuration.md#memory-regions). Choose based on whether packets are processed on the GPU or CPU and on the GPU class.
+7. :material-wrench: **`affinity`** · `integer (GPU ID / NUMA node)` · *required*: GPU device ID when `kind: device` or `kind: host_pinned`. NUMA node ID for CPU memory regions (`huge`, `host`).
+8. :material-package-variant: **`num_bufs`** · `integer` · *required*: Number of buffers in the region. Higher gives more time to process packets but uses more BAR1 space. Too low risks NIC drops (RX) or buffering latency (TX). A good starting point is 3×–5× the queue `batch_size`. For Raw Ethernet (`stream_type: "raw"`), `num_bufs` below 1.5× the NIC ring size deadlocks the worker, so `daqiri_init` auto-bumps such regions to 3× the ring (24576 with the default 8192) and logs a `WARN`.
+9. :material-package-variant: **`buf_size`** · `integer (bytes)` · *required*: Size of each buffer in the region. Should equal your maximum packet size, or smaller when chaining regions per packet (e.g. header-data split, see the [HDS walkthrough](#header-data-split-hds) below).
10. The `interfaces` section lists the NIC interfaces that will be configured for the application.
-11. :material-wrench: **`address`** · `string (PCIe BDF)` · *required* — PCIe bus address of this interface. **Must be changed for your system.** Both `tx_port` and `rx_port` may point to the same physical NIC for single-port closed-loop benches.
+11. :material-wrench: **`address`** · `string (PCIe BDF)` · *required*: PCIe bus address of this interface. **Must be changed for your system.** Both `tx_port` and `rx_port` may point to the same physical NIC for single-port closed-loop benches.
12. Each interface declares a `tx` (transmitting) and/or `rx` (receiving) section. Include only the side you're using on that port, or both if a single port carries traffic in both directions.
-13. The `queues` section lists per-direction queues. Queues are a core NIC concept: they handle the actual reception or transmission of packets. RX queues buffer incoming packets until the application processes them; TX queues hold outgoing packets waiting to be sent. The simplest setup uses one RX and one TX queue; using more queues allows parallel streams (each queue can be pinned to its own CPU core and memory region).
-14. :material-package-variant: **`batch_size`** · `integer (packets)` · *required* — Packets per burst. The RX path delivers packets to the application in batches of this size; the TX path should not send more packets than this per call.
-15. :material-wrench: **`cpu_core`** · `integer (CPU core ID)` · *required* — Core that this queue uses to poll the NIC. Ideally one [isolated core](system_configuration.md#step-5-isolate-cpu-cores) per queue. **Must match your system's available cores.**
+13. The `queues` section lists per-direction queues. Queues are a core NIC concept: they handle the actual reception or transmission of packets. RX queues buffer incoming packets until the application processes them, while TX queues hold outgoing packets waiting to be sent. The simplest setup uses one RX and one TX queue. Using more queues allows parallel streams (each queue can be pinned to its own CPU core and memory region).
+14. :material-package-variant: **`batch_size`** · `integer (packets)` · *required*: Packets per burst. The RX path delivers packets to the application in batches of this size. The TX path should not send more packets than this per call.
+15. :material-wrench: **`cpu_core`** · `integer (CPU core ID)` · *required*: Core that this queue uses to poll the NIC. Ideally one [isolated core](system_configuration.md#step-5-isolate-cpu-cores) per queue. **Must match your system's available cores.**
16. The list of memory regions where this queue will write/read packets. **Order matters:** the first region is used until one buffer fills (`buf_size`), then the next region is used, and so on until the packet is fully written/read. The [HDS walkthrough](#header-data-split-hds) below shows a chained example.
-17. **`offloads`** · `list of string` · *TX queues only* — Optional tasks offloaded to the NIC. **Supported:** `tx_eth_src` (the NIC inserts the Ethernet source MAC into outgoing headers). `daqiri_init()` fails if the NIC cannot install the offload flow rule. Note: IP, UDP, and Ethernet checksums/CRC are always done by the NIC and are not optional.
-18. :material-wrench: **`address`** · `string (PCIe BDF)` · *required* — PCIe bus address of the RX interface. May share the BDF with `tx_port` for single-port closed-loop benches, or be a different NIC for two-port setups. **Must be changed for your system.**
-19. **`flow_isolation`** · `boolean` · *default: `false`* — When `true`, static startup flows send unmatched traffic in their flow class back to the Linux kernel via fallback rules, while queues-only dynamic configs deliver no traffic to DAQIRI queues until dynamic rules are installed. Useful for letting the interface still handle ARP, ICMP, etc. while DAQIRI takes the application packets. When `false`, every packet hitting the interface must be processed (or dropped) by your application.
-20. The list of static startup flows. Flows route packets to a queue based on packet fields; queues-only configs may omit this list and add dynamic RX flows after initialization. For Raw Ethernet, each configured rule is programmed into the NIC during `daqiri_init()`; initialization fails if any rule or the send-to-kernel fallback (when `flow_isolation: true`) cannot be installed. Per interface, use only standard UDP/IP flows or only flex-item flows — not both.
-21. **`id`** · `integer` · *required* — Tag attached to packets that match this flow. Useful when multiple flows route to a single queue and the application needs to distinguish which rule matched.
+17. **`offloads`** · `list of string` · *TX queues only*: Optional tasks offloaded to the NIC. **Supported:** `tx_eth_src` (the NIC inserts the Ethernet source MAC into outgoing headers). `daqiri_init()` fails if the NIC cannot install the offload flow rule. Note: IP, UDP, and Ethernet checksums/CRC are always done by the NIC and are not optional.
+18. :material-wrench: **`address`** · `string (PCIe BDF)` · *required*: PCIe bus address of the RX interface. May share the BDF with `tx_port` for single-port closed-loop benches, or be a different NIC for two-port setups. **Must be changed for your system.**
+19. **`flow_isolation`** · `boolean` · *default: `false`*: When `true`, static startup flows send unmatched traffic in their flow class back to the Linux kernel via fallback rules, while queues-only dynamic configs deliver no traffic to DAQIRI queues until dynamic rules are installed. Useful for letting the interface still handle ARP, ICMP, etc. while DAQIRI takes the application packets. When `false`, every packet hitting the interface must be processed (or dropped) by your application.
+20. The list of static startup flows. Flows route packets to a queue based on packet fields. Queues-only configs may omit this list and add dynamic RX flows after initialization. For Raw Ethernet, each configured rule is programmed into the NIC during `daqiri_init()`, and initialization fails if any rule or the send-to-kernel fallback (when `flow_isolation: true`) cannot be installed. Per interface, use only standard UDP/IP flows or only flex-item flows, not both.
+21. **`id`** · `integer` · *required*: Tag attached to packets that match this flow. Useful when multiple flows route to a single queue and the application needs to distinguish which rule matched.
22. What to do with packets that match this flow. Existing configs can use the legacy single `action:` map with `type: queue` to send packets to the queue with the given `id`. New hardware transform flows use ordered `actions:`; RX transform flows can `vlan_pop` or `tunnel_decap` and must end with `type: queue`. That queue `id` must match an `rx.queues` entry on this interface; `daqiri_init()` rejects unknown queue IDs during config validation.
23. :material-package-variant: List of rules to match packets against. **All** rules must hold for a packet to match the flow. Currently supported keys: `udp_src` / `udp_dst` (UDP source/destination port numbers, integer), `ipv4_len` (full IPv4 packet length in bytes, integer). **Adjust to match your incoming traffic.**
-24. The `bench_rx` section is specific to the benchmark application. It is a list of application worker configs; list entries map to DAQIRI RX queues on the named interface, either by explicit `queue_id` or by queue-list order when `queue_id` is omitted. `cpu_core` pins the benchmark application's RX worker thread; it is separate from the DAQIRI queue `cpu_core` above, and can use the same core only when you intentionally want to share. In this base config there is one RX queue, so there is one entry. Other DAQIRI binaries (e.g. the reorder-quantize bench) may add fields here; see those configs for details.
-25. :material-package-variant: The `bench_tx` section configures the TX side of the benchmark: the benchmark application's TX worker core, packet sizes, and the Ethernet/IP/UDP header fields embedded in outgoing packets. It is a list for the same reason as `bench_rx`: each entry maps to a DAQIRI TX queue on the named interface, either by explicit `queue_id` or by queue-list order when `queue_id` is omitted. The `cpu_core` field pins the application TX thread. The `eth_dst_addr` is required when `flow_isolation` is enabled on the RX interface. The `ip_src_addr` / `ip_dst_addr` are only needed when traffic is routed across subnets — for a direct cable loopback, any value works. The `payload_size`, `header_size`, and UDP ports should match your application's packet format. Hardware TX VLAN push or tunnel encapsulation is configured under `daqiri.cfg.interfaces[].tx.flows`, not in `bench_tx`; application buffers remain the pre-encap packet.
+24. The `bench_rx` section is specific to the benchmark application. It is a list of application worker configs. List entries map to DAQIRI RX queues on the named interface, either by explicit `queue_id` or by queue-list order when `queue_id` is omitted. `cpu_core` pins the benchmark application's RX worker thread. It is separate from the DAQIRI queue `cpu_core` above, and can use the same core only when you intentionally want to share. In this base config there is one RX queue, so there is one entry. Other DAQIRI binaries (e.g. the reorder-quantize bench) may add fields here. See those configs for details.
+25. :material-package-variant: The `bench_tx` section configures the TX side of the benchmark: the benchmark application's TX worker core, packet sizes, and the Ethernet/IP/UDP header fields embedded in outgoing packets. It is a list for the same reason as `bench_rx`: each entry maps to a DAQIRI TX queue on the named interface, either by explicit `queue_id` or by queue-list order when `queue_id` is omitted. The `cpu_core` field pins the application TX thread. The `eth_dst_addr` is required when `flow_isolation` is enabled on the RX interface. The `ip_src_addr` / `ip_dst_addr` are only needed when traffic is routed across subnets. For a direct cable loopback, any value works. The `payload_size`, `header_size`, and UDP ports should match your application's packet format. Hardware TX VLAN push or tunnel encapsulation is configured under `daqiri.cfg.interfaces[].tx.flows`, not in `bench_tx`. Application buffers remain the pre-encap packet.
### Flow steering
-Raw Ethernet RX can steer packets into GPU queues, a flow-matched host queue, or a kernel fallback path for traffic that matches no rule. The animation below is a conceptual superset of the paths DAQIRI supports; the four-queue YAML example maps one UDP flow per GPU RX queue.
+Raw Ethernet RX can steer packets into GPU queues, a flow-matched host queue, or a kernel fallback path for traffic that matches no rule. The animation below is a conceptual superset of the paths DAQIRI supports. The four-queue YAML example maps one UDP flow per GPU RX queue.
@@ -273,7 +273,7 @@ For applications that parse small per-packet fields on the CPU while keeping the
-The canonical HDS config is [`daqiri_bench_raw_tx_rx_hds.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_tx_rx_hds.yaml). It builds on the base TX+RX config above; only the deltas are shown here.
+The canonical HDS config is [`daqiri_bench_raw_tx_rx_hds.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_tx_rx_hds.yaml). It builds on the base TX+RX config above. Only the deltas are shown here.
**New CPU memory regions, one per direction.** Headers land here.
@@ -302,8 +302,8 @@ memory_regions:
```
1. New region for headers. `kind: "huge"` puts buffers in CPU hugepages so the CPU can read them without touching GPU memory. Pair `Data_TX_CPU` and `Data_RX_CPU` with their GPU siblings to chain regions per packet.
-2. :material-package-variant: **`buf_size`** · `integer (bytes)` · *required* — In HDS, the **first region's `buf_size` is the split boundary**: bytes 0 to `buf_size − 1` go to the CPU region, the remainder spills into the next region. Size this to match your header length exactly (64 bytes for a typical Eth+IPv4+UDP header).
-3. :material-package-variant: **`buf_size`** · `integer (bytes)` · *required* — Size of each GPU buffer in HDS mode: payload-only (no longer the full packet size). The packet first fills 64 bytes in `Data_RX_CPU`, then the remaining 1000 bytes spill into `Data_RX_GPU`.
+2. :material-package-variant: **`buf_size`** · `integer (bytes)` · *required*: In HDS, the **first region's `buf_size` is the split boundary**: bytes 0 to `buf_size − 1` go to the CPU region, the remainder spills into the next region. Size this to match your header length exactly (64 bytes for a typical Eth+IPv4+UDP header).
+3. :material-package-variant: **`buf_size`** · `integer (bytes)` · *required*: Size of each GPU buffer in HDS mode: payload-only (no longer the full packet size). The packet first fills 64 bytes in `Data_RX_CPU`, then the remaining 1000 bytes spill into `Data_RX_GPU`.
**Chained `memory_regions:` per queue.** Order matters: header region first, payload region second. The NIC walks the list in order, filling each region's `buf_size` before moving on.
@@ -332,7 +332,7 @@ flows:
ipv4_len: 1050 # (1)!
```
-1. :material-package-variant: **`ipv4_len`** · `integer (bytes)` · *optional in the base config; recommended for HDS* — Match on the full IPv4 packet length. Required for HDS so the NIC can split deterministically. Set to the fixed packet length you expect.
+1. :material-package-variant: **`ipv4_len`** · `integer (bytes)` · *optional in the base config, recommended for HDS*: Match on the full IPv4 packet length. Required for HDS so the NIC can split deterministically. Set to the fixed packet length you expect.
**Match `bench_tx.payload_size` to the GPU region.** The TX side generates 1000-byte payloads to match `Data_RX_GPU.buf_size`:
@@ -356,7 +356,7 @@ For UDP workloads where packets arrive out-of-order and need to be placed at the
-The canonical reorder config is [`daqiri_bench_raw_tx_rx_reorder_seq_1024.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_tx_rx_reorder_seq_1024.yaml) (`seq_packets_per_batch` algorithm, GPU kernel, closed-loop TX+RX). It builds on the base TX+RX config above; only the deltas are shown here.
+The canonical reorder config is [`daqiri_bench_raw_tx_rx_reorder_seq_1024.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_tx_rx_reorder_seq_1024.yaml) (`seq_packets_per_batch` algorithm, GPU kernel, closed-loop TX+RX). It builds on the base TX+RX config above. Only the deltas are shown here.
**New `Reorder_RX_GPU` memory region.** A large, dedicated GPU region that holds one fully reordered batch.
@@ -379,11 +379,11 @@ memory_regions:
buf_size: 2031616 # (3)!
```
-1. :material-package-variant: **`num_bufs`** · `integer` · *required* — Shrunk from the base config's 51200 to 16384. Reorder works at smaller batches with smaller per-packet buffers (`buf_size: 2048` here vs. 8064 in the base), so the buffer pool is correspondingly smaller.
+1. :material-package-variant: **`num_bufs`** · `integer` · *required*: Shrunk from the base config's 51200 to 16384. Reorder works at smaller batches with smaller per-packet buffers (`buf_size: 2048` here vs. 8064 in the base), so the buffer pool is correspondingly smaller.
2. **New region for the kernel output.** Each buffer holds one fully reordered batch of `packets_per_batch` packets, payload-only (the header is stripped via `payload_byte_offset`).
-3. :material-package-variant: **`buf_size`** · `integer (bytes)` · *required* — Sized as `(source buf_size − payload_byte_offset) × packets_per_batch`. For this config: `(2048 − 64) × 1024 = 2,031,616` bytes. Re-derive when you change any of the three inputs.
+3. :material-package-variant: **`buf_size`** · `integer (bytes)` · *required*: Sized as `(source buf_size − payload_byte_offset) × packets_per_batch`. For this config: `(2048 − 64) × 1024 = 2,031,616` bytes. Re-derive when you change any of the three inputs.
-**Match queue `batch_size` to `packets_per_batch`.** The reorder kernel processes exactly one batch per invocation; the queue must hand it that many packets at once.
+**Match queue `batch_size` to `packets_per_batch`.** The reorder kernel processes exactly one batch per invocation. The queue must hand it that many packets at once.
```yaml hl_lines="3 4"
rx:
@@ -394,8 +394,8 @@ rx:
- "Data_RX_GPU"
```
-1. :material-package-variant: **`batch_size`** · `integer (packets)` · *required* — Must equal `packets_per_batch` in the reorder config below.
-2. :material-package-variant: **`timeout_us`** · `integer (microseconds)` · *default: none (waits forever)* — Maximum time the queue waits for a partial batch to fill before flushing. Without it, a stalled flow can stall the reorder kernel indefinitely.
+1. :material-package-variant: **`batch_size`** · `integer (packets)` · *required*: Must equal `packets_per_batch` in the reorder config below.
+2. :material-package-variant: **`timeout_us`** · `integer (microseconds)` · *default: none (waits forever)*: Maximum time the queue waits for a partial batch to fill before flushing. Without it, a stalled flow can stall the reorder kernel indefinitely.
**Flow `id` tags packets for the reorder config.** The reorder block selects packets to reorder by flow ID.
@@ -411,9 +411,9 @@ flows:
udp_dst: 5000
```
-1. **`id`** · `integer` · *required* — Flow tag attached to matching packets. Set to a non-zero value here so the `reorder_configs:` block below can reference it via `flow_ids:` to select which packets to reorder.
+1. **`id`** · `integer` · *required*: Flow tag attached to matching packets. Set to a non-zero value here so the `reorder_configs:` block below can reference it via `flow_ids:` to select which packets to reorder.
-**The `reorder_configs:` block.** The core of the feature — sits inside the `rx:` section alongside `queues` and `flows`.
+**The `reorder_configs:` block.** The core of the feature, it sits inside the `rx:` section alongside `queues` and `flows`.
```yaml hl_lines="5 11 12 13"
reorder_configs:
@@ -431,13 +431,13 @@ reorder_configs:
packets_per_batch: 1024 # (7)!
```
-1. **`reorder_type`** · `string` · *required* — Where the kernel runs. **Supported:** `"gpu"` (CUDA kernel, recommended), `"cpu"` (throughput-bounded; comparison baseline — see `daqiri_bench_raw_tx_rx_reorder_seq_1024_cpu.yaml`).
-2. **`memory_region`** · `string` · *required* — Name of the landing region for reordered output. Must match a region defined in the top-level `memory_regions:` (here, `Reorder_RX_GPU`).
-3. :material-package-variant: **`payload_byte_offset`** · `integer (bytes)` · *required* — Number of leading bytes (typically the header) to skip when copying packets into the reorder region. The kernel copies from this offset to the end of the source buffer.
+1. **`reorder_type`** · `string` · *required*: Where the kernel runs. **Supported:** `"gpu"` (CUDA kernel, recommended), `"cpu"` (throughput-bounded, comparison baseline, see `daqiri_bench_raw_tx_rx_reorder_seq_1024_cpu.yaml`).
+2. **`memory_region`** · `string` · *required*: Name of the landing region for reordered output. Must match a region defined in the top-level `memory_regions:` (here, `Reorder_RX_GPU`).
+3. :material-package-variant: **`payload_byte_offset`** · `integer (bytes)` · *required*: Number of leading bytes (typically the header) to skip when copying packets into the reorder region. The kernel copies from this offset to the end of the source buffer.
4. List of flow IDs whose packets feed this reorder config. Must match the `id` field of one or more `flows:` entries above.
-5. **`method`** — Algorithm choice. `seq_packets_per_batch` (used here) groups a fixed number of packets per batch, identified by a sequence number within the batch. The alternative `seq_batch_number` encodes the batch index directly in the seqno — see [`daqiri_bench_raw_tx_rx_reorder_quantize_seq_batch.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_tx_rx_reorder_quantize_seq_batch.yaml).
-6. :material-package-variant: **`bit_offset`** / **`bit_width`** · `integer (bits)` · *required* — Location and size of the sequence number within the packet. Here, the seqno starts at byte 64 (`bit_offset: 512` = 64 × 8) and is 32 bits wide — matching a `uint32` at the start of the UDP payload.
-7. :material-package-variant: **`packets_per_batch`** · `integer (packets)` · *required* — Number of packets the kernel groups per reordered batch. Must equal the queue `batch_size` above.
+5. **`method`**: Algorithm choice. `seq_packets_per_batch` (used here) groups a fixed number of packets per batch, identified by a sequence number within the batch. The alternative `seq_batch_number` encodes the batch index directly in the seqno. See [`daqiri_bench_raw_tx_rx_reorder_quantize_seq_batch.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_tx_rx_reorder_quantize_seq_batch.yaml).
+6. :material-package-variant: **`bit_offset`** / **`bit_width`** · `integer (bits)` · *required*: Location and size of the sequence number within the packet. Here, the seqno starts at byte 64 (`bit_offset: 512` = 64 × 8) and is 32 bits wide, matching a `uint32` at the start of the UDP payload.
+7. :material-package-variant: **`packets_per_batch`** · `integer (packets)` · *required*: Number of packets the kernel groups per reordered batch. Must equal the queue `batch_size` above.
**TX-side seqno injection.** The benchmark TX path writes a monotonic big-endian `uint32` into the configured payload offset.
@@ -452,7 +452,7 @@ bench_tx:
sequence_number_start: 0
```
-1. :material-package-variant: **`sequence_number_offset`** · `integer (bytes)` — Byte offset into the UDP payload where the TX path writes the monotonic seqno. Must align with `sequence_number.bit_offset` in the RX reorder config (after subtracting the header size). Here the seqno is at the very start of the payload.
+1. :material-package-variant: **`sequence_number_offset`** · `integer (bytes)`: Byte offset into the UDP payload where the TX path writes the monotonic seqno. Must align with `sequence_number.bit_offset` in the RX reorder config (after subtracting the header size). Here the seqno is at the very start of the payload.
The reorder bench runs on `daqiri_bench_raw_reorder_seq`:
@@ -467,6 +467,3 @@ When the wire format and compute format differ, the quantize variant adds an in-
Other reorder variants are listed under [question 2 of the decision tree above](#choosing-an-example-config): the CPU-kernel variant, the RX-only variants, and the `seq_batch_number` algorithm with in-kernel int4 → fp32 type conversion (runs on `daqiri_bench_raw_reorder_quantize`).
-
----
-**Previous:** [Raw Ethernet Benchmarking](../benchmarks/raw_benchmarking.md)
diff --git a/docs/tutorials/daqiri-holoscan-integration.md b/docs/tutorials/daqiri-holoscan-integration.md
index a18d58f..8c31838 100644
--- a/docs/tutorials/daqiri-holoscan-integration.md
+++ b/docs/tutorials/daqiri-holoscan-integration.md
@@ -27,16 +27,16 @@ Holoscan.
## Application Lifecycle
-A Holoscan application is a directed acyclic graph of operators — the fundamental
-units of work — connected by data flows and run by a scheduler. DAQIRI owns NIC
-setup, packet memory, and RX burst lifetimes; the operators in the graph pull
+A Holoscan application is a directed acyclic graph of operators (the fundamental
+units of work) connected by data flows and run by a scheduler. DAQIRI owns NIC
+setup, packet memory, and RX burst lifetimes. The operators in the graph pull
received data from DAQIRI and pass it downstream as tensors.
The recommended Holoscan pattern is to pass one YAML file into the application
with `app->config(...)`, read Holoscan operator parameters with `from_config(...)`,
and initialize DAQIRI from the same YAML during graph composition before any
DAQIRI-backed operators run. DAQIRI handles any configured GPU reorder/quantize
-plan internally; the Holoscan source operator is just the adapter that polls DAQIRI
+plan internally. The Holoscan source operator is just the adapter that polls DAQIRI
and emits completed batches to downstream operators.
```cpp
@@ -209,7 +209,7 @@ NIC while other workers run downstream operators. With `stop_on_deadlock: true`
scheduler returns from `app->run()` once no operator can make progress for
`stop_on_deadlock_timeout` ms (e.g. the stream ends); `SIGINT` (Ctrl-C) also stops
it. Either way control returns to `main()` and the teardown prints stats and shuts
-DAQIRI down. To bound a run explicitly — for a smoke test — attach a stop condition
+DAQIRI down. To bound a run explicitly, for a smoke test, attach a stop condition
such as `make_condition(num_batches)` to the operator in
`compose()`.
@@ -307,7 +307,7 @@ class DaqiriRxOp : public holoscan::Operator {
return;
}
- // The reorder/quantize kernel filled the batch asynchronously on our stream; the
+ // The reorder/quantize kernel filled the batch asynchronously on our stream, and the
// burst carries the completion event. Fence on it before reading the data.
if (burst->event != nullptr) {
cudaEventSynchronize(burst->event);
@@ -326,7 +326,7 @@ class DaqiriRxOp : public holoscan::Operator {
// Wrap that device buffer as a tensor and emit it zero-copy. The tensor takes
// ownership of the burst and frees it via the DLPack deleter once downstream is
- // done — so we do NOT free the burst here.
+ // done, so we do NOT free the burst here.
auto tensor = wrap_reorder_output_as_tensor(burst_guard.get(), batch, info);
burst_guard.release(); // Tensor's DLPack deleter now owns the burst.
@@ -355,11 +355,11 @@ Because the DAQIRI reorder plan reorders and quantizes the payloads on the GPU,
operator never touches individual packets: it dequeues a burst, checks the
`DAQIRI_BURST_FLAG_REORDERED` flag, waits on the burst's CUDA event, and reads one
contiguous output buffer. `daqiri::get_reorder_burst_info(...)` returns the batch
-layout — `packets_per_batch` sequence slots, `payload_len` output bytes per slot,
-`aggregate_len` total — so the operator can shape the tensor correctly.
+layout: `packets_per_batch` sequence slots, `payload_len` output bytes per slot, and
+`aggregate_len` total, so the operator can shape the tensor correctly.
-To process the stream differently — a different reorder layout, a custom
-quantization, or any other per-batch GPU transform — you can add your own CUDA kernel
+To process the stream differently (a different reorder layout, a custom
+quantization, or any other per-batch GPU transform), you can add your own CUDA kernel
instead of (or after) the built-in plan: run it on the operator's stream over the raw
RX payloads and emit its output buffer the same way. DAQIRI's reorder kernels live in
`src/kernels.cu` (build with `-DDAQIRI_REORDER_GPU_PROFILE=ON` for CUDA event timing)
@@ -417,7 +417,7 @@ std::shared_ptr DaqiriRxOp::wrap_reorder_output_as_tensor(
};
auto tensor = std::make_shared(dl.get());
- // Tensor construction succeeded; its DLPack deleter now owns dl and shape.
+ // Tensor construction succeeded. Its DLPack deleter now owns dl and shape.
shape.release();
dl.release();
return tensor;
@@ -426,7 +426,7 @@ std::shared_ptr DaqiriRxOp::wrap_reorder_output_as_tensor(
Downstream operators receive an fp32 `[packets_per_batch, elements_per_packet]` tensor
on the GPU, ready for inference or further processing with no host copy. The exact
-DLPack and tensor-construction details track the Holoscan SDK version; a complete,
+DLPack and tensor-construction details track the Holoscan SDK version. A complete,
build-tested implementation lives in the Holohub example (see [References](#references)).
## Sink Operator Skeleton
@@ -477,7 +477,7 @@ class TensorSinkOp : public holoscan::Operator {
- The standalone `daqiri_bench_raw_reorder_quantize` benchmark
(`examples/raw_reorder_quantize_bench.cpp`, config
`daqiri_bench_raw_tx_rx_reorder_quantize_seq_batch.yaml`) exercises the same
- reorder/quantize path outside Holoscan — a good way to validate the DAQIRI config
+ reorder/quantize path outside Holoscan, a good way to validate the DAQIRI config
first. See [Raw Ethernet Benchmarking](../benchmarks/raw_benchmarking.md).
- The landed Holohub reference app is
[`applications/daqiri_raw_ethernet_bench`](https://github.com/nvidia-holoscan/holohub/tree/main/applications/daqiri_raw_ethernet_bench).
diff --git a/docs/tutorials/system_configuration.md b/docs/tutorials/system_configuration.md
index 370c709..66c714d 100644
--- a/docs/tutorials/system_configuration.md
+++ b/docs/tutorials/system_configuration.md
@@ -5,7 +5,7 @@ hide:
# System Configuration
-DAQIRI requires an [**NVIDIA SmartNIC**](https://www.nvidia.com/en-us/networking/ethernet-adapters/) (ConnectX-6 Dx or later) and a CUDA-capable GPU. Two reference platforms are documented in this tutorial — pick the one closest to yours below:
+DAQIRI requires an [**NVIDIA SmartNIC**](https://www.nvidia.com/en-us/networking/ethernet-adapters/) (ConnectX-6 Dx or later) and a CUDA-capable GPU. Two reference platforms are documented in this tutorial. Pick the one closest to yours below:
- **IGX Orin** with a discrete GPU (e.g. [RTX 6000 Ada](https://www.nvidia.com/en-us/design-visualization/rtx-6000/)): peermem-based GPUDirect, a separate GPU BAR1, and a discrete-PCIe path between GPU and NIC. The originally-supported reference platform.
- **DGX Spark** (Grace Blackwell **GB10** superchip): unified CPU/GPU memory via NVLink-C2C, integrated **ConnectX-7**, no peermem, and GPUDirect via `kind: host_pinned` data buffers.
@@ -141,7 +141,7 @@ DAQIRI requires an [**NVIDIA SmartNIC**](https://www.nvidia.com/en-us/networking
**For DAQIRI, we want the NIC to use the ETH link layer.** To switch the link layer mode, there are two possible options:
1. On IGX Orin developer kits, you can switch that setting through the BIOS: [see IGX Orin documentation](https://docs.nvidia.com/igx-orin/user-guide/latest/switch-network-link.html).
- 2. On any system with a NVIDIA NIC (including the IGX Orin developer kits), you can run the commands below from a terminal:
+ 2. On any system with an NVIDIA NIC (including the IGX Orin developer kits), you can run the commands below from a terminal:
1. Identify the PCI address of your NVIDIA NIC
@@ -359,7 +359,7 @@ DAQIRI requires an [**NVIDIA SmartNIC**](https://www.nvidia.com/en-us/networking
The section below is for advanced users looking to extract more performance out of their system. You can choose to skip this section and return to it later if performance if your application is not satisfactory.
- While the configurations above are the minimum requirements to get a NIC and a NVIDIA GPU to communicate while bypassing the OS kernel stack, performance can be further improved in most scenarios by tuning the system as described below.
+ While the configurations above are the minimum requirements to get a NIC and an NVIDIA GPU to communicate while bypassing the OS kernel stack, performance can be further improved in most scenarios by tuning the system as described below.
The table below summarizes all optimization steps covered in this section, along with the corresponding `tune_system.py` flags and whether each setting can be made persistent across reboots. Use it as a checklist to track your progress.
@@ -367,19 +367,19 @@ DAQIRI requires an [**NVIDIA SmartNIC**](https://www.nvidia.com/en-us/networking
|------|-------------|--------------------|-------------|
| 1 | [PCIe topology](#step-1-ensure-ideal-pcie-topology) | `--check topo` | N/A (hardware) |
| 2 | [PCIe config (MPS/Speed)](#step-2-check-the-nics-pcie-configuration) | `--check mps` | N/A (hardware) |
- | 3 | [NIC MRRS](#step-3-maximize-the-nics-max-read-request-size-mrrs) | `--check mrrs` / `--set mrrs` | No — use a startup script |
- | 4 | [Hugepages](#step-4-enable-huge-pages) | `--check hugepages` | Yes — kernel bootline or `/etc/fstab` |
- | 5 | [CPU isolation](#step-5-isolate-cpu-cores) | `--check cmdline` | Yes — kernel bootline |
- | 6 | [CPU governor](#step-6-prevent-cpu-cores-from-going-idle) | `--check cpu-freq` | Yes — see persistent option in section |
- | 7 | [GPU clocks](#step-7-prevent-the-gpu-from-going-idle) | `--check gpu-clock` | Partial — `nvidia-smi -pm 1` persists driver; clock locks need a startup script |
- | 8 | [GPU BAR1 size](#step-8-maximize-gpu-bar1-size) | `--check bar1-size` | Yes — firmware flash |
- | 9 | [Jumbo frames (MTU)](#step-9-enable-jumbo-frames) | `--check mtu` | Yes — see persistent option in section |
+ | 3 | [NIC MRRS](#step-3-maximize-the-nics-max-read-request-size-mrrs) | `--check mrrs` / `--set mrrs` | No, use a startup script |
+ | 4 | [Hugepages](#step-4-enable-huge-pages) | `--check hugepages` | Yes, kernel bootline or `/etc/fstab` |
+ | 5 | [CPU isolation](#step-5-isolate-cpu-cores) | `--check cmdline` | Yes, kernel bootline |
+ | 6 | [CPU governor](#step-6-prevent-cpu-cores-from-going-idle) | `--check cpu-freq` | Yes, see persistent option in section |
+ | 7 | [GPU clocks](#step-7-prevent-the-gpu-from-going-idle) | `--check gpu-clock` | Partial. `nvidia-smi -pm 1` persists driver, clock locks need a startup script |
+ | 8 | [GPU BAR1 size](#step-8-maximize-gpu-bar1-size) | `--check bar1-size` | Yes, firmware flash |
+ | 9 | [Jumbo frames (MTU)](#step-9-enable-jumbo-frames) | `--check mtu` | Yes, see persistent option in section |
!!! tip "Plan your reboots"
- Several steps below require adding flags to the kernel bootline in `/etc/default/grub` (hugepages in [Enable Huge pages](#step-4-enable-huge-pages), CPU isolation in [Isolate CPU cores](#step-5-isolate-cpu-cores)). We recommend reading through both sections first and adding all the flags at once to avoid multiple reboots. Other items like MRRS, GPU clocks, and MTU can be applied at runtime but reset on reboot — consider scripting them or using a systemd service for persistence.
+ Several steps below require adding flags to the kernel bootline in `/etc/default/grub` (hugepages in [Enable Huge pages](#step-4-enable-huge-pages), CPU isolation in [Isolate CPU cores](#step-5-isolate-cpu-cores)). We recommend reading through both sections first and adding all the flags at once to avoid multiple reboots. Other items like MRRS, GPU clocks, and MTU can be applied at runtime but reset on reboot, so consider scripting them or using a systemd service for persistence.
- Before diving in each of the setups below, we provide a utility script as part of the DAQIRI library which provides an overview of the configurations that potentially need to be tuned on your system. The script (`python/tune_system.py`) is run on the host from a clone of the DAQIRI repo — it touches host PCIe/sysfs and is not intended to run inside the build container.
+ Before diving in each of the setups below, we provide a utility script as part of the DAQIRI library which provides an overview of the configurations that potentially need to be tuned on your system. The script (`python/tune_system.py`) is run on the host from a clone of the DAQIRI repo. It touches host PCIe/sysfs and is not intended to run inside the build container.
??? example "Work In Progress"
@@ -835,7 +835,7 @@ DAQIRI requires an [**NVIDIA SmartNIC**](https://www.nvidia.com/en-us/networking
cat /proc/cmdline | grep -e isolcpus -e irqaffinity -e nohz_full -e rcu_nocbs -e rcu_nocb_poll
```
- Decide which cores to isolate based on your configuration. We recommend one core per DAQIRI queue, plus one core per benchmark application worker thread (the `bench_tx` / `bench_rx` `cpu_core` fields), as a rule of thumb — these are separate busy-poll threads and should not share a core. First, identify your core IDs:
+ Decide which cores to isolate based on your configuration. We recommend one core per DAQIRI queue, plus one core per benchmark application worker thread (the `bench_tx` / `bench_rx` `cpu_core` fields), as a rule of thumb. These are separate busy-poll threads and should not share a core. First, identify your core IDs:
```bash
cat /proc/cpuinfo | grep processor
@@ -1151,7 +1151,7 @@ DAQIRI requires an [**NVIDIA SmartNIC**](https://www.nvidia.com/en-us/networking
1. Press `Join Now`.
2. Once approved, download the `Display Mode Selector` archive.
3. Unzip the archive.
- 4. Access your system without an X-server running. SSH into the machine, or switch to a Virtual Console (`Alt+F1`). You do not need to physically disconnect the monitor — the requirement is that no display server (X11/Wayland) is holding a lock on the NVIDIA driver.
+ 4. Access your system without an X-server running. SSH into the machine, or switch to a Virtual Console (`Alt+F1`). You do not need to physically disconnect the monitor. The requirement is that no display server (X11/Wayland) is holding a lock on the NVIDIA driver.
5. Go down the right OS and architecture folder for your system (`linux/aarch64` or `linux/x64`).
6. Run the `displaymodeselector` command like so:
@@ -1297,8 +1297,7 @@ DAQIRI requires an [**NVIDIA SmartNIC**](https://www.nvidia.com/en-us/networking
maxmtu 9978
```
- ---
- **Next:** [Benchmarking](../benchmarks/index.md) — choose and run your first DAQIRI benchmark
+ With the system tuned, continue to [Benchmarking](../benchmarks/index.md) to choose and run your first DAQIRI benchmark.
=== "DGX Spark"
@@ -1314,7 +1313,7 @@ DAQIRI requires an [**NVIDIA SmartNIC**](https://www.nvidia.com/en-us/networking
**Plug a cable into the chassis QSFP socket before debugging firmware, drivers, or BIOS.** The hotplug service brings the device back when a cable is detected.
- The hotplug behavior is implemented as a power/thermal management policy and coordinates with `nvidia-spark-mlnx-firmware-manager.service`. If you need the NIC alive without a cable for software-only testing, the override point is the scripts under `/opt/nvidia/dgx-spark-mlnx-hotplug` — read them before disabling.
+ The hotplug behavior is implemented as a power/thermal management policy and coordinates with `nvidia-spark-mlnx-firmware-manager.service`. If you need the NIC alive without a cable for software-only testing, the override point is the scripts under `/opt/nvidia/dgx-spark-mlnx-hotplug`. Read them before disabling.
### Port topology: 4 PFs, 2 ports, tied chassis sockets
@@ -1335,9 +1334,9 @@ DAQIRI requires an [**NVIDIA SmartNIC**](https://www.nvidia.com/en-us/networking
done
```
- !!! important "Single-machine loopback: same physical port = on-chip test; different ports = over-the-wire test"
+ !!! important "Single-machine loopback: same physical port = on-chip test, different ports = over-the-wire test"
- This distinction matters when you run a loopback benchmark within one DGX Spark, where TX and RX are two PFs on the same integrated CX-7. For two-device tests, pick ports according to the external cabling and the peer system's topology; the on-chip shortcut described here is specific to same-machine loopback.
+ This distinction matters when you run a loopback benchmark within one DGX Spark, where TX and RX are two PFs on the same integrated CX-7. For two-device tests, pick ports according to the external cabling and the peer system's topology. The on-chip shortcut described here is specific to same-machine loopback.
Which PF pair you choose decides **what you are actually measuring**. Because each physical port is exposed over both PCIe segments, two of the four BDFs map to the *same* physical port. Identify them on your own system with `phys_port_name`:
@@ -1359,18 +1358,18 @@ DAQIRI requires an [**NVIDIA SmartNIC**](https://www.nvidia.com/en-us/networking
Here two BDFs share each physical port (`mlx5_0` and `mlx5_2` are both **p0**). The two pairings measure different things:
- - **Same physical port** (e.g. `mlx5_0` ↔ `mlx5_2`, both p0) → TX/RX loop **on-chip** through the eswitch; traffic never reaches the cable. Physical-link packet counters stay flat while the vport counters (`tx_good_packets` / `rx_good_packets`) run at line rate. This is a software-path test.
+ - **Same physical port** (e.g. `mlx5_0` ↔ `mlx5_2`, both p0) → TX/RX loop **on-chip** through the eswitch. Traffic never reaches the cable. Physical-link packet counters stay flat while the vport counters (`tx_good_packets` / `rx_good_packets`) run at line rate. This is a software-path test.
- **Different physical ports** (e.g. `mlx5_0` p0 ↔ `mlx5_3` p1 `0002:01:00.1`, or `mlx5_0` ↔ `mlx5_1`) → TX/RX loop **over the wire**; physical-link packet counters rise to match the TX/RX counts. This is an over-the-wire test.
Confirm which case you got from the physical-link packet counters: near zero for on-chip, matching the TX/RX packet counts for over-the-wire. These counters count packets that reached the SerDes/QSFP side of the NIC rather than packets switched internally by the eswitch. The [daqiri bench](../benchmarks/raw_benchmarking.md)'s DPDK "Extended Stats" output reports them as `tx_phy_packets` / `rx_phy_packets`; `ethtool -S` and `mlnx_perf` report the same wire counters as `tx_packets_phy` / `rx_packets_phy`.
- `ethtool -m` reports identical `Connector: 0x23 No separable connector` on all 4 PFs and is **not** useful for distinguishing them; use `phys_port_name` above (the cable-yank carrier test confirms a cable is present but does **not** distinguish ports).
+ `ethtool -m` reports identical `Connector: 0x23 No separable connector` on all 4 PFs and is **not** useful for distinguishing them. Use `phys_port_name` above (the cable-yank carrier test confirms a cable is present but does **not** distinguish ports).
## System Setup for DAQIRI
### Check your NIC drivers
- Same as IGX — verify `ib_core` is loaded:
+ Same as IGX. Verify `ib_core` is loaded:
```bash
lsmod | grep ib_core
@@ -1392,7 +1391,7 @@ DAQIRI requires an [**NVIDIA SmartNIC**](https://www.nvidia.com/en-us/networking
### Configure the IP addresses of the NIC ports
- Spark uses NetworkManager. Create persistent `daqiri-tx` / `daqiri-rx` profiles that pin both the IP and the MTU (so [Step 9: Jumbo frames](#step-9-enable-jumbo-frames-already-covered) is folded in here). `daqiri-tx` is p0 (`enp1s0f0np0`) and `daqiri-rx` is p1 (`enP2p1s0f1np1`) — different physical ports, matching the over-the-wire benchmark example:
+ Spark uses NetworkManager. Create persistent `daqiri-tx` / `daqiri-rx` profiles that pin both the IP and the MTU (so [Step 9: Jumbo frames](#step-9-enable-jumbo-frames-already-covered) is folded in here). `daqiri-tx` is p0 (`enp1s0f0np0`) and `daqiri-rx` is p1 (`enP2p1s0f1np1`), different physical ports, matching the over-the-wire benchmark example:
```bash
sudo nmcli connection add type ethernet ifname enp1s0f0np0 con-name daqiri-tx \
@@ -1442,7 +1441,7 @@ DAQIRI requires an [**NVIDIA SmartNIC**](https://www.nvidia.com/en-us/networking
### Enable GPUDirect
- **No GPUDirect kernel-module setup is required on GB10.** Set `kind: "host_pinned"` in the YAML and you're done — there is no system-side step to perform. Buffers are allocated by DAQIRI via `cudaHostAlloc` (so they are CUDA-addressable) and registered with DPDK via `rte_extmem_register`. End-to-end TX↔RX over the QSFP loop with `kind: "host_pinned"`, `num_bufs: 51200`, `batch_size: 10240` reaches **~94 Gbps** unicast (verified against `main` 9ebd729, which contains [PR #41](https://github.com/nvidia/daqiri/pull/41)).
+ **No GPUDirect kernel-module setup is required on GB10.** Set `kind: "host_pinned"` in the YAML and you're done. There is no system-side step to perform. Buffers are allocated by DAQIRI via `cudaHostAlloc` (so they are CUDA-addressable) and registered with DPDK via `rte_extmem_register`. End-to-end TX↔RX over the QSFP loop with `kind: "host_pinned"`, `num_bufs: 51200`, `batch_size: 10240` reaches **~94 Gbps** unicast (verified against `main` 9ebd729, which contains [PR #41](https://github.com/nvidia/daqiri/pull/41)).
`kind: "huge"` works as a fallback at the same rate. `kind: "device"` does **not** work on GB10.
@@ -1470,25 +1469,25 @@ DAQIRI requires an [**NVIDIA SmartNIC**](https://www.nvidia.com/en-us/networking
| Step | Spark status | Notes |
|------|--------------|-------|
- | 1. PCIe topology | **N/A** | Single-SoC integrated GPU; no separable PCIe path GPU↔NIC |
+ | 1. PCIe topology | **N/A** | Single-SoC integrated GPU, no separable PCIe path GPU↔NIC |
| 2. PCIe MPS / Speed | unchanged | Same diagnostic commands; PCIe Gen5 native |
| 3. NIC MRRS | reshape | Use systemd unit + `setpci CAP_EXP+8.w` (capability-relative); **disable Secure Boot** |
| 4. Hugepages | reshape | Use a grub **drop-in** under `/etc/default/grub.d/`, not `/etc/default/grub` |
| 5. CPU isolation | reshape | Pin to big cores 16-19 (X925 cluster 1); folds into the same grub drop-in as Step 4 |
| 6. CPU governor | already set | Spark default is `performance` on all 20 cores |
| 7. GPU clocks | unchanged | Same systemd-unit recipe; GB10 max SM clock is 3003 MHz |
- | 8. GPU BAR1 size | **N/A** | Unified memory; no resizable BAR1 |
+ | 8. GPU BAR1 size | **N/A** | Unified memory, no resizable BAR1 |
| 9. Jumbo frames | folded into setup | MTU=9000 was set in the `daqiri-tx`/`daqiri-rx` nmcli profiles above |
- `tune_system.py --check all` on Spark suppresses the WARNs that are false positives on integrated GPUs (peermem, gpudirect, topology, BAR1) — see the source comments in [`python/tune_system.py`](https://github.com/nvidia/daqiri/blob/main/python/tune_system.py).
+ `tune_system.py --check all` on Spark suppresses the WARNs that are false positives on integrated GPUs (peermem, gpudirect, topology, BAR1). See the source comments in [`python/tune_system.py`](https://github.com/nvidia/daqiri/blob/main/python/tune_system.py).
- ### Step 1: PCIe topology — N/A on Spark
+ ### Step 1: PCIe topology (N/A on Spark)
`nvidia-smi topo -m` reports the integrated GPU as `SYS`-connected to the NIC PFs. This is structural, not tunable: there is no separable PCIe path GPU↔NIC on a single-SoC integrated GPU. `tune_system.py --check topo` recognizes integrated GPUs and reports INFO instead of WARNING for this case.
### Step 2: Check the NIC's PCIe configuration
- Same diagnostic commands as IGX — for each PF, query `MaxPayload` (DevCap vs DevCtl) and PCIe `Speed` (LnkCap vs LnkSta). Spark's CX-7 PFs negotiate PCIe Gen5 at the chip's native MPS; nothing to set.
+ Same diagnostic commands as IGX. For each PF, query `MaxPayload` (DevCap vs DevCtl) and PCIe `Speed` (LnkCap vs LnkSta). Spark's CX-7 PFs negotiate PCIe Gen5 at the chip's native MPS. Nothing to set.
```bash
for d in 0000:01:00.0 0000:01:00.1 0002:01:00.0 0002:01:00.1; do
@@ -1529,7 +1528,7 @@ DAQIRI requires an [**NVIDIA SmartNIC**](https://www.nvidia.com/en-us/networking
# Each line should print 5xxx (high nibble 5 = 4096-byte MRRS).
```
- ### Step 4: Enable Huge pages — grub drop-in pattern
+ ### Step 4: Enable Huge pages (grub drop-in pattern)
Spark composes its `GRUB_CMDLINE_LINUX` from drop-ins under `/etc/default/grub.d/`. Edit a new file rather than `/etc/default/grub` directly so Spark platform updates don't fight your changes. The shipped `daqiri_bench_raw_tx_rx_spark.yaml` needs ~4 GiB of hugepages (kind: HUGE dummy queues + DPDK per-pool overhead); 4 × 1 GiB pages is enough:
@@ -1547,7 +1546,7 @@ DAQIRI requires an [**NVIDIA SmartNIC**](https://www.nvidia.com/en-us/networking
sudo mkdir -p /mnt/huge
```
- Reboot once — Steps 4 and 5 land together.
+ Reboot once. Steps 4 and 5 land together.
```bash
sudo reboot
@@ -1576,7 +1575,7 @@ DAQIRI requires an [**NVIDIA SmartNIC**](https://www.nvidia.com/en-us/networking
cat /proc/cmdline | grep -oE "isolcpus=[^ ]+|nohz_full=[^ ]+|rcu_nocbs=[^ ]+|irqaffinity=[^ ]+"
```
- ### Step 6: CPU governor — already `performance` on Spark
+ ### Step 6: CPU governor (already `performance` on Spark)
Spark ships with `performance` on all 20 cores. Verify:
@@ -1595,13 +1594,13 @@ DAQIRI requires an [**NVIDIA SmartNIC**](https://www.nvidia.com/en-us/networking
sudo nvidia-smi -lgc=$(nvidia-smi --query-gpu=clocks.max.sm --format=csv,noheader,nounits)
```
- On a production GB10 the locked SM clock is 3003 MHz. If `nvidia-smi -pm 1` reports persistence mode is unsupported on this platform, the lock-clocks call still takes effect for the current driver session — fold it into a unit (omitting the `--lock-memory-clocks` line from the IGX recipe) and start it after reboot.
+ On a production GB10 the locked SM clock is 3003 MHz. If `nvidia-smi -pm 1` reports persistence mode is unsupported on this platform, the lock-clocks call still takes effect for the current driver session. Fold it into a unit (omitting the `--lock-memory-clocks` line from the IGX recipe) and start it after reboot.
- ### Step 8: GPU BAR1 size — N/A on Spark
+ ### Step 8: GPU BAR1 size (N/A on Spark)
- GB10 has unified CPU/GPU memory (NVLink-C2C coherent) — there is no resizable BAR1 to enlarge, so the entire IGX displaymodeselector / firmware-flash flow does not apply. `nvidia-smi -q | grep -A 3 BAR1` may print numbers but they are not actionable. `tune_system.py --check bar1-size` reports INFO instead of WARNING when an integrated GPU is detected.
+ GB10 has unified CPU/GPU memory (NVLink-C2C coherent). There is no resizable BAR1 to enlarge, so the entire IGX displaymodeselector / firmware-flash flow does not apply. `nvidia-smi -q | grep -A 3 BAR1` may print numbers but they are not actionable. `tune_system.py --check bar1-size` reports INFO instead of WARNING when an integrated GPU is detected.
- ### Step 9: Enable Jumbo frames — already covered
+ ### Step 9: Enable Jumbo frames (already covered)
The `daqiri-tx` / `daqiri-rx` nmcli profiles created in [Configure the IP addresses](#configure-the-ip-addresses-of-the-nic-ports_1) already pin `ethernet.mtu 9000`, so this step is a no-op on Spark. Verify:
@@ -1610,7 +1609,6 @@ DAQIRI requires an [**NVIDIA SmartNIC**](https://www.nvidia.com/en-us/networking
ip link show enP2p1s0f1np1 | grep -oE "mtu [0-9]+"
```
- ---
- **Next:** [Benchmarking](../benchmarks/index.md) — choose and run your first DAQIRI benchmark
+ With the system tuned, continue to [Benchmarking](../benchmarks/index.md) to choose and run your first DAQIRI benchmark.
diff --git a/mkdocs.yml b/mkdocs.yml
index 5982fcb..0ab4ecf 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -1,4 +1,4 @@
-site_name: DAQIRI — Data Acquisition for Integrated Real-time Instruments
+site_name: "DAQIRI: Data Acquisition for Integrated Real-time Instruments"
site_url: https://nvidia.github.io/daqiri/
repo_url: https://github.com/nvidia/daqiri
repo_name: nvidia/daqiri
@@ -9,20 +9,18 @@ theme:
logo: images/logo.svg
favicon: images/favicon-32.png
palette:
- - media: "(prefers-color-scheme: light)"
- scheme: default
+ - scheme: slate
primary: black
accent: light green
toggle:
- icon: material/brightness-7
- name: Switch to dark mode
- - media: "(prefers-color-scheme: dark)"
- scheme: slate
+ icon: material/brightness-4
+ name: Switch to light mode
+ - scheme: default
primary: black
accent: light green
toggle:
- icon: material/brightness-4
- name: Switch to light mode
+ icon: material/brightness-7
+ name: Switch to dark mode
font:
text: Inter
code: JetBrains Mono