#176 - Add ibverbs hardware packet pacing

AGENTS.md

@@ -35,7 +35,7 @@ There is no unit test suite. Verification is done via the benchmark executables
 
 | Executable | Source | Typical config |
 |---|---|---|
-| `daqiri_bench_raw_gpudirect` | `raw_gpudirect_bench.cpp` | `daqiri_bench_raw_tx_rx.yaml`, `daqiri_bench_raw_tx_rx_4q.yaml`, `daqiri_bench_raw_tx_rx_spark.yaml`, `daqiri_bench_raw_{tx,rx}_spark_xhost.yaml`, `daqiri_bench_raw_sw_loopback.yaml`, `daqiri_bench_raw_rx_multi_q.yaml`, `daqiri_bench_raw_tx_rx_vxlan.yaml`, `daqiri_bench_raw_tx_rx_vlan.yaml`, `daqiri_bench_raw_tx_rx_gre.yaml`, `daqiri_bench_raw_tx_rx_nvgre.yaml`, `daqiri_bench_raw_tx_rx_spark_mq.yaml` (mq base; `run_spark_mq_bench.sh` derives the 4 cells via `scripts/gen_spark_mq_config.py`), `daqiri_bench_raw_tx_rx_pacing.yaml` (per-queue `pacing_mbps`; DPDK engine only) |
+| `daqiri_bench_raw_gpudirect` | `raw_gpudirect_bench.cpp` | `daqiri_bench_raw_tx_rx.yaml`, `daqiri_bench_raw_tx_rx_4q.yaml`, `daqiri_bench_raw_tx_rx_spark.yaml`, `daqiri_bench_raw_{tx,rx}_spark_xhost.yaml`, `daqiri_bench_raw_sw_loopback.yaml`, `daqiri_bench_raw_rx_multi_q.yaml`, `daqiri_bench_raw_tx_rx_vxlan.yaml`, `daqiri_bench_raw_tx_rx_vlan.yaml`, `daqiri_bench_raw_tx_rx_gre.yaml`, `daqiri_bench_raw_tx_rx_nvgre.yaml`, `daqiri_bench_raw_tx_rx_spark_mq.yaml` (mq base; `run_spark_mq_bench.sh` derives the 4 cells via `scripts/gen_spark_mq_config.py`), `daqiri_bench_raw_tx_rx_pacing.yaml` (per-queue `pacing_mbps`; DPDK or ibverbs raw engine) |
 | `daqiri_example_dynamic_rx_flow` | `dynamic_rx_flow_example.cpp` | `daqiri_example_dynamic_rx_flow.yaml` — `flow_isolation: true` startup followed by runtime RX queue-steering and raw-engine decap/pop flow add/delete |
 | `daqiri_bench_raw_hds` | `raw_hds_bench.cpp` | `daqiri_bench_raw_tx_rx_hds.yaml` |
 | `daqiri_bench_raw_reorder_seq` | `raw_reorder_seq_bench.cpp` | `daqiri_bench_raw_tx_rx_reorder_seq_1024*.yaml`, `daqiri_bench_raw_rx_reorder_seq_*.yaml` |
@@ -81,7 +81,7 @@ The always-built socket engine implements Linux UDP/TCP streams directly. Applic
 
 The default `dpdk` raw engine (`src/engines/dpdk/`, `DpdkEngine`) programs RX steering, send-to-kernel fallbacks (`flow_isolation: true`), and `tx_eth_src` TX offloads via DPDK RTE Flow during `daqiri_init()`. Standard UDP/IP (group 3), flex-item (group 1) and eCPRI-over-Ethernet (group 2, EtherType `0xAEFE`, via `RTE_FLOW_ITEM_TYPE_ECPRI` matching message type and pc_id/rtc_id) RX flows use separate flow groups; `validate_config()` rejects mixing these flow classes per interface, duplicate `flex_item_id` values, and unknown `action.id` / `flex_item_id` values before NIC programming. The mlx5 PMD honors the native eCPRI flow item only under **firmware steering** (`dv_flow_en=1`); under HW steering (`dv_flow_en=2`, the default) the rule installs but silently never matches on ConnectX-class NICs. So `initialize()` auto-switches any interface carrying eCPRI RX flows to `dv_flow_en=1` (with a `WARN`), which means the async/template dynamic-RX-flow path is unavailable on that port. Flex-item parser handles are created per `(port, flex_item_id)` (scoped per interface). All programmed `rte_flow` rules and flex-item handles are tracked and destroyed in order on shutdown, init failure, and engine teardown (programmed flows → flex items → group-0 ETH jump rules). See `docs/benchmarks/raw_benchmarking.md` (Flow programming smoke test) for manual verification steps.
 
-The `ibverbs` raw engine (`src/engines/ibverbs/`, `IbverbsEngine`) drives a Mellanox/mlx5 Multi-Packet (striding) Receive Queue via **DevX** (`mlx5dv_devx_obj_create` against vendored PRM structs in `mlx5_prm_min.h`): a DevX CQ + striding RQ + TIR + `mlx5dv_dr` flow steering, with manual WQE/doorbell management and worker-driven cyclic refill. RX packets DMA strided into one pre-posted MR (host or GPU via `ibv_reg_dmabuf_mr`); a queue with >1 memory region instead uses a non-striding DevX *regular* RQ with multi-segment scatter WQEs for **physical** header-data split (header → CPU MR, payload → GPU MR). TX builds mlx5 send WQEs directly on a raw-packet QP's SQ (via `mlx5dv_init_obj`, bypassing `ibv_post_send`) from a slab of registered slots tracked by cyclic index counters, with NIC checksum offload and a `tx_eth_src` offload. It uses the libdpdk-free `daqiri::Ring`/`daqiri::ObjectPool` for the worker→app burst handoff (like the rdma engine — neither links DPDK) and drives the NIC through libibverbs/mlx5dv directly. Feature set: RX (MPRQ), TX, GPUDirect, physical/logical HDS, multi-queue 5-tuple flow steering with per-packet flow IDs, flex-item arbitrary-offset, IPv4-total-length and eCPRI-over-Ethernet (EtherType `0xAEFE`, message type + pc_id/rtc_id) flow matching (mlx5 flex parser / `misc_parameters_4`), per-packet RX hardware timestamps, accurate TX send scheduling (wait-on-time WAIT WQE), and GPU reorder/quantize. Because it uses the kernel netdev directly, `ensure_port_mtus` raises the netdev MTU at init to cover the configured frame size in either direction — RX (post-decap) and TX egress (post-encap) — sizing each direction with its own transform wire overhead (jumbo frames silently drop otherwise). Queues sharing a `cpu_core` are serviced round-robin by one poller thread.
+The `ibverbs` raw engine (`src/engines/ibverbs/`, `IbverbsEngine`) drives a Mellanox/mlx5 Multi-Packet (striding) Receive Queue via **DevX** (`mlx5dv_devx_obj_create` against vendored PRM structs in `mlx5_prm_min.h`): a DevX CQ + striding RQ + TIR + `mlx5dv_dr` flow steering, with manual WQE/doorbell management and worker-driven cyclic refill. RX packets DMA strided into one pre-posted MR (host or GPU via `ibv_reg_dmabuf_mr`); a queue with >1 memory region instead uses a non-striding DevX *regular* RQ with multi-segment scatter WQEs for **physical** header-data split (header → CPU MR, payload → GPU MR). TX builds mlx5 send WQEs directly on a raw-packet QP's SQ (via `mlx5dv_init_obj`, bypassing `ibv_post_send`) from a slab of registered slots tracked by cyclic index counters, with NIC checksum offload and a `tx_eth_src` offload. It uses the libdpdk-free `daqiri::Ring`/`daqiri::ObjectPool` for the worker→app burst handoff (like the rdma engine — neither links DPDK) and drives the NIC through libibverbs/mlx5dv directly. Feature set: RX (MPRQ), TX, GPUDirect, physical/logical HDS, multi-queue 5-tuple flow steering with per-packet flow IDs, flex-item arbitrary-offset, IPv4-total-length and eCPRI-over-Ethernet (EtherType `0xAEFE`, message type + pc_id/rtc_id) flow matching (mlx5 flex parser / `misc_parameters_4`), per-packet RX hardware timestamps, per-queue hardware packet pacing through the mlx5 QP rate table, accurate TX send scheduling (wait-on-time WAIT WQE), and GPU reorder/quantize. Packet pacing and accurate send are independent: `pacing_mbps` configures the QP's average rate, while `set_packet_tx_time()` emits WAIT WQEs for absolute per-packet times. Because it uses the kernel netdev directly, `ensure_port_mtus` raises the netdev MTU at init to cover the configured frame size in either direction — RX (post-decap) and TX egress (post-encap) — sizing each direction with its own transform wire overhead (jumbo frames silently drop otherwise). Queues sharing a `cpu_core` are serviced round-robin by one poller thread.
 
 ### Zero-copy / BurstParams
 All packet data flows through `BurstParams`, a batch of packets. Only pointers are passed between NIC, DAQIRI internals, and the application — the caller reads directly from the buffers the NIC DMA'd into. **The caller must explicitly free bursts**; a missed free drains the pool and produces `NO_FREE_BURST_BUFFERS` / `NO_FREE_PACKET_BUFFERS` errors and NIC drops. See `docs/concepts.md` (Zero-Copy Ownership) and `docs/api-reference/cpp.md` (free-function call patterns).

docs/api-reference/configuration.md

@@ -400,12 +400,14 @@ daqiri::set_reorder_cuda_stream("rx_port", "rx_reorder_0", stream);
   - values: `tx_eth_src` (auto-fill source MAC address)
 - **`pacing_mbps`**: Packet-pacing rate cap for this queue, in megabits per second of L2 frame
   bytes (the data the application transmits, excluding preamble/IFG/FCS). The NIC meters the queue
-  out so its long-run average TX rate stays at or below this value. The limit is enforced on an
-  average basis and idle gaps do not accumulate burst credit. `0` (the default) disables pacing
-  and sends at line rate. Supported only by the default `dpdk` raw engine on a NIC with hardware
-  send scheduling (ConnectX-7 or later); on devices without it, `pacing_mbps` is ignored with a
-  warning and TX runs at line rate. The `ibverbs` raw engine does **not** support `pacing_mbps`
-  and `daqiri_init()` fails if it is set on an `ibverbs` queue.
+  out so its long-run average TX rate stays at or below this value. `0` (the default) disables
+  pacing and sends at line rate. Both raw engines support this setting, with different capability
+  requirements: `dpdk` uses hardware send scheduling and falls back to line rate with a warning
+  when that offload is unavailable; `ibverbs` assigns the QP to an mlx5 hardware packet-pacing
+  rate-table entry and fails initialization if RAW_PACKET pacing is unavailable or the requested
+  rate is outside a range advertised by the device. Older drivers that omit the range defer bounds
+  checking to the provider when the rate is applied. The ibverbs engine leaves the optional burst
+  bound and typical-packet-size fields at their device defaults.
   - type: `integer`
   - default: `0`
 

docs/benchmarks/raw_benchmarking.md

@@ -260,10 +260,11 @@ flow programming test.
 To meter the transmit side at a fixed rate in hardware, set a per-queue `pacing_mbps` cap
 on the TX queue. [`daqiri_bench_raw_tx_rx_pacing.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_tx_rx_pacing.yaml)
 is the loopback config above with `pacing_mbps: 10000` (10 Gbps) on the TX queue. Pacing is
-supported only on the default DPDK engine. The `ibverbs` engine does not support `pacing_mbps`
-(init fails if it is set on an `ibverbs` queue). The NIC meters the queue out so its average
-TX rate stays at or below the configured value. The limit is enforced on an average basis and
-idle gaps do not accumulate burst credit.
+supported by both raw engines. The example uses the default DPDK engine; add
+`engine: "ibverbs"` beside `stream_type: "raw"` to exercise the ibverbs path. The NIC meters the
+queue out so its average TX rate stays at or below the configured value. The ibverbs path uses
+the mlx5 packet-pacing rate table rather than per-packet WAIT WQEs; firmware defaults determine
+the allowed burst size unless configured outside DAQIRI.
 
 ```bash
 /opt/daqiri/bin/daqiri_bench_raw_gpudirect /opt/daqiri/bin/daqiri_bench_raw_tx_rx_pacing.yaml --seconds 10
@@ -274,8 +275,11 @@ Validate the cap from the `RX complete:` line: `Gbps = bytes * 8 / seconds / 1e9
 link speed. Change `pacing_mbps` (or set it to `0` to disable pacing and send at line rate)
 and re-run to see the cap move.
 
-Pacing requires a Mellanox/mlx5 NIC with hardware send scheduling (ConnectX-7 or later). On
-devices without it, `pacing_mbps` is ignored with a warning at init and TX runs at line rate.
+The DPDK path requires hardware send scheduling; if unavailable, it logs a warning and runs at
+line rate. The ibverbs path requires packet-pacing support for RAW_PACKET QPs and, when the driver
+reports a supported range, a rate within that range. Older drivers that omit the range defer the
+check to the provider when applying the rate; unsupported requests still fail initialization
+instead of silently ignoring the cap.
 
 ## Tune RDMA SEND completion signaling
 

docs/tutorials/configuration-walkthrough.md

@@ -135,9 +135,9 @@ For a shorter selection guide, start with the [Benchmarking overview](../benchma
 ??? question "6. I need to cap (pace) the transmit rate in hardware"
     - [`daqiri_bench_raw_tx_rx_pacing.yaml`](https://github.com/nvidia/daqiri/blob/main/examples/daqiri_bench_raw_tx_rx_pacing.yaml) (runs on `daqiri_bench_raw_gpudirect`).
 
-    This is the base TX+RX template with a per-queue `pacing_mbps` cap added to the TX queue (it uses the default DPDK engine. Pacing is supported only on the DPDK engine. The `ibverbs` engine does not support `pacing_mbps`, and `daqiri_init()` fails if it is set on an `ibverbs` queue). The NIC meters the queue out so its average TX rate stays at or below the configured Mbps. The limit is enforced on an average basis and idle gaps do not accumulate burst credit. Set `pacing_mbps: 0` (or remove it) to send at line rate. Validate by computing the achieved rate from the benchmark's RX line (`Gbps = bytes * 8 / seconds / 1e9`) and confirming it tracks the configured cap. See the `pacing_mbps` key in the [TX queue configuration](../api-reference/configuration.md#transmit-configuration-tx).
+    This is the base TX+RX template with a per-queue `pacing_mbps` cap added to the TX queue. It uses the default DPDK engine; add `engine: "ibverbs"` beside `stream_type: "raw"` to use the mlx5 QP packet-pacing rate table instead. The NIC meters the queue out so its average TX rate stays at or below the configured Mbps. Set `pacing_mbps: 0` (or remove it) to send at line rate. Validate by computing the achieved rate from the benchmark's RX line (`Gbps = bytes * 8 / seconds / 1e9`) and confirming it tracks the configured cap. See the `pacing_mbps` key in the [TX queue configuration](../api-reference/configuration.md#transmit-configuration-tx).
 
-    *Requires: a Mellanox/mlx5 NIC with hardware send scheduling (ConnectX-7 or later). On devices without it, `pacing_mbps` is ignored with a warning and TX runs at line rate.*
+    *Requires the engine's pacing capability: DPDK warns and runs at line rate without its send-scheduling offload; ibverbs fails initialization unless the device advertises RAW_PACKET packet pacing and accepts the requested rate. If an older driver omits its supported range, the provider validates the rate when it is applied.*
 
 ## Annotated walkthrough
 

examples/daqiri_bench_raw_tx_rx_pacing.yaml

@@ -5,14 +5,14 @@
 # This is daqiri_bench_raw_tx_rx.yaml with a per-queue pacing_mbps cap added to
 # the TX queue (no "engine" key, so it uses the default raw engine, DPDK). The
 # NIC meters the queue out so its average TX rate stays at or below the
-# configured value; the limit is enforced on an average basis and idle gaps do
-# not accumulate burst credit. Pacing is supported only on the default DPDK raw
-# engine; the ibverbs engine does not support pacing_mbps (daqiri_init fails if
-# it is set on an ibverbs queue).
+# configured value. To exercise the ibverbs path, add `engine: "ibverbs"` beside
+# `stream_type: "raw"`; it programs the mlx5 hardware packet-pacing rate table
+# for the TX QP and uses the device defaults for burst sizing.
 #
-# Requirements: a Mellanox/mlx5 NIC with hardware send scheduling (ConnectX-7 or
-# later). On devices without it, pacing_mbps is ignored with a warning at init
-# and TX runs at line rate.
+# Requirements: the DPDK path needs its hardware send-scheduling offload and
+# warns/runs at line rate without it. The ibverbs path requires RAW_PACKET packet
+# pacing and an accepted rate; older drivers that omit their supported range
+# defer validation to the provider when the rate is applied.
 #
 # Validate: run daqiri_bench_raw_gpudirect with this config and compute the
 # achieved rate from the RX line's bytes/seconds (Gbps = bytes * 8 / seconds /

include/daqiri/types.h

@@ -631,8 +631,8 @@ struct RxQueueConfig {
 struct TxQueueConfig {
   CommonQueueConfig common_;
   // Packet pacing: average TX rate cap in megabits/sec (L2 frame bytes). 0
-  // disables pacing (line-rate). Honored only by engines/devices with accurate
-  // send scheduling (wait-on-time + real-time clock).
+  // disables pacing (line-rate). Honored only by engines/devices with hardware
+  // packet-pacing support.
   uint64_t pacing_mbps_ = 0;
 };