Skip to content

#176 - Add ibverbs hardware packet pacing#232

Merged
dleshchev merged 2 commits into
mainfrom
cburdick/ibverbs-packet-pacing
Jul 17, 2026
Merged

#176 - Add ibverbs hardware packet pacing#232
dleshchev merged 2 commits into
mainfrom
cburdick/ibverbs-packet-pacing

Conversation

@cliffburdick

Copy link
Copy Markdown
Collaborator

The ibverbs backend previously lacked packet pacing since the WAIT ON TIME method wasn't very reliable. This updated PR adds hardware pacing, which should be more accurate than the WAIT ON TIME method.

Signed-off-by: Cliff Burdick <cburdick@nvidia.com>
@greptile-apps

greptile-apps Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds hardware packet pacing to the ibverbs raw engine by calling ibv_modify_qp_rate_limit on the TX QP after it reaches RTS state, replacing the previous hard rejection of pacing_mbps on ibverbs queues. A follow-up commit in the same PR also handles older MLNX_OFED/rdma-core drivers that advertise RAW_PACKET pacing capability but leave the rate-range fields at zero, deferring bounds enforcement to the provider in that case.

  • configure_tx_pacing() is a self-contained new function: it overflow-guards the kbps conversion, checks the IBV_QPT_RAW_PACKET capability bit, conditionally validates against the advertised [min, max] range only when qp_rate_limit_max != 0, and applies the rate via ibv_modify_qp_rate_limit; max_burst_sz and typical_pkt_sz are left at zero for device defaults.
  • Failure cleanup in setup_tx_queue destroys the QP before the CQ (correct ibverbs teardown order) and returns the error; the broader engine teardown handles the remaining resources, consistent with every other failure path in that function.
  • All required doc-sync targets are updated in the same PR: AGENTS.md, docs/api-reference/configuration.md, docs/benchmarks/raw_benchmarking.md, docs/tutorials/configuration-walkthrough.md, and the example YAML comment block.

Confidence Score: 5/5

Safe to merge — the change is additive (removes a hard rejection, adds a well-guarded call path) and does not touch any shared or RX-side logic.

Both commits are DCO-signed and follow the #176 - … commit format. The new configure_tx_pacing function correctly sequences device-capability query → range validation (with the older-driver zero-max guard) → rate application, and the teardown on failure respects ibverbs object-destruction ordering (QP before CQ). Doc sync is thorough across AGENTS.md and all three relevant docs targets. No pre-existing patterns are broken and no new resource-management risks are introduced.

No files require special attention.

Important Files Changed

Filename Overview
src/engines/ibverbs/daqiri_ibverbs_engine.cpp Adds configure_tx_pacing() using ibv_modify_qp_rate_limit; removes the old hard-rejection of pacing_mbps on ibverbs; cleanup on pacing failure correctly destroys QP then CQ in the right order; logic is sound.
src/engines/ibverbs/daqiri_ibverbs_engine.h Adds configure_tx_pacing declaration to the TX-path section; declaration is consistent with the implementation signature.
include/daqiri/types.h Comment on pacing_mbps_ updated from 'wait-on-time + real-time clock' to 'hardware packet-pacing support'; no field or type changes.
AGENTS.md Benchmark table updated to reflect ibverbs pacing support; ibverbs engine description updated to describe QP rate-table pacing alongside WAIT WQE scheduling.
docs/api-reference/configuration.md pacing_mbps documentation rewritten to cover both raw engines, their capability requirements, and the older-driver deferral behavior.
docs/benchmarks/raw_benchmarking.md Pacing section updated to document ibverbs path using mlx5 rate table; DPDK-only restriction removed.
docs/tutorials/configuration-walkthrough.md FAQ entry for rate capping updated to document ibverbs path and its capability requirements.
examples/daqiri_bench_raw_tx_rx_pacing.yaml Header comment updated to document ibverbs pacing option and requirements; no YAML keys changed.

Reviews (2): Last reviewed commit: "#176 - Handle missing ibverbs pacing bou..." | Re-trigger Greptile

Comment thread src/engines/ibverbs/daqiri_ibverbs_engine.cpp Outdated
Signed-off-by: Cliff Burdick <cburdick@nvidia.com>
@dleshchev
dleshchev merged commit e3e97ae into main Jul 17, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants