Skip to content

mori-path.md

Latest commit

 

History

History
53 lines (40 loc) · 2.06 KB

mori-path.md

File metadata and controls

53 lines (40 loc) · 2.06 KB
Raw

MORI dma-buf path on MI355X

This is a separate transport case study from the Dynamo NixlConnector PoC. It records the MORI issue because the same MI355X/ionic memory-registration constraints informed the NIXL integration.

Failure sequence

  1. Plain ibv_reg_mr of a 512 MB GPU KV range failed with EINVAL on ionic.
  2. Redirecting registration to ibv_reg_dmabuf_mr made registration succeed.
  3. RDMA writes then returned IBV_WC_LOC_PROT_ERR or targeted the wrong bytes.
  4. A loopback test showed that HIP export plus offset zero is wrong for an interior range of a pooled allocation.
  5. HSA portable export returned the actual dma-buf offset; using it made the loopback and full two-node transfer succeed.

Reproduce the diagnostic

Inside a ROCm development container with /dev/kfd, /dev/dri, and /dev/infiniband passed through:

bash examples/mori/run_loopback.sh

The relevant comparison is:

  • whole allocation + HIP export: expected DATA_OK;
  • sub-allocation + HIP offset zero: failure or DATA_MISMATCH;
  • sub-allocation + real HSA offset: expected DATA_OK.

The shared interposer implementation is in scripts/ibv_dmabuf_shim.cpp. MORI resolves verbs functions with dlvsym, so the shim intercepts dlvsym but deliberately does not override dlsym. An earlier attempt referenced glibc's private _dl_sym; that symbol was not exported in the target image and broke every preloaded process. The current bootstrap uses the real, non-interposed dlsym to locate dlvsym safely.

Validated result

The offset-aware prototype completed a two-node DeepSeek-V4-Pro SGLang P/D run:

  • prefill TP=8 and decode TP=8;
  • 16/16 requests at concurrency 1;
  • 1,952 successful dma-buf registrations;
  • zero registration, LOC_PROT, or KV-transfer errors;
  • median TTFT 0.381 seconds;
  • median ITL 24.6 ms;
  • median end-to-end latency 8.22 seconds.

The upstream-quality implementation and regression test are in ROCm/mori#509. Prefer the upstream MORI fix over this preload prototype once available in the deployed release.