Allocate injects ROCR_VISIBLE_DEVICES without HIP_VISIBLE_DEVICES, so Ray workloads cannot start

[kenji-mido]

What happens

Workloads whose image bundles Ray fail to start. Reproduced with
rocm/vllm:latest and the log included:

RuntimeError: Please use HIP_VISIBLE_DEVICES instead of ROCR_VISIBLE_DEVICES
  /usr/local/lib/python3.12/dist-packages/ray/_private/accelerators/amd_gpu.py:46

vLLM imports Ray inside create_engine_config, so this happens at import time.

Whether an image is affected depends on Ray being installed, not on vLLM as
such: vllm/vllm-openai-rocm:latest ships without Ray and serves inference normally
under the same allocation.

How to reproduce

Importing Ray is enough to reproduce it — no inference server involved.

Prerequisites: a single AMD GPU node. HAMi's chart carries the AMD vGPU
resources from 2.10.0 on with the following procedure.

helm repo add hami-charts https://project-hami.github.io/HAMi/
helm upgrade --install hami hami-charts/hami --version 2.10.0 -n kube-system --create-namespace

helm upgrade --install amd-gpu \
  https://github.com/Project-HAMi/amd-device-plugin/releases/download/amd-gpu-helm-0.0.1/amd-gpu-0.0.1.tgz \
  -n kube-system --create-namespace \
  --set dp.securityContext.allowPrivilegeEscalation=null   # workaround for #16

Without that last flag the DaemonSet is rejected at admission and none of the following is reachable (reported in #16).

kubectl run ray-probe --image=rocm/vllm:latest --restart=Never --overrides='
{"spec":{"schedulerName":"hami-scheduler","containers":[{"name":"c",
"image":"rocm/vllm:latest","command":["python3","-c","import ray"],
"resources":{"limits":{"amd.com/gpu":"1","amd.com/gpumem":"4096",
"amd.com/gpucores":"25"}}}]}}'

# rocm/vllm:latest is ~9 GB; the first pull dominates the wait
kubectl wait --for=jsonpath='{.status.phase}'=Failed pod/ray-probe --timeout=60m
kubectl logs ray-probe

What the plugin injected into that pod (an example):

LD_AUDIT=/usr/local/vgpu/libamvgpu.so
HIP_DEVICE_MEMORY_LIMIT=4096m
ROCR_VISIBLE_DEVICES=GPU-d61e81b565653231
HSA_CU_MASK=0:0-75
RAY_EXPERIMENTAL_NOSET_ROCR_VISIBLE_DEVICES=1

HIP_VISIBLE_DEVICES is not in that list as in the error message.

Cause

Allocate injects ROCR_VISIBLE_DEVICES (plugin.go#L739) but not HIP_VISIBLE_DEVICES. Ray raises whenever it sees exactly that combination:

if (HIP_VISIBLE_DEVICES_ENV_VAR not in os.environ
        and "ROCR_VISIBLE_DEVICES" in os.environ):
    raise RuntimeError(...)

This is deliberate on Ray's side. ray-project/ray#53737 reported the conflict between the two variables and PR #53757 (merged 2025-06-20) relaxed the case where both are set — while leaving ROCr-only as an error.
So Ray has already revisited this code path and chose to keep it.

Suggested fix

Inject HIP_VISIBLE_DEVICES alongside ROCR_VISIBLE_DEVICES, listing the same
devices as container-local indices ("0" for one GPU, "0,1" for two).

Removing ROCR_VISIBLE_DEVICES instead would be wrong. The GPU index in
HSA_CU_MASK is not a physical one. ROCr renumbers whatever devices
ROCR_VISIBLE_DEVICES leaves visible, starting at 0, and the plugin builds the
mask against that renumbering.
That is ROCr's rule, stated where the variable is
parsed: "GPU indexes are taken post ROCR_VISIBLE_DEVICES reordering"
(ROCR-Runtime flag.cpp).
The plugin already relies on it:

// ROCr renumbers devices after this list is applied. HSA_CU_MASK uses
// those container-local indices, so it is built in the same order.
car.Envs["ROCR_VISIBLE_DEVICES"] = strings.Join(rocrVisibleDevices, ",")

Quickly verified on the same allocation: with HIP_VISIBLE_DEVICES=0 set, import ray
raises nothing and torch.cuda.mem_get_info() still reports the requested amount
rather than the card's entire amount, so the memory limit is unaffected.
Happy to send a PR with that change and the before/after runs.

Caveat: this was validated on a single-GPU node. Whether
HIP_VISIBLE_DEVICES=0..N-1 lines up for multi-GPU allocations is untested
here.

Environment

• device plugin chart/image 0.0.1
• HAMi chart 2.10.0 (projecthami/hami:v2.10.0) ·
• Kubernetes v1.36.4+k3s1
• MI300X VF (304 CUs, 196288 MiB)
• ROCm 10.0.0,
• rocm/vllm:latest (ray 2.52.1)

---

AI assistance disclosure: I used Claude Code to (double) fact-check claims against the
upstream sources and to draft wording. The final text was authored, edited, and
reviewed by me, and all technical claims were verified against the codebase and against
runs on the hardware named above.