fix: probe CUDA context on correct device in set_task_pid()

[maverick123123]

Problem

set_task_pid() in src/utils.c polls NVML processes on the first physical device where nvml_to_cuda_map(i) >= 0, but the probe context is always created on hardcoded CUDA device 0 via cuDevicePrimaryCtxRetain(&pctx, 0).

When CUDA_VISIBLE_DEVICES does not start with the pod's device 0 (e.g., CVD=1 or CVD=1,0), the polled NVML device and the probe context device are different physical GPUs. getextrapid() never finds the new PID, causing a spurious "host pid is error!" and subsequent fake Device 0 OOM due to incorrect memory accounting.

Fix

Save the cudaDev found in the NVML enumeration loop and use it for both cuDevicePrimaryCtxRetain and cuDevicePrimaryCtxRelease, ensuring the probe context runs on the same device being polled.

Also guard against the case where no NVML device maps to a CUDA device (probeDev remains -1) by returning early before retaining any context.

Testing

Verified on an 8x RTX 3090 machine with all CVD combinations:

CUDA_VISIBLE_DEVICES	Before fix	After fix
0	OK	OK
1	host pid is error	OK
0,1	OK	OK
1,0	host pid is error	OK

Related

Fixes #225

[hami-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: maverick123123
Once this PR has been reviewed and has the lgtm label, please assign archlitchi for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

Warning

Review limit reached

@maverick123123, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 10 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 284582d1-f21b-48e1-80db-d255aecceaca

📥 Commits

Reviewing files that changed from the base of the PR and between 52f33fc and ffcf6d7.

📒 Files selected for processing (1)

• src/utils.c

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[maverick123123]

@rootsongjc ready to merge

[mesutoezdil]

/lgtm

[iemAnshuman]

@maverick123123 ran the four CVD cases against #230 on 8×A100, driver 580.65.06, cold and with a warm holder. leak is gone, base leaves the probe context on GPU 1 for CVD=1 and 1,0, patched leaves nothing in all eight runs. but those same two still hit SET_TASK_PID FAILED, and the log says current processes num = 0 0 so nvml isnt seeing the probe on the device it polls at all, which I dont have an explanation for yet. tried CUDA_DEVICE_ORDER=PCI_BUS_ID in case CUDA and NVML disagreed on numbering, no change. you had all four passing on 3090 so maybe its this a100 or the 580 driver. not blocking, the leak fix stands on its own. logs and per case nvidia smi: https://gist.github.com/iemAnshuman/c3a0f5d8efc8871bc94816da1e7dcdd8

[iemAnshuman]

@maverick123123 followed this up with a small standalone test of just the set_task_pid() steps, on four A100. The 'current processes num = 0 0' I reported above does not happen there. 'cuDeviceGetUuid()' on the retained device and 'nvmlDeviceGetUUID()' on the polled handle give the same UUID on all nine layouts I tried, '0' '1' '0,1' '1,0' '2' '3' '2,3' and again with another process on the gpu, and the host pid turns up 0.062 to 0.071ms after the retain

So on that node, the device is right, and nvml publishes far too fast for the 330 s postinit budget to come into it.

I cant close the gap yet though. That node is driver 580.82.07 on A100-PCIE-40GB, where the failing runs were 580.65.06 on 8xA100, and the test runs the nvml steps on their own with no libvgpu.so. Three things differ her, which is why I can't say whether 'CVD=1' passing here is down to the test skipping libvgpu.so or to the different card and driver. Next thing I would try is the same probe under libvgpu.so on the box that failed.

logs and checksums: https://gist.github.com/iemAnshuman/e062b7297464fd07edefa9c350a72395

[maverick123123]

Thanks for the thorough testing across the A100 setups — really appreciate the detailed breakdown. Glad the leak fix holds across all cases. I'll keep an eye on the remaining SET_TASK_PID FAILED issue you flagged with CVD=1 / 1,0 on 580.65.06. Let me know if you find anything when testing with libvgpu.so on the failing node.