Skip to content

Add cuda_device_id placement knob for GPU decoders - #664

Open
kvmto wants to merge 3 commits into
NVIDIA:mainfrom
kvmto:decoder-gpu-pinning
Open

Add cuda_device_id placement knob for GPU decoders#664
kvmto wants to merge 3 commits into
NVIDIA:mainfrom
kvmto:decoder-gpu-pinning

Conversation

@kvmto

@kvmto kvmto commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

First PR of the #634 split (per @bmhowe23's review): cuda_device_id only — GPU decoder pinning, settable at construction via kwargs and the YAML realtime config. NUMA/mempolicy/cpu_affinity and thread-binding APIs are deferred to a follow-up draft PR (next release), Python interfaces for those to a third.

What it does

  • decoder::get() reads and validates cuda_device_id (negative or >= device count → std::runtime_error), persistently pins the constructing thread (cudaSetDevice, no restore), strips the key before the plugin constructor, and stores it (get_cuda_device_id(), -1 = unpinned).
  • Model: one thread owns one decoder. The thread that creates a decoder drives its decode calls; construction-time and lazy decode-time allocations all land on the pinned device with zero plugin changes — covers trt_decoder and the closed-source nv-qldpc-decoder transparently. This addresses the lazily-allocating-decoder concern from the Add hardware pinning for QEC decoders #634 review without per-call guards or new decode entry points.
  • decode_async() is the sole exception: its fresh std::async worker pins itself for the call's duration via a lib-private RAII CudaDeviceGuard (libs/qec/lib/hardware_guards.h, header-only; PR2 extends it).
  • YAML realtime config: cuda_device_id is a top-level decoder_config field (next to type/transport — placement knob common to any GPU decoder, not a per-decoder algorithm arg). Surfaced by prepare_decoder_params() for every decoder type; also exposed as decoder_config.cuda_device_id in Python.
  • Realtime session: the host dispatcher (one thread, all decoders) applies each decoder's device set-if-different (no restore) before enqueue and before DEVICE-mode graph capture; no-op for unpinned decoders.
  • Python kwargs work with no binding changes: qec.get_decoder("trt_decoder", H, cuda_device_id=1).

To place N decoders on N GPUs, create each decoder on its own thread (std::async / threading.Thread) — documented in realtime_decoding.rst.

Testing

  • 7 new C++ unit tests (DecoderCudaDeviceId.*): kwarg contract (absent/negative/out-of-range), key stripped before strict-validating plugin ctors, persistent pin observable after construction, two decoders on two GPUs from two threads, async worker pinning itself (verified RED→GREEN on a 2-GPU machine). GPU tests skip below the needed device count.
  • YAML: round-trip with the new field; prepare_decoder_params surfaces the knob for trt and non-trt types (ordering vs the trt-only early return is locked by test).
  • Python: kwargs error path (cuda_device_id named in the error) and valid-id construct+decode. Full test_decoder.py 61/61.
  • Full suites: test_decoders 52/52, test_decoders_yaml 20 pass / 2 skips (nv-qldpc-gated).

Notes for reviewers

  • Known pre-existing quirk (not introduced here): negative int kwargs from Python surface as an opaque bad_cast because hetMapFromKwargs stores Python ints as size_t (libs/core kwargs_utils.h) — the friendly negative-id message is only reachable from C++/YAML. Follow-up candidate in core.
  • nv-qldpc-decoder internal changes (mirroring the trt pattern) are a separate follow-up.

Supersedes #663 (same content, reopened from the fork).

🤖 Generated with Claude Code

kvmto and others added 2 commits July 9, 2026 17:31
Decoders can now be pinned to a specific CUDA device at construction
via a cuda_device_id parameter, settable through C++/Python kwargs
and the YAML realtime config (top-level decoder field, next to type/
transport).

Model: one thread owns one decoder. decoder::get() validates the id
(negative or >= device count throws), persistently pins the
constructing thread with cudaSetDevice (no restore), strips the key
before the plugin constructor, and stores it (get_cuda_device_id()).
Plugin constructors therefore allocate on the right device with zero
plugin changes -- this covers trt_decoder and the closed-source
nv-qldpc-decoder transparently.

decode_async() is the one exception: its fresh std::async worker pins
itself for the call's duration via a lib-private RAII CudaDeviceGuard
(libs/qec/lib/hardware_guards.h).

The realtime host dispatcher (one thread serving all decoders) applies
each decoder's device with set-if-different before enqueue and before
DEVICE-mode graph capture; no-op for unpinned decoders.

Tests: 7 C++ unit tests incl. 2-GPU placement and async-worker pinning,
YAML round-trip + prepare_decoder_params coverage, Python kwargs tests.

NUMA/mempolicy/cpu_affinity and thread-binding APIs are deferred to a
follow-up PR per review feedback on NVIDIA#634.

Signed-off-by: kvmto <kmato@nvidia.com>
@kvmto
kvmto marked this pull request as ready for review July 13, 2026 11:24
Signed-off-by: kvmto <kmato@nvidia.com>
decoders:
- id: 0
type: pymatching
cuda_device_id: 0 # optional: pin this decoder to a CUDA device

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on what I'm reading, the docs are saying to do this, but libs/qec/unittests/test_decoders_yaml.cpp flags this as invalid configuration. Am I missing something?

melody-ren added a commit that referenced this pull request Jul 13, 2026
Carrying the content of #664 , and its commit
7fe9529 in which the `cuda_device_id`
knob was introduced. This PR is meant to revert 664's scope change
introduced by 19b95bf.

This PR should be followed by #678 to enable support for the decoding
server.

PR description copied verbatim from #664 :
## Summary

First PR of the #634 split (per @bmhowe23's review): `cuda_device_id`
only — GPU decoder pinning, settable at construction via kwargs and the
YAML realtime config. NUMA/mempolicy/cpu_affinity and thread-binding
APIs are deferred to a follow-up draft PR (next release), Python
interfaces for those to a third.

## What it does

- **`decoder::get()`** reads and validates `cuda_device_id` (negative or
>= device count → `std::runtime_error`), **persistently pins the
constructing thread** (`cudaSetDevice`, no restore), strips the key
before the plugin constructor, and stores it (`get_cuda_device_id()`, -1
= unpinned).
- **Model: one thread owns one decoder.** The thread that creates a
decoder drives its decode calls; construction-time and lazy decode-time
allocations all land on the pinned device with **zero plugin changes** —
covers `trt_decoder` and the closed-source `nv-qldpc-decoder`
transparently. This addresses the lazily-allocating-decoder concern from
the #634 review without per-call guards or new decode entry points.
- **`decode_async()`** is the sole exception: its fresh `std::async`
worker pins itself for the call's duration via a lib-private RAII
`CudaDeviceGuard` (`libs/qec/lib/hardware_guards.h`, header-only; PR2
extends it).
- **YAML realtime config:** `cuda_device_id` is a top-level
`decoder_config` field (next to `type`/`transport` — placement knob
common to any GPU decoder, not a per-decoder algorithm arg). Surfaced by
`prepare_decoder_params()` for every decoder type; also exposed as
`decoder_config.cuda_device_id` in Python.
- **Realtime session:** the host dispatcher (one thread, all decoders)
applies each decoder's device set-if-different (no restore) before
enqueue and before DEVICE-mode graph capture; no-op for unpinned
decoders.
- **Python kwargs** work with no binding changes:
`qec.get_decoder("trt_decoder", H, cuda_device_id=1)`.

To place N decoders on N GPUs, create each decoder on its own thread
(`std::async` / `threading.Thread`) — documented in
`realtime_decoding.rst`.

## Testing

- 7 new C++ unit tests (`DecoderCudaDeviceId.*`): kwarg contract
(absent/negative/out-of-range), key stripped before strict-validating
plugin ctors, persistent pin observable after construction, **two
decoders on two GPUs from two threads**, async worker pinning itself
(verified RED→GREEN on a 2-GPU machine). GPU tests skip below the needed
device count.
- YAML: round-trip with the new field; `prepare_decoder_params` surfaces
the knob for trt and non-trt types (ordering vs the trt-only early
return is locked by test).
- Python: kwargs error path (`cuda_device_id` named in the error) and
valid-id construct+decode. Full `test_decoder.py` 61/61.
- Full suites: `test_decoders` 52/52, `test_decoders_yaml` 20 pass / 2
skips (nv-qldpc-gated).

## Notes for reviewers

- Known pre-existing quirk (not introduced here): negative int kwargs
from Python surface as an opaque `bad_cast` because `hetMapFromKwargs`
stores Python ints as `size_t` (`libs/core` `kwargs_utils.h`) — the
friendly negative-id message is only reachable from C++/YAML. Follow-up
candidate in core.
- `nv-qldpc-decoder` internal changes (mirroring the trt pattern) are a
separate follow-up.

Supersedes #663 (same content, reopened from the fork).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: kvmto <kmato@nvidia.com>
Signed-off-by: Melody Ren <melodyr@nvidia.com>
Co-authored-by: kvmto <kmato@nvidia.com>
melody-ren added a commit that referenced this pull request Jul 13, 2026
Summary:

- Worker threads in the decoding server now pin themselves to the
decoder's cuda_device_id before serving. Construction only pinned the
registry thread, so every worker was decoding on the default device
regardless of the pin prior to this patch.
- The direct call path (no realtime session) had the same problem:
configure_decoders leaves the thread on the last decoder's device, so
decoder 0 would decode on decoder N's GPU. Each decode now selects its
own decoder's device first.
- If a pin can't be honored we fail instead of decoding on the wrong
device: a worker that can't pin aborts server startup, and a
cudaSetDevice failure during dispatch returns an error response instead
of logging a warning and carrying on.
- Hololink: HOLOLINK_GPU_ID and cuda_device_id both name the GPU the
FPGA is attached to, so they must agree. Either one alone selects the
device; setting both to different values throws at transport creation.
Single decoder only, same as the rest of the gpu_roce path.
- Nothing changes if cuda_device_id is not set.
- Decoder construction is transactional: if a plugin constructor throws
after its device was selected, the calling thread's CUDA device is
restored to its previous value instead of being left on the failed
decoder's device. Originally first introduced in #687. Adopting the
change here.

Outdated: This is a follow-up PR to #664, which should be merged before
this PR

This PR depends on #690

---------

Signed-off-by: kvmto <kmato@nvidia.com>
Signed-off-by: Melody Ren <melodyr@nvidia.com>
Co-authored-by: kvmto <kmato@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants