Add cuda_device_id placement knob for GPU decoders - #664
Open
kvmto wants to merge 3 commits into
Open
Conversation
Decoders can now be pinned to a specific CUDA device at construction via a cuda_device_id parameter, settable through C++/Python kwargs and the YAML realtime config (top-level decoder field, next to type/ transport). Model: one thread owns one decoder. decoder::get() validates the id (negative or >= device count throws), persistently pins the constructing thread with cudaSetDevice (no restore), strips the key before the plugin constructor, and stores it (get_cuda_device_id()). Plugin constructors therefore allocate on the right device with zero plugin changes -- this covers trt_decoder and the closed-source nv-qldpc-decoder transparently. decode_async() is the one exception: its fresh std::async worker pins itself for the call's duration via a lib-private RAII CudaDeviceGuard (libs/qec/lib/hardware_guards.h). The realtime host dispatcher (one thread serving all decoders) applies each decoder's device with set-if-different before enqueue and before DEVICE-mode graph capture; no-op for unpinned decoders. Tests: 7 C++ unit tests incl. 2-GPU placement and async-worker pinning, YAML round-trip + prepare_decoder_params coverage, Python kwargs tests. NUMA/mempolicy/cpu_affinity and thread-binding APIs are deferred to a follow-up PR per review feedback on NVIDIA#634. Signed-off-by: kvmto <kmato@nvidia.com>
Signed-off-by: kvmto <kmato@nvidia.com>
18 tasks
bmhowe23
reviewed
Jul 13, 2026
| decoders: | ||
| - id: 0 | ||
| type: pymatching | ||
| cuda_device_id: 0 # optional: pin this decoder to a CUDA device |
Collaborator
There was a problem hiding this comment.
Based on what I'm reading, the docs are saying to do this, but libs/qec/unittests/test_decoders_yaml.cpp flags this as invalid configuration. Am I missing something?
melody-ren
added a commit
that referenced
this pull request
Jul 13, 2026
Carrying the content of #664 , and its commit 7fe9529 in which the `cuda_device_id` knob was introduced. This PR is meant to revert 664's scope change introduced by 19b95bf. This PR should be followed by #678 to enable support for the decoding server. PR description copied verbatim from #664 : ## Summary First PR of the #634 split (per @bmhowe23's review): `cuda_device_id` only — GPU decoder pinning, settable at construction via kwargs and the YAML realtime config. NUMA/mempolicy/cpu_affinity and thread-binding APIs are deferred to a follow-up draft PR (next release), Python interfaces for those to a third. ## What it does - **`decoder::get()`** reads and validates `cuda_device_id` (negative or >= device count → `std::runtime_error`), **persistently pins the constructing thread** (`cudaSetDevice`, no restore), strips the key before the plugin constructor, and stores it (`get_cuda_device_id()`, -1 = unpinned). - **Model: one thread owns one decoder.** The thread that creates a decoder drives its decode calls; construction-time and lazy decode-time allocations all land on the pinned device with **zero plugin changes** — covers `trt_decoder` and the closed-source `nv-qldpc-decoder` transparently. This addresses the lazily-allocating-decoder concern from the #634 review without per-call guards or new decode entry points. - **`decode_async()`** is the sole exception: its fresh `std::async` worker pins itself for the call's duration via a lib-private RAII `CudaDeviceGuard` (`libs/qec/lib/hardware_guards.h`, header-only; PR2 extends it). - **YAML realtime config:** `cuda_device_id` is a top-level `decoder_config` field (next to `type`/`transport` — placement knob common to any GPU decoder, not a per-decoder algorithm arg). Surfaced by `prepare_decoder_params()` for every decoder type; also exposed as `decoder_config.cuda_device_id` in Python. - **Realtime session:** the host dispatcher (one thread, all decoders) applies each decoder's device set-if-different (no restore) before enqueue and before DEVICE-mode graph capture; no-op for unpinned decoders. - **Python kwargs** work with no binding changes: `qec.get_decoder("trt_decoder", H, cuda_device_id=1)`. To place N decoders on N GPUs, create each decoder on its own thread (`std::async` / `threading.Thread`) — documented in `realtime_decoding.rst`. ## Testing - 7 new C++ unit tests (`DecoderCudaDeviceId.*`): kwarg contract (absent/negative/out-of-range), key stripped before strict-validating plugin ctors, persistent pin observable after construction, **two decoders on two GPUs from two threads**, async worker pinning itself (verified RED→GREEN on a 2-GPU machine). GPU tests skip below the needed device count. - YAML: round-trip with the new field; `prepare_decoder_params` surfaces the knob for trt and non-trt types (ordering vs the trt-only early return is locked by test). - Python: kwargs error path (`cuda_device_id` named in the error) and valid-id construct+decode. Full `test_decoder.py` 61/61. - Full suites: `test_decoders` 52/52, `test_decoders_yaml` 20 pass / 2 skips (nv-qldpc-gated). ## Notes for reviewers - Known pre-existing quirk (not introduced here): negative int kwargs from Python surface as an opaque `bad_cast` because `hetMapFromKwargs` stores Python ints as `size_t` (`libs/core` `kwargs_utils.h`) — the friendly negative-id message is only reachable from C++/YAML. Follow-up candidate in core. - `nv-qldpc-decoder` internal changes (mirroring the trt pattern) are a separate follow-up. Supersedes #663 (same content, reopened from the fork). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: kvmto <kmato@nvidia.com> Signed-off-by: Melody Ren <melodyr@nvidia.com> Co-authored-by: kvmto <kmato@nvidia.com>
melody-ren
added a commit
that referenced
this pull request
Jul 13, 2026
Summary: - Worker threads in the decoding server now pin themselves to the decoder's cuda_device_id before serving. Construction only pinned the registry thread, so every worker was decoding on the default device regardless of the pin prior to this patch. - The direct call path (no realtime session) had the same problem: configure_decoders leaves the thread on the last decoder's device, so decoder 0 would decode on decoder N's GPU. Each decode now selects its own decoder's device first. - If a pin can't be honored we fail instead of decoding on the wrong device: a worker that can't pin aborts server startup, and a cudaSetDevice failure during dispatch returns an error response instead of logging a warning and carrying on. - Hololink: HOLOLINK_GPU_ID and cuda_device_id both name the GPU the FPGA is attached to, so they must agree. Either one alone selects the device; setting both to different values throws at transport creation. Single decoder only, same as the rest of the gpu_roce path. - Nothing changes if cuda_device_id is not set. - Decoder construction is transactional: if a plugin constructor throws after its device was selected, the calling thread's CUDA device is restored to its previous value instead of being left on the failed decoder's device. Originally first introduced in #687. Adopting the change here. Outdated: This is a follow-up PR to #664, which should be merged before this PR This PR depends on #690 --------- Signed-off-by: kvmto <kmato@nvidia.com> Signed-off-by: Melody Ren <melodyr@nvidia.com> Co-authored-by: kvmto <kmato@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First PR of the #634 split (per @bmhowe23's review):
cuda_device_idonly — GPU decoder pinning, settable at construction via kwargs and the YAML realtime config. NUMA/mempolicy/cpu_affinity and thread-binding APIs are deferred to a follow-up draft PR (next release), Python interfaces for those to a third.What it does
decoder::get()reads and validatescuda_device_id(negative or >= device count →std::runtime_error), persistently pins the constructing thread (cudaSetDevice, no restore), strips the key before the plugin constructor, and stores it (get_cuda_device_id(), -1 = unpinned).trt_decoderand the closed-sourcenv-qldpc-decodertransparently. This addresses the lazily-allocating-decoder concern from the Add hardware pinning for QEC decoders #634 review without per-call guards or new decode entry points.decode_async()is the sole exception: its freshstd::asyncworker pins itself for the call's duration via a lib-private RAIICudaDeviceGuard(libs/qec/lib/hardware_guards.h, header-only; PR2 extends it).cuda_device_idis a top-leveldecoder_configfield (next totype/transport— placement knob common to any GPU decoder, not a per-decoder algorithm arg). Surfaced byprepare_decoder_params()for every decoder type; also exposed asdecoder_config.cuda_device_idin Python.qec.get_decoder("trt_decoder", H, cuda_device_id=1).To place N decoders on N GPUs, create each decoder on its own thread (
std::async/threading.Thread) — documented inrealtime_decoding.rst.Testing
DecoderCudaDeviceId.*): kwarg contract (absent/negative/out-of-range), key stripped before strict-validating plugin ctors, persistent pin observable after construction, two decoders on two GPUs from two threads, async worker pinning itself (verified RED→GREEN on a 2-GPU machine). GPU tests skip below the needed device count.prepare_decoder_paramssurfaces the knob for trt and non-trt types (ordering vs the trt-only early return is locked by test).cuda_device_idnamed in the error) and valid-id construct+decode. Fulltest_decoder.py61/61.test_decoders52/52,test_decoders_yaml20 pass / 2 skips (nv-qldpc-gated).Notes for reviewers
bad_castbecausehetMapFromKwargsstores Python ints assize_t(libs/corekwargs_utils.h) — the friendly negative-id message is only reachable from C++/YAML. Follow-up candidate in core.nv-qldpc-decoderinternal changes (mirroring the trt pattern) are a separate follow-up.Supersedes #663 (same content, reopened from the fork).
🤖 Generated with Claude Code