Add realtime_decoding_demo example: drive the delivered decoding server from an FPGA or a QPU kernel - #703
Merged
Conversation
…ng-server test Extend hsb_fpga_decoding_server_test.sh to drive the nv-qldpc Relay BP decoder through the decoding server's gpu_roce transport -- the self-relaunching device-graph scheduler (GpuRoceTransceiver): enqueue_syndromes / get_corrections / reset_decoder execute as DEVICE_CALLs on the GPU and the captured RelayBP decode graph fires device-side (fire-and-forget + tail-self-relaunch) when a window completes. This replaces hololink_qldpc_graph_decoder_bridge for the decoding-server flow; the old bridge test is untouched. hsb_fpga_decoding_server_test.sh: - --transport cpu_roce|gpu_roce, defaulted from the decoder profile (pymatching -> cpu_roce HOST_CALL path, nv-qldpc-decoder -> gpu_roce), plus --gpu, --proprietary-archive, --nv-qldpc-plugin. - Data generation (surface_code-4-yaml, fresh each run) adds --use-relay-bp for the nv-qldpc profile and injects `transport: gpu_roce` into the generated YAML: DecoderServer selects its transceiver from the per-decoder transport key, the generator omits non-default optionals, and the default (cpu_roce) resolves to the not-yet-implemented CpuRoceTransceiver stub. - gpu_roce server launch: Hololink parameters go via HOLOLINK_* env (the server's gpu_roce mode ignores the cpu_roce CLI flags), remote QP converted to decimal ($((qp)) -- GpuRoceConfig::from_env parses base-10 only), CUDA_MODULE_LOADING=EAGER as with the old bridge launcher, and readiness keyed on "QEC_DECODING_SERVER_READY gpu_roce" (the transceiver prints the QP/RKey/Buffer handshake before that line, without the bridge banner). - Build phase: builds gpu_roce_transceiver (HSB) and cudaq-realtime-bridge-hololink (cuda-quantum), wires CUDAQ_QEC_REALTIME_CUDEVICE_PROPRIETARY_ARCHIVE / GPU_ROCE_TRANSCEIVER_LIB / CUDAQ_REALTIME_BRIDGE_HOLOLINK_LIBRARY into the cudaqx configure, clears the CMake cache (find_library NOTFOUND staleness), and symlinks the nv-qldpc plugin into build/lib/decoder-plugins. All gated on the proprietary archive's presence so pymatching-only rigs build unchanged. The plugin symlink is also created opportunistically on --build-less runs. decoding-server-cqr/CMakeLists.txt (bug fix, separable): - CUDAQ_GPU_ROCE_AVAILABLE was set as CACHE INTERNAL on success, but the plain `set(... FALSE)` at the top of the file shadows the cache under CMP0126 (NEW), so the compile-definition gate for cudaq-qec-decoding-server never fired: the library was silently built WITHOUT gpu_roce (make_transport threw "CUDAQ_GPU_ROCE_AVAILABLE is not set" at runtime) while the sibling decoding_server target -- which reads the cache -- reported "gpu_roce transport enabled". Set the normal variable TRUE alongside the cache entry. Signed-off-by: Chuck Ketcham <cketcham@nvidia.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
Signed-off-by: Ben Howe <bhowe@nvidia.com>
With CUDAQ_GPU_ROCE_AVAILABLE now actually reaching the library target (the CMP0126 normal-vs-cache shadow fix), cudaq-qec-decoding-server links libdoca_gpunetio.so and libcudaq-realtime-bridge-hololink.so -- both of which reference the CUDA Driver API (cu*). Consumers of the static lib (test_decoding_server_core, test_decoders_yaml, decoding_server) inherit those dependencies, and on driverless build machines (CI containers: CUDA toolkit present, no GPU driver, no libcuda.so.1) their links fail with dozens of `undefined reference to cuStreamCreate` etc. -- the failure seen on all four QEC Build-and-test CI jobs. Link CUDA::cuda_driver alongside CUDA::cudart in the gpu_roce block: the CMake target resolves to the real driver where one is installed and to the toolkit's stubs (lib/stubs/libcuda.so) in driverless environments. This matches the existing precedent in unittests/utils (playback and the old bridge already link CUDA::cuda_driver). Signed-off-by: Chuck Ketcham <cketcham@nvidia.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
With CUDAQ_GPU_ROCE_AVAILABLE actually reaching the library target (the CMP0126 fix), every consumer of cudaq-qec-decoding-server (test_decoding_server_core, test_decoders_yaml, the CQR plugin) inherited runtime dependencies on libdoca_gpunetio.so and libcudaq-realtime-bridge-hololink.so, which require libcuda.so.1 at LOAD time. On driverless CI machines (CUDA toolkit + HSB + DOCA present, no GPU driver) all four QEC Build-and-test jobs failed: gtest_discover_tests executes the test binaries at build time and the loader cannot satisfy libcuda.so.1 -- link-time driver stubs satisfy ld but not the runtime loader. Restructure so the core library carries no GPU RoCE dependencies while keeping full CI build coverage of the HSB path: - New optional component cudaq-qec-decoding-server-gpuroce holds GpuRoceTransceiver.cpp plus GpuRoceFactory.cpp, the strong definition of the factory (cudaqx_qec_make_gpu_roce_transceiver) that DecodingServer.cpp now references weakly. All DOCA / hololink / CUDA-driver link deps move to this component. - make_transport(gpu_roce) calls the factory when it is linked in and throws a clear "GPU RoCE support is not linked into this binary" error otherwise; the ifdef and the direct GpuRoceTransceiver construction are gone from the core. - The scheduler hookup no longer dynamic_casts to the concrete type: ITransceiver grows a launch_device_scheduler(void*) virtual (default: no device scheduler), overridden by GpuRoceTransceiver to forward to launch_scheduler(). The config-driven constructor keys the hookup on the transport enum instead. - decoding_server links the component WHOLE_ARCHIVE in its proprietary-archive-gated gpu_roce block -- a weak reference does not pull archive members. - A link canary (cudaq-qec-decoding-server-gpuroce-linkcheck; never executed, not installed, not a test) preserves CI link validation of the component: a static archive alone never resolves symbols, and the one real consumer (the daemon's gpu_roce block) is gated on the proprietary cudevice archive, which CI does not provision. The canary forces full link resolution against the hololink / DOCA libraries with the driver stubs -- the same check that used to happen incidentally via the test binaries. - decoding_server's QEC_HAVE_GPU_ROCE_TRANSPORT gate is split from the proprietary-archive gate: the gpu_roce CLI branch references only the (now dependency-free) core DecodingServer API, so it compiles and links whenever HSB/DOCA are present -- CI now builds that branch too. The component + cudevice whole-archive link stays behind the archive gate, so the CI daemon remains loadable on driverless runners. Branch [2a] wraps server startup in try/catch, turning what used to be std::terminate on any gpu_roce bring-up failure into an "ERROR: gpu_roce startup failed: ..." message with exit 1. Net CI coverage: GpuRoceTransceiver.cpp compiles against the HSB/DOCA headers (component builds in `all`), its link resolves (canary), and the daemon's gpu_roce CLI branch compiles and links (split gate). No binary CI executes carries a libcuda.so.1 dependency. On dev rigs the daemon keeps full gpu_roce. As a bonus, the CQR plugin .so is loadable on driverless machines again (it too had inherited the deps). Signed-off-by: Chuck Ketcham <cketcham@nvidia.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
The server now contains no transport-specific code. --transport=<name> resolves to libcudaq-realtime-bridge-<name>.so next to the CUDA-Q realtime libraries (udp and cpu_roce ship there, incl. the TCP rendezvous and hsb_fpga QP exchange that previously lived inline here); --transport=/path/to/lib.so loads a partner's out-of-tree provider verbatim with no changes to this server. Unrecognized CLI arguments are forwarded to the provider's create(), so the existing flags (--port/--num-slots/--slot-size/--device/--local-ip/--qp_config/ --peer-ip/--remote-qp/--frame-size) keep working. Dispatch moves from a hand-rolled thread around cudaq_host_ring_dispatch_loop to the libcudaq-realtime dispatcher object (create/set_ringbuffer/set_function_table/set_control/start), the supported surface after cuda-quantum PR 4869; ring geometry comes from the provider's geometry query and readiness from its endpoint query (READY line format unchanged: port= hoisted first for the sscanf in test_decoding_server.cpp). gpu_roce is untouched: it predates the provider interface and takes a different dispatch shape (device-side scheduler; PR 670 is active in that code). CMake: the server no longer links the udp/cpu_roce transports, the host-dispatch archive, or ibverbs -- providers are runtime artifacts. Requires a CUDA-Q realtime install with bridge interface v2 (endpoint info + ring geometry) and the provider libraries. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
The Hololink transceiver bring-up moves behind the CUDA-Q realtime transport-provider interface: the constructor loads the provider (built-in libcudaq-realtime-bridge-hololink.so, or the library named by CUDAQ_REALTIME_BRIDGE_LIB) and adopts its RING_BUFFER context, geometry, and endpoint identity (v2 queries, with config-derived and banner fallbacks for a v1 provider). launch_scheduler()'s device-graph scheduler wiring is unchanged; the provider's launch() now owns the Hololink monitor thread, and disconnect()/destroy() replace direct hololink_close/destroy calls. Consequence: cudaq-qec-decoding-server no longer links Hololink / DOCA / HSB / ibverbs at all -- CUDAQ_GPU_ROCE_AVAILABLE now gates only on the CUDA-Q realtime headers, libcudaq-realtime.so, and the CUDA toolkit, so the gpu_roce path compiles on machines with none of the RDMA stack installed and fails loudly at runtime when the provider .so is absent. (Also applies PR 670's CMP0126 cache-shadowing fix so the compile definition actually fires.) Runtime validation on HSB rigs still required: this box has no DOCA / HSB, so the provider .so cannot be built or exercised here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
# Conflicts: # libs/qec/lib/realtime/decoding-server-cqr/CMakeLists.txt # libs/qec/tools/decoding-server/CMakeLists.txt Signed-off-by: Ben Howe <bhowe@nvidia.com>
…roce Follow-up to the bridge-provider refactor + PR 670 merge, closing the naming leaks that made the device path look like (and require the vocabulary of) a specific wire: - GpuRoceTransceiver/GpuRoceConfig -> DeviceGraphTransceiver/-Config (files, class, weak factory symbol cudaqx_qec_make_device_graph_transceiver, component cudaq-qec-decoding-server-device-graph, defines CUDAQ_QEC_DEVICE_GRAPH_AVAILABLE / QEC_HAVE_DEVICE_GRAPH_DISPATCH). It never was a transport: it is the GPU device-graph dispatch engine riding whatever bridge provider is loaded. - Per-decoder YAML key is now `dispatch: host|device_graph` (DecoderDispatch), naming the dispatch SHAPE instead of a wire. The legacy `transport: cpu_roce|gpu_roce` key and spellings remain input-only aliases. Fixes a subtle parser bug found while testing: the alias mapping must use two-arg mapOptional, or an absent legacy key resets the parsed dispatch value to the default. - Config env vars are QEC_DEVICE_GRAPH_* with the HOLOLINK_* spellings as fallback (values are forwarded to whatever provider is loaded, which need not be Hololink; existing orchestration scripts keep working). - decoding_server routes to the device-graph path from the YAML's declared dispatch shape, not from --transport=gpu_roce (kept as a legacy alias). The CLI/YAML mismatch that used to silently select a stub transceiver is now a loud, specific error, and a non-legacy --transport value selects the provider for the device path the same way it does for the host path. Validated: udp two-process suite 5/5, core 5/5; dispatch: device_graph, legacy transport: gpu_roce + HOLOLINK_* env, and the mismatch error all exercised end-to-end (device path fails at provider load on this box, as expected without the hololink provider .so). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
…split Everything here is new in this development cycle with no deployed consumers, so the transition aliases go: - YAML: the per-decoder legacy key `transport: cpu_roce|gpu_roce` and the wire-named enum spellings are gone; `dispatch: host|device_graph` is the only form. An old config now fails at parse time with "unknown key 'transport'" (and the server reports YAML parse errors cleanly instead of terminating on the exception). - CLI: `--transport=gpu_roce` is gone; --transport only ever names the wire/provider, and the device-graph path is selected solely by the config's dispatch key (default provider: built-in hololink). READY sentinel for that path is now `QEC_DECODING_SERVER_READY device_graph`. - Env: QEC_DEVICE_GRAPH_* only; the HOLOLINK_* fallback spellings are removed. - DeviceGraphTransceiver now REQUIRES a bridge-interface-v2 provider (ring geometry + endpoint identity queries); the v1-provider fallback paths are gone. - CpuRoceTransceiver (a stub whose constructor always threw) is deleted; DecoderDispatch::host in the standalone DecodingServer path throws a direct, accurate error instead. - hsb_fpga_decoding_server_test.sh updated to the new spellings (dispatch: device_graph injection, QEC_DEVICE_GRAPH_* env, no --transport for the device path, new READY sentinel). Validated: udp two-process suite 5/5, core 5/5; device path reached via dispatch: device_graph with QEC_DEVICE_GRAPH_* env; removed spellings all fail loudly (unknown YAML key; HOLOLINK_* env ignored with a clear missing-variable error). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
…n plan) Records the bridge-provider refactor end to end so it can be reproduced from the document alone: problem statement, after-architecture with the device-link vs dlopen design rules, the v2 bridge interface contract and the three providers, the server CLI/YAML/env contracts, the dispatch-vs- wire split, the ordered reproduction plan across both repos, validation gates (including the known pre-existing failure), and recorded follow-ups (rig validation, upstreaming, per-decoder rings, fan-in, vp_id). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
…r_id The QEC device wrappers now route every decoding RPC through cudaq::device_call's device-id overload with device_id == decoder_id. The CUDA-Q device_call runtime keys sessions, rings, and dispatchers by device id, so each decoder gets its own ring buffer and dispatcher: no shared ring, no head-of-line blocking between decoders, per-decoder backpressure. The payload keeps decoder_id as a cross-check (and for future multi-source fan-in). The CQR service plugin needs no change for the host path: per-device sessions share the same HOST_CALL handlers and the registry still routes by payload decoder_id. New app example surface_code-5-per-decoder-rings (+ test script, registered in ctest): two pymatching decoders decoded over two rings by one kernel, verified by (1) per-decoder corrections + dispatch count, (2) two per-device session-init lines in the runtime log, and (3) the per-device channel override (CUDAQ_DEVICE_CALL_CHANNEL= host_dispatch,1=device_dispatch) steering device 1 to the GPU channel while device 0 stays on its host ring. Pairs with cuda-quantum commit '[device-call] Per-device sessions on demand + per-device channel spec' (lazy per-device init; channel spec syntax; external channels keep one shared endpoint). Design doc per_decoder_rings_design.md records the topology, the CPU-ring + GPU-ring channel matrix, the recipe for the plugin's Gpu-mode session (the remaining piece for simultaneous CPU+GPU rings), and follow-ups. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
The server now opens ONE provider instance per decoder in the YAML -- own endpoint, own ring buffer, own dispatcher -- completing the two-process form of the one-ring-per-decoder topology (the product path; the earlier in-process demo only proved the runtime mechanism). Wire contract additions (single-decoder output unchanged apart from a ring0 token): - READY: 'QEC_DECODING_SERVER_READY port=<P0> transport=udp ring0=<P0> ring1=<P1> ...' -- leading port stays ring 0's for existing single-endpoint consumers, which keep working (all traffic lands on ring 0, payload-demuxed). - Shutdown: one 'QEC_DECODING_SERVER_RING decoder=<id> dispatched=<n>' line per ring, sampled AFTER dispatcher_stop (the loop flushes its stats counter on exit). New test DecodingServerTwoProcess.TwoProcessPerDecoderRings: two decoders, two udp endpoints, one caller wiring each decoder's device_call session to its own ring via device-scoped channel args (udp-port=<P0> udp-port.1=<P1>); asserts correct corrections AND >=3 dispatches on EACH ring -- proving per-ring traffic, not one shared wire. Full suite 6/6. Pairs with cuda-quantum commits 'Register bridge handles created through the cached-provider path' (second bridge instance on one provider was unusable) and 'Per-device external channel endpoints'. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
At run time the server is N independent 1-ring consumers constructed by
one setup loop: every decoder gets one bridge provider instance and one
ring (same CQR RPC wire format), and the decoder's dispatch: shape
selects the ring's consumer -- a host dispatcher-object thread, or the
CUDAQ device-graph scheduler. Host decoders and a device_graph decoder
now run simultaneously in one server; an all-device_graph config still
takes the standalone DecodingServer (HSB) path.
- DeviceGraphRingConsumer: the device-graph scheduler as a ring
CONSUMER (3 proprietary DEVICE_CALL entries + the decoder's captured
decode graph over a ring it does not own), extracted from
DeviceGraphTransceiver (which now delegates to it). Exposed through
a weak C ABI so server builds without the proprietary component link
and fail loudly at runtime instead.
- SessionRegistry accepts mixed dispatch shapes; the single-transceiver
DecodingServer paths still require uniformity (required_dispatch()
throws on mixed).
- CQR plugin exports cudaqx_qec_decoding_server_graph_resources(id) so
the server can wire the scheduler to a decoder hosted behind the
plugin (graphs are captured at session creation, before READY).
- Decoder YAML gains an optional per-decoder transport override
('transport: "udp --pinned-rings"', or 'hololink' for the builtin
provider slot). One EXTERNAL provider library per process (loader
limitation) is diagnosed with a clear error.
Validated locally: mixed config brings up ring0 (host) + ring1 (pinned
udp), READY publishes both rings, the non-graph decoder fails loudly at
the graph-capture requirement with clean teardown; full two-process
suite 6/6, per-decoder-rings example green. Positive decode on the
device ring needs a graph-capable decoder built against this tree's
ABI (see design doc follow-ups).
Pairs with cuda-quantum commits 'UDP transport external-rings API +
provider --pinned-rings' and 'Do not re-export transport-archive
symbols from the runtime' (ODR fix this work flushed out).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Ben Howe <bhowe@nvidia.com>
Replaces the per-decoder transport override added with the mixed-
dispatch server: the wire is deployment configuration and lives OUTSIDE
the decoders list. Transports differ between rings only by dispatch
shape (a device_graph ring must be GPU-pollable), so the top-level YAML
transport: section carries provider/args plus one shape-keyed
device_graph: override -- decoder entries carry no transport
information and stay portable across environments.
transport:
provider: udp
device_graph:
args: [--pinned-rings]
Per-ring precedence: shape override > section provider/args >
--transport CLI default; an explicit --transport still overrides the
section's provider for one-off experiments.
Validated: mixed config brings up both rings from the YAML alone (no
transport CLI); full two-process suite 6/6; per-decoder-rings example
and device_call tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Ben Howe <bhowe@nvidia.com>
… + WSL2 wall The sc4-cqr external channel guard wires per-decoder device_call sessions to their own rings via QEC_DECODING_SERVER_PORT_<id> -> udp-port.<id>= scoped args. Design doc records the positive-validation campaign: nv-qldpc rebuilt against this ABI (cuda-qx scratch merge), two-process per-decoder-ring CONTROL run decodes correctly with two host nv-qldpc rings; the mixed run's GPU scheduler serves DEVICE_CALLs over pinned-udp rings end to end (raw RPC datagram probes: reset + full-window enqueue round-trip status=0); the remaining decode-graph firing step is blocked by a platform limitation isolated with a minimal CUDA probe -- device-side cudaStreamGraphFireAndForget launches silently never execute on this WSL2 stack -- so the final step runs on the rig. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
…aunch The wedge is not a silent no-op: launching any kernel referencing device-side cudaGraphLaunch fails with cudaErrorNotSupported, visible only via cudaGetLastError() immediately after the launch (sync reports success). Documents the 10-second gating check for rig/CI machines. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
…s exonerated Rig data showed the original probe failing on target hardware too, which exposed the probe's API-contract violation: fire-and-forget device graph launch is only legal from a kernel inside a device-launched graph; a plain trigger kernel correctly gets cudaErrorNotSupported everywhere. The corrected two-graph probe (trigger inside a device-launched parent) PASSES on WSL2, native SASS and forced-JIT alike, and artifact arch coverage is ruled out. The wedge is a real pipeline issue, locally reproducible; next suspects recorded (triggered-graph instantiation flags/upload, tail relaunch, decode graph execution), plus the missing device-side error surfacing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
Three-way split with a sharpened boundary: - transport_provider_design.md now owns ALL deployment contracts in one place: alongside the provider ABI, section 4.2 gains the top-level YAML transport section (shape-keyed override + precedence), the multi-ring READY tokens and per-ring shutdown lines, the N-ring server flow, and the caller-side per-ring endpoint contract (key.<id>= scoped args, channel spec). Stale follow-ups refreshed. - per_decoder_rings_design.md slims to pure topology (rings, consumers, dispatch shapes, mixed server), pointing at the provider doc for contracts; the validation campaign narrative is replaced by a short status section. - per_decoder_rings_validation_notes.md (new) is the explicit campaign log: what is validated, the open decode-graph wedge with ordered suspects, the probe methodology including the invalid-probe lesson, the proprietary-rebuild recipe, and the build/runtime gotchas. No code changes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
Add transport_provider_design.md section 4.3 with complete minimal recipes for the nine deployment shapes covered by the 4.2 contracts (udp loopback, per-decoder rings, cpu_roce rendezvous + hsb_fpga, mixed dispatch local/HSB, all-device_graph, partner drop-in, in-process channels). Renumber the two-axis-split section to 4.4 and cross-reference the cookbook from per_decoder_rings_design.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
Root cause of the device-graph scheduler wedge: cooperative-launch co-residency starvation. The decode graph was captured with reserved_sms=0, sizing its cooperative grid for every SM; with the persistent dispatch graph resident, the device-side fire-and-forget launch queued forever and deadlocked at grid.sync(), stalling the tail self-relaunch. DecodingSession now captures with reserved_sms=1 (QEC_DECODE_GRAPH_RESERVED_SMS overrides, e.g. on rigs where Hololink RX/TX kernels are also resident). Same bug class, second instance (documented, plugin-side fix pending): HOST-path nv-qldpc decode launches unreserved cooperative kernels and hangs when a scheduler is resident -- in mixed deployments put a CPU decoder on host rings, or plumb reservation into the host decode. DeviceGraphRingConsumer logs the new trigger diagnostics (cudaq_dispatch_get_trigger_debug) at shutdown, and dispatched() now uses an async copy on a non-blocking stream -- a legacy default-stream cudaMemcpy synchronizes against the persistent scheduler graph and self-deadlocks. Proof (two-process, pinned-udp device ring, WSL2 laptop): sc4 app, 10 shots at p_spam=0.08, multi_error_lut host ring + nv-qldpc RelayBP device ring: trigger rc=0, fires=12 == tail_relaunches, both rings dispatched=72, decoder[1] corrections=2 logical_errors=0/10, clean teardown. Full regression suites stay green. Pairs with cuda-quantum commit 'Surface the device-side trigger-launch result'. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
…ss limit Pairs with cuda-quantum '[realtime] Key the bridge-provider loader by library name/path'. The server now passes each ring's resolved provider library to cudaq_bridge_create_from_library directly: no more CUDAQ_REALTIME_BRIDGE_LIB setenv side-channel for the per-ring path, no 'hololink is the builtin slot' special case, and rings may mix provider libraries freely -- the corresponding startup error and design-doc limitation are removed. DeviceGraphTransceiver likewise names its default provider library (hololink) as a plain string, with CUDAQ_REALTIME_BRIDGE_LIB still honored as the replacement override. Also note in --help that the installed cudaq-realtime is the source of truth for available providers and their arguments; the enumerated names and args are examples. Validated: two-process suite 4/4, per-decoder-rings app test, and the full mixed device-graph E2E (trigger rc=0, fires=10=tail_relaunches, both rings dispatched=60, real corrections on both decoders). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
decoding_server.cpp: - Set transport_from_cli when --transport is parsed: the documented "explicit --transport overrides the YAML transport section" precedence was dead code (the section's provider always won). - The all-device_graph path now resolves its provider with the same precedence as the per-ring loop (CLI > section device_graph override > section provider > built-in default) instead of consulting only --transport != udp. - The v1-provider endpoint fallback advertises the ring's own resolved provider, not the process default. - Guard --timeout and the provider port token against non-numeric input (clean error instead of an uncaught std::stoi/stoul exception). - Include DeviceGraphRingConsumer.h so the weak C-ABI redeclarations are compile-checked against the canonical prototypes. CQR components: - Consolidate the SM-reservation knob on the documented device-graph env family: DecodingSession reads QEC_DEVICE_GRAPH_RESERVED_SMS (was the undocumented QEC_DECODE_GRAPH_RESERVED_SMS), and the transceiver's parsed-but-unused reserved_sms field is removed. - Delete dead helpers left behind when the scheduler wiring moved into DeviceGraphRingConsumer (alloc_pinned_mapped, populate_device_call, GPU_CUDA_CHECK) -- they produced -Wunused-function noise. - DeviceGraphRingConsumer.h self-compiles with the component macro off (the C-ABI block's includes were inside the #ifdef). - Free the function table, shutdown flag, and stats allocations when the scheduler-stream create fails (the one error path without a cleanup ladder). - Fix a stale HOLOLINK_FRAME_SIZE error message (QEC_DEVICE_GRAPH_FRAME_SIZE) and a stale SessionRegistry comment; correct the READY-line doc: the leading port= token belongs to the first decoder listed in the YAML, not necessarily decoder id 0. Validated: two-process suite 4/4, per-decoder-rings app test, and the mixed device-graph E2E (trigger rc=0, fires=10=tail_relaunches, both rings dispatched=60, correct RelayBP decodes). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
… path The wire's identity lives in the YAML transport section; --transport now applies only to configs that intentionally leave the wire unspecified (one YAML reused across wires, selected per launch -- the pattern every two-process test uses). A config that names a provider combined with an explicit --transport is rejected at startup instead of resolved by a precedence rule, in both the mixed per-ring path and the standalone all-device_graph path. Help text and design docs updated to match. New coverage for the YAML path, which previously had none: - DecoderYAMLTest.TransportSectionAndMixedDispatch: round-trips the transport section (provider/args + device_graph shape override) with a host + device_graph decoder mix, and pins the exact YAML key spelling against a literal document. - DecodingServerTwoProcess.TwoProcessHostDispatchYamlTransportSection: full two-process decode where the server is launched WITHOUT --transport and the YAML names the provider and its args. - DecodingServerTwoProcess.TransportCliConflictsWithYamlSection: the conflict is rejected before READY with a nonzero exit and a pointed error message. - app_examples.surface_code-4-yaml-mixed-dispatch: the flagship mixed host+device_graph two-process flow as a gated ctest (registered when the server links the device-graph component; skips at runtime without a GPU or the nv-qldpc plugin). Asserts per-decoder rings on the READY line, per-decoder results in the app, a healthy scheduler (trigger rc=0, fires == tail_relaunches > 0), and traffic on both rings. ServerProcess (test harness) learns to launch without the CLI transport args, to fold stderr into the captured output, to report the child's exit code, and to bail out on EOF instead of burning the READY timeout. 29/29 in the decoding-server/YAML/session/app regression sweep, including the new tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
The after-diagram now reflects the one-ring-per-decoder consumers and the YAML transport section (--transport as fallback, conflict rejected). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
Signed-off-by: Ben Howe <bhowe@nvidia.com>
…ridge branch Main landed the squashed PR 670 plus NVIDIA#656/NVIDIA#673/NVIDIA#674/NVIDIA#678/NVIDIA#680/NVIDIA#683/NVIDIA#688/ NVIDIA#690/NVIDIA#691/NVIDIA#692. Resolution keeps this branch's design and ports main's content into it: - Naming: main's GpuRoce{Transceiver,Factory,LinkCheck} map 1:1 onto this branch's DeviceGraph* files (the de-transport-ing follow-up requested in the PR 670 review). Main's post-review fixes are ported into the renamed files: ring-size overflow + host-page-size alignment validation, gpu_id resolved from the decoder's cuda_device_id instead of an env var, the factory taking pinned_cuda_device, and the linkcheck exercising the new factory signature. - Schema: per-decoder 'dispatch: host|device_graph' plus the top-level transport section stay; main's per-decoder 'transport:' key does not return. cuda_device_id is adopted as a per-decoder placement knob (config struct + YAML trait), and the hsb script now injects 'dispatch: device_graph' + cuda_device_id and drives the server with QEC_DEVICE_GRAPH_* env and the device_graph READY sentinel. - Features adopted from main: decoder CUDA-device pinning (worker-thread pin via promise, graph-capture pin, hardware_guards.h), per-session decode counters + print_session_stats + QEC_DECODING_SERVER_STATS server-side stats printing, DecodingServer ctor exception safety, virtualized realtime decoder API (NVIDIA#674), CMP0126 fix, CUDA::cudart on the core lib, and the cudevice-archive dedup in the server link. - DecodingSession combines main's set_graph_capture_device with this branch's reserved-SMs decode-graph capture (QEC_DEVICE_GRAPH_RESERVED_SMS). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
Signed-off-by: Ben Howe <bhowe@nvidia.com>
Addresses PR 682 review feedback (the standalone transceiver was still hololink-shaped): - The transceiver no longer parses or formats wire-specific rendezvous tokens. It publishes the provider's interface-v2 endpoint description verbatim as one line -- 'QEC_DECODING_SERVER_ENDPOINT <key=value ...>' -- replacing the bespoke QP Number / RKey / Buffer Addr lines, and the qp/rkey/buffer_addr members and accessors are gone. The orchestration layer scrapes whatever tokens its wire uses (the HSB script now pulls qp=/rkey=/buffer_addr= from the endpoint line for the playback tool; a socket-style wire would pull port=). - Provider arguments are generic. The named QEC_DEVICE_GRAPH_* knobs (DEVICE, PEER_IP, REMOTE_QP, FRAME_SIZE, PAGE_SIZE, NUM_PAGES) each emit their --flag= only when set -- the constructor no longer requires HSB env, and no hololink-shaped defaults are forced on a foreign provider. --gpu= is the one always-passed argument (the decoder's cuda_device_id, not an env knob). QEC_DEVICE_GRAPH_PROVIDER_ARGS is a free-form pass-through appended verbatim after the named args, and the server's all-device_graph path fills it from the YAML transport section (section args + device_graph override), so a partner provider is configured entirely from the YAML. The ring-size / host-page alignment validation now gates on the geometry knobs being provided; authoritative geometry still comes from the provider's v2 query. Compile- and regression-verified locally (two-process suite, DecoderYAMLTest, per-decoder-rings and mixed-dispatch app tests, 26/26); the standalone path's endpoint handshake needs one HSB rig run of the updated script. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
Files: docs/sphinx/examples/qec/realtime_decoding_demo/run_realtime_decoding.sh
docs/sphinx/examples/qec/realtime_decoding_demo/README.md
docs/sphinx/examples_rst/qec/realtime_decoding_demo.rst
libs/qec/unittests/utils/hsb_fpga_decoding_server_test.sh
Adapt the example driver, its docs, and the hsb unittests script to the
merged transport-agnostic decoding server (bridge providers, one ring per
decoder, per-decoder dispatch). The example source needs no changes.
run_realtime_decoding.sh:
- Replace --transport with the server's two independent knobs, both derived
automatically: --wire udp|cpu_roce|hololink (the bridge-provider library
carrying syndromes into the server) and --dispatch host|device_graph (the
engine consuming each decoder's ring). Combinations this example has not
wired up are rejected with the reason: the qpu-kernel source is udp-only
for now (kernel-over-cpu_roce is planned follow-up work), device_graph
over udp (pinned shared rings) is not exposed, and device_graph serves
only nv-qldpc-decoder over the hololink wire. fpga + nv-qldpc +
--dispatch host is newly supported (GPU decode via host call over the
cpu_roce wire).
- Config injection writes the per-decoder `dispatch: device_graph` key
(replacing the retired `transport: gpu_roce`), still with cuda_device_id,
and verifies both landed.
- The device_graph server launch uses the QEC_DEVICE_GRAPH_* environment
(was HOLOLINK_*), drops the retired --transport=gpu_roce flag, and waits
for the "QEC_DECODING_SERVER_READY device_graph" sentinel.
- The RDMA handshake scrape reads the qp=/rkey=/buffer_addr= tokens from
whichever line carries them: the READY line on host dispatch (the bridge
provider's endpoint info rides it) or the device-graph transceiver's
dedicated QEC_DECODING_SERVER_ENDPOINT line. Replaces the retired
three-line "QP Number:/RKey:/Buffer Addr:" scrape.
- Pass/fail criteria additionally assert the new per-decoder ring counter
(QEC_DECODING_SERVER_RING decoder=0 dispatched > 0).
- The RoCE wire's provider token is passed hyphenated (cpu-roce): the server
resolves "libcudaq-realtime-bridge-<token>.so" literally and the built
provider soname is hyphenated.
hsb_fpga_decoding_server_test.sh:
- Pass the hyphenated cpu-roce provider token on the server command line.
- Scrape the handshake tokens from whichever line carries buffer_addr= (the
unconditional ENDPOINT-line wait broke the cpu_roce path, where the tokens
ride the READY line).
- Resolve the server's realtime lib dir via CUDAQ_REALTIME_LIB_DIR or the
conventional realtime build dirs, so the server dlopens the bridge-provider
API matching .cudaq_version instead of a stale build.
Docs (README + realtime_decoding_demo.rst): describe the wire/dispatch model
and the --wire/--dispatch overrides; the decoder table uses host /
device_graph dispatch vocabulary; the FPGA run section lists all three
decoder invocations explicitly.
Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
File: libs/qec/lib/realtime/decoding-server-cqr/DeviceGraphTransceiver.cpp
The transceiver configures its bridge provider by passing an argv-style
vector to cudaq_bridge_create_from_library(), but built that vector with the
first real option (--device=<nic>) at position 0. The provider's
parse_bridge_args follows the C argv convention and starts parsing at
argv[1], so --device was silently skipped on every start and the bridge fell
back to its compiled-in default device ("rocep1s0f0"). All remaining options
(--peer-ip, --remote-qp, --gpu, --page-size, --num-pages, --payload-size)
sat at index >= 1 and parsed correctly, so only the device was wrong and
nothing reported it.
On rigs where the default happens to name the cabled NIC the bug is
invisible. Where it does not, device_graph startup fails with
"gpu_roce_transceiver: Cannot find GID for RoCE v2": the default device's
GID table has only link-local entries, never the IPv4-mapped RoCE v2 GID the
transceiver searches for (confirmed via HSB debug logs showing "Opening DOCA
device with name: rocep1s0f0" while QEC_DEVICE_GRAPH_DEVICE named the cabled
port).
Prepend a program-name placeholder so the vector honors the argv[0]
convention and --device parses. The parser side is left alone: starting at
argv[1] is correct for its other callers, which pass a genuine argv[0].
Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
Signed-off-by: Ben Howe <bhowe@nvidia.com> # Conflicts: # libs/qec/lib/realtime/decoding-server-cqr/DecodingServer.cpp # libs/qec/lib/realtime/decoding-server-cqr/DecodingSession.cpp
Files: docs/sphinx/examples/qec/realtime_decoding_demo/run_realtime_decoding.sh
docs/sphinx/examples/qec/realtime_decoding_demo/README.md
docs/sphinx/examples_rst/qec/realtime_decoding_demo.rst
The qpu-kernel source can now drive the server over the cpu_roce wire
(--wire cpu_roce) with any of the three decoders, in addition to the
default udp wire, which is unchanged. No C++ changes: the example
binary's QEC_APP_CQR channel bring-up already carried the cpu_roce
client path; the script now selects it.
run_realtime_decoding.sh:
- The combination gate admits qpu-kernel + cpu_roce + host; qpu-kernel +
hololink and qpu-kernel device_graph dispatch stay rejected with their
reasons.
- RDMA topology comes from the same four env vars as every in-tree
cpu_roce test (CUDA-Q's CpuRoceChannelTester and the QEC two-process
test): CUDAQ_CPU_ROCE_TEST_{CHANNEL,DAEMON}_{DEVICE,IP}. The device
vars are required (the error lists the machine's IB devices); the IPs
default to 10.0.0.1 / 10.0.0.2.
- --setup-network now also covers this pair: per side it sweeps the IP
off any other interface (the duplicate-IP routing trap), configures
the port (reusing setup_port), and waits for the matching RoCE v2 GID
(the kernel populates it asynchronously after ip addr add). No ARP
seeding: the RDMA endpoints cross the TCP rendezvous.
- The cpu_roce server launch adds --device/--local-ip and pins the ring
geometry explicitly (--num-slots=8 --slot-size=256): the geometry is
part of the wire contract -- the kernel's channel writes straight into
the server's rings and hardcodes 8 x 256.
- The kernel invocation additionally carries
QEC_DECODING_SERVER_TRANSPORT=cpu_roce plus the channel-side topology
env; the READY scrape is tightened to the READY line (for cpu_roce the
port= token is the TCP rendezvous port).
- Pass/fail criteria are unchanged -- they are wire-agnostic.
Docs: README + rst gain the cpu_roce invocations for all three decoders
and the topology env-var block; the decoder table gets the udp/cpu_roce
wire vocabulary.
Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
…' into decoding_server_examples
18 tasks
…example New files: docs/sphinx/examples/qec/realtime_decoding_demo/surface_code_ising_realtime_decoding.cpp docs/sphinx/examples/qec/realtime_decoding_demo/gen_dsparse_from_memory_circuit.py Updated: docs/sphinx/examples/qec/realtime_decoding_demo/CMakeLists.txt docs/sphinx/examples/qec/realtime_decoding_demo/run_realtime_decoding.sh docs/sphinx/examples/qec/realtime_decoding_demo/README.md docs/sphinx/examples_rst/qec/realtime_decoding_demo.rst The example gains an opt-in fourth decoder profile: --decoder trt_decoder, the published Ising neural-network predecoder (a TensorRT engine built from its ONNX export) chained into a PyMatching global decoder, served by the same delivered decoding_server over all three shipped paths -- qpu-kernel over udp, qpu-kernel over cpu_roce, and the FPGA over cpu_roce (host dispatch on every path). The profile runs its own example source, a trimmed port of the in-tree surface_code-4-yaml app, because the Ising artifacts are bound to that lineage's conventions: orientation XV, per-round measurement layout [X-ancillas..., Z-ancillas...], and SPAM noise at the model's trained operating point (d=7, T=7, basis Z, p_spam=0.01). The port keeps the kernels and enqueue schedule byte-identical to the lineage (the artifact directory's D_sparse.txt maps Ising detector rows onto exactly that live measurement stream) and adds what the example needs on top: the cpu_roce device-call channel bring-up, a --seed flag, the in-process dispatch-count self-check, and --ising-artifacts-dir replacing the older --ising_bundle/--onnx_path pair (ONNX resolves to <dir>/model.onnx). The artifacts (six files: model.onnx, H_csr.bin, O_csr.bin, priors.bin, metadata.txt, D_sparse.txt) never ship with CUDA-QX: the weights are a gated Hugging Face download prepared through the NVIDIA/Ising-Decoding repository, and D_sparse.txt is generated against this example's own printed CNOT schedules with the gen_dsparse_from_memory_circuit.py copy shipped alongside (--ising-repo now required). The docs carry the full preparation recipe. An absent or incomplete artifact directory (or a missing TensorRT decoder plugin in the install prefix) makes the profile SKIP (exit 77), naming each missing file -- so CI and artifact-less machines are unaffected; the profile never runs in CI. run_realtime_decoding.sh: trt_decoder joins the decoder whitelist with its own binary pair selection; the profile pins d=7/rounds=7 and rejects conflicting explicit --distance/--num-rounds/--p-cnot (SPAM noise comes from the new --p-spam, default 0.01); the generated config is identity- checked (type + onnx_load_path); the server READY wait is extended for the TensorRT engine build; on the FPGA source the shot default is 56 -- the 512-frame playback BRAM ceiling at 9 frames/shot (8 syndrome slices + 1 corrections frame), the same fill-the-BRAM convention as the d3 default 85 -- and --spacing defaults to 5 ms for this profile: the 9-frame bursts overrun the server's 64-slot RX ring (~7 shots of buffer vs a measured ~35 us decode round trip) at the usual 10 us, corrupting the stream around shot 8-18. Also fixed here (bites every cpu_roce profile after a reboot): --setup-network's cpu_roce path now seeds static neighbor entries between the channel and daemon ports, the same-host analogue of the FPGA path's seed_fpga_neighbor. Same-host port pairs never ARP for each other's address (the kernel routes local IPs via lo and never probes the wire), so the RoCE path resolution failed with ibv_modify_qp(RTR) errno=110 once the neighbor state from the original bring-up was lost. Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
melody-ren
reviewed
Jul 22, 2026
run_realtime_decoding.sh: the nv-qldpc profiles cannot drain the server's 64-slot RX ring at the 10 us default playback spacing (measured on-rig: 171/510 frames captured on host dispatch, ~15/85 shots verified on device_graph, both looking like a broken FPGA/scheduler). Auto-pace them like the trt profile already does -- 5 ms on host dispatch, 100 us on device_graph (both measured-good) -- with an explicit --spacing always winning. Documented in the README and Sphinx page. hsb_fpga_decoding_server_test.sh: the 10.0.0.1 bridge-ip default is the emulate-loop convention; applied silently in real-FPGA mode it puts the server NIC on the wrong subnet and the only symptom is a hololink control-plane read_uint32 timeout that masquerades as dead hardware. Default the bridge IP by mode (10.0.0.1 emulate / 192.168.0.1 FPGA) and reject an explicit bridge-ip outside the FPGA's /24 with an error that names the failure mode. Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
cketcham2333
force-pushed
the
decoding_server_examples
branch
from
July 24, 2026 14:31
367b1cf to
65a6820
Compare
…mples Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
Two fixes from re-running the full FPGA matrix on this branch. hsb driver endpoint scrape: the upstream merge left the driver waiting for a dedicated QEC_DECODING_SERVER_ENDPOINT line, but only the device-graph transceiver prints one -- the host/cpu-roce path publishes the rendezvous tokens (qp=/rkey=/buffer_addr=) on the READY line itself, so the pymatching and trt profiles timed out at server startup and FAILed against the real FPGA. Wait for the buffer_addr= token instead and scrape whichever line carries it. Playback pacing: a decoder whose per-shot decode latency exceeds the playback tool's 10us default inter-shot spacing overruns the 64-slot RX ring (dropped enqueues mis-assemble a volume and the decode dies with an invalid-syndrome matching error; the ILA sample count comes up short). The slow profiles are trt_decoder (fixed per-invocation TensorRT engine overhead at batch 1) and nv-qldpc-decoder (iterative relay-BP); pymatching and multi_error_lut decode these volumes in microseconds and keep the unpaced default, which is part of what their profiles exercise. Auto-pace the slow profiles at a single measured 100us everywhere: the hsb driver gains the auto-pacing block (it had none -- its trt profile failed systematically at the 10us default), and run_realtime_decoding.sh drops its conservative 5ms values and its dispatch-type branch for the same 100us. README/rst updated to match. An explicit --spacing always wins in both scripts. Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
gen_dsparse_from_memory_circuit.py belonged to the pre-NVIDIA#713 Ising artifact recipe (manual Ising-Decoding clone, generate_test_data.py, ONNX export, then the shipped script for D_sparse.txt). NVIDIA#713 replaced all of that with prepare_ising_artifacts.py: one command that sparse-checkouts the pinned Ising revision, downloads the gated Hugging Face weights, exports the model, generates the matrices, captures the app's printed CNOT schedules, and writes a matching D_sparse.txt. Ship an adapted copy of prepare_ising_artifacts.py in the example directory (the demo stays self-contained; only the docstring, the --app help text, and the final usage hint differ from the app_examples original -- its prepare mode runs this example's generator, whose flags and schedule printout match what the script expects) and drop gen_dsparse_from_memory_circuit.py, whose logic NVIDIA#713 absorbed verbatim as the d-sparse subcommand. The rst recipe collapses from five manual steps to the one-command flow, and the README and app comments now name the new script. Verified end to end: the shipped script's full prepare mode (gated weights download, pinned sparse checkout, ONNX export, matrices, schedule capture from this example's generator) produced a bundle byte-identical to the known-good artifact directory across all six files, and the trt_decoder example passes on the udp and cpu_roce wires with both the pre-existing and the freshly prepared artifacts. Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
Collaborator
Author
|
/ok to test 9d846a5 |
bmhowe23
reviewed
Jul 24, 2026
…argv,
pacing semantics)
Six fixes from an external review of the example and its orchestration:
- Load-path precedence: --realtime-lib-dir now precedes the CUDA-Q prefix on
LD_LIBRARY_PATH. A CUDA-Q install can carry its own libcudaq-realtime.so,
and appending the explicit override after it let the prefix's copy shadow
the requested one (symptom: undefined symbol
cudaq_bridge_get_endpoint_info). Verified: --cudaq-prefix pointed at a
pre-v2 SDK with --realtime-lib-dir at the current bridge install now
PASSes.
- Build-time twin of the same bug: the example CMakeLists searches
CUDAQ_REALTIME_DIR before CUDAQ_INSTALL_DIR for cudaq-realtime-dispatch,
so an explicit separate realtime prefix cannot silently lose to the
CUDA-Q archive.
- FPGA subnet defaults: --bridge-ip now defaults to 192.168.0.1 (the FPGA
default's /24) instead of 10.0.0.1, and a guard rejects a bridge/FPGA
pair on different /24s up front -- --setup-network flushes the NIC, so
the mismatch previously surfaced minutes later as an opaque hololink
read_timeout_error. Mirrors the guard already in
hsb_fpga_decoding_server_test.sh.
- Argument parsing in surface_code_realtime_decoding.cpp: value-taking
options no longer read past argv, and malformed numbers exit with a
message instead of an uncaught std::invalid_argument ("--distance" alone
and "--distance abc" both reproduce a clean usage error now).
- Artifact-prep docs no longer claim "any machine": the pinned Ising export
runs GPU inference, so the recipe now says a machine with an NVIDIA GPU.
- --spacing semantics corrected everywhere: the playback engine's timer
fires once per FRAME (one BRAM window per frame), not per shot, so a shot
spans frames-per-shot x spacing. The playback tool's help, both
orchestration scripts' help and pacing comments, and the README/rst now
say per-frame (and drop the incorrect "9-frame burst" framing -- frames
are evenly spaced). All measured pacing values were validated under the
true semantics, so no defaults change.
Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
Collaborator
Author
|
/ok to test a8ffb2e |
Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
Collaborator
Author
|
/ok to test ae1383a |
melody-ren
approved these changes
Aug 3, 2026
…mples Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
Collaborator
Author
|
/ok to test 921e4bb |
The upstream/main merge auto-merged cleanly around three references this branch added to the playback tool by its pre-rename name (main renamed the target/binary in NVIDIA#754, this branch's additions were on untouched lines): - unittests/utils/CMakeLists.txt: install rule referenced the deleted hololink_fpga_syndrome_playback target (the CI configure failure). - run_realtime_decoding.sh: --install-prefix binary resolution pointed at the old binary name (would fail at runtime); plus two help-text mentions. - realtime_decoding_demo.rst: two doc mentions of the old tool name. Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
Collaborator
Author
|
/ok to test 5bb0ffc |
melody-ren
added a commit
to melody-ren/cudaqx
that referenced
this pull request
Aug 4, 2026
Bring in main's realtime_decoding_demo (NVIDIA#703 -- driving the delivered decoding server from a QPU kernel or FPGA) and its code directory, wired into our Realtime Decoding toctree rather than main's flat examples list. Signed-off-by: Melody Ren <melodyr@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a shipped example (
docs/sphinx/examples/qec/realtime_decoding_demo/) thatcompiles against the installed SDK — installed headers/libs only — and exercises
the delivered
decoding_serverend-to-end, plus its docs/CI registration and anFPGA RX-ring sizing fix in the unittests driver.
The example source is the refactored
surface_code-1.cppfrom #685 (new syndromeformat,
decoder_context_from_memory_circuit, seeded runs), copied with threeadaptations: the header comment, the usage line, and a true-width
--save_syndromecapture so the FPGA playback tool replays exact per-round measurement counts.
The example
One source, built two ways: a generator (decoder config + FPGA syndrome file) and
the lowered
-cqrkernel (-frealtime-lowering -DQEC_APP_CQR). One script, twosyndrome sources, three decoders (
pymatching,multi_error_lut,nv-qldpc-decoder):--source qpu-kernel— the lowered kernel streams syndromes to the server overUDP (no hardware;
QEC_DECODING_SERVER_PORTroutes the device_calls).--source fpga— the delivered playback tool streams syndromes from a realFPGA over RoCE (
cpu_roce/gpu_roce).Runs are seeded (
--seed, default 42), so results are reproducible. Pass/failmatches the existing surface-code tests: decoder-failure greps, the completion
proof, a
num_shots/50residual logical-error ceiling, an in-process dispatchcount of 0 (the two-process test's proof the decode stayed in the server), and a
num_shots*(num_rounds+3)server dispatch floor.CI
lib_qec.yamlbuilds the example against the installed SDK (enforcing theinstalled-headers-only rule) and runs a seeded 200-shot qpu-kernel pymatching
decode over UDP loopback on the CPU job — a real decode, no hardware, and
deterministic pass/fail.
FPGA ring fix (
hsb_fpga_decoding_server_test.sh)Fixes a #683 regression: the playback ring modulus (
--num-pages 512) no longermatches the cpu_roce server's 64-slot RX ring, starving the test after ~10 shots.
Both transports now run a ring equal to the HSB QP's 64-entry receive WQE depth
(deeper rings drop frames), with a fail-fast preflight of DOCA's host-page
alignment on 16K/64K-page hosts.