[QEC] Decoding server: serve HOST_CALL requests inline; retire the worker threads, queues, and CqrTransceiver by cketcham2333 · Pull Request #769 · NVIDIA/cudaqx

cketcham2333 · 2026-08-06T00:33:13Z

Summary

Retires the cross-thread request path in the QEC decoding server ("fix A" of the
decoding-server redesign). Every host RPC previously crossed two server threads:
shim → CqrTransceiver::inject (frame copy, token, pending map) → RpcDispatcher
re-parse/re-route → WorkItem copy → bounded queue → per-decoder worker thread →
decoder → transport.send → pending lookup → tx-slot memcpy → promise wakeup of
the parked dispatcher — two thread crossings and ~5 buffers per get_corrections.

Now the shim resolves the DecodingSession by the payload's decoder_id and
calls its handle_* method inline on the CUDAQ dispatcher thread: the rx
slot is parsed in place, the decoder runs on the calling thread, and the
response is written straight into the tx slot (magic release-stored last).
Zero thread crossings, zero copies, zero steady-state allocations — the same
shape as the GPU device-graph dispatch path and the legacy direct path
(host::enqueue_syndromes), whose validation ladder, capture hook, and pin
helper handle_enqueue now shares.

Net: +960 / −2,702 lines.

Measured latency (udp two-process rig, d3 surface code, pymatching, 500 shots)

RPC	before (median)	after (median)	p99 after
enqueue_syndromes	0.8 µs	0.10 µs	0.86 µs
get_corrections	1.6 µs	0.08 µs	0.10 µs
reset_decoder	4.8 µs	0.10 µs	0.10 µs

(get_corrections no longer contains the decode wait — the decode runs inline
in the volume-completing enqueue, whose tail carries the actual decode time.)

What's new

RpcSlot.h — single home for in-place slot parsing and in-place response
writing; deliberately device-portable (no allocation/exceptions/STL) so the
GPU dispatch kernel can later compile the same source.
DecodingSession — payload-level cores plus inline handle_* entry
points; corrections pack directly into the tx slot (no truncation: an
oversized result fails with INTERNAL_ERROR).
Debug-only single-caller guard — a scoped in-flight assert catches the
invalid two-rings-one-decoder topology loudly in debug builds and compiles
out entirely in release (invalid-config guards must not add hot-path
latency). Always-on enforcement is deliberately omitted: the HOST_CALL ABI
carries no ring identity to validate against, and multi-sender serialization
belongs to the multi-ring decoding-unit design.
pin_decode_device_cached — thread-local-cached CUDA pin; pin failures
surface at server bring-up via a create-time probe.
HopStats — slimmed to single-thread stage stats (parse/decode/respond);
QEC_DECODING_SERVER_HOP_STATS and the total metric keep their semantics.

What's deleted (~1,500 LOC)

CqrTransceiver.h (singleton transceiver: inject, tokens, pending map,
promises, inbox, frame builders)
RpcDispatcher + ResponseWriter (second-stage re-parse/re-route)
DecodingSession worker machinery (WorkItem, bounded queue, worker thread,
syndromes_dropped latch — backpressure is the transport ring; the
SYNDROMES_DROPPED/BUSY codes stay reserved on the wire but are no longer
emitted)
RoundAccumulator + SyndromeMappingTable (unreachable fragment-merge
machinery; the syndrome_mapping_id wire field stays — id 0 accepted,
others rejected with BAD_REQUEST)
SpinPolicy.h + QEC_DECODING_SERVER_SPIN_US (the cross-thread waits they
optimized no longer exist)
LoopbackTransceiver, the recv loop, install_dispatch_sink

Behavior notes

Enqueue response moves to the handler tail; client-visible timing is
unchanged (the tx flag publishes on handler return). The deferred-error
contract is unchanged.
A volume-completing enqueue runs its decode inline and blocks only its own
ring: decoder parallelism comes from per-decoder rings (one dispatcher
thread each). A shared ring still routes multiple decoders correctly but
serializes their decodes.
Test-harness updates accordingly: surface_code-4-yaml-test.sh exports the
per-decoder ring ports (all tokens required, fail-fast), and
hard-patch-pymatching runs 500 shots so its opportunistic overlap evidence
is statistically certain (the deterministic proof is the three-way-overlap
barrier test).
run_realtime_decoding.sh: fixes a hololink→hsb rename leftover
(--hololink → --hsb-ip) needed for --source fpga runs.

…rker threads, queues, and CqrTransceiver The host request path used to be: shim -> CqrTransceiver::inject (frame copy, process-unique token, pending_ registration) -> RpcDispatcher re-parse/re-route -> WorkItem copy -> bounded queue (depth 64) -> per-decoder worker thread -> decoder -> transport.send -> pending_ lookup -> memcpy tx slot -> promise wakeup of the parked dispatcher. Two thread crossings and ~5 buffers per get_corrections. Now dispatch_rpc resolves the DecodingSession by the payload's decoder_id (the HOST_CALL ABI carries no context pointer, and a shared ring may serve several decoders) and calls its handle_* method, which parses the rx slot in place, runs the decoder on the calling CUDAQ dispatcher thread, and writes the RPCResponse straight into the tx slot (magic release-stored last). Zero thread crossings, zero copies, zero steady-state allocations. Same shape as the GPU device-graph dispatch path and the legacy direct path (host::enqueue_syndromes), whose validation ladder, capture hook, and pin helper handle_enqueue now shares line for line. New/changed: - RpcSlot.h: single home for in-place slot parsing and in-place response writing (EnqueueView/parse_enqueue moved from CqrTransceiver.h; GetCorrections/Reset views, peek_decoder_id, write_response, ResultWriter). Deliberately device-portable (no allocation/exceptions/STL) so the GPU dispatch kernel can later compile the same source — one point of truth for the wire layout. - DecodingSession: payload-level cores (enqueue_core, get_corrections_core, reset_core) plus the inline handle_* entry points; get_corrections packs corrections directly into the tx slot, and a result larger than the slot fails with INTERNAL_ERROR (same no-truncation rule as the old CqrTransceiver::send). - The enqueue response moves to the handler tail (single write point). Client-visible timing is unchanged: the CUDAQ dispatcher publishes the tx flag only when the handler returns. A volume-completing enqueue runs its decode inline and blocks only this session's ring — decoder parallelism comes from per-decoder rings (one dispatcher thread each); a shared ring still routes correctly but serializes. - Deferred-error contract unchanged: enqueue failures latch shot_state=failed, respond OK, and surface as INTERNAL_ERROR at the next get_corrections. - Busy accounting moves into a BusyScope RAII in the handlers; cudaqx_qec_decoding_server_max_concurrent keeps its name and now measures sessions concurrently executing on dispatcher threads. - Single-caller tripwire: a debug-only scoped in-flight guard asserts that no two handlers execute on one session concurrently — catching the invalid two-rings-one-decoder wiring loudly while tolerating legal serial dispatcher-thread replacement. It compiles out entirely in release — invalid-config guards must not add hot-path latency. Enforced (always-on) protection is deliberately NOT added: the HOST_CALL ABI carries no ring identity to validate against, and two senders feeding one decoder is semantically broken (interleaved rounds decode wrong) regardless of memory safety — real multi-sender serialization is the multi-ring decoding-unit design (fix C). - pin_decode_device_cached (hardware_guards.h): thread_local-cached device pin (one compare per call steady-state; debug re-verify), and CUDA pin failures surface at bring-up via a CudaDeviceGuard probe in DecodingSession::create (was: the worker pin handshake). - The shim's duplicated --save_syndrome capture is deleted; capture now uses the same hooks as the legacy path, byte-identical output. Deleted (~1,500 LOC) — everything the inline path made unreachable; every layer kept "just in case" is a layer customers must read past: - CqrTransceiver.h: the singleton transceiver (inject, tokens, pending_ map, per-call promises, inbox deque, frame builders, write_ack). Correlation state is unnecessary when each call writes its own tx slot synchronously. - RpcDispatcher.h/.cpp + ResponseWriter: the second-stage re-parse/ re-route of a call the CUDAQ function table already routed. - DecodingSession worker machinery: WorkItem, bounded queue, worker thread, try_enqueue, worker_loop, send_response, and the syndromes_dropped latch — there is no queue to overflow; backpressure is the transport ring. SYNDROMES_DROPPED/BUSY stay reserved in the wire enum but are no longer emitted. - RoundAccumulator + SyndromeMappingTable: unreachable fragment-merge machinery (only the hardcoded identity table ever existed; vp_id was always 0). The syndrome_mapping_id wire field stays: id 0 accepted, anything else rejected with BAD_REQUEST instead of silently decoding as identity; non-identity mappings re-enter via the identity-aware decoder API (fix B) when they become configurable. - SpinPolicy.h + QEC_DECODING_SERVER_SPIN_US: both spin-then-block sites died with the cross-thread waits they optimized. - LoopbackTransceiver, the DecodingServer recv loop, and install_dispatch_sink: no implementors/users remain. DecodingServer shrinks to the device_graph lifecycle shell; run() parks until stop(). Instrumentation: HopStats.h drops the cross-thread hop machinery and becomes a stack-resident StageScope recording parse/decode/respond stages per request. QEC_DECODING_SERVER_HOP_STATS and the `total` metric keep their semantics (handler entry -> exit) so reports remain comparable; the volume-completing enqueue now shows its decode in stage_decode (previously hidden between the enqueue ACK and the get_corrections wait). QEC_PIN_RECV/QEC_PIN_WORKER die with their threads. Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>

Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>

vedika-saravanan · 2026-08-06T21:52:50Z

+                           static_cast<int32_t>(RpcStatus::BAD_REQUEST));
+      return;
+    }
+    DecodingSession *session = registry->find(decoder_id);


I may be overlooking an existing topology guarantee, but is explicit enforcement of the current one-ring-per-decoder assumption planned as a follow-up? The inline path now mutates DecodingSession directly, while the concurrent-entry check is debug-only. It would be helpful to document the current ownership constraint, or reject unsupported sharing clearly, as multi-ring support evolves.

vedika-saravanan · 2026-08-06T22:26:55Z

Thanks Chuck Overall, this is a strong simplification of the host dispatch path. I left one small question about documenting the current one-ring-per-decoder ownership assumption as multi-ring support evolves.

melody-ren

Thanks Chuck! Looks good to me. Just some minor nits

melody-ren · 2026-08-07T16:43:34Z

 // ---------------------------------------------------------------------------

 // The schema entries below register under the SAME function IDs the handlers
 // and CqrTransceiver route on: the kXFunctionId constants from


Some stale comments here: This PR removes CqrTransceiver

melody-ren · 2026-08-07T16:44:28Z


 # ---------------------------------------------------------------------------
 # CQR host-dispatch plugin (only when CUDAQ device_call headers are available):
 # CqrTransceiver bridges DeviceCallService handler callbacks to ITransceiver,


Stale comment about CqrTransceiver

melody-ren · 2026-08-07T16:51:31Z

+/// Write a header-only RPCResponse (no result payload) into \p tx_slot.
+/// The magic is release-stored LAST so the CUDAQ runtime sees a complete
+/// response before observing the magic word.
+inline void write_response(void *tx_slot, uint32_t request_id,


You probably already have a plan for this one, but here's a function that does nearly the same thing:

cudaqx/libs/qec/lib/realtime/decoding-server-cqr/decoding_server_cqr.cpp

Line 108 in ec5fc99

static void write_error_response(const void *rx_slot, void *tx_slot,

I could be mistaken though and they might be serving different purposes

cketcham2333 added 2 commits August 6, 2026 00:25

clang format

49d13fe

Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>

cketcham2333 marked this pull request as ready for review August 6, 2026 14:33

cketcham2333 requested review from bmhowe23, melody-ren and vedika-saravanan August 6, 2026 14:33

vedika-saravanan reviewed Aug 6, 2026

View reviewed changes

melody-ren reviewed Aug 7, 2026

View reviewed changes

Navigation Menu

Uh oh!

[QEC] Decoding server: serve HOST_CALL requests inline; retire the worker threads, queues, and CqrTransceiver - #769

[QEC] Decoding server: serve HOST_CALL requests inline; retire the worker threads, queues, and CqrTransceiver#769
cketcham2333 wants to merge 2 commits into
NVIDIA:mainfrom
cketcham2333:decoding-server-worker-thread-remov

cketcham2333 commented Aug 6, 2026

Uh oh!

vedika-saravanan Aug 6, 2026

Uh oh!

vedika-saravanan commented Aug 6, 2026 •

edited

Loading

Uh oh!

melody-ren left a comment

Uh oh!

melody-ren Aug 7, 2026

Uh oh!

melody-ren Aug 7, 2026

Uh oh!

melody-ren Aug 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

cketcham2333 commented Aug 6, 2026

Summary

Measured latency (udp two-process rig, d3 surface code, pymatching, 500 shots)

What's new

What's deleted (~1,500 LOC)

Behavior notes

Uh oh!

vedika-saravanan Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

vedika-saravanan commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

melody-ren left a comment

Choose a reason for hiding this comment

Uh oh!

melody-ren Aug 7, 2026

Choose a reason for hiding this comment

Uh oh!

melody-ren Aug 7, 2026

Choose a reason for hiding this comment

Uh oh!

melody-ren Aug 7, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

vedika-saravanan commented Aug 6, 2026 •

edited

Loading