Declarative decoder parameter schemas: pluggable realtime decoder configuration

[bmhowe23]

Declarative decoder parameter schemas: true third-party support for realtime decoding

TL;DR

config.cpp was a closed shop: every decoder's YAML parameters were hard-coded into
typed structs, a std::variant, and per-type LLVM YAML traits inside the core library.
A third-party decoder could implement the decoder interface from its own plugin, but
could never be configured through the realtime YAML without patching and rebuilding
CUDA-Q QEC.

This PR replaces all of that with one declarative schema per decoder, registered by the
same shared library that registers the decoder itself. The framework does the rest —
parsing, emission, validation, defaulting, Python conversion, and standalone JSON Schema
export are all generic. The old typed Python config classes remain available as
deprecated compatibility shims, and the pre-schema typed-config test suite runs against
them essentially unmodified. Net size: +3.9k/−2.2k lines, and the additions are
mostly tests, docs, and the deprecation layer; the core config machinery is smaller than
what it replaces.

Headline benefits

🔌 True third-party realtime decoders. A plugin makes itself fully YAML-configurable
with one registration in its own .so — zero framework changes, zero rebuilds:

register_decoder_schema({"my_decoder", {
    {"strength", k::f64},
    {"mode",     k::string, /*required=*/true},
}});

Proven end-to-end: a demo decoder was built against a sandbox cmake --install tree
using only installed headers, dropped into decoder-plugins/, and ran realtime decoding
with its own YAML parameters — no cudaqx source edits.

🧰 Maintainability: each parameter exists in exactly one place. Previously, adding
one decoder parameter meant touching ~6 places (struct field, variant traits, YAML
mapping, to_heterogeneous_map, Python binding, docs). Now it is one line in the
decoder's own schema. The core gets rid of ~900 lines of per-decoder ladders, and the
YAML walker, validator, and JSON exporter never change when decoders are added.

🐍 Usability: plain dicts and maps, with introspection. No more per-decoder config
classes to learn:

config.decoder_custom_args = {"error_rate_vec": rates, "merge_strategy": "smallest_weight"}
qec.decoder_param_schema("pymatching")   # discover any decoder's parameters at runtime

Values are converted to the schema's declared types (Python ints are fine for float
parameters, negative ints for int32 parameters), and mistakes fail with messages like
Parameter 'bp_seed' of 'nv-qldpc-decoder' expects a 32-bit int value.

✅ Validability, three ways.

• Parse time: unknown keys and missing required keys are rejected by the framework —
  decoders never write those checks.

• Programmatic: config.validate_custom_args() applies the same checks to dict/map-built
  configs before use; decoders can attach cross-field hooks (e.g. sliding_window rejects
  step_size > window_size at config time instead of deep in the constructor).

• Offline: qec.decoder_config_json_schema() exports a standard JSON Schema
  (draft 2020-12) built from whatever plugins are loaded, so CI and editors can lint
  config files without loading the library at all:

  python3 -c "import cudaq_qec; print(cudaq_qec.decoder_config_json_schema())" > schema.json
  check-jsonschema --schemafile schema.json my_config.yaml

🔁 Behavior preserved. YAML round-trips byte-match the old output; trt's default
global_decoder_params materialization, sliding_window's inner-decoder handling, and
configure_decoders status-code semantics are unchanged and covered by tests (31 C++ /
93 Python in the touched suites, all green).

Compatibility

Python: backward compatible, with deprecation warnings. The typed config classes
(nv_qldpc_decoder_config, trt_decoder_config, sliding_window_config,
pymatching_config, chromobius_config, etc.) remain available as pure-Python,
dict-backed shims (cudaq_qec/_compat.py) that emit a DeprecationWarning on
construction. Existing code works unchanged:

cfg = qec.nv_qldpc_decoder_config()   # DeprecationWarning
cfg.max_iterations = 50
dc.set_decoder_custom_args(cfg)       # same YAML as the dict form, byte-for-byte

The shims reproduce the old semantics: optional fields (unset reads as None, only set
fields are emitted), TypeError on wrongly-typed assignments, sliding_window's typed
inner-params fields collapsing into inner_decoder_params, trt's default
global_decoder_params materialization, and nested typed configs throughout. A shim
carries its own schema name, so unlike the dict path it converts correctly even before
decoder_config.type is assigned.

Proof: the pre-schema typed-config test suite is restored verbatim in
test_decoding_config_deprecated.py and passes against the shims — the only edits are
four tests whose final asserts read configs back (the one intentional API change, see
below), each marked inline, plus one smoke test that used no typed config and lives on
in the dict-based suite. The file notes it should be deleted together with _compat.py
when the deprecation period ends.

What does break:

• C++: the typed config structs, the std::variant decoder_custom_args, and their
  YAML traits are removed with no shim. C++ callers build a cudaqx::heterogeneous_map
  (the form every decoder constructor already consumes) and assign it directly.
• Python reads: decoder_config.decoder_custom_args now always returns a plain dict,
  never a typed object — code that read typed fields off a parsed config
  (dc.decoder_custom_args.max_iterations) must index the dict instead.
• The deprecated classes expose exactly the old attribute surface; parameters added
  after the removal (e.g. sliding_window's num_boundary_syndromes) are dict-only.

One structural consequence to be aware of: a decoder's YAML section is only parseable when
its plugin (and therefore its schema) is loaded. Hosts that parse-and-forward configs for
decoders they don't have locally need that decoder's plugin present. The nv-qldpc-decoder
schema is temporarily hosted in-tree until the proprietary plugin registers it itself.

Follow-ups (out of scope here)

• Move nv-qldpc-decoder / srelay_bp schema registration into the proprietary plugin.
• sparse_binary_matrix parameter kind with scipy pass-through (parked on a separate
  branch for its own PR).
• Optionally constrain discriminated sections (e.g. global_decoder) to decoder-role
  schemas rather than any registered name.
• Remove the Python deprecation shims (and their test file) after one release cycle.

🤖 Generated with Claude Code

[melody-ren]

Thanks Ben! Finally got rid of that ever changing config.cpp way of defining config. Good riddance.

Most likely not in scope for this PR, I think we still need a pre-decode job validator in addition to the schema checker such that we won't need live decoders to verify the config. That will be a bigger change and touching decoders too.

Regarding round-trip and config generation/verification: I'm not sure if the intended behaviour is to have a single source of truth for the config. Right now there are several paths for the config to reach the decoders. It can go through the realtime parsing, programmatic configure_decoders(), offline get_decoder (I might even be missing some). These paths do not share the same validation or normalization and might lead to a hole down the road (lesson learned from patching the pinning hardware PR). If unification is desired, it can go into a follow up though.

This is also related to round-trip. For example, a programmatic TRT config containing only global_decoder: pymatching does not contain global_decoder_params when first serialized, but after YAML round-trip it gains global_decoder_params: {}, so the objects and emitted YAML are unequal. The realtime decoder behaviour is still preserved because the constructor-facing path materializes the empty params before TRT construction; the difference is in config serialization, and the old typed YAML path included the empty section on the first emission. Similarly, a plugin field omitted from its registered schema can reach a local decoder unchanged but be dropped when the same configuration is serialized for a remote target. Some of the behaviour might be leftover from before and this PR or a follow up might be a good spot to clean up those "slightly different depending how you use it" behaviour.

[bmhowe23]

Most likely not in scope for this PR, I think we still need a pre-decode job validator in addition to the schema checker such that we won't need live decoders to verify the config. That will be a bigger change and touching decoders too.

Agreed that a full pre-decode job validator is out of scope here, but a chunk of it already works without live decoders:

• Cross-field checks with no decoder instantiation: each schema can attach a validate hook that runs at parse time and from validate_custom_args() - sliding_window uses it today (step_size <= window_size, non-empty error_rate_vec), and any plugin can register one from its own .so. Nothing is constructed to run these.
• Value-kind checking (added on this branch after your comment): validate_custom_args() now verifies every value is readable as its declared kind, so a config that validates is guaranteed to serialize - again with no decoder in the loop.
• Offline tooling: One could use decoder_config_json_schema() to generate a JSON schema which can be used to validate any YAML file against that schema. Granted, it doesn't have the validate hook, but it does cover some kind of document-style validation that requires no special hardware and no special CUDA-Q QEC libraries.

What genuinely needs the bigger change you're describing is validation that requires runtime context - H/O/D dimensions vs. block_size/syndrome_size vs. parameter values (e.g. error_rate_vec length), resource/capability checks, etc.
Today those live in the decoder constructors, so a pre-decode validator would need decoders to expose something like a static validate(config) alongside construction. That touches the decoder API and every implementation, so I would indeed rather save that for later.

Regarding round-trip and config generation/verification: I'm not sure if the intended behaviour is to have a single source of truth for the config. Right now there are several paths for the config to reach the decoders. It can go through the realtime parsing, programmatic configure_decoders(), offline get_decoder (I might even be missing some). These paths do not share the same validation or normalization and might lead to a hole down the road (lesson learned from patching the pinning hardware PR). If unification is desired, it can go into a follow up though.

Partially fixed on this branch now: the realtime paths are unified. Both YAML parsing and programmatic configure_decoders() run the same normalization (unknown keys warned-and-dropped, defaults materialized, values checked against their schema kinds), and emission serializes exactly the map the decoder constructor receives - so your trt example now emits global_decoder_params: {} on first emission and round-trips stably.

The path that's still outside the fence is offline get_decoder(name, H, opts), which takes a raw options map and never consults the schema. Agreed that unifying it is a potential follow-up - it changes behavior for existing callers and existing decoders, so it deserves separate thought.

I'm not sure if the intended behaviour is to have a single source of truth for the config.

I don't think we can ever get all the way down to a single source other than the live decoders, if for no other reason than that some decoding configurations may require resources that are able to be satisfied on some systems but not others. (I.e. perhaps some decoding configurations require 100 SMs but some GPUs don't have that many.) Therefore, all the additional validation checks are "just gravy" in my opinion, meant to catch as many errors as they can, but they are ultimately just "best effort" validation.

This is also related to round-trip. For example, a programmatic TRT config containing only global_decoder: pymatching does not contain global_decoder_params when first serialized, but after YAML round-trip it gains global_decoder_params: {}, so the objects and emitted YAML are unequal. The realtime decoder behaviour is still preserved because the constructor-facing path materializes the empty params before TRT construction; the difference is in config serialization, and the old typed YAML path included the empty section on the first emission. Similarly, a plugin field omitted from its registered schema can reach a local decoder unchanged but be dropped when the same configuration is serialized for a remote target. Some of the behaviour might be leftover from before and this PR or a follow up might be a good spot to clean up those "slightly different depending how you use it" behaviour.

Both examples are fixed in the latest commits. Emission now serializes the same normalized map the constructor-facing path produces (defaults materialized, non-schema keys warned-and-dropped), so:

• the trt config emits global_decoder_params: {} on the first emission, matching the old typed path, and emitted YAML is byte-stable across round trips;
• a non-schema key can no longer reach a local decoder while silently vanishing for a remote target - it's dropped (with a warning) from both, and validate_custom_args() hard-rejects it for callers who want an error instead.