Add topology-safe live decoder reconfiguration

[vedika-saravanan]

Summary

Reimplements live decoding-server configuration on top of the current one-ring-per-decoder architecture and runtime bridge providers.

• Reloads the canonical YAML file on SIGHUP without rebuilding transport providers, endpoints, or rings.
• Permits host-decoder configuration changes only when decoder IDs, per-ID dispatch shapes, provider configuration, and device-graph entries remain unchanged.
• Drains in-flight work at the documented between-shots safe point before replacing affected host decoder sessions.
• Rejects malformed or topology-changing configurations while preserving the active configuration.
• Returns NOT_READY during reload and when construction of a replacement decoder fails; a later valid reload can recover the unavailable decoder.

Validation

• Added two-process tests for a host decoder swap, malformed YAML rollback, topology-change rejection, and construction-failure recovery.
• Added operator documentation for SIGHUP behavior, invariants, and failure semantics.
• clang-format --dry-run --Werror passed for all changed C++ files.
• git diff --check passed.

Full CUDA-Q build and test coverage is delegated to GitHub CI because the local environment does not contain CUDA/CUDA-Q.

[bmhowe23]

This looks like an accidental checkin that should be removed.

[melody-ren]

Thanks Vedika! I have some questions about the gpu roce path. See inline comments please~

[melody-ren]

#679 added some nice validation helpers for config. We can call config.validate_custom_args() here to validate it

[melody-ren]

Later in SessionRegistry.cpp, we check for negative/out of range decoder IDs and mixed transports, which is a bit too late since invalid config should be rejected here before reaching SessionRegistry. I think we can just add those checks here too

[melody-ren]

I understand why the ServerConfig is stripped for the decoders, but what happens if the new config has server setup (e.g, transport is different) than the one currently serving? I think we need to check the server config's content before stripping and compare that to the active server's setting and reject if the transport restart is needed (For GPU RoCE path, I think we do need to reject per the current scope)

[melody-ren]

Three questions about this:
(1) We removed HOLOLINK_GPU_ID. Not sure we should also remove these hololink env vars for two reasons: they should come from the config and we want to use HSB in names instead. @cketcham2333 ?

(2) The transport level config saved in ServerConfig is stripped before being sent to the factory, which then reads the settings from from_env. We have basically two pathways for configuring gpu_roce. This env might be useful still for internal testing but the primary route, in my opinion, should default to the config coming down the wire and we need some sort of reconciliation if the two disagree. I favor that we remove the env var path entirely though

(3) Related to another comment about transport might be different in the config: the gpu_roce related settings are not checked or reread on SIGHUP since ServerConfig is stripped

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[vedika-saravanan]

/ok to test 04b2d38