Feature proposal: nflow doctor — automate the 5-step Verify Installation checklist

[Anilreddy2309]

Motivation

Looking at recent releases, air-gapped/offline cluster operation is clearly a priority right now — v1.1.1 ("air-gapped Slurm support, dual-mode runtime") and v1.1.2 ("air-gapped NeMo-RL v0.7.0 stack") both call it out explicitly, and nvflow/lib/runtime.py exists specifically to resolve Python interpreters differently depending on whether the container is airgapped or dev-mode.

INSTALL.md's "Verify Installation" (Step 6 of 6) — the thing every new cluster deployment is told to run before a real workflow — is currently five commands a user has to type and interpret by hand:

# 1. Test NeMo-Skills import
uv run python -c "from nemo_skills.pipeline.cli import generate; print('✅ OK')"

# 2. Check containers exist
ls -lh <PATH_TO_CONTAINERS>/*.sqsh

# 3. Test SSH to cluster (only if submitting from a local machine)
ssh -i <PATH_TO_SSH_KEY> <YOUR_USERNAME>@<YOUR_CLUSTER_LOGIN_NODE> "echo '✅ SSH OK'"

# 4. Test cluster config loads
uv run python -c "from omegaconf import OmegaConf; OmegaConf.load('cluster_configs/my_cluster.yaml'); print('✅ Config OK')"

# 5. List available stages
uv run nflow list-stages

For air-gapped deployments specifically, this is exactly where things go wrong silently: cluster_configs/containers.yaml requires six containers (nemo-rl, nemo-skills, vllm, vllm-grpo, nemo-gym, sglang); staging 5 of 6 doesn't surface as an error until a multi-day GRPO training job fails hours in, on a cluster with no internet to pull the missing image.

Proposal

A nflow doctor command that automates exactly this checklist — read-only, no side effects, no execution risk:

• NeMo-Skills importable
• All containers declared in the active cluster_configs/*.yaml present as .sqsh files
• Cluster config loads and parses
• SSH reachable (best-effort, skipped if not applicable)
• Stage registry populated (list-stages succeeds — also catches silent recipe-import failures, which is a separate existing failure mode nvflow.core.discovery already warns about on stderr)

Output as a pass/fail table in the same rich-table style the CLI already uses elsewhere (list-stages, validate).

Why I think this is worth doing

It's not a speculative "nice to have" — it automates a process the project has already written down by hand as mandatory, for the deployment mode currently getting the most release attention. It should reduce first-deployment support burden and catch a specific, expensive failure mode (partial container staging surfacing hours into a paid GPU job) earlier.

Happy to implement this if the approach sounds right — wanted to float the design first since it's a new command rather than a fix, in case there's a different direction preferred (e.g. folding these checks into nflow validate instead of a new subcommand, or a different check set).