feat(madengine): give the manifest a contract and multi-node runs an honest verdict

.gitignore

@@ -131,6 +131,11 @@ venv/
 docker/
 scripts/
 *.json
+# ... except the schemas madengine ships and loads at runtime. The blanket rule above is
+# for the model JSONs that land in a dev checkout; these are package data, and hatchling
+# picks files by VCS status, so an ignored schema would also be missing from the wheel.
+!src/madengine/schemas/*.json
+!src/madengine/deployment/presets/**/*.json
 .*_env/
 .vscode/
 

docs/cli-reference.md

@@ -522,9 +522,24 @@ madengine uses standard exit codes so scripts and CI (e.g. Jenkins) can detect s
 | `2` | `BUILD_FAILURE` | One or more image builds failed (e.g. Docker build error) |
 | `3` | `RUN_FAILURE` | One or more model executions failed |
 | `4` | `INVALID_ARGS` | Invalid command-line arguments or configuration |
+| `5` | `NO_METRIC` | The workload ran to completion but reported no performance metric |
 
 **Failure recording:** Pre-run failures (e.g. image pull, setup) and run failures are recorded in the performance table (`perf.csv`) with status `FAILURE`, so all attempted models appear in the CSV. The file is created automatically if missing.
 
+**`NO_METRIC` versus `RUN_FAILURE`:** a crashed workload and a workload that finished
+without producing a number are different problems — the first is a broken run, the second a
+broken contract between the model script and madengine — so they get different exit codes
+and the row in `perf.csv` reads `NO_METRIC` rather than `FAILURE`.
+
+**Multi-node verdicts:** every node records its exit code, and the submit node reads them
+all back. A metric outweighs an exit code, the same way it does for a single node: where a
+framework reports throughput from one rank only, the nodes that collected nothing exit
+non-zero on a perfectly healthy run, so a run that produced a metric is reported as a
+success with the node outcomes listed as warnings. With no metric anywhere, the node
+evidence is all there is and it decides: a node that exited non-zero or never reported at
+all gives `RUN_FAILURE` naming the node, while a clean set of exit codes and no metric
+gives `NO_METRIC`.
+
 **Example usage in scripts / CI:**
 
 ```bash

docs/configuration.md

@@ -252,6 +252,11 @@ This allows you to rebuild only changed models while maintaining references to e
 
 ### Environment Variables
 
+Host-side variables (`MODEL_DIR`, cache roots, `MAD_DOCKER_BUILDS`) can be collected in a
+shell env file that the manifest names via `deployment_config.env_file`; see
+[Environment file](deployment.md#environment-file-env_file). The variables below are the
+container's, and are set from the run configuration.
+
 Pass environment variables to containers:
 
 ```json

docs/deployment.md

@@ -252,6 +252,140 @@ The deployment target is automatically detected from the `slurm` key in the conf
 
 See [examples/slurm-configs/](../examples/slurm-configs/) for complete examples.
 
+### Environment file (`env_file`)
+
+A cluster run usually depends on a handful of host variables that say where things live —
+the model directory, the cache roots, the shared image store. Instead of requiring every
+operator to `source mad.env` in the shell that launches madengine, name the file in the
+build manifest and madengine loads it itself, on the submit node and on every worker:
+
+```json
+{
+  "deployment_config": {
+    "target": "slurm",
+    "env_file": "mad.env"
+  }
+}
+```
+
+A relative path is resolved against the manifest's directory, so a run directory holding
+the manifest and its `mad.env` side by side stays movable. The file is sourced with
+`bash`, exactly as `source mad.env` would, so `${VAR:-default}`, command substitution and
+conditionals all work — and, like the manifest that names it, the file is trusted input.
+Values in the file override what madengine inherited from the launching shell. madengine
+logs the names of the variables it applied, never their values, so an env file may carry
+tokens.
+
+A missing `env_file` is a fatal error at startup rather than a variable that silently
+resolves to the empty string mid-run.
+
+### Cluster profiles (`cluster_profile`)
+
+The SLURM presets describe the shape of a job — how many nodes, for how long. What they
+cannot describe is the cluster it lands on, so cluster facts used to be baked into the
+shape: the built-in multi-node preset sets `NCCL_IB_DISABLE=1` and
+`NCCL_SOCKET_IFNAME=eth0`, which puts a run on TCP over an interface that may not exist,
+on a fabric that may well be RoCE.
+
+A cluster profile carries those facts separately:
+
+```json
+{
+  "slurm": {
+    "nodes": 2,
+    "cluster_profile": ["roce-broadcom-thor2", "no-gpu-gres"]
+  }
+}
+```
+
+Profiles merge after the shape presets and before your own configuration, so a profile
+fixes a bad default and you can still override the profile. Several may be named because
+the facts are orthogonal: the fabric and whether the scheduler advertises GPU GRES are
+separate statements. A value of `null` in a profile removes a variable an earlier layer
+set — an interface name that does not exist on this cluster is worse than none at all.
+
+Bundled archetypes (`madengine/deployment/presets/slurm/clusters/`):
+
+| Profile | What it asserts |
+| --- | --- |
+| `roce-broadcom-thor2` | RoCEv2 over `bnxt_re` devices |
+| `roce-mellanox-cx7` | RoCEv2 over `mlx5` devices |
+| `infiniband-mellanox` | InfiniBand over `mlx5` devices |
+| `ethernet-tcp` | No RDMA fabric; collectives over TCP |
+| `no-gpu-gres` | `sbatch` rejects GPU directives (`GresTypes=(null)`) |
+
+Archetypes name hardware, never a particular cluster: anything site-specific — the
+partition, the account, the management interface — belongs in a site profile. Write one as
+a JSON file with the same shape, then name it by path, or by name after pointing
+`MADENGINE_CLUSTER_PROFILES` at the directory holding it. A site profile shadows a bundled
+one of the same name. `MADENGINE_CLUSTER_PROFILE` selects a profile for runs whose manifest
+names none.
+
+A profile that does not exist stops the run: a silent fallback to the wrong fabric costs
+more than a startup error.
+
+#### Submit nodes without GPUs
+
+A profile's `facts` block describes the compute nodes, which is what lets a login node with
+no GPUs, no ROCm and no `/dev/dri` prepare a job for nodes that have all three:
+
+```json
+{
+  "_description": "Our cluster.",
+  "facts": {
+    "gpu_vendor": "AMD",
+    "gpus_per_node": 8,
+    "gpu_architecture": "gfx942",
+    "gpu_product_name": "AMD Instinct MI300X",
+    "hip_version": "6.4"
+  }
+}
+```
+
+Without this, runtime context initialisation on a login node fails with `Unable to
+determine gpu vendor`. Facts are a fallback rather than an override: a node that can answer
+for itself still does, so on a heterogeneous partition the architecture recorded in results
+is the one that ran the work, not the one the profile expected.
+
+### Shared image store (`MAD_DOCKER_BUILDS`)
+
+For a multi-node run every worker needs the same image. Set `MAD_DOCKER_BUILDS` to a
+directory on shared storage (usually from the `env_file` above) and madengine saves the
+built image there once, then loads it from the tar on workers whose local image ID differs:
+
+```bash
+export MAD_DOCKER_BUILDS=/shared/MADstorage/docker_builds
+```
+
+Leaving it unset is supported but leaves image distribution to the operator: workers that
+do not already have the image cannot reconcile it, and the run fails on those nodes with a
+message saying so.
+
+### How a multi-node run is judged
+
+Every node writes its exit code to `<results_dir>/<model>/<job_id>/node_<rank>/node.status`
+before it copies anything else, so the outcome survives even when the artifacts do not. The
+submit node reads them all back, and weighs them the way a single-node run does — a metric
+outweighs an exit code:
+
+| What came back | Verdict |
+| --- | --- |
+| A metric, all nodes zero | success |
+| A metric, some node non-zero or silent | success, with the node outcomes as warnings |
+| No metric, some node non-zero or silent | `RUN_FAILURE`, naming the node |
+| No metric, all nodes zero | `NO_METRIC` |
+
+The warning row is not a technicality. Where a framework reports throughput from one rank
+only — Primus/Megatron reports from the last global rank — every other node finds no metric
+locally and exits non-zero on a completely healthy run. A verdict that trusted exit codes
+over results would fail every such run, so the exit codes are reported and the numbers are
+kept.
+
+For a workload where every rank really is expected to exit zero, set
+`slurm.kill_on_bad_exit` to tear the step down on the first bad exit instead of letting the
+survivors block on a peer that will never answer. It is off by default for the reason above:
+the node exiting non-zero may be the one whose peer holds the numbers.
+
 ### Multi-Node Training
 
 For distributed training across SLURM nodes:

pyproject.toml

@@ -25,6 +25,7 @@ dependencies = [
   "rich>=13.0.0",
   "click>=8.0.0",
   "jinja2>=3.0.0",
+  "jsonschema>=4.0.0",
   "pyyaml>=6.0",
   "kubernetes>=28.0.0",
   "pytest>=7.0",

src/madengine/cli/commands/run.py

@@ -280,16 +280,35 @@ def run(
             save_summary_with_feedback(execution_summary, summary_output, "Execution")
 
             failed_runs = len(execution_summary.get("failed_runs", []))
-            if failed_runs == 0:
+            no_metric_runs = len(execution_summary.get("no_metric_runs", []))
+            incomplete = execution_summary.get("incomplete")
+            for warning in execution_summary.get("warnings", []):
+                console.print(f"⚠️  [yellow]{warning}[/yellow]")
+            if incomplete:
                 console.print(
-                    "🎉 [bold green]All model executions completed successfully![/bold green]"
+                    f"💥 [bold red]Run did not complete and produced no metric: "
+                    f"{incomplete.get('reason', 'unknown reason')}[/bold red]"
                 )
-                raise typer.Exit(ExitCode.SUCCESS)
-            else:
+                raise typer.Exit(ExitCode.RUN_FAILURE)
+            if failed_runs:
                 console.print(
-                    f"💥 [bold red]Execution failed for {failed_runs} models[/bold red]"
+                    f"💥 [bold red]Execution failed for {failed_runs} "
+                    f"{'run' if failed_runs == 1 else 'runs'}[/bold red]"
                 )
                 raise typer.Exit(ExitCode.RUN_FAILURE)
+            if no_metric_runs:
+                # The workload ran to completion and produced nothing to measure. That is
+                # not a successful benchmark, and it is not the same failure as a crash:
+                # what broke is the contract between the model script and madengine.
+                console.print(
+                    f"📉 [bold yellow]{no_metric_runs} models ran but reported no "
+                    f"performance metric[/bold yellow]"
+                )
+                raise typer.Exit(ExitCode.NO_METRIC)
+            console.print(
+                "🎉 [bold green]All model executions completed successfully![/bold green]"
+            )
+            raise typer.Exit(ExitCode.SUCCESS)
 
         else:
             # MAD_CONTAINER_IMAGE handling is done in RunOrchestrator

src/madengine/cli/constants.py

@@ -17,6 +17,9 @@ class ExitCode(IntEnum):
     BUILD_FAILURE = 2
     RUN_FAILURE = 3
     INVALID_ARGS = 4
+    #: The workload finished but produced no performance metric. Distinct from
+    #: RUN_FAILURE so a caller can tell a crashed run from a broken result contract.
+    NO_METRIC = 5
 
 
 # Valid values for validation