docs: rewrite README as concise landing page and add architecture diagrams [WIP]

docs/README.md

@@ -35,7 +35,55 @@ Complete documentation for madengine - AI model automation and distributed bench
 
 ## 🏗️ Architecture
 
-The architecture diagram (Orchestration, Infrastructure, and Launcher layers) is in the [main README](../README.md#-architecture). Summary:
+The CLI drives orchestrators that discover and build models, then hand off to a local or distributed execution target, which runs the model under the appropriate launcher and emits performance data for reporting. (Same diagram as the [main README](../README.md#-architecture).)
+
+```mermaid
+flowchart TB
+    subgraph CLI["CLI Layer — Typer + Rich"]
+        C1[discover]
+        C2[build]
+        C3[run]
+        C4[report]
+        C5[database]
+    end
+
+    subgraph ORC["Orchestration Layer"]
+        O1[DiscoverModels]
+        O2[BuildOrchestrator]
+        O3[RunOrchestrator]
+        MAN[(build_manifest.json)]
+    end
+
+    subgraph EXEC["Execution / Deployment Layer"]
+        E1[ContainerRunner<br/>local Docker]
+        E2[DeploymentFactory]
+        K8S[Kubernetes Jobs]
+        SLURM[SLURM Jobs]
+    end
+
+    subgraph LAUNCH["Launcher Layer"]
+        T[Train: torchrun · DeepSpeed<br/>Megatron-LM · TorchTitan · Primus]
+        I[Infer: vLLM · SGLang · SGLang Disagg]
+    end
+
+    OUT[(perf.csv / JSON)]
+
+    C1 --> O1
+    C2 --> O2
+    C3 --> O3
+    O2 --> MAN --> O3
+    O1 --> O2
+    O3 --> E1
+    O3 --> E2
+    E2 --> K8S
+    E2 --> SLURM
+    E1 --> LAUNCH
+    K8S --> LAUNCH
+    SLURM --> LAUNCH
+    LAUNCH --> OUT
+    OUT --> C4
+    OUT --> C5
+```
 
 1. **CLI Layer** - User interface with 5 commands (discover, build, run, report, database)
 2. **Model Discovery** - Find and validate models from MAD package

docs/batch-build.md

@@ -224,11 +224,15 @@ Creates `build_manifest.json` with:
     }
   },
   "built_models": {...},
+  "context": {...},
+  "credentials_required": {...},
   "deployment_config": {...},
   "summary": {...}
 }
 ```
 
+> `deployment_config` is only written when `--additional-context` resolves to a non-local deployment (e.g. `slurm`, `k8s`/`kubernetes`, `distributed`, `vllm`, or non-empty `env_vars`). Plain local builds omit this key entirely.
+
 ## Best Practices
 
 1. **Version Control**: Keep batch manifests in version control for reproducibility

docs/cli-reference.md

@@ -233,10 +233,10 @@ madengine run [OPTIONS]
 | `--manifest-output` | | TEXT | `build_manifest.json` | Output file for build manifest (full workflow) |
 | `--summary-output` | `-s` | TEXT | `None` | Output file for summary JSON |
 | `--live-output` | `-l` | FLAG | `False` | Print output in real-time |
-| `--output` | `-o` | TEXT | `perf_entry.csv` | Performance output file |
+| `--output` | `-o` | TEXT | `perf.csv` | Performance output file |
 | `--ignore-deprecated` | | FLAG | `False` | Force run deprecated models |
 | `--data-config` | | TEXT | `data.json` | Custom data configuration file |
-| `--tools-config` | | TEXT | `tools.json` | Custom tools JSON configuration |
+| `--tools-config` | | TEXT | `./scripts/common/tools.json` | Custom tools JSON configuration |
 | `--sys-env-details` | | FLAG | `True` | Generate system config env details |
 | `--force-mirror-local` | | TEXT | `None` | Path to force local data mirroring |
 | `--disable-skip-gpu-arch` | | FLAG | `False` | Disable skipping models based on GPU architecture |
@@ -356,7 +356,7 @@ madengine run --tags model \
 
 **Performance Output:**
 
-Results are saved to CSV file (default: `perf_entry.csv`) with metrics including:
+Results are saved to CSV file (default: `perf.csv`) with metrics including:
 - Execution time
 - GPU utilization
 - Memory usage
@@ -384,20 +384,20 @@ madengine report to-html [OPTIONS]
 
 | Option | Short | Type | Required | Description |
 |--------|-------|------|----------|-------------|
-| `--csv-file` | | TEXT | **Yes** | Path to the CSV file to convert |
+| `--csv-file-path` | | TEXT | **Yes** | Path to the CSV file to convert |
 | `--verbose` | `-v` | FLAG | No | Enable verbose logging |
 
 **Examples:**
 
 ```bash
 # Convert CSV to HTML
-madengine report to-html --csv-file perf_entry.csv
+madengine report to-html --csv-file-path perf_entry.csv
 
 # With custom CSV file
-madengine report to-html --csv-file results/perf_mi300.csv
+madengine report to-html --csv-file-path results/perf_mi300.csv
 
 # Verbose output
-madengine report to-html --csv-file perf.csv --verbose
+madengine report to-html --csv-file-path perf.csv --verbose
 ```
 
 **Output:** Creates `{filename}.html` in the same directory as the CSV file.
@@ -444,7 +444,7 @@ madengine report to-email --directory ./results --verbose
 
 ### `database` - Upload to MongoDB
 
-Upload CSV performance data to MongoDB database.
+Upload CSV or JSON performance data to MongoDB (format is auto-detected).
 
 **Usage:**
 
@@ -456,32 +456,30 @@ madengine database [OPTIONS]
 
 | Option | Short | Type | Default | Required | Description |
 |--------|-------|------|---------|----------|-------------|
-| `--csv-file` | | TEXT | `perf_entry.csv` | No | Path to the CSV file to upload |
-| `--database-name` | `--db` | TEXT | `None` | **Yes** | Name of the MongoDB database |
-| `--collection-name` | `--collection` | TEXT | `None` | **Yes** | Name of the MongoDB collection |
-| `--verbose` | `-v` | FLAG | `False` | No | Enable verbose logging |
+| `--file` | `-f` | TEXT | `None` | **Yes** | Path to file (CSV or JSON, auto-detected) |
+| `--database` | `--db` | TEXT | `None` | **Yes** | MongoDB database name |
+| `--collection` | `-c` | TEXT | `None` | **Yes** | MongoDB collection name |
+| `--unique-key` | `-k` | TEXT | `None` | No | Unique field(s) for deduplication (comma-separated, auto-detected if not specified) |
+| `--batch-size` | | INT | `1000` | No | Batch size for bulk operations |
+| `--no-upsert` | | FLAG | `False` | No | Insert only (don't update existing documents) |
+| `--no-index` | | FLAG | `False` | No | Skip automatic index creation |
+| `--dry-run` | | FLAG | `False` | No | Validate without uploading |
+| `--verbose` | `-v` | FLAG | `False` | No | Verbose output |
 
 **Examples:**
 
 ```bash
-# Upload to MongoDB
-madengine database \
-  --csv-file perf_entry.csv \
-  --database-name mydb \
-  --collection-name results
+# Upload JSON with auto-detection
+madengine database -f perf_entry_super.json --db mydb -c perf_super
 
-# Short option names
-madengine database \
-  --csv-file perf.csv \
-  --db test \
-  --collection perf_data
+# Upload CSV with custom unique key
+madengine database -f perf.csv --db test -c results -k model,timestamp
+
+# Dry run to validate
+madengine database -f data.json --db test -c data --dry-run
 
 # With verbose output
-madengine database \
-  --csv-file perf.csv \
-  --db mydb \
-  --collection results \
-  --verbose
+madengine database -f perf.csv --db mydb -c results --verbose
 ```
 
 **Environment Variables:**
@@ -490,10 +488,12 @@ MongoDB connection details are read from environment variables:
 
 | Variable | Description | Example |
 |----------|-------------|---------|
-| `MONGO_HOST` | MongoDB host address | `localhost` or `mongodb.example.com` |
-| `MONGO_PORT` | MongoDB port | `27017` |
+| `MONGO_HOST` | MongoDB host address (default: `localhost`) | `localhost` or `mongodb.example.com` |
+| `MONGO_PORT` | MongoDB port (default: `27017`) | `27017` |
 | `MONGO_USER` | MongoDB username | `admin` |
 | `MONGO_PASSWORD` | MongoDB password | `secretpassword` |
+| `MONGO_AUTH_SOURCE` | MongoDB authentication database (default: `admin`) | `admin` |
+| `MONGO_TIMEOUT_MS` | Server selection timeout in milliseconds (default: `5000`) | `5000` |
 
 **Example Setup:**
 
@@ -504,7 +504,7 @@ export MONGO_USER=myuser
 export MONGO_PASSWORD=mypassword
 
 madengine database \
-  --csv-file perf_entry.csv \
+  --file perf_entry.csv \
   --db performance_db \
   --collection model_runs
 ```

docs/configuration.md

@@ -21,8 +21,7 @@ madengine run --tags model --additional-context-file config.json
 ```json
 {
   "gpu_vendor": "AMD",
-  "guest_os": "UBUNTU",
-  "timeout_multiplier": 2.0
+  "guest_os": "UBUNTU"
 }
 ```
 
@@ -179,13 +178,13 @@ Unknown `rocenv_mode` values fall back to `lite` with a warning.
 **Overrides** (recommended for CI):
 
 - **Additional context (host):** top-level `"MAD_ROCM_PATH": "/path/to/host/rocm"` — controls where madengine looks for host GPU tools (`rocminfo`, `amd-smi`, etc.).
-- **Additional context (container):** `"docker_env_vars": { "MAD_ROCM_PATH": "/path/inside/image" }` — sets the in-container `ROCM_PATH` for Docker runs. If omitted, at `run` time madengine uses the image OCI `Env` (`ROCM_PATH` / `ROCM_HOME`) if present, then an in-container probe, then defaults to `/opt/rocm`. The host-resolved path is **not** mirrored into the container.
+- **Additional context (container):** `"docker_env_vars": { "ROCM_PATH": "/path/inside/image" }` — sets the in-container `ROCM_PATH` for Docker runs. If omitted, at `run` time madengine uses the image OCI `Env` (`ROCM_PATH` / `ROCM_HOME`) if present, then an in-container probe, then defaults to `/opt/rocm`. The host-resolved path is **not** mirrored into the container.
 
 These two keys are independent, allowing host and container to use different ROCm installations without confusion.
 
 Precedence (host): top-level `MAD_ROCM_PATH` → auto-detect (unless disabled) → `ROCM_PATH` → `/opt/rocm`.
 
-Precedence (container, **local Docker `run`**, **AMD**): `docker_env_vars.MAD_ROCM_PATH` (maps to `ROCM_PATH` for the workload) or explicit `ROCM_PATH` in `docker_env_vars` → image OCI `Env` (`ROCM_PATH` / `ROCM_HOME`) → in-image probe → default `/opt/rocm` with a warning. Implemented in `ContainerRunner.run_container` after the run image is resolved.
+Precedence (container, **local Docker `run`**, **AMD**): explicit `ROCM_PATH` in `docker_env_vars` → image OCI `Env` (`ROCM_PATH` / `ROCM_HOME`) → in-image probe → default `/opt/rocm` with a warning. Implemented in `ContainerRunner.run_container` after the run image is resolved.
 
 This applies to the run phase; build uses build-only context (no GPU detection) but still honors `MAD_ROCM_PATH` in context when set.
 
@@ -328,13 +327,15 @@ Format: Comma-separated list with hyphen ranges.
 
 ### Timeout Settings
 
+Set a per-model timeout (seconds) in `models.json`:
+
 ```json
 {
-  "timeout_multiplier": 2.0
+  "timeout": 7200
 }
 ```
 
-Or use command-line option:
+Or use the command-line option, which overrides the model's timeout:
 
 ```bash
 madengine run --tags model --timeout 7200
@@ -388,7 +389,6 @@ Automatically applies (see presets under `src/madengine/deployment/presets/k8s/`
     "memory_limit": "64Gi",
     "cpu": "16",
     "cpu_limit": "32",
-    "service_account": "madengine-sa",
     "image_pull_policy": "Always",
     "ttl_seconds_after_finished": null,
     "allow_privileged_profiling": null,
@@ -409,7 +409,6 @@ Automatically applies (see presets under `src/madengine/deployment/presets/k8s/`
 - `memory_limit` - Memory limit (default: 2× memory request)
 - `cpu` - CPU cores request (default: auto-scaled by GPU count)
 - `cpu_limit` - CPU cores limit (default: 2× CPU request)
-- `service_account` - Service account name
 - `image_pull_policy` - `Always`, `IfNotPresent`, or `Never`
 - `ttl_seconds_after_finished` - Optional Job TTL in seconds (auto-delete finished Job); `null` to omit
 - `allow_privileged_profiling` - `null` means enable elevated `securityContext` when tools/profiling are configured; `true`/`false` to force
@@ -471,7 +470,7 @@ Automatically applies (see presets under `src/madengine/deployment/presets/k8s/`
 - `partition` - SLURM partition name (required)
 - `account` - Billing account
 - `qos` - Quality of Service
-- `gpus_per_node` - GPUs per node (default: 1)
+- `gpus_per_node` - GPUs per node (default: 8)
 - `nodes` - Number of nodes (default: 1)
 - `nodelist` - Comma-separated node names to run on (e.g. `"node01,node02"`); when set, job is restricted to these nodes and automatic node health preflight is skipped
 - `reservation` - SLURM reservation name; forwarded to srun health/cleanup commands and SBATCH directives
@@ -558,14 +557,12 @@ See [Launchers Guide](launchers.md) for details.
     "launcher": "vllm",
     "nnodes": 2,
     "nproc_per_node": 4
-  },
-  "vllm": {
-    "tensor_parallel_size": 4,
-    "pipeline_parallel_size": 1
   }
 }
 ```
 
+`nproc_per_node` is exported into the container as `VLLM_TENSOR_PARALLEL_SIZE`; there is no separate `vllm.*` config block for tensor/pipeline parallel sizing.
+
 ## Profiling Configuration
 
 ### Basic Profiling
@@ -656,14 +653,11 @@ Configure in `data.json` (MAD package root):
 
 ```json
 {
-  "data_sources": {
-    "model_data": {
-      "nas": {"path": "/home/datum"},
-      "minio": {"path": "s3://datasets/datum"},
-      "aws": {"path": "s3://datasets/datum"}
-    }
-  },
-  "mirrorlocal": "/tmp/local_mirror"
+  "model_data": {
+    "nas": {"path": "/home/datum", "mirrorlocal": "/tmp/local_mirror"},
+    "minio": {"path": "s3://datasets/datum"},
+    "aws": {"path": "s3://datasets/datum"}
+  }
 }
 ```
 
@@ -678,13 +672,13 @@ Configure in `credential.json` (MAD package root):
     "password": "your_token",
     "repository": "myorg"
   },
-  "AMD_GITHUB": {
+  "PUBLIC_GITHUB_ROCM_KEY": {
     "username": "github_username",
-    "password": "github_token"
+    "token": "github_token"
   },
   "MAD_AWS_S3": {
-    "username": "aws_access_key",
-    "password": "aws_secret_key"
+    "USERNAME": "aws_access_key",
+    "PASSWORD": "aws_secret_key"
   }
 }
 ```

docs/contributing.md

@@ -61,7 +61,7 @@ pytest
 pytest --cov=src/madengine --cov-report=html
 
 # Run specific test file
-pytest tests/test_cli.py
+pytest tests/unit/test_cli.py
 
 # Run tests matching pattern
 pytest -k "test_build"