diff --git a/README.md b/README.md index 9ff81e42..ed763456 100644 --- a/README.md +++ b/README.md @@ -9,45 +9,21 @@ [![Python](https://img.shields.io/badge/python-3.8%2B-blue.svg)](https://python.org) [![CI](https://img.shields.io/badge/CI-GitHub%20Actions-green.svg)](https://github.com/ROCm/madengine/actions) [![Code Style](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black) -[![Version](https://img.shields.io/badge/version-2.0-brightgreen.svg)](CHANGELOG.md) +[![Version](https://img.shields.io/badge/version-2.1.3-brightgreen.svg)](CHANGELOG.md) [![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE) > **AI model automation and benchmarking platform for local and distributed execution** -madengine is a modern CLI tool for running Large Language Models (LLMs) and Deep Learning models across local and distributed environments. Built for the [MAD (Model Automation and Dashboarding)](https://github.com/ROCm/MAD) ecosystem, it provides seamless execution from single GPUs to multi-node clusters. - -## πŸ“– Table of Contents - -- [Key Features](#-key-features) -- [Quick Start](#-quick-start) -- [Commands](#-commands) -- [Documentation](#-documentation) -- [Architecture](#-architecture) -- [Feature Matrix](#-feature-matrix) -- [Usage Examples](#-usage-examples) -- [Model Discovery](#-model-discovery) -- [Performance Profiling](#-performance-profiling) -- [Reporting and Database](#-reporting-and-database) -- [Installation](#-installation) -- [Tips & Best Practices](#-tips--best-practices) - - [Log error pattern scan](#log-error-pattern-scan) - - [Exit codes and CI](#exit-codes-and-ci) -- [Contributing](#-contributing) -- [License](#-license) -- [Links & Resources](#-links--resources) +madengine is a modern CLI tool for running Large Language Models (LLMs) and Deep Learning models across local and distributed environments. Built for the [MAD (Model Automation and Dashboarding)](https://github.com/ROCm/MAD) ecosystem, it provides seamless execution from single GPUs to multi-node clusters β€” with the same command working locally, on Kubernetes, and on SLURM. ## ✨ Key Features -- **πŸš€ Modern CLI** - Rich terminal output with Typer and Rich -- **🎯 Simple Deployment** - Run locally or deploy to Kubernetes/SLURM via configuration -- **πŸ”§ Distributed Launchers** - Full support for torchrun, DeepSpeed, Megatron-LM, TorchTitan, Primus, vLLM, SGLang -- **🐳 Container-Native** - Docker-based execution with GPU support (ROCm, CUDA) -- **πŸ“‚ ROCm Path** - Auto-detect **host** ROCm root (override with top-level `MAD_ROCM_PATH`); in-container `ROCM_PATH` is set independently via `docker_env_vars.MAD_ROCM_PATH` and resolved at Docker run (image OCI env + in-image probe, not host mirroring) β€” see [Configuration](docs/configuration.md#rocm-path-run-only) -- **πŸ“Š Performance Tools** - Integrated profiling with rocprof/rocprofv3, [rocm-trace-lite](https://github.com/sunway513/rocm-trace-lite) (RTL), rocblas, MIOpen, RCCL tracing -- **🎯 ROCprofv3 Profiles** - 8 pre-configured profiles for compute/memory/communication bottleneck analysis -- **πŸ” Environment Validation** - TheRock ROCm detection and validation tools -- **βš™οΈ Intelligent Defaults** - Minimal K8s configs with automatic preset application -- **πŸ“‹ Configurable log scan** - Optional `--additional-context` keys to disable or tune post-run log substring checks (see [Log error pattern scan](#log-error-pattern-scan)) +- **πŸš€ Modern CLI** β€” Rich terminal output with Typer and Rich +- **🎯 Simple Deployment** β€” Run locally or deploy to Kubernetes/SLURM by adding a config key; no code changes +- **πŸ”§ Distributed Launchers** β€” torchrun, DeepSpeed, Megatron-LM, TorchTitan, Primus, vLLM, SGLang +- **🐳 Container-Native** β€” Docker-based execution with GPU support (ROCm, CUDA) +- **πŸ“Š Performance Tools** β€” Integrated profiling with rocprof/rocprofv3, [rocm-trace-lite](https://github.com/sunway513/rocm-trace-lite), rocBLAS/MIOpen/RCCL tracing β†’ see [Profiling](docs/profiling.md) +- **βš™οΈ Intelligent Defaults** β€” Minimal configs auto-merged with presets; host/in-container ROCm path auto-detected β†’ see [Configuration](docs/configuration.md#rocm-path-run-only) ## πŸš€ Quick Start @@ -55,504 +31,186 @@ madengine is a modern CLI tool for running Large Language Models (LLMs) and Deep # Install madengine pip install git+https://github.com/ROCm/madengine.git -# Clone MAD package (required for models) +# Clone the MAD package (required for models) git clone https://github.com/ROCm/MAD.git && cd MAD # Discover available models madengine discover --tags dummy -# Run locally (full workflow: discover/build/run as configured by the model) +# Run locally (discover β†’ build β†’ run, as configured by the model) madengine run --tags dummy +``` + +> **Note:** For build operations `gpu_vendor` defaults to `AMD` and `guest_os` to `UBUNTU`. For non-AMD/Ubuntu environments, set them explicitly, e.g. `--additional-context '{"gpu_vendor": "NVIDIA", "guest_os": "CENTOS"}'`. + +**Results:** Performance data is written to `perf.csv` (and optionally `perf_entry.csv`), created automatically if missing. Failed runs are recorded with status `FAILURE` so every attempted model appears. See [Exit Codes](docs/cli-reference.md#exit-codes) for CI usage. If ROCm isn't auto-detected, set `MAD_ROCM_PATH` β€” see [Configuration](docs/configuration.md#rocm-path-run-only). -# Or with explicit configuration -madengine run --tags dummy \ - --additional-context '{"gpu_vendor": "AMD", "guest_os": "UBUNTU"}' +## πŸ—οΈ Architecture + +madengine is organized in layers: the CLI drives orchestrators that discover and build models, then hand off to a local or distributed execution target, which runs the model under the appropriate launcher and emits performance data for reporting. + +```mermaid +flowchart TB + subgraph CLI["CLI Layer β€” Typer + Rich"] + C1[discover] + C2[build] + C3[run] + C4[report] + C5[database] + end + + subgraph ORC["Orchestration Layer"] + O1[DiscoverModels] + O2[BuildOrchestrator] + O3[RunOrchestrator] + MAN[(build_manifest.json)] + end + + subgraph EXEC["Execution / Deployment Layer"] + E1[ContainerRunner
local Docker] + E2[DeploymentFactory] + K8S[Kubernetes Jobs] + SLURM[SLURM Jobs] + end + + subgraph LAUNCH["Launcher Layer"] + T[Train: torchrun Β· DeepSpeed
Megatron-LM Β· TorchTitan Β· Primus] + I[Infer: vLLM Β· SGLang Β· SGLang Disagg] + end + + OUT[(perf.csv / JSON)] + + C1 --> O1 + C2 --> O2 + C3 --> O3 + O2 --> MAN --> O3 + O1 --> O2 + O3 --> E1 + O3 --> E2 + E2 --> K8S + E2 --> SLURM + E1 --> LAUNCH + K8S --> LAUNCH + SLURM --> LAUNCH + LAUNCH --> OUT + OUT --> C4 + OUT --> C5 ``` -> **Note**: For build operations, `gpu_vendor` defaults to `AMD` and `guest_os` defaults to `UBUNTU` if not specified. For production deployments or non-AMD/Ubuntu environments, explicitly specify these values. +1. **CLI Layer** β€” five commands: `discover`, `build`, `run`, `report`, `database` +2. **Orchestration** β€” `DiscoverModels` finds models; `BuildOrchestrator` builds images and writes `build_manifest.json`; `RunOrchestrator` reads/triggers the build and infers the target +3. **Execution / Deployment** β€” local `ContainerRunner`, or `DeploymentFactory` β†’ Kubernetes / SLURM +4. **Launchers** β€” distributed training and inference frameworks +5. **Output & Post-Processing** β€” `perf.csv`/JSON results β†’ `report` (HTML/email) and `database` (MongoDB) -If auto-detection does not find your **host** ROCm root, set top-level `MAD_ROCM_PATH` in `--additional-context`. For a different ROCm root **inside the container**, set `docker_env_vars.MAD_ROCM_PATH` in additional context. If you omit it, madengine derives in-container `ROCM_PATH` when running Docker (from the image's baked-in env, then an in-container probe, then `/opt/rocm` β€” it does **not** copy the host path). You can also set `ROCM_PATH` / `MAD_AUTO_ROCM_PATH=0` for **host** behavior as documented in [docs/configuration.md](docs/configuration.md): +## πŸ”„ Workflow -```bash -# Override host ROCm root: -madengine run --tags dummy --additional-context '{"MAD_ROCM_PATH": "/path/to/rocm"}' -# or: export ROCM_PATH=/path/to/rocm && madengine run --tags dummy -# Override in-container ROCm root independently: -madengine run --tags dummy --additional-context '{"docker_env_vars": {"MAD_ROCM_PATH": "/path/in/container"}}' +The core pipeline is the same everywhere: discover models, build images once, run them against a target, then report. Build and run can be separated so images are built once (e.g. in CI) and reused across nodes. + +```mermaid +flowchart LR + D[discover
find models by tag] --> B[build
Docker images] + B --> M[(build_manifest.json)] + M --> R[run
infer target + execute] + R --> P[(perf.csv)] + P --> RP[report
HTML / email] + P --> DB[database
MongoDB] ``` -**Results:** Performance data is written to `perf.csv` (and optionally `perf_entry.csv`). The file is created automatically if missing. Failed runs (including pre-run setup failures) are recorded with status `FAILURE` so every attempted model appears in the table. See [Exit Codes](docs/cli-reference.md#exit-codes) for CI/script usage. +**Deployment target is inferred from the config** (Convention over Configuration) β€” no `deploy` flag needed: -## πŸ“‹ Commands +```mermaid +flowchart TD + A[additional_context] --> Q{which key?} + Q -->|k8s / kubernetes| K[Kubernetes deployment] + Q -->|slurm| S[SLURM deployment] + Q -->|neither| L[Local Docker execution] +``` -madengine provides five main commands for model automation and benchmarking: +## πŸ“‹ Commands | Command | Description | Use Case | |---------|-------------|----------| -| **[discover](#-model-discovery)** | Find available models | Model exploration and validation | -| **[build](#building-images)** | Build Docker images | Create containerized models | -| **[run](#-usage-examples)** | Execute models | Local and distributed execution | -| **[report](docs/cli-reference.md#report---generate-reports)** | Generate HTML reports | Convert CSV to viewable reports | -| **[database](docs/cli-reference.md#database---upload-to-mongodb)** | Upload to MongoDB | Store results in database | - -**Quick Start:** +| **[discover](docs/usage.md#model-discovery)** | Find available models | Model exploration and validation | +| **[build](docs/usage.md#build-workflow)** | Build Docker images | Create containerized models | +| **[run](docs/usage.md#run-workflow)** | Execute models | Local and distributed execution | +| **[report](docs/cli-reference.md#report---generate-reports)** | Generate HTML/email reports | Convert CSV to viewable reports | +| **[database](docs/cli-reference.md#database---upload-to-mongodb)** | Upload to MongoDB | Store results in a database | ```bash -# Discover models -madengine discover --tags dummy +madengine discover --tags dummy # Find models +madengine build --tags dummy # Build image (AMD/UBUNTU defaults) +madengine run --tags dummy # Run model +madengine report to-html --csv-file-path perf_entry.csv # Report +madengine database --file perf_entry.csv --db mydb --collection results # Upload +``` -# Build image (uses AMD/UBUNTU defaults) -madengine build --tags dummy +For all options and examples, see the **[CLI Reference](docs/cli-reference.md)**. -# Run model -madengine run --tags dummy +## πŸ’» Usage Examples -# For non-AMD/Ubuntu environments, specify explicitly: -# madengine build --tags dummy --additional-context '{"gpu_vendor": "NVIDIA", "guest_os": "CENTOS"}' +```bash +# Local, multi-GPU with torchrun (DDP/FSDP) +madengine run --tags model \ + --additional-context '{"docker_gpus": "0,1,2,3", + "distributed": {"launcher": "torchrun", "nproc_per_node": 4}}' -# Generate report -madengine report to-html --csv-file perf_entry.csv +# Kubernetes (minimal config, presets auto-applied) +madengine run --tags model \ + --additional-context '{"k8s": {"gpu_count": 2}}' -# Upload results -madengine database --csv-file perf_entry.csv --db mydb --collection results +# SLURM (build once, then deploy) +madengine build --tags model --registry gcr.io/myproject +madengine run --manifest-file build_manifest.json \ + --additional-context '{"slurm": {"partition": "gpu", "nodes": 4, "gpus_per_node": 8}, + "distributed": {"launcher": "torchtitan", "nnodes": 4, "nproc_per_node": 8}}' ``` -For detailed command options, see the **[CLI Command Reference](docs/cli-reference.md)**. +More local/K8s/SLURM/CI recipes: [Usage Guide](docs/usage.md) Β· [Configuration](docs/configuration.md) Β· [CLI Reference](docs/cli-reference.md). ## πŸ“š Documentation | Guide | Description | |-------|-------------| | [Installation](docs/installation.md) | Complete installation instructions | -| [Usage Guide](docs/usage.md) | Commands, workflows, and examples ([`--skip-model-run`](docs/usage.md#skip-model-run-after-build)) | +| [Usage Guide](docs/usage.md) | Commands, workflows, and examples | | **[CLI Reference](docs/cli-reference.md)** | **Detailed command options and examples** | +| [Configuration](docs/configuration.md) | Advanced options, ROCm path, log error scan | | [Deployment](docs/deployment.md) | Kubernetes and SLURM deployment | -| [Configuration](docs/configuration.md) | Advanced options; [run log error pattern scan](docs/configuration.md#run-phase-log-error-pattern-scan) | | [Batch Build](docs/batch-build.md) | Selective builds for CI/CD | -| [Launchers](docs/launchers.md) | Distributed training frameworks | +| [Launchers](docs/launchers.md) | Distributed frameworks + capability matrices | | [Profiling](docs/profiling.md) | Performance analysis tools | | [Contributing](docs/contributing.md) | How to contribute | -## πŸ—οΈ Architecture - -``` - β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” - β”‚ madengine CLI v2.0 (Typer + Rich) β”‚ - β”‚ discover β”‚ build β”‚ run β”‚ report β”‚ database β”‚ - β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ - β”‚ β”‚ β”‚ - β”‚ β”‚ β–Ό - β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ Orchestration Layer ───────────────────────────┐ - β”‚ β”‚ β”‚ Model Discovery (models.json / scripts/ get_models) β”‚ - β”‚ β”‚ β”‚ BuildOrchestrator Β· RunOrchestrator β”‚ - β”‚ └──│ | - β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ - β”‚ - β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ Infrastructure Layer ─────────────┐ - β”‚ β–Ό β–Ό β–Ό β”‚ - β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ - β”‚ β”‚ Local β”‚ β”‚ Kubernetes β”‚ β”‚ SLURM β”‚ β”‚ - β”‚ β”‚ Docker β”‚ β”‚ Jobs β”‚ β”‚ Jobs β”‚ β”‚ - β”‚ β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ - β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ - β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ - β–Ό - β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ Launcher Layer (Distribution) ──────────────────────┐ - β”‚ Train: torchrun Β· DeepSpeed Β· Megatron-LM Β· TorchTitan Β· Primus β”‚ - β”‚ Infer: vLLM Β· SGLang Β· SGLang Disagg β”‚ - β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ - β–Ό - β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” - β”‚ Performance (CSV/JSON) β”‚ - β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ - β”‚ - β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” - β–Ό β–Ό - β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” - β”‚ report β”‚ β”‚ database β”‚ - β”‚ to-html, to-email β”‚ β”‚ MongoDB upload β”‚ - β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ -``` - -**Component Flow:** - -1. **CLI Layer** - User interface with 5 commands (discover, build, run, report, database) -2. **Model Discovery** - Find and validate models from MAD package -3. **Orchestration** - BuildOrchestrator & RunOrchestrator manage workflows -4. **Execution Targets** - Local Docker, Kubernetes Jobs, or SLURM Jobs -5. **Distributed Launchers** - Training (torchrun, DeepSpeed, Megatron-LM, TorchTitan, Primus) and Inference (vLLM, SGLang) -6. **Performance Output** - CSV/JSON results with metrics -7. **Post-Processing** - Report generation (HTML/Email) and database upload (MongoDB) - -## 🎯 Feature Matrix - -### Supported Launchers & Infrastructure +## 🎯 Supported Launchers | Launcher | Local | Kubernetes | SLURM | Type | Key Features | |----------|-------|-----------|-------|------|--------------| | **torchrun** | βœ… | βœ… | βœ… | Training | PyTorch DDP/FSDP, elastic training | | **DeepSpeed** | βœ… | βœ… | βœ… | Training | ZeRO optimization, pipeline parallelism | | **Megatron-LM** | βœ… | βœ… | βœ… | Training | Tensor+Pipeline parallel, large transformers | -| **TorchTitan** | βœ… | βœ… | βœ… | Training | FSDP2+TP+PP+CP, Llama 3.1 (8B-405B) | -| **Primus** | βœ… | βœ… | βœ… | Training | Megatron / TorchTitan / MaxText via Primus YAML; `distributed.primus` | +| **TorchTitan** | βœ… | βœ… | βœ… | Training | FSDP2+TP+PP+CP, Llama 3.1 (8B–405B) | +| **Primus** | βœ… | βœ… | βœ… | Training | Megatron / TorchTitan / MaxText via Primus YAML | | **vLLM** | βœ… | βœ… | βœ… | Inference | v1 engine, PagedAttention, Ray cluster | | **SGLang** | βœ… | βœ… | βœ… | Inference | RadixAttention, structured generation | | **SGLang Disagg** | ❌ | βœ… | βœ… | Inference | Disaggregated prefill/decode, Mooncake, 3+ nodes | -**Note:** All launchers support single-GPU, multi-GPU (single node), and multi-node (where infrastructure allows). See [Launchers Guide](docs/launchers.md) for details. - -### Parallelism Capabilities - -| Launcher | Tensor Parallel (TP) | Pipeline Parallel (PP) | Data Parallel (DP) | Context Parallel (CP) | FSDP/ZeRO | Expert Parallel (EP) | Primary Use Case | -|----------|----------------------|------------------------|--------------------|------------------------|-----------|----------------------|------------------| -| **torchrun** | ❗Manual | ❌No | ❗Manual (DDP) | ❌No | ❗Manual (FSDP) | ❌No | General distributed training | -| **TorchTitan** | βœ…Auto | βœ…Auto | βœ…Auto (FSDP2) | ❗Manual | βœ…Auto (FSDP2) | ❌No | Large-scale LLM pre-training | -| **DeepSpeed** | ❗Manual | ❗Manual | βœ…Auto (ZeRO) | ❌No | βœ…Auto (ZeRO) | ❌No | Memory-efficient training | -| **Megatron-LM** | βœ…Auto | βœ…Auto | βœ…Implicit | βœ…Auto | ❌No | ❌No | Large transformer training | -| **Primus** | ❗Manual | ❗Manual | ❗Manual | ❗Manual | ❗Manual | ❌No | Unified pretrain (experiment YAML; backend-specific) | -| **vLLM** | βœ…Auto | SLURM: βœ…Auto (Multi) / K8s: ❗Disabled | βœ…Auto (Replicas) | ❌No | ❌No | ❗Manual | High-throughput inference | -| **SGLang** | βœ…Auto | SLURM: βœ…Auto (Multi) / K8s: ❗Disabled | ❗Limited | ❌No | ❌No | ❌No | Inference + structured gen | -| **SGLang PD Disagg** | βœ…Auto | ❌No | βœ…Role-based | ❌No | ❌No | ❌No | Optimized prefill/decode | +All launchers support single-GPU, multi-GPU, and multi-node (where infrastructure allows). See the [Launchers Guide](docs/launchers.md) for the full **parallelism** and **infrastructure** capability matrices. -**Legend:** βœ…Auto = supported and configured by madengine; ❗Manual = supported by launcher but requires user configuration; ❗Limited / ❗Disabled = launcher or platform limitation. See [Launchers Guide](docs/launchers.md) and [Configuration](docs/configuration.md) for details. +## πŸ“Š Profiling -### Infrastructure Capabilities - -| Feature | Local | Kubernetes | SLURM | -|---------|-------|-----------|-------| -| **Execution** | Docker containers | K8s Jobs | SLURM jobs | -| **Multi-Node** | ❌ | βœ… Indexed Jobs | βœ… Job arrays | -| **Resource Mgmt** | Manual | Declarative (YAML) | Batch scheduler | -| **Monitoring** | Docker logs | kubectl/dashboard | squeue/scontrol | -| **Auto-scaling** | ❌ | βœ… | ❌ | -| **Network** | Host | CNI plugin | InfiniBand/Ethernet | - -## πŸ’» Usage Examples - -### Local Execution - -```bash -# Single GPU -madengine run --tags dummy \ - --additional-context '{"gpu_vendor": "AMD", "guest_os": "UBUNTU"}' - -# Multi-GPU with torchrun (DDP/FSDP) -madengine run --tags model \ - --additional-context '{ - "gpu_vendor": "AMD", - "guest_os": "UBUNTU", - "docker_gpus": "0,1,2,3", - "distributed": { - "launcher": "torchrun", - "nproc_per_node": 4 - } - }' - -# With DeepSpeed (ZeRO optimization) -madengine run --tags model \ - --additional-context '{ - "gpu_vendor": "AMD", - "guest_os": "UBUNTU", - "docker_gpus": "all", - "distributed": { - "launcher": "deepspeed", - "nproc_per_node": 8 - } - }' -``` - -### Kubernetes Deployment +madengine ships integrated profiling for AMD ROCm β€” `rocprof`, eight pre-configured `rocprofv3` profiles (ROCm 7.0+), `rocm-trace-lite`, library tracing (rocBLAS/MIOpen/Tensile/RCCL), and power/VRAM monitors. Tools are stackable via `--additional-context '{"tools": [...]}'`. ```bash -# Minimal config (auto-defaults applied) -madengine run --tags model \ - --additional-context '{"k8s": {"gpu_count": 2}}' - -# Multi-node inference with vLLM -madengine run --tags model \ - --additional-context '{ - "k8s": { - "namespace": "ml-team", - "gpu_count": 8 - }, - "distributed": { - "launcher": "vllm", - "nnodes": 2, - "nproc_per_node": 4 - } - }' - -# SGLang with structured generation -madengine run --tags model \ - --additional-context '{ - "k8s": {"gpu_count": 4}, - "distributed": { - "launcher": "sglang", - "nproc_per_node": 4 - } - }' -``` - -### SLURM Deployment - -```bash -# Build phase (local or CI) -madengine build --tags model \ - --registry gcr.io/myproject \ - --additional-context '{"gpu_vendor": "AMD", "guest_os": "UBUNTU"}' - -# Deploy phase (on SLURM login node) -madengine run --manifest-file build_manifest.json \ - --additional-context '{ - "slurm": { - "partition": "gpu", - "nodes": 4, - "gpus_per_node": 8, - "time": "24:00:00" - }, - "distributed": { - "launcher": "torchtitan", - "nnodes": 4, - "nproc_per_node": 8 - } - }' -``` - -To run on **specific nodes**, set `nodelist` (comma-separated node names). When set, the job is restricted to those nodes and automatic node health preflight is skipped. Example: `"slurm": { "nodelist": "node01,node02", "nodes": 2, ... }`. See [Configuration](docs/configuration.md#slurm-deployment) and [examples/slurm-configs/basic/03-multi-node-basic-nodelist.json](examples/slurm-configs/basic/03-multi-node-basic-nodelist.json). - -### Common Workflows - -**Development β†’ Testing β†’ Production:** - -```bash -# 1. Develop locally with single GPU -madengine run --tags model \ - --additional-context '{"gpu_vendor": "AMD", "guest_os": "UBUNTU"}' - -# 2. Test multi-GPU locally -madengine run --tags model \ - --additional-context '{ - "gpu_vendor": "AMD", - "guest_os": "UBUNTU", - "docker_gpus": "0,1", - "distributed": {"launcher": "torchrun", "nproc_per_node": 2} - }' - -# 3. Build and push to registry -madengine build --tags model \ - --registry docker.io/myorg \ - --additional-context '{"gpu_vendor": "AMD", "guest_os": "UBUNTU"}' - -# 4. Deploy to Kubernetes -madengine run --manifest-file build_manifest.json -``` - -**CI/CD Pipeline:** - -```bash -# Batch build (selective rebuilds) -madengine build --batch-manifest batch.json \ - --registry docker.io/myorg - -# Run tests -madengine run --manifest-file build_manifest.json \ - --additional-context '{"k8s": {"namespace": "ci-test"}}' - -# Generate and email reports -madengine report to-email --directory ./results --output ci_report.html - -# Upload to database -madengine database --csv-file perf_entry.csv \ - --database-name ci_db --collection-name test_results -``` - -See [Usage Guide](docs/usage.md), [Configuration Guide](docs/configuration.md), and [CLI Reference](docs/cli-reference.md) for more examples. - -### Building Images - -```bash -# Build single model -madengine build --tags dummy \ - --additional-context '{"gpu_vendor": "AMD", "guest_os": "UBUNTU"}' - -# Build with registry (for distributed deployment) -madengine build --tags model1 model2 \ - --registry localhost:5000 \ - --additional-context '{"gpu_vendor": "AMD", "guest_os": "UBUNTU"}' - -# Build for multiple GPU architectures -madengine build --tags model \ - --target-archs gfx908 gfx90a gfx942 \ - --registry gcr.io/myproject - -# Batch build mode (selective builds for CI/CD) -madengine build --batch-manifest examples/build-manifest/batch.json \ - --registry docker.io/myorg - -# Clean rebuild (no Docker cache) -madengine build --tags model --clean-docker-cache \ - --additional-context '{"gpu_vendor": "AMD", "guest_os": "UBUNTU"}' -``` - -**Output:** Creates `build_manifest.json` with built image names and configurations. - -See [Batch Build Guide](docs/batch-build.md) and examples in [`examples/build-manifest/`](examples/build-manifest/). - -## πŸ” Model Discovery - -madengine discovers models from the MAD package using three methods: - -```bash -# Root models (models.json) -madengine discover --tags pyt_huggingface_bert - -# Directory-specific (scripts/{dir}/models.json), scoped tag: {dir}/{model_or_tag} -madengine discover --tags dummy2/model1 - -# Dynamic with parameters (scripts/{dir}/get_models_json.py) -madengine discover --tags dummy3/model3:batch_size=512 +madengine run --tags model --additional-context '{"tools": [{"name": "rocprofv3_compute"}]}' ``` -## πŸ“Š Performance Profiling - -madengine includes integrated profiling tools for AMD ROCm: - -```bash -# GPU profiling with rocprof -madengine run --tags model \ - --additional-context '{ - "gpu_vendor": "AMD", - "guest_os": "UBUNTU", - "tools": [{"name": "rocprof"}] - }' - -# ROCprofv3 (ROCm 7.0+) - Advanced profiling with pre-configured profiles -madengine run --tags model \ - --additional-context '{"tools": [{"name": "rocprofv3_compute"}]}' - -# Use configuration files for complex setups -madengine run --tags model \ - --additional-context-file examples/profiling-configs/rocprofv3_multi_gpu.json - -# Library tracing (rocBLAS, MIOpen, Tensile, RCCL) -madengine run --tags model \ - --additional-context '{"tools": [{"name": "rocblas_trace"}]}' - -# rocm-trace-lite β€” lightweight kernel dispatch trace (SQLite; no rocprofiler-sdk) -# Requires outbound HTTPS to GitHub on first run unless the wheel is baked into the image -# (see docs/profiling.md). Do not combine with rocprof / rocprofv3_* on the same run. -madengine run --tags model \ - --additional-context '{"gpu_vendor": "AMD", "guest_os": "UBUNTU", "tools": [{"name": "rocm_trace_lite"}]}' - -# Power and VRAM monitoring -madengine run --tags model \ - --additional-context '{"tools": [ - {"name": "gpu_info_power_profiler"}, - {"name": "gpu_info_vram_profiler"} - ]}' - -# Multiple tools (stackable) -madengine run --tags model \ - --additional-context '{"tools": [ - {"name": "rocprofv3_memory"}, - {"name": "rocblas_trace"}, - {"name": "gpu_info_power_profiler"} - ]}' -``` - -**Available Tools:** - -| Tool | Purpose | Output | -|------|---------|--------| -| `rocprof` | GPU kernel profiling | Kernel timings, occupancy | -| `rocprofv3_compute` | Compute-bound analysis (ROCm 7.0+) | ALU metrics, wave execution | -| `rocprofv3_memory` | Memory-bound analysis (ROCm 7.0+) | Cache hits, bandwidth | -| `rocprofv3_communication` | Multi-GPU communication (ROCm 7.0+) | RCCL traces, inter-GPU transfers | -| `rocprofv3_lightweight` | Minimal overhead profiling (ROCm 7.0+) | HIP and kernel traces | -| `rocm_trace_lite` | RTL **`lite`** mode β€” kernel dispatch trace (HSA, SQLite/RPD-style); [`rtl trace --mode lite`](https://sunway513.github.io/rocm-trace-lite/quickstart.html) via `rtl_trace_wrapper.sh` | `rocm_trace_lite_output/trace.db` (and optional `trace.json.gz`, `trace_summary.txt`) | -| `rocm_trace_lite_default` | RTL **`default`** mode β€” broader dispatch coverage; higher overhead than `lite` (same outputs paths) | Same as `rocm_trace_lite` | -| `rocblas_trace` | rocBLAS library calls | Function calls, arguments | -| `miopen_trace` | MIOpen library calls | Conv/pooling operations | -| `tensile_trace` | Tensile GEMM library | Matrix multiply details | -| `rccl_trace` | RCCL collective ops | Communication patterns | -| `gpu_info_power_profiler` | GPU power consumption | Power usage over time | -| `gpu_info_vram_profiler` | GPU memory usage | VRAM utilization | -| `therock_check` | TheRock ROCm validation | Installation detection | - -**ROCprofv3 Profiles** (ROCm 7.0+): - -madengine provides 8 pre-configured ROCprofv3 profiles for different bottleneck scenarios: - -- `rocprofv3_compute` - Compute-bound workloads (transformers, dense ops) -- `rocprofv3_memory` - Memory-bound workloads (large batches, high-res) -- `rocprofv3_communication` - Multi-GPU distributed training -- `rocprofv3_full` - Comprehensive profiling (all metrics, high overhead) -- `rocprofv3_lightweight` - Minimal overhead (production-friendly) -- `rocprofv3_perfetto` - Perfetto UI compatible traces -- `rocprofv3_api_overhead` - API call timing analysis -- `rocprofv3_pc_sampling` - Kernel hotspot identification - -See [`examples/profiling-configs/`](examples/profiling-configs/) for ready-to-use configuration files. - -**rocm-trace-lite (`rocm_trace_lite` / `rocm_trace_lite_default`):** - -- madengine runs workloads under `scripts/common/tools/rtl_trace_wrapper.sh`, which invokes the `rtl` CLI (or `python3 -m rocm_trace_lite.cli`) with **`RTL_MODE=lite`** or **`RTL_MODE=default`** and writes traces under `rocm_trace_lite_output/`. -- The trace **pre-script** installs the package from a **[GitHub Release wheel](https://github.com/sunway513/rocm-trace-lite/releases)** (not PyPI). By default it uses a **pinned** `linux_x86_64` wheel for reproducible installs. Set **`ROCM_TRACE_LITE_FOLLOW_LATEST=1`** to resolve the latest wheel via the GitHub API, or **`ROCM_TRACE_LITE_WHEEL_URL`** to a direct `.whl` URL for air-gapped installs or non-x86_64 platforms. -- Choose **either** `rocm_trace_lite` **or** rocprof / `rocprofv3_*` for a given runβ€”not both. Details: [Profiling Guide](docs/profiling.md) (section *rocm-trace-lite (RTL)*). - -**TheRock Validation:** - -```bash -# Validate TheRock installation (AMD's pip-based ROCm) -madengine run --tags dummy_therock \ - --additional-context '{"tools": [{"name": "therock_check"}]}' -``` - -See [Profiling Guide](docs/profiling.md) for detailed usage and analysis. - -## πŸ“Š Reporting and Database - -### Generate Reports - -Convert performance CSV files to HTML reports: - -```bash -# Single CSV to HTML -madengine report to-html --csv-file perf_entry.csv - -# Consolidated email report (all CSVs in directory) -madengine report to-email --directory ./results --output summary.html -``` - -### Upload to Database - -Store performance results in MongoDB: - -```bash -# Set MongoDB connection -export MONGO_HOST=mongodb.example.com -export MONGO_PORT=27017 -export MONGO_USER=myuser -export MONGO_PASSWORD=mypassword - -# Upload CSV to MongoDB -madengine database --csv-file perf_entry.csv \ - --database-name performance_db \ - --collection-name model_runs -``` - -**Use Cases:** -- Track performance over time -- Compare results across different configurations -- Build performance dashboards -- Automated CI/CD reporting - -See [CLI Reference](docs/cli-reference.md) for complete options. +See the [Profiling Guide](docs/profiling.md) and ready-to-use configs in [`examples/profiling-configs/`](examples/profiling-configs/). ## πŸ“¦ Installation ```bash -# Install madengine (all dependencies, including Kubernetes support, are included) +# Install (all dependencies, including Kubernetes support, included) pip install git+https://github.com/ROCm/madengine.git # Development installation @@ -560,147 +218,41 @@ git clone https://github.com/ROCm/madengine.git cd madengine && pip install -e . ``` -See [Installation Guide](docs/installation.md) for detailed instructions. - -## πŸ’‘ Tips & Best Practices - -### General Usage - -- **Use configuration files** for complex setups instead of long command lines -- **Test locally first** with single GPU before scaling to multi-node -- **Enable verbose logging** (`--verbose`) when debugging issues -- **Use `--live-output`** for real-time monitoring of long-running operations - -### Log error pattern scan - -After a local Docker run, madengine can scan the captured **run log** for common failure substrings (for example `RuntimeError:`, `CUDA out of memory`, `Traceback`). That helps catch hard failures when exit codes are ambiguous, but some workloads log benign `RuntimeError:` text while tests still pass. - -- **Disable** the scan when another signal is authoritative (e.g. pytest/JUnit inside the image): set `"log_error_pattern_scan": false` in `--additional-context` or in the model entry in `models.json`. See [Configuration β€” Run phase: log error pattern scan](docs/configuration.md#run-phase-log-error-pattern-scan). -- **Extend exclusions** with `log_error_benign_patterns` (list of strings), or **replace** the default pattern list with `log_error_patterns` (non-empty list of strings) for advanced cases. - -### CI / Jenkins - -- **Exit codes:** The CLI uses fixed exit codes (`ExitCode` in `madengine.cli.constants`, e.g. `SUCCESS=0`, `RUN_FAILURE=3`, `INVALID_ARGS=4`). Pipelines should treat **non-zero** as failure; no log scraping is required for pass/fail. -- **Streaming:** In Jenkins, avoid redirecting stdout only to a file (`> file`) without `tee` if you want the console to update during the run. Prefer `... 2>&1 | tee madengine.run.log` with `bash -o pipefail` so the step exit code is still from `madengine`. -- **Unbuffered Python:** If output still appears in chunks, set `PYTHONUNBUFFERED=1` (or `python -u`) for the `madengine` process. +See the [Installation Guide](docs/installation.md) for details. -### Build & Deployment +## πŸ’‘ Tips & Troubleshooting -- **Separate build and run phases** for distributed deployments -- **Skip model script:** `madengine run --tags … --skip-model-run` starts the container and runs `pre_scripts`, but skips the model script. Combine with `--keep-alive` for a live container ready for manual exec. Ignored with a warning on SLURM/K8s. See [Usage β€” Skip model run after build](docs/usage.md#skip-model-run-after-build). -- **Use registries** for multi-node execution (K8s/SLURM) -- **Use batch build mode** for CI/CD to optimize build times -- **Specify `--target-archs`** when building for multiple GPU architectures +- **Test locally first** with a single GPU before scaling to multi-node; use config files for complex setups. +- **Debugging:** add `--verbose --live-output`; keep a container for inspection with `--keep-alive`. +- **CI:** the CLI uses fixed [exit codes](docs/cli-reference.md#exit-codes) (`0` success, `2` build failure, `3` run failure, `4` invalid args) β€” no log scraping needed. +- **Log error scan** can flag benign `RuntimeError:` text; disable or tune it via `log_error_pattern_scan` / `log_error_benign_patterns` β€” see [Configuration](docs/configuration.md#run-phase-log-error-pattern-scan). -### Performance - -- **Start with small timeouts** and increase as needed -- **Use profiling tools** to identify bottlenecks -- **Monitor GPU utilization** with `gpu_info_power_profiler` -- **Profile library calls** with rocBLAS/MIOpen tracing - -### Exit codes and CI - -madengine uses consistent exit codes for scripts and CI (e.g. Jenkins): `0` = success, `1` = general failure, `2` = build failure, `3` = one or more run failures, `4` = invalid arguments. Failed runs are still written to `perf.csv` with status `FAILURE`. See [CLI Reference β€” Exit Codes](docs/cli-reference.md#exit-codes) for the full table and examples. - -### Troubleshooting - -```bash -# Check model is available -madengine discover --tags your_model - -# Verbose output for debugging -madengine run --tags model --verbose --live-output - -# Keep container alive for inspection -madengine run --tags model --keep-alive - -# Clean rebuild if build fails -madengine build --tags model --clean-docker-cache --verbose -``` - -**ROCm not in /opt/rocm:** Set top-level `MAD_ROCM_PATH` in `--additional-context` for the **host**; for **in-container** paths, set `docker_env_vars.MAD_ROCM_PATH`, or let madengine resolve `ROCM_PATH` at run from the image and probe (see [Configuration](docs/configuration.md#rocm-path-run-only)). - -**Common Issues:** -- **False failures with profiling**: If models show FAILURE but have performance metrics, see [Profiling Troubleshooting](docs/profiling.md#false-failure-detection-with-rocprof) -- **False failures from `RuntimeError:` in logs**: If the workload logs expected exception text but tests pass, disable or tune the scan with `log_error_pattern_scan` / `log_error_benign_patterns` β€” see [Configuration](docs/configuration.md#run-phase-log-error-pattern-scan) -- **ROCProf log errors**: Messages like `E20251230` are informational logs, not errors (fixed in v2.0+) -- **Configuration errors**: Validate JSON with `python -m json.tool your-config.json` +More: [Usage β€” Troubleshooting](docs/usage.md#troubleshooting) Β· [Profiling β€” False failures](docs/profiling.md#false-failure-detection-with-rocprof). ## 🀝 Contributing -We welcome contributions! See [Contributing Guide](docs/contributing.md) for details. +Contributions are welcome! See the [Contributing Guide](docs/contributing.md). ```bash git clone https://github.com/ROCm/madengine.git -cd madengine -python3 -m venv venv && source venv/bin/activate +cd madengine && python3 -m venv venv && source venv/bin/activate pip install -e . - -# Run all tests pytest - -# Run specific test module -pytest tests/unit/test_error_handling.py -v - -# Run error pattern tests -pytest tests/unit/test_error_handling.py::TestErrorPatternMatching -v ``` ## πŸ“„ License -MIT License - see [LICENSE](LICENSE) file for details. +MIT License β€” see [LICENSE](LICENSE). ## πŸ”— Links & Resources -### Documentation -- **[CLI Reference](docs/cli-reference.md)** - Complete command options -- **[Usage Guide](docs/usage.md)** - Workflows and examples -- **[Deployment Guide](docs/deployment.md)** - Kubernetes/SLURM deployment -- **[Configuration Guide](docs/configuration.md)** - Advanced configuration -- **[All Docs](docs/)** - Complete documentation index - -### External Resources -- **MAD Package**: https://github.com/ROCm/MAD -- **Issues & Support**: https://github.com/ROCm/madengine/issues -- **ROCm Documentation**: https://rocm.docs.amd.com/ - -### Getting Help - -**Command Help:** -```bash -madengine --help # Main help -madengine --help # Command-specific help -madengine report --help # Sub-app help -madengine report to-html --help # Sub-command help -``` - -**Quick Checks:** -```bash -# Verify installation -madengine --version - -# Discover available models -madengine discover - -# Check specific model -madengine discover --tags your_model --verbose -``` - -**Troubleshooting:** -- Check [CLI Reference](docs/cli-reference.md) for all command options -- Enable `--verbose` flag for detailed error messages -- See [Usage Guide](docs/usage.md) troubleshooting section -- Report issues: https://github.com/ROCm/madengine/issues +- **MAD Package:** https://github.com/ROCm/MAD +- **Issues & Support:** https://github.com/ROCm/madengine/issues +- **ROCm Documentation:** https://rocm.docs.amd.com/ +- **Command help:** `madengine --help` Β· `madengine --help` --- ## ⚠️ Migration Notice (v2.0.0+) -The CLI has been unified! Starting from v2.0.0: -- βœ… Use `madengine` (unified modern CLI with K8s, SLURM, distributed support) -- ❌ Legacy v1.x CLI has been removed - ---- - -**Code Quality**: Clean codebase with no dead code, comprehensive test coverage, and following Python best practices. +The CLI has been unified. Starting from v2.0.0, use `madengine` (with K8s, SLURM, and distributed support); the legacy v1.x CLI has been removed. diff --git a/docs/README.md b/docs/README.md index db31f762..1388a2ae 100644 --- a/docs/README.md +++ b/docs/README.md @@ -35,7 +35,55 @@ Complete documentation for madengine - AI model automation and distributed bench ## πŸ—οΈ Architecture -The architecture diagram (Orchestration, Infrastructure, and Launcher layers) is in the [main README](../README.md#-architecture). Summary: +The CLI drives orchestrators that discover and build models, then hand off to a local or distributed execution target, which runs the model under the appropriate launcher and emits performance data for reporting. (Same diagram as the [main README](../README.md#-architecture).) + +```mermaid +flowchart TB + subgraph CLI["CLI Layer β€” Typer + Rich"] + C1[discover] + C2[build] + C3[run] + C4[report] + C5[database] + end + + subgraph ORC["Orchestration Layer"] + O1[DiscoverModels] + O2[BuildOrchestrator] + O3[RunOrchestrator] + MAN[(build_manifest.json)] + end + + subgraph EXEC["Execution / Deployment Layer"] + E1[ContainerRunner
local Docker] + E2[DeploymentFactory] + K8S[Kubernetes Jobs] + SLURM[SLURM Jobs] + end + + subgraph LAUNCH["Launcher Layer"] + T[Train: torchrun Β· DeepSpeed
Megatron-LM Β· TorchTitan Β· Primus] + I[Infer: vLLM Β· SGLang Β· SGLang Disagg] + end + + OUT[(perf.csv / JSON)] + + C1 --> O1 + C2 --> O2 + C3 --> O3 + O2 --> MAN --> O3 + O1 --> O2 + O3 --> E1 + O3 --> E2 + E2 --> K8S + E2 --> SLURM + E1 --> LAUNCH + K8S --> LAUNCH + SLURM --> LAUNCH + LAUNCH --> OUT + OUT --> C4 + OUT --> C5 +``` 1. **CLI Layer** - User interface with 5 commands (discover, build, run, report, database) 2. **Model Discovery** - Find and validate models from MAD package diff --git a/docs/batch-build.md b/docs/batch-build.md index 24983d98..7bac329f 100644 --- a/docs/batch-build.md +++ b/docs/batch-build.md @@ -224,11 +224,15 @@ Creates `build_manifest.json` with: } }, "built_models": {...}, + "context": {...}, + "credentials_required": {...}, "deployment_config": {...}, "summary": {...} } ``` +> `deployment_config` is only written when `--additional-context` resolves to a non-local deployment (e.g. `slurm`, `k8s`/`kubernetes`, `distributed`, `vllm`, or non-empty `env_vars`). Plain local builds omit this key entirely. + ## Best Practices 1. **Version Control**: Keep batch manifests in version control for reproducibility diff --git a/docs/cli-reference.md b/docs/cli-reference.md index 842e0fdf..57fa9d31 100644 --- a/docs/cli-reference.md +++ b/docs/cli-reference.md @@ -233,10 +233,10 @@ madengine run [OPTIONS] | `--manifest-output` | | TEXT | `build_manifest.json` | Output file for build manifest (full workflow) | | `--summary-output` | `-s` | TEXT | `None` | Output file for summary JSON | | `--live-output` | `-l` | FLAG | `False` | Print output in real-time | -| `--output` | `-o` | TEXT | `perf_entry.csv` | Performance output file | +| `--output` | `-o` | TEXT | `perf.csv` | Performance output file | | `--ignore-deprecated` | | FLAG | `False` | Force run deprecated models | | `--data-config` | | TEXT | `data.json` | Custom data configuration file | -| `--tools-config` | | TEXT | `tools.json` | Custom tools JSON configuration | +| `--tools-config` | | TEXT | `./scripts/common/tools.json` | Custom tools JSON configuration | | `--sys-env-details` | | FLAG | `True` | Generate system config env details | | `--force-mirror-local` | | TEXT | `None` | Path to force local data mirroring | | `--disable-skip-gpu-arch` | | FLAG | `False` | Disable skipping models based on GPU architecture | @@ -356,7 +356,7 @@ madengine run --tags model \ **Performance Output:** -Results are saved to CSV file (default: `perf_entry.csv`) with metrics including: +Results are saved to CSV file (default: `perf.csv`) with metrics including: - Execution time - GPU utilization - Memory usage @@ -384,20 +384,20 @@ madengine report to-html [OPTIONS] | Option | Short | Type | Required | Description | |--------|-------|------|----------|-------------| -| `--csv-file` | | TEXT | **Yes** | Path to the CSV file to convert | +| `--csv-file-path` | | TEXT | **Yes** | Path to the CSV file to convert | | `--verbose` | `-v` | FLAG | No | Enable verbose logging | **Examples:** ```bash # Convert CSV to HTML -madengine report to-html --csv-file perf_entry.csv +madengine report to-html --csv-file-path perf_entry.csv # With custom CSV file -madengine report to-html --csv-file results/perf_mi300.csv +madengine report to-html --csv-file-path results/perf_mi300.csv # Verbose output -madengine report to-html --csv-file perf.csv --verbose +madengine report to-html --csv-file-path perf.csv --verbose ``` **Output:** Creates `{filename}.html` in the same directory as the CSV file. @@ -444,7 +444,7 @@ madengine report to-email --directory ./results --verbose ### `database` - Upload to MongoDB -Upload CSV performance data to MongoDB database. +Upload CSV or JSON performance data to MongoDB (format is auto-detected). **Usage:** @@ -456,32 +456,30 @@ madengine database [OPTIONS] | Option | Short | Type | Default | Required | Description | |--------|-------|------|---------|----------|-------------| -| `--csv-file` | | TEXT | `perf_entry.csv` | No | Path to the CSV file to upload | -| `--database-name` | `--db` | TEXT | `None` | **Yes** | Name of the MongoDB database | -| `--collection-name` | `--collection` | TEXT | `None` | **Yes** | Name of the MongoDB collection | -| `--verbose` | `-v` | FLAG | `False` | No | Enable verbose logging | +| `--file` | `-f` | TEXT | `None` | **Yes** | Path to file (CSV or JSON, auto-detected) | +| `--database` | `--db` | TEXT | `None` | **Yes** | MongoDB database name | +| `--collection` | `-c` | TEXT | `None` | **Yes** | MongoDB collection name | +| `--unique-key` | `-k` | TEXT | `None` | No | Unique field(s) for deduplication (comma-separated, auto-detected if not specified) | +| `--batch-size` | | INT | `1000` | No | Batch size for bulk operations | +| `--no-upsert` | | FLAG | `False` | No | Insert only (don't update existing documents) | +| `--no-index` | | FLAG | `False` | No | Skip automatic index creation | +| `--dry-run` | | FLAG | `False` | No | Validate without uploading | +| `--verbose` | `-v` | FLAG | `False` | No | Verbose output | **Examples:** ```bash -# Upload to MongoDB -madengine database \ - --csv-file perf_entry.csv \ - --database-name mydb \ - --collection-name results +# Upload JSON with auto-detection +madengine database -f perf_entry_super.json --db mydb -c perf_super -# Short option names -madengine database \ - --csv-file perf.csv \ - --db test \ - --collection perf_data +# Upload CSV with custom unique key +madengine database -f perf.csv --db test -c results -k model,timestamp + +# Dry run to validate +madengine database -f data.json --db test -c data --dry-run # With verbose output -madengine database \ - --csv-file perf.csv \ - --db mydb \ - --collection results \ - --verbose +madengine database -f perf.csv --db mydb -c results --verbose ``` **Environment Variables:** @@ -490,10 +488,12 @@ MongoDB connection details are read from environment variables: | Variable | Description | Example | |----------|-------------|---------| -| `MONGO_HOST` | MongoDB host address | `localhost` or `mongodb.example.com` | -| `MONGO_PORT` | MongoDB port | `27017` | +| `MONGO_HOST` | MongoDB host address (default: `localhost`) | `localhost` or `mongodb.example.com` | +| `MONGO_PORT` | MongoDB port (default: `27017`) | `27017` | | `MONGO_USER` | MongoDB username | `admin` | | `MONGO_PASSWORD` | MongoDB password | `secretpassword` | +| `MONGO_AUTH_SOURCE` | MongoDB authentication database (default: `admin`) | `admin` | +| `MONGO_TIMEOUT_MS` | Server selection timeout in milliseconds (default: `5000`) | `5000` | **Example Setup:** @@ -504,7 +504,7 @@ export MONGO_USER=myuser export MONGO_PASSWORD=mypassword madengine database \ - --csv-file perf_entry.csv \ + --file perf_entry.csv \ --db performance_db \ --collection model_runs ``` diff --git a/docs/configuration.md b/docs/configuration.md index 4831cc4f..63a014bd 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -21,8 +21,7 @@ madengine run --tags model --additional-context-file config.json ```json { "gpu_vendor": "AMD", - "guest_os": "UBUNTU", - "timeout_multiplier": 2.0 + "guest_os": "UBUNTU" } ``` @@ -179,13 +178,13 @@ Unknown `rocenv_mode` values fall back to `lite` with a warning. **Overrides** (recommended for CI): - **Additional context (host):** top-level `"MAD_ROCM_PATH": "/path/to/host/rocm"` β€” controls where madengine looks for host GPU tools (`rocminfo`, `amd-smi`, etc.). -- **Additional context (container):** `"docker_env_vars": { "MAD_ROCM_PATH": "/path/inside/image" }` β€” sets the in-container `ROCM_PATH` for Docker runs. If omitted, at `run` time madengine uses the image OCI `Env` (`ROCM_PATH` / `ROCM_HOME`) if present, then an in-container probe, then defaults to `/opt/rocm`. The host-resolved path is **not** mirrored into the container. +- **Additional context (container):** `"docker_env_vars": { "ROCM_PATH": "/path/inside/image" }` β€” sets the in-container `ROCM_PATH` for Docker runs. If omitted, at `run` time madengine uses the image OCI `Env` (`ROCM_PATH` / `ROCM_HOME`) if present, then an in-container probe, then defaults to `/opt/rocm`. The host-resolved path is **not** mirrored into the container. These two keys are independent, allowing host and container to use different ROCm installations without confusion. Precedence (host): top-level `MAD_ROCM_PATH` β†’ auto-detect (unless disabled) β†’ `ROCM_PATH` β†’ `/opt/rocm`. -Precedence (container, **local Docker `run`**, **AMD**): `docker_env_vars.MAD_ROCM_PATH` (maps to `ROCM_PATH` for the workload) or explicit `ROCM_PATH` in `docker_env_vars` β†’ image OCI `Env` (`ROCM_PATH` / `ROCM_HOME`) β†’ in-image probe β†’ default `/opt/rocm` with a warning. Implemented in `ContainerRunner.run_container` after the run image is resolved. +Precedence (container, **local Docker `run`**, **AMD**): explicit `ROCM_PATH` in `docker_env_vars` β†’ image OCI `Env` (`ROCM_PATH` / `ROCM_HOME`) β†’ in-image probe β†’ default `/opt/rocm` with a warning. Implemented in `ContainerRunner.run_container` after the run image is resolved. This applies to the run phase; build uses build-only context (no GPU detection) but still honors `MAD_ROCM_PATH` in context when set. @@ -328,13 +327,15 @@ Format: Comma-separated list with hyphen ranges. ### Timeout Settings +Set a per-model timeout (seconds) in `models.json`: + ```json { - "timeout_multiplier": 2.0 + "timeout": 7200 } ``` -Or use command-line option: +Or use the command-line option, which overrides the model's timeout: ```bash madengine run --tags model --timeout 7200 @@ -388,7 +389,6 @@ Automatically applies (see presets under `src/madengine/deployment/presets/k8s/` "memory_limit": "64Gi", "cpu": "16", "cpu_limit": "32", - "service_account": "madengine-sa", "image_pull_policy": "Always", "ttl_seconds_after_finished": null, "allow_privileged_profiling": null, @@ -409,7 +409,6 @@ Automatically applies (see presets under `src/madengine/deployment/presets/k8s/` - `memory_limit` - Memory limit (default: 2Γ— memory request) - `cpu` - CPU cores request (default: auto-scaled by GPU count) - `cpu_limit` - CPU cores limit (default: 2Γ— CPU request) -- `service_account` - Service account name - `image_pull_policy` - `Always`, `IfNotPresent`, or `Never` - `ttl_seconds_after_finished` - Optional Job TTL in seconds (auto-delete finished Job); `null` to omit - `allow_privileged_profiling` - `null` means enable elevated `securityContext` when tools/profiling are configured; `true`/`false` to force @@ -471,7 +470,7 @@ Automatically applies (see presets under `src/madengine/deployment/presets/k8s/` - `partition` - SLURM partition name (required) - `account` - Billing account - `qos` - Quality of Service -- `gpus_per_node` - GPUs per node (default: 1) +- `gpus_per_node` - GPUs per node (default: 8) - `nodes` - Number of nodes (default: 1) - `nodelist` - Comma-separated node names to run on (e.g. `"node01,node02"`); when set, job is restricted to these nodes and automatic node health preflight is skipped - `reservation` - SLURM reservation name; forwarded to srun health/cleanup commands and SBATCH directives @@ -558,14 +557,12 @@ See [Launchers Guide](launchers.md) for details. "launcher": "vllm", "nnodes": 2, "nproc_per_node": 4 - }, - "vllm": { - "tensor_parallel_size": 4, - "pipeline_parallel_size": 1 } } ``` +`nproc_per_node` is exported into the container as `VLLM_TENSOR_PARALLEL_SIZE`; there is no separate `vllm.*` config block for tensor/pipeline parallel sizing. + ## Profiling Configuration ### Basic Profiling @@ -656,14 +653,11 @@ Configure in `data.json` (MAD package root): ```json { - "data_sources": { - "model_data": { - "nas": {"path": "/home/datum"}, - "minio": {"path": "s3://datasets/datum"}, - "aws": {"path": "s3://datasets/datum"} - } - }, - "mirrorlocal": "/tmp/local_mirror" + "model_data": { + "nas": {"path": "/home/datum", "mirrorlocal": "/tmp/local_mirror"}, + "minio": {"path": "s3://datasets/datum"}, + "aws": {"path": "s3://datasets/datum"} + } } ``` @@ -678,13 +672,13 @@ Configure in `credential.json` (MAD package root): "password": "your_token", "repository": "myorg" }, - "AMD_GITHUB": { + "PUBLIC_GITHUB_ROCM_KEY": { "username": "github_username", - "password": "github_token" + "token": "github_token" }, "MAD_AWS_S3": { - "username": "aws_access_key", - "password": "aws_secret_key" + "USERNAME": "aws_access_key", + "PASSWORD": "aws_secret_key" } } ``` diff --git a/docs/contributing.md b/docs/contributing.md index 5e53a300..078b23bc 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -61,7 +61,7 @@ pytest pytest --cov=src/madengine --cov-report=html # Run specific test file -pytest tests/test_cli.py +pytest tests/unit/test_cli.py # Run tests matching pattern pytest -k "test_build" diff --git a/docs/deployment.md b/docs/deployment.md index fa03e7f5..c913b117 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -13,24 +13,28 @@ Deployment is configured via `--additional-context` and happens automatically du ## Deployment Workflow +Build once, then deploy the resulting manifest to any target: + +```mermaid +flowchart LR + B["1. Build Phase
(local or CI/CD)
madengine build --tags model"] --> M[(build_manifest.json
+ image in registry)] + M --> D["2. Deploy Phase
madengine run --manifest-file build_manifest.json
--additional-context '{...}'"] + D --> T{detect target} + T -->|k8s / kubernetes| K[K8s Job] + T -->|slurm| S[SLURM script] + T -->|neither| L[Local Docker] ``` -β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” -β”‚ 1. Build Phase (Local or CI/CD) β”‚ -β”‚ madengine build --tags model β”‚ -β”‚ β†’ Creates Docker image β”‚ -β”‚ β†’ Pushes to registry β”‚ -β”‚ β†’ Generates build_manifest.json β”‚ -β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ - ↓ -β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” -β”‚ 2. Deploy Phase (Run with Context) β”‚ -β”‚ madengine run β”‚ -β”‚ --manifest-file build_manifest.json β”‚ -β”‚ --additional-context '{"deploy":...}' β”‚ -β”‚ β†’ Detects deployment target β”‚ -β”‚ β†’ Creates K8s Job or SLURM script β”‚ -β”‚ β†’ Submits and monitors execution β”‚ -β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + +### Deployment Target Inference + +No explicit `deploy` field is required β€” the target is inferred from the config structure (Convention over Configuration): + +```mermaid +flowchart TD + A[additional_context] --> Q{which key?} + Q -->|k8s / kubernetes| K[Kubernetes deployment] + Q -->|slurm| S[SLURM deployment] + Q -->|neither| L[Local Docker execution] ``` ## Kubernetes Deployment @@ -90,11 +94,12 @@ The deployment target is automatically detected from the `k8s` key in the config } ``` -**Configuration Priority:** -1. User config (`--additional-context-file`) -2. Profile presets (single-gpu/multi-gpu) -3. GPU vendor presets (AMD/NVIDIA) -4. Base defaults +**Configuration Priority (lowest to highest precedence):** +1. Base defaults +2. GPU vendor preset (AMD/NVIDIA) +3. GPU vendor multi-GPU preset (AMD only, for multi-GPU/multi-node) +4. Profile preset (single-gpu/multi-gpu/multi-node) +5. User config (`--additional-context-file`) β€” highest precedence, applied last See [examples/k8s-configs/](../examples/k8s-configs/) for complete examples. @@ -229,9 +234,7 @@ The deployment target is automatically detected from the `slurm` key in the conf "qos": "normal", "gpus_per_node": 8, "nodes": 1, - "time": "24:00:00", - "mail_user": "user@example.com", - "mail_type": "ALL" + "time": "24:00:00" } } ``` @@ -245,10 +248,7 @@ The deployment target is automatically detected from the `slurm` key in the conf - `nodelist`: Comma-separated node names to run on (e.g. `"node01,node02"`); when set, job runs only on these nodes and node health preflight is skipped - `reservation`: SLURM reservation name; forwarded to srun health/cleanup commands - `time`: Wall time limit (HH:MM:SS) -- `mem`: Memory per node (e.g., "64G") - `exclusive`: Exclusive node access (default: `true`) -- `mail_user`: Email for job notifications -- `mail_type`: Notification types (BEGIN, END, FAIL, ALL) See [examples/slurm-configs/](../examples/slurm-configs/) for complete examples. diff --git a/docs/img/architecture_overview.png b/docs/img/architecture_overview.png deleted file mode 100755 index 7bf972b3..00000000 Binary files a/docs/img/architecture_overview.png and /dev/null differ diff --git a/docs/img/distributed_workflow.png b/docs/img/distributed_workflow.png deleted file mode 100755 index a6723b44..00000000 Binary files a/docs/img/distributed_workflow.png and /dev/null differ diff --git a/docs/installation.md b/docs/installation.md index c8c9eb4c..e4cbd658 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -91,7 +91,6 @@ madengine run --tags dummy \ ```bash # Check installation madengine --version -madengine --version # Test basic functionality (requires MAD package) cd /path/to/MAD diff --git a/docs/launchers.md b/docs/launchers.md index 227557aa..26121300 100644 --- a/docs/launchers.md +++ b/docs/launchers.md @@ -69,11 +69,13 @@ madengine run --manifest-file build_manifest.json "launcher": "torchrun", "nnodes": 2, "nproc_per_node": 8, - "master_port": 29500 + "port": 29500 } } ``` +Note: `distributed.port` sets the master port on **SLURM** (`distributed.get("port", 29500)`). On **Kubernetes**, the master port is instead read from a separate top-level `"launcher"` object (`{"launcher": {"master_port": 29500}}`), not from `distributed`. + **Features**: - Automatic rank assignment - NCCL backend for GPU communication @@ -136,7 +138,7 @@ madengine run --manifest-file build_manifest.json ```json { "distributed": { - "launcher": "megatron", + "launcher": "megatron-lm", "nnodes": 4, "nproc_per_node": 8 } @@ -276,8 +278,6 @@ Optional **`primus.backend`** (e.g. `MaxText`, `megatron`) emits `export BACKEND **Container image**: Prefer `docker/primus.ubuntu.amd.Dockerfile` with `COPY scripts/Primus/ /workspace/Primus/` and `PRIMUS_ROOT=/workspace/Primus`. On **Kubernetes**, the Job’s emptyDir hides image files under `/workspace`; madengine bundles `scripts/Primus/examples/...` into the ConfigMap as `Primus/examples/...` so the init container recreates `/workspace/Primus`. `run.sh` resolves `PRIMUS_ROOT` in that order (see script comments). **Examples**: -- SLURM: `examples/slurm-configs/minimal/primus-minimal.json` -- K8s: `examples/k8s-configs/minimal/primus-minimal.json` - K8s (Primus vs upstream workload API, MaxText caveats, TorchTitan/Megatron/MaxText sample JSON): `examples/k8s-configs/README.md` section **Primus on Kubernetes** --- @@ -314,13 +314,13 @@ Optional **`primus.backend`** (e.g. `MaxText`, `megatron`) emits `export BACKEND **Architecture**: - Single-node: TP across GPUs, no Ray - Multi-node (K8s): Data Parallelism with independent replicas per pod -- Multi-node (SLURM): TP + PP with Ray cluster +- Multi-node (SLURM): Data Parallelism (one vLLM serve per node, TP only on that node, no shared Ray cluster) **Environment Variables**: ```bash VLLM_TENSOR_PARALLEL_SIZE=4 VLLM_PIPELINE_PARALLEL_SIZE=1 -VLLM_DISTRIBUTED_BACKEND="auto" # or "ray" for multi-node +VLLM_DISTRIBUTED_BACKEND="auto" # or "none" for multi-node SLURM (data parallel) ``` **Examples**: @@ -435,7 +435,7 @@ SGLang Disaggregated separates inference into specialized node pools: | Total Nodes | Proxy | Prefill | Decode | |-------------|-------|---------|--------| | 3 | 1 | 1 (33%) | 1 (33%) | -| 5 | 1 | 2 (40%) | 2 (40%) | +| 5 | 1 | 1 (25%) | 3 (75%) | | 7 | 1 | 2 (29%) | 4 (57%) | | 11 | 1 | 4 (40%) | 6 (60%) | @@ -529,14 +529,14 @@ SGLANG_NODE_IPS="10.0.0.1,10.0.0.2,..." **Performance Tuning**: ```bash # Start with automatic split -madengine run --tags model --config minimal-config.json +madengine run --tags model --additional-context-file minimal-config.json # Monitor bottleneck (prefill latency vs decode throughput) # If prefill is bottleneck β†’ increase prefill nodes # If decode is bottleneck β†’ increase decode nodes # Apply custom split -madengine run --tags model --config custom-split-config.json +madengine run --tags model --additional-context-file custom-split-config.json ``` **Troubleshooting**: @@ -694,6 +694,34 @@ madengine run --manifest-file build_manifest.json --- +## Parallelism Capabilities + +How each launcher handles the various parallelism strategies. `βœ…Auto` = supported and configured by madengine; `❗Manual` = supported by the launcher but requires user configuration; `❗Limited` / `❗Disabled` = launcher or platform limitation. + +| Launcher | Tensor Parallel (TP) | Pipeline Parallel (PP) | Data Parallel (DP) | Context Parallel (CP) | FSDP/ZeRO | Expert Parallel (EP) | Primary Use Case | +|----------|----------------------|------------------------|--------------------|------------------------|-----------|----------------------|------------------| +| **torchrun** | ❗Manual | ❌No | ❗Manual (DDP) | ❌No | ❗Manual (FSDP) | ❌No | General distributed training | +| **TorchTitan** | βœ…Auto | βœ…Auto | βœ…Auto (FSDP2) | ❗Manual | βœ…Auto (FSDP2) | ❌No | Large-scale LLM pre-training | +| **DeepSpeed** | ❗Manual | ❗Manual | βœ…Auto (ZeRO) | ❌No | βœ…Auto (ZeRO) | ❌No | Memory-efficient training | +| **Megatron-LM** | βœ…Auto | βœ…Auto | βœ…Implicit | βœ…Auto | ❌No | ❌No | Large transformer training | +| **Primus** | ❗Manual | ❗Manual | ❗Manual | ❗Manual | ❗Manual | ❌No | Unified pretrain (experiment YAML; backend-specific) | +| **vLLM** | βœ…Auto | SLURM: βœ…Auto (Multi) / K8s: ❗Disabled | βœ…Auto (Replicas) | ❌No | ❌No | ❗Manual | High-throughput inference | +| **SGLang** | βœ…Auto | SLURM: βœ…Auto (Multi) / K8s: ❗Disabled | ❗Limited | ❌No | ❌No | ❌No | Inference + structured gen | +| **SGLang PD Disagg** | βœ…Auto | ❌No | βœ…Role-based | ❌No | ❌No | ❌No | Optimized prefill/decode | + +## Infrastructure Capabilities + +| Feature | Local | Kubernetes | SLURM | +|---------|-------|-----------|-------| +| **Execution** | Docker containers | K8s Jobs | SLURM jobs | +| **Multi-Node** | ❌ | βœ… Indexed Jobs | βœ… Job arrays | +| **Resource Mgmt** | Manual | Declarative (YAML) | Batch scheduler | +| **Monitoring** | Docker logs | kubectl/dashboard | squeue/scontrol | +| **Auto-scaling** | ❌ | βœ… | ❌ | +| **Network** | Host | CNI plugin | InfiniBand/Ethernet | + +--- + ## Configuration Best Practices ### 1. Launcher Selection diff --git a/docs/profiling.md b/docs/profiling.md index 5e421ec8..367313ef 100644 --- a/docs/profiling.md +++ b/docs/profiling.md @@ -93,6 +93,8 @@ madengine uses `rocprof_wrapper.sh` to automatically handle the transition betwe ``` 3. **Backward Compatibility:** The `--` works with both rocprof and rocprofv3, ensuring your configurations work across ROCm versions +**Related presets:** `rocprof_hip_only` (HIP trace only) and `rocprof_sys` (system trace only) wrap the same `rocprof_wrapper.sh` with fixed flags, for when you don't need a custom `cmd`. + **Example - Custom Command with Wrapper:** ```json { @@ -258,6 +260,8 @@ ROCprofv3 is the next-generation profiler for ROCm 7.0+ with enhanced features a - **Metrics**: Program counter sampling at 1000 Hz - **Output Format**: Perfetto trace with PC samples +**Other presets:** `rocprofv3` (bare `--runtime-trace` invocation, no wrapper), `rocprofv3_agent` (CSV stats/kernel trace via `rocprof_wrapper.sh`), and `rocprofv3_agent_counter` (same as `rocprofv3_agent` plus hardware counters from `counters/instruction_mix.txt`) are also available as `{"name": "..."}` tool entries. + #### Using Pre-Configured Profiles madengine provides ready-to-use configuration files in `examples/profiling-configs/`: @@ -321,6 +325,7 @@ Custom counter files are in `scripts/common/tools/counters/`: - `memory_bound.txt` - Cache and memory metrics - `communication_bound.txt` - PCIe and synchronization metrics - `full_profile.txt` - Comprehensive metrics +- `instruction_mix.txt` - Instruction mix metrics (used by `rocprofv3_agent_counter`) Create your own counter file: ```text @@ -360,6 +365,8 @@ Trace rocBLAS API calls and configurations: **Use Case:** Analyze BLAS operations, identify optimization opportunities +**Related:** `hipblaslt_trace` traces hipBLASLt calls the same way (sets `HIPBLASLT_TRACE=1`). + ### miopen_trace - MIOpen Library Tracing Trace MIOpen API calls for deep learning operations: @@ -509,9 +516,6 @@ Profile real-time GPU memory consumption: } ``` This will generate both `gpu_info_power_profiler_output.csv` and `gpu_info_vram_profiler_output.csv`. -- `SAMPLING_RATE` - Sampling interval in seconds -- `MODE` - Must be `"vram"` for this tool -- `DUAL-GCD` - Enable dual-GCD mode **Supported Platforms:** ROCm and CUDA @@ -666,8 +670,8 @@ madengine run --tags model \ { "name": "gpu_info_power_profiler", "env_vars": { - "DEVICE": "all", - "SAMPLING_RATE": "0.1" + "POWER_DEVICE": "all", + "POWER_SAMPLING_RATE": "0.1" } }, {"name": "rccl_trace"} @@ -777,7 +781,7 @@ Balance detail vs. overhead: { "name": "gpu_info_power_profiler", "env_vars": { - "SAMPLING_RATE": "1.0" // Less overhead, less detail + "POWER_SAMPLING_RATE": "1.0" // Less overhead, less detail } } ] @@ -923,15 +927,19 @@ Tool defaults are defined in `scripts/common/tools.json`: ```json { "rocprof": { - "cmd": "rocprof --hip-trace", - "env_vars": {} + "cmd": "bash ../scripts/common/tools/rocprof_wrapper.sh --runtime-trace --", + "env_vars": {}, + "post_scripts": [ + {"path": "scripts/common/post_scripts/trace.sh", "args": "rocprof"} + ] }, "gpu_info_power_profiler": { "env_vars": { - "DEVICE": "0", - "SAMPLING_RATE": "0.1", - "MODE": "power", - "DUAL-GCD": "false" + "POWER_DEVICE": "all", + "POWER_SAMPLING_RATE": "0.1", + "POWER_MODE": "power", + "POWER_DUAL_GCD": "false", + "POWER_OUTPUT_FILE": "gpu_info_power_profiler_output.csv" } } } @@ -956,8 +964,7 @@ Validate [TheRock](https://github.com/ROCm/TheRock) ROCm installations before ru ```bash madengine run --tags dummy_therock \ - --tools therock_check \ - --additional-context '{"gpu_vendor": "AMD", "guest_os": "UBUNTU"}' + --additional-context '{"gpu_vendor": "AMD", "guest_os": "UBUNTU", "tools": [{"name": "therock_check"}]}' ``` **Standalone detection:** diff --git a/docs/usage.md b/docs/usage.md index f06557bb..334cade0 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -46,7 +46,7 @@ madengine provides five main commands: | `build` | Build Docker images | `--tags`, `--registry`, `--batch-manifest` | | `run` | Execute models | `--tags`, `--manifest-file`, `--timeout` | | `report` | Generate HTML reports | `to-html`, `to-email` | -| `database` | Upload to MongoDB | `--csv-file`, `--database-name` | +| `database` | Upload to MongoDB | `--file`, `--db` | For complete command options and detailed examples, see **[CLI Command Reference](cli-reference.md)**. @@ -66,11 +66,11 @@ madengine run --tags model # madengine build --tags model --additional-context '{"gpu_vendor": "NVIDIA", "guest_os": "CENTOS"}' # Generate HTML report -madengine report to-html --csv-file perf_entry.csv +madengine report to-html --csv-file-path perf_entry.csv # Upload to MongoDB -madengine database --csv-file perf_entry.csv \ - --database-name mydb --collection-name results +madengine database --file perf_entry.csv \ + --database mydb --collection results ``` ## Model Discovery @@ -82,7 +82,7 @@ madengine supports three discovery methods: Central model definitions in MAD package root: ```bash -madengine discover --tags dummy pyt_huggingface_bert +madengine discover --tags dummy --tags pyt_huggingface_bert ``` ### 2. Directory-Specific Models @@ -117,18 +117,32 @@ Creates `build_manifest.json`: ```json { - "models": [ - { - "model_name": "my_model", - "image": "localhost:5000/my_model:20240115_123456", - "tag": "my_model" + "built_images": { + "ci-my_model_ubuntu": { + "model": "my_model", + "docker_image": "ci-my_model_ubuntu", + "dockerfile": "docker/my_model.ubuntu.amd.Dockerfile", + "build_duration": 42.3, + "registry": "localhost:5000" } - ], - "registry": "localhost:5000", - "build_timestamp": "2024-01-15T12:34:56Z" + }, + "built_models": { + "ci-my_model_ubuntu": { + "name": "my_model", + "dockerfile": "my_model", + "n_gpus": "1" + } + }, + "context": { + "gpu_vendor": "AMD", + "guest_os": "UBUNTU" + }, + "credentials_required": [] } ``` +`built_images` and `built_models` are both keyed by the built Docker image name. Depending on the build, the manifest may also include `deployment_config` and `summary` keys. + ### Build with Deployment Config Include deployment configuration: @@ -493,7 +507,7 @@ Convert performance CSV files to viewable HTML reports: ```bash # Single CSV to HTML -madengine report to-html --csv-file perf_entry.csv +madengine report to-html --csv-file-path perf_entry.csv # Result: Creates perf_entry.html in same directory ``` @@ -532,13 +546,13 @@ export MONGO_PASSWORD=secretpassword # Upload results madengine database \ - --csv-file perf_entry.csv \ - --database-name performance_tracking \ - --collection-name model_runs + --file perf_entry.csv \ + --database performance_tracking \ + --collection model_runs # Upload specific results madengine database \ - --csv-file results/perf_mi300.csv \ + --file results/perf_mi300.csv \ --db benchmarks \ --collection mi300_results ``` @@ -547,15 +561,15 @@ madengine database \ ```bash # 1. Run benchmarks -madengine run --tags model1 model2 model3 \ +madengine run --tags model1 --tags model2 --tags model3 \ --output perf_entry.csv # 2. Generate HTML report -madengine report to-html --csv-file perf_entry.csv +madengine report to-html --csv-file-path perf_entry.csv # 3. Upload to database madengine database \ - --csv-file perf_entry.csv \ + --file perf_entry.csv \ --db benchmarks \ --collection daily_runs @@ -764,7 +778,7 @@ if [ $? -eq 0 ]; then # Generate and upload results madengine report to-email --output ci_results.html madengine database \ - --csv-file perf.csv \ + --file perf.csv \ --db ci_results \ --collection ${CI_BUILD_ID} else diff --git a/examples/k8s-configs/README.md b/examples/k8s-configs/README.md index 40843ab3..fbd327dd 100644 --- a/examples/k8s-configs/README.md +++ b/examples/k8s-configs/README.md @@ -103,16 +103,16 @@ MODEL_DIR=tests/fixtures/dummy madengine run \ ```bash # For single GPU testing -cp examples/k8s-configs/01-single-node-single-gpu.json my-config.json +cp examples/k8s-configs/basic/01-native-single-node-single-gpu.json my-config.json # For multi-GPU (2 GPUs) -cp examples/k8s-configs/02-single-node-multi-gpu.json my-config.json +cp examples/k8s-configs/basic/02-torchrun-single-node-multi-gpu.json my-config.json # For multi-node distributed (2 nodes Γ— 2 GPUs) -cp examples/k8s-configs/03-multi-node-basic.json my-config.json +cp examples/k8s-configs/basic/03-torchrun-multi-node-basic.json my-config.json # For data provider with auto-PVC -cp examples/k8s-configs/06-data-provider-with-pvc.json my-config.json +cp examples/k8s-configs/basic/06-data-provider-with-pvc.json my-config.json ``` #### 2. Customize for Your Cluster (Optional) @@ -158,10 +158,10 @@ Located in [`minimal/`](minimal/) directory: | File | Description | GPU Count | |------|-------------|-----------| -| [`minimal/single-gpu-minimal.json`](minimal/single-gpu-minimal.json) | Single GPU with auto-defaults | 1 | -| [`minimal/multi-gpu-minimal.json`](minimal/multi-gpu-minimal.json) | Multi-GPU with auto-defaults | 2 | -| [`minimal/multi-node-minimal.json`](minimal/multi-node-minimal.json) | Multi-node with auto-defaults | 2Γ—2 | -| [`minimal/nvidia-gpu-minimal.json`](minimal/nvidia-gpu-minimal.json) | NVIDIA GPUs with auto-defaults | 4 | +| [`minimal/torchrun-single-gpu-minimal.json`](minimal/torchrun-single-gpu-minimal.json) | Single GPU with auto-defaults | 1 | +| [`minimal/torchrun-multi-gpu-minimal.json`](minimal/torchrun-multi-gpu-minimal.json) | Multi-GPU with auto-defaults | 2 | +| [`minimal/torchrun-multi-node-minimal.json`](minimal/torchrun-multi-node-minimal.json) | Multi-node with auto-defaults | 2Γ—2 | +| [`minimal/torchrun-nvidia-gpu-minimal.json`](minimal/torchrun-nvidia-gpu-minimal.json) | NVIDIA GPUs with auto-defaults | 4 | | [`minimal/custom-namespace-minimal.json`](minimal/custom-namespace-minimal.json) | Shows override examples | 1 | **Distributed Launchers:** @@ -169,7 +169,6 @@ Located in [`minimal/`](minimal/) directory: | File | Launcher | Description | GPUs | |------|----------|-------------|------| | [`minimal/torchtitan-single-node-minimal.json`](minimal/torchtitan-single-node-minimal.json) | TorchTitan | LLM pre-training (single-node) | 8 | -| [`minimal/primus-minimal.json`](minimal/primus-minimal.json) | primus | Primus pretrain (edit `distributed.primus.config_path`) | 2 | | [`minimal/vllm-single-node-minimal.json`](minimal/vllm-single-node-minimal.json) | vLLM | LLM inference (single-node) | 4 | | [`minimal/sglang-single-node-minimal.json`](minimal/sglang-single-node-minimal.json) | SGLang | LLM inference (single-node) | 4 | @@ -211,24 +210,24 @@ Set `distributed.primus.config_path` to your YAML under the Primus repo layout. **MaxText caveat:** For **multi-node** MaxText, Primus `run_pretrain.sh` may run **in-container `apt` installs** (InfiniBand-related packages). Many clusters disallow that unless the image is pre-baked or policy allows it. madengine logs a **warning** when MaxText is detected (`backend` or path) and `nnodes > 1`. -Primus examples under [`basic/`](basic/): [`primus-single-node-multi-gpu.json`](basic/primus-single-node-multi-gpu.json) (one pod, multi-GPU) and [`primus-multi-node.json`](basic/primus-multi-node.json) (Indexed Job). The same files work for TorchTitan, Megatron, and MaxText: set `distributed.primus.config_path` to your experiment YAML, and rely on madengine’s `BACKEND` inference from the model name (`primus_pretrain/__...`) or set `distributed.primus.backend` only when you need an explicit override. Use `docker_env_vars.HF_TOKEN` as a placeholder or runtime secrets β€” do not commit real tokens. +To use Primus, start from any `distributed.launcher: "torchrun"` example under [`basic/`](basic/) or [`minimal/`](minimal/) and change `distributed.launcher` to `"primus"`, adding a nested `distributed.primus.config_path` pointing at your experiment YAML (see [Distributed Execution Fields](#distributed-execution-fields) below). This works for TorchTitan, Megatron, and MaxText: set `distributed.primus.config_path` to your experiment YAML, and rely on madengine's `BACKEND` inference from the model name (`primus_pretrain/__...`) or set `distributed.primus.backend` only when you need an explicit override. Use `docker_env_vars.HF_TOKEN` as a placeholder or runtime secrets β€” do not commit real tokens. ### Full Configs (Reference Examples) Complete configurations showing all available fields: -**Training Configs:** +**Training Configs (basic/):** | File | GPUs | Nodes | Launcher | Use Case | |------|------|-------|----------|----------| -| [`01-single-node-single-gpu.json`](01-single-node-single-gpu.json) | 1 | 1 | None | Basic testing, small models | -| [`01-single-node-single-gpu-tools.json`](01-single-node-single-gpu-tools.json) | 1 | 1 | None | Single GPU + monitoring | -| [`02-single-node-multi-gpu.json`](02-single-node-multi-gpu.json) | 2 | 1 | torchrun | Multi-GPU training | -| [`02-single-node-multi-gpu-tools.json`](02-single-node-multi-gpu-tools.json) | 2 | 1 | torchrun | Multi-GPU + monitoring | -| [`03-multi-node-basic.json`](03-multi-node-basic.json) | 2/node | 2 | torchrun | Multi-node basics (4 GPUs total) | -| [`04-multi-node-advanced.json`](04-multi-node-advanced.json) | 2/node | 4 | torchrun | Production multi-node (8 GPUs) | -| [`05-nvidia-gpu-example.json`](05-nvidia-gpu-example.json) | 4 | 1 | torchrun | NVIDIA GPUs (A100, H100) | -| [`06-data-provider-with-pvc.json`](06-data-provider-with-pvc.json) | 2 | 1+ | torchrun | **Data provider with auto-PVC** | +| [`basic/01-native-single-node-single-gpu.json`](basic/01-native-single-node-single-gpu.json) | 1 | 1 | None | Basic testing, small models | +| [`basic/01-native-single-node-single-gpu-tools.json`](basic/01-native-single-node-single-gpu-tools.json) | 1 | 1 | None | Single GPU + monitoring | +| [`basic/02-torchrun-single-node-multi-gpu.json`](basic/02-torchrun-single-node-multi-gpu.json) | 2 | 1 | torchrun | Multi-GPU training | +| [`basic/02-torchrun-single-node-multi-gpu-tools.json`](basic/02-torchrun-single-node-multi-gpu-tools.json) | 2 | 1 | torchrun | Multi-GPU + monitoring | +| [`basic/03-torchrun-multi-node-basic.json`](basic/03-torchrun-multi-node-basic.json) | 2/node | 2 | torchrun | Multi-node basics (4 GPUs total) | +| [`basic/04-torchrun-multi-node-advanced.json`](basic/04-torchrun-multi-node-advanced.json) | 2/node | 4 | torchrun | Production multi-node (8 GPUs) | +| [`basic/05-torchrun-nvidia-gpu-example.json`](basic/05-torchrun-nvidia-gpu-example.json) | 4 | 1 | torchrun | NVIDIA GPUs (A100, H100) | +| [`basic/06-data-provider-with-pvc.json`](basic/06-data-provider-with-pvc.json) | 2 | 1+ | torchrun | **Data provider with auto-PVC** | **Distributed Launcher Configs (basic/):** @@ -237,8 +236,6 @@ Complete configurations showing all available fields: | [`basic/torchtitan-multi-node-basic.json`](basic/torchtitan-multi-node-basic.json) | 8/node | 4 | TorchTitan | Llama 3.1 70B+ training | | [`basic/vllm-multi-node-basic.json`](basic/vllm-multi-node-basic.json) | 4/node | 2 | vLLM | High-throughput inference | | [`basic/sglang-multi-node-basic.json`](basic/sglang-multi-node-basic.json) | 4/node | 2 | SGLang | Distributed inference | -| [`basic/primus-single-node-multi-gpu.json`](basic/primus-single-node-multi-gpu.json) | 8 | 1 | primus | Primus pretrain (single pod; edit `primus.config_path`) | -| [`basic/primus-multi-node.json`](basic/primus-multi-node.json) | 8/node | 2+ | primus | Primus pretrain (multi-pod; edit `nnodes`, `primus.config_path`) | --- @@ -248,26 +245,26 @@ Complete configurations showing all available fields: | Scenario | Config File | GPUs | Nodes | |----------|-------------|------|-------| -| **Quick test** | `01-single-node-single-gpu.json` | 1 | 1 | -| **Single GPU benchmark** | `01-single-node-single-gpu-tools.json` | 1 | 1 | -| **Multi-GPU (2 GPUs)** | `02-single-node-multi-gpu.json` | 2 | 1 | -| **Multi-GPU + monitoring** | `02-single-node-multi-gpu-tools.json` | 2 | 1 | -| **Multi-node (4 GPUs)** | `03-multi-node-basic.json` | 2Γ—2 | 2 | -| **Multi-node (8 GPUs)** | `04-multi-node-advanced.json` | 2Γ—4 | 4 | -| **NVIDIA GPUs** | `05-nvidia-gpu-example.json` | 4 | 1 | -| **With data download** | `06-data-provider-with-pvc.json` | 2 | 1+ | +| **Quick test** | `basic/01-native-single-node-single-gpu.json` | 1 | 1 | +| **Single GPU benchmark** | `basic/01-native-single-node-single-gpu-tools.json` | 1 | 1 | +| **Multi-GPU (2 GPUs)** | `basic/02-torchrun-single-node-multi-gpu.json` | 2 | 1 | +| **Multi-GPU + monitoring** | `basic/02-torchrun-single-node-multi-gpu-tools.json` | 2 | 1 | +| **Multi-node (4 GPUs)** | `basic/03-torchrun-multi-node-basic.json` | 2Γ—2 | 2 | +| **Multi-node (8 GPUs)** | `basic/04-torchrun-multi-node-advanced.json` | 2Γ—4 | 4 | +| **NVIDIA GPUs** | `basic/05-torchrun-nvidia-gpu-example.json` | 4 | 1 | +| **With data download** | `basic/06-data-provider-with-pvc.json` | 2 | 1+ | ### By Use Case | Use Case | Recommended Config | |----------|-------------------| -| **Development/Testing** | `01-single-node-single-gpu.json` | -| **Small models (BERT, ResNet)** | `01-single-node-single-gpu.json` | -| **Medium models (GPT-2, Stable Diffusion)** | `02-single-node-multi-gpu.json` | -| **Large models (LLaMA-13B)** | `03-multi-node-basic.json` | -| **Very large models (LLaMA-70B+)** | `04-multi-node-advanced.json` | -| **Models requiring datasets** | `06-data-provider-with-pvc.json` | -| **Busy/shared clusters** | `02-single-node-multi-gpu.json` (2 GPUs) | +| **Development/Testing** | `basic/01-native-single-node-single-gpu.json` | +| **Small models (BERT, ResNet)** | `basic/01-native-single-node-single-gpu.json` | +| **Medium models (GPT-2, Stable Diffusion)** | `basic/02-torchrun-single-node-multi-gpu.json` | +| **Large models (LLaMA-13B)** | `basic/03-torchrun-multi-node-basic.json` | +| **Very large models (LLaMA-70B+)** | `basic/04-torchrun-multi-node-advanced.json` | +| **Models requiring datasets** | `basic/06-data-provider-with-pvc.json` | +| **Busy/shared clusters** | `basic/02-torchrun-single-node-multi-gpu.json` (2 GPUs) | --- @@ -278,7 +275,7 @@ Complete configurations showing all available fields: ```bash MODEL_DIR=tests/fixtures/dummy madengine build \ --tags dummy \ - --additional-context-file examples/k8s-configs/01-single-node-single-gpu.json \ + --additional-context-file examples/k8s-configs/basic/01-native-single-node-single-gpu.json \ --registry dockerhub MODEL_DIR=tests/fixtures/dummy madengine run \ @@ -291,7 +288,7 @@ MODEL_DIR=tests/fixtures/dummy madengine run \ ```bash MODEL_DIR=tests/fixtures/dummy madengine build \ --tags dummy_torchrun \ - --additional-context-file examples/k8s-configs/02-single-node-multi-gpu.json \ + --additional-context-file examples/k8s-configs/basic/02-torchrun-single-node-multi-gpu.json \ --registry dockerhub MODEL_DIR=tests/fixtures/dummy madengine run \ @@ -304,7 +301,7 @@ MODEL_DIR=tests/fixtures/dummy madengine run \ ```bash MODEL_DIR=tests/fixtures/dummy madengine build \ --tags dummy_torchrun \ - --additional-context-file examples/k8s-configs/03-multi-node-basic.json \ + --additional-context-file examples/k8s-configs/basic/03-torchrun-multi-node-basic.json \ --registry dockerhub MODEL_DIR=tests/fixtures/dummy madengine run \ @@ -317,7 +314,7 @@ MODEL_DIR=tests/fixtures/dummy madengine run \ ```bash MODEL_DIR=tests/fixtures/dummy madengine build \ --tags dummy_torchrun_data_minio \ - --additional-context-file examples/k8s-configs/06-data-provider-with-pvc.json \ + --additional-context-file examples/k8s-configs/basic/06-data-provider-with-pvc.json \ --registry dockerhub MODEL_DIR=tests/fixtures/dummy madengine run \ @@ -347,7 +344,7 @@ kubectl get pvc madengine-shared-data **Step 1: Use data provider config** ```bash madengine build --tags dummy_torchrun_data_minio \ - --additional-context-file examples/k8s-configs/06-data-provider-with-pvc.json \ + --additional-context-file examples/k8s-configs/basic/06-data-provider-with-pvc.json \ --registry dockerhub ``` @@ -465,7 +462,6 @@ To use an existing PVC instead of auto-creation: "_comment": "Description of this configuration", "gpu_vendor": "AMD|NVIDIA", "guest_os": "UBUNTU", - "deploy": "k8s", "k8s": { "kubeconfig": "~/.kube/config", @@ -483,8 +479,7 @@ To use an existing PVC instead of auto-creation: "node_selector": {}, "tolerations": [], - "data_pvc": null, // Optional: for data providers - "results_pvc": null // Optional: custom results storage + "data_pvc": null // Optional: for data providers }, "distributed": { @@ -512,10 +507,10 @@ To use an existing PVC instead of auto-creation: | Field | Type | Required | Description | |-------|------|----------|-------------| -| `gpu_vendor` | string | **Yes** | `"AMD"` or `"NVIDIA"` | -| `guest_os` | string | **Yes** | `"UBUNTU"`, `"RHEL"`, etc. | -| `deploy` | string | **Yes** | Must be `"k8s"` | -| `k8s` | object | **Yes** | Kubernetes configuration | +| `gpu_vendor` | string | No (default `"AMD"`) | `"AMD"` or `"NVIDIA"` | +| `guest_os` | string | No (default `"UBUNTU"`) | `"UBUNTU"` or `"CENTOS"` | +| `deploy` | string | No | Not needed β€” deploy target is auto-inferred from presence of the `k8s`/`kubernetes` key (or `slurm`); if set, it must agree with the inferred target | +| `k8s` | object | **Yes** | Kubernetes configuration (presence of this key is what triggers K8s deployment) | | `distributed` | object | No | Distributed training (for torchrun) | | `env_vars` | object | No | Custom environment variables | | `debug` | boolean | No | Enable debug mode (saves manifests) | @@ -551,7 +546,9 @@ To use an existing PVC instead of auto-creation: |-------|------|---------|-------------| | `image_pull_policy` | string | `"Always"` | `"Always"`, `"IfNotPresent"`, or `"Never"` | | `backoff_limit` | integer | `3` | Retry attempts before marking failed | -| `host_ipc` | boolean | `false` | Enable shared memory (required for multi-node) | +| `allow_privileged_profiling` | boolean | `null` | Whether the pod runs with the elevated privileges profiling tools need. If unset (`null`), it's auto-enabled when a `tools` config is present; set explicitly to `true`/`false` to override | + +**Note:** `host_ipc` is **not** user-configurable β€” madengine always sets it automatically to `nnodes > 1` (enabled for multi-node jobs, disabled otherwise). Setting `k8s.host_ipc` in your config has no effect; it is not read anywhere. **Optional - Node Selection:** @@ -565,15 +562,18 @@ To use an existing PVC instead of auto-creation: | Field | Type | Default | Description | |-------|------|---------|-------------| | `data_pvc` | string | `null` | Data PVC name (auto-created if using data provider) | -| `results_pvc` | string | `null` | Results PVC name (auto-created by default) | | `storage_class` | string | **`nfs-banff`** (preset, since 2.0.3) | Generic broad fallback for both the data PVC and the single-node results PVC when no more-specific key is set | | `nfs_storage_class` | string | **`nfs-banff`** (preset) | RWX class for shared-data / multi-node results | | `local_path_storage_class` | string | `null` (not in preset since 2.0.3; was **`local-path`** in ≀ 2.0.2) | Optional RWO class for single-node `{job}-results`. Still honoured for backward compatibility | | `data_storage_class` | string | **`nfs-banff`** (preset) | Overrides SC for shared-data only | | `single_node_results_storage_class` | string | `null` | Overrides single-node results SC (falls back to `local_path_storage_class`, then `storage_class`) | | `multi_node_results_storage_class` | string | `null` | Overrides multi-node results SC (`nfs_storage_class` if unset) | +| `results_storage_size` | string | `"10Gi"` | Size of the per-job `{job}-results` PVC | +| `data_storage_size` | string | `"100Gi"` | Size of the shared `madengine-shared-data` PVC | | `recreate_shared_data_pvc` | boolean | **`false`** (preset) | If `true`, delete `madengine-shared-data` before create (data loss) | +**Note:** `results_pvc` (the per-job results PVC name) is **not** user-configurable β€” madengine always creates and uses `{job_name}-results`. There is no config key that overrides this name. + #### Distributed Execution Fields Configuration for distributed workloads (training and inference): @@ -650,7 +650,7 @@ CPU: 16 (request), 32 (limit) GPUs: 2 per node (4 total) Memory: 64Gi per node CPU: 16 per node -host_ipc: true (required!) +host_ipc: enabled automatically (nnodes > 1) ``` **Multi-Node Advanced (4 nodes Γ— 2 GPUs):** @@ -658,7 +658,7 @@ host_ipc: true (required!) GPUs: 2 per node (8 total) Memory: 128Gi per node CPU: 24 per node -host_ipc: true +host_ipc: enabled automatically (nnodes > 1) PVCs: Recommended for data and results ``` @@ -714,7 +714,7 @@ Write durable outputs under `/results//` in the container so each re - `HSA_ENABLE_SDMA=0` - Disable SDMA for better P2P **For multi-node:** -- `host_ipc: true` - Required for shared memory +- Shared memory (`host_ipc`) is enabled automatically when `distributed.nnodes > 1` β€” no config needed - `HSA_FORCE_FINE_GRAIN_PCIE=1` - Cross-node communication - `TORCH_NCCL_ASYNC_ERROR_HANDLING=1` - Better error handling @@ -833,14 +833,7 @@ NCCL WARN ... Unable to find NCCL communicator **Solutions:** -1. **Enable host_ipc:** -```json -{ - "k8s": { - "host_ipc": true // Required for multi-node! - } -} -``` +1. **Host IPC is automatic β€” no config needed:** madengine always enables shared memory (`hostIPC`) for the pod whenever `distributed.nnodes > 1`; there is no `host_ipc` config key to set, and setting one has no effect. If communication is still failing, confirm `nnodes` in your `distributed` config is actually `> 1` (a value of `1` disables host IPC). 2. **Verify headless service:** ```bash @@ -1008,7 +1001,7 @@ Use Case: Multi-GPU training, testing on busy clusters Configuration: 2 nodes Γ— 2 GPUs per node Memory: 64Gi per node CPU: 16 per node -host_ipc: true (required!) +host_ipc: enabled automatically (nnodes > 1) Use Case: Distributed training development ``` @@ -1017,7 +1010,7 @@ Use Case: Distributed training development Configuration: 4 nodes Γ— 2 GPUs per node Memory: 128Gi per node CPU: 24 per node -host_ipc: true +host_ipc: enabled automatically (nnodes > 1) PVCs: Recommended Use Case: Large-scale production training ``` @@ -1031,7 +1024,7 @@ Use Case: Large-scale production training ```bash # Use minimal config (defaults for everything) madengine build --tags dummy \ - --additional-context-file examples/k8s-configs/01-single-node-single-gpu.json \ + --additional-context-file examples/k8s-configs/basic/01-native-single-node-single-gpu.json \ --registry dockerhub madengine run --manifest-file build_manifest.json @@ -1042,7 +1035,7 @@ madengine run --manifest-file build_manifest.json ```bash # Use 2 GPUs to avoid scheduling conflicts madengine build --tags resnet50 \ - --additional-context-file examples/k8s-configs/02-single-node-multi-gpu.json \ + --additional-context-file examples/k8s-configs/basic/02-torchrun-single-node-multi-gpu.json \ --registry dockerhub madengine run --manifest-file build_manifest.json --live-output @@ -1053,7 +1046,7 @@ madengine run --manifest-file build_manifest.json --live-output ```bash # Multi-node for large models madengine build --tags llama_13b \ - --additional-context-file examples/k8s-configs/03-multi-node-basic.json \ + --additional-context-file examples/k8s-configs/basic/03-torchrun-multi-node-basic.json \ --registry dockerhub madengine run --manifest-file build_manifest.json --live-output @@ -1064,7 +1057,7 @@ madengine run --manifest-file build_manifest.json --live-output ```bash # Data provider with auto-PVC madengine build --tags bert_large \ - --additional-context-file examples/k8s-configs/06-data-provider-with-pvc.json \ + --additional-context-file examples/k8s-configs/basic/06-data-provider-with-pvc.json \ --registry dockerhub madengine run --manifest-file build_manifest.json --live-output @@ -1079,7 +1072,7 @@ kubectl exec -- ls -lh /data/ ```bash # Use *-tools.json variant for monitoring madengine build --tags model \ - --additional-context-file examples/k8s-configs/02-single-node-multi-gpu-tools.json \ + --additional-context-file examples/k8s-configs/basic/02-torchrun-single-node-multi-gpu-tools.json \ --registry dockerhub madengine run --manifest-file build_manifest.json --live-output @@ -1096,7 +1089,7 @@ kubectl cp :/results/gpu_info_*.csv ./ ```bash # Copy closest match -cp examples/k8s-configs/02-single-node-multi-gpu.json my-custom-config.json +cp examples/k8s-configs/basic/02-torchrun-single-node-multi-gpu.json my-custom-config.json # Edit vim my-custom-config.json @@ -1216,25 +1209,25 @@ kubectl logs | grep NCCL ## πŸŽ“ Learning Path ### Level 1: Beginner -1. Start with `01-single-node-single-gpu.json` +1. Start with `basic/01-native-single-node-single-gpu.json` 2. Test on single GPU 3. Understand basic K8s concepts 4. Monitor logs and results ### Level 2: Intermediate -1. Try `02-single-node-multi-gpu.json` +1. Try `basic/02-torchrun-single-node-multi-gpu.json` 2. Learn distributed execution with torchrun (training workloads) 3. Understand NCCL configuration 4. Profile GPU utilization ### Level 3: Advanced -1. Deploy `03-multi-node-basic.json` +1. Deploy `basic/03-torchrun-multi-node-basic.json` 2. Master multi-node networking 3. Optimize NCCL parameters 4. Use PVCs for data and results ### Level 4: Expert -1. Customize `04-multi-node-advanced.json` +1. Customize `basic/04-torchrun-multi-node-advanced.json` 2. Fine-tune for your cluster 3. Implement node affinity and tolerations 4. Scale to 8+ nodes @@ -1250,7 +1243,7 @@ Before deploying to production: - [ ] Set appropriate memory and CPU limits - [ ] Configured node selectors (if needed) - [ ] Set NCCL environment variables -- [ ] Enabled `host_ipc` for multi-node +- [ ] Confirmed `distributed.nnodes > 1` for multi-node (host IPC is enabled automatically) - [ ] Tested with small batch size first - [ ] Configured PVCs for data (if using data providers) - [ ] Set up monitoring and logging @@ -1271,15 +1264,17 @@ Before deploying to production: ``` examples/k8s-configs/ -β”œβ”€β”€ README.md # This file -β”œβ”€β”€ 01-single-node-single-gpu.json # 1 GPU, basic -β”œβ”€β”€ 01-single-node-single-gpu-tools.json # 1 GPU + monitoring -β”œβ”€β”€ 02-single-node-multi-gpu.json # 2 GPUs, distributed -β”œβ”€β”€ 02-single-node-multi-gpu-tools.json # 2 GPUs + monitoring -β”œβ”€β”€ 03-multi-node-basic.json # 2 nodes Γ— 2 GPUs -β”œβ”€β”€ 04-multi-node-advanced.json # 4 nodes Γ— 2 GPUs -β”œβ”€β”€ 05-nvidia-gpu-example.json # NVIDIA GPUs -└── 06-data-provider-with-pvc.json # Data provider + auto-PVC +β”œβ”€β”€ README.md # This file +β”œβ”€β”€ basic/ +β”‚ β”œβ”€β”€ 01-native-single-node-single-gpu.json # 1 GPU, basic +β”‚ β”œβ”€β”€ 01-native-single-node-single-gpu-tools.json # 1 GPU + monitoring +β”‚ β”œβ”€β”€ 02-torchrun-single-node-multi-gpu.json # 2 GPUs, distributed +β”‚ β”œβ”€β”€ 02-torchrun-single-node-multi-gpu-tools.json # 2 GPUs + monitoring +β”‚ β”œβ”€β”€ 03-torchrun-multi-node-basic.json # 2 nodes Γ— 2 GPUs +β”‚ β”œβ”€β”€ 04-torchrun-multi-node-advanced.json # 4 nodes Γ— 2 GPUs +β”‚ β”œβ”€β”€ 05-torchrun-nvidia-gpu-example.json # NVIDIA GPUs +β”‚ └── 06-data-provider-with-pvc.json # Data provider + auto-PVC +└── minimal/ # Minimal configs (see minimal/README.md) ``` --- diff --git a/examples/k8s-configs/basic/02-torchrun-single-node-multi-gpu-tools.json b/examples/k8s-configs/basic/02-torchrun-single-node-multi-gpu-tools.json index 3c5f80ae..a231ab14 100644 --- a/examples/k8s-configs/basic/02-torchrun-single-node-multi-gpu-tools.json +++ b/examples/k8s-configs/basic/02-torchrun-single-node-multi-gpu-tools.json @@ -28,7 +28,7 @@ "launcher": "torchrun", "nnodes": 1, "nproc_per_node": 2, - "master_port": 29500 + "port": 29500 }, "env_vars": { diff --git a/examples/k8s-configs/basic/02-torchrun-single-node-multi-gpu.json b/examples/k8s-configs/basic/02-torchrun-single-node-multi-gpu.json index be0d7c5e..3fdf6d26 100644 --- a/examples/k8s-configs/basic/02-torchrun-single-node-multi-gpu.json +++ b/examples/k8s-configs/basic/02-torchrun-single-node-multi-gpu.json @@ -27,7 +27,7 @@ "launcher": "torchrun", "nnodes": 1, "nproc_per_node": 2, - "master_port": 29500 + "port": 29500 }, "env_vars": { diff --git a/examples/k8s-configs/basic/03-torchrun-multi-node-basic.json b/examples/k8s-configs/basic/03-torchrun-multi-node-basic.json index 0c2205f9..6db11b10 100644 --- a/examples/k8s-configs/basic/03-torchrun-multi-node-basic.json +++ b/examples/k8s-configs/basic/03-torchrun-multi-node-basic.json @@ -28,7 +28,7 @@ "launcher": "torchrun", "nnodes": 2, "nproc_per_node": 2, - "master_port": 29500 + "port": 29500 }, "env_vars": { diff --git a/examples/k8s-configs/basic/04-torchrun-multi-node-advanced.json b/examples/k8s-configs/basic/04-torchrun-multi-node-advanced.json index 5560ffab..bee06a54 100644 --- a/examples/k8s-configs/basic/04-torchrun-multi-node-advanced.json +++ b/examples/k8s-configs/basic/04-torchrun-multi-node-advanced.json @@ -54,7 +54,7 @@ "launcher": "torchrun", "nnodes": 4, "nproc_per_node": 2, - "master_port": 29500 + "port": 29500 }, "env_vars": { diff --git a/examples/k8s-configs/basic/05-torchrun-nvidia-gpu-example.json b/examples/k8s-configs/basic/05-torchrun-nvidia-gpu-example.json index 7c087acc..6d2f37d0 100644 --- a/examples/k8s-configs/basic/05-torchrun-nvidia-gpu-example.json +++ b/examples/k8s-configs/basic/05-torchrun-nvidia-gpu-example.json @@ -31,7 +31,7 @@ "launcher": "torchrun", "nnodes": 1, "nproc_per_node": 4, - "master_port": 29500 + "port": 29500 }, "env_vars": { diff --git a/examples/k8s-configs/basic/06-data-provider-with-pvc.json b/examples/k8s-configs/basic/06-data-provider-with-pvc.json index 9bd2e47f..4c52f17e 100644 --- a/examples/k8s-configs/basic/06-data-provider-with-pvc.json +++ b/examples/k8s-configs/basic/06-data-provider-with-pvc.json @@ -36,7 +36,7 @@ "nnodes": 1, "nproc_per_node": 2, - "master_port": 29500 + "port": 29500 }, "env_vars": { diff --git a/examples/k8s-configs/basic/megatron-lm-multi-node-basic.json b/examples/k8s-configs/basic/megatron-lm-multi-node-basic.json index e059ba08..b0c0efd5 100644 --- a/examples/k8s-configs/basic/megatron-lm-multi-node-basic.json +++ b/examples/k8s-configs/basic/megatron-lm-multi-node-basic.json @@ -18,10 +18,10 @@ }, "distributed": { - "launcher": "megatron", + "launcher": "megatron-lm", "nnodes": 4, "nproc_per_node": 8, - "master_port": 29500 + "port": 29500 }, "env_vars": { diff --git a/examples/k8s-configs/basic/sglang-disagg-custom-split.json b/examples/k8s-configs/basic/sglang-disagg-custom-split.json index 49aeecb1..97cd7509 100644 --- a/examples/k8s-configs/basic/sglang-disagg-custom-split.json +++ b/examples/k8s-configs/basic/sglang-disagg-custom-split.json @@ -29,7 +29,7 @@ "launcher": "sglang-disagg", "nnodes": 7, "nproc_per_node": 8, - "master_port": 29500, + "port": 29500, "sglang_disagg": { "prefill_nodes": 4, "decode_nodes": 2 diff --git a/examples/k8s-configs/basic/sglang-disagg-multi-node-basic.json b/examples/k8s-configs/basic/sglang-disagg-multi-node-basic.json index c16fd342..25096a9e 100644 --- a/examples/k8s-configs/basic/sglang-disagg-multi-node-basic.json +++ b/examples/k8s-configs/basic/sglang-disagg-multi-node-basic.json @@ -28,7 +28,7 @@ "launcher": "sglang-disagg", "nnodes": 5, "nproc_per_node": 8, - "master_port": 29500 + "port": 29500 }, "context": { diff --git a/examples/k8s-configs/basic/sglang-multi-node-basic.json b/examples/k8s-configs/basic/sglang-multi-node-basic.json index b693260e..59dab1af 100644 --- a/examples/k8s-configs/basic/sglang-multi-node-basic.json +++ b/examples/k8s-configs/basic/sglang-multi-node-basic.json @@ -22,7 +22,7 @@ "launcher": "sglang", "nnodes": 2, "nproc_per_node": 4, - "master_port": 29500 + "port": 29500 }, "context": { diff --git a/examples/k8s-configs/basic/torchtitan-multi-node-basic.json b/examples/k8s-configs/basic/torchtitan-multi-node-basic.json index e350605d..db4d1190 100644 --- a/examples/k8s-configs/basic/torchtitan-multi-node-basic.json +++ b/examples/k8s-configs/basic/torchtitan-multi-node-basic.json @@ -22,7 +22,7 @@ "launcher": "torchtitan", "nnodes": 4, "nproc_per_node": 8, - "master_port": 29500 + "port": 29500 }, "context": { diff --git a/examples/k8s-configs/basic/vllm-multi-node-basic.json b/examples/k8s-configs/basic/vllm-multi-node-basic.json index 4c1b61c9..eb9f4cae 100644 --- a/examples/k8s-configs/basic/vllm-multi-node-basic.json +++ b/examples/k8s-configs/basic/vllm-multi-node-basic.json @@ -22,7 +22,7 @@ "launcher": "vllm", "nnodes": 2, "nproc_per_node": 4, - "master_port": 29500 + "port": 29500 }, "context": { diff --git a/examples/k8s-configs/minimal/megatron-lm-exclude-node.json b/examples/k8s-configs/minimal/megatron-lm-exclude-node.json index 793431a2..09e69e8b 100644 --- a/examples/k8s-configs/minimal/megatron-lm-exclude-node.json +++ b/examples/k8s-configs/minimal/megatron-lm-exclude-node.json @@ -24,7 +24,7 @@ }, "distributed": { - "launcher": "megatron", + "launcher": "megatron-lm", "nnodes": 1, "nproc_per_node": 2 }, diff --git a/examples/k8s-configs/minimal/megatron-lm-minimal.json b/examples/k8s-configs/minimal/megatron-lm-minimal.json index 43266e01..d033a26d 100644 --- a/examples/k8s-configs/minimal/megatron-lm-minimal.json +++ b/examples/k8s-configs/minimal/megatron-lm-minimal.json @@ -14,7 +14,7 @@ }, "distributed": { - "launcher": "megatron", + "launcher": "megatron-lm", "nnodes": 1, "nproc_per_node": 2 }, diff --git a/examples/k8s-configs/minimal/megatron-lm-optimized.json b/examples/k8s-configs/minimal/megatron-lm-optimized.json index 29559308..bfeba1b4 100644 --- a/examples/k8s-configs/minimal/megatron-lm-optimized.json +++ b/examples/k8s-configs/minimal/megatron-lm-optimized.json @@ -29,10 +29,10 @@ "distributed": { "enabled": true, "backend": "nccl", - "launcher": "megatron", + "launcher": "megatron-lm", "nnodes": 1, "nproc_per_node": 2, - "master_port": 29500 + "port": 29500 }, "env_vars": { diff --git a/examples/slurm-configs/README.md b/examples/slurm-configs/README.md index e89f016f..0100fe54 100644 --- a/examples/slurm-configs/README.md +++ b/examples/slurm-configs/README.md @@ -42,7 +42,7 @@ The deployment type is **inferred** from the configuration structure: | File | Description | Nodes | GPUs | Use Case | |------|-------------|-------|------|----------| -| `01-torchrun-single-node-single-gpu.json` | Single GPU training | 1 | 1 | Quick tests, small models | +| `01-single-node-single-gpu.json` | Single GPU training | 1 | 1 | Quick tests, small models | | `02-single-node-multi-gpu.json` | Single node, 8 GPUs | 1 | 8 | Single-node distributed workload | | `03-multi-node-basic.json` | 2 nodes, 8 GPUs each | 2 | 16 | Multi-node distributed workload | | `03-multi-node-basic-nodelist.json` | Same as 03 with `nodelist` | 2 | 16 | Pin job to specific nodes (e.g. node01,node02) | @@ -58,15 +58,13 @@ The deployment type is **inferred** from the configuration structure: ### Minimal Examples (`minimal/`) Stripped-down configurations showing only essential fields: -- `single-gpu-minimal.json` - Minimal single GPU config -- `multi-gpu-minimal.json` - Minimal 8 GPU config -- `multi-node-minimal.json` - Minimal 2-node config -- `primus-minimal.json` - Minimal Primus pretrain (`distributed.launcher: "primus"`; edit `primus.config_path`) +- `torchrun-single-gpu-minimal.json` - Minimal single GPU config +- `torchrun-multi-gpu-minimal.json` - Minimal 8 GPU config +- `torchrun-multi-node-minimal.json` - Minimal 2-node config - `vllm-single-node-minimal.json` - Minimal vLLM single-node - `vllm-multi-node-minimal.json` - Minimal vLLM multi-node - `slurm-multi-minimal.json` - Minimal slurm_multi self-managed launcher (3 nodes) -For Primus options and environment variables, see [Launchers Guide](../../docs/launchers.md#5-primus). For slurm_multi escape-hatch launcher, see [Launchers Guide](../../docs/launchers.md#9-slurm_multi-self-managed-escape-hatch). ## πŸ”„ Configuration Workflow @@ -127,7 +125,7 @@ ssh user@hpc-cluster.example.com # Phase 1: Build with configuration MODEL_DIR=models/my-model madengine build \ --tags model_tag \ - --additional-context-file examples/slurm-configs/03-multi-node-basic.json \ + --additional-context-file examples/slurm-configs/basic/03-multi-node-basic.json \ --manifest-output build_manifest.json # Phase 2: Run from manifest @@ -146,7 +144,7 @@ For quick tests without custom `env_vars`: ```bash madengine run --tags model_tag \ - --additional-context-file examples/slurm-configs/minimal/single-gpu-minimal.json + --additional-context-file examples/slurm-configs/minimal/torchrun-single-gpu-minimal.json ``` ### 3. CLI Override @@ -168,7 +166,7 @@ madengine run --tags model_tag \ ```bash # Use base config, override specific fields madengine run --tags model_tag \ - --additional-context-file examples/slurm-configs/03-multi-node-basic.json \ + --additional-context-file examples/slurm-configs/basic/03-multi-node-basic.json \ --additional-context '{"slurm": {"nodes": 4, "time": "48:00:00"}}' ``` @@ -392,13 +390,23 @@ madengine uses intelligent multi-layer configuration merging: "qos": "high", // Quality of Service "account": "project-name", // SLURM account "network_interface": "ib0", // Network interface (ib0/eth0) - "modules": ["rocm/5.7.0"] // Environment modules to load + "modules": ["rocm/5.7.0"], // Environment modules to load + "enable_node_check": true, // Run node health preflight before submission (default: true) + "auto_cleanup_nodes": false, // Automatically clean up unhealthy nodes found during preflight (default: false) + "allow_submit_without_clean_nodes": false, // Submit even if fewer clean nodes than requested are found (default: false) + "verbose_node_check": false // Print detailed node health-check output (default: false) } } ``` **nodelist**: When set to a comma-separated list of node names (e.g. `"node01,node02"`), the job runs only on those nodes. Automatic node health preflight is skipped when `nodelist` is set. +**Node Health Preflight**: Before submitting multi-node jobs, madengine runs a health check across candidate nodes and pins the job to the healthy ones (skipped if `nodelist` is set). +- **`enable_node_check`** (default `true`): Enables/disables the preflight health check. +- **`auto_cleanup_nodes`** (default `false`): Automatically attempts to clean up unhealthy nodes found during preflight. +- **`allow_submit_without_clean_nodes`** (default `false`): If fewer clean nodes are found than requested, submit anyway instead of failing. +- **`verbose_node_check`** (default `false`): Print detailed health-check output during preflight. + ### Distributed Execution Section ```json @@ -452,7 +460,7 @@ For slurm_multi, the model's `.slurm` script runs on baremetal and manages Docke ```bash madengine run --tags my_model \ - --additional-context-file examples/slurm-configs/minimal/single-gpu-minimal.json + --additional-context-file examples/slurm-configs/minimal/torchrun-single-gpu-minimal.json ``` ### Multi-Node Training @@ -461,7 +469,7 @@ madengine run --tags my_model \ # Build with config MODEL_DIR=models/my-model madengine build \ --tags training \ - --additional-context-file examples/slurm-configs/03-multi-node-basic.json + --additional-context-file examples/slurm-configs/basic/03-multi-node-basic.json # Run from manifest MODEL_DIR=models/my-model madengine run \ @@ -498,7 +506,7 @@ MODEL_DIR=models/llama2-70b madengine run \ ```bash madengine build --tags my_model \ - --additional-context-file examples/slurm-configs/04-multi-node-advanced.json + --additional-context-file examples/slurm-configs/basic/04-multi-node-advanced.json madengine run --manifest-file build_manifest.json ``` @@ -759,7 +767,7 @@ module load python/3.9 # 3. Build with configuration MODEL_DIR=models/my-model madengine build \ --tags llama2_training \ - --additional-context-file examples/slurm-configs/03-multi-node-basic.json \ + --additional-context-file examples/slurm-configs/basic/03-multi-node-basic.json \ --manifest-output build_manifest.json # 4. Run from manifest diff --git a/examples/slurm-configs/basic/09-megatron-lm-multi-node.json b/examples/slurm-configs/basic/09-megatron-lm-multi-node.json index 84e3c3f6..bb718a20 100644 --- a/examples/slurm-configs/basic/09-megatron-lm-multi-node.json +++ b/examples/slurm-configs/basic/09-megatron-lm-multi-node.json @@ -17,10 +17,10 @@ }, "distributed": { - "launcher": "megatron", + "launcher": "megatron-lm", "nnodes": 4, "nproc_per_node": 8, - "master_port": 29500 + "port": 29500 }, "env_vars": { diff --git a/examples/slurm-configs/minimal/megatron-lm-minimal.json b/examples/slurm-configs/minimal/megatron-lm-minimal.json index 9480359e..3461f71a 100644 --- a/examples/slurm-configs/minimal/megatron-lm-minimal.json +++ b/examples/slurm-configs/minimal/megatron-lm-minimal.json @@ -14,7 +14,7 @@ }, "distributed": { - "launcher": "megatron", + "launcher": "megatron-lm", "nnodes": 1, "nproc_per_node": 2 }, diff --git a/src/madengine/database/README.md b/src/madengine/database/README.md index 2c8e5f9f..eec76ccb 100644 --- a/src/madengine/database/README.md +++ b/src/madengine/database/README.md @@ -1,81 +1,152 @@ -# Database Layer (Future MongoDB Ingestion) +# Database Layer -**Status**: Planned for future development -**Purpose**: Modern data ingestion API for local and distributed deployments +**Status**: Active +**Purpose**: Upload CSV/JSON performance results to MongoDB --- -## 🎯 Objective +## 🎯 Responsibility -This directory is reserved for a future unified database ingestion layer that will support: -- MongoDB data persistence -- Local result storage -- Distributed data collection from build and run phases -- Unified API for performance metrics ingestion +This module implements MongoDB ingestion for madengine results (e.g. `perf.csv`, +`perf_entry.csv`, or arbitrary JSON documents). It is a single, self-contained +file β€” `mongodb.py` β€” with no sub-packages. It handles: ---- +- Auto-detecting file format (CSV vs JSON) +- Loading files with native type preservation (numbers, bools, nested + JSON-in-CSV-cell strings) +- Transforming/normalizing documents (metadata stamping, numpy/pandas type + cleanup) +- Auto-detecting unique fields for deduplication when not specified +- Bulk upload to MongoDB with batching, upsert, and automatic indexing -## πŸ“‹ Current State +It is wired directly into the CLI via `madengine database` (see +`src/madengine/cli/commands/database.py`). -⚠️ **Not yet implemented**. This directory is a placeholder for future development. +--- -For current database operations, use the existing `db/` package which handles MySQL operations via SSH. +## πŸ“¦ Components (`mongodb.py`) + +### Configuration + +- **`MongoDBConfig`** β€” dataclass holding `host`, `port`, `username`, + `password`, `auth_source`, `timeout_ms`. `MongoDBConfig.from_env()` builds + one from `MONGO_HOST`, `MONGO_PORT`, `MONGO_USER`, `MONGO_PASSWORD`, + `MONGO_AUTH_SOURCE`, `MONGO_TIMEOUT_MS`. The `.uri` property builds the + `mongodb://` connection string. +- **`UploadOptions`** β€” dataclass controlling upload behavior: + `unique_fields`, `upsert`, `batch_size`, `ordered`, `create_indexes`, + `index_fields`, `add_metadata`, `metadata_prefix`, `validate_schema` + (reserved field, currently unused by the implementation), `dry_run`. +- **`UploadResult`** β€” dataclass returned by every upload: `status` + (`"success"` / `"partial"` / `"failed"`), `documents_read`, + `documents_processed`, `documents_inserted`, `documents_updated`, + `documents_failed`, `errors`, `duration_seconds`. Has a + `print_summary()` method that renders a formatted Rich summary. + +### File loading (Strategy pattern) + +- **`DocumentLoader`** (ABC) β€” defines `load(file_path)` and + `infer_schema(documents)`. +- **`JSONLoader`** β€” loads a JSON object or array of objects, preserving + native types. +- **`CSVLoader`** β€” loads via `pandas.read_csv`, preserving native types and + attempting to parse cell values that look like JSON (`{...}` / `[...]`) + back into dicts/lists. +- **`detect_file_format(file_path)`** β€” picks `FileFormat.CSV` or + `FileFormat.JSON` from the extension, falling back to content sniffing. +- **`get_loader(file_format)`** β€” returns the right loader instance. + +### Transformation + +- **`DocumentTransformer`** β€” takes `UploadOptions` and: + - `transform(documents)` adds metadata (`_meta_uploaded_at`, + `created_date`) and normalizes types (numpy scalars β†’ Python, pandas + `Timestamp` β†’ `datetime`, `NaN` β†’ `None`). + - `infer_unique_fields(documents)` guesses a dedup key by checking + candidate fields (`model`, `name`, `id`, `timestamp`, `date`, + `pipeline`) for uniqueness across a sample of documents. + +### Upload + +- **`MongoDBUploader`** β€” connection + bulk-write class, usable as a context + manager (`with MongoDBUploader(config) as uploader:`). + - `connect()` / `disconnect()` + - `upload(documents, database_name, collection_name, options)` β†’ + `UploadResult`. Creates indexes (if `options.create_indexes`) then does + either a plain `insert_many` (when no `unique_fields`/`upsert`) or a + batched `bulk_write` of `UpdateOne(..., upsert=True)` operations keyed + on `unique_fields`. + +### Entry points + +- **`upload_file_to_mongodb(file_path, database_name, collection_name, config=None, options=None) -> UploadResult`** + β€” the main entry point. Detects format, loads, auto-infers unique fields + if not given, transforms, honors `dry_run` (returns without connecting to + MongoDB), then uploads. +- **`upload_csv_to_mongodb(csv_file_path, database_name, collection_name, mongo_config=None) -> Dict[str, Any]`** + β€” deprecated wrapper around `upload_file_to_mongodb` that returns a legacy + dict shape instead of `UploadResult`. +- **`MongoDBHandler`** β€” deprecated class-based wrapper (`MongoDBHandler(args).run() -> bool`) + kept for backward compatibility with old argparse-style call sites. --- -## πŸ—‚οΈ Legacy MySQL Tools (Removed) +## πŸ”— CLI mapping (`madengine database`) -**MySQL support has been removed from madengine**. The following tools are no longer available: +`src/madengine/cli/commands/database.py` is a thin Typer wrapper around +`upload_file_to_mongodb`: -| File | Purpose | Status | -|------|---------|--------| -| ~~`tools/create_table_db.py`~~ | MySQL table creation | **REMOVED** | -| ~~`tools/update_table_db.py`~~ | MySQL table updates | **REMOVED** | -| ~~`db/` package~~ | MySQL operations via SSH | **REMOVED** | +| CLI flag | Maps to | +|---|---| +| `--file` / `-f` | `file_path` | +| `--database` / `--db` | `database_name` | +| `--collection` / `-c` | `collection_name` | +| `--unique-key` / `-k` (comma-separated) | `UploadOptions.unique_fields` | +| `--batch-size` | `UploadOptions.batch_size` | +| `--no-upsert` | `UploadOptions.upsert = False` | +| `--no-index` | `UploadOptions.create_indexes = False` | +| `--dry-run` | `UploadOptions.dry_run` | -For database operations, use MongoDB via the `database` command in the new CLI or legacy `mad.py`. +Connection config always comes from `MongoDBConfig.from_env()` (the CLI does +not expose host/port/credential flags β€” set `MONGO_HOST`, `MONGO_PORT`, +`MONGO_USER`, `MONGO_PASSWORD` instead). See `madengine database --help` or +`docs/cli-reference.md` for the full flag reference. --- -## πŸš€ Future Implementation Plan - -When implemented, this layer will provide: - -### **1. MongoDB Client** (`mongodb_client.py`) -```python -from madengine.database.mongodb_client import MongoDBClient +## πŸš€ Usage -# Connect to local or remote MongoDB -client = MongoDBClient(connection_string="mongodb://localhost:27017") +**Via the CLI:** -# Ingest build results -client.ingest_build_results(build_manifest) +```bash +export MONGO_HOST=localhost +export MONGO_USER=admin +export MONGO_PASSWORD=secret -# Ingest run results -client.ingest_run_results(run_summary) +madengine database -f perf.csv --db madengine --collection results -k model,timestamp +madengine database -f perf_entry.json --db madengine --collection results --dry-run ``` -### **2. Local Storage** (`local_storage.py`) -```python -from madengine.database.local_storage import LocalStorage - -# Store results locally (JSON, Parquet, etc.) -storage = LocalStorage(base_path="./madengine_results") -storage.save_results(results_dict) -``` +**Via the Python API:** -### **3. Unified API** (`api.py`) ```python -from madengine.database import ingest_results - -# Works with both local and distributed deployments -ingest_results( - results=run_summary, - target="mongodb", # or "local", "mysql" - config={"connection": "mongodb://..."} +from madengine.database import upload_file_to_mongodb, MongoDBConfig, UploadOptions + +result = upload_file_to_mongodb( + file_path="perf.csv", + database_name="madengine", + collection_name="results", + config=MongoDBConfig.from_env(), + options=UploadOptions(unique_fields=["model", "timestamp"], batch_size=500), ) + +result.print_summary() +print(result.status, result.documents_inserted, result.documents_updated) ``` +`config` and `options` are both optional β€” omit them to use +`MongoDBConfig.from_env()` and default `UploadOptions()`. + --- ## πŸ“¦ Difference from `db/` Package (Removed) @@ -87,28 +158,23 @@ ingest_results( | **Transport** | SSH tunnel | Direct connection | | **Status** | **REMOVED** | Active | ---- - -## πŸ”„ Migration Status - MySQL support has been fully removed from madengine: -1. βœ… **Phase 1**: Removed `db/` package (MySQL operations) -2. βœ… **Phase 2**: Removed `tools/create_table_db.py` and `tools/update_table_db.py` -3. βœ… **Phase 3**: Removed `utils/ssh_to_db.py` (SSH to MySQL host) -4. βœ… **Phase 4**: Removed MySQL dependencies (`mysql-connector-python`, `pymysql`) +1. βœ… Removed `db/` package (MySQL operations) +2. βœ… Removed `tools/create_table_db.py` and `tools/update_table_db.py` +3. βœ… Removed `utils/ssh_to_db.py` (SSH to MySQL host) +4. βœ… Removed MySQL dependencies (`mysql-connector-python`, `pymysql`) -**Current state**: Only MongoDB support remains via the `database/` package. +**Current state**: Only MongoDB support remains, via this `database/` package. --- ## πŸ“š References -- **MongoDB package**: `src/madengine/database/mongodb.py` -- **CLI database command**: `madengine database --help` +- **Implementation**: `src/madengine/database/mongodb.py` +- **CLI command**: `src/madengine/cli/commands/database.py`, `madengine database --help` --- -**Last Updated**: November 30, 2025 +**Last Updated**: 2026-08-05 **Maintainer**: madengine Team - diff --git a/src/madengine/execution/README.md b/src/madengine/execution/README.md index 1277cfa3..6a51f274 100644 --- a/src/madengine/execution/README.md +++ b/src/madengine/execution/README.md @@ -44,7 +44,7 @@ result = builder.build_image( # Build all models results = builder.build_all_models( - models_list=[model1, model2, model3], + models=[model1, model2, model3], target_archs=["gfx90a", "gfx942"] ) @@ -52,6 +52,10 @@ results = builder.build_all_models( builder.export_build_manifest(output_file="build_manifest.json") ``` +### **`dockerfile_utils.py`** + +Helper functions for multi-architecture Dockerfile parsing (e.g., `parse_dockerfile_gpu_variables`, `normalize_architecture_name`, `is_target_arch_compatible_with_variable`, `is_compilation_arch_compatible`) used by `docker_builder.py`. + ### **`container_runner.py`** Runs Docker containers locally for model execution. @@ -73,16 +77,19 @@ runner = ContainerRunner(context, data, console) # Run model in container result = runner.run_container( model_info=model_dict, - model_docker=docker_client, - gpu_ids="0,1", + docker_image="model1:latest", timeout=3600 ) # Result includes status, metrics, logs -print(result["status"]) # "successful", "failed", "timeout" -print(result["duration"]) +print(result["status"]) # "SUCCESS", "FAILURE", "SKIPPED" +print(result["test_duration"]) ``` +### **`container_runner_helpers.py`** + +Backs `container_runner.py`'s timeout management and error-detection features (e.g., `resolve_log_error_scan_config`, `log_text_has_error_pattern`, `resolve_run_status`, `resolve_run_timeout`, `make_run_log_file_path`). + --- ## πŸ—οΈ Architecture diff --git a/src/madengine/reporting/README.md b/src/madengine/reporting/README.md index 33d8e5a4..ac3ec49e 100644 --- a/src/madengine/reporting/README.md +++ b/src/madengine/reporting/README.md @@ -26,43 +26,41 @@ from madengine.reporting.update_perf_csv import update_perf_csv, flatten_tags # Update CSV with new results update_perf_csv( - perf_json_path="results.json", - output_csv="performance.csv" + perf_csv="performance.csv", + single_result="results.json", ) -# Flatten nested tags for CSV export -flattened = flatten_tags(perf_entry) +# Flatten nested tags in place (no return value) +flatten_tags(perf_entry) # mutates perf_entry in place ``` ---- - -## πŸ—‚οΈ Legacy Reporting Tools +### **`update_perf_super.py`** -The following legacy-only reporting tools remain in `tools/`: +Maintains `perf_super.json`, a cumulative superset performance record that also +captures matched config data, and provides conversion to `perf_super.csv` / +`perf_entry_super.json` / `perf_entry_super.csv`. -| File | Purpose | Used By | Status | -|------|---------|---------|--------| -| `tools/csv_to_html.py` | Convert CSV to HTML | `mad.py`, `run_models.py` | Legacy only | -| `tools/csv_to_email.py` | Email CSV reports | `mad.py` | Legacy only | +**Used by:** +- βœ… `execution/container_runner.py` (modern madengine CLI) -These tools are **NOT** used by the modern `madengine` CLI. +### **`csv_to_html.py`** ---- +Converts a single CSV file to an HTML table. Provides the `ConvertCsvToHtml` +handler class (`ConvertCsvToHtml.__init__(self, args: argparse.Namespace)`, +`.run(self) -> bool`) used by the CLI. -## πŸ“‹ Architecture Decision - -**Why is `update_perf_csv.py` in `reporting/` instead of `tools/`?** +**Used by:** +- βœ… `cli/commands/report.py` (backs `madengine report to-html`) -1. βœ… **Shared across architectures**: Used by both legacy and new CLI -2. βœ… **Active development**: Not deprecated, actively maintained -3. βœ… **Clear responsibility**: Performance data processing -4. βœ… **Semantic clarity**: Reporting is a distinct concern +### **`csv_to_email.py`** -**Why are other CSV tools still in `tools/`?** +Converts all CSV files in a directory into a single consolidated HTML report +suitable for emailing. Provides the `ConvertCsvToEmail` handler class +(`ConvertCsvToEmail.__init__(self, args: argparse.Namespace)`, `.run(self) -> bool`) +used by the CLI. -- They are **not used** by the modern `madengine` CLI -- Kept for backward compatibility only -- Will be deprecated when legacy CLI is retired +**Used by:** +- βœ… `cli/commands/report.py` (backs `madengine report to-email`) --- @@ -74,10 +72,10 @@ These tools are **NOT** used by the modern `madengine` CLI. from madengine.reporting.update_perf_csv import update_perf_csv # After model execution completes +perf_csv = "/path/to/performance.csv" results_json = "/path/to/results.json" -output_csv = "/path/to/performance.csv" -update_perf_csv(results_json, output_csv) +update_perf_csv(perf_csv, single_result=results_json) ``` ### **Legacy madengine** (via `run_models.py` or `mad.py`) @@ -107,6 +105,10 @@ Performance CSV (Optional) CSV β†’ Email (legacy only) ``` +**Note:** As a side effect, `update_perf_csv()` (and the `handle_*_result()` helpers it +calls) also always write/append `perf_entry.csv` and `perf_entry.json` with the +latest result, regardless of the output file passed in. + --- ## πŸ§ͺ Testing