diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 687228b..66016a2 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -169,7 +169,7 @@ jobs: run: | modelforge deployments --environment ci-demo > cli-deployments.json modelforge predict ci-demo 10 > cli-prediction.json - - name: Run recruiter deployment proof + - name: Validate deployment lifecycle run: >- python demo/proof_scenario.py --environment ci-proof diff --git a/.github/workflows/demo-proof.yml b/.github/workflows/demo-proof.yml index 53eb4df..888b24a 100644 --- a/.github/workflows/demo-proof.yml +++ b/.github/workflows/demo-proof.yml @@ -6,7 +6,7 @@ on: environment: description: "ModelForge serving environment name used for the proof." required: false - default: "recruiter-demo" + default: "validation-demo" permissions: contents: read diff --git a/.gitignore b/.gitignore index b671fc9..ec50201 100644 --- a/.gitignore +++ b/.gitignore @@ -49,5 +49,5 @@ override.tf.json *_override.tf *_override.tf.json -# Generated demo proof evidence +# Generated deployment validation evidence demo-proof-evidence/ diff --git a/Makefile b/Makefile index ac8f58d..92fb5fa 100644 --- a/Makefile +++ b/Makefile @@ -26,7 +26,7 @@ demo: python demo/bootstrap_demo.py --environment demo --weight 20 proof: - python demo/proof_scenario.py --environment recruiter-demo --output-dir demo-proof-evidence + python demo/proof_scenario.py --environment validation-demo --output-dir demo-proof-evidence benchmark: python benchmarks/bootstrap_external.py --environment benchmark-go diff --git a/README.md b/README.md index 1a78f56..1e23275 100644 --- a/README.md +++ b/README.md @@ -391,7 +391,7 @@ be captured on controlled hardware. The latest validation also smoke-tests the installed CLI, public landing page, multi-tenant SaaS console, auth configuration, Docker Compose stack, external Go -runtime, and the full recruiter deployment proof. In that proof, v2 received +runtime, and the full deployment validation scenario. In that scenario, v2 received real weighted canary traffic and was promoted, v1 was restored through rollback, and an intentionally regressed v3 produced 185.7% output drift against a 50% threshold, was recorded FAILED, and left v1 as the authoritative final target. @@ -410,8 +410,8 @@ This creates the foundation for measuring reliability and performance rather tha - [External runtime contract](docs/runtime-protocol.md) - [Public-cloud deployment contract](docs/cloud-deployment.md) - [AWS ECS/Fargate reference deployment](infra/aws/README.md) -- [Recruiter deployment proof](docs/demo-proof.md) -- [Recruiter and interview brief](docs/recruiting.md) +- [Deployment validation scenario](docs/demo-proof.md) +- [Platform overview](docs/platform-overview.md) Common workflows are available through the root `Makefile`: diff --git a/demo/proof_scenario.py b/demo/proof_scenario.py index bd5b1e5..972cee0 100644 --- a/demo/proof_scenario.py +++ b/demo/proof_scenario.py @@ -1,4 +1,4 @@ -"""Run and capture a recruiter-visible ModelForge deployment proof.""" +"""Run and capture a ModelForge deployment validation scenario.""" from __future__ import annotations @@ -256,7 +256,7 @@ def run_proof( json={ "name": model_name, "description": ( - "Recruiter-proof model for canary promotion, rollback, " + "Validation model for canary promotion, rollback, " "and regression abort." ), }, @@ -575,7 +575,7 @@ def write_evidence( summary: dict[str, Any], samples: list[dict[str, Any]], ) -> None: - """Write machine-readable and recruiter-readable proof artifacts.""" + """Write machine-readable evidence and a human-readable report.""" output_dir.mkdir(parents=True, exist_ok=True) (output_dir / "proof-summary.json").write_text( diff --git a/docs/demo-proof.md b/docs/demo-proof.md index 7e1d068..2147120 100644 --- a/docs/demo-proof.md +++ b/docs/demo-proof.md @@ -1,4 +1,4 @@ -# Recruiter Demo Proof +# Deployment Validation Scenario This scenario is the shortest way to demonstrate ModelForge as a working ML deployment platform rather than a collection of API endpoints. @@ -31,7 +31,7 @@ docker compose up --build --wait --detach Run the proof: ~~~bash -python demo/proof_scenario.py --environment recruiter-demo --output-dir demo-proof-evidence +python demo/proof_scenario.py --environment validation-demo --output-dir demo-proof-evidence ~~~ The command exits non-zero if any lifecycle assertion fails. @@ -45,7 +45,7 @@ export MODELFORGE_URL="https://YOUR_MODELFORGE_HOST" export MODELFORGE_API_KEY="mf_live_..." export MODELFORGE_WORKSPACE="YOUR_WORKSPACE_SLUG" -python demo/proof_scenario.py --environment recruiter-demo --output-dir demo-proof-evidence +python demo/proof_scenario.py --environment validation-demo --output-dir demo-proof-evidence ~~~ The raw API key is used only as an HTTP Authorization header. It is never @@ -57,8 +57,7 @@ The output directory contains: - proof-summary.json — compact machine-readable lifecycle evidence; - traffic-samples.jsonl — request-level model-version and traffic-lane proof; -- proof-report.md — recruiter-readable summary suitable for a GitHub Actions - artifact or README screenshot. +- proof-report.md — human-readable summary of the deployment lifecycle checks. The proof deliberately includes both a successful and unsuccessful release: diff --git a/docs/platform-overview.md b/docs/platform-overview.md new file mode 100644 index 0000000..313f427 --- /dev/null +++ b/docs/platform-overview.md @@ -0,0 +1,39 @@ +# Platform Overview + +ModelForge manages the lifecycle of machine learning models from artifact +registration through deployment and inference. It combines a FastAPI control +plane, MySQL-backed model and deployment state, a browser console, an +authenticated CLI, and a standalone Go inference runtime. + +## Model lifecycle + +Model versions reference immutable artifacts with checksums. Deployments have +explicit lifecycle states, and each serving environment has an authoritative +target. Weighted canaries can receive live traffic before promotion. Operators +can abort a canary or roll back to a previous stable version. + +## Runtime integration + +The control plane supports in-process PyTorch and ONNX runtimes and an external +runtime protocol. The Go service implements that protocol in a separate +process. Other model frameworks can integrate through the same contract +without adding framework-specific inference logic to the control plane. + +## Reliability and access + +The serving path includes bounded retries, circuit breakers, health checks, +automatic stable fallback, and Prometheus metrics. Workspaces scope data and +API keys. Browser access uses OIDC Authorization Code with PKCE, opaque +sessions, CSRF protection, and a strict Content Security Policy. + +## Validation and deployment + +The repository includes a reproducible [deployment validation +scenario](demo-proof.md) covering stable serving, weighted canary traffic, +promotion, rollback, regression detection, failed-canary abort, and final +stable-target preservation. Benchmark results in the README were measured in +CI and should not be treated as production capacity estimates. + +The [AWS reference deployment](../infra/aws/README.md) uses Terraform for +ECS/Fargate, RDS MySQL, S3, and supporting infrastructure. It is a deployment +reference; the public service has not yet been launched. diff --git a/docs/recruiting.md b/docs/recruiting.md deleted file mode 100644 index 13a02bc..0000000 --- a/docs/recruiting.md +++ /dev/null @@ -1,84 +0,0 @@ -# ModelForge Recruiting Brief - -## 30-second pitch - -ModelForge is a reliability-aware ML deployment platform I built to explore the systems problems that appear after model training. It combines a Python/FastAPI control plane, MySQL-backed model and deployment state, framework-specific in-process runtimes, and a standalone Go inference runtime behind a framework-agnostic protocol. The platform supports immutable artifacts, transactional promotion and rollback, weighted canary releases, retries, circuit breakers, automatic stable fallback, Prometheus metrics, S3-compatible artifact storage, multi-tenant workspaces, secure OIDC browser sessions, a public SaaS console, an authenticated CLI, production configuration preflight, and a validated Terraform AWS reference deployment. The current post-merge CI checkpoint passes 164 Python tests and validates the full stack end to end. - -## Resume bullets - -- Built **ModelForge**, a reliability-aware ML deployment platform in **Python/FastAPI, MySQL, SQLAlchemy, Docker, and Go**, supporting immutable model versioning, deployment lifecycle management, PyTorch/ONNX execution, and framework-independent external runtimes. -- Designed a framework-agnostic runtime boundary with **bounded retries, exponential backoff, circuit breakers, health checks, Prometheus metrics, weighted canary releases, rollback, and automatic stable fallback** so future ML frameworks can integrate without control-plane rewrites. -- Shipped a **multi-tenant SaaS console + authenticated CLI + reproducible deployment proof**, with OIDC/PKCE browser sessions, hashed workspace API keys, strict CSP, S3-compatible artifacts, and a validated AWS ECS/Fargate + RDS + S3 Terraform reference; CI passes **164 Python tests** and the proof exercises weighted canary traffic, promotion, rollback, regression detection, failed-canary abort, and stable-target preservation through the public API. - -## One-line project description - -Production-style, multi-tenant ML deployment platform with framework-independent runtimes, immutable cloud artifacts, safe canary rollout/rollback, runtime resilience, observability, secure OIDC/API-key access, and cross-language Go inference. - -## What makes the project technically strong - -### 1. Framework independence - -The control plane does not need framework-specific branches for every future model format. A new framework can implement the documented external runtime contract and register a framework identifier through configuration. - -### 2. Reliability is part of the serving path - -Retries, backoff, circuit breakers, canary routing, rollback, automatic fallback, health inspection, and metrics are implemented as platform behavior rather than demo-only scripts. - -### 3. Model identity is immutable - -Deployments point to immutable model versions and verified artifacts rather than mutable filesystem locations. - -### 4. Cross-language execution is real - -The reference Go runtime runs as a separate HTTP service and participates in the same deployment/inference flow as in-process runtimes. - -### 5. Claims are measured - -The repository includes reproducible load generation, fault injection, CI artifacts, p50/p95/p99 latency, throughput, error-rate, and traffic-lane reporting. A recruiter-proof harness also exercises stable serving, weighted canary traffic, promotion, rollback, regression detection, canary abort, and final stable-target preservation through the public API. - -## 60-second interview answer - -I built ModelForge because training a model is only one piece of an ML system. I wanted to understand what happens when you need to version artifacts, move traffic safely, isolate framework dependencies, recover from runtime failures, and prove which model actually served a request. - -The control plane is FastAPI with MySQL and SQLAlchemy. Models are registered as immutable versions with checksums. Deployments have explicit lifecycle state, and environments have authoritative serving targets. For execution, ModelForge supports in-process runtimes like PyTorch and ONNX, but the important design choice is the external runtime contract. The Go runtime proves the control plane can serve through another language and process without adding Go-specific inference logic to the core. - -Then I added reliability: retries, exponential backoff, circuit breakers, health inspection, weighted canaries, promotion, abort, rollback, and automatic stable fallback. I also turned the control plane into a multi-tenant SaaS foundation with workspaces, hashed API keys, OIDC Authorization Code + PKCE browser sessions, CSRF protection, strict CSP, and S3-compatible artifacts. The product has a public landing page, browser console, CLI, fault injection, and benchmark harness. The validated public-product checkpoint passes 164 Python tests and exercises the live Compose stack end to end. - -## Good interview follow-up topics - -- Why use immutable model-version identity instead of mutable artifact paths? -- How does transactional deployment state prevent split-brain serving decisions? -- Why isolate some runtimes out of process? -- What should happen when a canary runtime fails after receiving traffic? -- Why are circuit breakers per runtime instead of global? -- How would the current process-local cache evolve into a distributed serving system? -- What guarantees does the external runtime contract intentionally provide, and what is left runtime-specific? -- Why are CI benchmark numbers presented as validation evidence rather than production capacity claims? - -## Demo script - -1. Start the stack with `docker compose up --build --wait --detach`. -2. Run `make proof`. -3. Open `http://localhost:8000/` to show the public product, then enter `/dashboard` for the SaaS console. -4. Open `demo-proof-evidence/proof-report.md` and show that v2 received real weighted canary traffic before promotion. -5. Show that the proof promoted v2 and then rolled back to v1 through the deployment lifecycle. -6. Show the intentionally regressed v3 canary: the proof detects its output drift, aborts it, and ModelForge records the deployment as `FAILED` with a failure reason. -7. Show that the final environment target still points to stable v1 with no canary configured. -8. Run `modelforge runtimes` to show the Go runtime health and circuit state, then connect the same lifecycle to the AWS/Terraform/release-automation story. - -The same proof harness accepts `MODELFORGE_URL`, `MODELFORGE_API_KEY`, and -`MODELFORGE_WORKSPACE`, so the identical scenario can be run against a hosted -multi-tenant deployment without modifying the script. - -## Validated metrics to use carefully - -- 164 Python tests passed; 3 skipped in the standard Python CI job. -- Full deployment proof passed in the live Compose stack: v2 canary promotion, - rollback to v1, v3 output drift of 185.7% against a 50% threshold, v3 marked - FAILED, and v1 preserved as the final active target. -- Go formatting, tests, and vet checks passed. -- 200/200 benchmark requests succeeded with 0% error. -- 133.5 requests/s in the post-merge GitHub-hosted CI smoke run. -- p50 137.3 ms, p95 231.2 ms, p99 322.8 ms end-to-end latency in that same run. - -Always qualify the throughput/latency values as **GitHub-hosted CI smoke results**, not universal production limits. diff --git a/tests/test_demo_proof.py b/tests/test_demo_proof.py index c2cb272..7135b8a 100644 --- a/tests/test_demo_proof.py +++ b/tests/test_demo_proof.py @@ -1,4 +1,4 @@ -"""Unit tests for the recruiter-visible deployment proof harness.""" +"""Unit tests for the deployment validation scenario.""" from __future__ import annotations