From 651adabcdf091f229860765173a2fd92bb945de8 Mon Sep 17 00:00:00 2001 From: Michael Baffour Awuah Date: Sat, 19 Sep 2026 17:05:59 -0400 Subject: [PATCH 01/10] Add hosted launch verification gate --- .github/workflows/demo-proof.yml | 8 +- Makefile | 5 +- docs/hosted-demo.md | 128 ++++++++++++++++ scripts/verify_hosted.py | 244 +++++++++++++++++++++++++++++++ tests/test_verify_hosted.py | 167 +++++++++++++++++++++ 5 files changed, 547 insertions(+), 5 deletions(-) create mode 100644 docs/hosted-demo.md create mode 100644 scripts/verify_hosted.py create mode 100644 tests/test_verify_hosted.py diff --git a/.github/workflows/demo-proof.yml b/.github/workflows/demo-proof.yml index 53eb4df..7516e39 100644 --- a/.github/workflows/demo-proof.yml +++ b/.github/workflows/demo-proof.yml @@ -55,13 +55,12 @@ jobs: - name: Install proof client run: python -m pip install -e . - - name: Verify hosted service health + - name: Verify hosted production surface shell: bash run: | set -euo pipefail - BASE_URL="${MODELFORGE_URL%/}" - curl --fail --silent --show-error "${BASE_URL}/health" >/dev/null - curl --fail --silent --show-error "${BASE_URL}/ready" >/dev/null + python scripts/verify_hosted.py \ + --output hosted-release-evidence.json - name: Run hosted deployment proof shell: bash @@ -75,6 +74,7 @@ jobs: with: name: modelforge-hosted-demo-proof-${{ github.run_id }} path: | + hosted-release-evidence.json hosted-demo-proof/ hosted-demo-proof.log if-no-files-found: ignore diff --git a/Makefile b/Makefile index ac8f58d..87b5009 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -.PHONY: install lint test go-test check up down demo proof benchmark clean +.PHONY: install lint test go-test check up down demo proof hosted-verify benchmark clean install: python -m pip install -e ".[dev,pytorch,onnx]" @@ -28,6 +28,9 @@ demo: proof: python demo/proof_scenario.py --environment recruiter-demo --output-dir demo-proof-evidence +hosted-verify: + python scripts/verify_hosted.py + benchmark: python benchmarks/bootstrap_external.py --environment benchmark-go python benchmarks/load_test.py --environment benchmark-go --requests 5000 --concurrency 50 --warmup 100 --output benchmark-results.json diff --git a/docs/hosted-demo.md b/docs/hosted-demo.md new file mode 100644 index 0000000..c7c214e --- /dev/null +++ b/docs/hosted-demo.md @@ -0,0 +1,128 @@ +# Hosted Demo Launch + +This is the final ModelForge proof step. The repository already validates the +full platform locally and in CI; this run proves the same lifecycle against the +real public HTTPS deployment. + +## What must exist first + +Before running the hosted proof, the AWS reference stack must be provisioned and +activated: + +- public HTTPS DNS name and ACM certificate; +- ECS/Fargate API and Go-runtime services with desired count greater than zero; +- RDS MySQL; +- S3 artifact bucket; +- populated runtime and metrics secrets; +- configured OIDC application/callback; +- successful database migration task; +- successful Release AWS workflow. + +The AWS bootstrap and release sequence is documented in infra/aws/README.md. + +## First browser login + +Open the production URL in a browser and sign in through the configured OIDC +provider. + +Create or select the workspace that will be used for recruiter proof. In that +workspace create a developer API key. The raw mf_live_ secret is shown only +once. + +Use a dedicated demo workspace instead of a personal production workspace when +possible. The proof intentionally creates immutable model versions and +deployment records. + +## GitHub production environment + +Configure these non-secret variables: + +~~~text +MODELFORGE_PUBLIC_URL=https://YOUR_PUBLIC_HOST +MODELFORGE_DEMO_WORKSPACE=YOUR_WORKSPACE_SLUG +~~~ + +Configure this environment secret: + +~~~text +MODELFORGE_DEMO_API_KEY=mf_live_... +~~~ + +The same protected production environment may already contain the AWS release +variables used by release-aws.yml. + +## Verify the hosted surface + +You can run the verifier locally without changing data: + +~~~bash +MODELFORGE_URL=https://YOUR_PUBLIC_HOST \ +MODELFORGE_API_KEY=mf_live_... \ +MODELFORGE_WORKSPACE=YOUR_WORKSPACE_SLUG \ +python scripts/verify_hosted.py +~~~ + +It checks HTTPS, health/readiness, OIDC mode, HSTS, strict CSP, +unauthenticated-request rejection, protected metrics, workspace API-key +identity, runtime inventory, and external-runtime health. The report never +includes the raw API key. + +## Run the final hosted proof + +In GitHub: + +1. Open Actions. +2. Select Hosted Demo Proof. +3. Choose Run workflow from main. +4. Keep the default serving environment or choose another isolated name. +5. Run it. + +The workflow first runs scripts/verify_hosted.py, then executes the same +full-lifecycle proof used by local CI: + +~~~text +stable v1 + -> v2 weighted canary + -> real traffic through both lanes + -> promote v2 + -> rollback to v1 + -> v3 regression canary + -> detect output drift + -> abort v3 as FAILED + -> verify v1 remains authoritative +~~~ + +A successful run uploads one GitHub Actions artifact containing: + +~~~text +hosted-release-evidence.json +hosted-demo-proof/proof-summary.json +hosted-demo-proof/traffic-samples.jsonl +hosted-demo-proof/proof-report.md +hosted-demo-proof.log +~~~ + +That artifact is the evidence to cite when the README says the public hosted +demo passed. + +## Portfolio capture + +After the hosted proof is green, capture real screenshots of: + +1. the public ModelForge landing page; +2. signed-in workspace overview; +3. model registry with the proof versions; +4. stable/canary deployment state; +5. runtime fleet showing the external Go runtime healthy; +6. the GitHub Hosted Demo Proof run and PASS artifact. + +Do not fabricate or mock these screenshots. They should come from the real +hosted deployment and real proof run. + +## Final claim boundary + +Before the hosted workflow passes, describe ModelForge as having a validated AWS +deployment architecture and hosted-proof workflow. + +After it passes, it is accurate to say the identical deployment lifecycle was +executed successfully against the public hosted ModelForge service. diff --git a/scripts/verify_hosted.py b/scripts/verify_hosted.py new file mode 100644 index 0000000..3bcb2b0 --- /dev/null +++ b/scripts/verify_hosted.py @@ -0,0 +1,244 @@ +"""Verify that a public ModelForge deployment is production-ready.""" + +from __future__ import annotations + +import argparse +import json +import os +from datetime import UTC, datetime +from pathlib import Path +from typing import Any + +import httpx + + +def _expect_status( + response: httpx.Response, + expected: int, + *, + label: str, +) -> Any: + if response.status_code != expected: + raise RuntimeError( + f"{label} returned HTTP {response.status_code}: {response.text}" + ) + if not response.content: + return None + content_type = response.headers.get("content-type", "") + if "application/json" in content_type: + return response.json() + return response.text + + +def _require_header( + response: httpx.Response, + name: str, + expected: str | None = None, +) -> str: + value = response.headers.get(name) + if not value: + raise RuntimeError(f"Missing required security header: {name}.") + if expected is not None and value != expected: + raise RuntimeError( + f"Security header {name} was {value!r}; expected {expected!r}." + ) + return value + + +def verify_hosted_release( + *, + base_url: str, + api_key: str, + workspace: str, + timeout_seconds: float = 10.0, + transport: httpx.BaseTransport | None = None, +) -> dict[str, Any]: + """Verify public security, auth, tenancy, runtime health, and readiness.""" + + normalized_url = base_url.rstrip("/") + parsed = httpx.URL(normalized_url) + if parsed.scheme != "https": + raise ValueError("Hosted ModelForge verification requires an HTTPS URL.") + if not parsed.host: + raise ValueError("Hosted ModelForge verification requires a public host.") + if not api_key.startswith("mf_live_"): + raise ValueError("Hosted verification requires a ModelForge API key.") + if not workspace.strip(): + raise ValueError("Hosted verification requires a workspace selector.") + + started_at = datetime.now(UTC) + common = { + "base_url": normalized_url, + "timeout": timeout_seconds, + "transport": transport, + "follow_redirects": False, + } + + with httpx.Client(**common) as public_client: + health = _expect_status(public_client.get("/health"), 200, label="health") + readiness = _expect_status( + public_client.get("/ready"), 200, label="readiness" + ) + auth_config = _expect_status( + public_client.get("/auth/config"), 200, label="auth config" + ) + landing_response = public_client.get("/") + landing = _expect_status( + landing_response, 200, label="public landing page" + ) + unauthenticated_identity = public_client.get("/auth/me") + metrics_without_token = public_client.get("/metrics") + + if health.get("status") != "healthy": + raise RuntimeError(f"Unexpected health payload: {health!r}") + if readiness.get("status") != "ready": + raise RuntimeError(f"Unexpected readiness payload: {readiness!r}") + if auth_config.get("mode") != "oidc": + raise RuntimeError("Hosted deployment is not running in OIDC auth mode.") + if auth_config.get("browser_login_enabled") is not True: + raise RuntimeError("Hosted browser login is not enabled.") + if "ModelForge" not in landing: + raise RuntimeError("Public landing page did not identify ModelForge.") + if unauthenticated_identity.status_code != 401: + raise RuntimeError( + "/auth/me must reject unauthenticated hosted requests with HTTP 401." + ) + if metrics_without_token.status_code != 401: + raise RuntimeError("/metrics must require a bearer token in production.") + + hsts = _require_header(landing_response, "strict-transport-security") + if "max-age=31536000" not in hsts or "includeSubDomains" not in hsts: + raise RuntimeError("Hosted HSTS policy is weaker than the production baseline.") + _require_header(landing_response, "x-content-type-options", "nosniff") + _require_header(landing_response, "x-frame-options", "DENY") + csp = _require_header(landing_response, "content-security-policy") + if "script-src 'self'" not in csp or "style-src 'self'" not in csp: + raise RuntimeError("Hosted CSP does not restrict scripts/styles to self.") + if "'unsafe-inline'" in csp: + raise RuntimeError("Hosted product CSP unexpectedly permits unsafe inline code.") + + authenticated_headers = { + "Authorization": f"Bearer {api_key}", + "X-ModelForge-Workspace": workspace, + } + with httpx.Client(**common, headers=authenticated_headers) as authenticated_client: + identity = _expect_status( + authenticated_client.get("/auth/me"), + 200, + label="authenticated identity", + ) + models = _expect_status( + authenticated_client.get("/models"), 200, label="model registry" + ) + deployments = _expect_status( + authenticated_client.get("/deployments"), 200, label="deployments" + ) + runtimes = _expect_status( + authenticated_client.get("/runtimes"), 200, label="runtime inventory" + ) + runtime_health = _expect_status( + authenticated_client.get("/runtimes/health"), + 200, + label="runtime health", + ) + + if identity.get("auth_type") != "api_key": + raise RuntimeError("Hosted API-key request did not authenticate as api_key.") + if identity.get("workspace_slug") != workspace: + raise RuntimeError( + "Hosted API key resolved to a different workspace than requested." + ) + + external_inventory = runtimes.get("external") or [] + if not external_inventory: + raise RuntimeError("Hosted deployment exposes no external runtime.") + external_health = runtime_health.get("external") or [] + if not external_health: + raise RuntimeError("Hosted deployment returned no external runtime health.") + + unhealthy = [ + runtime + for runtime in external_health + if runtime.get("status") != "healthy" + ] + if unhealthy: + raise RuntimeError( + "Hosted external runtime health is not fully green: " + + json.dumps(unhealthy, sort_keys=True) + ) + + finished_at = datetime.now(UTC) + return { + "status": "PASS", + "base_url": normalized_url, + "workspace": workspace, + "started_at": started_at.isoformat(), + "finished_at": finished_at.isoformat(), + "duration_seconds": (finished_at - started_at).total_seconds(), + "checks": { + "health": health.get("status"), + "readiness": readiness.get("status"), + "auth_mode": auth_config.get("mode"), + "browser_login_enabled": auth_config.get("browser_login_enabled"), + "unauthenticated_identity_status": unauthenticated_identity.status_code, + "unauthenticated_metrics_status": metrics_without_token.status_code, + "hsts": hsts, + "content_security_policy": csp, + "identity": { + "auth_type": identity.get("auth_type"), + "workspace_slug": identity.get("workspace_slug"), + "role": identity.get("role"), + }, + "models_visible": len(models), + "deployments_visible": len(deployments), + "runtime_inventory": { + "in_process": len(runtimes.get("in_process") or []), + "external": len(external_inventory), + }, + "external_runtime_health": external_health, + }, + } + + +def main() -> None: + parser = argparse.ArgumentParser( + description="Verify a hosted ModelForge production deployment.", + ) + parser.add_argument( + "--base-url", + default=os.getenv("MODELFORGE_URL"), + required=os.getenv("MODELFORGE_URL") is None, + ) + parser.add_argument( + "--api-key", + default=os.getenv("MODELFORGE_API_KEY"), + required=os.getenv("MODELFORGE_API_KEY") is None, + ) + parser.add_argument( + "--workspace", + default=os.getenv("MODELFORGE_WORKSPACE"), + required=os.getenv("MODELFORGE_WORKSPACE") is None, + ) + parser.add_argument("--timeout-seconds", type=float, default=10.0) + parser.add_argument( + "--output", + type=Path, + default=Path("hosted-release-evidence.json"), + ) + args = parser.parse_args() + + report = verify_hosted_release( + base_url=args.base_url, + api_key=args.api_key, + workspace=args.workspace, + timeout_seconds=args.timeout_seconds, + ) + args.output.write_text( + json.dumps(report, indent=2, sort_keys=True) + "\n", + encoding="utf-8", + ) + print(json.dumps(report, indent=2, sort_keys=True)) + + +if __name__ == "__main__": + main() diff --git a/tests/test_verify_hosted.py b/tests/test_verify_hosted.py new file mode 100644 index 0000000..e8ddab5 --- /dev/null +++ b/tests/test_verify_hosted.py @@ -0,0 +1,167 @@ +"""Tests for hosted production verification.""" + +from __future__ import annotations + +import json + +import httpx +import pytest + +from scripts.verify_hosted import verify_hosted_release + + +SECURITY_HEADERS = { + "Strict-Transport-Security": "max-age=31536000; includeSubDomains", + "X-Content-Type-Options": "nosniff", + "X-Frame-Options": "DENY", + "Content-Security-Policy": ( + "default-src 'self'; style-src 'self'; script-src 'self'; " + "connect-src 'self'; img-src 'self' data:; frame-ancestors 'none'; " + "base-uri 'self'; form-action 'self';" + ), +} + + +def hosted_transport( + *, + runtime_status: str = "healthy", + metrics_status: int = 401, +) -> httpx.MockTransport: + def handler(request: httpx.Request) -> httpx.Response: + path = request.url.path + authorized = request.headers.get("Authorization") == "Bearer mf_live_test" + workspace = request.headers.get("X-ModelForge-Workspace") + + if path == "/health": + return httpx.Response(200, json={"status": "healthy"}, request=request) + if path == "/ready": + return httpx.Response(200, json={"status": "ready"}, request=request) + if path == "/auth/config": + return httpx.Response( + 200, + json={ + "mode": "oidc", + "browser_login_enabled": True, + "login_url": "/auth/login", + }, + request=request, + ) + if path == "/": + return httpx.Response( + 200, + text="ModelForge", + headers=SECURITY_HEADERS, + request=request, + ) + if path == "/metrics": + return httpx.Response( + metrics_status, + json={"detail": "Metrics authentication required."}, + request=request, + ) + if path == "/auth/me": + if not authorized: + return httpx.Response( + 401, + json={"detail": "Authentication is required."}, + request=request, + ) + return httpx.Response( + 200, + json={ + "auth_type": "api_key", + "workspace_id": 9, + "workspace_slug": workspace, + "role": "developer", + }, + request=request, + ) + if path == "/models": + return httpx.Response(200, json=[], request=request) + if path == "/deployments": + return httpx.Response(200, json=[], request=request) + if path == "/runtimes": + return httpx.Response( + 200, + json={ + "in_process": [ + {"framework": "modelforge-json", "mode": "in_process"} + ], + "external": [ + { + "framework": "go-linear", + "mode": "external", + "runtime": "modelforge-go-runtime", + "circuit_state": "closed", + "consecutive_failures": 0, + } + ], + }, + request=request, + ) + if path == "/runtimes/health": + return httpx.Response( + 200, + json={ + "external": [ + { + "framework": "go-linear", + "runtime": "modelforge-go-runtime", + "status": runtime_status, + "circuit_state": "closed", + "consecutive_failures": 0, + } + ] + }, + request=request, + ) + return httpx.Response(404, request=request) + + return httpx.MockTransport(handler) + + +def test_hosted_verifier_passes_secure_production_surface() -> None: + report = verify_hosted_release( + base_url="https://modelforge.example", + api_key="mf_live_test", + workspace="demo", + transport=hosted_transport(), + ) + + assert report["status"] == "PASS" + assert report["workspace"] == "demo" + assert report["checks"]["auth_mode"] == "oidc" + assert report["checks"]["unauthenticated_metrics_status"] == 401 + assert report["checks"]["runtime_inventory"]["external"] == 1 + assert report["checks"]["external_runtime_health"][0]["status"] == "healthy" + assert "mf_live_test" not in json.dumps(report) + + +def test_hosted_verifier_rejects_http() -> None: + with pytest.raises(ValueError, match="HTTPS"): + verify_hosted_release( + base_url="http://modelforge.example", + api_key="mf_live_test", + workspace="demo", + transport=hosted_transport(), + ) + + +def test_hosted_verifier_requires_protected_metrics() -> None: + with pytest.raises(RuntimeError, match="/metrics"): + verify_hosted_release( + base_url="https://modelforge.example", + api_key="mf_live_test", + workspace="demo", + transport=hosted_transport(metrics_status=200), + ) + + +def test_hosted_verifier_rejects_unhealthy_external_runtime() -> None: + with pytest.raises(RuntimeError, match="runtime health"): + verify_hosted_release( + base_url="https://modelforge.example", + api_key="mf_live_test", + workspace="demo", + transport=hosted_transport(runtime_status="unavailable"), + ) From 55808d4e1b58a44da96e5e167a04461a4a5d446d Mon Sep 17 00:00:00 2001 From: Michael Baffour Awuah Date: Sat, 19 Sep 2026 17:09:16 -0400 Subject: [PATCH 02/10] Package hosted verification logic cleanly --- scripts/verify_hosted.py | 241 +------------------------------ src/modelforge/hosted_verify.py | 244 ++++++++++++++++++++++++++++++++ tests/test_verify_hosted.py | 2 +- 3 files changed, 247 insertions(+), 240 deletions(-) create mode 100644 src/modelforge/hosted_verify.py diff --git a/scripts/verify_hosted.py b/scripts/verify_hosted.py index 3bcb2b0..1d5c8f1 100644 --- a/scripts/verify_hosted.py +++ b/scripts/verify_hosted.py @@ -1,243 +1,6 @@ -"""Verify that a public ModelForge deployment is production-ready.""" +"""Executable wrapper for hosted ModelForge verification.""" -from __future__ import annotations - -import argparse -import json -import os -from datetime import UTC, datetime -from pathlib import Path -from typing import Any - -import httpx - - -def _expect_status( - response: httpx.Response, - expected: int, - *, - label: str, -) -> Any: - if response.status_code != expected: - raise RuntimeError( - f"{label} returned HTTP {response.status_code}: {response.text}" - ) - if not response.content: - return None - content_type = response.headers.get("content-type", "") - if "application/json" in content_type: - return response.json() - return response.text - - -def _require_header( - response: httpx.Response, - name: str, - expected: str | None = None, -) -> str: - value = response.headers.get(name) - if not value: - raise RuntimeError(f"Missing required security header: {name}.") - if expected is not None and value != expected: - raise RuntimeError( - f"Security header {name} was {value!r}; expected {expected!r}." - ) - return value - - -def verify_hosted_release( - *, - base_url: str, - api_key: str, - workspace: str, - timeout_seconds: float = 10.0, - transport: httpx.BaseTransport | None = None, -) -> dict[str, Any]: - """Verify public security, auth, tenancy, runtime health, and readiness.""" - - normalized_url = base_url.rstrip("/") - parsed = httpx.URL(normalized_url) - if parsed.scheme != "https": - raise ValueError("Hosted ModelForge verification requires an HTTPS URL.") - if not parsed.host: - raise ValueError("Hosted ModelForge verification requires a public host.") - if not api_key.startswith("mf_live_"): - raise ValueError("Hosted verification requires a ModelForge API key.") - if not workspace.strip(): - raise ValueError("Hosted verification requires a workspace selector.") - - started_at = datetime.now(UTC) - common = { - "base_url": normalized_url, - "timeout": timeout_seconds, - "transport": transport, - "follow_redirects": False, - } - - with httpx.Client(**common) as public_client: - health = _expect_status(public_client.get("/health"), 200, label="health") - readiness = _expect_status( - public_client.get("/ready"), 200, label="readiness" - ) - auth_config = _expect_status( - public_client.get("/auth/config"), 200, label="auth config" - ) - landing_response = public_client.get("/") - landing = _expect_status( - landing_response, 200, label="public landing page" - ) - unauthenticated_identity = public_client.get("/auth/me") - metrics_without_token = public_client.get("/metrics") - - if health.get("status") != "healthy": - raise RuntimeError(f"Unexpected health payload: {health!r}") - if readiness.get("status") != "ready": - raise RuntimeError(f"Unexpected readiness payload: {readiness!r}") - if auth_config.get("mode") != "oidc": - raise RuntimeError("Hosted deployment is not running in OIDC auth mode.") - if auth_config.get("browser_login_enabled") is not True: - raise RuntimeError("Hosted browser login is not enabled.") - if "ModelForge" not in landing: - raise RuntimeError("Public landing page did not identify ModelForge.") - if unauthenticated_identity.status_code != 401: - raise RuntimeError( - "/auth/me must reject unauthenticated hosted requests with HTTP 401." - ) - if metrics_without_token.status_code != 401: - raise RuntimeError("/metrics must require a bearer token in production.") - - hsts = _require_header(landing_response, "strict-transport-security") - if "max-age=31536000" not in hsts or "includeSubDomains" not in hsts: - raise RuntimeError("Hosted HSTS policy is weaker than the production baseline.") - _require_header(landing_response, "x-content-type-options", "nosniff") - _require_header(landing_response, "x-frame-options", "DENY") - csp = _require_header(landing_response, "content-security-policy") - if "script-src 'self'" not in csp or "style-src 'self'" not in csp: - raise RuntimeError("Hosted CSP does not restrict scripts/styles to self.") - if "'unsafe-inline'" in csp: - raise RuntimeError("Hosted product CSP unexpectedly permits unsafe inline code.") - - authenticated_headers = { - "Authorization": f"Bearer {api_key}", - "X-ModelForge-Workspace": workspace, - } - with httpx.Client(**common, headers=authenticated_headers) as authenticated_client: - identity = _expect_status( - authenticated_client.get("/auth/me"), - 200, - label="authenticated identity", - ) - models = _expect_status( - authenticated_client.get("/models"), 200, label="model registry" - ) - deployments = _expect_status( - authenticated_client.get("/deployments"), 200, label="deployments" - ) - runtimes = _expect_status( - authenticated_client.get("/runtimes"), 200, label="runtime inventory" - ) - runtime_health = _expect_status( - authenticated_client.get("/runtimes/health"), - 200, - label="runtime health", - ) - - if identity.get("auth_type") != "api_key": - raise RuntimeError("Hosted API-key request did not authenticate as api_key.") - if identity.get("workspace_slug") != workspace: - raise RuntimeError( - "Hosted API key resolved to a different workspace than requested." - ) - - external_inventory = runtimes.get("external") or [] - if not external_inventory: - raise RuntimeError("Hosted deployment exposes no external runtime.") - external_health = runtime_health.get("external") or [] - if not external_health: - raise RuntimeError("Hosted deployment returned no external runtime health.") - - unhealthy = [ - runtime - for runtime in external_health - if runtime.get("status") != "healthy" - ] - if unhealthy: - raise RuntimeError( - "Hosted external runtime health is not fully green: " - + json.dumps(unhealthy, sort_keys=True) - ) - - finished_at = datetime.now(UTC) - return { - "status": "PASS", - "base_url": normalized_url, - "workspace": workspace, - "started_at": started_at.isoformat(), - "finished_at": finished_at.isoformat(), - "duration_seconds": (finished_at - started_at).total_seconds(), - "checks": { - "health": health.get("status"), - "readiness": readiness.get("status"), - "auth_mode": auth_config.get("mode"), - "browser_login_enabled": auth_config.get("browser_login_enabled"), - "unauthenticated_identity_status": unauthenticated_identity.status_code, - "unauthenticated_metrics_status": metrics_without_token.status_code, - "hsts": hsts, - "content_security_policy": csp, - "identity": { - "auth_type": identity.get("auth_type"), - "workspace_slug": identity.get("workspace_slug"), - "role": identity.get("role"), - }, - "models_visible": len(models), - "deployments_visible": len(deployments), - "runtime_inventory": { - "in_process": len(runtimes.get("in_process") or []), - "external": len(external_inventory), - }, - "external_runtime_health": external_health, - }, - } - - -def main() -> None: - parser = argparse.ArgumentParser( - description="Verify a hosted ModelForge production deployment.", - ) - parser.add_argument( - "--base-url", - default=os.getenv("MODELFORGE_URL"), - required=os.getenv("MODELFORGE_URL") is None, - ) - parser.add_argument( - "--api-key", - default=os.getenv("MODELFORGE_API_KEY"), - required=os.getenv("MODELFORGE_API_KEY") is None, - ) - parser.add_argument( - "--workspace", - default=os.getenv("MODELFORGE_WORKSPACE"), - required=os.getenv("MODELFORGE_WORKSPACE") is None, - ) - parser.add_argument("--timeout-seconds", type=float, default=10.0) - parser.add_argument( - "--output", - type=Path, - default=Path("hosted-release-evidence.json"), - ) - args = parser.parse_args() - - report = verify_hosted_release( - base_url=args.base_url, - api_key=args.api_key, - workspace=args.workspace, - timeout_seconds=args.timeout_seconds, - ) - args.output.write_text( - json.dumps(report, indent=2, sort_keys=True) + "\n", - encoding="utf-8", - ) - print(json.dumps(report, indent=2, sort_keys=True)) +from modelforge.hosted_verify import main if __name__ == "__main__": diff --git a/src/modelforge/hosted_verify.py b/src/modelforge/hosted_verify.py new file mode 100644 index 0000000..3bcb2b0 --- /dev/null +++ b/src/modelforge/hosted_verify.py @@ -0,0 +1,244 @@ +"""Verify that a public ModelForge deployment is production-ready.""" + +from __future__ import annotations + +import argparse +import json +import os +from datetime import UTC, datetime +from pathlib import Path +from typing import Any + +import httpx + + +def _expect_status( + response: httpx.Response, + expected: int, + *, + label: str, +) -> Any: + if response.status_code != expected: + raise RuntimeError( + f"{label} returned HTTP {response.status_code}: {response.text}" + ) + if not response.content: + return None + content_type = response.headers.get("content-type", "") + if "application/json" in content_type: + return response.json() + return response.text + + +def _require_header( + response: httpx.Response, + name: str, + expected: str | None = None, +) -> str: + value = response.headers.get(name) + if not value: + raise RuntimeError(f"Missing required security header: {name}.") + if expected is not None and value != expected: + raise RuntimeError( + f"Security header {name} was {value!r}; expected {expected!r}." + ) + return value + + +def verify_hosted_release( + *, + base_url: str, + api_key: str, + workspace: str, + timeout_seconds: float = 10.0, + transport: httpx.BaseTransport | None = None, +) -> dict[str, Any]: + """Verify public security, auth, tenancy, runtime health, and readiness.""" + + normalized_url = base_url.rstrip("/") + parsed = httpx.URL(normalized_url) + if parsed.scheme != "https": + raise ValueError("Hosted ModelForge verification requires an HTTPS URL.") + if not parsed.host: + raise ValueError("Hosted ModelForge verification requires a public host.") + if not api_key.startswith("mf_live_"): + raise ValueError("Hosted verification requires a ModelForge API key.") + if not workspace.strip(): + raise ValueError("Hosted verification requires a workspace selector.") + + started_at = datetime.now(UTC) + common = { + "base_url": normalized_url, + "timeout": timeout_seconds, + "transport": transport, + "follow_redirects": False, + } + + with httpx.Client(**common) as public_client: + health = _expect_status(public_client.get("/health"), 200, label="health") + readiness = _expect_status( + public_client.get("/ready"), 200, label="readiness" + ) + auth_config = _expect_status( + public_client.get("/auth/config"), 200, label="auth config" + ) + landing_response = public_client.get("/") + landing = _expect_status( + landing_response, 200, label="public landing page" + ) + unauthenticated_identity = public_client.get("/auth/me") + metrics_without_token = public_client.get("/metrics") + + if health.get("status") != "healthy": + raise RuntimeError(f"Unexpected health payload: {health!r}") + if readiness.get("status") != "ready": + raise RuntimeError(f"Unexpected readiness payload: {readiness!r}") + if auth_config.get("mode") != "oidc": + raise RuntimeError("Hosted deployment is not running in OIDC auth mode.") + if auth_config.get("browser_login_enabled") is not True: + raise RuntimeError("Hosted browser login is not enabled.") + if "ModelForge" not in landing: + raise RuntimeError("Public landing page did not identify ModelForge.") + if unauthenticated_identity.status_code != 401: + raise RuntimeError( + "/auth/me must reject unauthenticated hosted requests with HTTP 401." + ) + if metrics_without_token.status_code != 401: + raise RuntimeError("/metrics must require a bearer token in production.") + + hsts = _require_header(landing_response, "strict-transport-security") + if "max-age=31536000" not in hsts or "includeSubDomains" not in hsts: + raise RuntimeError("Hosted HSTS policy is weaker than the production baseline.") + _require_header(landing_response, "x-content-type-options", "nosniff") + _require_header(landing_response, "x-frame-options", "DENY") + csp = _require_header(landing_response, "content-security-policy") + if "script-src 'self'" not in csp or "style-src 'self'" not in csp: + raise RuntimeError("Hosted CSP does not restrict scripts/styles to self.") + if "'unsafe-inline'" in csp: + raise RuntimeError("Hosted product CSP unexpectedly permits unsafe inline code.") + + authenticated_headers = { + "Authorization": f"Bearer {api_key}", + "X-ModelForge-Workspace": workspace, + } + with httpx.Client(**common, headers=authenticated_headers) as authenticated_client: + identity = _expect_status( + authenticated_client.get("/auth/me"), + 200, + label="authenticated identity", + ) + models = _expect_status( + authenticated_client.get("/models"), 200, label="model registry" + ) + deployments = _expect_status( + authenticated_client.get("/deployments"), 200, label="deployments" + ) + runtimes = _expect_status( + authenticated_client.get("/runtimes"), 200, label="runtime inventory" + ) + runtime_health = _expect_status( + authenticated_client.get("/runtimes/health"), + 200, + label="runtime health", + ) + + if identity.get("auth_type") != "api_key": + raise RuntimeError("Hosted API-key request did not authenticate as api_key.") + if identity.get("workspace_slug") != workspace: + raise RuntimeError( + "Hosted API key resolved to a different workspace than requested." + ) + + external_inventory = runtimes.get("external") or [] + if not external_inventory: + raise RuntimeError("Hosted deployment exposes no external runtime.") + external_health = runtime_health.get("external") or [] + if not external_health: + raise RuntimeError("Hosted deployment returned no external runtime health.") + + unhealthy = [ + runtime + for runtime in external_health + if runtime.get("status") != "healthy" + ] + if unhealthy: + raise RuntimeError( + "Hosted external runtime health is not fully green: " + + json.dumps(unhealthy, sort_keys=True) + ) + + finished_at = datetime.now(UTC) + return { + "status": "PASS", + "base_url": normalized_url, + "workspace": workspace, + "started_at": started_at.isoformat(), + "finished_at": finished_at.isoformat(), + "duration_seconds": (finished_at - started_at).total_seconds(), + "checks": { + "health": health.get("status"), + "readiness": readiness.get("status"), + "auth_mode": auth_config.get("mode"), + "browser_login_enabled": auth_config.get("browser_login_enabled"), + "unauthenticated_identity_status": unauthenticated_identity.status_code, + "unauthenticated_metrics_status": metrics_without_token.status_code, + "hsts": hsts, + "content_security_policy": csp, + "identity": { + "auth_type": identity.get("auth_type"), + "workspace_slug": identity.get("workspace_slug"), + "role": identity.get("role"), + }, + "models_visible": len(models), + "deployments_visible": len(deployments), + "runtime_inventory": { + "in_process": len(runtimes.get("in_process") or []), + "external": len(external_inventory), + }, + "external_runtime_health": external_health, + }, + } + + +def main() -> None: + parser = argparse.ArgumentParser( + description="Verify a hosted ModelForge production deployment.", + ) + parser.add_argument( + "--base-url", + default=os.getenv("MODELFORGE_URL"), + required=os.getenv("MODELFORGE_URL") is None, + ) + parser.add_argument( + "--api-key", + default=os.getenv("MODELFORGE_API_KEY"), + required=os.getenv("MODELFORGE_API_KEY") is None, + ) + parser.add_argument( + "--workspace", + default=os.getenv("MODELFORGE_WORKSPACE"), + required=os.getenv("MODELFORGE_WORKSPACE") is None, + ) + parser.add_argument("--timeout-seconds", type=float, default=10.0) + parser.add_argument( + "--output", + type=Path, + default=Path("hosted-release-evidence.json"), + ) + args = parser.parse_args() + + report = verify_hosted_release( + base_url=args.base_url, + api_key=args.api_key, + workspace=args.workspace, + timeout_seconds=args.timeout_seconds, + ) + args.output.write_text( + json.dumps(report, indent=2, sort_keys=True) + "\n", + encoding="utf-8", + ) + print(json.dumps(report, indent=2, sort_keys=True)) + + +if __name__ == "__main__": + main() diff --git a/tests/test_verify_hosted.py b/tests/test_verify_hosted.py index e8ddab5..0fa8394 100644 --- a/tests/test_verify_hosted.py +++ b/tests/test_verify_hosted.py @@ -7,7 +7,7 @@ import httpx import pytest -from scripts.verify_hosted import verify_hosted_release +from modelforge.hosted_verify import verify_hosted_release SECURITY_HEADERS = { From 56df7c58b51bc0abdd00511719f67267952b114b Mon Sep 17 00:00:00 2001 From: Michael Baffour Awuah Date: Sat, 19 Sep 2026 17:12:49 -0400 Subject: [PATCH 03/10] Align hosted verifier import layout --- tests/test_verify_hosted.py | 1 - 1 file changed, 1 deletion(-) diff --git a/tests/test_verify_hosted.py b/tests/test_verify_hosted.py index 0fa8394..5aa157c 100644 --- a/tests/test_verify_hosted.py +++ b/tests/test_verify_hosted.py @@ -6,7 +6,6 @@ import httpx import pytest - from modelforge.hosted_verify import verify_hosted_release From 021f77803cf4d10c12215337388403095bd4337c Mon Sep 17 00:00:00 2001 From: Michael Baffour Awuah Date: Sat, 19 Sep 2026 17:12:52 -0400 Subject: [PATCH 04/10] Run packaged hosted verifier module --- .github/workflows/demo-proof.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/demo-proof.yml b/.github/workflows/demo-proof.yml index 7516e39..638833c 100644 --- a/.github/workflows/demo-proof.yml +++ b/.github/workflows/demo-proof.yml @@ -59,7 +59,7 @@ jobs: shell: bash run: | set -euo pipefail - python scripts/verify_hosted.py \ + python -m modelforge.hosted_verify \ --output hosted-release-evidence.json - name: Run hosted deployment proof From 143fcd17f82cb72a9c23008abe2ed183a5688fc1 Mon Sep 17 00:00:00 2001 From: Michael Baffour Awuah Date: Sat, 19 Sep 2026 17:12:56 -0400 Subject: [PATCH 05/10] Use packaged verifier from Make --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 87b5009..1e68923 100644 --- a/Makefile +++ b/Makefile @@ -29,7 +29,7 @@ proof: python demo/proof_scenario.py --environment recruiter-demo --output-dir demo-proof-evidence hosted-verify: - python scripts/verify_hosted.py + python -m modelforge.hosted_verify benchmark: python benchmarks/bootstrap_external.py --environment benchmark-go From b241db0e9c6b23d0cdd219364a7256edaa9730c2 Mon Sep 17 00:00:00 2001 From: Michael Baffour Awuah Date: Sat, 19 Sep 2026 17:12:59 -0400 Subject: [PATCH 06/10] Document packaged hosted verifier command --- docs/hosted-demo.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/hosted-demo.md b/docs/hosted-demo.md index c7c214e..594545a 100644 --- a/docs/hosted-demo.md +++ b/docs/hosted-demo.md @@ -59,7 +59,7 @@ You can run the verifier locally without changing data: MODELFORGE_URL=https://YOUR_PUBLIC_HOST \ MODELFORGE_API_KEY=mf_live_... \ MODELFORGE_WORKSPACE=YOUR_WORKSPACE_SLUG \ -python scripts/verify_hosted.py +python -m modelforge.hosted_verify ~~~ It checks HTTPS, health/readiness, OIDC mode, HSTS, strict CSP, @@ -77,7 +77,7 @@ In GitHub: 4. Keep the default serving environment or choose another isolated name. 5. Run it. -The workflow first runs scripts/verify_hosted.py, then executes the same +The workflow first runs the packaged hosted verifier, then executes the same full-lifecycle proof used by local CI: ~~~text From a323b9be257191c2e1325f502ec7a88802eb3150 Mon Sep 17 00:00:00 2001 From: Michael Baffour Awuah Date: Sat, 19 Sep 2026 17:13:02 -0400 Subject: [PATCH 07/10] Remove redundant hosted verifier wrapper --- scripts/verify_hosted.py | 7 ------- 1 file changed, 7 deletions(-) delete mode 100644 scripts/verify_hosted.py diff --git a/scripts/verify_hosted.py b/scripts/verify_hosted.py deleted file mode 100644 index 1d5c8f1..0000000 --- a/scripts/verify_hosted.py +++ /dev/null @@ -1,7 +0,0 @@ -"""Executable wrapper for hosted ModelForge verification.""" - -from modelforge.hosted_verify import main - - -if __name__ == "__main__": - main() From 8aca5f88a6fce9a2234120421655b98311b6424b Mon Sep 17 00:00:00 2001 From: Michael Baffour Awuah Date: Sat, 19 Sep 2026 17:17:13 -0400 Subject: [PATCH 08/10] Fix hosted verifier import grouping --- tests/test_verify_hosted.py | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/test_verify_hosted.py b/tests/test_verify_hosted.py index 5aa157c..0fa8394 100644 --- a/tests/test_verify_hosted.py +++ b/tests/test_verify_hosted.py @@ -6,6 +6,7 @@ import httpx import pytest + from modelforge.hosted_verify import verify_hosted_release From 770f111cc3a1431c79df9b6947ef8b82049b11d0 Mon Sep 17 00:00:00 2001 From: Michael Baffour Awuah Date: Sat, 19 Sep 2026 17:20:47 -0400 Subject: [PATCH 09/10] Match hosted verifier test spacing --- tests/test_verify_hosted.py | 1 - 1 file changed, 1 deletion(-) diff --git a/tests/test_verify_hosted.py b/tests/test_verify_hosted.py index 0fa8394..ca657ba 100644 --- a/tests/test_verify_hosted.py +++ b/tests/test_verify_hosted.py @@ -9,7 +9,6 @@ from modelforge.hosted_verify import verify_hosted_release - SECURITY_HEADERS = { "Strict-Transport-Security": "max-age=31536000; includeSubDomains", "X-Content-Type-Options": "nosniff", From dd85bb8a65a293030d62d5557bd4604d80989f38 Mon Sep 17 00:00:00 2001 From: Michael Baffour Awuah Date: Sat, 19 Sep 2026 17:26:26 -0400 Subject: [PATCH 10/10] Sync hosted launch proof documentation --- README.md | 21 +++++++++++++++++++-- docs/recruiting.md | 15 +++++++++++---- 2 files changed, 30 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 1a78f56..e605089 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ Rather than treating model serving as a single `/predict` endpoint, ModelForge t | Signal | Current validated result | |---|---| -| Python test suite | **164 passed**, 3 skipped | +| Python test suite | **168 passed**, 3 skipped | | Go validation | `gofmt`, `go test ./...`, and `go vet ./...` green | | CI benchmark | **200 / 200 successful requests**, 0% errors | | Throughput | **133.5 requests/s** | @@ -42,6 +42,23 @@ intentionally regressed canary, automated abort, and final stable-target verification. It writes JSON/JSONL/Markdown evidence to `demo-proof-evidence/` and exits non-zero if any lifecycle assertion fails. +For a real public deployment, ModelForge also includes a non-mutating hosted +release gate: + +```bash +MODELFORGE_URL=https://YOUR_PUBLIC_HOST \ +MODELFORGE_API_KEY=mf_live_... \ +MODELFORGE_WORKSPACE=YOUR_WORKSPACE_SLUG \ +make hosted-verify +``` + +It verifies HTTPS/HSTS/CSP, OIDC mode, unauthenticated access controls, +workspace API-key identity, readiness, runtime inventory, and external-runtime +health before the hosted lifecycle proof runs. See +[`docs/hosted-demo.md`](docs/hosted-demo.md). A successful public hosted run +is intentionally **not claimed** until the `Hosted Demo Proof` workflow passes +against the real deployment. + For the smaller interactive stable + canary seed: ```bash @@ -472,7 +489,7 @@ Reliability and performance characteristics should be demonstrated through tests ## Testing -The post-merge validation checkpoint currently reports **164 Python tests passed, 3 skipped**, with Go formatting, unit tests, and vet checks green. The project includes tests across the major system boundaries: +The post-merge validation checkpoint currently reports **168 Python tests passed, 3 skipped**, with Go formatting, unit tests, and vet checks green. The project includes tests across the major system boundaries: ```text Artifact storage diff --git a/docs/recruiting.md b/docs/recruiting.md index 13a02bc..262dac3 100644 --- a/docs/recruiting.md +++ b/docs/recruiting.md @@ -2,13 +2,13 @@ ## 30-second pitch -ModelForge is a reliability-aware ML deployment platform I built to explore the systems problems that appear after model training. It combines a Python/FastAPI control plane, MySQL-backed model and deployment state, framework-specific in-process runtimes, and a standalone Go inference runtime behind a framework-agnostic protocol. The platform supports immutable artifacts, transactional promotion and rollback, weighted canary releases, retries, circuit breakers, automatic stable fallback, Prometheus metrics, S3-compatible artifact storage, multi-tenant workspaces, secure OIDC browser sessions, a public SaaS console, an authenticated CLI, production configuration preflight, and a validated Terraform AWS reference deployment. The current post-merge CI checkpoint passes 164 Python tests and validates the full stack end to end. +ModelForge is a reliability-aware ML deployment platform I built to explore the systems problems that appear after model training. It combines a Python/FastAPI control plane, MySQL-backed model and deployment state, framework-specific in-process runtimes, and a standalone Go inference runtime behind a framework-agnostic protocol. The platform supports immutable artifacts, transactional promotion and rollback, weighted canary releases, retries, circuit breakers, automatic stable fallback, Prometheus metrics, S3-compatible artifact storage, multi-tenant workspaces, secure OIDC browser sessions, a public SaaS console, an authenticated CLI, production configuration preflight, and a validated Terraform AWS reference deployment. The current post-merge CI checkpoint passes 168 Python tests and validates the full stack end to end. ## Resume bullets - Built **ModelForge**, a reliability-aware ML deployment platform in **Python/FastAPI, MySQL, SQLAlchemy, Docker, and Go**, supporting immutable model versioning, deployment lifecycle management, PyTorch/ONNX execution, and framework-independent external runtimes. - Designed a framework-agnostic runtime boundary with **bounded retries, exponential backoff, circuit breakers, health checks, Prometheus metrics, weighted canary releases, rollback, and automatic stable fallback** so future ML frameworks can integrate without control-plane rewrites. -- Shipped a **multi-tenant SaaS console + authenticated CLI + reproducible deployment proof**, with OIDC/PKCE browser sessions, hashed workspace API keys, strict CSP, S3-compatible artifacts, and a validated AWS ECS/Fargate + RDS + S3 Terraform reference; CI passes **164 Python tests** and the proof exercises weighted canary traffic, promotion, rollback, regression detection, failed-canary abort, and stable-target preservation through the public API. +- Shipped a **multi-tenant SaaS console + authenticated CLI + reproducible deployment proof**, with OIDC/PKCE browser sessions, hashed workspace API keys, strict CSP, S3-compatible artifacts, and a validated AWS ECS/Fargate + RDS + S3 Terraform reference; CI passes **168 Python tests** and the proof exercises weighted canary traffic, promotion, rollback, regression detection, failed-canary abort, and stable-target preservation through the public API. ## One-line project description @@ -34,6 +34,13 @@ The reference Go runtime runs as a separate HTTP service and participates in the ### 5. Claims are measured +The final validation layer includes a hosted release verifier that checks the +public HTTPS/security/auth/runtime surface before running the same deployment +proof against a real cloud instance. Until that hosted workflow passes, the +repository deliberately claims validated AWS architecture and hosted-proof +readiness rather than a completed public deployment. + + The repository includes reproducible load generation, fault injection, CI artifacts, p50/p95/p99 latency, throughput, error-rate, and traffic-lane reporting. A recruiter-proof harness also exercises stable serving, weighted canary traffic, promotion, rollback, regression detection, canary abort, and final stable-target preservation through the public API. ## 60-second interview answer @@ -42,7 +49,7 @@ I built ModelForge because training a model is only one piece of an ML system. I The control plane is FastAPI with MySQL and SQLAlchemy. Models are registered as immutable versions with checksums. Deployments have explicit lifecycle state, and environments have authoritative serving targets. For execution, ModelForge supports in-process runtimes like PyTorch and ONNX, but the important design choice is the external runtime contract. The Go runtime proves the control plane can serve through another language and process without adding Go-specific inference logic to the core. -Then I added reliability: retries, exponential backoff, circuit breakers, health inspection, weighted canaries, promotion, abort, rollback, and automatic stable fallback. I also turned the control plane into a multi-tenant SaaS foundation with workspaces, hashed API keys, OIDC Authorization Code + PKCE browser sessions, CSRF protection, strict CSP, and S3-compatible artifacts. The product has a public landing page, browser console, CLI, fault injection, and benchmark harness. The validated public-product checkpoint passes 164 Python tests and exercises the live Compose stack end to end. +Then I added reliability: retries, exponential backoff, circuit breakers, health inspection, weighted canaries, promotion, abort, rollback, and automatic stable fallback. I also turned the control plane into a multi-tenant SaaS foundation with workspaces, hashed API keys, OIDC Authorization Code + PKCE browser sessions, CSRF protection, strict CSP, and S3-compatible artifacts. The product has a public landing page, browser console, CLI, fault injection, and benchmark harness. The validated public-product checkpoint passes 168 Python tests and exercises the live Compose stack end to end. ## Good interview follow-up topics @@ -72,7 +79,7 @@ multi-tenant deployment without modifying the script. ## Validated metrics to use carefully -- 164 Python tests passed; 3 skipped in the standard Python CI job. +- 168 Python tests passed; 3 skipped in the standard Python CI job. - Full deployment proof passed in the live Compose stack: v2 canary promotion, rollback to v1, v3 output drift of 185.7% against a 50% threshold, v3 marked FAILED, and v1 preserved as the final active target.