Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .github/workflows/demo-proof.yml
Original file line number Diff line number Diff line change
Expand Up @@ -55,13 +55,12 @@ jobs:
- name: Install proof client
run: python -m pip install -e .

- name: Verify hosted service health
- name: Verify hosted production surface
shell: bash
run: |
set -euo pipefail
BASE_URL="${MODELFORGE_URL%/}"
curl --fail --silent --show-error "${BASE_URL}/health" >/dev/null
curl --fail --silent --show-error "${BASE_URL}/ready" >/dev/null
python -m modelforge.hosted_verify \
--output hosted-release-evidence.json

- name: Run hosted deployment proof
shell: bash
Expand All @@ -75,6 +74,7 @@ jobs:
with:
name: modelforge-hosted-demo-proof-${{ github.run_id }}
path: |
hosted-release-evidence.json
hosted-demo-proof/
hosted-demo-proof.log
if-no-files-found: ignore
Expand Down
5 changes: 4 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.PHONY: install lint test go-test check up down demo proof benchmark clean
.PHONY: install lint test go-test check up down demo proof hosted-verify benchmark clean

install:
python -m pip install -e ".[dev,pytorch,onnx]"
Expand Down Expand Up @@ -28,6 +28,9 @@ demo:
proof:
python demo/proof_scenario.py --environment recruiter-demo --output-dir demo-proof-evidence

hosted-verify:
python -m modelforge.hosted_verify

benchmark:
python benchmarks/bootstrap_external.py --environment benchmark-go
python benchmarks/load_test.py --environment benchmark-go --requests 5000 --concurrency 50 --warmup 100 --output benchmark-results.json
Expand Down
21 changes: 19 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Rather than treating model serving as a single `/predict` endpoint, ModelForge t

| Signal | Current validated result |
|---|---|
| Python test suite | **164 passed**, 3 skipped |
| Python test suite | **168 passed**, 3 skipped |
| Go validation | `gofmt`, `go test ./...`, and `go vet ./...` green |
| CI benchmark | **200 / 200 successful requests**, 0% errors |
| Throughput | **133.5 requests/s** |
Expand Down Expand Up @@ -42,6 +42,23 @@ intentionally regressed canary, automated abort, and final stable-target
verification. It writes JSON/JSONL/Markdown evidence to
`demo-proof-evidence/` and exits non-zero if any lifecycle assertion fails.

For a real public deployment, ModelForge also includes a non-mutating hosted
release gate:

```bash
MODELFORGE_URL=https://YOUR_PUBLIC_HOST \
MODELFORGE_API_KEY=mf_live_... \
MODELFORGE_WORKSPACE=YOUR_WORKSPACE_SLUG \
make hosted-verify
```

It verifies HTTPS/HSTS/CSP, OIDC mode, unauthenticated access controls,
workspace API-key identity, readiness, runtime inventory, and external-runtime
health before the hosted lifecycle proof runs. See
[`docs/hosted-demo.md`](docs/hosted-demo.md). A successful public hosted run
is intentionally **not claimed** until the `Hosted Demo Proof` workflow passes
against the real deployment.

For the smaller interactive stable + canary seed:

```bash
Expand Down Expand Up @@ -472,7 +489,7 @@ Reliability and performance characteristics should be demonstrated through tests

## Testing

The post-merge validation checkpoint currently reports **164 Python tests passed, 3 skipped**, with Go formatting, unit tests, and vet checks green. The project includes tests across the major system boundaries:
The post-merge validation checkpoint currently reports **168 Python tests passed, 3 skipped**, with Go formatting, unit tests, and vet checks green. The project includes tests across the major system boundaries:

```text
Artifact storage
Expand Down
128 changes: 128 additions & 0 deletions docs/hosted-demo.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
# Hosted Demo Launch

This is the final ModelForge proof step. The repository already validates the
full platform locally and in CI; this run proves the same lifecycle against the
real public HTTPS deployment.

## What must exist first

Before running the hosted proof, the AWS reference stack must be provisioned and
activated:

- public HTTPS DNS name and ACM certificate;
- ECS/Fargate API and Go-runtime services with desired count greater than zero;
- RDS MySQL;
- S3 artifact bucket;
- populated runtime and metrics secrets;
- configured OIDC application/callback;
- successful database migration task;
- successful Release AWS workflow.

The AWS bootstrap and release sequence is documented in infra/aws/README.md.

## First browser login

Open the production URL in a browser and sign in through the configured OIDC
provider.

Create or select the workspace that will be used for recruiter proof. In that
workspace create a developer API key. The raw mf_live_ secret is shown only
once.

Use a dedicated demo workspace instead of a personal production workspace when
possible. The proof intentionally creates immutable model versions and
deployment records.

## GitHub production environment

Configure these non-secret variables:

~~~text
MODELFORGE_PUBLIC_URL=https://YOUR_PUBLIC_HOST
MODELFORGE_DEMO_WORKSPACE=YOUR_WORKSPACE_SLUG
~~~

Configure this environment secret:

~~~text
MODELFORGE_DEMO_API_KEY=mf_live_...
~~~

The same protected production environment may already contain the AWS release
variables used by release-aws.yml.

## Verify the hosted surface

You can run the verifier locally without changing data:

~~~bash
MODELFORGE_URL=https://YOUR_PUBLIC_HOST \
MODELFORGE_API_KEY=mf_live_... \
MODELFORGE_WORKSPACE=YOUR_WORKSPACE_SLUG \
python -m modelforge.hosted_verify
~~~

It checks HTTPS, health/readiness, OIDC mode, HSTS, strict CSP,
unauthenticated-request rejection, protected metrics, workspace API-key
identity, runtime inventory, and external-runtime health. The report never
includes the raw API key.

## Run the final hosted proof

In GitHub:

1. Open Actions.
2. Select Hosted Demo Proof.
3. Choose Run workflow from main.
4. Keep the default serving environment or choose another isolated name.
5. Run it.

The workflow first runs the packaged hosted verifier, then executes the same
full-lifecycle proof used by local CI:

~~~text
stable v1
-> v2 weighted canary
-> real traffic through both lanes
-> promote v2
-> rollback to v1
-> v3 regression canary
-> detect output drift
-> abort v3 as FAILED
-> verify v1 remains authoritative
~~~

A successful run uploads one GitHub Actions artifact containing:

~~~text
hosted-release-evidence.json
hosted-demo-proof/proof-summary.json
hosted-demo-proof/traffic-samples.jsonl
hosted-demo-proof/proof-report.md
hosted-demo-proof.log
~~~

That artifact is the evidence to cite when the README says the public hosted
demo passed.

## Portfolio capture

After the hosted proof is green, capture real screenshots of:

1. the public ModelForge landing page;
2. signed-in workspace overview;
3. model registry with the proof versions;
4. stable/canary deployment state;
5. runtime fleet showing the external Go runtime healthy;
6. the GitHub Hosted Demo Proof run and PASS artifact.

Do not fabricate or mock these screenshots. They should come from the real
hosted deployment and real proof run.

## Final claim boundary

Before the hosted workflow passes, describe ModelForge as having a validated AWS
deployment architecture and hosted-proof workflow.

After it passes, it is accurate to say the identical deployment lifecycle was
executed successfully against the public hosted ModelForge service.
15 changes: 11 additions & 4 deletions docs/recruiting.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,13 @@

## 30-second pitch

ModelForge is a reliability-aware ML deployment platform I built to explore the systems problems that appear after model training. It combines a Python/FastAPI control plane, MySQL-backed model and deployment state, framework-specific in-process runtimes, and a standalone Go inference runtime behind a framework-agnostic protocol. The platform supports immutable artifacts, transactional promotion and rollback, weighted canary releases, retries, circuit breakers, automatic stable fallback, Prometheus metrics, S3-compatible artifact storage, multi-tenant workspaces, secure OIDC browser sessions, a public SaaS console, an authenticated CLI, production configuration preflight, and a validated Terraform AWS reference deployment. The current post-merge CI checkpoint passes 164 Python tests and validates the full stack end to end.
ModelForge is a reliability-aware ML deployment platform I built to explore the systems problems that appear after model training. It combines a Python/FastAPI control plane, MySQL-backed model and deployment state, framework-specific in-process runtimes, and a standalone Go inference runtime behind a framework-agnostic protocol. The platform supports immutable artifacts, transactional promotion and rollback, weighted canary releases, retries, circuit breakers, automatic stable fallback, Prometheus metrics, S3-compatible artifact storage, multi-tenant workspaces, secure OIDC browser sessions, a public SaaS console, an authenticated CLI, production configuration preflight, and a validated Terraform AWS reference deployment. The current post-merge CI checkpoint passes 168 Python tests and validates the full stack end to end.

## Resume bullets

- Built **ModelForge**, a reliability-aware ML deployment platform in **Python/FastAPI, MySQL, SQLAlchemy, Docker, and Go**, supporting immutable model versioning, deployment lifecycle management, PyTorch/ONNX execution, and framework-independent external runtimes.
- Designed a framework-agnostic runtime boundary with **bounded retries, exponential backoff, circuit breakers, health checks, Prometheus metrics, weighted canary releases, rollback, and automatic stable fallback** so future ML frameworks can integrate without control-plane rewrites.
- Shipped a **multi-tenant SaaS console + authenticated CLI + reproducible deployment proof**, with OIDC/PKCE browser sessions, hashed workspace API keys, strict CSP, S3-compatible artifacts, and a validated AWS ECS/Fargate + RDS + S3 Terraform reference; CI passes **164 Python tests** and the proof exercises weighted canary traffic, promotion, rollback, regression detection, failed-canary abort, and stable-target preservation through the public API.
- Shipped a **multi-tenant SaaS console + authenticated CLI + reproducible deployment proof**, with OIDC/PKCE browser sessions, hashed workspace API keys, strict CSP, S3-compatible artifacts, and a validated AWS ECS/Fargate + RDS + S3 Terraform reference; CI passes **168 Python tests** and the proof exercises weighted canary traffic, promotion, rollback, regression detection, failed-canary abort, and stable-target preservation through the public API.

## One-line project description

Expand All @@ -34,6 +34,13 @@ The reference Go runtime runs as a separate HTTP service and participates in the

### 5. Claims are measured

The final validation layer includes a hosted release verifier that checks the
public HTTPS/security/auth/runtime surface before running the same deployment
proof against a real cloud instance. Until that hosted workflow passes, the
repository deliberately claims validated AWS architecture and hosted-proof
readiness rather than a completed public deployment.


The repository includes reproducible load generation, fault injection, CI artifacts, p50/p95/p99 latency, throughput, error-rate, and traffic-lane reporting. A recruiter-proof harness also exercises stable serving, weighted canary traffic, promotion, rollback, regression detection, canary abort, and final stable-target preservation through the public API.

## 60-second interview answer
Expand All @@ -42,7 +49,7 @@ I built ModelForge because training a model is only one piece of an ML system. I

The control plane is FastAPI with MySQL and SQLAlchemy. Models are registered as immutable versions with checksums. Deployments have explicit lifecycle state, and environments have authoritative serving targets. For execution, ModelForge supports in-process runtimes like PyTorch and ONNX, but the important design choice is the external runtime contract. The Go runtime proves the control plane can serve through another language and process without adding Go-specific inference logic to the core.

Then I added reliability: retries, exponential backoff, circuit breakers, health inspection, weighted canaries, promotion, abort, rollback, and automatic stable fallback. I also turned the control plane into a multi-tenant SaaS foundation with workspaces, hashed API keys, OIDC Authorization Code + PKCE browser sessions, CSRF protection, strict CSP, and S3-compatible artifacts. The product has a public landing page, browser console, CLI, fault injection, and benchmark harness. The validated public-product checkpoint passes 164 Python tests and exercises the live Compose stack end to end.
Then I added reliability: retries, exponential backoff, circuit breakers, health inspection, weighted canaries, promotion, abort, rollback, and automatic stable fallback. I also turned the control plane into a multi-tenant SaaS foundation with workspaces, hashed API keys, OIDC Authorization Code + PKCE browser sessions, CSRF protection, strict CSP, and S3-compatible artifacts. The product has a public landing page, browser console, CLI, fault injection, and benchmark harness. The validated public-product checkpoint passes 168 Python tests and exercises the live Compose stack end to end.

## Good interview follow-up topics

Expand Down Expand Up @@ -72,7 +79,7 @@ multi-tenant deployment without modifying the script.

## Validated metrics to use carefully

- 164 Python tests passed; 3 skipped in the standard Python CI job.
- 168 Python tests passed; 3 skipped in the standard Python CI job.
- Full deployment proof passed in the live Compose stack: v2 canary promotion,
rollback to v1, v3 output drift of 185.7% against a 50% threshold, v3 marked
FAILED, and v1 preserved as the final active target.
Expand Down
Loading
Loading