ML Inference Service (starter)

This is a minimal starter repo demonstrating a tiny PyTorch model served with FastAPI and containerized with Docker. Use it as a resume/demo project to show model serving, latency profiling, and containerization.

Quick start (local, without Docker)

1. Create a virtualenv and install requirements

   python -m venv .venv; ..venv\Scripts\Activate.ps1; pip install -r requirements.txt

2. Run the app

   uvicorn app.main:app --host 0.0.0.0 --port 8000 --workers 1

3. Smoke test

PowerShell (recommended on Windows):

check health

Invoke-RestMethod -Method Get -Uri http://localhost:8000/health

example predict

$body = @{ features = @(0.1,0.0,0.2,0.3,0.9,0.5,0.1,0.7,0.2,0.4,0.6,0.8,0.0,0.1,0.3,0.2) } | ConvertTo-Json Invoke-WebRequest -Method Post -Uri http://localhost:8000/predict -Body $body -ContentType 'application/json'

Or using curl (cross-platform):

curl -X GET http://localhost:8000/health

curl -X POST http://localhost:8000/predict -H "Content-Type: application/json" -d '{"features":[0.1,0.0,0.2,0.3,0.9,0.5,0.1,0.7,0.2,0.4,0.6,0.8,0.0,0.1,0.3,0.2]}'

You can also run the included PowerShell smoke-test script:

.\scripts\test.ps1

Docker

Build and run (Docker)

docker build -t ml-infer-service:latest . docker run --rm -p 8000:8000 ml-infer-service:latest

If docker is not available on your machine (you saw "The term 'docker' is not recognized"), you can run the app directly with Python or use the helper script which prefers Docker but falls back to the venv/python:

run helper (will use Docker if present, otherwise runs uvicorn with venv or system python)

.\scripts\run.ps1

Troubleshooting tips:

• Ensure Docker Desktop is installed and running on Windows. After installation, open a new PowerShell so docker is in PATH.
• If you don't want to install Docker, create and use the venv and run uvicorn directly (see Quick start).

Load testing (quick)

Install autocannon (node):

npm install -g autocannon

Or use the bundled Python load tester (no Node required). First install requests into your venv:

..venv\Scripts\python -m pip install requests

Then run the tester (example: 50 concurrency for 30s):

..venv\Scripts\python .\scripts\load_test.py --url http://localhost:8000/predict --concurrency 50 --duration 30

If you prefer autocannon, run:

autocannon -d 30 -c 50 -m POST -H "Content-Type: application/json" -b '{"features":[0.1,0.0,0.2,0.3,0.9,0.5,0.1,0.7,0.2,0.4,0.6,0.8,0.0,0.1,0.3,0.2]}' http://localhost:8000/predict

Resume bullets you can add (after measuring):

• Containerized ML inference service (FastAPI + Docker); handled 50 rps locally with p99 ≈ 2100 ms (2.1 s).
• Implemented /health and /predict endpoints, request validation, and basic server-timing profiling.

Measured performance (local quick test):

• Test command: .\scripts\load_test.py --url http://localhost:8000/predict --concurrency 50 --duration 10
• Total requests: 250 (50 concurrency × 10s)
• Throughput: ~25 req/s (overall during test)
• p50 = 2057 ms, p90 = 2069 ms, p95 = 2072 ms, p99 = 2080 ms, avg ≈ 2057 ms