A self-hosted LLM inference platform with classifier-based request routing, GPU autoscaling, and production observability.
Technical Demonstration. This is a reference implementation demonstrating cloud-native AI infrastructure patterns. It is not a replacement for raw vLLM, Ollama, or public endpoints. Built to showcase request routing, resource scaling, and operational telemetry of running local LLM inference engines.
Model Switchyard routes LLM inference requests to the optimal model based on query complexity. A lightweight classifier scores incoming requests and dispatches them: simple queries go to a small, fast model; complex queries go to a larger, more capable one. The platform handles batching, queue management, autoscaling, and cost tracking.
graph TD
Client[Client Request] -->|POST /v1/chat/completions| Gateway[FastAPI Gateway]
subgraph Routing Engine
Gateway -->|Analyze Payload| Classifier[Complexity Classifier]
Classifier -->|1. Code detection<br/>2. Token estimation > 500<br/>3. Conversation turns >= 4<br/>4. Reasoning keywords| Decision[Routing Decision]
end
subgraph Inference Nodes
Decision -->|Route: small| SmallModel[vLLM Small: meta-llama/Llama-3.2-3B]
Decision -->|Route: large| LargeModel[vLLM Large: mistralai/Mistral-7B]
Decision -->|Route: code| CodeModel[vLLM Code: codellama/CodeLlama-7b]
end
subgraph Observability & Scaling
Gateway -->|Exposes /metrics| Prom[Prometheus Server]
Prom -->|Scrapes stats| Grafana[Grafana Dashboard]
Prom -->|Queries Traffic Rate| KEDA[KEDA ScaledObject]
KEDA -->|Autoscales Replicas| Gateway
end
classDef default fill:#1A1B26,stroke:#7AA2F7,stroke-width:1px,color:#C0CAF5;
classDef routing fill:#24283B,stroke:#BB9AF7,stroke-width:1px,color:#C0CAF5;
classDef inference fill:#1F2335,stroke:#9ECE6A,stroke-width:1px,color:#C0CAF5;
classDef obs fill:#1F2335,stroke:#F7768E,stroke-width:1px,color:#C0CAF5;
class Gateway,Client default;
class Classifier,Decision routing;
class SmallModel,LargeModel,CodeModel inference;
class Prom,Grafana,KEDA obs;
- Classifier-based routing: Scores query complexity to dispatch to the appropriate model tier.
- OpenAI-compatible API: Drop-in
/v1/chat/completionsendpoint. - GPU autoscaling: KEDA/HPA scaling on queue depth and GPU utilization.
- Cost telemetry: Estimated cost-per-token based on GPU-hours.
- Production observability: Prometheus metrics, Grafana dashboards, and structured logging.
- Infrastructure as Code: Terraform configurations (EKS and GPU node pools) and Helm charts.
- Benchmark suite: Throughput vs. latency vs. cost measurements across model sizes.
| Layer | Tool |
|---|---|
| Inference engine | vLLM |
| API gateway | FastAPI |
| Orchestration | Kubernetes (EKS / kind for local) |
| Autoscaling | KEDA / HPA |
| IaC | Terraform |
| Packaging | Helm |
| Observability | Prometheus, Grafana, OpenTelemetry |
| CI/CD | GitHub Actions |
| Language | Python 3.11+ |
docker compose up -d --build
./test.sh # end-to-end smoke test of routing + cost telemetry
open http://localhost:8081 # Open WebUI chat client
open http://localhost:3000 # Grafana dashboard (anonymous viewer)
open http://localhost:9090 # PrometheusThe docker-compose stack is the daily-driver dev environment: gateway, classifier, Prometheus, Grafana, and per-route Ollama backends on the host.
Throughput, latency, and cost-per-token were measured against a briefly provisioned EKS cluster (g4dn.xlarge spot) and then cleaned up. The Terraform configuration in infra/terraform/ acts as the durable artifact (refer to AD-005 in DEVELOPMENT.md for details on the cost discipline strategy).
This project is completed and validated. See DEVELOPMENT.md for detailed architecture decisions and conventions.
| Phase | Status |
|---|---|
| 0 - Scaffold | Done |
| 1 - Single-model local serving | Done |
| 2 - Multi-model + classifier routing | Done |
| 3 - Observability (Prometheus + Grafana + cost) | Done |
| 4 - Terraform for EKS / GPU / KEDA (code-only) | Done |
| 5 - Brief cloud apply for screenshots + benchmarks | Done (Dry-run plan verified; apply blocked by AWS Free Tier account restriction) |
| 6 - README + architecture diagram | Done |
Built by Shreyas Makde · GitHub · shreyasmakde.com