ComputeBench

ComputeBench is an open-source tool for comparing GPU cloud providers using real ML workloads instead of advertised $/GPU-hour.

Status: alpha / experimental. Architecture is provider-agnostic; the first real adapter is RunPod. Most “results” you can produce today without credits are mock or dry-run simulations, not validated cloud measurements.

Why it exists

Advertised GPU price is a weak decision signal. A cheaper GPU that is slower on your workload can cost more per unit of useful work. ComputeBench runs the same workload spec across providers and ranks by workload economics (e.g. effective $/1M output tokens), when real measurements are available.

What works today

• Local CLI (computebench run|prepare|providers|runpod-verify)
• Mock providers with different price/performance curves
• RunPod adapter with dry-run, paid opt-in, cleanup, session timeout
• Primary workload: vLLM inference on a pinned TinyLlama revision + committed prompts
• Secondary: CUDA matmul smoke test (not for ranking)
• JSONL local result storage with per-request + environment metadata
• Unit tests with fakes (no live cloud calls)

What is not yet validated

• Public cross-provider rankings from real measured runs
• Broad provider coverage (only RunPod is a live adapter so far)
• Production multi-tenant SaaS, billing, or marketplace features
• Full async load generation / production serving SLOs
• Independence from cold-start effects (pip install, HF download, image pull)

Architecture overview

workload.yaml
    → CLI
    → runner (repeats, store, ranking)
    → Provider adapter (provision → run → cleanup)
    → benchmark harness (vLLM or cuda_smoke)

Provider SDKs stay behind adapters. Ranking uses standardized metrics, not provider APIs.

Safe local quickstart ($0)

cd computebench
PYTHONPATH=src python3 -m computebench prepare examples/dry_run/vllm_inference.yaml
PYTHONPATH=src python3 -m computebench run examples/dry_run/vllm_inference.yaml --dry-run -v
PYTHONPATH=src python3 -m computebench run examples/dry_run/two_provider_vllm.yaml --dry-run --markdown
PYTHONPATH=src python3 -m pytest -q

Optional install:

python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
computebench run examples/dry_run/vllm_inference.yaml --dry-run

Example dry-run

PYTHONPATH=src python3 -m computebench run examples/dry_run/vllm_inference.yaml --dry-run

You should see a table and a note that mock/dry-run rows are not real cloud evidence.

Live-provider warning (charges money)

Creating GPU pods incurs charges. Required opt-in:

export RUNPOD_API_KEY=...          # never commit
export RUNPOD_ALLOW_PAID=1         # required opt-in
unset RUNPOD_DRY_RUN COMPUTEBENCH_DRY_RUN
PYTHONPATH=src python3 -m computebench run examples/live/vllm_inference_runpod.yaml -v

Without the matching *_ALLOW_PAID=1 flag, ComputeBench refuses to create billable resources. Default examples under examples/dry_run/ are non-billable. Live sessions honor RUNPOD_MAX_SESSION_S / VAST_MAX_SESSION_S (default 3600).

Provider docs: docs/RUNPOD.md · docs/VAST.md · docs/LIGHTNING.md · docs/FIRST_LIVE_RUN.md.

Current provider support

Provider	Mode
budget_spot, steady_cloud, turbo_cluster	Mock only
runpod	Dry-run + live (opt-in RUNPOD_ALLOW_PAID=1)
vast	Dry-run + live (opt-in VAST_ALLOW_PAID=1)
lightning	Dry-run + live Studio CPU↔GPU (opt-in LIGHTNING_ALLOW_PAID=1)

Current benchmark support

Workload	Notes
vllm_inference / frozen vllm_inference_v1	Primary. See docs/BENCHMARK_V1.md
cuda_smoke	Fast infra probe; not for provider economics claims
legacy llm_inference mocks	Still supported for simple demos

Methodology: docs/VLLM_BENCHMARK.md · Integrity: docs/BENCHMARK_INTEGRITY.md · First live run: docs/FIRST_LIVE_RUN.md

Output / metrics

Aggregates include provisioning time, setup/load time, benchmark wall time, throughput, latency percentiles, TTFT (when available), failures, estimated run cost, and effective $/1M output tokens. JSONL also stores per_request, environment, and config pins.

Repository structure

src/computebench/     CLI, runner, metrics, providers, benchmarks
examples/dry_run/     Safe $0 examples
examples/live/        Paid examples (see WARNING.md)
docs/                 RunPod, vLLM methodology, integrity
tests/                Offline unit tests
results/              Local outputs only (gitignored artifacts)

Adding a provider

1. Implement providers/base.py methods (provision, run_llm_inference, cleanup).
2. Register the adapter in providers/registry.py.
3. Keep SDK/pricing quirks inside the adapter.
4. Gate billable creates behind an explicit env opt-in (YAML alone must not spend).
5. Provide a dry-run path that never creates or deletes paid resources.
6. Add fake-client unit tests (no network charges).

Details: CONTRIBUTING.md.

Adding a benchmark

1. Prefer a provider-agnostic harness under benchmarks/.
2. Commit a fixed prompt dataset when prompts matter.
3. Pin model id and revision, plus engine version.
4. Persist environment + config metadata in stored results.
5. Keep smoke probes separate from the primary ranking workload.

Methodology: docs/VLLM_BENCHMARK.md.

Benchmark methodology & integrity

• Methodology: docs/VLLM_BENCHMARK.md
• Integrity / disclosure: docs/BENCHMARK_INTEGRITY.md

Public comparisons should only use real measured runs, with methodology published. Pricing and availability change; early methodology may evolve.

Limitations

• Alpha quality; APIs and specs may change
• One live provider adapter so far
• Batch concurrency model (not full async loadgen)
• Cold installs/downloads can dominate cost and time
• Catalog prices are snapshots

Roadmap (informal)

• More provider adapters behind the same interface
• Stronger TTFT/loadgen fidelity
• Optional result signing / run manifests
• Caching guidance to reduce cold-start bias

Contributing

See CONTRIBUTING.md.

License

Apache License 2.0 — see LICENSE.