Prove your local LLM fits in memory — or get a loud refusal instead of a silent OOM.
fitsproof is for people running LLMs on consumer hardware (4–8 GB VRAM / 16–32 GB RAM): a class of machine every mainstream engine either ignores or silently falls back from. It predicts peak memory for your machine from an on-device calibration, enforces a declared budget (admit / degrade loudly / refuse), and proves it with a measured stress harness.
python3 -m venv .venv && . .venv/bin/activate
pip install "git+https://github.com/AnnasMazhar/fitsproof.git"
fitsproof admit --budget-gb 4
fitsproof admit --budget-gb 0.001 # REFUSED — names the binding constraint, exit code 2Python 3.11+. Everything runs offline after install. Once the PyPI release is published
(pending — pip install fitsproof is not yet available), use the Git-URL form above.
Stress harness result — 25 configs, zero budget violations, zero silent mode changes:
$ fitsproof stress
ADMITTED: 0.039 GB predicted peak <= 4.000 GB budget (margin: 3961.0 MB)
Stress harness: 25 configs, 0 violations, 0 silent mode changes. Margin: min=3909.4 MB, median=3909.7 MB, max=3913.1 MB.
If this is useful, star the repo.
The claim above says fitsproof predicts peak memory. How accurately? Measured on this
machine, offline, against the in-repo reference model (reproduce:
python scripts/calibration_demo.py):
$ python scripts/calibration_demo.py
=== Calibration demo ===
bandwidth: 1.92 GB/s
gemm: 37.19 GFLOPS
RAM: 33.5 GB
bandwidth_utilisation: 0.0487
MAPE (held-out): 46.1%
CI (95%): [46.1%, 46.1%] ← n_held_out=1: degenerate interval (not a range); see docs/ADOPTION.md F-3
n_train=2, n_held_out=1
Note: n_held_out=1 — the CI is a point, not an interval. Collect n >= 10 held-out
measurements for a meaningful interval (Davison & Hinkley 1997, §2.4). The MAPE itself is still valid.
The bandwidth reading above (1.92 GB/s) is low because the box was under load when
this transcript was recorded. MAPE varies with bandwidth measurement: across sessions
on this machine the observed range is ~30–65% (documented in docs/ADOPTION.md
F-3). Run calibration_demo.py yourself; your number will differ.
The [46.1%, 46.1%] interval is a degenerate point because the calibration run uses
only 3 measurements (n_train=2, n_held_out=1). Bootstrap CI requires n ≥ 2 held-out
samples to produce a genuine range. With n_held_out=1, every resample returns the
same single value, so lower == upper. The MAPE itself (46.1%) is valid; the CI is
not informative at n=1. See docs/ADOPTION.md F-3 for the full analysis and closure
path (Davison & Hinkley 1997, §2.4).
And against a real 4.3 GB model through ollama (raw transcripts in docs/ADOPTION.md §2):
predicted peak: 7.219 GB ollama observed resident: 4.4 GB error: +64% (over-prediction)
How to read this honestly:
- The error is one-sided, in the safe direction for a refusal gate. fitsproof
over-predicts: if it admits a config, the real load fits with margin. The dangerous
direction — predicting a fit that then OOMs — has not been observed
(
docs/RESEARCH.md, Pass 3 falsification check 1). - It is not free. A budget between the true footprint (4.4 GB) and the prediction
(7.22 GB) gets a false degradation. Held-out n is small (1) and the MAPE is large.
This is the top adoption risk — documented as finding F-1 in
docs/ADOPTION.md§5, not hidden. - Prediction is the least-proven of the three pillars. Enforcement (
admit) and proof (stress,verify) are measured directly; the prediction feeds them conservatively.
# SURFACE: client
from fitsproof.client import DoesNotFit, FitsproofClient
client = FitsproofClient()
plan = client.plan(context_len=512, budget_bytes="4GiB")
record = client.admit(plan)
print(record.message)
try:
client.admit(client.plan(context_len=512, budget_bytes="1MiB"))
except DoesNotFit as e:
print("refused:", e)
assert client.metrics()["ram_gb"] > 0# SURFACE: guard
from fitsproof.client import DoesNotFit, guard
loaded = []
@guard(budget="1MiB") # the reference model needs ~40 MB: this config cannot fit
def load_model():
loaded.append("allocated")
try:
load_model()
except DoesNotFit as e:
print(e)
assert loaded == [] # the wrapped callable was provably never invoked# SURFACE: server
import json
import socket
import time
import urllib.request
from fitsproof.engine.model import get_reference_bundle
from fitsproof.engine.server import start_server
from fitsproof.engine.transformer import Transformer
cfg, weights = get_reference_bundle()
with socket.socket() as s:
s.bind(("127.0.0.1", 0))
port = s.getsockname()[1]
srv = start_server(Transformer(cfg, weights), cfg, host="127.0.0.1", port=port, block=False)
# Your existing client, pointed at fitsproof:
# from openai import OpenAI
# client = OpenAI(base_url=f"http://127.0.0.1:{port}/v1", api_key="unused")
payload = {
"messages": [{"role": "user", "content": "hi"}],
"max_tokens": 5,
"temperature": 0.0,
"stream": False,
}
req = urllib.request.Request(
f"http://127.0.0.1:{port}/v1/chat/completions",
data=json.dumps(payload).encode(),
headers={"Content-Type": "application/json"},
)
for _ in range(30):
try:
with urllib.request.urlopen(req, timeout=30) as resp:
body = json.loads(resp.read())
break
except Exception:
time.sleep(0.2)
else:
raise RuntimeError("server did not start")
assert body["choices"][0]["message"]["content"]
assert body["fitsproof"]["admission"] in ("admitted", "degraded")
srv.shutdown()Every response carries a fitsproof field with the admission record. A request whose predicted
peak exceeds the server's budget is refused with HTTP 503 and the binding constraint named —
never a silent OOM.
# SURFACE: mcp
import json
import subprocess
import sys
messages = [
{"jsonrpc": "2.0", "id": 1, "method": "initialize", "params": {}},
{"jsonrpc": "2.0", "id": 2, "method": "tools/list", "params": {}},
{
"jsonrpc": "2.0",
"id": 3,
"method": "tools/call",
"params": {"name": "admit", "arguments": {"budget": "4GiB", "context_len": 512}},
},
]
proc = subprocess.run(
[sys.executable, "-m", "fitsproof.cli", "mcp"],
input="\n".join(json.dumps(m) for m in messages) + "\n",
capture_output=True,
text=True,
timeout=120,
)
replies = [json.loads(line) for line in proc.stdout.splitlines() if line.strip()]
assert replies[0]["result"]["serverInfo"]["name"] == "fitsproof-mcp"
tool_names = {t["name"] for t in replies[1]["result"]["tools"]}
assert {"probe", "plan", "admit"} <= tool_names
assert replies[2]["result"]["isError"] is FalseRegister it with any MCP host as fitsproof mcp (stdio).
fitsproof is not a new CUDA kernel. It does not claim speed superiority over
llama.cpp, vLLM, or KTransformers. Those engines are faster, more mature, cover
more hardware, and support more models. COMPARISONS.md names each one and states
where it beats us.
The engine exists to make the contract real and testable: a NumPy-only correctness-first runtime that runs offline with no GPU and no CUDA toolkit.
plan and admit are distinct commands. plan describes the prediction without enforcing;
admit enforces it (exit 2 on refusal). Use plan to inspect before committing to the gate.
fitsproof --version # print the package version (also -V)
fitsproof probe # measure this machine (bandwidth, GEMM, RAM/VRAM)
fitsproof plan # show predicted peak, CI, tok/s, verdict — does NOT enforce (always exit 0)
fitsproof admit # enforce the contract — ADMITTED / DEGRADED / REFUSED (exit 2 on refusal)
fitsproof verify # measure peak RSS during generation, assert <= budget
fitsproof stress # >=20 configs: zero violations, zero silent mode changes
fitsproof serve # OpenAI-compatible HTTP server (alias: fitsproof server)
fitsproof mcp # MCP server (stdio): plan / admit / probe tools
fitsproof pareto # measured Pareto frontier over (quant, context)
Example — inspect the prediction, then enforce:
$ fitsproof plan --budget-gb 4
predicted peak: 0.042 GB (95% CI: [0.033, 0.050] GB)
predicted tok/s: 104.0 (95% CI: [72.8, 135.1])
budget: 4.000 GB
verdict: fits
$ fitsproof admit --budget-gb 4
ADMITTED: 0.042 GB predicted peak <= 4.000 GB budget (margin: 3958.3 MB)
$ fitsproof plan --budget-gb 0.001
predicted peak: 0.042 GB (95% CI: [0.033, 0.050] GB)
predicted tok/s: 106.3 (95% CI: [74.4, 138.2])
budget: 0.001 GB
verdict: does_not_fit
degradation options:
[does not fit] Use int8_sym quantisation instead of none -> 0.011 GB (400.0 tok/s)
[does not fit] Use int4_sym quantisation instead of none -> 0.006 GB (741.6 tok/s)
...
# plan exits 0 — it describes; admit enforces (exit 2 on does_not_fit)
src/fitsproof/
engine/ NumPy-only transformer runtime
model.py Bundle format + in-repo reference model (6L/384H/GQA, ~38MB fp32)
attention.py MHA + GQA + RoPE + KV cache
quant.py int8_sym, int8_asym, int4_sym, int4_asym weight quantisation
sampling.py greedy, temperature, top-k, top-p (seeded RNG)
transformer.py RMSNorm + SwiGLU FFN + full forward pass
speculative.py speculative decoding (Leviathan et al. 2023)
server.py OpenAI-compatible HTTP server (stdlib only)
contract/ Resource contract enforcement
probe.py Machine characterisation from measurement (STREAM triad, GEMM)
cost.py Analytical roofline cost model
calibrate.py Fit model constants from on-device measurements; MAPE + CI
plan.py Compute verdict: fits / fits_with_degradation / does_not_fit
admit.py Enforcement point — emits a record on every mode change
verify.py Proof harness — measures peak RSS vs declared budget
pareto.py Pareto frontier sweep over (quant, context) configurations
client.py FitsproofClient, DoesNotFit, @guard decorator
mcp.py MCP stdio server (probe / plan / admit tools)
cli.py fitsproof command
These are honest. A repo with no stated limitations is not credible.
-
No CUDA kernels. The engine is NumPy-only. It is correctness-first and slow vs llama.cpp or vLLM. That is intentional; the product is the contract, not speed.
-
Calibration is CPU-only. The cost model is calibrated from DRAM bandwidth and NumPy GEMM measurements. It does not account for GPU memory hierarchy or compute rooflines.
-
Prediction error is large and one-sided. Held-out MAPE is ~30–65% at reference scale (n_held_out=1, varies with machine load at measurement time — see F-3 in
docs/ADOPTION.md), and on a real GGUF model (gemma3:4b) the peak is over-predicted by +64% (7.22 GB predicted vs 4.4 GB observed):head_dimdefaults to hidden/heads and embeddings are accounted in fp32. Refusals stay safe (the error is conservative), but budgets between the true and predicted footprint get false degradations. Full analysis:docs/ADOPTION.mdF-1. -
Bootstrap CI degenerates to a point at n_held_out=1. The calibration demo uses 3 measurements (n_train=2, n_held_out=1). With a single held-out sample, every bootstrap resample returns the same value — lower == upper. The MAPE itself is valid; the interval is not informative at n=1. Collecting n ≥ 10 held-out measurements produces a genuine range (Davison & Hinkley 1997, §2.4;
docs/ADOPTION.mdF-3). The[X%, X%]in the calibration demo output is annotated at runtime to explain the degeneracy. -
Contract covers memory, not latency SLOs. fitsproof enforces a peak RSS budget; it does not guarantee latency targets (tok/s predictions are estimates).
-
KV cache bandwidth not included in decode formula. At long contexts, KV cache streaming adds to the bandwidth cost. The current formula is accurate for the reference model and short contexts; it underpredicts decode time at very long contexts.
-
RSS measurement is per-config delta, not absolute. The proof harness reads
/proc/self/statusVmRSS (current RSS) before and after each run and reports the delta. It cannot attribute RSS held across calls (e.g. NumPy arena memory) to any single configuration, so the margin figures are conservative rather than exact. The per-config peak is the maximum of pre- and post-call samples. -
Boundary safety margin.
admitreturns NEAR_BOUNDARY (exit 1) when the predicted peak is within 50 MiB (52 MB) of the declared budget. This covers a ~25% prediction overrun on a small model (~70 MB weights × 1.25 = ~17 MB overrun, well under the margin). Configurations flagged as NEAR_BOUNDARY should be validated withfitsproof verifybefore use. Admission at a tight budget does not guarantee the proof harness will pass. -
Speculative decoding equality only holds at temperature=0. Probabilistic acceptance (temperature > 0) requires rejection sampling (Algorithm 1 of Leviathan et al. 2023), not implemented.
-
No NTK-aware RoPE scaling. Context lengths beyond
max_seq_lenwill degrade. -
In-repo reference model is randomly initialised. It generates valid token sequences but not coherent text. It exists to run all contract tests offline.
See COMPARISONS.md for the full table with star counts and release dates. Short version:
| Tool | What it does better than fitsproof |
|---|---|
| llama.cpp | Mature, broad model support, fast CPU kernels, broad quant support |
| vLLM | GPU serving, high throughput, PagedAttention, broad model support |
| KTransformers | CPU/GPU hybrid MoE, AMX kernels, runs 671B on ~14 GB VRAM |
| ridgepoint | Calibrated VRAM/roofline prediction for GPU (A100/H100) |
| Strata | Consumer packaging, one-click install |
| aura | Kernel-level (cgroup v2 / Win32 Job Object) budget enforcement for local LLMs |
fitsproof's position: aura enforces budgets at the OS level, but none of the above pairs
enforcement with an on-device calibrated prediction (held-out MAPE published) and a
measured proof of compliance — the stress harness asserting measured <= budget over
25 configurations. That is the claim.
docs/demo.sh is a 15-second end-to-end demo. Recording instructions are in its header.
See CONTRIBUTING.md. Every test file must name the fault it detects, and core tests
must cite their research source.
MIT — see LICENSE.