Measure what your AI inference actually costs.
Most teams running AI in production know exactly one number: the monthly cloud bill. They can't tell you cost per endpoint, per feature, or per prediction. When the bill doubles, they can't say why.
This tells you.
Two ways to use it. Pick whichever you are comfortable with.
One file, no install. Standard library only. No network code in it — grep it. It writes JSON lines to a file on your own disk and nothing leaves the machine.
https://github.com/qaisermehdi3-coder/qvunex/blob/main/qvunex_single.py
Run it with no API key and no network first, to see what the output looks like:
python3 qvunex_single.py --demo
Or the package, if you would rather:
pip install qvunexfrom qvunex import meter
@meter("checkout-classifier")
def predict(batch):
return model(batch)qvunex report
qvunex checklistThat's the whole integration.
The method behind this section is written up on one page, so you can follow it with or without this tool: METHOD.md, how to price one GPU wake, billed time against served work.
The server already counts every request it finishes. Save its counters before and after your traffic, then compare:
curl -s localhost:8000/metrics > before.txt
# ... run your traffic ...
curl -s localhost:8000/metrics > after.txt
python3 qvunex_single.py --reconcile before.txt after.txt --gpu-rate 0.83--gpu-rate is what the server costs per hour. You get three things:
- Meter against server: requests, prompt tokens and output tokens on both sides, and the gap. Streamed calls that came back with no usage are recovered as a total from the server's own count.
- Server speed: prompt reading and answer writing per request, and output for the whole server over the window.
- GPU cost: the hourly rate over the window, idle time included, per request and per million output tokens.
The file still opens no network connection: curl saves the counters and the file reads them.
Tested live on a Colab T4, vLLM 0.27.1, Qwen2.5-0.5B, 25 September 2026:
- Meter and server agreed exactly in all 8 windows. The decode token count matched vLLM's separate inter-token gap counter, 504 = 504.
- 8 requests one at a time against 8 at once, three rounds each: per-request decode 169-170 against 122-124 tokens/s, while the whole server went 151-154 against 701-708 tokens/s.
So price by the whole-server figure. Priced from per-request speed, that load looks about 5.7x more expensive than it was. And skip the first window after a server starts: in an earlier run, 24 requests 8 at a time took 12.8 s in the first window and 1.0 to 1.6 s in later windows on the same server.
Every wake is a cold start, and the boot is billed too. So take the first
snapshot as soon as the server answers, the second just before teardown, and
pass the whole billed time of the wake (create to delete) as --window, not
the time between the snapshots:
curl -s localhost:8000/metrics > before.txt # right after the health check
# ... serve this wake's requests ...
curl -s localhost:8000/metrics > after.txt # just before teardown
python3 qvunex_single.py --reconcile before.txt after.txt --gpu-rate 1.72 --window 161Tested live on a Colab T4, vLLM 0.27.1, 26 September 2026: one cold boot, 8 requests, clock started before the server launched. The counters already exist at boot, so the first snapshot works. Meter and server agreed exactly. The server took about 155 s to answer and served all 8 requests in the few seconds after, so about 96% of the paid time was boot: priced at $1.72/hour, $0.0096 per request. Priced from the snapshots alone, the same wake would look at least 14 times cheaper. The fewer requests a wake serves, the more the boot dominates.
Where one wake's bill went. Pass the whole billed time as --window,
plus --boot (billing start to the first snapshot) and --tail (second
snapshot to billing end, the cooldown before the server is torn down):
python3 qvunex_single.py --reconcile before.txt after.txt --gpu-rate 2.689 --window 226 --boot 160.2 --tail 60The report splits the bill into boot, the serving window between the snapshots, and the tail, in seconds, share and dollars. A part you leave out is shown as unknown, never as 0. Tested live on a Colab T4, vLLM 0.27.1, 2 October 2026: one cold wake, 8 requests, then 60 s answering only health checks, like a cooldown timer. Boot 160.2 s (70.9%), serving 5.8 s (2.6%), tail 60.0 s (26.6%), meter and server equal. On a real setup the tail is set by timers and can be much longer. In one published GKE run (KEDA 300 s cooldown, then the autoscaler's 600 s unneeded time), the tail was about 62% of the GPU node's time, and serving about 1%.
Paid, nothing served. If a paid window finishes 0 requests, the report
says so in a line of its own, with the dollar amount, instead of "unknown".
This is the failure where something keeps a scale-to-zero server awake, like
a gateway polling /v1/models every few minutes. Tested live on a Colab T4,
vLLM 0.27.1, 29 September 2026: a 60.1 s window with six /v1/models calls and
six /health calls, and no chat requests. vLLM counted 0 requests, 0 prompt
tokens and 0 output tokens for them, and the report printed
PAID, NOTHING SERVED $0.044867 for 60.1 s with 0 requests (priced at
$2.689/hour). So those polls leave no trace in vLLM's request counters. The
only sign is paid time with nothing served.
Two servers on one GPU. Each vLLM server keeps its own counters, so work stays attributed to the right server even when several share a GPU. Tested live on a Colab T4, vLLM 0.27.1, 29 September 2026: two servers on the same T4 as two processes (5450 and 5888 MiB; no MIG, no MPS, no Kubernetes), 8 requests to one and 3 to the other in the same window. Each server counted only its own work, 8 requests / 288 prompt / 91 output tokens and 3 / 108 / 30, and both matched what the client recorded exactly. Not tested yet: Kubernetes time-slicing itself, or a GPU monitor such as DCGM side by side.
Which hours were paid for nothing. On a server that stays up, save the
counters every few minutes and hand them all to --idle. Each window between
two snapshots is marked served, PAID, NOTHING SERVED, or unknown (a counter
missing, or a server restart that reset the counters, is never counted as
idle). Name each file by the Unix time, as below, and the window lengths come
from the names:
mkdir -p snaps
curl -s localhost:8000/metrics > snaps/$(date +%s).txt # repeat every 5 minutes
python3 qvunex_single.py --idle snaps/*.txt --gpu-rate 2.689Tested live on a Colab T4, vLLM 0.27.1, 3 October 2026: 6 snapshots 30 s
apart, with a planned order of idle, busy, idle, idle, busy (busy = 4 chat
requests; idle = only a /health call). The report matched the plan window
for window: windows 2 and 5 served 4 requests each (40 and 62 output tokens),
windows 1, 3 and 4 were PAID, NOTHING SERVED. Of 150 s, 60 s served
($0.044817) and 90 s, 60%, paid for nothing ($0.067225), at $2.689/hour;
0 s unknown.
Check these numbers yourself in about 10 minutes, on a free Colab T4: Open the notebook in Colab. It installs vLLM, loads this file from GitHub, and runs both measurements. Run from GitHub on a different Colab machine on 28 September 2026, it gave: meter and server equal again; boot about 165 s of 171 s billed, $0.0102 per request; per-request decode 122-123 against 695-701 tokens/s for the whole server at 8 requests at once. Colab will warn that the notebook was not authored by Google; read the three code cells first, they are short.
The same two measurements on an H100. On 28 September 2026 they ran on a rented 1x H100 SXM 80GB (vast.ai, 20 vCPUs, $2.689/hour, the vLLM 0.27.1 image), from a terminal instead of Colab:
mkdir -p /content && cd /content
curl -sL -o reproduce_t4.ipynb https://raw.githubusercontent.com/qaisermehdi3-coder/qvunex/main/notebooks/reproduce_t4.ipynb
curl -sL -o qvunex_single.py https://raw.githubusercontent.com/qaisermehdi3-coder/qvunex/main/qvunex_single.py
python3 -c "import json;nb=json.load(open('reproduce_t4.ipynb'));open('run.py','w').write(chr(10).join(''.join(c['source']) for c in nb['cells'][2:]).replace('RATE = 1.72','RATE = 2.689'))"
python3 run.pyMeter and server were equal in every window. The server answered after about 75 s of 77.3 s billed, so about 97% of the wake was boot: $0.0072 per request. One at a time, per-request decode was 648-649 tokens/s and the whole server 589-593. At 8 requests at once, per-request decode was 549-573 tokens/s and the whole server 3,194-3,348. That is 5.7-5.8x, the same ratio as on the T4.
SGLang too. Start SGLang with --enable-metrics and the same commands
work on its /metrics (port 30000 by default); the report names which server
the counters came from. It reads SGLang's sglang:num_requests_total,
sglang:prompt_tokens_total and sglang:generation_tokens_total, streamed and
not streamed added together. Per-request speed is not read for SGLang yet,
only the whole-server figure.
Tested live on a Colab T4, SGLang 0.5.21, 4 October 2026. 8 chat requests
moved its counters by exactly what the client got back: 8 requests, 264 prompt
and 80 output tokens. Then, through this file's wrapper, 4 chat requests and 2
streams sent without stream_options: the meter recorded 6 calls, 2 with no
usage; the server counted 6 requests, and --reconcile showed requests +0 and
a token gap of +66 prompt and +64 output, the two streams. --idle over three
20 s windows planned idle, busy, idle marked them exactly that way.
Two things differ from vLLM. SGLang's /health runs a one-token generation on
the GPU every time it is called (its own code says so), yet the request
counters stayed still through six /health calls and through the idle windows
above. So on SGLang a health check uses the GPU and still leaves no trace in
the counts. And SGLang sends itself one warmup request at startup, which is
already counted when the server first answers (1 request, 6 prompt, 8 output
on that run): take the first snapshot after the server answers, and that
request falls before your window.
One SGLang cold start, and why "ready" is not the end of boot. Same T4, SGLang 0.5.21, 5 October 2026, clock started before the server launched:
boot (to /health 200) 386.7 s 60.5% of which capturing prefill CUDA graphs 287.7 s
serving window 192.8 s 30.1% 8 requests, meter and server equal 8 / 264 / 80
tail (health checks) 60.1 s 9.4%
billed 639.6 s
The serving window is not what it looks like. SGLang logged "ready to roll"
and /health said 200, then the first chat request took about 176 s, the
second about 16 s, and the other six finished within a second. Eight more
requests on the same server afterwards took 0.08 to 0.22 s each. So about 563
of the 640 billed seconds, 88%, went by before the first user got an answer,
though a split taken at /health would call only 60% of it boot. The log does
not say what those 176 s were spent on, so the cause is not claimed here. On
vLLM 0.27.1 on the same card, the 8 requests after ready took 5.8 s together.
The same log also shows each /health call in the tail as a one-token
prefill on the GPU, while the request counters stayed still.
Other servers are not covered yet.
Wrap the client and name the unit of work. Two lines.
from qvunex import wrap, task
client = wrap(anthropic.Anthropic()) # or OpenAI
with task("outbound email"):
...your existing code, unchanged...wrap matters more than it looks. Decorating your own functions only sees the
calls you wrote; wrapping the client also catches the calls your framework makes
on your behalf — which is where the spend hides when one request fans out into
six sub-agent calls.
To see the output before wiring anything up — no API key, no network, no spend:
qvunex demo --apiOn your own workload, give it a rate card (copy tools/prices.example.txt to
~/.qvunex/prices.txt) and run qvunex report:
--------------------------------------------------------------------
COST PER FINISHED TASK
--------------------------------------------------------------------
task n mean p50 p95 calls
outbound email 12 $0.13 $0.13 $0.16 42.1
support triage 5 $0.02 $0.02 $0.02 1.0
retries 1 call(s), $0.03 the same work paid for twice
--------------------------------------------------------------------
COST BY ENDPOINT
--------------------------------------------------------------------
endpoint calls tokens $ share per call
classify intent 480 $1.04 62.0% $0.0022
research 12 $0.27 16.2% $0.02
draft 13 $0.27 16.0% $0.02
--------------------------------------------------------------------
WHAT THIS SUGGESTS
--------------------------------------------------------------------
* 'classify intent' is 63.4% of token spend while being the cheapest
single call you make ($0.0022). It is the 480 runs, not the price.
* 'classify intent' wrote 408,000 tokens to the prompt cache and read
none back. A write costs more than not caching at all, so that is a
pure surcharge.
Four things there are hard to get any other way:
- Cost per finished task, not per request. Per-task cost cannot be reconstructed afterwards from per-call billing data — the id has to be attached at call time or the number is a guess.
- The spread. The mean and the p95 are different questions. An average hides the tasks that actually hurt.
- Cache writes counted apart from reads. A write costs more than not caching (1.25x input on the 5-minute TTL, 2x on the hour); a read costs 0.1x. Folded into one "input tokens" figure, a cold run and a warm run look the same.
- Retries and fallbacks as their own line. A retry is the same work paid for twice. A fallback is a model you didn't ask for answering, at its price. Both are real spend that logging the successful attempt drops on the floor.
Providers disagree about whether reasoning tokens sit inside the output count or on top of it, and neither raises an error if you assume the wrong one.
Measured against gemini-3.6-flash, three calls, using Google's own reported
total:
17 + 144 + 589 = 750
4,632 + 26 + 236 = 4,894
4,632 + 32 + 238 = 4,902
total = prompt + output + thinking, exactly, every time. So on Google, thinking
is billed on top of the output count. OpenAI is the reverse — reasoning_tokens
is a subset of completion_tokens, and adding it again inflates the bill.
Anthropic folds thinking into output_tokens and never reports it apart.
On that first call thinking was 589 of 750 tokens. Anything reading the output count alone would show a fifth of what you were charged.
So the convention is recorded per call, not assumed globally, and pricing acts
on what was recorded. Reproduce it yourself with a free key:
tools/gemini_check.py.
Where a provider reports its own total, that is the one number qvunex did not
derive. Every call carries it, the parts are added up and compared, and a
mismatch prints UNACCOUNTED along with a line saying every cost below it is
short by at least that much.
The gap is recorded, never corrected. We do not know which side is wrong, and quietly adjusting a number to make a check pass is how a meter starts lying.
A model with no entry in your rate card is reported as unpriced and left out rather than estimated.
wrap catches every call a task makes, including the ones your framework makes
for you. It does not tell you which sub-agent made each one.
If your harness doesn't say who is calling — and most SDKs don't put that on the wire — no amount of wrapping invents it. You get every call, correctly totalled, with the task it belongs to. You do not get a per-sub-agent breakdown unless you label the steps yourself.
So the honest scope:
- You own your loop — label the steps and attribution works end to end.
- Managed agent platform — per-session totals are the ceiling. Nobody can do better from outside, and a tool claiming otherwise is guessing.
A call whose owner cannot be determined is recorded as unattributed and shown on its own line. It is never folded into the parent task, because a total that looks complete and isn't is worse than a gap you can see.
====================================================================
QVUNEX METER REPORT
====================================================================
window 3600.0 s
calls 41,209
inferences 329,672
gpu rate $0.7500/hr x 1 device(s)
window cost $0.75
CPKI $0.0023 per 1,000 inferences
--------------------------------------------------------------------
COST BY ENDPOINT
--------------------------------------------------------------------
endpoint inf share cost CPKI
image-classifier 251,104 78.1% $0.59 $0.0023
text-embedder 62,336 18.6% $0.14 $0.0022
thumbnail-scorer 16,232 3.3% $0.02 $0.0015
--------------------------------------------------------------------
WASTE
--------------------------------------------------------------------
mean GPU utilisation 31.4%
time below 5% util 38.2%
cost of that idle time $0.29 <-- paid for, not used
duty cycle 44.1% time actually inside inference
batch efficiency 19.0% mean batch vs max seen (32)
Plus latency percentiles per endpoint, memory headroom, and a short list of things worth looking at — fired only on defensible thresholds, because a meter that cries wolf stops being read.
Two cost figures are only comparable if they were produced under the same conditions,
and those conditions almost never get published. qvunex checklist fills in what the
meter observed and prints everything else as MISSING, loudly:
04 WORKLOAD
------------------------------------------------------------------
4.1 batch_size / concurrency [observed]
batch 1-32, mean 6.2 (6 distinct sizes)
4.2 input_tokens / output_tokens
>> MISSING - qvunex cannot observe this. State it yourself.
Tell it what it cannot see, and those fields fill in too:
qvunex.configure(rate_usd_hour=0.35, context={
"engine": "vLLM 0.27.1",
"cuda_graphs": True,
"model_id": "Qwen/Qwen2.5-1.5B-Instruct",
"dtype": "fp16 weights, fp16 compute",
})Why this is in the tool rather than in a document: I published a benchmark showing 4-bit quantization was 24.8% more expensive than fp16, then re-ran it with CUDA graphs enabled and measured 48.3% cheaper. One unreported flag, 73 percentage points, same card, same afternoon. A checklist nobody fills in is not a standard.
Full field list and the evidence behind each one: https://gist.github.com/qaisermehdi3-coder/b00f296641681695daf90e5a500d0d23
Local only. Every measurement is appended to a JSONL file on your own disk. This package contains no network code — grep it. That isn't a limitation, it's the point: a read-only tool that never phones home gets adopted in an afternoon instead of surviving a six-month security review.
No dependencies. Pure standard library. GPU stats come from shelling out to
nvidia-smi, which exists wherever an NVIDIA GPU does. Nothing for anyone's
platform team to approve.
Never breaks the caller. Any failure inside the meter is swallowed. Your
function's exceptions propagate untouched, and a call that raises is still
recorded with ok=False. A measurement tool that can take down production is a
tool nobody installs twice.
Cheap. The hot path is a perf_counter pair, a dict, and an append to a
buffered list. No I/O per call.
Honest. Costs are attributed, not measured: we know total GPU-hours and each endpoint's share of busy time, and we divide. Under concurrency, summed call duration exceeds wall time — so duty cycle is capped and the overlap is flagged in the output rather than quietly normalised. Without a GPU price, cost is reported as unavailable, never estimated from a default.
Set your GPU price to get costs. Everything else has a working default.
import qvunex
qvunex.configure(
path="~/.qvunex/events.jsonl", # where the corpus lives
rate_usd_hour=0.75, # what you pay per GPU-hour
sample_interval=1.0, # device polling, seconds
)Or by environment, which is usually easier in a container:
QVUNEX_PATH=/data/qvunex.jsonl
QVUNEX_RATE_USD_HOUR=0.75
QVUNEX_DISABLED=1 # hard off, decorator becomes a passthrough
For code that isn't shaped like a function:
with meter.span("batch-job", batch=len(items)):
process(items)Batch size is inferred from the first argument (.shape[0], then len(),
then 1). Override it when that guess is wrong:
@meter("ranker", batch=lambda args, kwargs: len(kwargs["docs"]))
def rank(*, query, docs): ...qvunex report # default corpus
qvunex report /data/events.jsonl # a specific file
qvunex report --rate 0.75 # override the recorded price
qvunex report --json # raw analysis for your own tooling
qvunex checklist # comparability fields, gaps marked MISSING
qvunex demo # synthetic workload + report, no setupEvents are JSONL, one object per line, schema-versioned in schema.py. Three
record types: session, call, gpu.
Keep these files. The corpus is the asset — cross-workload measurement data can't be collected retroactively, and no cloud vendor can assemble it, because no cloud vendor is neutral enough to be allowed to measure its competitors.
v0.2. Measures what it claims to measure and nothing more. Precision headroom and input-triviality analysis are deliberately not here yet — those change your model, and this release only observes.
qvunex checklist exits non-zero when fields are missing, so you can wire it into
CI and fail a build that would publish an uncomparable number.
Apache-2.0.