inference-latency-debugger

A diagnostic toolkit for ML inference latency — isolate where time actually goes, profile the stage that's slow, then benchmark quantization, pruning, ONNX, and horizontal scaling against the same model.

Most latency write-ups post one speedup number and stop. This repo runs every technique three times where the result was inconsistent, reports every run instead of the cleanest one, and uses percentiles instead of means, because on real hardware a mean from a single run can flip sign between reruns of identical code.

Read the full write-up on EmiTechLogic → How to Debug Latency and Throughput Issues in ML Inference

What It Does

Request
   |
   v
Diagnose (4 stages)  --> Profile (torch.profiler)  --> Optimize            --> Scale
naive vs fixed            which operator is slow        quantize / prune       worker pool +
preprocessing/batch/                                    / ONNX                load balancer
postprocess
   |                                                        |                     |
   +--------------------------------------------------------+---------------------+
                                    |
                                    v
                    bench_utils.py: warmup + percentile harness
                    (every number below comes through here)

Nine scripts, one shared model, one benchmarking harness:

Script	Job
model_def.py	Shared 528K-parameter MLP used across every benchmark
bench_utils.py	Warmup + percentile timing harness (p50/p95/p99, not just mean)
diagnose_causes.py	Isolates 4 common latency causes: naive vs fixed, each timed
profiler_demo.py	torch.profiler breakdown of the forward pass, operator by operator
batch_vs_realtime.py	Latency/throughput sweep across batch sizes 1 through 256
quantization_demo.py	Dynamic INT8 quantization vs fp32, run 3x to check consistency
pruning_demo.py	40% unstructured L1 pruning vs dense, run 3x
onnx_demo.py	ONNX Runtime vs torch eager mode, run 3x
scaling_demo.py	Worker-pool horizontal scaling, includes a real process-spawn bug and its fix
test_latency_debug.py	5-test suite validating correctness of every technique above

Installation

git clone https://github.com/Emmimal/inference-latency-debugger.git
cd inference-latency-debugger
pip install -r requirements.txt

requirements.txt:

torch>=2.13
onnx>=1.22
onnxruntime>=1.24
numpy>=2.4
pytest>=9.1

No GPU required. Every benchmark in this repo runs CPU-only, including quantization, which is CPU-only in PyTorch by design.

Quick Start

python model_def.py          # sanity check: should print "params: 528138"
python diagnose_causes.py    # isolates the 4 most common latency causes
python profiler_demo.py      # torch.profiler breakdown of the forward pass

Each script is self-contained and writes its own results to a .json file in the working directory. Run them in any order after the sanity check passes.

Running the Benchmarks

Order	Script	What It Shows
1	diagnose_causes.py	Python-loop preprocessing vs vectorized, unbatched vs batched forward, .item() sync vs vectorized postprocessing
2	profiler_demo.py	Which operator inside the forward pass is actually expensive (aten::addmm, in this model)
3	batch_vs_realtime.py	Latency/throughput tradeoff across 9 batch sizes, including where throughput stops climbing
4	quantization_demo.py	Dynamic INT8 vs fp32, 3 runs, reports the run-to-run sign flip rather than one number
5	pruning_demo.py	40% unstructured pruning vs dense, 3 runs, shows why unstructured pruning doesn't speed up dense kernels
6	onnx_demo.py	ONNX Runtime vs torch eager, 3 runs, includes the dynamo=False fallback needed on PyTorch 2.13 without onnxscript
7	scaling_demo.py	Worker-pool scaling, before and after fixing a process-spawn bug that was measuring startup cost instead of inference
—	test_latency_debug.py	pytest test_latency_debug.py -v — validates correctness of every technique above

Configuration Reference

Key constants, if you want to point these scripts at your own model:

SEED = 42                          # every script
BATCH_SIZES_SWEPT = [1, 2, 4, 8, 16, 32, 64, 128, 256]   # batch_vs_realtime.py
PRUNING_SPARSITY = 0.4             # pruning_demo.py, global unstructured L1
QUANTIZATION_DTYPE = "qint8"       # quantization_demo.py, dynamic quantization
ONNX_OPSET = 17                    # onnx_demo.py
SCALING_TOTAL_REQUESTS = 40000     # scaling_demo.py, post-fix version
SCALING_WORKER_COUNTS = [1, 2, 4, 8]

Swap model_def.build_model() for your own model and every script downstream should work unchanged, since they all import from model_def and bench_utils.

Project Structure

inference-latency-debugger/
├── model_def.py
├── bench_utils.py
├── diagnose_causes.py
├── profiler_demo.py
├── batch_vs_realtime.py
├── quantization_demo.py
├── pruning_demo.py
├── onnx_demo.py
├── scaling_demo.py
├── test_latency_debug.py
├── requirements.txt
└── README.md

Results Summary (4-core Windows, Python 3.12.10, PyTorch 2.13.0)

Technique	Result	Consistent across reruns?
Batching (1 row vs 64-row calls)	8.4x faster	Yes
ONNX Runtime vs torch eager	1.18x–1.30x faster	Yes
Dynamic INT8 quantization	18% slower to 29% faster, ~wash by p50	No
40% unstructured pruning	0.85x–1.01x (no speedup)	Yes (consistently no gain)
Horizontal scaling (1→8 workers, post-fix)	Roughly flat	Yes (flat, on shared hardware)

Full numbers, including all three runs for quantization, pruning, and ONNX, and the before/after scaling fix, are in the article linked above.

When to Use This

Worth running against your own model if you:

• Have a latency complaint and haven't yet isolated which pipeline stage is slow
• Want to know whether quantization or ONNX will actually help your hardware before committing engineering time to either
• Need to sanity-check a horizontal-scaling plan before provisioning more workers than your infrastructure can actually parallelize

Skip it if you already know exactly which operator or stage is the bottleneck. Go straight to fixing that stage instead of re-running this diagnostic sequence.

Known Limitations

• All benchmarks run against one synthetic 528K-parameter MLP. Ratios here (8.4x for batching, etc.) are properties of this model's size and shape, not universal constants — a transformer will show different proportions.
• Quantization and scaling results are hardware- and even machine-load-dependent. The quantization sign flip and the flat scaling curve in this repo were measured on a shared development laptop, not a dedicated benchmarking box.
• No GPU benchmarks. Dynamic quantization specifically is CPU-only in PyTorch; none of these numbers apply to GPU-served models.
• onnx_demo.py requires dynamo=False on PyTorch 2.13+ unless you also install onnxscript, which isn't a repo dependency by default.
• Structured pruning (which does speed up dense inference) isn't implemented here — only unstructured pruning, which this repo shows does not help.

License

MIT