An analytical benchmark for speculative decoding at batch sizes greater than 1. Measures how acceptance rate, optimal gamma, and speedup change with batch size, draft quality heterogeneity, and serving mode.
For full methodology and design decisions see design.md.
- Does speculative decoding maintain its gains beyond batch_size=1?
- What is the optimal speculation depth for different batch configurations?
- Where does speculative decoding become slower than greedy?
7 batch sizes x 7 gammas x 4 quality modes x 2 serving modes
Batch sizes: 1, 2, 4, 8, 16, 32, 64 Gammas: 1, 2, 4, 6, 8, 12, 16 Quality modes: homogeneous_good, homogeneous_mid, heterogeneous, bimodal Serving modes: static_batch, continuous_refill
homogeneous_good (q~0.85): γ=2: 1.80x γ=4: 2.03x γ=6: 2.09x γ=8: 1.92x γ=12: 1.69x γ=16: 1.45x
homogeneous_mid (q0.65):
γ=2: 1.47x γ=4: 1.40x γ=6: 1.24x
γ=8: 1.08x γ=12: 0.85x γ=16: 0.69x
CROSSOVER at gamma10
homogeneous_good: gamma=4-6 peak=2.09x homogeneous_mid: gamma=2 peak=1.50x heterogeneous: gamma=2 peak=1.47x bimodal: gamma=2 peak=1.49x
homogeneous_good, gamma=4: bs=1: 2.03x bs=4: 2.00x bs=16: 2.04x bs=64: 2.09x
bimodal, gamma=2: bs=1: 1.37x bs=4: 1.46x bs=16: 1.48x bs=64: 1.49x
Speculative decoding scales to batch>1 with no degradation. Peak speedup is 2.09x at both batch_size=1 and batch_size=64. Continuous refill slightly improves speedup at larger batches.
Gamma is the critical tuning knob, not batch size.
Optimal gamma depends on draft quality, not on batch size.
Good draft (q0.85): gamma=4-6. Medium draft (q0.65): gamma=2.
Over-speculating hurts moderate-quality drafts. At quality=0.65 and gamma=12: 15% SLOWER than greedy. At gamma=16: 31% SLOWER than greedy. Crossover point: approximately gamma=10 for q~0.65.
Batch heterogeneity caps speedup at approximately 1.5x. Bimodal and heterogeneous quality distributions are bounded regardless of gamma or batch size. The weakest-accepting request limits the batch.
Gamma=2 is the safe default for unknown draft quality. Provides 1.37-1.50x speedup across all quality modes without regression risk. Gamma=4 is optimal only when draft quality is confirmed to be high.
batched-speculative-decoding-bench/ ├── src/ │ ├── init.py │ ├── config.py │ ├── workload.py │ ├── acceptance.py │ ├── baseline.py │ ├── speculative.py │ ├── metrics.py │ └── analysis.py ├── results/ ├── plots/ ├── LICENSE ├── design.md ├── README.md ├── requirements.txt └── run.py
python3 -m venv venv source venv/bin/activate pip install -r requirements.txt python run.py
Outputs: results/results.csv results/summary.txt plots/speedup_vs_batch.png plots/utilization_vs_batch.png plots/divergence_penalty.png plots/gamma_batch_heatmap.png
Closes the speculative decoding series:
speculative-decoding-impl: linear draft+verify at batch=1 tree-speculative-decoding: tree draft verification at batch=1 batched-speculative-decoding-bench: batch>1, gamma tuning, crossover analysis
MIT License. See LICENSE for details.
Joao Felipe De Souza 2026