Calibrated simulation benchmark for real-time complexity-based routing in LLM serving.
Given an incoming request, the router decides whether to send it to a small model or a large model before executing inference.
Which routing signal gives the best trade-off between cost savings, quality preservation, and hard-request protection?
Three prior benchmarks studied related phenomena in isolation:
- inference-time-scaling-bench: harder tasks need more compute
- draft-model-selection-bench: real cost difference between small and large models
- serving-cost-model-v2: decode dominates total serving cost, not model size alone
This benchmark connects them into a deployable decision problem:
Given a request arriving now, which model should serve it?
The key insight is that decode dominates cost at 82-91% of total serving cost. This means routing savings are largest not just for easy requests, but for easy requests with long expected outputs.
It combines difficulty signal with output-length awareness:
score = perplexity_signal - 0.30 * output_length_signal
Route to small model if score is below threshold.
At exactly the same number of offloaded requests:
| Workload | Smart score | Random mean | Advantage |
|---|---|---|---|
| long_easy | 52.39 | 34.15 | +18.24 |
| easy_heavy | 43.23 | 29.58 | +13.66 |
| balanced_mix | 28.22 | 16.12 | +12.11 |
| ambiguous_middle | 26.98 | 15.25 | +11.73 |
| short_hard | 18.05 | 7.36 | +10.69 |
| hard_heavy | 13.25 | 3.16 | +10.09 |
This confirms that selection quality matters as much as offload volume. Routing the right requests is as important as routing enough of them.
| Workload | Router | Savings | Quality drop | Risk |
|---|---|---|---|---|
| long_easy | perplexity_based_router | 56.3% | 2.8% | safe |
| easy_heavy | confidence_based_router | 46.6% | 3.0% | safe |
| balanced_mix | perplexity_length_router | 32.3% | 2.6% | safe |
| short_hard | perplexity_length_router | 27.0% | 2.9% | moderate |
| hard_heavy | perplexity_length_router | 23.8% | 2.9% | moderate |
| ambiguous_middle | perplexity_based_router | 21.4% | 2.1% | safe |
| Workload | Router | Savings | Quality drop |
|---|---|---|---|
| long_easy | perplexity_length_router | 72.0% | 4.85% |
| easy_heavy | confidence_based_router | 58.4% | 4.57% |
| balanced_mix | confidence_based_router | 42.3% | 4.34% |
| short_hard | perplexity_length_router | 36.9% | 4.83% |
| hard_heavy | perplexity_based_router | 31.6% | 4.79% |
| ambiguous_middle | perplexity_based_router | 40.3% | 4.76% |
In 4 of 6 workloads, it sends 100% of hard requests to the small model. It is never the best or second-best deployable policy.
At matched offload count, smart routing achieves 2-5pp lower quality drop than random selection. It routes the right requests, not just more requests.
| Policy | Kind | Signal |
|---|---|---|
| always_large | baseline | no signal |
| always_small | baseline | no signal |
| random_router | control | matched offload fraction |
| length_based_router | deployable | prompt length |
| perplexity_based_router | deployable | draft model perplexity |
| perplexity_length_router | deployable | perplexity + output length |
| confidence_based_router | deployable | draft model confidence |
| oracle_router | upper bound | true difficulty score |
| Workload | Description |
|---|---|
| easy_heavy | 60% easy requests |
| balanced_mix | 35/35/30 easy/medium/hard split |
| hard_heavy | 60% hard requests |
| short_hard | all prompts short, 50% hard tasks |
| long_easy | 70% easy, all prompts and outputs long |
| ambiguous_middle | 60% medium difficulty, high signal noise |
cd ~/dev/model-routing-complexity-bench
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
python -u run.py
Runtime: approximately 5 seconds. No GPU required.
results/
summary.csv
best_configs.csv
best_deployable.csv
decision_map.csv
quality_budget_best.csv
deployment_recommendations.csv
random_control.csv
plots/
01_cost_vs_quality_*.png
02_misroutes_*.png
03_output_vs_savings_*.png
04_frontier_*_*.png
05_exact_random_control_*.png
06_deployment_recommendations_*.png
model-routing-complexity-bench/
+-- src/
| +-- config.py
| +-- workload.py
| +-- models.py
| +-- router.py
| +-- simulator.py
| +-- bench.py
| +-- analysis.py
+-- results/
+-- plots/
+-- run.py
+-- README.md
+-- summary.txt
+-- design.md
+-- LICENSE
+-- requirements.txt
This project is a calibrated simulation, not an end-to-end serving benchmark.
Each request has a latent difficulty score, observable routing signals, and model-specific cost and quality estimates based on calibrated parameters.
The random router control uses exactly k matched offloads repeated 30 times, providing a clean baseline to isolate the value of signal quality from offload volume.
For full design details, see design.md.
- Python 3.10+
- NumPy >= 1.26.0
- Pandas >= 2.0.0
- Matplotlib >= 3.8.0
No GPU required.
This project does not run real LLM inference.
Main limitations:
- synthetic request distributions
- proxy quality model rather than measured task accuracy
- estimated router overhead rather than measured latency
- single model pair evaluated (1.5B vs 7B)
- no batching or queue effects modeled
These are acceptable because the benchmark targets relative comparison of routing signal quality, not absolute production forecasting.
See design.md for a full discussion.
If you need a simple default routing policy:
- use perplexity_length_router as the primary router
- use perplexity_based_router as a simpler fallback
- avoid length_based_router in any workload with short hard requests
- set threshold to match your quality budget using the deployment recommendations table
- monitor hard misroute rate in production as the primary safety signal
- design.md -- detailed design rationale and modeling assumptions
- summary.txt -- concise high-level summary of findings
- LICENSE -- MIT License
MIT License -- Copyright (c) 2026 Joao Felipe De Souza
Joao Felipe De Souza
2026