A local-first, self-learning AI load balancer for heterogeneous Windows laptops.
SiliconRoute benchmarks ONNX models on every compute chip in a Windows laptop (CPU, integrated GPU, discrete GPU, and NPU), stores every measurement in SQLite, fits hardware performance models (command overhead, cache and DRAM bandwidth, compute throughput), and routes each inference task to the best chip for the user's goal. The Fastest goal is evaluated below across three workload types (sustained, idle, cold start); the Battery, Balanced and Cool goals are implemented but not yet evaluated.
Important
While baseline evaluation was conducted on hybrid x86 hardware, SiliconRoute's primary development focus is ARM-based Copilot+ systems. Active hardware benchmarking and profiling is currently underway on the Qualcomm Snapdragon X platform (including Snapdragon X Elite and upcoming Snapdragon X2 systems), evaluating heterogeneous execution across Qualcomm Oryon ARM CPU cores, DirectML GPU acceleration, and dedicated Qualcomm Hexagon NPU offload via ONNX Runtime's QNNExecutionProvider.
Modern laptops are heterogeneous systems with multiple compute devices: multi-core CPUs with large L3 caches, integrated GPUs (iGPUs) sharing system memory, and high-power discrete GPUs (dGPUs) with dedicated VRAM.
Standard AI deployment frameworks make static assumptions:
- "Always run on the discrete GPU" — slow for small models and cold starts. Even with the GPU warm, a tiny model (
mlp-256w-4l, batch 1) takes 0.142 ms per call on the RTX 5070 but only 0.017 ms on the Ryzen CPU, so the CPU is 8.4x faster. After an idle period or on a cold start, the GPU's disadvantage grows (see Method). - "Always run on the CPU" — misses large speedups on big sustained workloads: on the largest MLP tested (
mlp-3072w-4l, batch 1) the RTX is 6.4x faster than the CPU. - "Software controls power" — software cannot control how power is delivered inside a laptop; hardware and firmware do. What software can control is which chip does the work. SiliconRoute's energy- and heat-aware goals use that lever, but their effect has not been measured yet.
SiliconRoute operates in four phases:
-
Empirical Measurement: Times real ONNX inference workloads on each hardware provider (
CPUExecutionProvider,DmlExecutionProviderper DirectML adapter). Collects live system telemetry (NVML dGPU power, clocks, P-states, temperatures; WMI battery discharge rates; psutil memory and CPU utilization). Stores raw sample timings, bootstrap 95% confidence intervals, and inter-session stability.- Measurement Conditions: All benchmarks were conducted with the laptop plugged into AC power under the high-performance Windows 'Turbo' power scheme (preventing clock throttling), with NVIDIA driver 616.56 and ONNX Runtime 1.24.4.
-
Inference Timing Scope: Timings are end-to-end execution times including host-to-device and device-to-host CPU-GPU tensor copies (default ORT
RunOptions), reflecting true application response times rather than isolated compute kernel duration.
-
Physical Hardware Modeling: Fits multiple analytical scaling equations (
$F_1$ Classic Roofline,$F_2$ Two-Level Cache Roofline,$F_3$ Log-Linear,$F_4$ Family-Specific Roofline) using Non-Negative Least Squares (NNLS) with$1/t$ relative weighting. Selects the optimal model per device via Leave-One-Out (LOO) Mean Absolute Percentage Error (MAPE). -
Multi-Objective Routing:
- Uses empirical measurements first for previously profiled configurations (
WorkloadMeasurementrecords). - Falls back to measured family slowdown ratios or hardware model predictions for novel models or batch sizes.
- Applies live hardware context rules:
- Low Battery: When unplugged and battery is under 30% , switches to the Battery goal (implemented; not yet tested on battery).
-
Empirical Wake & Cold-Start Slowdowns: Wake penalties and cold-start slowdowns are measured empirically across workloads (vs latest sustained median):
-
idle_loaded: CPU 1.76x [0.71x –14.42x , n=20 ], AMD Radeon 610M 1.62x [0.98x –11.96x , n=20 ], NVIDIA RTX 5070 2.14x [0.51x –12.22x , n=20 ] -
cold_start: CPU 41.7x [0.93x –357.94x , n=28 ], AMD Radeon 610M 27.9x [1.40x –447.51x , n=28 ], NVIDIA RTX 5070 230.5x [2.40x –580.13x , n=28 ]
-
- Cold-Start Rule: For cold starts (session creation + first inference), defaults to CPU unless an accelerator is predicted >30% lower latency, because in our measurements accelerators were rarely faster on cold starts.
- Session Volatility Bands: when the top two chips score within 15% of each other, prefers the chip with lower measured session-to-session variability.
- Uses empirical measurements first for previously profiled configurations (
- Verification & Self-Audit: Verifies decisions on physical hardware, computing exact regret vs optimal.
All numbers sourced from results/final/manifest.json and generated from data/final/siliconroute_final.db.
For subsequent x86 benchmark updates and unseen model evaluations, see results/v1_1/x86/.
Evaluated across 8 sustained, 8 idle-loaded, and 8 cold-start tasks on real laptop hardware (AMD Ryzen 9 8940HX, AMD Radeon 610M, NVIDIA GeForce RTX 5070 Laptop GPU):
| Strategy | Decisions Won | Total Decisions | Accuracy (%) | Mean Regret (%) | P90 Regret (%) |
|---|---|---|---|---|---|
| SiliconRoute | 22 | 24 | 91.7% | 1.75% | 0.00% |
| Always-CPU | 15 | 24 | 62.5% | 113.06% | 420.56% |
| Always-RTX | 9 | 24 | 37.5% | 304.93% | 784.34% |
| Fit-Only Router | 19 | 24 | 79.2% | 31.42% | 12.50% |
- SiliconRoute achieved 91.7% decision accuracy, reducing average slowdown vs optimal to 1.75% .
- Always-RTX suffered 304.93% mean regret, because the RTX loses badly on small models and cold starts.
- Always-CPU suffered 113.06% mean regret, because it misses the RTX's large speedups on big sustained workloads.
- The p90 regret of 0.00% reflects that 22 of 24 choices had zero regret; both misses were cold starts of the largest model, where all chips were close.
Detailed Workload Breakdown (Sustained, Idle-Loaded, Cold-Start)
- Sustained (Warm Runs): SiliconRoute 8 / 8 (100.0% , 0.00% regret), Always-CPU 4 / 8 (50.0% , 195.52% regret), Always-RTX 4 / 8 (50.0% , 177.37% regret), Fit-Only 8 / 8 (100.0% , 0.00% regret).
- Idle-Loaded (Warm Session, 10s Idle Gap): SiliconRoute 8 / 8 (100.0% , 0.00% regret), Always-CPU 3 / 8 (37.5% , 143.65% regret), Always-RTX 5 / 8 (62.5% , 215.22% regret), Fit-Only 7 / 8 (87.5% , 1.86% regret).
- Cold-Start (Fresh Session + 1st Inference): SiliconRoute 6 / 8 (75.0% , 5.25% regret), Always-CPU 8 / 8 (100.0% , 0.00% regret), Always-RTX 0 / 8 (0.0% , 522.22% regret), Fit-Only 4 / 8 (50.0% , 92.41% regret).
With the Cold-Start Rule active (choose CPU unless another chip is predicted >30% lower latency):
- SiliconRoute: 6 / 8 wins (75.0% accuracy), 3.00% mean regret, 11.38% p90 regret.
- Always-CPU: 6 / 8 wins (75.0% accuracy), 3.00% mean regret.
- Always-RTX: 0 / 8 wins (0.0% accuracy), 575.15% mean regret.
- Fit-Only Router: 4 / 8 wins (50.0% accuracy), 109.26% mean regret.
- With the rule active, SiliconRoute chose the CPU in all 8 decisions, so it matched the always-CPU baseline exactly. Both misses were cold starts of the largest model, where the Radeon 610M was narrowly fastest.
- CPU Beats RTX on Small Tasks: For lightweight models (e.g.
mlp-256w-4lat batch 1), in session 225, the CPU executes in 0.017 ms (run 839), while the RTX 5070 takes 0.142 ms (run 841) (hypothesis: per-call DirectML dispatch and host-device transfer overhead dominates for tiny models). The CPU is 8.4x faster. - GPU Identity Swaps on Hybrid Laptops: DirectML adapter numbering was not stable on this laptop: in 12 runs, adapter indices 0 and 1 were swapped (the "Radeon" ran at RTX speed and vice versa). Hypothesis: the enumeration order depends on the discrete GPU's power state. SiliconRoute now identifies GPUs in-process via DXGI (PCI vendor IDs
0x10DE/0x1002and LUIDs), confirmed by a 300 ms NVML load fingerprint; the swapped runs are flagged and excluded. - Batch-8 DirectML Anomaly: On DirectML GPUs, latency does not scale smoothly with batch size around batch 8 (hypothesis: the driver switches kernels or algorithms). In session 87,
conv-96c-4ltakes 13.5 ms at B=8 (run 622) vs 2.4 ms at B=4 (run 621) on RTX 5070 (a 5.6x slowdown). SiliconRoute captures this via empirical lookup rather than relying purely on linear roofline fits. - Loading Time Dominates Cold Starts: In session 194, sustained RTX inference on
mlp-3072w-4ltakes 0.655 ms (run 814) vs CPU 4.20 ms (run 812). But under cold start, in Decision 144, the RTX cold start (session creation + first inference) takes 219.9 ms (WorkloadMeasurement 132, a 335.7x overhead vs sustained), while CPU completes in 215.8 ms (WorkloadMeasurement 130); the Radeon 610M was fastest in that decision, so the cold-start winner varies. Similarly, in headline Decision 124, CPU cold start completes in 189.8 ms vs RTX 202.8 ms (a 309.6x overhead).
- FP32 Only: All synthetic and profiled models currently use float32 precision. FP16 and INT8 tensor core acceleration are not yet profiled.
- DirectML Only on Windows: GPU execution uses
DmlExecutionProvideracross both AMD and NVIDIA GPUs; native CUDA or TensorRT execution providers are not used. - Synthetic Model Families: Benchmarks utilize parameterized MLP and Convolution architectures; real-world Transformer and LLM attention topologies are planned.
- Profiled Configurations: The evaluation used only previously profiled configurations; for new models or batch sizes the router relies on the fitted models, whose average LOO MAPE is ~22.3% .
- Session Variability: The same configuration can differ between sessions by up to 139.9% (RTX); the cause has not been established.
- Energy Telemetry: Whole-system energy is available via WMI battery discharge rate when unplugged, and GPU package energy is available via NVML on NVIDIA. CPU package power (RAPL) and AMD iGPU power are not accessible via standard unprivileged Windows APIs.
- Energy Goals Not Evaluated: Only one GPU energy measurement exists, so the Battery and Balanced goals currently fall back to latency, and the Cool goal (GPU temperature penalty) is untested. CPU and Radeon 610M energy require battery-drain measurements (planned).
- Single Laptop: All results come from one laptop (AMD Ryzen 9 8940HX, AMD Radeon 610M, NVIDIA GeForce RTX 5070 Laptop GPU).
- Real AI Models: MobileNet, ResNet and a small transformer are planned for v1.1.
- ARM Architecture & Snapdragon X Testing: Active hardware benchmarking and profiling is currently underway on the Qualcomm Snapdragon X platform (including Snapdragon X Elite and upcoming Snapdragon X2 systems). Heterogeneous load balancing across ARM CPU cores, DirectML GPU, and the Qualcomm Hexagon NPU via QNN Execution Provider is our primary development focus.
- Author: Tanmay — defined the problem, the measurement method and the project rules, and reviewed every development phase.
- AI Assistance: Most code was written with Google Antigravity (an AI coding agent). Planning, specifications and independent audits of the database were done with Anthropic's Claude.
- Integrity Protocol: Early in development, an AI-generated report contained fabricated benchmark numbers; a physics sanity check caught it. Since then, every published number is computed from the frozen database by
scripts/metrics.pyand checked bytests/test_published_numbers.py, agent reports are printed from logs only, and the database has been audited independently (raw samples vs. stored statistics, recomputed results, provenance of every headline number). Thresholds such as the cold-start rule are configuration settings, not measurements.
Double-click start.bat or run:
start.batThis automatically initializes the Python 3.12 virtual environment, installs dependencies, launches the FastAPI server at http://127.0.0.1:8000, and opens the dark-mode dashboard in your browser.
Run the reproduction script (about a minute on the reference laptop):
python scripts/reproduce.pyThis benchmarks CPU vs RTX on small and large MLPs at batch 1 and 32, measures sustained vs cold-start latency, and prints reproduced numbers side-by-side with published values with percentage differences.
python -m pytest -q- Frozen Database: data/final/siliconroute_final.db
- Size: 1,044,480 bytes
- SHA256:
98542cffb7264eee537d218e1863ccbbc174495e0bf13eb244ee22cd6f034cf5 - Published Results & Manifest: Located in results/final/.
SiliconRoute's primary long-term strategic objective is heterogeneous AI load balancing on ARM-based Windows laptops (Copilot+ PCs), with active hardware benchmarking currently underway on Qualcomm Snapdragon X series silicon:
- Active Snapdragon Hardware Testing: We are actively profiling and evaluating execution behavior on Qualcomm Snapdragon hardware (including Snapdragon X Elite and upcoming Snapdragon X2 systems). Testing covers Qualcomm Oryon ARM CPU cores, DirectML GPU acceleration, and dedicated Hexagon NPU offload via ONNX Runtime's
QNNExecutionProvider. - Primary Goal — Tri-Silicon Load Balancing: On modern ARM laptops, the dedicated NPU provides massive energy efficiency for neural inference, but can incur model loading and compilation overhead. SiliconRoute is expanding its physical roofline models to dynamically predict the optimal execution path across Oryon CPU cores, DirectML GPU, and Hexagon NPU under dynamic thermal and battery constraints.
- Battery-First Real-World Workloads: ARM laptops prioritize battery endurance. Intelligent routing prevents power-hungry GPU wake-ups when the NPU or CPU can complete the inference with minimal energy consumption.