[lslusarczyk]

XPU graphs now serve a dynamic batch, record Inductor kernels, and win only the batch sizes where they are faster.

Grid of batch sizes

The XPU graph backend records graphs for batch sizes 1, 2, 4, and every multiple of 8, up to the largest batch size seen during tuning.
max_recorded_batch_size caps that size at 256, which is 35 graphs.
vLLM and TensorRT-LLM use the same grid.

A call pads its batch with zeros to the smallest recorded size that holds it.
A larger batch replays in chunks.
All graphs share one memory pool and one input buffer.
Tuning runs eager once on the largest batch and compares every graph with prefixes of it, full and padded.

Inductor kernels

XPUGraphBackendConfig(compile_with_inductor=True) records the kernels Torch Inductor generates.
Replay runs the fused kernels without a Python launch per kernel.
It is the XPU counterpart of torch.compile(mode="reduce-overhead") on CUDA.
Inductor compiles three times for all recorded sizes.

Routing by batch size

HighestThroughputStrategy kept one backend for every batch size, chosen by peak throughput.
An XPU graph lost the small batch sizes it was faster at.
Another backend now takes a range of batch sizes when it is at least 10% faster there.
BatchSizeRouterBackend holds the routed backends, sends each call by its batch size, and saves them in a checkpoint.

Profiling fix

Saturation counted the first call of a new batch size, which can compile.
Inductor looked saturated and stopped profiling at batch size 8.
The stop strategy now judges the measured window, like the reported throughput.

Performance

Latency in ms on Arc Pro B70, fp32, torch 2.14.1+xpu, by batch size.
Auto is HighestThroughputStrategy over all four backends.

Model	Backend	1	3	8	17	64
MLP 24x256	eager	0.85	0.84	0.83	0.82	0.83
	Inductor	0.56	0.61	0.59	0.61	0.59
	XPU graph	0.15	0.17	0.17	0.17	0.18
	XPU graph + Inductor	0.12	0.13	0.13	0.14	0.13
	auto	0.14	0.14	0.13	0.13	0.13
MobileNetV3-S	eager	2.09	2.11	2.12	2.21	6.24
	Inductor	1.42	1.47	1.47	1.48	2.88
	XPU graph	0.45	0.62	0.80	1.94	6.06
	XPU graph + Inductor	0.28	0.38	0.50	1.09	3.37
	auto	0.28	0.37	0.50	1.09	3.48
ResNet-18	eager	1.11	1.67	3.45	10.83	28.79
	Inductor	0.83	1.34	2.66	7.84	20.77
	XPU graph	0.78	1.79	3.36	10.99	28.96
	XPU graph + Inductor	0.63	1.49	2.68	7.53	21.04
	auto	0.62	1.52	2.66	6.76	18.21
ResNet-50	eager	2.18	3.49	8.22	23.97	75.38
	Inductor	1.49	2.64	5.89	15.61	46.17
	XPU graph	1.51	3.79	8.00	25.06	75.00
	XPU graph + Inductor	1.18	3.04	5.91	16.06	46.26
	auto	1.19	2.66	5.91	15.64	46.18
ViT-B/16	eager	3.74	7.83	19.77	41.71	156.60
	Inductor	3.50	7.41	18.97	40.68	152.07
	XPU graph	3.60	10.98	20.04	58.50	155.54
	XPU graph + Inductor	3.39	10.49	19.01	57.22	152.48
	auto	3.52	7.46	19.11	41.09	152.28

XPU graph + Inductor is the fastest single backend for launch-bound models.
It ties Inductor on large batches, except where padding adds rows: ViT batch 17 runs the graph for 24.
Auto lands on the fastest backend at each batch size.
On ResNet-18 it routes 1 to 4 to XPU graph + Inductor and 5 and up to Inductor.