perf(kernels): budget cublasLt tuning repeats by weight traffic

[FeathBow]

Description

gemm_lt_tune_cuda times every heuristic candidate for a decode GEMM shape and caches the winner. It spent a flat 20 timed repeats per candidate regardless of shape.

Timing noise is roughly a fixed cost per launch, so the repeats a candidate needs for a stable ranking fall as the GEMM streams more weight. A flat count overpays on the large projections and buys nothing extra on the small ones. This budgets the repeats by weight traffic instead:

clamp(1 GiB / (M * K * 2), 5, 20)

The upper bound is the count it replaces, so no shape is tuned more than before. As a pure function of BF16 weight bytes the rule describes its own blast radius: a projection keeps 20 repeats below about 52.5 MiB, falls through the range above that, and sits at the floor of five past about 186.2 MiB.

Reported by the build itself:

checkpoint	shape	M×K	weight	repeats
Qwen3-4B	lm_head	151936×2560	741.9 MiB	5
Qwen3-4B	gate_up, down, q, o, kv	—	≤ 47.5 MiB	20
Qwen3.5-4B	lm_head	248077×2560	1211.3 MiB	5
Qwen3.5-4B	gate_up	18432×2560	90.0 MiB	11
Qwen3.5-4B	down, and the five attention/linear shapes	—	≤ 45.0 MiB	20

Qwen3.5's ten tuning calls resolve to eight distinct (M, K); the tuner caches on (device, M, N, K), so calls sharing a shape share one tuned result. Qwen3-4B's lm_head M is vocab_size, Qwen3.5's is selection_vocab. The DFlash draft path keeps its attention shape at 20 and crosses the threshold in its context projection only at five or more target layers.

gemm_lt_tune_cuda now rejects a non-positive M, N, or K at its entry, so the helper works on validated dimensions.

What motivated this: on Qwen3-4B, warm startup spent 1594 ms of 3073 ms in this tuner, against 908 ms in the weight load that two earlier PRs optimised.

Test Env

Single GPU (sm_89, x86_64), CUDA 12.9, Qwen3-4B and Qwen3.5-4B, page cache warm. Both A/B binaries are built from this commit and its parent on 35f990d.

Covered: the two 4B checkpoints on sm_89 — every shape whose repeat count changes there, plus the golden gate for both model lines.

Not covered: Qwen3-8B and Qwen3.5-9B/27B.

Verification

A/B — five order-rotated interleaved pairs.

pair	flat 20	byte budget	delta
1	3192 ms	2438 ms	754 ms (23.6%)
2	3212 ms	2525 ms	687 ms (21.4%)
3	3207 ms	2514 ms	693 ms (21.6%)
4	3180 ms	2431 ms	749 ms (23.6%)
5	3276 ms	2403 ms	873 ms (26.6%)

Paired median 749 ms (23.6%) off warm HTTP-ready, 5/5 pairs. The weight-load phase is unchanged — 872–906 ms flat against 809–926 ms budgeted, overlapping ranges with means 9 ms apart — so the saving sits in tuning.

A second five-pair session on the same parent measured a paired median of 800 ms for a behaviourally identical revision. Pooling both gives ten pairs spanning 687–873 ms at a median of 774 ms, which is this harness's resolution.