kv-cache : spend the partial residency budget on the slowest link first

[Piggidragon]

Overview

Depends on #67. That PR's commit is included in this branch because the base repository has
no branch to target. Only the last commit, kv-cache : spend the partial residency budget on the slowest link first, belongs to this PR. Merge #67 first and this diff reduces to that commit.

Builds on the residency PR. There, --kv-gpu-layers spreads round-robin over the owning devices, so
each device gives up the same number of layers. That is the right default only when the devices are
alike.

A device-resident layer saves the host-to-device transfer it would otherwise cost every token, and
that transfer costs most where the link is slowest. On an asymmetric pair the two links differ by a
lot:

llama_pick_gpu_resident_layers: CUDA0: host-to-device 21.7 GB/s   (RTX 4070, gen4 x16)
llama_pick_gpu_resident_layers: CUDA1: host-to-device  3.3 GB/s   (RTX 3060, gen3 x4)

A factor of 6.6, so a layer kept off the 3060 is worth 6.6 kept off the 4070.

Measure the host-to-device bandwidth once at startup, through the same path the cache is delivered
on, and spend the budget on the slowest device first. Devices within 15% of each other stay one group
and keep the round-robin, so nothing changes for a symmetric machine.

Testing

Qwen3.8-27B-UD-Q5_K_M, 15216-token prompt, -c 20480 -n 128 -ngl 99, f16 cache,
-sm layer -nkvo --kv-cpu-pinned --recurrent-state-offload. The model has 16 owned attention layers,
8 per GPU. "round-robin" is the parent PR, "slowest first" is this one.

--kv-gpu-layers	pp round-robin	pp slowest first	tg round-robin	tg slowest first
0	611.67	611.23	4.34	4.34
2	620.45	627.17	4.79	5.20
4	629.41	644.61	5.35	6.50
8	649.38	979.28	6.97	12.97
16 (all)	979.95	977.19	17.77	17.77

At 8 layers, the point where the 3060 owns none of the host-resident cache, the two orders cost the
same memory and differ by +86% generation and +51% prefill. Both converge at 0 and at 16,
where every device gives up every layer, which is the expected sanity check.

test-llama-archs -s 1 at 1, 2, 3 and 4 virtual CUDA devices: passes. Built with
-DLLAMA_FATAL_WARNINGS=ON.

Not done here

Deriving --attn-split or --tensor-split from the same measurement. The measurement is there and
the rounding would land on a useful value for this pair, but it changes the reduction order for every
multi-GPU user, so it stays out until it has run on more than one machine.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES - implemented by an agent on my instruction, see the Assisted-by: commit trailer.

[GenerelSchwerz]

Automated preliminary review by Codex; the repository owner plans a separate manual review.

Verdict: FAIL.

Blocking

• The current live-base synthetic merge does not compile at src/llama-model.cpp:2532: specialized_placement is undeclared. Compiler output from Linux x64, Linux ARM64, Windows ARM64, and WASM all reaches this diagnostic.
• llama_dev_h2d_bandwidth() always allocates its source with ggml_backend_dev_host_buffer_type() (src/llama-model.cpp:2245), while the real KV cache uses pageable CPU storage unless kv_cpu_pinned is enabled (src/llama-kv-cache.cpp:21; the default is false at src/llama-context.cpp:4077). The ranking therefore measures a different transfer path for the default configuration.
• The slow-link-first loop at src/llama-model.cpp:2338 spends every eligible layer on the slowest group before moving on, but considers neither device free memory nor each layer's actual KV size/layout, including SWA versus full-cache and MLA K-only cases. Allocation is attempted only later and fails outright at src/llama-kv-cache.cpp:382; there is no retry or redistribution, so this policy can turn an otherwise working configuration into a startup allocation failure.

Will slow review

• The automatic multi-device placement policy at src/llama-model.cpp:2316 has no linked issue or design agreement. The added test at tests/test-llama-archs.cpp:1047 only proves one MTP layer leaves host storage; it does not cover asymmetric link ranking, default pageable storage, memory pressure, SWA/MLA, or fallback behavior. This PR is also stacked on kv-cache : resolve the partial KV residency set once for the model #67, which has a FAIL review verdict.

Developmental progress

• Actual residency is now reconciled back into cparams.kv_gpu_layers at src/llama-model.cpp:2833, and MTP coverage was added at tests/test-llama-archs.cpp:1047. Those are useful fixes, but integration and policy correctness remain unacceptable.

Reviewed head: fe9617e