ggml-meta : split a host-resident KV cache by head

[Piggidragon]

Fixes --split-mode tensor together with --no-kv-offload, which on any model with more than
one KV head either aborts or silently returns wrong output. Adds --attn-split and makes
--kv-gpu-layers usable, both of which turn a host-resident cache from unusable into the
fastest option for a long context on an asymmetric pair of GPUs.

Replaces #48, which fixed one model shape and carried unrelated history.

The defect

With -sm tensor, ggml-backend-meta asks the model's split-state callback how each tensor is
divided across devices, but only for tensors outside a compute buffer:

if (ggml_backend_buffer_get_usage(tensor->buffer) != GGML_BACKEND_BUFFER_USAGE_COMPUTE && tensor->view_src == nullptr) {

With -nkvo the cache lives in host memory, so it is not a tensor of the meta backend at all.
The scheduler copies it in, and that copy is a leaf in the compute buffer. It never reaches the
callback and falls through to MIRRORED while the queries stay split by head, so both devices
hold every KV head but only half the query heads. handle_flash_attn_ext then takes its
kv_mirrored branch, which is correct only for MLA with one KV head, and each device's
renumbered queries attend the other device's keys.

Whether that aborts or only produces garbage depends on whether the per-device query head count
stays a multiple of the KV head count.

The commits

1. ggml-meta : split a host-resident KV cache by head - offer a copied-in compute-buffer
   leaf to the split-state callback under the name it was copied from, and split it on the head
   axis. The scheduler names such a copy <backend>#cache_k_l0 (view) (permuted)#0; undoing
   that decoration is the scheduler's own convention, so it is undone in the meta backend. The
   copy arrives permuted as [head_dim, n_kv, n_head_kv, n_stream], so the cache's segments and
   granularity are rescaled to whole heads. Every other copied-in leaf keeps the mirrored
   fallback.
2. llama : keep the recurrent state on device under split mode tensor - a linear-attention
   op packs the state it writes back together with its output, so that split does not line up
   with the one a host cache expects. On device it stays consistent; host-side the two orders
   disagree and produce plausible but wrong tokens. The state is small (~150 MiB on
   Qwen3.8-27B) and it is the attention cache that -nkvo exists to move.
3. kv-cache : spread partial GPU residency across devices - --kv-gpu-layers N took the
   first N layers in layer order, filling one device and leaving the others' free memory unused.
   It now takes one layer per owning device in turn.
4. kv-cache : apply partial GPU residency to every cache layout - --kv-gpu-layers was
   ignored by every cache made of several sub-caches (iSWA, DSA, DSV4, MSA), because each
   sub-cache would have taken the full budget. Resolved once for the whole model instead.
5. kv-cache : keep a recurrent layer out of the partial residency budget - a hybrid model
   counted every layer, so most of the budget went to layers the attention cache does not own.
   On Qwen3.5, --kv-gpu-layers 4 delivered one layer.
6. kv-cache : spend the partial residency budget on the slowest link first - a resident
   layer saves the transfer it would cost, which is worth most where the link is slowest. The
   host-to-device bandwidth is measured once through the same path the cache is delivered on,
   and the slowest device's layers are taken first. Devices within 15% stay one group and keep
   the round-robin of commit 3.
7. llama : add an attention split separate from the tensor split - --attn-split (-as).
   The share a device gets of the attention heads decides both how much of a host cache it
   receives and how much attention work it does, and neither has to follow the memory split.
   Unset, it follows --tensor-split and nothing changes.
8. llama : split a tied output projection under split mode tensor - a model without
   output.weight reuses the embedding table under the table's name. That copy landed in a meta
   buffer, missed pattern_output_weight and fell through to MIRRORED, so every device held
   the whole table and ran the whole projection.

Testing

Two GPUs, RTX 4070 (gen4 x16) + RTX 3060 (gen3 x4), CUDA, NCCL. Greedy, --temp 0 --top-k 1 --seed 1, 48 tokens, -c 4096, -fit off, -ts 50,50 for -sm tensor, sha256 prefix of the
output. "before" is the branch point, 43aaa59b6.

All numbers below were measured at stock clocks. An earlier revision of this description was
measured with the 4070 overclocked, which is why the throughput figures have moved by a few
percent.

-sm tensor -nkvo, before and after:

model	before	after
gemma-4-26B-A4B (attention MoE)	wrong: 3fdf2d9f40ad5d8c	fixed
Qwen3.8-27B-UD-IQ2_M (hybrid)	aborts, ggml-backend-meta.cpp:826	fixed
Qwen3.8-27B-UD-Q4_K_XL (hybrid)	aborts, ggml-backend-meta.cpp:826	fixed
Muse-Glimmer-30B-UD-Q5_K_M (DFlash2)	wrong: f89017c318699020	fixed
Ornith-1.5-35B-A3B (MTP)	aborts, ggml-backend-meta.cpp:826	fixed
gemma-4, -ts 55,45 -nkvo	aborts, fattn.cu:373 GGML_ASSERT(Q->ne[2] % K->ne[2] == 0)	fixed

"fixed" means the hash equals that model's -sm tensor hash:

model	-sm layer	-sm layer -nkvo	-sm tensor	-sm tensor -nkvo
gemma-4-26B-A4B	bdda82c6574fb090	bdda82c6574fb090	ed21fa6254900f95	ed21fa6254900f95
Qwen3.8-27B-UD-IQ2_M	dc9c9b89a57e8ac2	dc9c9b89a57e8ac2	74bc62ba2d48fd7a	74bc62ba2d48fd7a
Qwen3.8-27B-UD-Q4_K_XL	0ba93868baa7ddda	0ba93868baa7ddda	0ba93868baa7ddda	0ba93868baa7ddda
Muse-Glimmer-30B-UD-Q5_K_M	1b0c0da6c4bf450c	1b0c0da6c4bf450c	1b0c0da6c4bf450c	1b0c0da6c4bf450c
Ornith-1.5-35B-A3B	8372a5cb22e6f1c1	8372a5cb22e6f1c1	eab8b94ee9cb4376	eab8b94ee9cb4376

Nothing changes where it already worked: all ten -sm layer and -sm tensor hashes above are
identical on 43aaa59b6. Uneven splits agree too - on gemma-4, -ts 55,45, 60,40, 45,55
and 65,35 each give the same hash with and without -nkvo.

The head split works in bytes off nb[2], so it does not depend on the cache type. On
Qwen3.8-27B-UD-Q5_K_M, -sm tensor, -sm tensor -nkvo and -sm tensor -nkvo --kv-gpu-layers 4
all give 8dfc3fab3519a246 with -ctk q8_0 -ctv q8_0, and the same again with q4_0.

Perplexity, -c 2048 over a 15216-token slice of this repository's docs. Unmodified build of
43aaa59b6 against this branch, identical to four decimals:

model	-sm layer	-sm tensor
gemma-4-26B-A4B	148.6177 both	153.0402 both
Qwen3.8-27B-UD-IQ2_M	4.0475 both	4.0464 both

and moving the cache to host memory, pinning it, or keeping part of it on the device leaves the
result unchanged:

config	gemma-4	Qwen3.8-27B-UD-IQ2_M
-sm tensor	153.0402	4.0464
-sm tensor -nkvo	153.0402	4.0464
-sm tensor -nkvo --kv-cpu-pinned	153.0402	-
-sm tensor -nkvo --kv-gpu-layers 4	-	4.0464
-sm layer -nkvo	148.9252	-
-sm layer -nkvo --kv-gpu-layers 8	148.9252	-

Performance

llama-bench, -ngl 99 -ts 50/50 -r 3. -sm tensor -nkvo did not run before this PR.

model	sm	nkvo	pp512	tg128
Qwen3.8-27B-UD-IQ2_M	layer	0	750.47	26.35
	layer	1	532.75	8.70
	tensor	0	609.67	30.85
	tensor	1	589.09	25.68
gemma-4-26B-A4B	layer	0	2638.81	103.47
	layer	1	1938.25	42.98
	tensor	0	1691.01	91.64
	tensor	1	1318.80	36.36

Which split mode wins is model-dependent, not a property of MoE.

Where the residency budget goes

The case this is for: a dense model that has to stay on the GPUs, and a context large enough
that the cache no longer fits beside it. Qwen3.8-27B-UD-Q5_K_M (19.8 GB), -ngl 99 -n 128,
prompt filled to the context. The two links are not alike and the startup measurement reports
what they do:

llama_pick_gpu_resident_layers: CUDA0: host-to-device 24.2 GB/s   (gen4 x16)
llama_pick_gpu_resident_layers: CUDA1: host-to-device  3.3 GB/s   (gen3 x4)

A factor of 7.3, so a layer kept off the 3060 is worth 7.3 kept off the 4070. -c 20480,
15216-token prompt, f16 cache, -sm layer -nkvo --kv-cpu-pinned --recurrent-state-offload. The
model has 16 owned attention layers, 8 per GPU; the round-robin column is this branch with
commit 6 reverted:

--kv-gpu-layers	round-robin (commit 3)	slowest link first (commit 6)
0	4.37	4.37
2	4.83	5.24
4	5.39	6.54
8	7.01	13.00
16 (all)	17.76	17.76

At 8 layers, the point where the 3060 owns none of the host-resident cache, the two orders cost
the same memory and differ by +85%. Prefill moves with it, 645.77 to 959.89 t/s. Both orders
converge at 16, where each takes every layer. A device-resident cache reaches 17.78.

-c 65536, 58466-token prompt, where the cache cannot be device-resident at all:

cache	--kv-gpu-layers	pp	tg
f16	0	414.70	1.39
q8_0	0	476.15	2.39
q8_0	8, round-robin	516.27	4.11
q8_0	8, slowest first	808.46	9.06
q8_0	16 (all)	804.09	14.45

8 layers cost 1088 MiB, all of it on the 3060. Against the f16 baseline that is 6.5x decode;
against the same budget spread round-robin, +120%.

A quantized cache is a transport saving here, not a kernel cost: device-resident, f16 is
slightly faster than q8_0 (17.78 against 17.41); host-resident at -c 65536 it is +72% (1.39
against 2.39). The link is the limit. Halving the bytes also halves what a resident layer costs.

Under -sm tensor the budget only pays near-complete

Same model and context, -sm tensor -ts 50,50 -nkvo, q8_0:

--kv-gpu-layers	pp	tg
0	425.92	2.41
8	464.38	4.29
16 (all)	507.38	19.82

Tensor parallelism splits every layer's cache across both devices, so a partly resident cache
still sends the remaining layers to the slow device; only the last layer removes the transport.
Commit 6 changes nothing here - all layers report the same meta device, so there is one group.
--kv-gpu-layers under -sm tensor is worth setting to the full attention layer count or not
at all.

At full residency -sm tensor beats -sm layer on generation (19.82 against 14.45) and loses
on prefill (507.38 against 804.09).

Giving the heads a ratio of their own

The other way out is to stop sending the slow device half the cache. Same prompt, -sm tensor -ts 50,50 -nkvo, q8_0:

--attn-split	--kv-gpu-layers	pp	tg
(follows -ts)	0	425.92	2.41
1,0	0	493.99	7.36
1,0	8	505.62	10.69

3.1x from the head split alone, and 4.4x combined with the residency budget. Perplexity is
unmoved: on Qwen3.8-27B-UD-Q5_K_M at -c 2048, -sm layer gives 3.7843, -sm tensor 3.7860
and -sm tensor -as 1,0 3.7824, against an error bar of +/- 0.096.

The share is rounded to whole heads, and the rounding is what the granularity rules already
enforce so that GQA stays consistent. The cache copy and the query tensor do not always round to
the same share: Qwen3.8-27B has n_head_kv = 4 and a query tensor fused with a gate, and
-as 0.75,0.25 there lands on 3,1, not on 2,2. It is a distinct working share, identical to
an explicit -as 3,1 (3.7831 perplexity for both) and measurably between the two neighbours -
3.47 t/s against 2.41 for 2,2 and 7.36 for 4,0 on the run above.

Setting --attn-split when it is not used costs nothing. Against this branch with commit 7
reverted, same configs:

config	without commit 7	with
-sm tensor -nkvo	426.19 / 2.42	425.92 / 2.41
-sm tensor -nkvo --kv-gpu-layers 16	507.70 / 19.83	507.38 / 19.82

--tensor-split wants the fast device

With a device-resident cache both devices run concurrently inside a layer and the collective is
a barrier, so the layer costs max(share_i / rate_i) and the optimum sits near the compute
ratio, not at the memory ratio. The default splits by free memory, which on two 12 GiB cards is
50/50 regardless of speed. llama-bench -r 3, -sm tensor:

gemma-4-26B-A4B:

-ts	pp512	tg128
40/60	1582.41	86.02
50/50	1693.96	91.87
55/45	1788.10	96.56
60/40	1791.29	97.67
65/35	1881.30	101.80
70/30	1881.71	101.87
75/25	does not fit

Qwen3.8-27B-UD-Q5_K_M, which sits much closer to the memory limit:

-ts	pp512	tg128
50/50	604.35	25.75
55/45	641.44	27.98
58/42	652.46	28.91
60/40	660.10	29.19
62/38	657.86	28.75
64/36	does not fit

+11% prefill and +11 to +13% generation from an argument. On both models the optimum is a real
compute optimum below the memory limit and lands near the 4070:3060 ratio, not at the limit
itself: gemma-4 flattens after 65/35 and stops fitting at 75/25, Qwen3.8 peaks at 60/40, is past
it at 62/38 and stops fitting at 64/36.

With a host-resident cache the answer differs, because -as 1,0 already puts the cache on the
fast card and there is no room left to move weight there as well. -c 65536, -as 1,0 --kv-gpu-layers 8:

-ts	pp	tg
50,50	505.62	10.69
45,55	483.09	10.36
40,60	472.28	10.17
55,45	does not fit
60,40	does not fit

Under -sm layer the answer is the opposite again, because there the devices run one after the
other and -ts is the layer count. Same model and prompt, --kv-gpu-layers 16:

-ts	pp	tg
50,50	804.09	14.45
55,45	918.80	14.88
57,43	941.78	14.94

+17% prefill for nothing but an argument. 60,40 no longer fits.

Most of Qwen3.8's -sm layer -nkvo deficit is the recurrent state, not the attention cache:
-c 4096 -n 128, 7.62 t/s against 17.24 with --recurrent-state-offload. -sm tensor gets
that for free after commit 2.

Not fixed here

• The asynchronous head-split delivery for sched: pipeline the delivery of a host-resident KV cache #39. That PR pipelines the host-to-device
  delivery of the cache and lists the strided head-split write as the one piece missing for
  -sm tensor; commit 1 supplies the synchronous form it needs.
• Deriving --attn-split automatically from the measured bandwidth. The measurement is
  already there and the rounding would land on 1,0 for this pair, but it changes the reduction
  order for every multi-GPU user, so it stays opt-in until it has run on more than one machine.
• llama_params_fit for SPLIT_MODE_TENSOR, so -ts has to be set by hand. The sweep
  above gives it a second motive: the optimum is worth +11% and it cannot be made a default
  without a feasibility check against free memory. The margin is narrow - on Qwen3.8-27B-Q5_K_M
  the optimum is 60/40 and 64/36 no longer fits.
• A host-resident recurrent state under -sm tensor (commit 2 sidesteps it).
• An iSWA cache split by tensor still keeps the whole cache in host memory, because a mix of
  device- and host-resident layers degrades there. On gemma-4, -sm tensor -nkvo, one, two or
  four resident layers all give bit-identical and worse perplexity while 0 and "all" agree, so
  something switches as soon as the cache is mixed. It is not offload_attn_compute and not an
  unpinned host buffer; layer split and a plain non-iSWA cache under tensor split are both
  unaffected. Root-causing it needs a look at how the meta backend maps externally created views
  onto its per-buffer compute containers, which carries its own FIXME already.

Earlier revisions of this description carried four rejected optimizations and their
measurements. They are preserved in a comment below rather than here.

AI usage disclosure: yes, see the commit trailers.

[Piggidragon]

Archive: rejected optimizations and open work

Moved out of the description to keep it to what this PR actually changes. Nothing here is part
of the diff. These numbers were measured with the RTX 4070 overclocked (+200 MHz core,
+1500 MHz memory) and have not been re-measured at stock clocks, unlike everything in the
description. The verdicts do not depend on the few percent that costs, but the absolute figures
are a few percent high.

Same two GPUs, llama-bench -r 3 unless noted.

Rejected: even FFN split boundaries

get_split_granularity uses lcm(blck_size, 128) for the FFN and the device boundary is
rounded down, so a segment that is not a multiple of 128*n_devices splits unevenly. On gemma-4
(n_ff_exp 704, ffn_down_exps Q8_0) that is {256, 448} per layer instead of {352, 352},
and every layer ends in a collective, so the larger share sets the cost. Lowering the
granularity until the segment divides evenly gives {352, 352}; rounding the boundary to the
nearest multiple instead of down gives {384, 320}.

	pp512	tg128
baseline {256, 448}	1686.4	85.40
even {352, 352}	1620.86	85.70
nearest {384, 320}	1660.32	86.07

Both lose prefill and neither gains generation. -sm tensor prefill on this model is not
FFN-width-bound, and the coarser slices run better kernels. The imbalance is real but it is not
what costs anything.

Rejected: size the compute buffer per device

ggml_backend_meta_buffer_type_get_alloc_size returns the full tensor size, alloc_buffer
allocates that on every device, and init_tensor_impl places each device's slice at the logical
offset. So under -sm tensor every device carries a compute buffer sized for the whole, unsplit
graph. gemma-4 at -c 4096:

-b/-ub	-sm layer, both devices	-sm tensor, per device x2	overhead
512	118.57 + 560.70 = 679 MiB	534.02 x2 = 1068 MiB	+389 MiB
2048	498.28 + 2266.81 = 2765 MiB	2142.08 x2 = 4284 MiB	+1519 MiB

Not implemented. It is memory only, never speed; the fix has to change what get_alloc_size may
assume about a tensor that has no buffer yet, which is a change to the meta backend's allocation
contract with a silent-corruption failure mode; and a user can recover the same bytes with a
smaller -ub. Recorded as a known property.

Rejected: pipeline parallelism with a host-resident cache

pipeline_parallel is gated on cparams.offload_kqv, which -nkvo clears even when
cparams.offload_attn_compute keeps attention on the device, so -sm layer -nkvo --kv-cpu-pinned never pipelines. Relaxing the gate to offload_kqv || offload_attn_compute
does enable it, and it buys nothing.

The control settles it. On -sm layer with a device-resident cache, where upstream already
enables pipelining, turning it off changes nothing (16k-token prompt, -c 8192):

	pp	tg
pipeline on	929.39	18.82
pipeline off	922.21	18.76

The scheduler's n_copies only overlaps graph inputs, and a host-resident cache does not
reach a split as one. With the gate relaxed, -nkvo --kv-cpu-pinned measured 511.94 to 509.50
pp and 5.76 to 5.62 tg while peak VRAM rose by about 1360 MiB per device. Whatever -nkvo costs
at prefill, it is not the lost pipelining.

Rejected: backend sampling under -sm tensor

llama_context::set_sampler refuses a sampler under SPLIT_MODE_TENSOR (upstream ad27757).
Dropping that guard is not enough. This PR splits output.weight on axis 1, so the logit row is
split across the vocabulary, and every per-row sampler op trips ggml-backend-meta.cpp:564,
GGML_ASSERT(src_ss[0].axis != GGML_BACKEND_SPLIT_AXIS_0). The meta backend has no all-gather.
The one split state that puts a whole logit row on every device is MIRRORED.

Qwen3.8-27B-UD-Q5_K_M, -sm tensor -ts 50/50 -fa on -c 4096, server /completion, 256 tokens:

output.weight	backend sampling	tg	VRAM CUDA0 + CUDA1
axis 1 (this PR)	off	25.23	9772 + 9522 MiB
MIRRORED	off	24.36	10270 + 10020 MiB

Mirroring the head costs 3.4% generation and 996 MiB before backend sampling wins anything back.
That is commit 8 run backwards. What it wins, on Qwen3-0.6B-Q8_0, 512 tokens:

-sm	output.weight	backend sampling	tg
layer	(n/a)	off	318.8
layer	(n/a)	on	349.0 (+9.5%)
tensor	axis 1	off	214.9
tensor	MIRRORED	off	207.7
tensor	MIRRORED	on	211.8 (+2.0%)

+9.5% under -sm layer is the ceiling, on a 0.6B model. The saving is a fixed ~0.27 ms per
token; on a 27B at 41 ms per token it is below 1%, against the 3.4% the mirrored head costs. It
is also not clean: with the head mirrored, the first sampled request aborts at
ggml-backend-meta.cpp:901, because build_sampling pads t_logits and a ubatch with no output
rows leaves result_output with zero elements, which has no defined split state.

Rejected: per (layer, device) residency for --kv-gpu-layers

Letting a layer be device-resident on one device and host-resident on the other. --attn-split
already reaches that point and pays no memory for it. Qwen3.8-27B-UD-Q5_K_M, -c 65536,
58780-token prompt, q8_0, -nkvo --kv-cpu-pinned -rso -sm tensor -ts 50,50 -n 128:

--attn-split	--kv-gpu-layers	VRAM CUDA0 + CUDA1	pp	tg
(follows -ts)	0	10106 + 9852	426.94	2.40
1,0	0	10724 + 9230	495.54	7.35
(follows -ts)	4	10376 + 10122	444.37	3.07
1,0	4	11266 + 9228	502.47	8.74
(follows -ts)	8	10646 + 10392	464.56	4.26
1,0	8	11808 + 9228	507.82	10.71
(follows -ts)	12	10916 + 10662	484.93	6.98
1,0	12	cudaMalloc failed on CUDA0
(follows -ts)	16	11152 + 10898	507.56	19.71

The pairs match on total VRAM to within 4 MiB, so those rows are equal-budget comparisons.
-as 1,0 at 8 layers is exactly what the finer knob would deliver at that budget. Below it,
-as 1,0 wins outright because it drops the slow link for free while a residency budget has to
buy its way out.

One narrow band is left: between half and full residency -as 1,0 runs out of room on the fast
device while the even split still pays the slow link. Per-device residency would reach about
14 t/s there against the even split's 6.98 - modelled, not measurable today. Against that stands
mixing buffer types inside a meta buffer or staging a slice per subgraph, both new machinery in
ggml-backend-meta.cpp. Rejected.

Rejected: overlapping the AllReduce with compute

Built and measured. Worth +3.0% prefill, regresses the one MoE tested, and takes an intermittent
illegal memory access.

Skipping the AllReduce entirely (wrong output, timing only) on Qwen3.8-27B-UD-Q5_K_M, a
13500-token prompt, -sm tensor -ts 50/50 -c 20480: prefill goes from 23443 ms to 14573 ms. The
collective is 38% of prefill and it is link bound, about 21 MiB per direction per boundary
over the gen3 x4 link, 10.5 ms measured against 12.8 ms of pure transfer. Hiding all of it would
be +61%.

Only the tail projection of a subgraph can go in that shadow; everything else is upstream of the
collective and the next subgraph is downstream. A longer suffix is not safe to chunk, because
ggml-alloc stacks the intermediates inside a suffix on top of each other. The tail is 3150 ms
of the 14573 ms of compute, so hiding all of it caps the change at +11%.

393 lines were built: a split collective in ggml-backend.h
(comm_allreduce_tensor_begin/end), a NCCL implementation on its own stream with an event pair
per slot, and chunked tail MUL_MAT in ggml-backend-meta.cpp, software pipelined at depth 1.

model	prefill off	prefill on	delta	tg
Qwen3.8-27B-UD-Q5_K_M	575.4	592.7	+3.0%	24.66 to 24.67
Qwen3.8-27B-UD-IQ2_M	578.3	596.2	+3.1%	29.42 to 29.55
gemma-4-26B-A4B	1522.6	1500.6	-1.4%	80.2 to 80.1

The pipelining is what pays, not the chunking: baseline 569.6, chunked tail with the one-shot
collective 572.5, chunked tail with the split collective 592.1. The split collective is
bit-identical to the one-shot one.

gemma-4 takes an illegal memory access in 2 to 4 runs out of 12, always at the end of prefill.
It needs both the chunked tail and a collective (AllReduce disabled: 0/14; tail computed whole:
0/10). Not the split collective, not NCCL, not CUDA graphs, not the view node.
compute-sanitizer memcheck reports nothing. That is the shape of a latent synchronization
hazard in the existing AllReduce path that the chunking exposes rather than causes. Not
localized.

Rejected. The measurement does point somewhere: hiding the collective properly means
micro-batching the ubatch one level up, in llama, where a whole layer of compute sits between
one boundary and the next. That is a different and much larger change.

Still open

Everything under "Not fixed here" in the description still stands. These come out of the
measurements above, cheapest first.

• -as and -kvgl for llama-bench. Both exist in common/arg.cpp but llama-bench
  only takes -sm, -ts, -ub, -ngl and -fa, so the two sweeps that produced the
  largest wins had to be run through llama-completion. sched: pipeline the delivery of a host-resident KV cache #39 already adds -kvcp and -rso
  to llama-bench; these belong next to them.
• Sweep -ub under -sm tensor. Never measured. It is also the only lever a user has
  against the per-device compute buffer above.

[GenerelSchwerz]

Review of frozen head cf91605fe54d279e5f4ffa341ca22d572961a8bc against declared base 43aaa59b6c1d9b33e686f393607b660047a68e19.

Blocking

1. The PR fails the scope quick-reject gate.

The documented upstream issue #27757 concerns host-resident KV under tensor parallelism, but this 395-line diff also adds a public attention-split API, a runtime bandwidth benchmark and new residency policy, tied-output handling, recurrent-state policy changes, meta-buffer OOM semantics, and an unrelated CLAUDE.md mode change. This conflicts with the fork requirements that features begin with an issue, public API additions meet a higher bar, and unrelated changes use separate PRs (CONTRIBUTING.md lines 31-52).

Keep the host-cache correctness fix and its regression test focused here. Move the attention-split API, bandwidth/residency policy, tied-output optimization, allocation behavior, and documentation cleanup into separately justified PRs.

2. The copied-cache name parser fails on supported high-GPU-count configurations.

Scheduler copy names are stored in a 64-byte fixed field and formatted as <backend>#<source>#<copy> here. With eight short CUDA device names, Meta(CUDA0,CUDA1,CUDA2,CUDA3,CUDA4,CUDA5,CUDA6,CUDA7)#cache_k_l42#0 is 67 characters and truncates to Meta(CUDA0,CUDA1,CUDA2,CUDA3,CUDA4,CUDA5,CUDA6,CUDA7)#cache_k_l.

The new prefix parser returns cache_k_l. Because the cache regex permits zero digits (\d*) at line 382, it matches, after which line 488 calls std::stoul("") and throws. With still longer backend names, matching instead fails and the cache silently returns to MIRRORED, recreating the wrong-head behavior.

This is not theoretical topology: upstream has reports involving 8-10 CUDA devices. The design should preserve source-tensor identity in scheduler metadata instead of recovering it from a lossy display name. Requiring \d+ and checking conversion would prevent the exception but would not restore truncated identity.

3. --spec-draft-kv-gpu-layers no longer selects any MTP/NextN cache layer.

The option explicitly represents independently owned draft KV layers. Speculative setup disables full KV offload and copies the requested count here, then creates an MTP context here.

The new global picker only considers IDs below hparams.n_layer() lines 2355-2371. MTP caches explicitly accept only IDs at or above hparams.n_layer(), for example GLM-DSA/DeepSeek32 and the default MTP path here. The sets are disjoint, so membership-based placement leaves every MTP cache layer on the host while reporting the requested count as successful.

The declared base selected the first N layers after applying the cache filter (base lines 242-251, base placement), so this is a regression. Selection must use n_layer_all and the actual context/cache filters, or remain inside each cache with a shared budget abstraction.

4. Current head does not pass warning-as-error builds.

llama_memory_placement_options gained a third member here, but the aggregate initializer omits it. Required ARM64 CI fails with -Werror=missing-field-initializers at that location (CI job). I reproduced the identical failure locally on x86 with LLAMA_FATAL_WARNINGS=ON. Initialize the set explicitly or construct and assign the options without a partial aggregate.

Will slow the review

1. No regression coverage was added for the new behavior.

Existing test-llama-archs creates default context parameters, leaving KV offload enabled lines 357-387. It therefore cannot exercise the central -sm tensor -nkvo path, the truncated copied-cache name, partial/MTP residency, or attn_split.

This area has strong upstream precedent: an earlier meta tensor-split change was reverted after a crash, with a maintainer explicitly requesting tensor-split CI coverage (comment. The corrected implementation #27574 added MQA, GQA, per-head fixtures, and 1-4 virtual-device CI. Reuse tests/test-llama-archs.cpp rather than adding a new file, and cover host KV, an 8-device name case, differing K/V head dimensions, custom attention split, and MTP draft residency.

2. The new parser repeats a known maximum-device off-by-one.

split_arg.size() >= llama_max_devices() rejects exactly the number of entries the public array supports. Upstream #28028 identifies the same inherited tensor-split check as a bug and adds boundary tests. Use > and cover N and N+1 entries.

3. Several added comments violate the concise, no-forced-wrapping rule.

Examples include the four-line copy-name explanation here and host-cache explanation here. Reduce these to short why-only comments after the design is simplified.

Validation

• Normal CUDA build passed, but emitted the missing-initializer warning.
• test-arg-parser: passed.
• Full CPU test-llama-archs -s 1: passed.
• Full two-virtual-device CUDA architecture matrix: passed.
• Focused Llama CUDA runs with 2 and 8 virtual devices: passed, but do not use host KV and therefore do not exercise the copied-cache defect.
• git diff --check and an added-line ASCII scan passed.
• Current self-hosted GPU CI jobs were canceled after approximately 24 hours, so they provide no conclusive backend coverage.
• Comparison was against current pristine upstream at 159b7414; no equivalent host-KV or attention-split implementation has landed.
• AI disclosure and Assisted-by: usage are correct.
• No actionable security findings or additional nits survived verification.

[Piggidragon]

Splitting this up as requested. Closing in favour of seven focused PRs, all against llama/dev,
each re-measured from scratch against its own base rather than reusing the numbers here.

PR	change	size
#66	ggml-meta : split a host-resident KV cache by head - the correctness fix, plus its regression test	198/24
#65	llama : split a tied output projection under split mode tensor	4/1
#64	ggml : report allocation failure from the meta buffer type	22/8
#67	kv-cache : resolve the partial KV residency set once for the model	169/40
#68	kv-cache : spend the partial residency budget on the slowest link first (needs #67)	+91/13
#69	llama : add an attention split separate from the tensor split (needs #66)	+65/2
#63	docs : link CLAUDE.md to AGENTS.md	1/1

#66, #65, #64, #67 and #63 are independent of each other and can be reviewed in any order.

On the review points from this PR:

• Scope. ggml-meta : split a host-resident KV cache by head #66 is now only the host-cache defect and its test. The attention-split API, the
  bandwidth/residency policy, the tied-output handling, the allocation semantics and the docs change
  are each separately justified.
• Truncated copy names. ggml-meta : split a host-resident KV cache by head #66 stops the source name being the part that gets cut: the scheduler now
  trims the backend label instead, since the source name is what identifies the copy. The cache
  patterns also require \d+ rather than \d*, so a truncated name cannot match and reach
  std::stoul(""). ci/run.sh gains an 8-device run that exercises it.
• --spec-draft-kv-gpu-layers / MTP. kv-cache : resolve the partial KV residency set once for the model #67 selects over n_layer_all and the requesting context
  type, so an MTP context resolves its own nextn layers. tests/test-llama-archs.cpp gains
  test_mtp_kv_residency, which fails if the budget is resolved against the main context's range.
• Warning-as-error build. The placement options are assigned member-wise instead of through a
  partial aggregate initializer. All seven branches build with -DLLAMA_FATAL_WARNINGS=ON.
• Regression coverage. ggml-meta : split a host-resident KV cache by head #66 runs the tensor-split matrix a second time with the cache in host
  memory, and llama : add an attention split separate from the tensor split #69 a third time with a custom attention split. Both reuse
  tests/test-llama-archs.cpp; no new test file.
• llama_max_devices() off-by-one. llama : add an attention split separate from the tensor split #69 uses > rather than >=.
• Comment length. The long copy-name and host-cache comments are cut down.

One item from this PR's description did not survive re-measurement and has been dropped rather than
carried over: the claim that a hybrid model spent the residency budget on recurrent layers. On
llama/dev a hybrid reports 4 of 4 requested owned attention layers device-resident, because each
cache already applies its own layer filter. That fault only existed inside this PR's own commit
sequence. #67 still excludes recurrent layers, because resolving one budget for the whole model
would otherwise introduce it.

Still open, and stated as a limitation in #66 rather than claimed fixed: an iSWA cache under
-sm tensor does not reach exact parity between a host- and a device-resident cache. On
gemma-4-26B-A4B the perplexity is 227.2279 device-resident against 235.0273 host-resident, down from
5.96e10 before the fix. A non-iSWA model reaches parity exactly (Qwen3.8-27B-UD-IQ2_M: 4.6940 both).