ggml-meta : split a host-resident KV cache by head

[Piggidragon]

Overview

--split-mode tensor together with --no-kv-offload either aborts or silently returns wrong
output on any model with more than one KV head. This fixes that.

This is the correctness part of #57, which was too large to review as one change. #57 can be closed
in favour of this set:

PR	change	depends on
this one	host-resident KV cache split by head	-
#65	tied output projection under -sm tensor	-
#64	meta buffer type reports allocation failure	-
#67	partial KV residency resolved once for the model	-
#68	residency budget spent on the slowest link first	#67
#69	--attn-split	this one
#63	CLAUDE.md symlink	-

The defect

With -sm tensor, ggml-backend-meta asks the model's split-state callback how each tensor is
divided across devices, but only for tensors outside a compute buffer:

if (ggml_backend_buffer_get_usage(tensor->buffer) != GGML_BACKEND_BUFFER_USAGE_COMPUTE && tensor->view_src == nullptr) {

With -nkvo the cache lives in host memory, so it is not a tensor of the meta backend at all. The
scheduler copies it in, and that copy is a leaf in the compute buffer. It never reaches the callback
and falls through to MIRRORED while the queries stay split by head, so both devices hold every KV
head but only half the query heads. handle_flash_attn_ext then takes its kv_mirrored branch,
which is correct only for MLA with one KV head, and each device's renumbered queries attend the
other device's keys.

Whether that aborts or only produces garbage depends on whether the per-device query head count
stays a multiple of the KV head count.

The commits

1. ggml : keep the source name of a scheduler copy - a copy is named <backend>#<source>#<copy>
   in a name field of fixed size. The meta backend's label lists every device, so with enough devices
   the source name is what gets cut and the consumer can no longer tell which tensor the copy came
   from. Cut the label instead.
2. ggml-meta : split a host-resident KV cache by head - offer a copied-in compute-buffer leaf
   to the split-state callback under the name it was copied from, and split it on the head axis. The
   copy arrives permuted as [head_dim, n_kv, n_head_kv, n_stream], so the cache's segments and
   granularity are rescaled to whole heads. Every other copied-in leaf keeps the mirrored fallback.
3. llama : split a V tensor on its own head size - the KV granularity was derived from the query
   granularity through n_gqa, which assumes the V side has the head size of the K side.
4. ggml : read back a strided row split from a meta buffer - a fused QKV puts Kcur and Vcur in a
   strided view that a host-resident cache reads back through the meta buffer.
5. llama : keep the recurrent state on device under split mode tensor - a linear-attention op
   packs the state it writes back together with its output, so that split does not line up with the
   one a host-resident state expects. Needed for any hybrid or recurrent model to run at all here.
6. tests : cover a host-resident KV cache split by tensor - see below.

Testing

Two GPUs, RTX 4070 (gen4 x16) + RTX 3060 (gen3 x4), CUDA, NCCL, stock clocks. Greedy,
--temp 0 --top-k 1 -s 1, 48 tokens, -c 4096, -fit off, -ts 50,50 for -sm tensor, sha256
prefix of the output. "before" is llama/dev at 9dc93fd62.

-sm tensor -nkvo, before and after:

model	before	after
gemma-4-26B-A4B (iSWA)	wrong: defcc467489b9fe9	e07cb42d45be171f
Qwen3.8-27B-UD-IQ2_M (hybrid)	aborts, ggml-backend-meta.cpp:826	92e1787dfbff1354
Qwen3.8-27B-UD-Q4_K_XL (hybrid)	aborts, ggml-backend-meta.cpp:826	361ca91089116566
Muse-Glimmer-30B-UD-Q5_K_M (DFlash2)	wrong: a80adbb9cefb1279	9b50b14b17935032
Ornith-1.5-35B-A3B (MTP)	aborts, ggml-backend-meta.cpp:826	58f6e8d41f1747f0
gemma-4, -ts 55,45 -nkvo	aborts, fattn.cu:506 GGML_ASSERT(Q->ne[2] % K->ne[2] == 0)	1bc54856ec2b7c82

Each "after" hash equals that model's -sm tensor hash, so the host-resident cache now agrees with
the device-resident one.

Nothing changes where it already worked. All 15 configurations below are bit-identical before and
after:

model	-sm layer	-sm layer -nkvo	-sm tensor
gemma-4-26B-A4B	1243fdca3e993373	adbdcb93dcd4bf77	e07cb42d45be171f
Qwen3.8-27B-UD-IQ2_M	1872d08d9c815611	1872d08d9c815611	92e1787dfbff1354
Qwen3.8-27B-UD-Q4_K_XL	361ca91089116566	361ca91089116566	361ca91089116566
Muse-Glimmer-30B-UD-Q5_K_M	287db7134353eb01	287db7134353eb01	9b50b14b17935032
Ornith-1.5-35B-A3B	58f6e8d41f1747f0	58f6e8d41f1747f0	58f6e8d41f1747f0

gemma-4 is the one model where -sm layer and -sm layer -nkvo differ, before and after alike. The
perplexity above puts that at 0.10%, i.e. one token flipping on float noise rather than a defect.

Uneven splits

gemma-4, -sm tensor, with and without -nkvo. Identical in every case, and none of them abort:

-ts	device-resident	host-resident
55,45	1bc54856ec2b7c82	1bc54856ec2b7c82
60,40	bff10e2e7c2eddfe	bff10e2e7c2eddfe
45,55	be56a72280d7ff0e	be56a72280d7ff0e
65,35	fa424d57e41011a2	fa424d57e41011a2

Perplexity

-c 2048 over a 15216-token slice of this repository's docs, llama-perplexity, 7 chunks.

model	config	before	after
gemma-4-26B-A4B	-sm layer	244.1475	244.1475
	-sm tensor	227.2279	227.2279
	-sm tensor -nkvo	59629416583.1496	235.0273
Qwen3.8-27B-UD-IQ2_M	-sm layer	4.6942	4.6942
	-sm tensor	4.6940	4.6940
	-sm tensor -nkvo	aborts	4.6940

Every device-resident configuration is unchanged to four decimals. Qwen3.8 host-resident now matches
its device-resident value exactly.

gemma-4 is iSWA and does not reach that identity: 235.0273 against 227.2279, a 3.4% gap, from
5.96e10 before. The output is coherent and the greedy hashes above match, so this is not the
mirrored-head defect. It is listed under "not fixed here" rather than claimed as solved.

The gap is specific to -sm tensor. Under -sm layer the same model and cache reach parity:

gemma-4-26B-A4B	device-resident	host-resident	gap
-sm layer	244.1475	244.3848	0.10%
-sm tensor	227.2279	235.0273	3.43%

so a host-resident iSWA cache is not generally at fault; something in the tensor-split path is. That
0.10% is also what makes the two -sm layer hashes in the table below differ - a single greedy token
flipping on float noise, not a defect.

Throughput

llama-bench -ngl 99 -ts 50/50 -r 3. On llama/dev the -sm tensor -nkvo row aborts the whole
llama-bench process, which is why there is no before column for it.

model	sm	nkvo	pp512 before	pp512 after	tg128 before	tg128 after
Qwen3.8-27B-UD-IQ2_M	layer	0	754.47	755.87	26.39	26.38
	layer	1	533.32	532.98	8.66	8.68
	tensor	0	610.03	610.71	30.95	30.96
	tensor	1	aborts	588.78	aborts	25.61
gemma-4-26B-A4B	tensor	0	1687.72	1685.83	85.86	85.84
	layer	0	-	2732.69	-	103.83
	layer	1	-	1941.45	-	42.87
	tensor	1	aborts	1339.58	aborts	34.97

Nothing that worked before moves outside run-to-run noise. On Qwen3.8 a host-resident cache under
-sm tensor reaches 25.61 tg against 8.68 under -sm layer, which is what makes the option usable.

Regression test

tests/test-llama-archs.cpp ran the tensor-split matrix with the default context parameters, which
leave KV offload on, so it could not reach this path at all. It now runs the meta configuration a
second time with offload_kqv = false, and ci/run.sh adds an 8-device run, where the scheduler
copy name is long enough to be truncated.

Every model in the matrix aborts or mismatches on llama/dev in the Meta -nkvo column and passes
with this PR. test-llama-archs -s 1 at 1, 2, 3 and 4 virtual CUDA devices and the 8-device
-a llama run all pass. Built with -DLLAMA_FATAL_WARNINGS=ON.

Not fixed here

• The asynchronous head-split delivery for sched: pipeline the delivery of a host-resident KV cache #39. That PR pipelines the host-to-device delivery of the
  cache and lists the strided head-split write as the one piece missing for -sm tensor; commit 2
  supplies the synchronous form it needs.
• A host-resident recurrent state under -sm tensor (commit 5 sidesteps it).
• An iSWA cache under -sm tensor does not reach parity with a device-resident cache, as measured
  above (3.43%, against 0.10% under -sm layer). The same combination has a second symptom: a mix of
  device- and host-resident iSWA layers degrades, which is why kv-cache : resolve the partial KV residency set once for the model #67 refuses --kv-gpu-layers for iSWA
  under -sm tensor. Both are probably the one root cause, and the likely place to look is how the
  meta backend maps externally created views onto its per-buffer compute containers, which carries
  its own FIXME already. Not root-caused here.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES - implemented by an agent on my instruction, see the Assisted-by: commit trailers.

[GenerelSchwerz]

Automated preliminary review by Codex; the repository owner plans a separate manual review.

Verdict: CONDITIONAL PASS.

Blocking

• None. No blocking implementation defect was proven.

Will slow review

• The multi-stream repair at ggml/src/ggml-backend-meta.cpp:1469-1483 now iterates ne[3], but the added host-KV matrix leaves n_seq_max at its default of 1 (tests/test-llama-archs.cpp:397-403, src/llama-context.cpp:4051-4057). Add direct n_seq_max > 1 coverage for this path. The feature-relevant CUDA and Metal jobs that invoke ci/run.sh:305-324 are currently queued with no executed steps; they have not failed.
• Gemma4/iSWA remains skipped at tests/test-llama-archs.cpp:1349-1351, while the reported tensor-split host/device KV perplexities are 235.0273 and 227.2279, a 3.43% gap. Track this explicitly, gate the affected combination, or resolve it before treating coverage as complete.
• ggml/src/ggml-backend-meta.cpp:488-507 reconstructs source identity from the scheduler tensor display-name format created at ggml/src/ggml-backend.cpp:1037-1047. This is an undocumented grammar shared across components; make the identity contract explicit or cover it directly.
• The new strided paths call 2D transfers per device and stream (ggml/src/ggml-backend-meta.cpp:1477-1479, ggml/src/ggml-backend-meta.cpp:1634-1635). Backends without native 2D transfer fall back to one synchronous set/get per row at ggml/src/ggml-backend.cpp:371-395, which can amplify calls substantially. This needs measurements or an explicit support/performance note.
• The recurrent workaround at src/llama-context.cpp:148-153 overrides requested placement and can increase VRAM use. Make that behavior explicit or gated, or track the underlying host-resident state incompatibility.

Nits

• None.

Developmental progress

• Good progress since predecessor ggml-meta : split a host-resident KV cache by head #57 was split: scheduler-copy name truncation is addressed and exercised with eight devices (ggml/src/ggml-backend.cpp:1037-1047, ci/run.sh:315-316); V uses its own head size (src/llama-model.cpp:780-789); strided cache transfers are handled (ggml/src/ggml-backend-meta.cpp:1458-1484, ggml/src/ggml-backend-meta.cpp:1616-1639); and recurrent corruption is avoided (src/llama-context.cpp:148-153).

Reviewed head: 9db019e