llama-bench placement flags, two host-KV fixes, and H1/H2/H3/H5/H13 measured

[Piggidragon]

Overview

Two fixes, one tooling addition, and the measured verdicts on five hypotheses from
docs/dense-kv-hypotheses.md. Draft, because the last section is a behaviour question that
is not mine to answer.

Fix: quantized KV with -nkvo aborted on many models

Not --split-mode tensor and not what #57 fixes; it reproduces on a single device with no
split mode.

stage_store_rows() casts the current rows to the cache type before storing them into a
host-resident cache. That cast is a GGML_OP_CPY whose output is a quantized tensor in the
compute buffer, so get_alloc_size sizes it. CUDA pads a quantized tensor whose row is not a
multiple of MATRIX_ROW_PADDING (512), and GGML_OP_CPY was missing from
ggml_backend_op_alloc_size_may_expand(), so the allocation assert fired at context creation.

model	n_embd_k_gqa	-nkvo with q8_0
Qwen3.8-27B	1024	worked
Muse-Glimmer-30B	256	aborted

f16 unaffected, every quantized pair aborted. One line, following the instruction in
[TAG_ALLOC_SIZE_EXPAND]. The allocation only grows, so nothing changes numerically:
test-backend-ops -o CPY 246/246, Qwen3.8 tg unchanged at 32.9 t/s, and Muse-Glimmer greedy
output byte-identical between a device- and a host-resident cache.

Not behind a flag on purpose: the prior behaviour is an abort, so there is no configuration
where the old path was preferable.

Fix: H13, a context shift roped the whole allocated cache

build_graph_shift() viewed get_size()*n_stream instead of the live cells. On a
host-resident cache the ROPE runs on the CPU backend over the entire allocation, so the cost
was set by -c rather than by the context in use.

Muse-Glimmer-30B, host-resident cache, 256 live cells, cost above a plain decode:

-c	plain decode	before	after
8192	49.3 ms	66.8 ms	19.6 ms
32768	49.1 ms	197.8 ms	16.1 ms
131072	49.3 ms	687.2 ms	16.6 ms

At -c 131072 one shift cost 14x a plain decode for 256 live tokens, and the server takes a
shift on every cache-reuse seq_add, not only on overflow. The remaining cost no longer grows
with -c.

The doc's stated risk turned out to be void, which is why this is not behind a flag either: a
cell at or past used_max_p1() is empty, and set_input_k_shift() already wrote a shift of 0
for an empty cell, so the old graph was applying an identity rotation to the tail. Skipping it
is exactly equivalent. Logits after a shift are bit-identical at -c 8192, 32768 and 131072.
Several streams keep the old path, because their per-stream extents are not contiguous and one
rope view cannot express them.

llama-bench: -kvgl, -kvcp, -rso

llama-bench accepted -nkvo and -ub and none of the other placement flags, so the
host-resident KV configuration space could not be swept from it at all. This adds
-kvgl / --kv-gpu-layers, -kvcp / --kv-cpu-pinned and -rso /
--recurrent-state-offload, following the existing -nkvo pattern exactly: they sweep with
, lists, they show up as result columns (kvgl, kvcp, rso) only when set, and they map
straight onto the matching llama_context_params fields. No behaviour changes anywhere else.

-kvgl was the flag the doc asked for. -rso turned out to be needed first, for the reason
in the last section.

Machine baseline

RTX 4070, gen4 x16, 16 host threads, CUDA 13.3.

path	GB/s
host to device, pinned, 256 MiB	24.61
host to device, pageable	12.63
host DRAM read, 1 thread	16.31
host DRAM read, 8 threads	42.73

The doc assumed host DRAM is "several times" PCIe bandwidth. It is 1.74x here, and that
turns out to decide two of the four results.

Model under test for everything below: Qwen3.8-27B-UD-IQ2_M, 9.6 GiB, all weights device
resident, 16 owned attention layers, q8_0 K and V, so 34 KiB of window per context token per
decode step.

H5, host-side attention: rejected

tg64 at depth, -ub 512, 3 reps:

depth	device KV	host KV, device attn	host KV, host attn
2048	36.84	32.93	29.78
4096	36.43	29.92	25.32
8192	35.57	25.14	19.41
16384	33.83	18.98	13.30
32768	30.69	12.71	8.13

There is no crossover. Shipping the window wins at every context and the margin grows from
10.6% to 56.3%.

Fitting ms/token against depth separates the fixed cost from the per-context-token cost. All
three arms agree on the fixed cost to within 0.8 ms, which is what makes the slopes worth
comparing:

arm	fixed	ms per 1k of context	implied bandwidth
device KV	26.78 ms	0.177	197 GB/s of VRAM
host KV, device attn	27.15 ms	1.573	22.1 GB/s over the link
host KV, host attn	27.61 ms	2.912	12.0 GB/s of host DRAM

The delivery runs at 90% of the pinned link. The host attention path runs at 28% of host DRAM
read, and it is flat under -t 4,8,16 (13.32, 13.28, 13.31 t/s at depth 16384), so it is not
thread-starved; it is running at about single-thread DRAM speed whatever it is given.

Consequences, both recorded in the doc: H14 is parked, because it would replace a
22.1 GB/s delivery with a 12.0 GB/s host read, and even a perfect CPU kernel at the full
42.8 GB/s ceiling only buys 1.9x on a machine where the raw ratio is 1.74x. H10 is
parked, because there is no crossover for a per-layer policy to find.

Fix: llama-bench reserved logits for a whole batch

llama_context sets n_outputs_max to n_batch unless a tool asks for less, so the prompt
reserve allocated logits for every token of a ubatch. On this 248320 token vocabulary that is
0.947 MiB per ubatch token. llama-server sets n_outputs_max to its slot count and never
paid it; perplexity sets it to n_batch because it genuinely reads every output;
llama-bench said nothing and so reserved a buffer it never fills, because it decodes with
llama_batch_get_one, which marks only the last token of the batch.

CUDA0 compute buffer, Qwen3.8-27B, 4096 token prompt:

ub	before	after
512	505 MiB	132 MiB
2048	2054 MiB	568 MiB

-ub 4096 now fits where it used to fail to allocate. Throughput is unchanged, since the
reserve only sized a buffer that was never filled: pp at -ub 512 with a host-resident cache
is 953.57 t/s against 953.30 before.

H1, prefill transport against ubatch: rejected, on the size of the prize

32768 token prompt, -b 8192, -rso 1, 2 reps, measured after the fix above:

ub	host KV	device KV	transport penalty	pp buffer
512	953.57	994.67	4.1%	286 MiB
2048	980.75	996.42	1.6%	568 MiB
4096	975.66	985.81	1.0%	1056 MiB
8192	out of memory	out of memory	-	1952 MiB needed

The mechanism is real: raising ub divides the prompt transport, and the host-versus-device
penalty falls from 4.1% to 1.0%. What kills it is that the whole penalty is 4.1% at 32k, so
1/ubatch divides a term that is already small, and the gain plateaus - 512 -> 2048 is
+2.9%, 2048 -> 4096 is -0.5%. Prefill is compute-bound long before transport matters.

This corrects an earlier version of this PR. It claimed ub 512 -> 2048 cost 1549 MiB of
compute buffer, more than the whole KV cache, and that the buffer grew at 15x the
n_ff * ub * 4 term. That was the llama-bench artifact above, not a real cost. After the fix
the growth is about 0.28 MiB per ubatch token, a small multiple of the FFN intermediate, so
the doc's original mechanism was right and my correction to it was wrong. H1's verdict is
unchanged; its second reason is not.

H7, H9 and H12: sized out, not implemented

A host-resident decode graph has 34 splits against a device-resident one's 2, two per owned
attention layer - exactly the structure these three items target. The H5 fits price it: the
intercepts are 26.78 ms device against 27.15 ms host, so all 32 extra splits and the staging
round trips together cost 0.37 ms, about 11.6 us each. That is 1.2% of a decode at 2k and
0.47% at 32k, shrinking with context. H12's padded tail is another 0.20 ms.

Three changes the doc itself calls risky, for under 1.3% combined, none of which touch the
delivered bytes that H5 shows are the entire cost. Not worth it.

H2, unified cache across slots: mechanism confirmed, claim overstated

llama-batched-bench -kvu -pps, shared 8192-token prompt, 64 tokens per slot, aggregate tg:

slots	host KV	device KV	host / device	ms per decode graph, host - device
1	24.98	35.28	0.708	11.69
2	44.21	59.28	0.746	11.50
4	72.31	89.82	0.805	10.79
8	99.01	113.90	0.869	10.56

The last column is the hypothesis and it holds exactly: the delivery costs about 11 ms per
graph regardless of how many slots feed it, and at n_kv = 8256 the cost model predicts
11.68 ms against 11.69 measured.

What does not hold is "close to linearly": aggregate rises 3.96x for 8x the slots, because
compute grows too. The useful form of the result is the third column. The host-KV penalty
falls from 29% at one slot to 13% at eight, and host-resident KV scales better with slots
than a device-resident cache does (3.96x against 3.23x). A multi-slot server is where
host-resident KV is least bad, and the single-stream numbers are its worst case. That is the
opposite of the usual intuition and is not in any current default or doc.

H3, speculative decoding under host KV: confirmed, and the biggest effect measured

27312-token prompt, 256 generated tokens, greedy at --temp 0 with a fixed seed, DFlash2
sidecar as the draft, --spec-draft-kv-gpu-layers 99 in the host arm. Speedup is against
that arm's own n_max = 0:

--spec-draft-n-max	device KV	speedup	host KV	speedup	ratio
0	21.9	1.00	4.8	1.00	-
2	32.0	1.46	9.6	2.00	1.37
4	32.4	1.48	11.0	2.29	1.55
8	27.9	1.27	11.2	2.33	1.83

The last column is the hypothesis. Speculation is worth 1.37x to 1.83x more under a
host-resident cache, and the gap widens with draft length. The curve shapes differ for the
reason the mechanism gives: the device arm peaks at 4 and regresses at 8 because the draft
starts costing more than the verification saves, while the host arm keeps climbing because
the single delivery it amortizes dominates everything else.

Acceptance per draft is identical between arms by construction, same model, prompt, seed and
temperature, rather than by measurement.

One caveat on the absolute numbers. The DFlash sidecar shares output.weight with the target
and has to sit on the same device stack, and target plus draft do not fit on the 4070 alone,
so this ran with the model split across both cards. The second card is gen3 x4, about
3.5 GB/s, which is why the host baseline is 4.8 t/s here against 12.7 t/s single-card in H5.
Both arms share the topology so the ratio holds, but the left columns are a slow-second-link
case. Incidentally that is its own finding: a host-resident cache behind gen3 x4 is about 7x
worse than behind gen4 x16.

Muse-Glimmer, now that it runs

tg64 at depth, across a 4070 and a 3060:

depth	device KV	host KV	penalty
2048	19.67	15.73	20%
8192	19.23	13.69	29%
32768	17.71	9.18	48%

Read the host column as a slow-link case: the 3060 is gen3 x4, about 3.5 GB/s against the
4070's 24.6, and it holds half the layers.

This also corrects the cost model in the doc. Density is the wrong axis; the sliding window
dominates it. Muse-Glimmer is fully dense at 52 attention layers, but 39 are SWA-2048 and
n_embd_k_gqa is 256, so beyond the window only 13 layers grow: 7072 B per context token
against the hybrid reference's 34816 B. The fully dense model is 4.9x cheaper per context
token, not "roughly 4x worse".

What this adds up to

The actionable result is a configuration, not a code change. Host-resident KV is far less bad
than the numbers usually quoted for it, but the wins are in H2 and H3, not in moving
attention to the host (H5) or in larger ubatches (H1). Run it with several slots, a unified
cache and a draft model. Nothing in the defaults or documentation currently says so, which is
what H11 is for and what I would do next.

The one that is not in this diff: -nkvo also moves the recurrent state

llama-model.cpp:2215 builds the recurrent memory with
cparams.offload_kqv || cparams.recurrent_state_offload. On a hybrid model, -nkvo therefore
moves the gated-delta-net state to host memory along with the attention KV, and every
recurrent layer's state ops go to the CPU backend.

At 2k of context, -nkvo --kv-cpu-pinned:

recurrent state	tg t/s	ms/token
host, the current default	15.99	62.54
device, --recurrent-state-offload	32.93	30.37

32.2 ms per token, at every context length, and llama_memory_recurrent reports what it buys
back as 149.62 MiB for one sequence, independent of -c. On the same run the attention KV is
8.5 MiB per 256 cells, so 149.62 MiB is about 4500 cells of cache.

I think the default is wrong for one sequence and I am not sure it is wrong at high
-np, where the state scales with n_seq_max and the argument gets weaker. So this PR only
documents it and adds the flag to llama-bench. Changing the default is a user-visible
behaviour change and it is your call, not mine.

The immediate practical consequence either way: any -nkvo number ever taken on a hybrid
model without --recurrent-state-offload is off by about 2x, including anything in this
repository's history.

Additional information

Everything above is in docs/dense-kv-hypotheses.md alongside the hypotheses it settles, so
the file stays the single place the backlog lives. Raw CSVs are not committed; the exact
commands are in each measured block.

Not measured yet, and next in the doc's own order: H11 (--fit for dense models), which is
what turns the H2 and H3 results into something a user gets without reading this file.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES. Claude Opus 5 ran the benchmark sweeps, wrote the llama-bench
  flag plumbing against the existing -nkvo pattern, and drafted this description and the
  measured blocks in the doc. The hypotheses, the cost model they are tested against, and the
  decision not to touch the recurrent_state_offload default are mine.