sched: pipeline the delivery of a host-resident KV cache - #39

Open

Piggidragon wants to merge 37 commits into

GenerelSchwerz:llama/devfrom

Piggidragon:kv/pipelined-transport

Commits on Sep 8, 2026

sched: pipeline the delivery of a host-resident KV cache
Piggidragon
committed
d14138b
docs: record what tensor parallelism still needs, and fix the repro scripts
Piggidragon
committed
fbe9cdd
sched: report where the pipelined token goes, and fix the budget check
Piggidragon
committed
22b865a
sched : name what the ordered path still copies, and why it costs what it does
Piggidragon
committed
499c10a
repro : make the exactness tasks independent of each other
Piggidragon
committed
fac757c
sched: fix pipelined transport fallback paths
Piggidragon
committed
cc1afb7
sched: fix pipelined transport review issues
Piggidragon
committed
2ca196e
sched: allocate transport ring entries the way the backend would
Piggidragon
committed
6e2fe44
sched: never let the transport ring starve the graph
Piggidragon
committed
df5f1f7
docs: re-measure the transport gates on the current head
Piggidragon
committed
2811703
docs: say where the transport stops paying, and why the budget is 128 MiB
Piggidragon
committed
9a12d97
sched: deliver a multi-stream KV window one stream at a time
Piggidragon
committed
34851f6
docs: measure the quant invariance, and what parallel sequences cost
Piggidragon
committed
9b92b9c
sched: take a staged window's stream span from the whole tensor
Piggidragon
committed
f5971b6
sched: wait for the previous graph before staging a window again
Piggidragon
committed
9bcfcac
sched: wait for the consumer before freeing the transport ring
Piggidragon
committed
9b0d2f2
llama: note that a cache sharing cells keeps no stable prefix
Piggidragon
committed
f619d58
llama-bench: stop on an out-of-range -kvpd or -kvpb
Piggidragon
committed
b87c74b
sched: wait on the slot release events, not on the consumer backend
Piggidragon
committed
499d2b9
sched: deliver a staged window with ggml_backend_tensor_set_2d_async
Piggidragon
committed
763f706
ggml: say what a stable prefix covers
Piggidragon
committed
feeb332
repro: gate the multi-stream delivery on concurrent output
Piggidragon
committed
950e7fb
docs, tests: unwrap the hard-wrapped comments
Piggidragon
committed
7edce45
docs: re-measure on this head, and say which head each number is from
Piggidragon
committed
2ce9c7b
sched: release an idle transport ring, and grow a slot in powers of two
Piggidragon
committed
8eaa67c
arg, llama : bound --kv-pipeline-budget
Piggidragon
committed
c7b2e8e
llama : do not clear the KV buffers under a running decode
Piggidragon
committed
56e8076
llama : wait for the decode before clearing the memory buffers
Piggidragon
committed
7b9fdb2
sched : fix the review issues of the pipelined transport
Piggidragon
committed
ed6bc86
sched, llama: refinements to pipelined transport handling

Piggidragon
and
claude
committed
3f41a75
sched, llama : fix the second review of the pipelined transport
Piggidragon
committed
bbdd08f
sched, llama : fix the third review of the pipelined transport
Piggidragon
committed
b0d9c49
sched, llama : pack the multi-stream transport ring, make pipelining opt-in
Piggidragon
committed
f79205b
sched, tests : reconcile the ring with the merged ordered range copy
Piggidragon
committed
d359a23

Commits on Sep 9, 2026

docs : re-measure the parallel and packing tables on the rebased base
Piggidragon
committed
da23680

Commits on Sep 10, 2026

sched, llama : fix the fourth review of the pipelined transport
Piggidragon
committed
47347c0

Commits on Sep 13, 2026

Merge llama/dev into kv/pipelined-transport
Piggidragon
committed
1d10dc0