sched: pipeline the delivery of a host-resident KV cache - #39
sched: pipeline the delivery of a host-resident KV cache#39Piggidragon wants to merge 14 commits into
Conversation
|
Review of head 175236b: changes requested before merge. The CUDA performance result looks credible, and the declining relative gain is consistent with the overlap ceiling: ordered time is approximately copy + compute, while pipelined time approaches max(copy, compute) plus residual overhead. As context grows, KV traffic grows while the amount of compute available to hide it does not grow at the same rate. NCU is not needed to explain that curve. A short nsys trace is useful after the correctness fixes, but it is not a substitute for them. Blocking defects1. The incomplete meta/tensor-parallel path is now reachableThe generic eligibility check selects any non-CPU backend with async tensor set and event interfaces: llama.cpp/ggml/src/ggml-backend.cpp Lines 2724 to 2755 in 175236b This PR adds those interfaces to the meta backend: llama.cpp/ggml/src/ggml-backend-meta.cpp Lines 180 to 267 in 175236b llama.cpp/ggml/src/ggml-backend-meta.cpp Lines 2530 to 2546 in 175236b A normal partial-prefix delivery sends the changing tail with a nonzero destination offset: llama.cpp/ggml/src/ggml-backend.cpp Lines 2341 to 2353 in 175236b Meta hard-asserts that the offset is zero and also requires whole-chunk granularity: llama.cpp/ggml/src/ggml-backend-meta.cpp Lines 1912 to 1949 in 175236b The design document says the strided delivery is still missing and the meta work has not been run: llama.cpp/docs/kv-transport-pipelining.md Lines 390 to 438 in 175236b This means the documentation claim that tensor parallelism remains ordered does not match the code. A host-KV tensor-parallel run can enter the pipeline and assert. Fix: explicitly exclude GGML_BACKEND_DEVICE_TYPE_META for now. Prefer moving the untested meta event and transfer groundwork into the later change that implements and validates the strided head-split write. Removing only the offset assertion is not sufficient. 2. Partial per-backend setup failure leaves stale plan membershipTransfer backend or event creation can fail here: llama.cpp/ggml/src/ggml-backend.cpp Lines 1794 to 1835 in 175236b The caller only sets r->n_staged to zero: llama.cpp/ggml/src/ggml-backend.cpp Lines 2014 to 2018 in 175236b It does not clear split_order and input_staged for that backend, unlike the budget, headroom, and allocation failure paths. If another backend succeeds, global staging remains enabled and address assignment reaches the failed backend: llama.cpp/ggml/src/ggml-backend.cpp Lines 2119 to 2143 in 175236b That path calls ggml_backend_buffer_get_base with a null ring buffer. Fix: factor one backend-decline helper that releases the ring and clears n_staged, every affected split_order entry, and every corresponding input_staged entry. Use it for every failure path. Add failure injection for one backend failing while another succeeds. 3. Crossing the budget permanently disables the featureRing eligibility rejects a backend once over_budget is set: llama.cpp/ggml/src/ggml-backend.cpp Lines 1664 to 1679 in 175236b The budget and headroom paths set that flag permanently: llama.cpp/ggml/src/ggml-backend.cpp Lines 1990 to 2011 in 175236b llama.cpp/ggml/src/ggml-backend.cpp Lines 2032 to 2051 in 175236b The assumption that a context only grows is not valid for a long-running server. The KV view size is derived from the highest currently used cell and can shrink after a sequence is removed: llama.cpp/src/llama-kv-cache.cpp Lines 1334 to 1347 in 175236b Once over_budget is set, the initial eligibility check prevents later small graphs from even reaching the calculation that could reconsider the ring. One long request can therefore force all later short requests on the same context onto the ordered path. Fix: re-evaluate the budget for each new graph. Keep warning suppression separate from eligibility. If allocation churn is a concern, use hysteresis rather than a permanent latch. 4. The public setters can invalidate an allocated graphDepth is documented as pre-allocation, but the restriction is not enforced. Budget has no timing restriction: llama.cpp/ggml/include/ggml-backend.h Lines 339 to 353 in 175236b Both setters can free ring storage: llama.cpp/ggml/src/ggml-backend.cpp Lines 2691 to 2717 in 175236b llama.cpp/ggml/src/ggml-backend.cpp Lines 2769 to 2785 in 175236b Ring freeing does not invalidate the staged input-copy data and buffer fields: llama.cpp/ggml/src/ggml-backend.cpp Lines 1746 to 1768 in 175236b Calling either setter after graph allocation can therefore leave the graph pointing into freed ring storage. Fix: make the configuration immutable once allocation starts and enforce that state. Passing transport configuration at scheduler construction would be safer than mutable exported setters. At minimum, budget needs the same enforced precondition as depth. 5. The claimed ggml_tensor size preservation is false on some 32-bit ABIsThe parent reserves eight trailing bytes: Lines 698 to 705 in 71b2b1b The PR replaces that with size_t while claiming sizeof(ggml_tensor) does not change: Lines 698 to 723 in 175236b That is true on LP64, but on an ILP32 ABI such as i386, size_t is four bytes and the structure can shrink from 256 to 252 bytes. Fix: preserve an eight-byte storage slot, for example with a fixed-size union or representation, and add layout assertions for representative 32-bit and 64-bit targets. The memory-cap arithmetic also needs overflow checks. On 32-bit, sufficiently large MiB values can wrap to zero, which means uncapped: llama.cpp/src/llama-context.cpp Lines 785 to 793 in 175236b llama.cpp/ggml/src/ggml-backend.cpp Lines 2769 to 2775 in 175236b Use checked parsing, checked MiB conversion, and checked ring addition/multiplication. Major merge risks6. Backend eligibility assumes stronger event semantics than the API providesDepth 1 is enabled by default for a host-resident cache: Lines 581 to 588 in 175236b The eligibility test uses function-pointer presence as proof of nonblocking stream-to-stream ordering. That is not true for all selected backends. SYCL blocks the host in event->wait(): llama.cpp/ggml/src/ggml-sycl/ggml-sycl.cpp Lines 5815 to 5826 in 175236b WebGPU event wait calls host synchronization: llama.cpp/ggml/src/ggml-webgpu/ggml-webgpu.cpp Lines 3560 to 3572 in 175236b This contradicts the design requirement that recycling not stop the host and can produce serialization or regressions. Fix: add a capability that specifically guarantees nonblocking event waits, or keep the feature opt-in until each backend is validated. For the current fork, a temporary CUDA-only gate is safer than generic pointer-based eligibility. 7. The late-tail path does not order against a general async producerThe original path either uses a backend-aware async copy or synchronizes the producer and consumer before a blocking copy: llama.cpp/ggml/src/ggml-backend.cpp Lines 2472 to 2495 in 175236b The staged late-tail path directly calls set_tensor_async from input->data without coordinating with input_backend: llama.cpp/ggml/src/ggml-backend.cpp Lines 2335 to 2354 in 175236b The current llama KV producer is CPU-oriented, but the new ggml API is generic. An asynchronous producer can still be writing the tail when the destination starts reading it. Fix: either restrict staging to persistent CPU-produced tensors and document that restriction, or preserve producer ordering for the late region. 8. Inputs with no stable prefix are still redirected into the ringThe public documentation says only inputs carrying a stable prefix are eligible: llama.cpp/ggml/include/ggml-backend.h Lines 326 to 345 in 175236b Actual ring membership deliberately ignores the prefix: llama.cpp/ggml/src/ggml-backend.cpp Lines 1704 to 1735 in 175236b This includes transposed V, which is explicitly assigned a zero prefix: llama.cpp/src/llama-kv-cache.cpp Lines 1617 to 1626 in 175236b Those inputs consume ring capacity and use the new event path while contributing no overlap. This can cause transposed-V configurations to cross the budget earlier despite the documentation saying they stay untouched. Fix: use a static transport-candidate annotation for reserve-time membership and a separate per-evaluation stable byte count. Validation and instrumentationThere are no automated test changes. The exactness harness prints hashes but does not compare them: llama.cpp/docs/repro/r4-kv-pipeline-exact.sh Lines 17 to 34 in 175236b llama.cpp/docs/repro/r4-kv-pipeline-exact.py Lines 61 to 87 in 175236b It only fails on a request error or detected cache reuse. It is a manual measurement harness, not an exactness gate. Reuse the existing test infrastructure rather than adding a new test file. At minimum cover:
Some profiling claims are also not reproducible from this head. The document says debug level 3 showed individual copy costs: llama.cpp/docs/kv-transport-pipelining.md Lines 294 to 308 in 175236b The implementation logs only tensor name and size: llama.cpp/ggml/src/ggml-backend.cpp Lines 2368 to 2377 in 175236b Also, n_stop_recycle increments when a stream wait is enqueued, not when look-ahead actually stops: llama.cpp/ggml/src/ggml-backend.cpp Lines 2173 to 2180 in 175236b The public stats getter exposes only deliveries and early/late bytes, not all the counters described by the document: llama.cpp/ggml/src/ggml-backend.cpp Lines 2788 to 2794 in 175236b Restore the instrumentation that produced the documented numbers or narrow the claims. Commit cleanupThe first two commits use Co-Authored-By for Claude. Repository policy requires Assisted-by for agent assistance: Later commits use the required trailer. Rewrite the first two before merge. Recommended development order
|
With --no-kv-offload the attention history lives in host RAM and reaches the accelerator on every decode token. The scheduler issued that transfer on the consumer's own stream immediately before the kernels that read it, so a token cost copy + compute in series. The bytes and the attention operations are unchanged; only the point at which the transfer is issued moves. Greedy output is byte-identical to the ordered path -- verified against a build without these changes, at every look-ahead tested, single GPU and layer-split across two. Three pieces, each load-bearing: - ggml_tensor::stable_prefix records how many leading bytes of a tensor's storage the graph about to run will not write. The KV window is not stable for a whole graph -- a CPU split writes this ubatch's rows into it between one layer's attention and the next -- but everything below the lowest written row is, and at decode depth that is essentially all of it. llama_kv_cache sets it from apply_ubatch(), before the graph is built and allocated, so the plan and the deliveries are decided against the same write position even when the graph is reused; build_graph_shift() clears it. - A staging ring the graph allocator cannot reach. ggml-alloc may recycle a graph-owned input copy after its last graph-level consumer while a look-ahead transfer is still in flight. The scheduler allocates the ring itself and points the staged copies at it before allocation; a ready/release event pair per slot carries the handover in each direction. Every eligible accelerator gets its own ring, cursor and budget, so a layer-split model pipelines on each device and a device with no room falls back alone. - A look-ahead that stays clear of the ring's tail. A delivery L splits ahead recycles the slot of the split L - n_slots back, so n_slots == L + 1 recycles the split just enqueued and still running. The ring keeps two slots of margin, deliveries are issued after a split is enqueued rather than before, and slot recycling is ordered stream to stream rather than through the host. Each of those three alone costs the entire gain while still producing correct output. --kv-pipeline-depth N, default 1, 0 restores the ordered path exactly. It only engages where a host-resident cache produces the deliveries. Because a host-resident cache exists to keep device memory free, the staging is capped outright by --kv-pipeline-budget (default 128 MiB per device) rather than by a fraction of what happens to be free. A ring is (N + 2) slots of one attention layer's K and V over the whole context, so it grows with the context: 27 MiB at 4k, 213 MiB at 32k, 1.7 GiB at 256k. Past the cap the scheduler declines and keeps the ordered path, and declining costs nothing -- the check runs before anything is allocated, the decision is latched because a context only grows, and the transfer backend is created lazily and released with the ring. Single GPU (RTX 4070, Qwen3.8-27B-UD-IQ2_M, -nkvo --kv-cpu-pinned, q8_0 K/V), A/B/A/B with reversed arm order: depth ordered pipelined gain 4,096 31.7324, 31.7363 37.0889, 37.0741 +16.9% 16,384 19.6765, 19.6854 31.5352, 31.5807 +60.4% 32,768 13.0264, 13.0254 15.5325, 15.5329 +19.3% (needs a raised budget) Server decode behind an 18,422-token prompt: 18.468 -> 30.685 t/s, +66.2%. Two GPUs (RTX 4070 + RTX 3060, Qwen3.8-27B-UD-Q5_K_M, -sm layer), both rings engaged: 13.06 -> 18.05 t/s at 4,096 and 6.86 -> 9.75 t/s at 16,384. The gain narrows with depth because compute is a shrinking share of the token, so there is less to hide the copy behind. That is arithmetic, not an implementation limit, and more look-ahead makes it worse rather than better. Tensor parallelism keeps the ordered path: the scheduler sees one meta backend there and the ring is a byte arena, while a meta buffer places tensors as per-device slices rather than at offsets. It declines rather than staging into something it cannot address. docs/kv-transport-pipelining.md carries the design, the numbers and the limits; docs/repro/ carries the scripts that produced them. Assisted-by: Claude Opus 5
…cripts llama-bench does not expose --kv-cpu-pinned or --recurrent-state-offload the way llama-server does, so two of the reproduction scripts passed flags the binary rejects. They now probe --help and pass only what it takes. The feature doc gains what is actually left: why -sm tensor keeps the ordered path (no events in the meta layer, and a ring that is a byte arena while a meta buffer places tensors as per-device slices), the correctness problem underneath it that is not this feature's, and the rest of the open list -- the transient device-memory peak, the --kv-gpu-layers comparison, and the exactness harness's dependence on baseline determinism it does not have. The decline for a backend that cannot record events now says so by name rather than falling into the generic "no backend supports" line, because the backend it catches is the meta backend and the next person to look will want to know that. Assisted-by: Claude Opus 5
The host-time breakdown the feature doc describes had no code behind it: the counters existed but nothing accumulated or printed them. GGML_SCHED_TRANSPORT_DEBUG=2 now reports the split loop as a mean over each 128 graphs, with the bytes the ordered path still moves and why the look-ahead stopped; =3 names the tensors that are still on it. That is what found the rest: 40 blocking copies a token moving 0.4 MiB, 32 of them the device-to-host KV store. The budget warning now reports what the ring costs at the full context next to what it costs now, so --kv-pipeline-budget can be sized against the number that matters. It is still applied per graph: enforcing the projection would refuse the ring for every large -c even when the window never gets near it. llama-bench gains -kvcp and -rso. Without them a host-resident run measures something else entirely -- 9.02 against 19.43 t/s ordered at 16,384 -- and the repro scripts had been silently dropping both since llama-bench lost them. The exactness harness gives every task a nonce derived from its own name and length, so no two share a prefix the server can restore, and fails a task whose prompt_n says one was reused anyway. Assisted-by: Claude Opus 5
…t it does GGML_SCHED_TRANSPORT_DEBUG=3 now reports what each remaining blocking copy cost, not just its name. It turns out one of them is almost all of it: attn_inp_k_rot, 256 KiB, 18 us on the ordered path and 3.4 ms behind one split of look-ahead. That is the copy engine, not latency. A blocking copy waits for the deliveries already queued on it, and two staged splits at 22.0 GB/s is 3.6 ms. Issuing the delivery in pieces does not help, the engine is FIFO across streams. Putting the copy on the consumer's stream so the host never blocks moves the time into the consumer wait and leaves throughput alone. The doc records both, so the next person does not spend the afternoon on it again. Assisted-by: Claude Opus 5
records@18432 was giving different answers across otherwise identical N = 0 runs, which made it useless as a gate and looked like the pipeline breaking exactness. It is not the task: asked on its own with the prompt cache off it returns the same hash three times running, at -c 32768 and at -c 65536. It is the harness. All eight tasks share one server with prompt caching on, and records@18432 is about 29.6k tokens with a task of about the same size ahead of it, so the two do not both fit in a 32,768 cache and placement depended on what was still resident. The nonce stops a prefix being restored, it does not stop the pressure. cache_prompt=false does. Two independent N = 0 passes now agree on all eight tasks, and N = 0, N = 1 and N = 4 agree on all eight. Assisted-by: Claude Opus 5
Restrict staging to annotated CUDA inputs, make backend decline complete, re-evaluate budgets, freeze scheduler configuration, and preserve tensor layout. Add regressions for prefix changes and fallback behavior. Assisted-by: OpenAI Codex
175236b to
2f70cbe
Compare
|
Addressed on head 2f70cbe.
The existing test-alloc target now covers partial, zero, and changing prefixes on a reused graph; depth 0; budget recovery; post-allocation setters; partial setup failure; meta exclusion; annotation requirement; and non-CUDA exclusion. Local validation on the final commit:
The branch is rebased onto llama/dev at 01b141f. The obsolete meta commit was removed and the first two commit trailers now use Assisted-by. CI is queued on the updated head. I could not capture the requested nsys traces locally because this environment has no working NVIDIA driver. The CUDA CI job is queued, but it does not replace the runtime trace. |
Review findings
🤖 Generated with Claude Code |
Assisted-by: OpenAI Codex
Assisted-by: OpenAI Codex
|
AI-assisted review of current head
For an eventual pristine llama.cpp port, this also needs to be stacked after the fork's host-KV-storage/accelerator-attention separation. Upstream currently moves the entire attention region to CPU when KV offload is disabled, so this scheduler patch alone has no useful host-to-accelerator path. Local checks passed: |
The ring laid its entries out with ggml_nbytes() and bound them by writing data and buffer directly. A buffer type may ask for more than ggml_nbytes() for a tensor -- CUDA does for a quantized one, and MMQ clears that padding -- so an entry could reach into the next one. Entries are now sized with ggml_backend_buft_get_alloc_size() and bound with ggml_backend_tensor_alloc(), which also gives them the buffer's own initialization and its bounds check. test-alloc gets a dummy buffer type whose get_alloc_size exceeds ggml_nbytes, and a two-entry ring test that checks the entries stay inside the ring and out of each other, that every byte of an entry is delivered once from the matching source offset, and that nothing waits on an event before it is recorded. llama-bench takes -kvpb/--kv-pipeline-budget and reports it. The repro scripts pass 512 and now fail closed: they refuse a build without -kvcp, -rso or -kvpb instead of dropping the option, and every arm propagates its status. The llama-bench table in the doc was measured before the budget existed, so it says so, and the 32,768 row is marked as needing a re-measurement. Assisted-by: Claude Opus 5
github-actions
Bot
added
documentation
From the moe-cache-drafting branch: --moe-expert-cache-size, --moe-expert-cache-l2-pinned-mb, --experimental-logs, the automatic grouped decode / prefetch / bias residency behaviour, and the layer-split-only and speculative interactions. Marked as coming from an unmerged branch, like the PR GenerelSchwerz#57 and GenerelSchwerz#39 material. Assisted-by: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_013Bs926e6SdsxnkrqqKq5HP
|
Review of head Blocking1. The optional ring can consume memory required by the graphThe transport planner allocates and retains the ring before graph allocation: llama.cpp/ggml/src/ggml-backend.cpp Lines 2223 to 2258 in 0933834 Only afterward does graph allocation fall back to llama.cpp/ggml/src/ggml-backend.cpp Lines 2393 to 2433 in 0933834 This can turn a graph that fits without pipelining into
Configuration is then locked here, so the caller cannot disable the ring and retry on the same scheduler: llama.cpp/ggml/src/ggml-backend.cpp Lines 3093 to 3107 in 0933834 The graph-allocation failure path should discard the current graph's optional ring bindings and retry reservation/allocation once on the ordered path. 2. The previously reported zero-input defect remains unresolvedWhen a split exists but the total number of split inputs is zero, llama.cpp/ggml/src/ggml-backend.cpp Lines 2036 to 2054 in 0933834 The new llama.cpp/tests/test-alloc.cpp Lines 1468 to 1478 in 0933834 The test passes in Release, but the exact head fails under UBSan: This is follow-up to the existing finding, not independent duplicate feedback: Handle Will slow the review1. The PR description materially contradicts the current headThe description still says that three meta-backend prerequisites ship here and that budget decline is latched. The current implementation explicitly excludes meta and requires the exact llama.cpp/ggml/src/ggml-backend.cpp Lines 2925 to 2955 in 0933834 The current design document instead says smaller later graphs are reconsidered: llama.cpp/docs/kv-transport-pipelining.md Lines 257 to 274 in 0933834 The description's measurement presentation also conflicts with the document, which says the 32k row predates the current budget behavior and needs remeasurement on this head: llama.cpp/docs/kv-transport-pipelining.md Lines 135 to 139 in 0933834 Update the description so reviewers are evaluating the implementation and validation that actually remain in this revision. 2. Changed prose and public comments extensively violate the no-hard-wrap ruleThe repository rule is explicit here: Lines 79 to 90 in 0933834 Examples include:
This is widespread rather than an isolated nit. Existing validation concernI am not duplicating the already-raised request for real CUDA data and ordering validation. It remains unresolved on this exact head: the current CUDA CI job was canceled without running, and commit Canceled job: https://github.com/GenerelSchwerz/llama.cpp/actions/runs/33486643492/job/99788163320 Existing owner feedback: Upstream comparisonNo equivalent implementation exists in current pristine upstream. The closest relevant work supports parts of the approach but does not remove the blockers above:
The scope gate otherwise passes. Fork PR #31 documented R4 and its design decision before implementation, and neither upstream ggml-org#21067 nor ggml-org#27311 is a direct duplicate. Upstream issue ggml-org#27757 concerns the separate tensor-parallel host-KV correctness defect. Verification
No additional security, lifetime, event-order, public-API, or backend-dispatch finding survived verification. |
The rings are laid out and allocated before the graph is, so a device that can hold the graph alone but not the graph next to a ring turned into GGML_STATUS_ALLOC_FAILED. The configuration is locked by then, so the caller could not turn the ring off and retry either. When graph reservation fails the rings are now released and the reservation is retried once on the ordered path, and that scheduler keeps the ordered path from then on. A plan over a split list with no inputs left input_staged unallocated and passed it to memset, which UBSan reports even at size 0. Such a plan stages nothing, so it now returns after putting every split back on the ordered path. test-alloc gets a device capacity on the dummy backend and a test that sizes it to hold the graph or the ring but not both. The prose and public comments this branch added were hard-wrapped to a fixed column, against the repository rule. They are unwrapped, one sentence per line. Assisted-by: Claude Opus 5
The llama-bench table predated the budget and said so, and the context sweep and the exactness gate were last run before the ring allocation and binding changed. All three are re-run on an RTX 4070 with a CUDA build of this head, at --kv-pipeline-budget 512. Greedy server output is identical at depth 0, 1 and 4 across all eight tasks. Throughput is +16.1% at 4,096, +56.6% at 16,384, +20.1% at 32,768 and +13.4% at 65,536, for +28, +104, +206 and +410 MiB of device memory. The 131,072 and 262,144 arms are not re-measured and say so. The server table is replaced with the four 18,432-prefill tasks of the exactness gate, which is what this head was actually run on; the copy/compute breakdown keeps its earlier numbers and says which head they came from. Assisted-by: Claude Opus 5
ab1c7ae to
72df633
Compare
… MiB The context sweep is measured with the budget raised, so its deep rows read as default behaviour when they are not: past 20,556 rows of window the default declines and those depths stay ordered. Say so, and add a finer sweep that puts the peak at 16,384 rows and shows the gain per MiB falling off as 1/rows^2 above it. The peak is where copy and compute are equal, and the ring size there works out to (n_slots / n_attn) * compute * BW - the bytes per row cancel, so the budget is quant-invariant. That predicts 101 MiB against the 102 MiB measured, which is what the 128 MiB default is sized against. Assisted-by: Claude Opus 5
A window over several streams is one view of a tensor whose streams sit end to end, and both the prefix and the delivery treated it as one flat byte range. That made the lowest-writing stream cap the stable prefix for every stream above it, and it copied the cells between one stream's window and the next, which the graph never reads. The prefix is now counted within a stream, and a staged input whose last dimension indexes streams is delivered as one range per stream. A window over one stream keeps the single flat range it had, so single-sequence timing and bytes are unchanged. Behind 8 slots of a non-unified cache this takes decode from 72.18 to 112.70 t/s, against 70.61 ordered. At one slot it measures 35.31 against 35.32 before. test_transport_multi_stream_ranges pins the delivery: every stream's window covered once from its own source offset, the unread cells between them never moved, and the early and late bytes split as the prefix says. Concurrent slots cannot be gated on output the way one sequence can - their batching varies between runs, so the same depth gives different greedy output - which is why this is a unit test. Assisted-by: Claude Opus 5
The q4_0 arm of the context sweep moves the peak from 16,384 rows to 32,768 and leaves the ring at it at 108 MiB against 102, which is what "the bytes per row cancel" claims. It was derived before and is measured now. Add the parallel numbers and say what the 8-slot row depends on: run on its own the unified ring fits, and only after a sweep has allocated for 1, 2 and 4 slots in the same process does the headroom guard refuse it. Assisted-by: Claude Opus 5
2 participants
Pipelines the host-to-device delivery of a host-resident KV cache: the transfer is issued one split ahead, on a transfer stream of its own, so a decode token stops paying transfer and attention in series.
Supersedes #38, which targeted
beellama/dev; this is the same work rebased ontollama/dev.What it does
Three pieces, each load-bearing:
ggml_tensor::stable_prefixrecords it on the tensor that owns the storage;llama_kv_cache::update_stable_prefixes()sets it fromapply_ubatch(), before the graph is built and allocated, so the plan and the deliveries are decided against the same write position even when the graph is reused.build_graph_shift()clears it.ggml-alloc's reach.ggml-allocmay recycle a graph-owned input copy after its last graph-level consumer while a look-ahead transfer is still in flight - that is what made an earlier cross-layer prefetch experiment non-exact. The scheduler allocates its own ring and points the staged copies at it before allocation; aready/releaseevent pair per slot carries the handover each way. Every eligible accelerator gets its own ring, cursor and budget, so a layer-split model pipelines on each device and a device with no room falls back alone.Lsplits ahead recycles the slot of the splitL - n_slotsback, son_slots == L + 1recycles the split just enqueued and still running. The ring keeps 2 slots of margin, deliveries are issued after a split is enqueued, and slot recycling is ordered stream-to-stream, not through the host. Each of those three alone costs the entire gain while still producing correct output - the first working version measured +0.5% and looked like "the copy just doesn't overlap".--kv-pipeline-depth N(default 1,0= ordered path exactly). Only engages where a host-resident cache produces the deliveries; a device-resident run never creates the transport. Deeper look-ahead is worse at every depth measured: at a 19,246-token prompt, 29.83 t/s atN = 1, 28.44 atN = 2, 25.93 atN = 4.CUDA only. The scheduler enables the ring for a device whose backend registry is named
CUDA, that has asynchronous transfers and events, and whose default buffer type the scheduler was configured with. Meta devices are excluded outright, so-sm tensorkeeps the ordered path. Everything else - SYCL, WebGPU, Vulkan, the CPU - is untouched and stays ordered.Measurements
Re-measured on this head, on an RTX 4070 (sm_89, 11,902 MiB usable), driver 610.57.04 / CUDA 13.3, i5-13400F,
Qwen3.8-27B-UD-IQ2_M.gguf,-ngl 99 -sm none -mg 0 -t 3 -fa on -ctk q8_0 -ctv q8_0 -b 512 -ub 512, host residency-nkvo --kv-cpu-pinned --recurrent-state-offload,--kv-pipeline-budget 512, everything undertaskset -c 0,2,4.llama-bench, A/B/A/B with reversed arm order (docs/repro/r4-kv-pipeline-ab.sh), both passes shown, plus peak device memory from the context sweep:llama-server, greedy, the four 18,432-prefill tasks of the exactness gate at-c 32768:Depth 4 is slower than depth 1 on all four (27.037, 26.584, 15.934, 15.857), which is why
N = 1is the default.Where the token goes, and why the gain narrows
GGML_SCHED_TRANSPORT_DEBUG=2is implemented in this PR (the counters existed but nothing accumulated or printed them). Per decode graph behind a 19,246-token prompt:ggml_backend_tensor_copy644 MiB in the 28.30 ms the ordered arm spends on the same bytes is 22.0 GB/s, and
nvidia-smireports the card at gen4 x16 - about 88% of what the link does in practice. The pipeline came within 5% of themax(copy, compute)ceiling at both 19k and 48k. The narrowing gain is therefore arithmetic: copy grows with context, compute does not, and once copy dominates there is only compute left to hide behind it.=3names the tensors still on the ordered path.Two attempts to recover the last 3.57 ms, both recorded in the doc as negative results: issuing the delivery in pieces so a blocking copy can interleave does nothing (the copy engine is FIFO across streams, and small pieces cost throughput), and putting the copy on the consumer's stream moves the time into the consumer wait without changing throughput. That 3.57 ms is almost entirely one 256 KiB graph input queued behind the deliveries - it is bandwidth, not latency.
The lever is bytes, not the link
-ctk q4_0 -ctv q4_0halves the traffic, and what that is worth depends on which side of the crossover you are:+5.6% at 19,246 (the consumer wait is 27.31 ms at q8_0 and 27.26 at q4_0 - the pipeline had already reached the compute floor, so the removed bytes were bytes nothing waited for) and +56.6% at 48,042, where copy still dominates.
The ring also wins per MiB against
--kv-gpu-layers. At 19,246 with-c 32768, a device-resident layer costs ~68 MiB and the ring ~205 MiB:--kv-gpu-layers48Four device-resident layers are worth +14.3% on the ordered path and +1.4% on the pipelined one, for more memory than the ring costs.
Pinning is worth as much as the pipeline and is off by default. Behind a 13,128-token prompt: 21.582 -> 32.252 pinned, 14.945 -> 22.709 unpinned.
Parallel sequences
Measured with
llama-batched-bench, 2,048 prompt tokens per sequence,-c 32768 -np 8, generation t/s:-nplThe 8-slot row is measured with each arm run on its own. Taken as the last step of a sweep that has already run 1, 2 and 4 slots in the same process, the unified ring is refused for headroom and that arm reads 83.88 instead - the guard reacting to what the process has already allocated rather than to the configuration.
Without the per-stream delivery a non-unified cache pipelines almost nothing: 6.6% of the delivery goes early at 8 slots, because the prefix stops at the lowest stream's head, and the delivery also carries the unread cells between the streams. With it, 8 slots go from 72.18 to 112.70 t/s against 70.61 ordered, and one slot measures 35.31 against 35.32. A non-unified cache is the default, so this is the configuration most
-np > 1runs were in.Concurrent slots cannot be gated on output the way one sequence can. Their batching varies between runs, so the same build at the same depth produces different greedy output - three runs at
N = 0gave three different hashes.test_transport_multi_stream_rangesstands in for it: every stream's window covered once from its own source offset, the cells between them never moved, and the early and late bytes split as the prefix says.Device memory
A host-resident cache exists to keep device memory free, so the staging is capped outright by
--kv-pipeline-budget(default 128 MiB per device), not by a fraction of what happens to be free. A ring is(N + 2)slots of one attention layer's K+V over the whole context, so it grows linearly with context.The cap is applied to what the current graph needs, and the warning reports the full-context figure alongside it so the budget can be sized against the number that matters. Enforcing the projection instead was tried and reverted: it refuses the ring for every large
-ceven when the window never gets near it, which at-c 32768turned the feature off by default.A budget or headroom decline is per graph, not latched. The plan is remade for every graph, so a context that outgrows the budget gives the ring back and a later, smaller live window takes it again. The cost is that transient, and it is bounded by the budget the user already authorised. Declining is otherwise free: the ring and the transfer backend's device context are both released, and the transfer backend is created lazily in the first place.
The ring never starves the graph. The rings are laid out and allocated before the graph is, and the configuration is locked by then, so a device that can hold the graph alone but not the graph next to a ring used to fail allocation outright with no way for the caller to retry. If graph reservation fails, the rings are now released and the reservation is retried once on the ordered path, and that scheduler keeps the ordered path from then on.
test_transport_releases_ring_for_graphsizes a dummy device to hold the graph or the ring but not both.The headroom guard was measured and left alone. The ring declines unless it can leave
GGML_SCHED_TRANSPORT_HEADROOM(512 MiB) free. The compute buffer on this configuration is 617 MiB and this fork resizes compute buffers at run time, so the guard is about the size of the thing it exists to leave room for. Two candidate changes were checked and neither survives: sizing it against the compute buffer instead of a constant is stricter here (617 against 512), and lowering it is not supported by the cases that looked like evidence for it - run on their own, both the 8-slot unified ring and-d 131072fit and pipeline at the shipped 512 MiB. The guard refuses only under accumulated pressure inside one process, which is what it is for.Tried and reverted: planning the ring during reserve so
ggml-allocwould not budget blocks for the staged copies. It reclaims nothing, andtest_transport_fallback_keeps_allocator_planpins the invariant: the compute buffer is the same size on both arms.Validation
N = 0,N = 1andN = 4- all eight tasks (prose, source code, JSON records, dialogue, at 2k and 18k), re-run on this head with a CUDA build:docs/repro/r4-kv-pipeline-exact.sh 0 1 4exits 0 with every hash matching depth 0.N = 0passes agree on all eight, which was not true before this PR.records@18432used to give different hashes across otherwise identical runs and looked like the pipeline breaking exactness. It is not the task: asked on its own with the prompt cache off it returns the same hash three times running, at-c 32768and-c 65536. It was the harness - all eight tasks share one server with prompt caching on, andrecords@18432is ~29.6k tokens with a task of about the same size ahead of it, so the two do not both fit in a 32,768 cache and placement depended on what was still resident. The harness now setscache_prompt: false, and gives every task a nonce so no two share a restorable prefix.test-alloccovers entry allocation and bounds against a buffer type whoseget_alloc_sizeexceedsggml_nbytes, per-byte delivery coverage from the matching source offset, event ordering (nothing waits on an event before it is recorded), the empty graph, the budget decline and recovery, per-backend partial failure, the meta and non-CUDA exclusions, and the ring-starvation fallback. Clean under UBSan.test_transport_multi_stream_rangescovers the per-stream delivery, which the byte-identical gate structurally cannot: that gate is single-sequence, and concurrent slots are not reproducible between runs.llama_contextpasses a depth of 0.Tensor parallelism
-sm tensoris not pipelined, and nothing in this PR moves it closer. The scheduler excludes meta devices explicitly and requires theCUDAregistry name, so a tensor-parallel run keeps the ordered path. Enabling it needs a validated strided head-split write for a host-resident cache, which is not here. An earlier revision of this branch carried three meta-backend prerequisites; they are not in this revision.Independently of that,
-sm tensortogether with--no-kv-offloadis currently incorrect, which is a pre-existing fork defect and not caused by this PR. Fixed separately in #48; reported upstream as ggml-org#27757.Same build, same prompt, greedy:
-sm layer+-nkvo3e127464e8b901a7-sm tensor+ device KV3e127464e8b901a7-sm tensor+-nkvo9b82be0158a2fa4dCause. TP splits attention by head, but a host-resident cache is one undivided tensor, so the scheduler's copy is classified
MIRROREDand the whole window goes to both devices. With 24 query heads split 12/12 and 4 KV heads mirrored, the kernel derives the GQA ratio from the tensors it is handed - 12/4 = 3 instead of 6 - and the second device's queries, renumbered from 0, read the first device's keys. With an uneven split the same fault surfaces as a crash:fattn.cu:371 GGML_ASSERT(Q->ne[2] % K->ne[2] == 0), since 24 split 13/11 is not divisible by 4.Still to do
-sm tensor+--no-kv-offload(ggml-meta : split a host-resident KV cache by head #48). Until then it is wrong, not slow.-sm tensorcan be pipelined at all.Re-run on this head:
r4-kv-pipeline-exact.sh(exit 0),r4-kv-pipeline-ab.sh(exit 0),r4-kv-pipeline-context-sweep.sh(exit 0),test-allocin Release, UBSan and against the CUDA build,git diff --check.🤖 Generated with Claude Code
https://claude.ai/code/session_01DLsim6XPFPodyRQ9cRVdG1