ggml: add optional asynchronous weight prefetching

[giveen]

Overview

Backports the asynchronous weight-prefetch work from ggml-org#21067 and integrates it with this fork's MoE cache scheduler changes.

• Adds --prefetch-weights 0|1, default off
• Uses a dedicated copy stream and events to overlap host-to-device weight transfers with compute
• Adds prefetch_weights reporting to llama-bench
• Currently advertises copy-stream support on CUDA only

Review status

The branch is current with feature/turboquant-kv-cache. Fork-specific CUDA performance and correctness validation is still required before merge, particularly with MoE cache enabled.

AI usage disclosure: Yes

[TheTom]

I reviewed the current head after the base merge. The latest commit only brings the branch current; the prefetch implementation itself is unchanged.

I am holding this one for evidence, not because the idea is unwelcome. This changes the central graph splitter, allocation lifetimes, and cross-stream event ordering. The upstream source PR is still open and conflicting, and an upstream reviewer specifically called out event synchronization as unresolved. Its performance reports are also mixed, with MoE workloads sometimes moving more data and regressing.

That matters more in this fork because prefetch and the GPU MoE cache now share the scheduler. Only CUDA advertises copy_stream, so generic CI can prove compilation but cannot prove the feature.

Please post controlled llama-bench results on CUDA with identical model and flags for --prefetch-weights 0 and 1:

• one partially offloaded dense model
• one MoE model with the same CPU expert placement
• MoE cache off and on for the MoE case
• prompt processing and generation results, at least 5 repeats
• peak VRAM and the exact command lines

Also run a deterministic output or perplexity comparison with prefetch off/on. A longer generation run is useful for catching event-ordering failures that a short benchmark can miss.

The current WebGPU wasm failure is unrelated. That job mixed wasm32 system libraries into a wasm64 link. A fresh matrix is running after the base updates.

I also cleaned up the title and PR description so the scope and current review status are clear.

[giveen]

@TheTom I believe the evidence does not pan this out as a feasible idea.

pr/prefetch-weights - Evidence Summary and Verdict

Date: 2026-08-20. Branch: pr/prefetch-weights @ 2d5f621.
Hardware: RTX 5090 (32 GiB, CC 12.0), build 10515.

Verdict: kill the prefetch idea as implemented

The measured evidence shows the mechanism targets the wrong bottleneck in
every decode regime, and its cost lands on this fork's flagship feature
(the MoE expert cache). The moe-cache commits in this branch are strong and
should be pursued separately.

What prefetch was supposed to do

--prefetch-weights double-buffers host-resident weight copies across
scheduler splits (ggml-backend.cpp, ggml_backend_sched_set_prefetch_weights),
reserving next-split weight copies before compute so CPU->GPU transfer
overlaps with compute. Only backends advertising copy_stream get the copy
backend/events; Metal reports false, so the feature is CUDA-only by
construction (and unified-memory Macs have no per-token copies to hide).

Measured results

All runs: llama-bench, identical model/flags per pair, q8_0 KV cache,
fa on, 5 reps unless noted.

MoE model: Laguna-S-2.1 118B.A8B UD-Q5_K_M (81.82 GiB), experts on CPU

Placement pinned with -ngl 49 -ncmoe 9999.

cache	pw	pp512 (t/s)	tg128 (t/s)
off	0	89.54 ± 0.83	9.58 ± 0.08
off	1	72.46 ± 0.11 (-19%)	9.29 ± 0.29
auto	0	90.16 ± 0.76	26.12 ± 0.81
auto	1	73.39 ± 0.12 (-18.6%)	24.54 ± 0.61 (-6.1%)

Notes:

• tg arms used --n-gen-warmup 256 so the cache working set converged
  before timing. Without it, an earlier run showed a misleading "prefetch
  helps tg" artifact (19.04 ± 6.63) that vanished once converged.
• pw=1 runs also received LESS cache capacity from fit: ~16.0-16.3 GiB of
  projected pools vs 19.6 GiB for pw=0. The double-buffer reservations
  consume VRAM before pool allocation. The two features fight over the
  same memory.

Dense model: Qwen3.8-27B UD-Q5_K_XL (19.43 GiB)

Offload sweep, -n 64, partial results before stopping the run:

ngl	pp512 pw=0 -> pw=1	tg pw=0 vs pw=1
999 (fully resident)	3180 -> 3158 (noise)	66.37 vs 65.88 (-0.7%, noise)
40	612 -> 641 (+4.7%)	4.31 vs (not reached)
32	489 -> 507 (+3.7%)	3.27 vs 3.24 (none)
24	409 -> 424 (+3.7%)	2.66 vs 2.67 (none)
8	312 -> 315 (noise)	1.98 vs 1.97 (none)

Why the idea fails

1. The premise targets the wrong bottleneck for decode. With partially
   offloaded layers, decode time is dominated by CPU matvec compute on the
   spilled layers, not by the PCIe copy. Hiding copy latency behind compute
   that is not the bottleneck buys nothing: zero tg effect at every offload
   depth measured (ngl 8 through 40).
2. Where overlap does engage (prompt processing), the win is small and
   narrow: +3.7 to +4.7% pp on partially-offloaded dense only. Users in
   that regime can already address it by quanting down or repacking.
3. The cost lands on the fork's flagship feature. With moe-cache active,
   prefetch costs -18.6% pp and -6.1% tg, and steals ~3.6 GiB from cache
   pools (~18% less expert coverage). Same flag, opposite and much larger
   signs depending on workload.
4. Complexity price is permanent: changes to the central graph splitter,
   allocation lifetime extensions (keepalive views), and cross-stream event
   ordering - the exact area an upstream reviewer flagged as unresolved.
   Unjustifiable maintenance risk for a flag that loses more often than it
   wins.

Scorecard

workload	pp	tg
dense, fully resident	free (noise)	free (-0.7%)
dense, partial offload	+4%	0
MoE + cache	-18.6%	-6.1%
MoE, cache off	-19%	0

What survives

• moe-cache: strong positive. 9.58 -> 26.12 t/s tg128 (~2.7x) on Laguna
  with matched placement, pp unchanged. These commits should be separated
  from this PR and pursued on their own merits.
• Dormant-cost control: fully-resident dense shows prefetch is free when
  unused (-0.7% tg, pp within noise). Default-off behavior is safe.

Outstanding items if anyone re-litigates

• Perplexity / deterministic output comparison, prefetch off vs on
• Peak VRAM sampling during runs (pool capacity numbers above are from fit
  logs; nvidia-smi would make them explicit)
• -v log excerpts showing [moe-cache] enabled + hit stats for cache arms
• A scoped variant could theoretically survive: auto-disable when a cache
  session exists, documented dense-pp-only. Low expected value; different PR.

Mac note (from earlier scoping)

Prefetch cannot help Apple Silicon: Metal reports copy_stream = false
(ggml-metal.cpp), so the scheduler never creates the copy backend/events,
and unified memory has no per-token host->device weight copies to hide.
The Mac-relevant feature is the moe-cache (functional on Metal since commit
905499a, unbenchmarked there - test plan in Part 2 below).

---

Part 2 - MoE cache on Apple Silicon (Metal) - still wanted

Build with -DGGML_METAL=ON. Confirm engagement BEFORE timing: -v log must
show [moe-cache] enabled after pool allocation, no disable-gate lines,
nonzero hits in teardown stats.

A/B arms, identical except cache:

# off
... --fit off --moe-cache off -c 8192 -ngl 99 -fa on ...
# auto
... --fit on  --moe-cache auto -c 8192 -ngl 99 -fa on ...

Same fixed workload, one warmup request discarded, 3+ timed runs, report
cold AND settled t/s. Correctness: llama-perplexity off vs on (CUDA
reference matched within error bars: 2.7996 vs 2.7987) plus one long
coherent generation.

Metal v1 caveats: synchronous fills (no background worker yet), fused SwiGLU
falls back to CPU path, win mechanism is GPU-matvec-beats-CPU-matvec over
unified memory (narrower margin than CUDA's PCIe avoidance).

[TheTom]

Agreed. This is a good kill decision. The warmup correction matters, and the MoE cache interaction settles it: a narrow 4% dense prefill win does not justify the scheduler and event complexity when the same flag costs 18 to 19% MoE prefill, 6% decode, and about 3.6 GiB of cache capacity. Thanks for doing the full A/B and closing it on evidence. Keeping the MoE cache work separate is the right outcome.