Eval bug: DeepSeek-V4-Flash lightning-indexer materialises a ctx-proportional (4 GiB/layer) score buffer

[TacoTakumi]

Name and Version

$ llama-cli --version
version: 9924 (90e0f5c)
built with GNU 13.3.0 for Linux x86_64

Operating systems

Linux

GGML backends

CUDA

Hardware

GPU: 5x NVIDIA CUDA (2x RTX 3090, RTX 5060 Ti, 2x RTX 4060 Ti), 96 GiB total VRAM
CPU: Intel Core i5-13600K (14 cores / 20 threads)
RAM: 128 GB DDR4-3200
OS: Linux x86_64 (Ubuntu 24.04)

Models

DeepSeek-V4-Flash (DSV4-Flash), GGUF, MXFP4 quant, 5 shards.

Problem description & steps to reproduce

Edit (root cause found): (see comment below) The driver is not the KV cache and not a host-side allocation as such, it is the DSV4 lightning-indexer compute buffer. The scores lid_kq = ggml_mul_mat(indexer_k, indexer_q) in build_lid_top_k() are materialised full-width per CSA layer (4 GiB/layer at ctx 131072, 512 MiB at 16384 -- exactly ctx-proportional, x21 layers) just to pick a top-512, and are not covered by flash-attention. The host-RAM growth is a second-order effect of MoE expert-offload under memory pressure. Full analysis + GGML_SCHED_DEBUG=2 evidence in the followup comment below.

With my setup, loading DeepSeek V4 Flash with --ctx-size 131072 uses about 110.5 GiB of system RAM on current master (90e0f5c, v9924) vs 71 GiB on the older fairydreaming be00e7771 build with identical flags. That's about 38-40 GiB more and it grows with context, roughly 0.33 MiB per token. GPU VRAM is comparable across builds at matched splits (~82 GiB old vs ~85 GiB new), so the increase is host side, and it reproduces on unmodified master.

Binary: Commit: RSS @ ctx 131072:
fairydreaming be00e7771 (v9854) 71 GiB
master with #25402 15f7321 (v9907) 110 GiB
clean master 90e0f5c (v9924) 110.5 GiB
local stall-fix branch ec1c342da (v9910) 107 GiB

Regression ~+38 GiB at ctx 131072, ~0.33 MiB per token of ctx.
GPU VRAM comparable across builds (~82-86 GiB total; run-to-run --fit variation).

Reproduction:

llama-server --ctx-size 131072 --ubatch-size 512 --fit on --fit-ctx 131072
--fit-target 1024,1024,2048,2048,2048 --no-mmap --flash-attn on --parallel 1

--fit-target is my 5-device split, adjust for your setup
env: GGML_CUDA_NO_PINNED=1 ; model: DeepSeek-V4-Flash MXFP4
poll /health until {"status":"ok"}, then read VmRSS from /proc//status
repeat per commit, GPUs drained between runs

Both my local build ec1c342da (107 vs 110 GiB) and #25402 (a checkpointing PR, now closed) are exonerated. I think the problem is somewhere between be00e7771 and 90e0f5c. I'll likely dig deeper when I have the cycles but wanted to get this out there.

I searched open and closed issues and found no existing report, the closest ones (#25408, #24265, #21690) are distinct.

The measurements and tracker search were aided by Claude.

First Bad Commit

Not bisected. Last known good: be00e7771 (v9854, 71 GiB). Known bad: 90e0f5c (v9924, 110.5 GiB). First bad commit is somewhere in that range.

Relevant log output

Logs

# Load-time host RSS at /health OK, ctx 131072, identical llama-server flags.
# RSS = VmRSS from /proc/<pid>/status. VRAM = per-device used at load (MiB, 5 GPUs).
# GPUs drained between runs.

be00e7771  v9854  old fairydreaming base
  RSS  = 71 GiB
  VRAM = 22079 22686 13163 13265 13160 MiB   (total ~82 GiB)

15f732183  v9907  new base
  RSS  = 110 GiB
  VRAM = 22859 22824 13407 13823  1904 MiB   (total ~73 GiB; the 5th device took a
         degenerate --fit split from VRAM residue, so ~11 GiB spilled to host here)

ec1c342da  v9910  stall-fix branch on the new base
  RSS  = 107 GiB
  VRAM = 22857 22826 13045 13989 13908 MiB   (total ~85 GiB)

90e0f5cfc  v9924  clean current master (separately drained run)
  RSS  = 110.5 GiB

# Cleanest matched-VRAM comparison: be00e7771 (71 GiB at ~82 GiB VRAM) vs the new-base
# build ec1c342da (107 GiB at ~85 GiB VRAM). VRAM is comparable, slightly higher on the
# new build, yet host RSS is +36 GiB. The ~110 GiB new-lineage figure and the
# ~0.33 MiB/token slope reproduce across three independent new-base builds vs the old
# base. The 15f732183 row's low VRAM total is the noted degenerate split and is not the
# basis of the claim.

[TacoTakumi]

Root cause: the lightning-indexer (lid_kq) score matrix, not the KV cache

Followed this up on the two builds. The extra memory is not a host-side allocation per se and not the KV cache -- it is the DSV4 compute (graph) buffer, and it is fully reproducible with stock flags on any hardware via GGML_SCHED_DEBUG=2. The host-RAM growth I first reported is a second-order effect (below).

The KV caches are identical and small

At load (-lv 6), every DSV4 KV cache is tiny and does not scale the way the RSS does: the non-SWA/"raw" cache is 131072 cells, 0 layers -> 0 MiB; SWA 32.25 MiB, CSA 672 MiB, HCA 20 MiB, lightning-indexer 168 MiB. So the KV cache is not where the growth is.

The compute buffer is, and it is ctx-proportional

sched_reserve at --ctx-size 131072 --ubatch-size 512 --flash-attn on:

graph nodes  = 33689
compute buffer (summed over devices) ~= 55 GiB

Sweeping context shows it scales linearly:

--ctx-size	compute buffer (total)
16384	~17 GiB
131072	~55 GiB

+38 GiB across that range = ~0.34 MiB per token of context.

The single tensor responsible

GGML_SCHED_DEBUG=2 shows the dominant tensors are the lightning-indexer dense KQ scores, indexer_kq = ggml_mul_mat(indexer_k, indexer_q) in build_lid_top_k() (then ggml_relu), named lid_kq-<layer>:

ctx 131072:  lid_kq-2 .. lid_kq-42  = 4096 MiB each   (x21 CSA layers)
ctx  16384:  lid_kq-*               =  512 MiB each

Exactly 8x for 8x context. These are by far the largest tensors in the graph (next is ~1088 MiB). The chain per CSA layer is:

lid_kq (4 GiB dense scores) -> relu -> * weights -> sum_rows
   -> lid_score (64 MiB) -> + mask -> lid_top_k (1 MiB)

i.e. the full [n_kv x n_indexer_head x n_tokens] score matrix is materialised per layer only to be reduced to a top-512 selection, and this path is not covered by flash-attention. With 21 CSA layers this is the ctx-proportional blow-up.

Why it showed up as host RAM

On a memory-constrained multi-GPU box with MoE expert-offload (--fit), the oversized compute buffer consumes VRAM, and the offloader compensates by pushing more expert weights to system RAM -- so VRAM stays about constant while host RSS grows ~38 GiB at ctx 131072. On a setup without CPU expert-offload the same buffer surfaces as VRAM pressure / OOM instead. Either way the driver is lid_kq.

Note on the bisect range in the issue

be00e7771 is fairydreaming's out-of-tree DSV4 branch, not a master ancestor (merge-base 6f4f53f2b); master's DSV4 was added by #24162. So be00e7771..90e0f5cfc spans a branch divergence -- the comparison is really fairy-impl vs master-impl, and a single linear "regressing commit" probably does not exist. For reference, the fairydreaming-based build keeps this compute buffer at ~5 GiB (graph ~9.6k nodes vs 33.7k on master), so it evidently bounds/streams the indexer scoring where master materialises it full-width. I have not pinned the exact place that differs.

Suggested direction

Compute the indexer top-k in a chunked/streaming pass (score a window, keep a running top-k) rather than materialising the full-context score per CSA layer.

Happy to attach the full GGML_SCHED_DEBUG=2 dumps at both context sizes if useful.

[TacoTakumi]

cc @fairydreaming

Pinned: master is missing the fused ggml_lightning_indexer op

Followed the thread from my previous comment (I'd said I hadn't pinned where the fairydreaming build differs). It is now exact.

Graph-dumping both builds at ctx 131072 (GGML_SCHED_DEBUG=2), the attention dispatch is identical (21 CSA + 20 HCA + 2 raw layers on both). The only difference: the fairydreaming build has zero lid_kq tensors (largest tensor ~1088 MiB), while master has 21 x 4096 MiB.

The reason is in build_lid_top_k() (src/models/deepseek4.cpp). The fairydreaming branch computes the indexer score with a fused op:

indexer_score = ggml_lightning_indexer(ctx0, indexer_q, indexer_k, indexer_weights,
                                        1.0f/sqrtf(n_embd_indexer_head),
                                        1.0f/sqrtf(n_indexer_head));

which streams the score and never materialises the [n_lid x n_indexer_head x n_tokens] intermediate. Master has no such op, so build_lid_top_k() runs the unfused fallback -- ggml_mul_mat -> lid_kq (4 GiB/CSA layer at ctx 131072) -> relu -> mul -> sum_rows -> top-k.

ggml_lightning_indexer does not exist anywhere in master's ggml/ at all. In the fairydreaming branch it is a full op: GGML_OP_LIGHTNING_INDEXER + ggml.c creation (output shape [k->ne[2], q->ne[2], 1, q->ne[3]]), a CPU kernel (ggml_compute_forward_lightning_indexer, ~70 lines), and a CUDA kernel (ggml-cuda/lightning-indexer.cu, ~500 lines).

So the fix is to bring that fused op (or an equivalent streaming top-k) into master and switch build_lid_top_k() to it. The unfused path is correct but pays a ctx-proportional score buffer per CSA layer that the fused op avoids entirely.

[v0l0dia]

version: 9988 (f4253ef96)
built with GNU 14.2.0 for Linux x86_64
nvcc: Cuda compilation tools, release 13.3, V13.3.33
Build cuda_13.3.r13.3/compiler.37862127_0

Can confirm that the very same antirez q2 quant runs w/o issues on single RTX-6KPro with 512K full context using @fairydreaming fork, but fails to load with v9988.

llama-server --kv-unified --ctx-size 512000 --n_predict 64000 --parallel 1 --n-gpu-layers 999 --batch-size 4096 --ubatch-size 2048 --flash-attn on --no-mmap

Some LID-related logs:

0.15.243.718 W resolve_fused_ops: layer 2 is assigned to device CUDA0 but Lightning Indexer is assigned to device CPU (usually due to missing support)
0.15.243.721 W resolve_fused_ops: Lightning Indexer not supported, set to disabled
0.15.250.649 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 130527.19 MiB on device 0: cudaMalloc failed: out of memory
0.15.250.653 E ggml_gallocr_reserve_n_impl: failed to allocate CUDA0 buffer of size 136867676672

[TacoTakumi]

Thanks for the input. Yes, it's the same root cause. The fused indexer op on master (GGML_OP_LIGHTNING_INDEXER, #24231) is CPU-only, so on CUDA it gets disabled and the fallback materialises the dense indexer scores, which scale with context. A CUDA implementation of the fused op would fix this.

[fairydreaming]

PR is incoming: #25545

Will probably take some time to apply all review suggestions and merge. But you can use it now, should work.

[TacoTakumi]

@fairydreaming Thank you!

[fairydreaming]

#25545 is merged now, the problem should be fixed (Metal and Vulkan are still affected though).

[TacoTakumi]

Thanks, @fairydreaming. I rebuilt from master (505b1ed) and loaded DeepSeek-V4-Flash MXFP4 at ctx 131072. Host RSS dropped back down to 72 GiB from 110. Closing.