DeepSeek-V4: quantized K-cache (--cache-type-k q8_0) produces garbage on all backends — attention rotation forces the model off its sparse attention paths

[Mermiges]

Summary

DeepSeek-V4(-Flash) with a quantized K-cache (--cache-type-k q8_0) produces confident gibberish on every backend (CPU and CUDA verified identically) while the server stays healthy — no error, no abort. With --cache-type-k f16 output is correct. Root-caused to the attention-rotation (Hadamard incoherence) path forcing the model off its sparse attention implementation.

Minimal reproduction

Any DeepSeek-V4-Flash GGUF (tested: two independently sha256-verified unsloth quants, UD-Q4_K_XL and Q4_K_M-XL):

llama-server -m DeepSeek-V4-Flash-UD-Q4_K_XL-00001-of-00005.gguf -fa on --cache-type-k q8_0 --cache-type-v q8_0
# CPU-only (-ngl 0) is sufficient to reproduce
curl -s http://127.0.0.1:8080/completion -d '{"prompt":"The capital of France is","n_predict":12,"temperature":0}'

Result: " newcom wrt wrtêu newcom dialog大有lean…" (identical signature on CPU and 8× V100 CUDA).
With --cache-type-k f16 (V may stay q8_0): " Paris." — correct.

Root cause

1. A quantized K/V cache enables the Hadamard incoherence rotation (attn_rot_k/attn_rot_v), allocating self_k_rot.
2. DeepSeek-V4's attention dispatch (models/deepseek4.cpp, build_attention) requires self_k_rot == nullptr to take its designed sparse CSA/HCA/lightning-indexer paths; a non-null self_k_rot diverts every affected layer to build_raw_attention.
3. build_raw_attention applies the rotation with a plain ggml_mul_mat instead of the block-reshaping ggml_mul_mat_aux the standard path uses, and never un-rotates the MLA V-is-a-view-of-K output before the v_mla up-projection (compare build_attn's cur = ggml_mul_mat_aux(cur, v_rot)).

Graph-level, therefore backend-independent — which matches the identical CPU/CUDA corruption. f16-K never enables the rotation, hence unaffected.

Isolation proof (the kernels are fine)

test-backend-ops cases added at hsk=576/hsv=512 (V as a sub-view of K) with q8_0: FLASH_ATTN_EXT 56/56 PASS and SET_ROWS (quantize-on-write) 2/2 PASS, CUDA-vs-CPU — the FA read kernels and the K-cache write kernel are numerically correct in isolation. The defect is purely the rotation path composition in the live graph.

Additional discriminator: disabling all rotations at runtime (patch adding an env override, incl. the forced-indexer rotation) with q8_0 K/V → coherent on CPU.

Live verification of the minimal fix

Disabling the KV Hadamard rotation for LLM_ARCH_DEEPSEEK4 (so quantized K flows through the model's designed sparse attention as plain q8_0) fixes it — full 4-cell K/V ∈ {q8_0,f16}² matrix coherent at temp 0, CPU and CUDA:

K	V	before	after
q8_0	q8_0	garbage	" Paris."
q8_0	f16	garbage	" Paris."
f16	q8_0	correct	correct (unchanged)
f16	f16	correct	correct (unchanged)

Suggested fixes (in increasing scope)

a. Disable the KV rotation for DEEPSEEK4 until the sparse paths support it (minimal; what our branch does).
b. Make build_csa_lid_attention / build_hca_attention rotation-aware (apply + undo) instead of asserting self_k_rot == nullptr.
c. Fix build_raw_attention to use ggml_mul_mat_aux and to un-rotate the V output for the MLA V-is-K-view case.

(We have (b)/(c) in progress as well and can PR whichever direction maintainers prefer.)

Provenance

• Introduced by the interaction of the DeepSeek-V4 support (DeepSeek V4  #24162) with the quantized-KV attention rotation (Feature Request: Implement Fast Walsh Hadamard Transform #21352/kv-cache : support attention rotation for heterogeneous iSWA #21513/CUDA: add fast walsh-hadamard transform #23615/vulkan: fast path for walsh-hadamard transform #23687); present on current master by code inspection.
• Fix branch with regression tests: https://github.com/Mermiges/llama.cpp/tree/ds4-volta-fix (commits 6c6f71f89 tests, b5bec9221 fix, c57d18583 guard note).
• Full research record (evidence matrices, raw logs, measured V100 numbers): https://github.com/ProprietaryLegal/deepseek-v4-flash-v100

Happy to open a PR for option (a) immediately or (b)/(c) once validated.

[Mermiges]

Update: fix options (b) and (c) are now implemented and verified on branch
https://github.com/Mermiges/llama.cpp/tree/ds4-rotation-aware (commits 07a24f492 + 82553cf08, stacked on the minimal fix):

• build_raw_attention corrected: block-wise rotation via ggml_mul_mat_aux (the plain ggml_mul_mat was a shape error that never actually rotated) + un-rotation of the attention output before the output projection.
• build_csa_lid_attention / build_hca_attention made rotation-aware (asserts removed; Q and the raw-cache K rotated; output un-rotated; a rotated copy written to the CSA/HCA caches while the persisted compressor state stays un-rotated). Quantized-K layers stay on their trained sparse paths.
• Rotation re-enabled for DEEPSEEK4 by default; LLAMA_ATTN_ROT_DISABLE=1 retained as opt-out.

Verification (V100/sm_70 + CUDA-vs-CPU op parity): full K/V ∈ {q8_0,f16}² matrix coherent at temp 0 with rotation ON; FLASH_ATTN_EXT 56/56 and SET_ROWS 15/15 unchanged. Measured rotation cost: ~6.8% decode throughput (q8/q8, single card).

Notably, the rotation earns that cost: in a needle-in-haystack screen at ~6000 prompt tokens with --cache-type-k q8_0, rotation-ON retrieved the needle while rotation-OFF (plain q8_0-K, i.e. fix (a) alone) missed it and dropped two probes — consistent with the incoherence rotation's intended error-spreading. (Single unseeded run — directional; we'd want seeded repeats + perplexity vs f16-K before treating it as decision-grade.)

Known gaps before a PR: hint-path behavior on non-CUDA backends (Vulkan/Metal/SYCL) unverified; a defensive assert that the raw/CSA/HCA caches share one rotation block size; a composite rotate→attend→un-rotate op-test for CI. Happy to open the PR for (a) as the safe immediate fix, or (b)+(c) once maintainers weigh in on the preferred direction.