Misc. bug: llama-quantize fails on DeepSeek-V4 — i32 ffn_gate_tid2eid routing table not excluded from quantization

[KikoCis]

Summary

llama-quantize fails on DeepSeek-V4 models because the arch's ffn_gate_tid2eid tensor is i32 and isn't excluded from quantization.

Repro

Convert any DeepSeek-V4 checkpoint (e.g. deepseek-ai/DeepSeek-V4-Flash-DSpark) and quantize it:

python convert_hf_to_gguf.py DeepSeek-V4-Flash-DSpark --outfile v4-q8.gguf --outtype q8_0 --fp8-as-q8
llama-quantize --allow-requantize v4-q8.gguf v4-q4.gguf Q4_K_M

Output:

[  20/1328] blk.0.ffn_gate_shexp.weight  - [4096, 2048, 1, 1], type = q8_0, converting to q4_K .. size = 8.50 MiB -> 4.50 MiB
[  21/1328] blk.0.ffn_gate_tid2eid.weight - [   6, 129280, 1, 1], type = i32,
llama_model_quantize: failed to quantize: cannot dequantize/convert tensor type i32

(thrown at src/llama-quant.cpp:228)

Cause

blk.N.ffn_gate_tid2eid.weight (LLM_TENSOR_FFN_GATE_TID2EID, declared in src/llama-arch.cpp and created in src/models/deepseek4.cpp as {n_expert_used, n_vocab}) is a token-id → expert-id index table, not weights. It's i32, so llama_tensor_quantize_impl can't convert it.

llama_tensor_get_type() in src/llama-quant.cpp already excludes the other MoE routing tensor:

quantize &= name.find("ffn_gate_inp.weight") == std::string::npos;

but tid2eid was never added when DeepSeek-V4 landed (#24162) — understandable, since that work targeted inference.

Suggested fix

One line, next to the existing router exclusion:

quantize &= name.find("ffn_gate_inp.weight") == std::string::npos;
quantize &= name.find("tid2eid") == std::string::npos;   // DeepSeek-V4: i32 token-id -> expert-id table

With this, the tensor is copied through verbatim (2.959 MiB) and quantization completes.

I'm happy to open a PR if that's useful — just say the word (or feel free to take the line directly, it's trivial).

Notes

• Probably went unnoticed because the public V4 releases already ship quantized (FP8 attention + MXFP4 experts, ~4.4 BPW), so there's little reason to run llama-quantize on them. It does matter if you want to push the routed experts below 4 bits.
• Tested on 505b1ed1, macOS/Metal.

Thanks for the V4 support — it landed fast and the conversion path (incl. --fp8-as-q8) worked flawlessly otherwise.

[fairydreaming]

@KikoCis Just a friendly reminder that your quantization command will likely make the model significantly dumber. The problem is that llama-quantize will try to quantize most of the model tensors.

If you check for example unsloth DeepSeek V4 quants they have almost all the model tensors untouched (kept in high-resolution f32, f16 or Q8_0), only huge expert tensors that take most of the disk space are quantized more.

[KikoCis]

Thanks @fairydreaming — you're right, and I appreciate the nudge. A flat quantize crushes the small, sensitive tensors (attention, embeddings, norms) for almost no disk saving on a MoE, where the experts are the real weight. Keeping those high and only pushing the expert tensors down (à la Unsloth's UD / --tensor-type) is clearly the way.

For context, I wasn't actually shipping that quant — I was requantizing from a community Q8_0 just to test, and it turned out pointless regardless: its experts are already mxfp4, so a flat Q4_K up-converts them and the file grows. The tid2eid report was just to stop the quantize from erroring out; good to see #25787 handling it. Thanks again for taking the time.