Qwen3.8 dense silently generates repeating garbage with quantized KV cache
Summary
On gmlx 0.4.6, enabling uniform 8-bit KV-cache quantization causes Qwen3.8-27B dense to generate deterministic repeating or corrupted output.
The corruption reproduces with:
- MTP disabled
- APC disabled
- No custom chat template
- A single request
- A freshly started server
The same GGUF, prompt, and sampling configuration produces coherent output when KV quantization is disabled.
The default gmlx-owned Qwen path silently generates corrupted text. Setting GMLX_QWEN_OWNED=0 instead exposes a cache-layout exception:
Generation failed: 'tuple' object has no attribute 'shape'
This may be related to the quantized hybrid-cache extraction/merge problems discussed in mlx-vlm issue #1322, although that issue is marked fixed and covered Qwen3.6 MoE. This reproduction uses Qwen3.8 dense through gmlx.
Environment
Hardware: Apple M5 Pro, 24 GB unified memory
OS: macOS 27.0
gmlx 0.4.6
mlx 0.32.1
mlx-kquant 0.4.4
mlx-lm 0.31.3
gguf 0.19.0
gmlx doctor output:
gmlx 0.4.6 doctor
PASS runtime mlx 0.32.1, mlx-kquant 0.4.4, mlx-lm 0.31.3, gguf 0.19.0, metal ok
PASS kernels sdpa_vector, sdpa_decode_gqa
WARN config no config found (gmlx init); searched: gmlx.yaml, ~/.config/gmlx/gmlx.yaml, ~/.gmlx.yaml
SKIP models no config
SKIP server no background server (gmlx serve starts one)
PASS launcher
SKIP hf token no token
PASS memory 24 GB RAM
PASS disk
Model:
unsloth/Qwen3.8-27B-GGUF
Qwen3.8-27B-UD-Q3_K_XL.gguf
gmlx detects it as:
model_type=qwen3_5
hidden=5120
layers=64
heads=24
kv_heads=4
family=qwen3.6
Minimal reproduction
Start a server with Q8 KV, APC disabled, MTP disabled, and no custom template:
APC_ENABLED=0 \
GMLX_CACHE_LIMIT_GB=1 \
GMLX_OVERCOMMIT=1 \
KV_BITS=8 \
KV_GROUP_SIZE=64 \
MAX_KV_SIZE=4096 \
QUANTIZED_KV_START=0 \
gmlx serve \
"/path/to/Qwen3.8-27B-UD-Q3_K_XL.gguf" \
--host 127.0.0.1 \
--port 8080 \
--foreground \
--no-menubar
Send this request:
curl -s http://127.0.0.1:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "qwen3.8-27b",
"messages": [{
"role": "user",
"content": "Explain why speculative decoding accelerates language-model inference, including one limitation, in one concise technical paragraph."
}],
"temperature": 1.0,
"max_tokens": 128,
"seed": 4242,
"stream": false
}' | jqActual result
The request succeeds at the HTTP level but returns corrupted, repetitive reasoning:
jid _ 8 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0...
Other Q8 runs produced patterns including:
CDATActrlctrlctrlctrlctrlctrlctrlctrl...
angep://://://://://://://://://://...
This is not a small quality regression. The output is unusable and generally repeats a token or short pattern until max_tokens.
F16 control
Start the same model without KV quantization:
GMLX_OVERCOMMIT=1 \
MAX_KV_SIZE=4096 \
gmlx serve \
"/path/to/Qwen3.8-27B-UD-Q3_K_XL.gguf" \
--host 127.0.0.1 \
--port 8080 \
--foreground \
--no-menubar
The identical request produces coherent reasoning. Measured control performance was approximately:
Prefill: 197 tok/s
Decode: 16.8-16.9 tok/s
Peak memory: 13.36 GB
Configuration matrix
| KV cache |
APC |
MTP |
Qwen implementation |
Result |
| F16 |
default |
off |
gmlx-owned |
Coherent |
| F16 |
default |
exact |
gmlx-owned |
Coherent |
| F16 |
default |
stochastic |
gmlx-owned |
Coherent |
| Q8 |
default |
exact |
gmlx-owned |
Repeating garbage |
| Q8 |
default |
stochastic |
gmlx-owned |
Repeating garbage |
| Q8 |
off |
off |
gmlx-owned |
Repeating garbage |
| Q8 |
default |
stochastic |
stock (GMLX_QWEN_OWNED=0) |
Server exception: tuple has no .shape |
This appears to rule out MTP, stochastic sampling, APC restoration, the custom Jinja template, prompt caching, and multi-request concurrency as the sole cause.
Stock-Qwen diagnostic
With the same Q8 settings plus:
the request fails instead of silently corrupting:
{
"error": {
"type": "server_error",
"message": "Generation failed: 'tuple' object has no attribute 'shape'"
}
}This resembles a mismatch between a quantized cache represented as a tuple and code expecting a dense array with a .shape attribute. The gmlx-owned path appears to avoid the exception but still produces invalid model state or output.
MTP control and performance
With F16 KV, native stochastic MTP remains coherent and provides a valid speedup:
Run 1: 24.3 tok/s
Run 2: 21.7 tok/s
Run 3: 23.3 tok/s
Mean: 23.1 tok/s
Median: 23.3 tok/s
Peak memory: approximately 14.17 GB
Therefore, native MTP itself works with this model when the KV cache is not quantized.
Expected behavior
Q8 KV should produce coherent output close to the F16-KV control. If quantized KV is not currently supported for this architecture or serving path, gmlx should reject the configuration at model load rather than silently return corrupted completions.
Possible relationship
Potentially related upstream issue:
That issue describes BatchQuantizedKVCache lacking extraction and merge behavior required by Qwen's hybrid per-row decode path. The reported consequence—losing prompt-cache state, resetting the offset, and producing identical logits or repeating tokens—closely resembles this reproduction.
However, #1322 is closed and concerns Qwen3.6 MoE. This report concerns Qwen3.8 dense through gmlx 0.4.6, so it may be a separate dense-model variant, a regression in gmlx's owned Qwen forward path, an affected dependency, or a continuous-batching integration issue.
Suggested safety behavior
Until the underlying path is fixed, it may be safer to fail model loading when quantized KV is enabled for this Qwen hybrid serving path. Silent coherent-looking corruption is significantly harder to detect than an explicit unsupported-configuration error.
This report was generated by an AI and reviewed for accuracy by a human who performed the tests.
Note from human: I'm not actually all that experienced with AI haha, but that seems like what happened. If you don't want me to use AI in future replies (if you do reply) lmk and I'll stop. love the project by the way, got a pretty nice 20-30% decode boost with this!
Qwen3.8 dense silently generates repeating garbage with quantized KV cache
Summary
On gmlx 0.4.6, enabling uniform 8-bit KV-cache quantization causes Qwen3.8-27B dense to generate deterministic repeating or corrupted output.
The corruption reproduces with:
The same GGUF, prompt, and sampling configuration produces coherent output when KV quantization is disabled.
The default gmlx-owned Qwen path silently generates corrupted text. Setting
GMLX_QWEN_OWNED=0instead exposes a cache-layout exception:This may be related to the quantized hybrid-cache extraction/merge problems discussed in mlx-vlm issue #1322, although that issue is marked fixed and covered Qwen3.6 MoE. This reproduction uses Qwen3.8 dense through gmlx.
Environment
gmlx doctoroutput:Model:
gmlx detects it as:
Minimal reproduction
Start a server with Q8 KV, APC disabled, MTP disabled, and no custom template:
APC_ENABLED=0 \ GMLX_CACHE_LIMIT_GB=1 \ GMLX_OVERCOMMIT=1 \ KV_BITS=8 \ KV_GROUP_SIZE=64 \ MAX_KV_SIZE=4096 \ QUANTIZED_KV_START=0 \ gmlx serve \ "/path/to/Qwen3.8-27B-UD-Q3_K_XL.gguf" \ --host 127.0.0.1 \ --port 8080 \ --foreground \ --no-menubarSend this request:
Actual result
The request succeeds at the HTTP level but returns corrupted, repetitive reasoning:
Other Q8 runs produced patterns including:
This is not a small quality regression. The output is unusable and generally repeats a token or short pattern until
max_tokens.F16 control
Start the same model without KV quantization:
GMLX_OVERCOMMIT=1 \ MAX_KV_SIZE=4096 \ gmlx serve \ "/path/to/Qwen3.8-27B-UD-Q3_K_XL.gguf" \ --host 127.0.0.1 \ --port 8080 \ --foreground \ --no-menubarThe identical request produces coherent reasoning. Measured control performance was approximately:
Configuration matrix
GMLX_QWEN_OWNED=0).shapeThis appears to rule out MTP, stochastic sampling, APC restoration, the custom Jinja template, prompt caching, and multi-request concurrency as the sole cause.
Stock-Qwen diagnostic
With the same Q8 settings plus:
the request fails instead of silently corrupting:
{ "error": { "type": "server_error", "message": "Generation failed: 'tuple' object has no attribute 'shape'" } }This resembles a mismatch between a quantized cache represented as a tuple and code expecting a dense array with a
.shapeattribute. The gmlx-owned path appears to avoid the exception but still produces invalid model state or output.MTP control and performance
With F16 KV, native stochastic MTP remains coherent and provides a valid speedup:
Therefore, native MTP itself works with this model when the KV cache is not quantized.
Expected behavior
Q8 KV should produce coherent output close to the F16-KV control. If quantized KV is not currently supported for this architecture or serving path, gmlx should reject the configuration at model load rather than silently return corrupted completions.
Possible relationship
Potentially related upstream issue:
That issue describes
BatchQuantizedKVCachelacking extraction and merge behavior required by Qwen's hybrid per-row decode path. The reported consequence—losing prompt-cache state, resetting the offset, and producing identical logits or repeating tokens—closely resembles this reproduction.However, #1322 is closed and concerns Qwen3.6 MoE. This report concerns Qwen3.8 dense through gmlx 0.4.6, so it may be a separate dense-model variant, a regression in gmlx's owned Qwen forward path, an affected dependency, or a continuous-batching integration issue.
Suggested safety behavior
Until the underlying path is fixed, it may be safer to fail model loading when quantized KV is enabled for this Qwen hybrid serving path. Silent coherent-looking corruption is significantly harder to detect than an explicit unsupported-configuration error.
This report was generated by an AI and reviewed for accuracy by a human who performed the tests.
Note from human: I'm not actually all that experienced with AI haha, but that seems like what happened. If you don't want me to use AI in future replies (if you do reply) lmk and I'll stop. love the project by the way, got a pretty nice 20-30% decode boost with this!