Eval bug: DFlash Speculative Decoding Crash: GGML_ASSERT(buffer) Failure

[bestbug456]

Name and Version

./llama-cli --version
WARNING: radv is not a conformant Vulkan implementation, testing use only.
version: 9849 (799fcc0)
built with GNU 11.4.0 for Linux x86_64

Operating systems

Linux

GGML backends

Vulkan

Hardware

AMD Radeon RX 9070

Models

unsloth/Qwen3.6-27B-Q4_K_M and williamliao/Qwen3.6-27B-DFlash-IQ4_XS

Problem description & steps to reproduce

When using DFlash speculative decoding (--spec-type draft-dflash) with context sizes non default with --cache-type-k and --cache-type-v q8_0 the server crash

#0  0x00007f3c9fa9fff2 in ?? () from /usr/lib/libc.so.6
#1  0x00007f3c9fa9403c in ?? () from /usr/lib/libc.so.6
#2  0x00007f3c9fa94084 in ?? () from /usr/lib/libc.so.6
#3  0x00007f3c9fb0494f in wait4 () from /usr/lib/libc.so.6
#4  0x00007f3ca0151c3b in ggml_print_backtrace () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libggml-base.so.0
#5  0x00007f3ca0151dd2 in ggml_abort () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libggml-base.so.0
#6  0x00007f3ca0169670 in ggml_backend_buffer_get_type () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libggml-base.so.0
#7  0x00007f3ca01696fd in ggml_backend_buffer_is_host () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libggml-base.so.0
#8  0x00007f3c9f12e1d7 in llama_kv_cache::set_input_k_rot(ggml_tensor*) const () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama.so.0
#9  0x00007f3c9f117d9e in llm_graph_input_attn_kv_iswa::set_input(llama_ubatch const*) () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama.so.0
#10 0x00007f3c9f11bd00 in llm_graph_result::set_inputs(llama_ubatch const*) () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama.so.0
#11 0x00007f3c9f0e7941 in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama.so.0
#12 0x00007f3c9f0ee60a in llama_context::decode(llama_batch const&) () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama.so.0
#13 0x00007f3c9f0efe30 in llama_decode () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama.so.0
#14 0x00007f3c9f6d7ce7 in common_speculative_impl_draft_dflash::process(llama_batch const&) () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama-common.so.0
#15 0x00007f3c9f6cbdbd in common_speculative_process(common_speculative*, llama_batch const&) () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama-common.so.0
#16 0x00007f3ca039381b in server_context_impl::decode(int&, int, llama_batch&) () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama-server-impl.so
#17 0x00007f3ca0395b1d in server_context_impl::update_slots() () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama-server-impl.so
#18 0x00007f3ca033e2f1 in server_queue::start_loop(long) () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama-server-impl.so
#19 0x00007f3ca02ddc12 in llama_server(int, char**) () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama-server-impl.so
#20 0x00007f3c9fa27741 in ?? () from /usr/lib/libc.so.6
#21 0x00007f3c9fa27879 in __libc_start_main () from /usr/lib/libc.so.6
#22 0x00005621fd94e295 in _start ()

I've try to analyze the crash with the support of a local LLM without using dflash. This is the result analysis written by the LLM used:

The Call Chain

1. common_speculative_impl_draft_dflash::process(batch) calls llama_decode(ctx_dft, batch) on the draft model context
2. llama_context::process_ubatch() builds and executes the computation graph
3. llm_graph_input_attn_kv_iswa::set_input() sets up graph inputs for attention
4. Inside set_input(), the code checks if (self_k_rot) and calls mctx->get_base()->set_input_k_rot(self_k_rot)
5. set_input_k_rot() calls ggml_backend_buffer_is_host(self_k_rot->buffer) — buffer is NULL

Why the Buffer is NULL

The self_k_rot tensor is built in src/llama-graph.cpp:2510:

inp->self_k_rot = mctx_cur->get_base()->build_input_k_rot(ctx0);

In src/llama-kv-cache.cpp:1314-1333:

ggml_tensor * llama_kv_cache::build_input_k_rot(ggml_context * ctx) const {
    ggml_tensor * res = nullptr;

    if (attn_rot_k) {
        int nrot = 64;
        do {
            nrot *= 2;
        } while (n_embd_head_k_all % nrot == 0);
        nrot /= 2;

        res = ggml_new_tensor_2d(ctx, GGML_TYPE_F32, nrot, nrot);
        ggml_set_input(res);
        ggml_set_name(res, "attn_inp_k_rot");
    }

    return res;
}

ggml_new_tensor_2d() creates a tensor with buffer == NULL. It is marked as an input (ggml_set_input) but is never allocated. In normal decoding, the graph allocation step happens before set_input_k_rot() is called, so the buffer is valid. In the DFlash path, this allocation is missing.

Why attn_rot_k is True

The attn_rot_k flag is set during KV cache initialization (llama-kv-cache.cpp:290-294):

attn_rot_k =
    !attn_rot_disable &&
    n_embd_head_k_all > 0 &&
    ggml_is_quantized(type_k) &&
    hparams.n_embd_head_k() % 64 == 0;

For the Qwen3.6-27B model with --cache-type-k q8_0:

• ggml_is_quantized(q8_0) → true
• n_embd_head_k() = 128, 128 % 64 == 0 → true
• attn_rot_disable → false (default)
• Result: attn_rot_k = true → tensor is created but not allocated

Why It Only Crashes at Context > 4096

At small contexts, the graph building and execution phases overlap within the same llama_decode() call, so the tensor happens to be allocated before set_input_k_rot() runs. At larger contexts (>4096), the graph execution path diverges — set_input_k_rot() is called before the tensor allocation step completes, exposing the NULL buffer.

Why Only DFlash (Not EAGLE/EAGLE3)

DFlash creates a separate ctx_dft (draft model context) with its own llama_kv_cache. The graph building happens during initial setup, but set_input_k_rot() is invoked later during llama_decode(ctx_dft) — between these phases, the tensor allocation is not guaranteed. EAGLE and EAGLE3 use different graph paths that don't trigger llm_graph_input_attn_kv_iswa::set_input().

I try to dig in order to understand what are the possible workaround (beside removing the kv parameter) and I can confirm that if set the environment variable LLAMA_ATTN_ROT_DISABLE=1 before launching llama-server resolve temporary the crash (paying the fact that now the kv does not have the Walsh-Hadamard rotation). I can confirm that also doing this resolve the crash.

I then try to understand what are the possible fix on the codebase to try to resolve, again with the help of the LLM. This is the output:

Option A: Ensure Proper Allocation (Preferred)

In common/speculative.cpp, in common_speculative_impl_draft_dflash::process(), ensure the graph is allocated before the first llama_decode(ctx_dft):

// Before calling llama_decode(ctx_dft, ...), ensure graph inputs are allocated
// This may require calling a graph finalization step on ctx_dft

Option B: Defensive Null Check (Quick Fix)

In src/llama-kv-cache.cpp:1673-1674, add a null guard:

void llama_kv_cache::set_input_k_rot(ggml_tensor * dst) const {
    if (dst->buffer == nullptr) return;  // skip if not yet allocated
    GGML_ASSERT(ggml_backend_buffer_is_host(dst->buffer));
    ...
}

This is less ideal because it silently skips the rotation data copy, but it prevents the crash.

At this point I would like a confirmation from someone else about this finding and the resolution fix. If is confirmed I'm willing to try to propose a patch but I would like to understand where is the limit about using the LLM for the patch, like what can and what can not be used for since my knowledge in cpp is very little.

First Bad Commit

No response

Relevant log output

Logs

0x00007f3c9fa9fff2 in ?? () from /usr/lib/libc.so.6
#0  0x00007f3c9fa9fff2 in ?? () from /usr/lib/libc.so.6
#1  0x00007f3c9fa9403c in ?? () from /usr/lib/libc.so.6
#2  0x00007f3c9fa94084 in ?? () from /usr/lib/libc.so.6
#3  0x00007f3c9fb0494f in wait4 () from /usr/lib/libc.so.6
#4  0x00007f3ca0151c3b in ggml_print_backtrace () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libggml-base.so.0
#5  0x00007f3ca0151dd2 in ggml_abort () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libggml-base.so.0
#6  0x00007f3ca0169670 in ggml_backend_buffer_get_type () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libggml-base.so.0
#7  0x00007f3ca01696fd in ggml_backend_buffer_is_host () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libggml-base.so.0
#8  0x00007f3c9f12e1d7 in llama_kv_cache::set_input_k_rot(ggml_tensor*) const () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama.so.0
#9  0x00007f3c9f117d9e in llm_graph_input_attn_kv_iswa::set_input(llama_ubatch const*) () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama.so.0
#10 0x00007f3c9f11bd00 in llm_graph_result::set_inputs(llama_ubatch const*) () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama.so.0
#11 0x00007f3c9f0e7941 in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama.so.0
#12 0x00007f3c9f0ee60a in llama_context::decode(llama_batch const&) () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama.so.0
#13 0x00007f3c9f0efe30 in llama_decode () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama.so.0
#14 0x00007f3c9f6d7ce7 in common_speculative_impl_draft_dflash::process(llama_batch const&) () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama-common.so.0
#15 0x00007f3c9f6cbdbd in common_speculative_process(common_speculative*, llama_batch const&) () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama-common.so.0
#16 0x00007f3ca039381b in server_context_impl::decode(int&, int, llama_batch&) () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama-server-impl.so
#17 0x00007f3ca0395b1d in server_context_impl::update_slots() () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama-server-impl.so
#18 0x00007f3ca033e2f1 in server_queue::start_loop(long) () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama-server-impl.so
#19 0x00007f3ca02ddc12 in llama_server(int, char**) () from /home/bestbug/Documents/llama-vulkan/llama-b9849/libllama-server-impl.so
#20 0x00007f3c9fa27741 in ?? () from /usr/lib/libc.so.6
#21 0x00007f3c9fa27879 in __libc_start_main () from /usr/lib/libc.so.6
#22 0x00005621fd94e295 in _start ()

[liminfei-amd]

I ran into the same GGML_ASSERT(buffer) on an RX 9070 (gfx1201, Vulkan) with a DFlash draft and --cache-type-k q8_0 --cache-type-k-draft q8_0, and it looks like a missing buffer guard rather than anything DFlash-specific.

set_input_k_rot / set_input_v_rot in set_input (both llm_graph_input_attn_kv and the ISWA variant) are only guarded by if (self_k_rot). That tensor's buffer is left unallocated when a graph stores K/V without attending — the same case the neighbouring kq_mask inputs already handle (their comment even names the DFlash KV-injection pass). set_input_k_rot then calls ggml_backend_buffer_is_host(dst->buffer) on a NULL buffer and asserts.

I opened #25215, which adds the same && ->buffer guard to the four k_rot/v_rot inputs. Before: abort on the first draft decode; after: it runs to completion with speculative decoding still active, and a regular causal (non-DFlash) model is unaffected. Happy to adjust it, or if you'd rather fold the guard into the DFlash work directly that's fine by me.

[bestbug456]

Hello @liminfei-amd I keep an eye when the PR is merged to see if the problem is indeed resolved, thanks for taking time to answer to my question! :)

[ruixiang63]

Can you confirm if the PR resolves the issue from your side as well? @bestbug456

[bestbug456]

Hello everyone, yes indeed now I can have the kv quantized! Thanks a lot, now I've to figure out why with dflash I'm yielding a -50% in performance, but this is another story. Thanks @liminfei-amd for the patch! :)

[liminfei-amd]

Thanks for the follow-up, @bestbug456 — glad KV-quantized is working now!

Just to confirm for the record: did the #25215 patch resolve the GGML_ASSERT(buffer) crash for you (i.e. you're on a build with that fix applied)? A clear confirmation would help @ruixiang63 decide how to proceed with #25215. The -50% dflash perf you mention sounds like a separate issue — probably best tracked in its own report.

Thanks again!

[bestbug456]

@liminfei-amd yes apologies for lack of clarity, I indeed when on your proposal, build and verify that the crash was gone (and indeed it was the case). I'm also aware that the problem that I'm having is not related to this specific report :)

Thanks to you! :)

[ruixiang63]

After merging PR #25823, the quantized KV cache should achieve an acceptance rate similar to the non-quantized version.