Eval bug: GGML_ASSERT in ggml_cuda_op_concat with DeepSeek V4 + quantized KV cache (non-contiguous src)

[adubovikov]

Name and Version

version: 9939 (92b187c)
built with GNU 13.3.0 for Linux aarch64

Also reproduced on a build from 7a63fde. The backtrace below is from a build from early July, where the same condition trips a slightly different assert (see "Problem description").

Operating systems

Linux

GGML backends

CUDA, RPC

Hardware

2x NVIDIA GB10 (aarch64, CUDA 13), 2-node RPC cluster

Models

DeepSeek-V4-Flash UD-Q4_K_XL (unsloth GGUF)

Problem description & steps to reproduce

Enabling KV cache quantization (--cache-type-k q8_0 --cache-type-v q8_0) for DeepSeek V4 crashes the CUDA backend during prompt processing. The same setup with default f16 KV cache works.

Root cause: the DeepSeek V4 graph (#24162, lightning indexer #24231) feeds a non-contiguous view of the quantized KV cache into GGML_OP_CONCAT. The quantized branch of ggml_cuda_op_concat (ggml/src/ggml-cuda/concat.cu) only supports contiguous inputs: on current master this trips GGML_ASSERT(ggml_is_contiguous(src1)); on the early-July build from the backtrace below the same case reaches the assert inside concat_cuda (GGML_ASSERT(!ggml_is_quantized(src0->type)), concat.cu:207). The non-quantized branch handles non-contiguous inputs fine.

The CUDA supports_op mirrors the contiguity requirement, but the RPC backend's supports_op returns true unconditionally, so on an RPC worker the op reaches CUDA and aborts. Note the CPU backend has the same assert in ggml_compute_forward_concat, so a single-node setup where the scheduler falls back to CPU for this op would likely abort as well (not verified).

Repro:

# worker:
./build/bin/ggml-rpc-server -H 10.10.10.2 -p 50052

# head:
./build/bin/llama-server \
  --model DeepSeek-V4-Flash-UD-Q4_K_XL-00001-of-00005.gguf \
  --rpc 10.10.10.2:50052 \
  --ctx-size 196608 --parallel 2 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --no-mmap --jinja --port 8080

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model":"x","messages":[{"role":"user","content":"hello"}]}'

The worker aborts, then the head node fails in ggml_backend_rpc_buffer_get_tensor.

I have a working fix on a branch: pack non-contiguous quantized inputs into a contiguous scratch buffer first (a single cudaMemcpy2DAsync when the row stride is uniform, otherwise a small packing kernel in one launch), then reuse the existing byte-level concat path; the contiguity asserts are relaxed to ggml_is_contiguous_rows (CUDA, supports_op, and the CPU backend, whose generic path already handles strided rows). With it:

• test-backend-ops -o CONCAT passes 252/252, including new non-contiguous quantized cases, on GB10 (aarch64, CUDA 13) and RTX 4070 (x86_64, CUDA 13.1);
• the setup above survives 33K-token prompts and multi-turn conversations, prompt processing stays at ~194 t/s at 33K ctx (vs f16 KV cache degrading 210 -> 82 t/s past ~18K on the same cluster).

Branch: https://github.com/adubovikov/llama.cpp/tree/cuda-concat-noncont-quant — happy to open a PR if the approach looks acceptable.

Disclosure: this report and the fix branch were prepared with heavy AI (LLM agent) assistance; the debugging, reproduction and benchmarks were run on real hardware as described. Given the project's AI usage policy, feel free to treat the branch as a reference for a maintainer-authored fix instead of a PR.

First Bad Commit

Not bisected; the op pattern comes with DeepSeek V4 support (#24162).

Relevant log output

Logs

ggml_backend_cuda_graph_compute: CUDA graph warmup complete
/home/shurik/llama.cpp/ggml/src/ggml-cuda/concat.cu:207: GGML_ASSERT(!ggml_is_quantized(src0->type)) failed
[New LWP 4511]
[New LWP 4507]
...
#0  0x0000f7a805cd7b74 in __GI___wait4 (pid=4528, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
#1  0x0000b9542e95595c in ggml_print_backtrace ()
#2  0x0000b9542e955b00 in ggml_abort ()
#3  0x0000b9542e63b50c in ggml_cuda_op_concat(ggml_backend_cuda_context&, ggml_tensor*) ()
#4  0x0000b9542df4d66c in ggml_backend_cuda_graph_compute(ggml_backend*, ggml_cgraph*) ()
#5  0x0000b9542e96df10 in ggml_backend_graph_compute ()
#6  0x0000b9542e94b680 in rpc_server::graph_compute(std::vector<unsigned char, std::allocator<unsigned char> > const&) ()
#7  0x0000b9542e94fca4 in rpc_serve_client(std::vector<ggml_backend*, std::allocator<ggml_backend*> > const&, char const*, std::shared_ptr<socket_t>) ()
#8  0x0000b9542e952030 in ggml_backend_rpc_start_server ()
#9  0x0000b9542de24a08 in main ()
[Inferior 1 (process 4506) detached]
./notmux-spark2-eae8-start-worker.sh: line 12:  4506 Aborted                 (core dumped) $HOME/llama.cpp/ggml-rpc-server -H $BIND_HOST -p $PORT -c

[fairydreaming]

This is my fault, it didn't occur to me that with multiple streams concatenated KV cache tensors wouldn't be contiguous. 😢

But I think it may be fixed in a simpler way. The current CUDA implementation already iterates over last dimension if concat dim != 3, so we can simply relax tensor contiguity requirement for this case to the first 3 dimensions and everything should work without any major changes (boolean conditions that currently force contiguity would need a split into two cases, contiguity required for dim == 3 and relaxed case for dim != 3).

But in GGML I currently don't see a way to easily check for contiguity of N inner dimensions. @ggerganov I'd need something like:

// returns the number of contiguous inner dimensions
GGML_API int  ggml_n_dims_contiguous(const struct ggml_tensor * tensor);

or generalized equivalent of ggml_is_contiguous_n() for this case like ggml_is_contiguous_m_n(). Let me know what would be the preferred solution.

Edit: maybe a pair of:

GGML_API int  ggml_n_dims_contiguous_inner(const struct ggml_tensor * tensor);
GGML_API int  ggml_n_dims_contiguous_outer(const struct ggml_tensor * tensor); # ok maybe not a good idea if GGML_MAX_DIMS is ever going to change

[adubovikov]

the current patch in my repo is stable. Since 3 days no crash.

[ggerganov]

@fairydreaming ggml_is_contiguous_m_n() sounds good.