Eval bug: llama.cpp (AMD GPU): partial layers run on CPU when loading dflash model, cannot fully offload to GPU

[rykerzhou]

Name and Version

.\llama-cli.exe --version
ggml_cuda_init: found 1 ROCm devices (Total VRAM: 89976 MiB):
Device 0: AMD Radeon(TM) 8060S Graphics, gfx1151 (0x1151), VMM: no, Wave Size: 32, VRAM: 89976 MiB
version: 9459 (07ac3ce)
built with Clang 22.0.0 for Windows AMD64

Operating systems

Windows

GGML backends

HIP

Hardware

OS: Microsoft Windows 11 专业版 10.0.26200 26200
CPU: AMD RYZEN AI MAX+ 395 w/ Radeon 8060S
GPU: AMD Radeon(TM) 8060S Graphics 4293918720 32.0.31007.5012

Models

Qwen3.6-27B-Q5_K_S.gguf
Qwen3.6-27B-DFlash-Q4_K_M.gguf

Problem description & steps to reproduce

.\llama-server.exe -m C:\Users\xh\Desktop\work\code\novamax\data\models_dir\llm\unsloth\Qwen3.6-27B-GGUF\Qwen3.6-27B-Q5_K_S.gguf --mmproj C:\Users\xh\Desktop\work\code\novamax\data\models_dir\llm\unsloth\Qwen3.5-27B-GGUF\mmproj-BF16.gguf --no-mmproj-offload --spec-draft-model C:\Users\xh\Downloads\Qwen3.6-27B-DFlash-Q4_K_M.gguf --spec-type dflash --spec-dflash-cross-ctx 1024 --host 0.0.0.0 --port 1234 --parallel 1 --kv-unified --n-gpu-layers all --spec-draft-ngl all -b 2048 -ub 512 --ctx-size 102400 --cache-type-k q5_0 --cache-type-v q4_1 --flash-attn on --cache-ram 0 --jinja --no-mmap --mlock --temperature 0.6 --reasoning off --top-p 1.0 --top-k 20 --min-p 0.0 --spec-draft-n-max 8

On an AMD GPU system, when loading and running dflash model with llama.cpp, only part of the model layers are offloaded to GPU, while the rest still execute on CPU. GPU usage is partial and CPU load remains high, resulting in slower inference speed.

dflash_model.log

First Bad Commit

No response

Relevant log output

Logs

[Anbeeld]

Enabled fused GDN probing on ROCm, should help with AMD performance, but it's experimental for now as I don't own AMD GPU myself and can't test it. Please try latest v0.3.0 and report back.

[ElioDonato]

Name and Version

llama-server --version
version: 10026 (8b00fe115)
built with GNU 15.2.1 for Linux x86_64

Operating System

• Distribution: Artix Linux (runit)
• Kernel: Linux llm-server 7.0.10-artix1-1 #1 SMP PREEMPT_DYNAMIC Sat, 23 May 2026 18:01:41 +0000 x86_64 GNU/Linux

GGML Backend

HIP (ROCm)

Hardware

• CPU: AMD RYZEN AI MAX+ 395 w/ Radeon 8060S
• GPU: AMD Radeon(TM) 8060S Graphics (gfx1151)

Models

• Target: Qwen3.6-27B-Q5_K_S.gguf
• Drafter: Qwen3.6-27B-DFlash-Q5_K_M.gguf

Problem Description

During generation with dflash speculative decoding, the GPU backend fails for key operations (allocate_prefill_gpu, allocate_hidden_gpu, allocate_tape_gpu), falling back to CPU and causing very low generation speed. The same hardware and models worked fine with an older build of beellama.cpp.

Steps to Reproduce

Build (using Strix Halo Toolboxes ROCm nightlies as a base)

cmake -S . -B build \
  -DGGML_HIP=ON \
  -DAMDGPU_TARGETS=gfx1151 \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_RPC=ON \
  -DLLAMA_HIP_UMA=ON

cmake --build build --config Release -- -j$(nproc)
cmake --install build --config Release

Container Run (Podman)

podman run --rm -it \
  --name llama-rocm7-nightlies \
  --network host \
  --device=/dev/dri --device=/dev/kfd \
  --group-add=video --group-add=render \
  --security-opt=seccomp=unconfined \
  -v /home/llm/models:/models \
  -v /home/llm/.cache/llama-cpp:/.cache \
  llama-rocm7-nightlies

Server Command

llama-server \
  --models-preset /models/config.ini \
  --host 0.0.0.0 --port 8080 \
  --models-max 4 --no-webui \
  -cram -1 -lcd /.cache \
  --no-mmap -fa on -dio

Configuration (config.ini excerpt)

[Qwen3.6-large]
load-on-startup = true
model = /models/Qwen3.6-27B-GGUF/Qwen3.6-27B-Q5_K_S.gguf
spec-draft-model = /models/Qwen3.6-27B-DFlash-GGUF/Qwen3.6-27B-DFlash-Q5_K_M.gguf
spec-type = dflash
spec-dflash-cross-ctx = 1024
spec-draft-ngl = all
ctx-size = 262144
parallel = 2
kv-unified = true
ctk = turbo4
ctv = turbo4

Log Excerpt (showing the failure)

[49689] 0.50.119.798 W allocate_prefill_gpu: no GPU backend for prefill layer 1 device ROCm0; using callback fallback
[49689] 0.50.834.311 W dflash: accepted target-hidden full-KV commit failed; clearing drafter KV suffix for seq=0 start=14
[49689] 0.50.834.348 W dflash: drafter K/V projection cache unavailable; using full-window K/V projection
[49689] 0.51.149.197 W allocate_hidden_gpu: no GPU backend for hidden layer 1 device ROCm0; using callback hidden fallback
[49689] 0.51.149.202 W allocate_tape_gpu: no GPU backend for recurrent layer 0 device ROCm0; falling back to CPU tape
...
[49689] 0.59.382.088 I slot print_timing: prompt eval time =  1148.87 ms / 267 tokens (232.40 t/s)
[49689] 0.59.382.092 I slot print_timing:        eval time =  8168.80 ms /  67 tokens (  8.20 t/s)
[49689] 0.59.382.099 I slot print_timing: draft acceptance = 0.05586 (31 accepted / 555 generated)
...
[49689] 0.59.129.524 W allocate_tape_gpu: no GPU backend for recurrent layer 0 device ROCm0; falling back to CPU tape
[49689] 0.59.382.088 I slot print_timing: id  0 | task 0 | prompt eval time =    1148.87 ms /   267 tokens (    4.30 ms per token,   232.40 tokens per second)
[49689] 0.59.382.092 I slot print_timing: id  0 | task 0 |        eval time =    8168.80 ms /    67 tokens (  121.92 ms per token,     8.20 tokens per second)
[49689] 0.59.382.092 I slot print_timing: id  0 | task 0 |       total time =    9317.66 ms /   334 tokens
[49689] 0.59.382.098 I slot print_timing: id  0 | task 0 |    graphs reused =         36
[49689] 0.59.382.099 I slot print_timing: id  0 | task 0 | draft acceptance = 0.05586 (   31 accepted /   555 generated)
[49689] 0.59.382.100 I slot print_timing: id  0 | task 0 | adaptive dm: fringe=0.00 n_max=15
[49689] 0.59.406.898 I statistics           dflash: #calls(b,g,a) =    1     37     37, #gen drafts =     37, #acc drafts =    20, #gen tokens =    555, #acc tokens =    31, dur(b,g,a) = 0.002, 829.354, 0.005 ms
[49689] 0.59.407.103 I slot      release: id  0 | task 0 | stop processing: n_tokens = 335, truncated = 0
[49689] 0.59.407.158 I srv  update_slots: all slots are idle

[Anbeeld]

@ElioDonato thanks for the report, a misalignment appeared after upstream changed how they gate stuff by GPU presence checks. Should be fixed in the latest v0.3.0, please try it.