Eval bug: Decreased perfomance because of input layers on CPU for AMD Strix Halo

[Vasili-Sk]

Name and Version

version 1 (1705d43)
build with Clang 23.0.0 for Windows

Operating systems

Windows

GGML backends

HIP

Hardware

AMD Strix Halo 128GB

Models

Qwen 3.6 35B MoE Q8

Problem description & steps to reproduce

I was fighting with 30% cpu usage and decreased GPU utilization on AMD Strix halo machine when running Qwen 3.6 35B MoE Q8 generation on ROCm. Vulkan does not have that problem and needs only ~5% cpu to run tg.
GLM found this issue in src/llama-model.cpp around line 1300:

// there is very little benefit to offloading the input layer, so always keep it on the CPU
pimpl->dev_input = { cpu_dev, &pimpl->cpu_buft_list };

This keeps tok_embd on CPU even with -ngl 99, which forces GET_ROWS(tok_embd, inp_tokens) onto the CPU backend at the start of every forward pass. On Stix Halo where CPU and GPU share same DDR5 it hurts a lot.

Possible fix is to mirror the output-layer policy for the input layer when full offload is requested.

Before fix:
41.33 t/s tg (30% CPU usage)

After full GPU offload:
47.24 t/s tg (2% CPU usage)

First Bad Commit

No response

Relevant log output

n/a

[Mushoz]

Oh this would explain why ROCm is behind compared to Vulkan on token generation speeds. Good find!

[0cc4m]

Why would this affect ROCm but not Vulkan? It also runs on CPU for Vulkan then.

[pwilkin]

Dunno, different buffer management? Might be a similar issue as to how --mmap is unusable on ROCm right now while it works fine on Vulkan.

[Vasili-Sk]

Sorry im not that yet deep into llm engine programming, but this is what my llm says:
Vulkan does not suffer because its async H2D copy actually works (copy_from_host=true), so 2nd split only adds minimal overhead.
Rocm HIP backend hardcodes copy_from_host=false (with comment 'this change is not beneficial for hip') forcing a fallback path that does sync(source) + sync(dst) + copy per input tensor - 10 inputs * 2 syncs = 20 syncs per token, what kills TG perfomance.

[pwilkin]

Yeah, makes sense. Basically, it makes sense to only offload input tensors to CPU for non-integrated devices.

[Vasili-Sk]

Dunno, different buffer management? Might be a similar issue as to how --mmap is unusable on ROCm right now while it works fine on Vulkan.

there are few issues there too, from my attempts to hack around windows memory limits i found that mapping actually does not map but copies memory to gpu area, doubling memory usage basically

[0cc4m]

Yeah, makes sense. Basically, it makes sense to only offload input tensors to CPU for non-integrated devices.

First step should be to check the ROCm backend's assumption about async copies, not conditional llama.cpp core changes.

[pwilkin]

Why would you not want to offload input tensors to the device in case of iGPU? The only rationale for not doing that is to save memory, that doesn't apply in case of iGPU systems.

[0cc4m]

Why would you not want to fix async behaviour of the backend? I'm not saying this should not be looked at, just that first priority for fixes is always within the backend.