Eval bug: HIP/ROCm on gfx1151, wrong logits (not a crash), triggered by prompts longer than n_ubatch.

[leok7v]

Name and Version

version: 0.3.0-dev (build 10729, commit 458681e)
built with GNU 13.3.0 for Linux x86_64

Operating systems

Linux

GGML backends

HIP

Hardware

CPU AMD Ryzen AI MAX+ 395 w/ Radeon 8060S (Strix Halo APU)
GPU Radeon 8060S Graphics, gfx1151 (RDNA 3.5, integrated)
VRAM 98304 MiB carved out to the GPU
ROCm 7.2.1
HIP 7.2.53211-e1a6bc5663
compiler AMD clang version 22.0.0git (roc-7.2.1 26084 f58b06dce1f9c15707c5f808fd002e18c2accf7e)
kernel Linux 7.0.0-30-generic x86_64
distro Ubuntu 24.04.4 LTS

build:
cmake -B build -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 -DCMAKE_BUILD_TYPE=Release
cmake --build build

HIP compile flags actually used:
-O3 -DNDEBUG -std=gnu++17 --offload-arch=gfx1151 -fPIC

Vulkan build used for comparison (same machine, same GPU):
cmake -B build-vk -DGGML_VULKAN=ON -DCMAKE_BUILD_TYPE=Release

Models

https://huggingface.co/unsloth/Qwen3.5-0.8B-GGUF (Qwen3.5-0.8B-Q4_K_M.gguf, 508 MB)
https://huggingface.co/google/gemma-4-E2B-it-qat-q4_0-gguf (gemma-4-E2B_q4_0-it.gguf, 3.3 GB)

Also observed, same machine, same HIP build:

• gemma-4-E2B-it BF16, Q8_0, Q4_K_M and Q4_0, all four from
  https://huggingface.co/unsloth/gemma-4-E2B-it-GGUF
• gemma-4-31B-it Q4_K_M (a local file; its size does not match the copy
  currently published by unsloth, so I am not citing a URL for it)

The BF16 case is the useful one for narrowing: BF16 involves no
dequantisation, so this is not a quantised-matmul kernel. Perplexity from
llama-perplexity on WikiText-2, same 2-item set, default n_ubatch, HIP
against Vulkan:

    BF16      84.7476  vs  13.8167
    Q8_0     110.2795  vs  14.4350
    Q4_K_M    92.7499  vs  12.9609
    Q4_0     112.7851  vs  12.1284

The 31B case is how this was first noticed: it read ppl 93.86 on HIP against
7.40 on Vulkan, which looks like a weak model rather than a broken backend.

Problem description & steps to reproduce

Any prompt longer than n_ubatch produces badly wrong logits. No error, no
warning, no crash: the numbers look plausible. Default n_ubatch is 512, so in
practice almost every real prompt is affected.

repro.cpp

// Minimal reproducer, public llama.h only.
//
// Decodes one prompt twice with two contexts that differ in exactly one
// parameter: n_ubatch. Everything else is identical, including the token ids.
// A correct backend must return the same distribution both times.
//
//   ./download.sh
//   ./repro /tmp/Qwen3.5-0.8B-Q4_K_M.gguf 99      qwen35 architecture
//   ./repro /tmp/gemma-4-E2B_q4_0-it.gguf 99      gemma4 architecture
//
// Pass 0 as the second argument to run on CPU.
//
// Compares the logits at every position and prints how often the two runs
// disagree about the most likely next token, the mean and worst KL divergence
// per position, and the largest absolute logit difference.

#include "llama.h"

#include <cmath>
#include <cstdio>
#include <cstring>
#include <algorithm>
#include <string>
#include <vector>

static std::vector<llama_token> make_prompt(const llama_vocab * vocab,
                                            int want) {
    std::string text = "The quick brown fox jumps over the lazy dog. ";
    while ((int) text.size() < want * 6) {
        text += "Numbers and words follow in a long unremarkable sentence. ";
    }
    std::vector<llama_token> out(want * 4);
    const int n = llama_tokenize(vocab, text.c_str(), (int32_t) text.size(),
                                 out.data(), (int32_t) out.size(), true, false);
    out.resize(n > want ? want : (n < 0 ? 0 : n));
    return out;
}

static std::vector<float> run(llama_model * model,
                              const std::vector<llama_token> & toks,
                              uint32_t n_batch, uint32_t n_ubatch, int ngl) {
    llama_context_params cp = llama_context_default_params();
    cp.n_ctx     = 4096;
    cp.n_batch   = n_batch;
    cp.n_ubatch  = n_ubatch;
    cp.n_threads = 8;
    (void) ngl;
    llama_context * ctx = llama_init_from_model(model, cp);
    std::vector<float> last;
    if (ctx != nullptr) {
        const int n = (int) toks.size();
        llama_batch batch = llama_batch_init(n, 0, 1);
        batch.n_tokens = n;
        for (int i = 0; i < n; ++i) {
            batch.token   [i]    = toks[i];
            batch.pos     [i]    = i;
            batch.n_seq_id[i]    = 1;
            batch.seq_id  [i][0] = 0;
            batch.logits  [i]    = 1;
        }
        const int rc = llama_decode(ctx, batch);
        if (rc == 0) {
            const llama_vocab * v = llama_model_get_vocab(model);
            const int n_vocab = llama_vocab_n_tokens(v);
            last.resize((size_t) n * (size_t) n_vocab);
            for (int i = 0; i < n; ++i) {
                const float * lg = llama_get_logits_ith(ctx, i);
                if (lg != nullptr) {
                    std::copy(lg, lg + n_vocab,
                              last.begin() + (size_t) i * n_vocab);
                }
            }
        } else {
            fprintf(stderr, "llama_decode failed: %d\n", rc);
        }
        llama_batch_free(batch);
        llama_free(ctx);
    }
    return last;
}

static double kl_div(const std::vector<float> & p, const std::vector<float> & q) {
    double mp = p[0];
    double mq = q[0];
    for (size_t i = 1; i < p.size(); ++i) {
        if (p[i] > mp) { mp = p[i]; }
        if (q[i] > mq) { mq = q[i]; }
    }
    double sp = 0.0;
    double sq = 0.0;
    for (size_t i = 0; i < p.size(); ++i) {
        sp += std::exp((double) p[i] - mp);
        sq += std::exp((double) q[i] - mq);
    }
    const double lp = mp + std::log(sp);
    const double lq = mq + std::log(sq);
    double kl = 0.0;
    for (size_t i = 0; i < p.size(); ++i) {
        const double a = (double) p[i] - lp;
        const double b = (double) q[i] - lq;
        kl += std::exp(a) * (a - b);
    }
    return kl;
}

static int argmax(const std::vector<float> & v) {
    int best = 0;
    for (size_t i = 1; i < v.size(); ++i) {
        if (v[i] > v[best]) { best = (int) i; }
    }
    return best;
}

int main(int argc, char ** argv) {
    int status = 1;
    if (argc < 2) {
        fprintf(stderr, "usage: %s <model.gguf> [n_gpu_layers]\n", argv[0]);
    } else {
        const int ngl = argc > 2 ? atoi(argv[2]) : 99;
        llama_backend_init();
        llama_model_params mp = llama_model_default_params();
        mp.n_gpu_layers = ngl;
        llama_model * model = llama_model_load_from_file(argv[1], mp);
        if (model == nullptr) {
            fprintf(stderr, "failed to load %s\n", argv[1]);
        } else {
            const llama_vocab * vocab = llama_model_get_vocab(model);
            const std::vector<llama_token> toks = make_prompt(vocab, 768);
            printf("model      %s\n", argv[1]);
            printf("n_gpu_layers %d\n", ngl);
            printf("prompt     %zu tokens\n", toks.size());
            const std::vector<float> a = run(model, toks, 2048, 2048, ngl);
            const std::vector<float> b = run(model, toks, 2048,  512, ngl);
            if (a.size() == b.size() && !a.empty()) {
                const int nv = llama_vocab_n_tokens(vocab);
                const int np = (int) (a.size() / (size_t) nv);
                double worst = 0.0;
                double kl    = 0.0;
                double klmax = 0.0;
                int    diff  = 0;
                for (int t = 0; t < np; ++t) {
                    const std::vector<float> pa(a.begin() + (size_t) t * nv,
                                                a.begin() + (size_t) (t + 1) * nv);
                    const std::vector<float> pb(b.begin() + (size_t) t * nv,
                                                b.begin() + (size_t) (t + 1) * nv);
                    for (int i = 0; i < nv; ++i) {
                        const double d = std::fabs((double) pa[i] - (double) pb[i]);
                        if (d > worst) { worst = d; }
                    }
                    const double k = kl_div(pa, pb);
                    kl += k;
                    if (k > klmax) { klmax = k; }
                    if (argmax(pa) != argmax(pb)) { diff++; }
                }
                kl /= (double) np;
                const int ia = 0;
                const int ib = 0;
                (void) ia; (void) ib;
                printf("positions compared      %d\n", np);
                printf("argmax differs at       %d of %d positions (%.2f%%)\n",
                       diff, np, 100.0 * diff / np);
                printf("mean KL per position    %.8f\n", kl);
                printf("worst KL at a position  %.8f\n", klmax);
                printf("\nn_ubatch 2048 (prompt fits, no split)\n");
                printf("n_ubatch  512 (prompt is split into 2 micro-batches)\n");
                printf("max |logit difference|  %.6f\n", worst);
                printf("\n%s\n", kl > 0.01
                       ? "FAIL: n_ubatch changed the distribution."
                       : "ok: n_ubatch left the distribution intact.");
                status = kl > 0.01 ? 2 : 0;
            } else {
                fprintf(stderr, "decode produced no logits\n");
            }
            llama_model_free(model);
        }
        llama_backend_free();
    }
    return status;
}

CMakeLists.txt

cmake_minimum_required(VERSION 3.14)
project(repro CXX)
set(LLAMA_BUILD_COMMON   OFF CACHE BOOL "" FORCE)
set(LLAMA_BUILD_TESTS    OFF CACHE BOOL "" FORCE)
set(LLAMA_BUILD_TOOLS    OFF CACHE BOOL "" FORCE)
set(LLAMA_BUILD_EXAMPLES OFF CACHE BOOL "" FORCE)
set(LLAMA_BUILD_SERVER   OFF CACHE BOOL "" FORCE)
add_subdirectory(llama.cpp)
add_executable(repro repro.cpp)
target_link_libraries(repro PRIVATE llama)
target_compile_features(repro PRIVATE cxx_std_17)

curl -L -C - -o /tmp/Qwen3.5-0.8B-Q4_K_M.gguf \
  https://huggingface.co/unsloth/Qwen3.5-0.8B-GGUF/resolve/main/Qwen3.5-0.8B-Q4_K_M.gguf

curl -L -C - -o /tmp/gemma-4-E2B_q4_0-it.gguf \
  https://huggingface.co/google/gemma-4-E2B-it-qat-q4_0-gguf/resolve/main/gemma-4-E2B_q4_0-it.gguf

ls -l /tmp/Qwen3.5-0.8B-Q4_K_M.gguf /tmp/gemma-4-E2B_q4_0-it.gguf

./repro /tmp/Qwen3.5-0.8B-Q4_K_M.gguf 99      qwen35 architecture
./repro /tmp/gemma-4-E2B_q4_0-it.gguf 99      gemma4 architecture

Two models of different architectures, 768-token prompt, n_ubatch 2048
against 512.

/tmp/Qwen3.5-0.8B-Q4_K_M.gguf (qwen35), from
https://huggingface.co/unsloth/Qwen3.5-0.8B-GGUF

backend                     argmax differs   mean KL      worst KL   max|dlogit|
HIP (upstream)              15/768  1.95%    0.07477930   7.8480584    14.109375
Vulkan                       0/768  0.00%    0.00000187   0.0001609     0.304688
CPU                          0/768  0.00%    0.00000000   0.0000000     0.000000
HIP + one-line fix           0/768  0.00%    0.00048698   0.0728900     1.828125

/tmp/gemma-4-E2B_q4_0-it.gguf (gemma4), from
https://huggingface.co/google/gemma-4-E2B-it-qat-q4_0-gguf

backend                     argmax differs   mean KL      worst KL   max|dlogit|
HIP (upstream)             170/768 22.14%    1.17912350  18.3745292    40.190187
Vulkan                       0/768  0.00%    0.00000318   0.0004254     0.877997
HIP + one-line fix           1/768  0.13%    0.00008664   0.0066724     2.086188

Both architectures are affected and both are fixed by the same one-line
change. gemma-4 is the worse of the two: upstream HIP picks a different most
likely token at 22 percent of positions. CPU and Vulkan are bit-exact or near
it across the split.

This is not specific to how any one program calls the API. In
src/llama-context.cpp:

cparams.n_ubatch = std::min(cparams.n_batch, params.n_ubatch == 0 ? params.n_batch : params.n_ubatch);

n_ubatch defaults to 512. llama_kv_cache::init_batch then splits the batch
by n_ubatch and llama_context::decode loops over process_ubatch. That is
the ordinary prompt-processing path, so any caller submitting more than
n_ubatch tokens in one llama_decode is affected, including the shipped
tools and the server.

Severity scales with the number of splits

Perplexity over one sequence, one llama_decode, a ~350-token prompt of
gemma-4-E2B BF16, changing only n_ubatch. Measured with a small in-house
scorer rather than an upstream tool, so treat it as illustrating the shape of
the failure rather than as the primary evidence:

-ub  64   ->     792.9211      (6 micro-batches)
-ub 128   ->   80446.7741      (3 micro-batches)
-ub 256   ->    1614.8257      (2 micro-batches)
-ub 384   ->      10.0350      (no split)  CORRECT
-ub 512   ->      10.0350      (no split)  CORRECT

The corruption is not a fixed offset. It compounds with each additional
micro-batch.

Not caught by test-backend-ops

test-backend-ops passes 14564 of 14565, because it does not exercise the
host-buffer placement path. The one failure is unrelated and much too small to
explain the above, but is a genuine failure on this target:

TOPK_MOE(ne=[31,22,1,1],n_expert_used=8,with_norm=1,bias_probs=0,
         gating_func=0,scale_w=0.000000)
[VIEW] ERR = 0.000382444 > 0.000000100   FAIL

What it is not

Each tested and ruled out:

• Not a quantisation kernel. BF16 is affected and involves no dequantisation.
  Q8_0, Q4_K_M and Q4_0 are affected equally.
• Not flash attention. Broken with -fa on, off and auto.
• Not HIP graphs. GGML_CUDA_DISABLE_GRAPHS=1 changes nothing.
• Not the matmul path. GGML_CUDA_FORCE_MMQ=1 and GGML_CUDA_FORCE_CUBLAS=1
  change the wrong value without fixing it.
• Not KV offload alone. -nkvo changes the wrong value without fixing it.
• Not model or architecture specific. Reproduced on gemma-4 E2B and on
  Qwen3.5-0.8B, which has no sliding-window attention.
• Not non-determinism. Every configuration reproduces bit-identically, so this
  is incorrect data rather than a race.
• Not new at HEAD. Built and reproduced at a94d563 (2026-08-13), the point
  AMD's fork branched from.

Related issues, and why this is not a duplicate of them

GitHub suggests three. None describe this failure, which is silent numerical
corruption on a single GPU with no crash.

• Misc. bug: ggml-cuda: restore prop.integrated for HIP builds; #16308 hardcode breaks iGPU classification and supports_buft for AMD APUs #23977 is the origin, not a duplicate. Its fix, PR ggml-cuda : restore prop.integrated on HIP builds #24233, is what
  introduced this. See the First Bad Commit section below.

• Misc. bug: Incapable IGP crashes llama.cpp when capable GPU is present #23152, incapable IGP crashes llama.cpp when a capable GPU is present.
  A crash, on Windows, with a discrete GPU also present. Here there is one GPU,
  nothing crashes, and the run completes with wrong numbers.

• Misc. bug: gfx1152 broken unless I manually export HSA_OVERRIDE_GFX_VERSION=11.5.0 #19949, gfx1152 broken unless HSA_OVERRIDE_GFX_VERSION=11.5.0 is set.
  A segfault on a different chip caused by runtime chip recognition. Setting
  that override on gfx1151 does not help and aborts instead, since it
  contradicts the compiled --offload-arch. Unrelated.

• Eval bug: Broken/no Gemma 3n output on CUDA (Nvidia Jetson Orin Nano) #15034 is the same class of corruption on the CUDA side and is why
  integrated is disabled there. It is the closest relative of this report.

First Bad Commit

This might be a regression with a traceable history, and the naive fix may undo
someone else's bug fix.

• Eval bug: Broken/no Gemma 3n output on CUDA (Nvidia Jetson Orin Nano) #15034 (2025-08): Gemma 3n produced garbled output on a Jetson Orin Nano, an
  integrated GPU. Fixed by PR cuda : Disable host buffers on integrated GPUs (#15034) #16308, which hardcoded integrated = false.
• Misc. bug: ggml-cuda: restore prop.integrated for HIP builds; #16308 hardcode breaks iGPU classification and supports_buft for AMD APUs #23977 (2026-06): that hardcode broke iGPU classification on AMD APUs, where
  a machine with both an iGPU and a dGPU had work scheduled onto both as if
  both were discrete. The reporter traced it to
  ggml_backend_cuda_device_supports_buft returning the wrong value and
  proposed restoring prop.integrated for HIP.
• PR ggml-cuda : restore prop.integrated on HIP builds #24233 (2026-07-16) implemented that and is the commit that introduced the
  #if defined(GGML_USE_HIP) branch now in the tree.

Possible Root cause

ggml/src/ggml-cuda/ggml-cuda.cu, device init:

#if defined(GGML_USE_HIP)
        info.devices[id].integrated = prop.integrated;
#else
        info.devices[id].integrated = false; // Temporarily disabled due to issues with corrupted output (e.g. #15034)
#endif

integrated is disabled for CUDA because it produced corrupted output
(#15034), and re-enabled for HIP by PR #24233 to fix iGPU classification
(#23977). gfx1151 (Strix Halo) is an integrated UMA part, so HIP reports
integrated = true and takes the path CUDA disabled for the corruption
reason. See the history bullets above: both branches exist for good reasons
and the flag is being asked to mean two different things.

That flag then lets ggml_backend_cuda_device_supports_buft accept pinned host
buffers as GPU-usable:

static bool ggml_backend_cuda_device_supports_buft(ggml_backend_dev_t dev, ggml_backend_buffer_type_t buft) {
    ggml_backend_cuda_device_context * dev_ctx = (ggml_backend_cuda_device_context *) dev->context;
    const bool integrated = ggml_cuda_info().devices[dev_ctx->device].integrated;
    return (ggml_backend_buft_is_cuda(buft) && buft->device == dev) || (integrated && ggml_backend_buft_is_cuda_host(buft));
}

So deleting the HIP branch would restore correctness here and re-break
#23977. That is oneline-fix.patch, and it is included only to demonstrate
that this flag is the cause.

--- a/ggml/src/ggml-cuda/ggml-cuda.cu
+++ b/ggml/src/ggml-cuda/ggml-cuda.cu
@@ -305,11 +305,7 @@
-#if defined(GGML_USE_HIP)
-        info.devices[id].integrated = prop.integrated;
-#else
         info.devices[id].integrated = false; // Temporarily disabled due to issues with corrupted output (e.g. #15034)
-#endif

It is not the recommended fix.

Proposed fix

Keep integrated for device classification, which is what #23977 needs, and
stop treating host buffers as GPU-usable, which is what causes the corruption.
Those are two separate concerns sharing one flag.

AMD's fork does exactly this, in commit 865374b on branch gfx11, and it is
amd-fix.patch here:

static bool ggml_cuda_is_gfx1151(const int device) {
    return ggml_cuda_info().devices[device].cc == GGML_CUDA_CC_RDNA3_5 + 1;
}

static bool ggml_backend_cuda_device_supports_buft(ggml_backend_dev_t dev, ggml_backend_buffer_type_t buft) {
    ggml_backend_cuda_device_context * dev_ctx = (ggml_backend_cuda_device_context *) dev->context;
    const bool integrated         = ggml_cuda_info().devices[dev_ctx->device].integrated;
    const bool direct_host_access = integrated && !ggml_cuda_is_gfx1151(dev_ctx->device);
    return (ggml_backend_buft_is_cuda(buft) && buft->device == dev) ||
           (direct_host_access && ggml_backend_buft_is_cuda_host(buft));
}

Whether the exclusion should be gfx1151 only, or all HIP integrated targets, or
whether the underlying coherency problem can be fixed instead, is a question
for people who know the driver. We can only report that gfx1151 is affected and
that this shape of fix resolves it.

Relevant log output

Logs

# corpus: standard WikiText-2 test split
#   curl -sLO https://huggingface.co/datasets/ggml-org/ci/resolve/main/wikitext-2-raw-v1.zip
#   unzip wikitext-2-raw-v1.zip
#
# Only -ub changes between the two runs of each pair. Everything else is identical.

### HIP build, Qwen3.5-0.8B-Q4_K_M

$ llama-perplexity -m Qwen3.5-0.8B-Q4_K_M.gguf -f wiki.test.raw \
    -c 2048 -b 2048 -ub 512  --chunks 8 -ngl 99 -fa off
Final estimate: PPL = 1565.9828 +/- 87.11104

$ llama-perplexity -m Qwen3.5-0.8B-Q4_K_M.gguf -f wiki.test.raw \
    -c 2048 -b 2048 -ub 2048 --chunks 8 -ngl 99 -fa off
Final estimate: PPL = 13.7743 +/- 0.45458

### HIP build, gemma-4-E2B q4_0

$ llama-perplexity -m gemma-4-E2B_q4_0-it.gguf -f wiki.test.raw \
    -c 2048 -b 2048 -ub 512  --chunks 8 -ngl 99 -fa off
perplexity: calculating perplexity over 8 chunks, n_ctx=2048, batch_size=2048, n_seq=1
[1]6030.1563,[2]7605.1239,[3]8028.6236,[4]8303.2742,[5]7819.0489,[6]6628.9725,[7]6531.4533,[8]6473.3124,
Final estimate: PPL = 6473.3124 +/- 382.43584

$ llama-perplexity -m gemma-4-E2B_q4_0-it.gguf -f wiki.test.raw \
    -c 2048 -b 2048 -ub 2048 --chunks 8 -ngl 99 -fa off
Final estimate: PPL = 37.1891 +/- 1.62087

### Vulkan build, same machine, same GPU, same files

$ llama-perplexity -m Qwen3.5-0.8B-Q4_K_M.gguf ... -ub 512
Final estimate: PPL = 13.7938 +/- 0.45571
$ llama-perplexity -m Qwen3.5-0.8B-Q4_K_M.gguf ... -ub 2048
Final estimate: PPL = 13.7891 +/- 0.45529

$ llama-perplexity -m gemma-4-E2B_q4_0-it.gguf ... -ub 512
Final estimate: PPL = 37.0287 +/- 1.61418
$ llama-perplexity -m gemma-4-E2B_q4_0-it.gguf ... -ub 2048
Final estimate: PPL = 37.0170 +/- 1.61392

### Summary

              -ub 512      -ub 2048     ratio
HIP    Qwen    1565.9828     13.7743    113.7x
HIP    gemma   6473.3124     37.1891    174.1x
Vulkan Qwen      13.7938     13.7891      1.00x
Vulkan gemma     37.0287     37.0170      1.00x

Vulkan is flat. HIP is not. -ub is the only variable.

[Error00101]

Have the same gfx1151 (Strix Halo - RDNA3.5)
llama.cpp is compiled from source as a fat binary with all backends included (currently on: 7798007)

Devices

./llama-perplexity_fixed --list-devices

Available devices:
ROCm0: AMD Radeon 8060S Graphics (98304 MiB, 33936 MiB free)
Vulkan0: AMD Radeon 8060S Graphics (RADV STRIX_HALO) (114688 MiB, 19584 MiB free)
BLAS: OpenBLAS (0 MiB, 0 MiB free)

I did some slight testing and perplexity explodes when using the unfixed ROCm version.
The fix (865374b) was implemented by me and it works.

All models are run with the default params the only thing that changes is the backend
Summary:

---Qwen3.5---
./llama-perplexity_unfixed -m Qwen3.5-2B-Q8_0.gguf -f wiki.test.raw --device Vulkan0
1.23.662.709 I Final estimate: PPL = 12.8604 +/- 0.09557
./llama-perplexity_unfixed -m Qwen3.5-2B-Q8_0.gguf -f wiki.test.raw --device ROCm0
1.21.966.651 I Final estimate: PPL = 2929.2808 +/- 40.87135
./llama-perplexity_fixed -m Qwen3.5-2B-Q8_0.gguf -f wiki.test.raw --device ROCm0
1.20.640.104 I Final estimate: PPL = 12.8427 +/- 0.09539

---Gemma3---
./llama-perplexity_unfixed -m gemma-3-1b-it-Q4_K_M.gguf -f wiki.test.raw --device Vulkan0
0.48.547.746 I Final estimate: PPL = 29.8956 +/- 0.28932
./llama-perplexity_unfixed -m gemma-3-1b-it-Q4_K_M.gguf -f wiki.test.raw --device ROCm0
0.48.254.117 I Final estimate: PPL = 62725.6405 +/- 1175.69898
./llama-perplexity_fixed -m gemma-3-1b-it-Q4_K_M.gguf -f wiki.test.raw --device ROCm0
0.49.064.989 I Final estimate: PPL = 29.8499 +/- 0.28871

---MiniCPM5---
./llama-perplexity_unfixed -m MiniCPM5-1B.Q8_0.gguf -f wiki.test.raw --device Vulkan0
0.38.082.394 I Final estimate: PPL = 26.5569 +/- 0.22316
./llama-perplexity_unfixed -m MiniCPM5-1B.Q8_0.gguf -f wiki.test.raw --device ROCm0
0.38.951.182 I Final estimate: PPL = 168760.6023 +/- 1948.17555
./llama-perplexity_fixed -m MiniCPM5-1B.Q8_0.gguf -f wiki.test.raw --device ROCm0
0.39.607.795 I Final estimate: PPL = 26.5444 +/- 0.22294

I normaly run DeepseekV4-0731 with ROCm. I observed substantialy improved answers from it (far less rambly and more precise with its answers) but i have no benchmark for that. If needed i will provide a benchmark for that as well.

If other tests are needed i am happy to help.

Edit: I found the RPC option (Sorry!)