Eval bug: -ot and -ts/dev_layer() placement disagreement crashes RPC decode

[ldumancas]

Name and Version

$./llama-cli --version
version: 9949 (049326a00)
built with GNU 14.2.0 for Linux x86_64

Operating systems

Linux

GGML backends

RPC, CPU

Hardware

2-node cluster, each node is a Proxmox LXC container on a Geekom A6 (AMD Ryzen 7 6800H, 32 GB DDR5-4800). CPU-only, no GPU. Coordinator container capped at 27648 MiB/14 cores, worker container capped at 18432 MiB/10 cores (Linux memory cgroup + cpuset enforced by host). Nodes connected over LAN; the coordinator's own layer share is routed through a loopback ggml-rpc-server on 127.0.0.1, the worker takes a real network hop.

Models

Qwen3.6-35B-A3B - https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF/blob/main/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf

Problem description & steps to reproduce

Launching llama.cpp with Qwen3.6-35B-A3B across a 2-node dual-RPC topology with a -ts value that doesn't match the real -ot split ratio crashes during the server's own internal startup warm-up decode before any request is served.

It crashes with this RPC scheduler crash signature: ggml-rpc.cpp:498: Remote RPC server crashed or returned malformed response despite sched_reserve completing cleanly first.

Right before crashing, llama.cpp resolve_fused_ops logs show:

resolve_fused_ops: layer 21 is assigned to device RPC1 but fused Gated Delta Net (autoregressive)
is assigned to device RPC0 (usually due to missing support)
resolve_fused_ops: fused Gated Delta Net (autoregressive) not supported, set to disabled
resolve_fused_ops: layer 21 is assigned to device RPC1 but fused Gated Delta Net (chunked)
is assigned to device RPC0 (usually due to missing support)
resolve_fused_ops: fused Gated Delta Net (chunked) not supported, set to disabled

but I believe that the message is misleading. -ot's real weight placement disagrees with dev_layer()'s automatic placement. -ot sets a hard boundary at layer 25. dev_layer(), driven here by -ts 1,1, computes its own boundary near layer 20. For layers 20 through 24, the two systems disagree about which machine owns them.

I added some logging right after resolve_fused_ops warnings and the RPC graph scheduler logged 65 [add_tensor] CROSS-ENDPOINT MISMATCH errors on layer 21's tensors and Gated-Delta-Net intermediate compute nodes. Each reported a tensor reached via one RPC socket but required on another.

Commands for crashing state

# worker
./ggml-rpc-server -H <worker-ip> -p 50052 -t 8
# coordinator's own loopback device
./ggml-rpc-server -H 127.0.0.1 -p 50052 -t 6
# coordinator - crashes
GGML_RPC_DEBUG=1 ./llama-server \
  -m Qwen3.6-35B-A3B-UD-Q4_K_M.gguf \
  --rpc 127.0.0.1:50052,<worker-ip>:50052 \
  -ngl 99 -ts 1,1 \
  -ot "blk\.([0-9]|1[0-9]|2[0-4])\..*=RPC0[127.0.0.1:50052],blk\.(2[5-9]|3[0-9])\..*=RPC0[<worker-ip>:50052]" \
  -c 262144 -t 2 -np 1 -fit off --no-warmup -lv 4

Commands for working state

# same as above except for -ts 3,2 instead of -ts 1,1
# worker
./ggml-rpc-server -H <worker-ip> -p 50052 -t 8
# coordinator's own loopback device
./ggml-rpc-server -H 127.0.0.1 -p 50052 -t 6
# coordinator - works
GGML_RPC_DEBUG=1 ./llama-server \
  -m Qwen3.6-35B-A3B-UD-Q4_K_M.gguf \
  --rpc 127.0.0.1:50052,<worker-ip>:50052 \
  -ngl 99 -ts 3,2 \
  -ot "blk\.([0-9]|1[0-9]|2[0-4])\..*=RPC0[127.0.0.1:50052],blk\.(2[5-9]|3[0-9])\..*=RPC0[<worker-ip>:50052]" \
  -c 262144 -t 2 -np 1 -fit off --no-warmup -lv 4

I believe that my findings are that -ot and -ts/dev_layer() have an undocumented dependency that needs to be respected.

-ot's help text: --override-tensor <tensor name pattern>=<buffer type>,... override tensor buffer type
-ts's help text: --tensor-split N0,N1,N2,... fraction of the model to offload to each GPU, comma-separated list of proportions, e.g. 3,1

There is nothing that documents the coupling of -ot and -ts and I couldn't find anything that stated -ot only touches weights while another independent mechanism governs KV-cache and fused-op placement. Additionally -ts's help text doesn't mention RPC devices at all even though it drives RPC device placement.

This looks like the same underlying mechanism as #24492, fixed in #25232. MTP draft/target KV-cache sharing also hits the model.dev_layer(il) disagreement but through a different trigger. Root cause from the PR: "the shared tensor's buffer type and the layer's buffer type differ, and the scheduler later fails to place the tensor and aborts."

Given the coupling, I would expect a startup error or logging to indicate the misconfiguration. Maybe -ot and dev_layer() need some sort of cross validation? Is -ot's weight-only scope intentional or an oversight?

First Bad Commit

Not a regression. -ot/--override-tensor was added in #11397 as an override to weight-tensor buffer selection only. The automatic dev_layer/get_layer_buft_list table it disagrees with was introduced in #10026 and was untouched by the same commit. The check that surfaces the warning, resolve_fused_ops is much newer, #24646.

Relevant log output

Logs

# A. Known-good baseline (-ts 3,2, matching -ot):
load_tensors:   CPU_Mapped model buffer size =   515.31 MiB
load_tensors: RPC0[127.0.0.1:50052] model buffer size = 12553.92 MiB
load_tensors: RPC0[192.168.1.127:50052] model buffer size =  8029.41 MiB
llama_kv_cache: RPC0[127.0.0.1:50052] KV buffer size =  3072.00 MiB
llama_kv_cache: RPC0[192.168.1.127:50052] KV buffer size =  2048.00 MiB
llama_memory_recurrent: RPC0[127.0.0.1:50052] RS buffer size =    39.78 MiB
llama_memory_recurrent: RPC0[192.168.1.127:50052] RS buffer size =    23.03 MiB
resolve_fused_ops: Flash Attention enabled
resolve_fused_ops: resolving fused Gated Delta Net support:
resolve_fused_ops: fused Gated Delta Net (autoregressive) enabled
resolve_fused_ops: fused Gated Delta Net (chunked) enabled

# B. Broken config (-ts 1,1 against the unchanged -ot)
# Weight placement is identical to the good config (confirms -ot alone is unaffected by -ts):
load_tensors:   CPU_Mapped model buffer size =   515.31 MiB
load_tensors: RPC0[127.0.0.1:50052] model buffer size = 12553.92 MiB
load_tensors: RPC0[192.168.1.127:50052] model buffer size =  8029.41 MiB

# KV-cache and recurrent-state (SSM) buffers split evenly instead of 3:2 — confirms -ts is what dev_layer() actually consults:
llama_kv_cache: RPC0[127.0.0.1:50052] KV buffer size =  2560.00 MiB
llama_kv_cache: RPC0[192.168.1.127:50052] KV buffer size =  2560.00 MiB
llama_memory_recurrent: RPC0[127.0.0.1:50052] RS buffer size =    33.50 MiB
llama_memory_recurrent: RPC0[192.168.1.127:50052] RS buffer size =    29.31 MiB

# Mismatch
resolve_fused_ops: Flash Attention enabled
resolve_fused_ops: resolving fused Gated Delta Net support:
W resolve_fused_ops: layer 21 is assigned to device RPC1 but fused Gated Delta Net (autoregressive) is assigned to device RPC0 (usually due to missing support)
W resolve_fused_ops: fused Gated Delta Net (autoregressive) not supported, set to disabled
W resolve_fused_ops: layer 21 is assigned to device RPC1 but fused Gated Delta Net (chunked) is assigned to device RPC0 (usually due to missing support)
W resolve_fused_ops: fused Gated Delta Net (chunked) not supported, set to disabled

# Crash
ggml-rpc.cpp:498: Remote RPC server crashed or returned malformed response
E recv failed (bytes_recv=0, size_to_recv=8)
libggml-base.so.0(ggml_abort+0x11e)
libggml-rpc.so.0(+0xa9ec)
libggml-base.so.0(ggml_backend_tensor_copy+0x156)
libggml-base.so.0(ggml_backend_sched_graph_compute_async+0x1de)
libllama.so.0(llama_context::graph_compute(ggml_cgraph*, bool)+0xa1)
libllama.so.0(llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&)+0xea)
libllama.so.0(llama_context::decode(llama_batch const&)+0x368)
libllama.so.0(llama_decode+0xb)
libllama-server-impl.so(server_context_impl::load_model(common_params&)+0x8dc)

# From the logging I added in a local patch
E [add_tensor] CROSS-ENDPOINT MISMATCH: tensor 'blk.21.attn_qkv.weight' reached via src[] has buffer on sock=0x5819a9d53290, target connection sock=0x5819a9d533f0
E [add_tensor] CROSS-ENDPOINT MISMATCH: tensor 'blk.21.ssm_alpha.weight' reached via src[] has buffer on sock=0x5819a9d53290, target connection sock=0x5819a9d533f0
E [add_tensor] CROSS-ENDPOINT MISMATCH: tensor 'blk.21.ssm_a' reached via src[] has buffer on sock=0x5819a9d53290, target connection sock=0x5819a9d533f0
E [add_tensor] CROSS-ENDPOINT MISMATCH: tensor 'decay_mask-21' reached via src[] has buffer on sock=0x5819a9d53290, target connection sock=0x5819a9d533f0
E [add_tensor] CROSS-ENDPOINT MISMATCH: tensor 'attn_pre_solve-21' reached via src[] has buffer on sock=0x5819a9d53290, target connection sock=0x5819a9d533f0
E [add_tensor] CROSS-ENDPOINT MISMATCH: tensor 'g_in-21' reached via src[] has buffer on sock=0x5819a9d53290, target connection sock=0x5819a9d533f0

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.