llama-hot-experts: pin hottest MoE experts in RAM via --pin-hot-experts

[Ghimli]

Feature: --pin-hot-experts — prevent MoE expert eviction from RAM

Problem

Large MoE models that exceed system RAM are typically loaded via mmap. The OS page cache then evicts infrequently accessed pages to make room for active workloads. In MoE models, each routing step activates only a subset of experts, causing the OS to evict the other experts from RAM. When those evicted experts are needed later, they must be paged back in from disk, causing severe latency spikes and degraded throughput.

Solution

--pin-hot-experts N dynamically tracks expert usage across inference requests and uses mlock() to pin the top-N most frequently used ("hot") experts in RAM, preventing OS eviction. This keeps the critical-path experts resident while allowing cold experts to be paged normally.

Dense parts are mlocked automatically

Before any hot experts are pinned, all dense (non-MoE-expert) tensors that reside in host memory are mlocked in place. Dense parts (embeddings, attention/FFN weights, RMSNorm, output projection, router weights, etc.) are used on every single token — they are the hottest data by definition. This happens unconditionally when --pin-hot-experts is enabled and mlock is supported, consuming the global budget first so that hot experts only ever get the leftover budget.

Key design points

• Real-time tracking: Expert usage is counted per-layer at runtime via a tensor callback on ffn_moe_topk tensors. Every token's expert routing decision updates a global std::set ranking ordered by hit count.
• On-the-fly eviction: When a new expert's count exceeds the coldest currently-pinned expert, the cold expert is immediately munlock()'d and the new expert is mlock()'d. No periodic re-evaluation — decisions happen at routing time.
• Budget cap: --pin-hot-experts-budget-mib limits total pinned memory to avoid exhausting RAM. Budget is checked before each mlock() call. Dense tensors consume the budget first.
• Fail-safe rollback: If a new expert fails to lock (budget exhausted, mlock error), the evicted expert is re-pinned to keep data structures and locked pages consistent.
• Stats reporting: --pin-hot-experts-stats-interval N prints per-layer expert stats every N router observations (visible at -lv 4).
• Thread-safe: Uses a mutex-protected std::set for the global ranking and std::unique_ptr with a custom deleter for automatic munlock on eviction.
• Selective pinning: Only hot experts are mlocked. Unlike --load-mode mmap+mlock which locks the entire model, --load-mode mmap+pin locks only the experts that are actually used (plus all dense parts automatically).

CLI arguments

Argument	Default	Description
--pin-hot-experts <n>	0 (disabled)	Number of globally-hot experts to pin in RAM
--pin-hot-experts-budget-mib <n>	0 (unlimited)	Maximum pinned memory in MiB
--pin-hot-experts-stats-interval <n>	0 (disabled)	Print expert stats every N tokens
--load-mode mmap+pin	—	mmap without global mlock; designed to pair with --pin-hot-experts

Example

llama-server \
    -m /path/to/moe-model-00001-of-00006.gguf \
    --n-gpu-layers 999 --cpu-moe \
    --pin-hot-experts 190 \
    --pin-hot-experts-budget-mib 130000 \
    --pin-hot-experts-stats-interval 2000 \
    -lv 4

# Or with the dedicated load mode (mmap, no global mlock)
llama-server \
    -m /path/to/moe-model-00001-of-00006.gguf \
    --load-mode mmap+pin \
    --pin-hot-experts 190 \
    --pin-hot-experts-budget-mib 130000

Naming convention

• C API fields use underscores: n_pin_hot_experts, n_pin_hot_experts_budget_bytes, n_pin_hot_experts_stats_interval
• CLI args use hyphens: --pin-hot-experts, --pin-hot-experts-budget-mib, --pin-hot-experts-stats-interval
• Load mode: --load-mode mmap+pin (LLAMA_LOAD_MODE_MMAP_PIN) — mmap without global mlock, intended for use with --pin-hot-experts

Files changed

• include/llama.h — public API fields in llama_context_params, new LLAMA_LOAD_MODE_MMAP_PIN enum
• src/llama.cpp — load mode name/parser for mmap+pin
• src/llama-model.cpp — use_mlock logic (MMAP_PIN does NOT trigger global mlock)
• src/llama-hot-experts.cpp / src/llama-hot-experts.h — core implementation (new files)
• src/llama-context.cpp / src/llama-context.h / src/llama-cparams.h — context integration
• common/arg.cpp / common/common.cpp / common/common.h — CLI argument handling
• src/CMakeLists.txt — build integration

Hardware / testing environment

• CPU: Intel Core i9-13900KF (8P + 16E, 32 threads, up to 5.8 GHz)
• GPU: NVIDIA GeForce RTX 4090 (24 GB VRAM)
• RAM: 152 GiB DDR5
• Storage: NVMe SSD (3.6 TB)
• OS: Ubuntu 24.04, Linux 7.0.0, x86_64
• Compiler: GCC 15.2.0, CMake 4.2.3
• Driver: NVIDIA 595.71.05

AI usage disclosure

Yes — AI assistance was used during development. I can explain every line of code in this PR.

Related

• Discussion HotPin: Solving the MoE RAM problem – selective expert pinning for llama.cpp #25469 — "HotPin: Solving the MoE RAM problem" (separate external project that inspired this feature)

Baseline test (no pinning, plain mmap):

/home/ghimli/llama/build/bin/llama-server
-m "/home/ghimli/rootmodels/GLM-5.2-UD-Q2_K_XL-00001-of-00007.gguf"
--ctx-size 131072
--threads 32
--flash-attn on
-np 1
--fit off
--no-warmup
--cache-type-k q8_0
--cache-type-v q8_0
--host 127.0.0.1
--port 7777
--jinja
--reasoning-preserve
--cors-origins http://localhost:7777,http://127.0.0.1:7777
--n-gpu-layers 999
--cpu-moe
-lv 4

============================================
Benchmarking llama-server on 127.0.0.1:7777

Rep 1/5... total=195.422266s prompt=1544tok gen=128tok
Rep 2/5... total=65.140298s prompt=1544tok gen=128tok
Rep 3/5... total=53.022765s prompt=1544tok gen=128tok
Rep 4/5... total=49.189438s prompt=1544tok gen=128tok
Rep 5/5... total=51.243147s prompt=1544tok gen=128tok

Results (5 repetitions):

Prompt tok/s: avg=24.4 stdev=9.7
Gen tok/s: avg=2.0 stdev=0.8

Raw data:
Rep 1: prompt=7.9 tok/s, gen=0.6 tok/s
Rep 2: prompt=23.7 tok/s, gen=1.9 tok/s
Rep 3: prompt=29.1 tok/s, gen=2.4 tok/s
Rep 4: prompt=31.3 tok/s, gen=2.6 tok/s
Rep 5: prompt=30.1 tok/s, gen=2.4 tok/s

Hot experts pinned test (--load-mode mmap+pin):

/home/ghimli/llamahotpin/build/bin/llama-server
-m "/home/ghimli/rootmodels/GLM-5.2-UD-Q2_K_XL-00001-of-00007.gguf"
--ctx-size 131072
--threads 32
--flash-attn on
-np 1
--fit off
--no-warmup
--cache-type-k q8_0
--cache-type-v q8_0
--host 127.0.0.1
--port 7777
--jinja
--reasoning-preserve
--cors-origins http://localhost:7777,http://127.0.0.1:7777
--n-gpu-layers 999
--cpu-moe
-lv 4
--load-mode mmap+pin
--pin-hot-experts-stats-interval 2000
--pin-hot-experts 160

============================================
Benchmarking llama-server on 127.0.0.1:7777

Rep 1/5... total=217.214807s prompt=1544tok gen=128tok
Rep 2/5... total=46.257479s prompt=1544tok gen=128tok
Rep 3/5... total=34.168737s prompt=1544tok gen=128tok
Rep 4/5... total=30.620768s prompt=1544tok gen=128tok
Rep 5/5... total=29.886910s prompt=1544tok gen=128tok

Results (5 repetitions):

Prompt tok/s: avg=37.5 stdev=18.5
Gen tok/s: avg=3.0 stdev=1.5

Raw data:
Rep 1: prompt=7.1 tok/s, gen=0.5 tok/s
Rep 2: prompt=33.3 tok/s, gen=2.7 tok/s
Rep 3: prompt=45.1 tok/s, gen=3.7 tok/s
Rep 4: prompt=50.4 tok/s, gen=4.1 tok/s
Rep 5: prompt=51.6 tok/s, gen=4.2 tok/s

I [pin-hot-experts] obs=48300 | locked=139854.38 MiB | moe_layers=75 | pinned=12000/12000 (global, N=160 x layers=75) | distinct (layer,expert) seen=19141 | pinned count range=[53, 657] | per-layer: {L3=218, L4=224, L5=221, L6=211, L7=210, L8=211, L9=218, L10=206, L11=207, L12=179, L13=208, L14=186, L15=158, L16=171, L17=147, L18=97, L19=111, L20=113, L21=128, L22=141, L23=190, L24=189, L25=178, L26=179, L27=182, L28=158, L29=155, L30=154, L31=144, L32=143, L33=144, L34=143, L35=145, L36=136, L37=140, L38=147, L39=155, L40=159, L41=160, L42=157, L43=146, L44=166, L45=147, L46=172, L47=165, L48=163, L49=163, L50=154, L51=143, L52=153, L53=149, L54=140, L55=143, L56=130, L57=145, L58=139, L59=156, L60=133, L61=149, L62=160, L63=151, L64=153, L65=170, L66=145, L67=155, L68=163, L69=169, L70=153, L71=151, L72=150, L73=173, L74=175, L75=164, L76=166, L77=23}

pinned count range=[53, 657] <- all hot experts used more than 53 times are mlocked, and all cold experts used less than 53 times are just mmapped and mostly read from disk when filesystem cache miss. Hottest expert was used 657 times

[pwilkin]

Interesting, so this is something like an intermediate solution between normal --mmap and --mlock?

[Ghimli]

Yes. Partially mlock the most frequently used MoE experts to minimize disk I/O and accelerate token processing for large models that don't fit entirely in RAM. pon., 20 lip 2026, 21:14 użytkownik Piotr Wilkin (ilintar) < ***@***.***> napisał:

…

*pwilkin* left a comment (ggml-org/llama.cpp#25932) <#25932 (comment)> Interesting, so this is something like an intermediate solution between normal --mmap and --mlock?

[Ghimli]

Hi @CISC @ggerganov — could a maintainer please:

1. Approve the 6 CI workflow runs (they're stuck in action_required state for fork PRs)
2. Swap the examples label → enhancement (this is a new feature, not an example)

The code is rebased on latest master, builds cleanly, and the --load-mode mmap+pin bug is fixed (it no longer mlocks all weights). Thanks!

[Green-Sky]

i wonder if it should not be called "hot" instead of "pin", since pin == lock .

[Ghimli]

Moved to PR #26414