feat(moe): add heterogeneous NVMe expert streaming tier

[davide221]

Summary

• add a bounded, model-neutral NVMe capacity tier for routed MoE experts
• separate physical byte layout, numerical expert specification, and native routed batches so model adapters remain small
• support tensor-major GGUF, standard split GGUF shards, and optional expert-major records
• cache persistent expert graphs and adaptive GPU expert slots above the SSD scheduler
• support unchanged Strix Halo-only execution
• support concurrent R9700-primary + Strix-capacity route ownership, with independent SSD/cache pipelines
• complete the Kimi-K3 text adapter from its native router through split-GGUF expert execution

The model's native router remains authoritative. Placement and caching decide where selected expert contributions execute; they do not substitute routes.

Dual-owner execution

MoeStreamDualOwnerExecutor composes two ordinary stream engines:

• each (layer, expert) has one stable owner
• an explicit MoeHybridPlacement plan selects primary-owned hot experts; a deterministic hash is the bring-up fallback
• R9700 and Strix launch their selected routed branches concurrently
• each owner has independent io_uring, pinned staging, device cache, and persistent expert graphs
• activation-sized partials are added after both complete
• a persistent secondary worker removes thread create/join from the layer loop
• batches that route entirely to one owner bypass the rendezvous

For a dual-owner layer, the measured target is therefore max(primary branch, secondary branch) rather than their sum.

Kimi-K3 integration

• loads standard -NNNNN-of-NNNNN.gguf sets, including the normal case where only shard 1 has global metadata
• allocates dense, shared, router, latent-projection, recurrent/KV, and output tensors on the selected primary GPU
• leaves routed gate/up/down stacks file-backed and records their exact shard, offset, size, and quantization type
• runs KDA, absorbed MLA, Attention Residuals, dense layer 0, latent routed MoE with SiTU, shared experts, tokenizer, and sampling
• passes native route IDs, weights, and activation-sized inputs to the common streamed evaluator
• sizes both owner caches after resident weights/state, with independent memory queries and safety reserves
• owns SSD descriptors, both GPU backends, and the persistent worker correctly across init, park/unpark, and shutdown

Production Kimi defaults to file-backed routed experts. Resident mode remains a deterministic oracle for small fixtures.

Deployment modes

Strix Halo only

Leaving DFLASH_MOE_TP_GPU unset keeps resident text tensors, the warm expert cache, and selected-expert compute on the selected Strix device. No R9700 or peer access is required.

Full Lucebox

The speed-oriented topology is:

• R9700 primary: dense/main path and profile-selected hot routed experts
• Strix secondary: remaining routed experts and the larger capacity cache
• SSD: true misses for both bounded owner caches

Set DFLASH_MOE_TP_GPU=<strix-gpu> and select the R9700 with --target-device hip:<r9700-gpu>. This is functional ownership within every routed layer, not a contiguous layer split, so it does not use --target-devices.

This routed-owner feature does not solve placement of a non-routed core larger than R9700 VRAM. The current full Kimi quant has approximately 58 GiB of non-routed text weights, so running that exact topology at full scale also requires a separate non-routed tensor/layer plan. The Strix-only SSD path remains capacity-safe today.

Validation

Lucebox4/gfx1151:

• 326/326 server unit tests pass

• the model-neutral streamed MoE GPU numerical test passes

• the real two-shard Kimi-K3-0.40B-MXFP4 fixture runs with native MXFP4 expert tensors

• resident and file-backed execution produce the same eight greedy output IDs:

  318 57195 11 1459 387 1495 2189 261

• a forced 1 MiB device cache causes 163 evictions, proving the result uses the split-GGUF SSD path

• the same single-device adapter reaches 157.49 tok/s decode on the tiny fixture; this is not a full-model estimate

Lucebox5/gfx1201+gfx1151:

• R9700 primary + Strix secondary initializes two independent io_uring/cache/compute engines

• stable ownership exercises both devices; top-2 layers that naturally land on one owner take the direct path

• dual-owner and single-R9700 runs produce the same eight greedy IDs for the test prompt:

  289 646 108263 30 374 5058 667 13

• after warm-up, traced dual-branch layer wall time matches the slower branch to within a few microseconds

• two pre-existing Qwen servers occupied both GPUs during qualification, so no controlled end-to-end dual-owner speedup is claimed from this tiny fixture

The generic data plane was separately qualified at 4.202 GiB/s raw scheduler throughput, 4.275 GiB/s SSD-to-Strix, and 2.940 GiB/s SSD-to-R9700. A full Kimi IQ1_S-shaped fully-cold workload sustains 0.430 storage-limited token/s at 3.804 GiB/s.

Current boundaries

• text-only; the vision tower is out of scope
• Kimi graph execution is correctness-first and token-sequential, not yet fused/captured for full-model speed
• the two routed branches overlap, but Kimi still joins activation-sized partials through host-visible memory rather than PR-505's device-resident peer join
• full Kimi on R9700 primary still needs non-routed tensor/layer placement
• the full 594 GB Kimi quant has not been staged on Lucebox4: about 513 GB is free, while roughly 650 GB is needed safely for shards and run headroom

Controls

• DFLASH_MOE_TP_GPU=<secondary-gpu>
• DFLASH_MOE_PLACEMENT=<placement.json>
• DFLASH_MOE_PRIMARY_SHARE_PER_MILLE=<0..1000>
• DFLASH_MOE_DUAL_STREAM_TRACE=1
• DFLASH_MOE_NVME_BACKEND=auto|uring|pread|mmap
• DFLASH_MOE_NVME_DEVICE_CACHE_MB=<MiB>
• DFLASH_MOE_NVME_SLOTS=<count>
• DFLASH_MOE_NVME_IO_THREADS=<count>
• DFLASH_MOE_NVME_GRAPH_CACHE=<count>

See server/docs/MOE_NVME_STREAMING.md and server/docs/KIMI_K3_HETERO.md for architecture, operation, measurements, and qualification details.

[davide221]

Follow-up validation at 9abec86:

• Added the typed operator policy --moe-storage auto|resident|ssd, with CLI > DFLASH_MOE_STORAGE > deprecated compatibility environment precedence and fail-fast capability checks.
• Added server/scripts/benchmark_kimi_k3_deployments.py, which serially compares Strix-only+SSD with Strix+R9700+SSD, records cold/warm request timing plus per-owner NVMe/cache/graph telemetry, and fails on deterministic output divergence.
• On the R9700+Strix host, the feature gate passes 159 assertions and the harness passes all 6 unit tests.
• Both deployment profiles completed against the two-shard Kimi MXFP4 fixture with identical deterministic output, two real io_uring owner streams in the heterogeneous profile, and zero I/O errors or timeouts.

The published IQ1_S checkpoint defaults to Strix as primary in this comparison because its approximately 58 GiB non-routed allocation does not fit the 32 GiB R9700. R9700-primary remains an explicit harness option for a future non-routed placement or smaller compiled checkpoint.

Fresh CI for the exact head SHA: https://github.com/Luce-Org/lucebox/actions/runs/30671619387

[davide221]

Lucebox3 qualification: adaptive fused resident decode

Commit 3450da3 adds a model-neutral hot-path optimization for routed MoE decode.

• For single-token decode, when every selected expert is already resident in the device cache, all routed expert branches plus their weighted reduction execute in one persistent backend graph.
• If any selected expert misses cache, execution automatically retains the existing overlapped SSD transfer/compute pipeline.
• Exact native router IDs and weights are unchanged.
• Supports separate or fused gate/up projections, SiTU, SwiGLU, clamped SwiGLU, F32, and MXFP4.
• A/B control: DFLASH_MOE_NVME_FUSED_DECODE=0|1.

Paired Kimi-K3-shaped routed-core microbenchmarks (896 experts, top-16, 92 MoE layers, 3584 to 3072 to 3584, IQ1_S, SiTU; one cold token followed by 49 route-reuse tokens):

Device	Fused off	Fused on	Gain
Strix Halo gfx1151	5.770 tok/s	6.913 tok/s	+19.8%
RTX 3090 sm_86	6.327 tok/s	7.364 tok/s	+16.4%

Graph submissions fell from 73,600 to 5,980 over the run: 1,472 cold first-token expert graphs plus 4,508 fused layer graphs. A 0%-cache-hit guard run activated zero fused launches and retained the original cold path.

Fresh HIP/gfx1151 and CUDA/sm_86 builds pass the expanded numerical test suite, including F32/MXFP4, cold fallback, cache-pointer rebinding, and changed routes/weights.

These are routed-core measurements, not full-model Kimi generation results.

[davide221]

Mixed-vendor update (Lucebox4):

• rebased the draft on merged PR feat(ds4): run CUDA and HIP expert parallelism in one process #570 and now reuse its single-process HIP-primary / isolated-CUDA-peer runtime
• made SSD staging backend-neutral: pinned host buffers, asynchronous H2D, transfer events, and the second upload stream now come from the selected backend module
• Kimi accepts DFLASH_MOE_TP_BACKEND=cuda with DFLASH_MOE_TP_GPU=0, so hip:0 and cuda:0 are distinct routed owners in one process
• changed the benchmark harness to backend-qualified endpoints and verified the exact hip:0 + cuda:0 launch plan
• mixed build succeeds on Lucebox4; scheduler, expert-package, and streamed numerical suites all pass on Strix

CUDA execution and the full heterogeneous Kimi measurement remain pending because the Lucebox4 RTX 3090 reported Xid 79 / fallen off bus. Driver reset and PCI function reset did not recover it; a reboot or power cycle is required. No mixed-vendor speed claim is made yet.

[davide221]

Lucebox4 post-reset qualification (2026-08-04):

• ordinary reboot was blocked by the wedged NVIDIA probe; a synced emergency kernel reboot completed and the host returned through Tailscale in about 50 seconds
• RTX 3090 recovered on driver 595.84 with 24,373 MiB free and no new Xid entries
• all mixed HIP/CUDA runtime cases pass
• the CUDA-peer test_moe_stream_compute suite passes with zero I/O errors/timeouts
• the real Kimi loader selected the intended single-process topology: primary=hip:0, secondary=cuda:0

The full run was stopped during startup after auto-sizing a 59.24 GiB Strix cache: whole-pool zero initialization remained active beyond 180 seconds and left only about 5.5 GiB RAM headroom. This is now a concrete software issue to fix before reporting heterogeneous Kimi throughput; it is not an RTX failure.

[davide221]

Cache/allocation update (24045990)

• Fixed the Strix UMA cache-startup stall. The generic ggml memory query may expose system-available memory, while one cache allocation is limited by the runtime-native pool; auto-sizing now caps against both budgets.
• Removed the eager clear of the entire multi-GiB expert cache. Quantized row padding is now zeroed lazily with a small pinned staging buffer, preserving numerical correctness without faulting the whole managed allocation at startup.
• Added cache-first decode scheduling: resident experts execute while admitted SSD misses are in flight, while contributions are reduced in their original deterministic order.
• Added reproducible direct-I/O, slot-count, and cache-first A/B controls to the Kimi deployment benchmark.

Full Kimi-K3 UD-IQ1_S, Lucebox4 mixed HIP/CUDA, one NVMe, identical 29-token prompt + 8-token completion:

mode	prefill	decode	wall	reported decode
cache-first off	94.54 s	23.51 s	118.08 s	0.3 tok/s
cache-first on	82.35 s	19.56 s	101.94 s	0.4 tok/s

That is 16.8% lower decode time and 13.7% lower wall time, with byte-identical generated text. Automatic caches now initialize directly at approximately 32.7 GiB on Strix and 21.6 GiB on CUDA instead of probing/hanging on an unsafe approximately 59 GiB Strix allocation.

Validation on Lucebox4: mixed HIP/CUDA interop and native-memory query pass; streamed-expert numerical tests pass independently on HIP and CUDA; NVMe scheduler/package tests pass; benchmark script tests pass (7/7).

The allocation bug is resolved. The remaining approximately 0.4 target tok/s limit is the single-NVMe, tensor-major three-read expert layout, not cache initialization.