Kimi K3 CPU decode optimizations: MXFP4 kernels, parallel expert streaming, NUMA placement

[neilcouture]

Summary

Optimization series for Kimi K3 decode on a dual-socket target box, measured against the
real 2.8T checkpoint at every gate. Headline (from PERFORMANCE.md's full-effort scoreboard):
decode goes from 36.8 s/token at the Phase 0 baseline to 16.9 s/token after the
kernel/streaming phases, and 1.26 s/token in live serving with the NUMA work — with
logits bit-identical across every scheduling configuration tested, pinned by per-run
FNV-1a fingerprints.

What's included

• MXFP4 matmul: scalar kernel hygiene (bit-exact), then an AVX-512 tier
  (within-tolerance), benchmarked in benches/kernels.rs. This was the single biggest
  known K3 perf lever (MXFP4 was scalar-only).
• Expert streaming: parallel miss loads, --preload-experts, pinning skipped at full
  residency, mlock failure logged once. Preload startup: ~30 min → ~9 min.
• latent_moe: across-expert parallelism, v2 flattened to (expert × row-block) tasks.
• NUMA (src/numa.rs, --numa / --numa-threads): topology + placement primitives,
  pinned per-socket pools, expert home nodes, flattened domain dispatch, attention split,
  QTSharded/DenseQT row-sharding for lm_head (2.7× on that bucket).
• Synthetic K3-at-scale fixture generator (src/bin/gen_k3_synth.rs) so scheduling
  changes can be gated without the 2.8T checkpoint.
• PERFORMANCE.md: every phase logged with exact commands and hardware, including the
  attempts that were measured and reverted (KDA-projection sharding, target-cpu=native).

Correctness

• Scheduling-only changes are accepted only on identical logits fingerprints
  (bit-identical logits run-to-run); kernel changes are bit-exact or within-tolerance as
  documented per phase.
• Timing methodology follows the repo convention: teacher_forced_decode_bench only,
  never free-running greedy decode.

Reproducing

cargo run --release --example teacher_forced_decode_bench -- \
    --model <kimi-k3-dir> --steps 30 --expert-cache 896
cargo bench --bench kernels -- mxfp4

 numactl --interleave=all ./target/release/rabbit --model /data/hf/hub/kimi-k3 --serve --port 8000 --expert-cache 896 --no-usage-cache --preload-experts --threads 48 --numa --numa-threads 192