We build the software that lets inference and training run across accelerators from different vendors — NVIDIA GPUs, FuriosaAI RNGD and Rebellions ATOM NPUs, CPUs and GB10 unified memory — on Kubernetes and llm-d.
ai-rm is the integration hub: five technical layers connected by rm-core contracts, pinned component versions and reproducible measurements.
| Layer | Repository | What it does |
|---|---|---|
| L1 | rm-accel | Discovers GPUs and NPUs on each node and publishes them as AcceleratorProfiles and node labels |
| L2 | hetero-npu-p2p | Same-host KV transport between NPUs of different vendors (RNGD → ATOM) |
| L2 | npu-offload | MoE expert offloading: hot experts in NPU HBM, cold experts on the host |
| L2 | onggi | Content-addressed weights and compiled artifacts: deduplication, node caches, peers, file delta apply and read-only CSI volumes |
| L3 | rngd-pd | Prefill/decode-disaggregated serving across RNGD and ATOM as llm-d pods |
| L3 | npu-train | LoRA and full fine-tuning on inference-only NPUs, with A100-matching quality |
| L4 | hetero-llm-d | A heterogeneous Kubernetes + llm-d cluster: gateway, EPP, model store, monitoring |
| L4 | rm-controller | Topology-aware placement of prefill/decode pools |
| L5 | lmd-top | Terminal console for observing and operating llm-d clusters |
| L5 | rm-bench | End-to-end load, real-trace replay and SLO measurement |
Onggi (옹기) stores shared bytes once across model checkpoints and precompiled accelerator variants, then restores the required files through node caches, peers and Kubernetes CSI. Its P2 implementation includes POSIX/S3 storage, hash verification, garbage collection and delta file restoration. The compile farm and automatic prefetch from placement/autoscaling signals remain planned.
Onggi delivers files; serving runtimes select compatible prepared graphs and perform supported NPU-memory updates. A batch or sequence-length change still needs a compatible precompiled variant. This separates artifact storage from the runtime capabilities of RNGD external weights, ATOM device-input weights and ATOM embedded-weight graphs.
aicomp hosts Optimus Prime (opt_prime), a PyTorch parallelization framework
(pipeline, data and tensor parallelism, LoRA, an inference engine with explicit prefill/decode) that is being extended into the unified
inference runtime for the engine pieces above, and FASOP, a cost-based search for parallelization strategies on
heterogeneous GPU clusters (HPDC '24).
- LLMServingSim — a simulator for heterogeneous and disaggregated LLM serving infrastructure
- LLMServingOptimizer — design-space exploration and planning for LLM serving clusters
Browse by topic: ai-rm, ai-rm-l1 … ai-rm-l5, optimus-prime-source.