rm-accel

Accelerator abstraction for heterogeneous AI infrastructure

English · 한국어

Quick start · ai-rm integration · vLLM-compatible API · Architecture · Docs

---

Formerly rm_abstract_layer. The Python package stays rm_abstract, and the node daemon keeps tagging its metrics component="rm_abstract_layer" so existing dashboards and queries keep working.

rm-accel is the L1 accelerator abstraction of ai-rm. It discovers GPUs and NPUs (FuriosaAI RNGD, Rebellions ATOM) on each node, describes them through the rm-core Accelerator model and publishes them to Kubernetes as AcceleratorProfile objects and node labels, which placement (llm-d EPP) and observation (lmd-top) read.

It also provides a vLLM-compatible Python API that runs the same code on GPU or CPU with switchable serving engines (vLLM, Triton, TorchServe).

Maturity: prototype (Alpha). Current release v0.5.0, Helm chart 0.6.0 — see the changelog.

✨ Highlights

Discover devices Read-only vendor tools first (nvidia-smi, furiosa-smi, rbln-stat), SDK-based detection as fallback. Device's own index, model, memory and datasheet peak TFLOPS.

Publish to Kubernetes A node daemon (DaemonSet) applies one AcceleratorProfile per device, labels nodes airm.ai-computing.io/<vendor>.<model> and exports airm_accelerators.

One API, many backends Change from vllm import LLM to from rm_abstract import LLM; pick the device and engine by argument or environment variable.

🚀 Quick start

git clone https://github.com/ai-computing/rm-accel.git && cd rm-accel
pip install -r requirements/requirements.txt
pip install -e .

python -m rm_abstract.airm.daemon --once --dry-run                  # devices on this node, as k8s objects
python -m rm_abstract.airm.daemon --provider sim --once --dry-run   # same, with simulated devices
python -m rm_core.conformance airm.accelerator_providers rm-abstract  # contract check (PASS / SKIP / FAIL)

Installing through ai-rm: make setup COMPONENTS=rm-accel in the ai-rm umbrella (steps come from ai-rm.yaml).

🔌 ai-rm integration

AcceleratorProvider adapter

rm_abstract.airm implements the rm-core v0.1 AcceleratorProvider contract and registers it as rm-abstract in the airm.accelerator_providers entry-point group:

from rm_core import registry

provider = registry.load("airm.accelerator_providers", "rm-abstract")
for acc in provider.discover():
    print(acc.id, acc.vendor, acc.model, acc.memory_bytes, acc.peak_tflops)
# node-a/furiosa:0  furiosa  RNGD  ...  {'bf16': 256.0, 'fp8': 512.0}

Field	Value
id	<node>/<vendor>:<index>; index is the device's own number (npu3 → furiosa:3)
node name	AIRM_NODE_NAME (set from spec.nodeName in the DaemonSet), else the hostname
peak_tflops	Only where the vendor datasheet states it (RNGD, ATOM), with peak_source; unknown otherwise
CPUs	Not reported; a node without accelerators yields []

Node daemon

python -m rm_abstract.airm.daemon discovers periodically and

• applies one AcceleratorProfile (airm.ai-computing.io/v1alpha1) per device, deleting profiles of devices that stayed missing for AIRM_REMOVE_AFTER consecutive rounds,
• patches node labels airm.ai-computing.io/<vendor>.<model>=<count> and airm.ai-computing.io/managed=true,
• serves airm_accelerators{vendor,model} on /metrics.

Card occupancy

A device plugin only counts the cards it gave to pods. A process started on the host, outside Kubernetes, can hold a card the plugin still reports free, and a pod placed there fails with EBUSY. So each round the provider also finds who holds every card, read-only:

• holders from furiosa-smi ps --format json (RNGD), the Rebellions driver's /sys/class/rebellions/rbln<N>/ctx_info (what rbln-stat shows) and rbln-stat --json contexts, and open compute device nodes in /proc/<pid>/fd;
• each holder is a pod (managed: k8s, kubepods in /proc/<pid>/cgroup) or a host process (managed: host).

spec.extra key	Value
inUse	"true" / "false"
holders	JSON list [{"pid": 37839, "command": "python3", "managed": "host"}]; executable basename only, no arguments or users
health, healthy	"busy", "false" when a host process holds the card. rm-controller skips cards with healthy: false, so it never places on them. A card held only by pods stays available: the device plugin accounts for it

The node also gets airm.ai-computing.io/host-held=<n>, and /metrics adds rm_accel_accelerators_in_use and rm_accel_accelerators_host_held ({vendor,model}). Seeing host processes needs the host PID namespace (chart value hostPID, default on). Without it the daemon logs a warning and publishes no occupancy keys, rather than calling every card free.

Variable	Default	Meaning
AIRM_NODE_NAME	hostname	Node name used in ids and labels
AIRM_PROVIDER	rm-abstract	Provider entry point (sim for a demo)
AIRM_INTERVAL_S	60	Discovery interval
AIRM_METRICS_PORT	9400	/metrics port, 0 disables
AIRM_REMOVE_AFTER	3	Missed rounds before a profile is deleted (avoids flapping on a transient tool failure)
AIRM_OCCUPANCY	1	Report card occupancy (0 disables)

Applying to a cluster needs the Kubernetes client: pip install -e ".[k8s]".

Helm chart

deploy/helm/rm-abstract-node installs the daemon as a DaemonSet with a ServiceAccount and ClusterRole (acceleratorprofiles CRUD, nodes get/patch). The AcceleratorProfile CRD ships with rm-core and is installed by llm-d-setup / the ai-rm umbrella chart. The image is ghcr.io/ai-computing/rm-accel (tag = chart appVersion), built from the Dockerfile and published by the Image workflow on master and on v* tags.

helm template rm-abstract-node deploy/helm/rm-abstract-node                      # published image, host tools
helm template rm-abstract-node deploy/helm/rm-abstract-node --set provider=sim --set hostBin= --set hostDev=false
docker build -t rm-accel:dev . && docker run --rm rm-accel:dev --provider sim --once --dry-run

Chart 0.6.0: image, container runtime and site extension points

Value	Default	Meaning
privileged	true	Vendor tools read the device nodes
hostDev	true	Mount the host's /dev live, so a device the driver re-creates (e.g. an ATOM released by a pod) stays in discovery
hostBin	/usr/bin	Host directory with the vendor tools, mounted read-only at /host/usr/bin (on the image's PATH); "" disables
hostPID	true	Host PID namespace, to see processes holding a card (card occupancy)
occupancy	true	Report card occupancy (AIRM_OCCUPANCY)
command	[]	Override the daemon command (default python -m rm_abstract.airm.daemon)
extraEnv	[]	{name, value} entries appended to the daemon's env
initContainers	[]	Pod init containers, verbatim
extraVolumes / extraVolumeMounts	[]	Pod volumes / daemon volume mounts, verbatim

The image carries rm-abstract (without torch), rm-core, the Kubernetes client and numpy, but no vendor tools: those must match the host's driver, so they come from the host through hostBin. The root filesystem is read-only with a writable /tmp (rbln-stat unpacks itself under TMPDIR). The extension points are no longer needed for the published image; they remain for sites with their own image or packages. See values.yaml and the changelog.

🔁 vLLM-compatible API

# from vllm import LLM, SamplingParams
from rm_abstract import LLM, SamplingParams

llm = LLM(model="gpt2")                                  # auto: GPU > CPU
llm = LLM(model="gpt2", device="cpu")                    # or RM_DEVICE=cpu
llm = LLM(model="gpt2", engine="triton")                 # or RM_ENGINE=triton
outputs = llm.generate(["Hello, I am"], SamplingParams(max_tokens=50))
print(outputs[0].outputs[0].text)

Runtime control, engines and errors

import rm_abstract

rm_abstract.init(device="auto")
print(rm_abstract.get_available_resources())   # {'nvidia': True, 'cpu': True}
rm_abstract.switch_device("cpu")

Engine	Managed for you
vLLM (default)	—
Triton	HuggingFace → ONNX conversion, model repository, Docker container start/stop
TorchServe	MAR packaging, server start, model registration, cleanup

from rm_abstract.exceptions import BackendNotAvailableError

try:
    llm = LLM(model="gpt2", device="nvidia")
except BackendNotAvailableError:
    llm = LLM(model="gpt2", device="cpu")

Installation extras: .[gpu], .[triton], .[torchserve], .[serving], .[all] (see requirements guide). More in quick start and serving engines.

Support matrix

Device	Discovery & AcceleratorProfile	Inference through LLM
NVIDIA GPU	✅	✅ vLLM · Triton · TorchServe
CPU	— (not reported as an accelerator)	✅ transformers · TorchServe
FuriosaAI RNGD	✅	🧪 plugin present, not validated
Rebellions ATOM	✅	🧪 plugin present, not validated

🏗 Architecture

flowchart LR
    subgraph node["Node"]
      tools["nvidia-smi · furiosa-smi · rbln-stat"] --> prov["rm_abstract.airm<br/>AcceleratorProvider"]
      det["HardwareDetector<br/>(SDK fallback)"] --> prov
      prov --> d["node daemon"]
    end
    d -- "AcceleratorProfile · node labels" --> k8s[("Kubernetes API")]
    d -. "airm_accelerators" .-> prom[("Prometheus")]
    k8s --> epp["llm-d EPP / placement"]
    k8s --> top["lmd-top"]

Loading

src/rm_abstract/
├── airm/          # ai-rm protocol v0.1: AcceleratorProvider, node daemon
├── resources/     # device plugins: gpu/nvidia, npu/furiosa, npu/rebellions, cpu
├── engines/       # serving engines: vLLM, Triton, TorchServe, model conversion
├── common/        # hardware detection, input/output adapters
├── compat/        # vLLM / transformers compatibility layers
├── registry/      # plugin registry and discovery
├── core/          # configuration, DeviceFlowController
└── llm.py         # vLLM-compatible LLM proxy

🧪 Development

pip install -e ".[dev]"
pytest -m "not gpu and not slow" tests/unit     # quick tests, no accelerator needed
ruff check src tests && black --check src tests
python tools/check_sensitive.py .               # internal addresses, tokens, keys
bash scripts/run_examples.sh                    # run every example (needs models / devices)

Guides: contributing · adding a device · adding an engine.

🗺 Roadmap

• rm-core v0.1 AcceleratorProvider, node daemon, Helm chart
• Node daemon container image ghcr.io/ai-computing/rm-accel
• Validated RNGD / ATOM inference through the LLM API
• Engine-facing backend adapters for Optimus Prime (see the ai-rm engine plan)
• Wider device model: partitions, interconnect topology, live utilization

📚 Documentation

Use	Quick start · Serving engines · Requirements
Understand	Architecture · Project structure · Changelog
Extend	Contributing · Adding resources · Adding engines
ai-rm	Umbrella · rm-core protocol · Component card

Older test reports and slides: docs/archive.

📄 License

BSD-3-Clause. NPU backends need the vendors' own SDKs (Rebellions RBLN SDK, FuriosaAI SDK) under their license terms; this project does not redistribute them.

Third-party licenses

Package	License
PyTorch	BSD-3-Clause
vLLM	Apache-2.0
Triton Inference Server	BSD-3-Clause
TorchServe	Apache-2.0
HuggingFace Transformers	Apache-2.0
vllm-rbln · optimum-rbln	Apache-2.0
FuriosaAI SDK	Apache-2.0

NPU	SDK	License	Access
Rebellions ATOM	RBLN SDK (compiler, driver, runtime)	Proprietary	Request access
FuriosaAI RNGD	FuriosaAI SDK (driver, firmware, runtime)	Proprietary (EULA)	furiosa.ai

_{Part of ai-rm · ai-computing team · ETRI}