[gspark-etri]

Background

In the ai-rm end-to-end run (ai-computing/ai-rm#32, docs/baselines.md HUB-34), two training processes started directly on the second RNGD node, outside Kubernetes, held both of its cards. The Furiosa device plugin (allocatable 2, none allocated) and rm-accel's AcceleratorProfiles still showed both cards as free. The scheduler and rm-controller could therefore pick them, and the decode worker failed with EBUSY.

Changes

• rm_abstract/airm/occupancy.py (new) finds who holds each card. Everything it does is read-only.
  • Furiosa: furiosa-smi ps --format json. dev_name looks like npu0:[0, 7].
  • Rebellions: the driver's /sys/class/rebellions/rbln<N>/ctx_info, which is the source rbln-stat reads. rbln-stat --json contexts is used as well, but only when its npu index maps to exactly one device. rbln-stat restarts npu at 0 in each group, so on a 4-card host every device reports npu: 0 and a context cannot be matched to a card from that field alone.
  • /proc/<pid>/fd links to compute device nodes (/dev/rngd/npu<N>ch*|pe*, /dev/rbln<N>). Management and BAR nodes are ignored because exporters open those too.
  • Each holder is classified from /proc/<pid>/cgroup: a path containing kubepods means k8s, anything else means host. Only the executable's basename is kept. Arguments, users and hostnames are dropped. Monitoring tools such as furiosa-smi and rbln-stat are not counted as holders.
  • Host processes are only visible from the host PID namespace. This is detected by checking whether PID 1's cgroup contains kubepods. When they are not visible, the daemon logs a warning and publishes no occupancy keys rather than reporting every card as free.
• The provider adds these keys to each accelerator's extra, which ends up in the AcceleratorProfile spec.extra. All values are strings, as the CRD requires.
  • inUse: "true" or "false".
  • holders: a JSON list such as [{"pid":…,"command":"python3","managed":"host"}].
  • A card held by a host process also gets health: busy and healthy: "false". rm-controller's policy.healthy() already skips healthy: false, so it needs no changes. busy is not in its UNHEALTHY list; it is there to describe the state.
  • A card held only by pods stays available. The device plugin already accounts for those cards.
  • AIRM_OCCUPANCY=0 turns all of this off.
• The daemon adds the node label airm.ai-computing.io/host-held=<n>, which the daemon owns and prunes. It also serves two new metrics on /metrics: rm_accel_accelerators_in_use and rm_accel_accelerators_host_held, with labels {component,vendor,model}. They use an rm-accel prefix because rm_core.metrics.METRICS does not list them. Promoting them to shared metrics would be an rm-core change.
• Chart 0.6.0 adds two values: hostPID: true, needed for furiosa-smi ps and /proc to see host processes, and occupancy: true, which sets AIRM_OCCUPANCY. The release is 0.5.0 (pyproject, __version__, chart appVersion). No tag is pushed.
• README in EN and KO (new "Card occupancy" section, env and chart tables), and CHANGELOG.
• Tests: tests/unit/test_occupancy.py (13 tests) with fixtures recorded from the nodes in tests/fixtures/occupancy/. Ids, serials and argument paths are replaced with placeholders. The existing test_airm.py fixture now sets AIRM_OCCUPANCY=0 so it stays deterministic on Linux CI.

Verification

Node tools: furiosa-smi 2026.4.0 (firmware 2026.4.0), rbln-stat 3.0.0 (KMD 3.0.0), Python 3.10.12 on both nodes, cgroup v2 + containerd (systemd driver). Comparison: ai-rm master 9446511, rm-core v0.1.2.

Real tool formats, read-only. These are the formats behind the fixtures:

• On the second RNGD node's host, furiosa-smi ps --format json returned two entries: {"pid":37839,"dev_name":"npu0:[0, 7]",…} and {"pid":38166,"dev_name":"npu1:[0, 7]",…}. The holders' cgroup is 0::/user.slice/user-1001.slice/session-4.scope, which is classified as a host process.
• In the same node's existing rm-abstract-node pod, which runs without hostPID, furiosa-smi ps --format json returned []. This is the root cause: the pod cannot see host processes.
• On the RNGD+ATOM node, rbln-stat --json showed "contexts": [] and every device at "npu": 0. The context JSON keys (ctx_id, npu, process, pid, priority, ptid, memalloc, status, util_info) and the ctx_info line format (pid, ctx_id, priority, ptid, memalloc, status, util) were read from rbln-stat 3.0.0's bundled rblndevice.py bytecode. No ATOM was in use while recording, so rbln-stat.contexts.json is constructed in that shape and labelled as such.

Unit tests

• Windows, Python 3.11.9, rm-core v0.1.2: python -m pytest tests/unit gave 113 passed and 1 skipped. The skipped test is the fd scan, which needs symlinks.
• Linux, Python 3.10: pytest tests/unit/test_occupancy.py tests/unit/test_airm.py gave 30 passed, fd scan included.
• black --check, ruff check and tools/check_sensitive.py all passed. helm lint passed, and helm template renders hostPID: true / AIRM_OCCUPANCY "1" by default and false / "0" with the values turned off.

Live, read-only dry-run of this branch (python -m rm_abstract.airm.daemon --once --dry-run, source copied to a temp dir and removed afterwards; the training processes were not touched):

where	result
second RNGD node, host PID namespace	furiosa:0 and furiosa:1 are inUse=true health=busy healthy=false, holders [{"pid":37839,"command":"python3","managed":"host"}] and [{"pid":38166,…}]. Label host-held=2.
RNGD+ATOM node, host PID namespace	RNGD 0/1/3 and ATOM 0-3 are all inUse=false holders=[]. Label host-held=0.
existing rm-abstract-node pods on both nodes (kubectl exec, deployed chart has no hostPID)	Warning "host processes are not visible (run with hostPID): occupancy not reported". No occupancy keys are published, so cards are not wrongly reported as free.

rm-controller excludes the host-held cards. rm-controller from ai-rm master was run with run_once(..., dry_run=True) through its kube.Kubectl parsing, fed the 11 profiles above with both nodes Ready. The spec asked for 2 prefill and 3 decode RNGD over tcp.

• Profiles as rm-accel 0.4.2 published them (occupancy keys stripped): decode was placed on rngd-node-2/furiosa:0 and rngd-node-2/furiosa:1. That reproduces the EBUSY placement.
• Profiles from this branch: no free furiosa.RNGD x2 for prefill and furiosa.RNGD x3 for decode (tcp); skipped unhealthy rngd-node-2/furiosa:0, rngd-node-2/furiosa:1.
• With 1 prefill and 2 decode: prefill went to rngd-atom-node/furiosa:0 and decode to rngd-atom-node/furiosa:1,3. Only free cards were used.

Compatibility and limits

• Additive: new spec.extra keys, one label and two metrics. Profiles still parse with rm-core from_profile. Deployments that set hostPID: false or occupancy: false behave as before, apart from the warning.
• hostPID: true lets the daemon, which is already privileged, see all host processes. It only reads cmdline, cgroup and fd links.
• Not verified live:
  • k8s-held cards. No NPU pod was running anywhere in the cluster during the run. The case is covered by the real kubepods cgroup line in the fixtures.
  • A busy ATOM. All contexts were empty.
• The deployed DaemonSet needs this chart (hostPID) before the cluster sees occupancy. Until then the live pods publish nothing new, as shown above.
• NVIDIA GPUs get inUse/holders only from the fd scan, which has no /dev/nvidia<N> pattern yet, so they always read free. GPUs are out of scope for Cards held by host processes outside Kubernetes are reported as free #9.
• A host process that opens a card only briefly can mark it busy for one round (60 s). Known monitoring tools are excluded.
• health: busy is not in rm-controller's UNHEALTHY list. Exclusion relies on healthy: "false". Adding busy there, and showing "busy" rather than "unhealthy" in the controller's reason, would be an optional ai-rm follow-up.

Related Issue

Closes #9

🤖 Generated with Claude Code