Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@

## [Unreleased]

### ai-rm protocol v0.1 (read-only, 2026-10-19)
- Infra ▸ Devices shows an ai-rm panel when the cluster has any ai-rm source: `AcceleratorProfile` (`airm.ai-computing.io/v1alpha1`) vendor·model·memory·peak with its source (the selected device's profile is marked), `airm_*` Prometheus series with rm-core v0.1 names and units (vendor-derived `tok_s`/`req_s`/`power_w`/`accelerators` shown as `[vendor]` when no airm series exists), and the decisions in ConfigMap `airm-placement` (`LMD_AIRM_NS`, default `llm-d`). The device PWR column prefers `airm_power_w{scope="card"}`. Without the CRD, series and ConfigMap the view is unchanged and nothing is warned. Write actions are unchanged.

### Failing containers explain themselves (2026-10-19)
- Containers that are crash-looping, failing to pull, OOMKilled, or restarted in the last 10 minutes are found cluster-wide (from the pod list the inventory already reads) and shown with the line that explains them: the waiting message, the termination message, or — for a few pods per cycle, cached per restart — `logs --previous`. Fatal/panic lines win over later noise; logfmt `msg=… error=…` is reduced to its message.
- Known causes get a fix: image without a build for the node's architecture (`exec format error`), inotify instances exhausted, disk full, config apiVersion or CLI flag rejected by the image (noting a floating tag), the node unable to reach the API server, OOMKilled, missing tag / denied pull. Failures on a NotReady node blame the node and sort last.
Expand Down
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -235,6 +235,9 @@ python3 -m unittest discover -s tests -p 'test_*.py' # 설치 스크립트
| `LMD_PROM` | Prometheus `host:port` (평문 HTTP, NodePort 또는 port-forward) |
| `LMD_NS` | 모델 서버·EPP가 있는 네임스페이스 (기본 `llm-serving`) |
| `LMD_GRAFANA` | Grafana 기본 URL (`:graf` 명령) |
| `LMD_AIRM_NS` | ai-rm ConfigMap `airm-placement`의 네임스페이스 (기본 `llm-d`) |

**ai-rm protocol v0.1 (읽기 전용).** Infra ▸ Devices 아래에 ai-rm 패널이 나온다: `AcceleratorProfile`(`airm.ai-computing.io/v1alpha1`)의 벤더·모델·메모리·peak(출처), Prometheus `airm_*` 지표(rm-core `metrics.py`의 이름·단위, 없으면 기존 vendor 값을 `[vendor]`로 표시), ConfigMap `airm-placement`의 배치 결정과 이유. 장치별 PWR 열은 `airm_power_w{scope="card"}`가 있으면 그 값을 쓴다. CRD·지표·ConfigMap이 모두 없으면 패널은 나오지 않는다. `airm-placement`의 각 data 키는 워크로드 이름, 값은 `{"accelerator_ids": [...], "kv_transport": "...", "reason": "..."}`(JSON 또는 YAML)이다.

## Contributing & license

Expand Down
3 changes: 3 additions & 0 deletions ai-rm.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
name: lmd-top
protocol: "0.1"
layer: l5-ops
kind: tool
summary: llm-d 클러스터의 가속기·서빙·EPP 라우팅·SLO를 한 화면에서 보고 조정하는 터미널 관측·운영 콘솔(ai-rm 관측 콘솔)
Expand All @@ -19,3 +20,5 @@ interfaces:
provides: [Metrics, Placement] # --json 상태·KPI 관측, heterogeneous-capacity EPP 정책(contrib/epp-score-observer)
uses: [Metrics, Accelerator, Workload, Placement] # Prometheus 지표, 가속기 팩·device plugin 리소스, serving/학습 Job, InferencePool/EPP
kpi: [이종자원수, SLO, 전력효율]
adapters: [] # Rust; reads rm-core formats (AcceleratorProfile, airm_* series, airm-placement) without importing rm-core
metrics: [] # reads airm_* only, reports none
3 changes: 3 additions & 0 deletions cluster.example.env
Original file line number Diff line number Diff line change
Expand Up @@ -12,3 +12,6 @@ LMD_NS=llm-serving

# Grafana base URL used for dashboard links
LMD_GRAFANA=http://<grafana-host>:<port>

# Namespace of the ai-rm ConfigMap airm-placement (read-only placement decisions)
LMD_AIRM_NS=llm-d
Loading
Loading