draft: hardware partition and XCP capacity

README.md

@@ -73,6 +73,140 @@ Verify registration:
 kubectl get node <node-name> -o jsonpath='{.metadata.annotations.hami\.io/node-amd-register}'
 ```
 
+## Operating modes
+
+The plugin follows HAMi's Ascend vNPU model: one plugin registration, per-pod mode
+selection. Every device is registered in both forms and the scheduler picks the
+form per pod based on the pod annotation `hami.io/amd-mode`:
+
+- absent or `rocm` (default): soft mode. The device is published as a CU-maskable
+  device (Count 10) and the scheduler allocates CU slices via the amd-hami-core
+  hook library (`HSA_CU_MASK`, `HIP_DEVICE_MEMORY_LIMIT`, `LD_AUDIT`).
+- `spx`, `cpx`, `dpx` or `qpx`: hard partition mode. The same XCP compute
+  partition is published as a whole device (Count 1, ID `<rocr-uuid>#<mode>`)
+  and the scheduler allocates it exclusively without a CU mask. The node must be
+  switched to the matching compute partition profile first, e.g.:
+
+  ```bash
+  echo spx > /sys/class/drm/card0/device/current_compute_partition
+  echo nps1 > /sys/class/drm/card0/device/current_memory_partition
+  reboot   # partition changes require a reset
+  ```
+
+  Physical Instinct parts expose XCP compute partitions and serve both modes,
+  discrete GPUs and AI MAX APUs (MI300A/MI350A) alike. Virtio Instinct devices
+  (SR-IOV VFs) and partition-less embedded APUs register no hard entries and
+  only serve rocm mode; `hami.io/amd-mode: spx` cannot be scheduled there.
+
+The register annotation (`hami.io/node-amd-register`) carries one entry per
+form: soft entries have `Mode` empty, hard entries carry `Mode` equal to the
+compute partition type. The scheduler branches on `Mode` exactly like HAMi
+branches on the NVIDIA `MigMode` and the Ascend `huawei.com/vnpu-mode` values.
+Allocated devices must be written back as the published `DeviceInfo.ID`
+(`amd-smi` UUID for whole GPUs, `GPU-<unique_id>` or `<rocr-uuid>#<mode>` for
+partitions); the plugin resolves all of them.
+
+Whole-GPU devices whose KFD `unique_id` is 0 (embedded APUs without a PCI
+Device Serial Number) cannot be addressed by `GPU-<unique_id>`; ROCr addresses
+them by agent index and the plugin publishes that index as the ROCr-visible
+id. Such nodes expose no XCP partitions and only serve rocm mode; parts with a
+PCI Device Serial Number register by `GPU-<unique_id>` instead.
+
+### Partition profiles
+
+An Instinct GPU can be carved into compute partitions; which partitions exist
+and how they split the silicon depends on the selected profile. MI355X-class
+GPUs (8 XCCs) support four profiles, reported per GPU by
+`amd-smi partition -g <gpu-id> --json`:
+
+| profile | type       | memory caps | partitions x XCC |
+|---------|------------|-------------|-------------------|
+| 0       | SPX (default) | NPS1     | 1 x 8             |
+| 1       | DPX        | NPS1,NPS2   | 2 x 4             |
+| 2       | QPX        | NPS1        | 4 x 2             |
+| 3       | CPX        | NPS1        | 8 x 1             |
+
+- `profile` is the `profile_index` passed to the AMD SMI setter
+  (`amdsmi_set_gpu_accelerator_partition_profile`). The amd-smi CLI marks the
+  current profile with `*` (SPX here).
+- `partitions x XCC` is what the profile yields: e.g. QPX splits the GPU into
+  4 partitions of 2 XCCs each. Every partition appears as an XCP device
+  (`/sys/devices/platform/amdgpu_xcp_*`) and is registered by the plugin as a
+  hard entry of that mode.
+- `memory caps` lists the NPS modes the profile can run with (DPX also works
+  under NPS2; the others are NPS1-only on this part).
+
+Changing the profile requires the GPU to be idle (no workloads; see the
+`amdsmi_set_gpu_*_partition` docs). Compute partition changes are per-GPU and
+take effect live - the XCP devices appear without a reset. A memory partition
+change requires an amdgpu driver reload, which resets the device nodes of
+every GPU on the node: that is a node-wide quiet-window operation, not a
+single-GPU one.
+
+Each XCP entry is published with an even share of the whole-GPU capacity:
+VRAM and CU count divided by the number of XCP partitions of that GPU. Floor
+division is used, so the advertised values under-commit rather than
+over-commit. Soft entries keep the whole-GPU capacity; soft mode slices it
+with CU masks.
+
+The plugin registers the partitions of the mode each GPU is currently in, and
+advertises the available profiles per GPU (`partitionProfiles` in
+DeviceInfo.CustomInfo of the register annotation) so schedulers can see which
+modes a GPU can be switched to.
+
+### Profile storage
+
+Profile data is stored in this plugin's register annotation
+(`hami.io/node-amd-register`), in the same shape HAMi's NVIDIA device plugin
+uses for MIG profiles (that plugin lives in the Project-HAMi/HAMi scheduler
+repository, not here). It queries NVML at registration and publishes per-GPU
+MIG profile capacity as `migProfiles` inside each device entry of
+`hami.io/node-nvidia-register`; the scheduler reads the annotation and never
+queries hardware. This plugin does the same on the AMD side: it discovers the
+profiles from AMD SMI at registration and publishes them per whole GPU under
+`custominfo.partitionProfiles`, one entry per profile with `profile_index`,
+partition type, memory caps, partition count and XCC per partition.
+
+Both MIG and AMD partition switching require an idle GPU. HAMi's NVIDIA
+plugin start-up refuses to reset MIG-enabled GPUs that still have running
+allocations, and the AMD SMI setter rejects the switch when workloads are
+running. What differs is when switching happens and what it costs:
+
+- MIG: enablement is a one-time node-level operation at plugin start-up and
+  resets the GPU; afterwards instances are carved on demand from the fixed
+  NVML profile menu.
+- AMD: a compute partition switch is per-GPU and live - the XCP devices
+  appear without a reset; a memory partition switch requires a node-wide
+  amdgpu driver reload.
+
+In both cases the stored profiles are a static menu of what the silicon can
+do. The annotation carries the current mode (the device entries themselves,
+with their `Mode` and `partitionProfile` fields) alongside the modes the GPU
+can be switched to (`custominfo.partitionProfiles`). Allocation state stays
+in the pod annotations, as XCP device IDs for hard partitions and
+`hami.io/amd-cu-allocated` CU ranges in soft mode.
+
+### Memory partitions are not an allocation mechanism
+
+Memory partitions (NPS1/NPS2/NPS4/NPS8) are a NUMA-domain knob: they control
+how the GPU's memory is interleaved across NUMA nodes at the system level,
+not which pods get which bytes. They are fixed at boot (BIOS/PSP) and
+changing them requires an amdgpu driver reload that resets the device nodes
+of every GPU on the node.
+
+HAMi does not read or change the memory partition. Per-pod memory limits are
+enforced in software (`HIP_DEVICE_MEMORY_LIMIT` and the `libamvgpu.so`
+hook), so the NPS mode has no effect on what the scheduler can allocate.
+The allocation-relevant memory number is the per-XCP slice described above,
+derived at plugin start from the whole-GPU capacity and the current
+partition count.
+
+Using memory partitions for scheduling would only make sense if workloads
+needed OS-level NUMA locality per partition (NPS2 halves the NUMA node
+size). For HAMi's slicing model that benefit is rarely worth the cost - a
+boot-time setting and a node-wide driver reload to change - so the plugin
+treats NPS as a fixed node property.
+
 ## Memory-isolation compatibility
 
 CU isolation and device visibility use ROCr interfaces and are independent of the workload image's libc. Fractional-memory enforcement is different: it depends on loading `/usr/local/vgpu/libamvgpu.so` through glibc `LD_AUDIT`.

cmd/k8s-device-plugin/main.go

@@ -76,9 +76,12 @@ func getResourceList(resourceNamingStrategy ResourceNamingStrategy) ([]string, e
 			}
 		}
 	} else {
-		// Heterogeneous node reports resources based on partition types if strategy is mixed. Heterogeneous is not allowed if Strategy is single
+		// Heterogeneous node reports resources based on partition types if strategy is mixed. With the single
+		// strategy the kubelet list still carries every device, so mixed styles (e.g. one busy GPU stuck in spx
+		// while the rest flipped to qpx) must not abort the plugin; the devices register under "gpu" as-is.
 		if resourceNamingStrategy == StrategySingle {
-			return resources, fmt.Errorf("Partitions of different styles across GPUs in a node is not supported with single strategy. Please start device plugin with mixed strategy")
+			glog.Warningf("Partitions of different styles across GPUs in a node; reporting all devices under %q", "gpu")
+			resources = []string{"gpu"}
 		} else if resourceNamingStrategy == StrategyMixed {
 			for partitionType, count := range partitionCountMap {
 				if count > 0 {

docs/user-guide/configuration.md

@@ -9,6 +9,35 @@ The device plugin can be configured using the following environment variables:
 | Environment Variable | Type | Default | Description |
 |-----|------|---------|-------------|
 | `AMD_GPU_DEVICE_COUNT` | Integer | Auto-detected | Number of AMD GPUs available on the node |
+| `OPERATING_MODE` | String | `cu` | Registration mode: `cu` (soft whole-GPU + CU slices) or `partition` (hard compute partitions). Overridable per node via the `hami.io/amd-operating-mode` node annotation. |
+
+## Operating Modes
+
+The plugin registers AMD GPUs in one of two modes:
+
+- **cu mode (default)**: each whole GPU registers as a soft device (default
+  split count, `Mode` empty) and is sliced by CU and memory. This is the
+  historical HAMi behavior.
+- **partition mode**: devices register as hard compute partitions with a
+  `#<compute-type>` suffix (`GPU-xxx#spx`, `GPU-xxx#qpx`), `Mode` set to the
+  partition type and `Count: 1`. The kubelet-facing device list publishes
+  only the hard entries (no soft splits). Whole GPUs with more than one
+  partition (dpx/qpx/cpx) are replaced by their `amdgpu_xcp_*` partitions;
+  single-partition (spx) whole GPUs register as one hard device.
+
+Mode resolution, lowest to highest precedence:
+
+1. Default `cu` when nothing is set.
+2. `OPERATING_MODE` environment variable (chart value `dp.operatingMode`).
+3. The `hami.io/amd-operating-mode` node annotation, which overrides the env
+   per node (the NVIDIA `nvidia.com/mig.config` pattern).
+
+The annotation is read once at plugin start; changing it requires a plugin
+pod restart. The plugin logs the resolved mode as `operating mode: <mode>`.
+
+Partition counts are per GPU: `amd-smi set -g <id> --compute-partition qpx`
+flips a single GPU; mix spx and qpx GPUs on one node. See
+[MI355X Partitioning](mi355x-partitioning.md) for kernel caveats.
 
 ### Why Limit GPU Exposure?
 

docs/user-guide/mi355x-partitioning.md

@@ -0,0 +1,73 @@
+# MI355X (gfx950) Partitioning
+
+The MI355X splits into hardware compute partitions with `amd-smi set`:
+
+| Profile | Partitions per GPU | CUs per partition | Mode suffix |
+|---------|--------------------|-------------------|-------------|
+| spx     | 1                  | 256               | `#spx`      |
+| dpx     | 2                  | 128               | `#dpx`      |
+| qpx     | 4                  | 64                | `#qpx`      |
+| cpx     | 8                  | 32                | `#cpx`      |
+
+Each GPU has 256 CUs and 294896 MiB of VRAM (288 GB). Partition mode
+requires the plugin's `partition` operating mode; see
+[Configuration](configuration.md#operating-modes).
+
+## Registration by kernel
+
+Whether the plugin registers whole GPUs or `amdgpu_xcp_*` partitions depends
+on the kernel's KFD topology, not on the plugin:
+
+- **Kernels whose KFD topology exposes XCP render minors** (the platform
+  devices under `/sys/devices/platform/amdgpu_xcp_*` have `renderD` minors
+  listed in `/sys/class/kfd/kfd/topology/nodes`): each partition registers
+  as a hard `#<mode>` device with an even share of VRAM and CUs (qpx: 4
+  devices of 73728 MiB and 64 CUs per GPU). This is the full XCP path.
+- **Kernels without XCP render minors in KFD topology** (observed on
+  kernel 6.8.0-136 in spx mode): the plugin's KFD-validity gates drop every
+  `amdgpu_xcp_*` entry. Only the 8 whole-GPU nodes register. Note that the
+  KFD topology changes after a hardware partition flip: on 6.8.0-136, after
+  the plugin flips a GPU to qpx, the XGMI reset re-probes KFD and 3 of the 4
+  partition nodes per GPU appear (21 XCP entries for 7 qpx GPUs, verified
+  live). The missing 4th node is dropped by the same render-minor gate.
+  Additionally, the partitions of one GPU share a single ROCr unique_id on
+  this kernel, so the scheduler sees 3 devices with one ID and can only
+  allocate one partition per GPU safely; distinct per-partition unique_ids
+  need a kernel/driver that reports them.
+
+## Known kernel caveat: 6.8.0-136
+
+> On 6.8.0-136 the KFD topology lacks XCP render minors, so QPX/DPX nodes
+> register one hard whole-GPU `#qpx` entry per GPU (full 256 CU / 294896 MiB
+> capacity), not 4 XCP partitions. The registration and capacity math are
+> ready for XCPs on a kernel that exposes them; until then a QPX pod
+> requesting 64 cores / 73728 MiB still pins one GPU and gets memory- and
+> CU-limited correctly.
+
+The capacity divisor for XCP entries comes from the AMD SMI partition
+profile's `NumPartitions` for the current compute type, not from counting
+`amdgpu_xcp_*` children (gfx950 exposes 7 XCD chiplets per GPU regardless of
+partition mode, so child counts are wrong divisors).
+
+## Verifying on a node
+
+```bash
+# KFD topology node count and simd counts; XCP partitions appear as nodes
+# with simd_count 256 (qpx) and a render node in the same topology.
+cat /sys/class/kfd/kfd/topology/nodes/*/properties | grep -E "simd_count|gfx_target_version"
+
+# Platform partition devices and their render minors:
+ls /sys/devices/platform/amdgpu_xcp_*/drm/
+
+# Current compute partition per GPU:
+amd-smi static -g 0 | grep "Compute Part"
+
+# Flip one GPU to qpx (per GPU, not per node):
+amd-smi set -g 0 --compute-partition qpx
+```
+
+After a plugin restart, the node annotation
+`hami.io/node-amd-register` shows the registered devices: soft entries
+(`Count: 10`, empty `Mode`) in cu mode, hard entries (`Count: 1`, `Mode:
+qpx`, `#qpx` suffix) in partition mode. The plugin log reports
+`operating mode: <mode>` and `ListAndWatch: sending N split devices`.

example/vllm-qpx-serve/README.md

@@ -0,0 +1,52 @@
+# vLLM serving on QPX hard partitions (MI355X)
+
+QPX splits each MI355X GPU into 4 compute partitions of 64 CUs and
+73728 MiB (288 GB / 4). Pods request a whole partition by asking for
+the full per-partition memory and core count:
+
+- `amd.com/gpu: 1` (one partition)
+- `amd.com/gpumem: 73728` (MiB, one QPX partition)
+- `amd.com/gpucores: 64` (CUs, one QPX partition)
+
+## Prerequisites
+
+1. Device plugin in `partition` mode with QPX as the target partition.
+   The plugin flips the GPUs itself through the AMD SMI API at startup;
+   no manual `amd-smi set` on the node is needed. Either set the chart
+   values (`dp.operatingMode: partition`, `dp.computePartition: qpx`),
+   or per node:
+
+   ```
+   kubectl annotate node <gpu-node> hami.io/amd-operating-mode=partition
+   kubectl annotate node <gpu-node> hami.io/amd-compute-partition=qpx
+   kubectl delete pod -n kube-system -l ... # restart the device-plugin pod
+   ```
+
+   The annotations override the chart values on that node (the NVIDIA
+   `nvidia.com/mig.config`-style per-node switch). GPUs with running
+   workloads fail to flip and keep their current mode; the plugin logs
+   each per-GPU result and registers whatever mode the hardware is in.
+
+2. HAMi scheduler device config with the `amd` section (see
+   device-config.yaml). On kernels whose KFD topology does not expose
+   amdgpu_xcp_ nodes (e.g. 6.8.0-136), each GPU registers one hard
+   `#qpx` device instead of 4 XCP partitions; requesting a full
+   partition still pins one GPU and limits memory and CUs correctly.
+
+## Deploy
+
+```
+kubectl apply -f hf_token.yaml
+kubectl apply -f deployment.yaml
+kubectl apply -f service.yaml
+```
+
+## Test
+
+```
+kubectl get svc
+curl http://<CLUSTER-IP>:80/v1/models
+curl http://<CLUSTER-IP>:80/v1/completions \
+  -H "Content-Type: application/json" \
+  -d '{"model": "mistralai/Mistral-7B-v0.3", "prompt": "San Francisco is a", "max_tokens": 7, "temperature": 0}'
+```

example/vllm-qpx-serve/deployment.yaml

@@ -0,0 +1,63 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: mistral-7b
+  namespace: default
+  labels:
+    app: mistral-7b
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: mistral-7b
+  template:
+    metadata:
+      labels:
+        app: mistral-7b
+    spec:
+      volumes:
+      # vLLM needs to access the host's shared memory for tensor parallel inference.
+      - name: shm
+        emptyDir:
+          medium: Memory
+          sizeLimit: "8Gi"
+      hostNetwork: true
+      hostIPC: true
+      containers:
+      - name: mistral-7b
+        image: rocm/vllm:rocm6.2_mi300_ubuntu20.04_py3.9_vllm_0.6.4
+        securityContext:
+          seccompProfile:
+            type: Unconfined
+          runAsGroup: 44
+          capabilities:
+            add:
+            - SYS_PTRACE
+        command: ["/bin/sh", "-c"]
+        args: [
+          "vllm serve mistralai/Mistral-7B-v0.3 --port 8888 --trust-remote-code --enable-chunked-prefill --max_num_batched_tokens 1024"
+        ]
+        env:
+        - name: HUGGING_FACE_HUB_TOKEN
+          valueFrom:
+            secretKeyRef:
+              name: hf-token-secret
+              key: token
+        ports:
+        - containerPort: 8888
+        resources:
+          limits:
+            cpu: "10"
+            memory: 20G
+            amd.com/gpu: "1"
+            amd.com/gpumem: "73728"
+            amd.com/gpucores: "64"
+          requests:
+            cpu: "6"
+            memory: 6G
+            amd.com/gpu: "1"
+            amd.com/gpumem: "73728"
+            amd.com/gpucores: "64"
+        volumeMounts:
+        - name: shm
+          mountPath: /dev/shm