diff --git a/design/cpu-inference.md b/design/cpu-inference.md new file mode 100644 index 000000000..10c8c8705 --- /dev/null +++ b/design/cpu-inference.md @@ -0,0 +1,294 @@ +# Serving models on CPUs + +**Status:** Draft +**Date:** October 2026 +**Author:** Nicholas Thomson + +This document proposes how Modelplane provisions node pools with no GPU and +serves models on their CPUs, using +[dra-driver-cpu](https://github.com/kubernetes-sigs/dra-driver-cpu) to make +those CPUs claimable. + +## Summary + +I propose four changes, each usable on its own: + +1. An InferenceClass may omit `accelerator` from its provisioning block. A pool + whose class has none is composed with a new `CPU` role: a standard node + image, no GPU driver and no GPU attachment. +2. `CPU` pools carry a taint of their own, which engine pods tolerate, so that + system workloads stay off them as they stay off GPU pools today. +3. The serving stack installs dra-driver-cpu on a cluster when one of its pools + uses a class with a `dra.cpu` device, and only on those pools. +4. Later, a device request may ask for part of a device's capacity, so that + more than one replica can share a CPU node. + +Changes 1 to 3 are enough to serve a model on CPUs on every provider. Change 4 +is about packing, and can wait. + +## Background + +A small model can be served on CPUs alone, which makes a test target that +costs a fraction of a GPU node. + +Modelplane places an engine only on a device it can claim through DRA. The +scheduler rejects an engine none of whose members match a `claim: DRA` device, +and a `Synthetic` device doesn't count. The only DRA driver a serving stack +installs is NVIDIA's, so a node with no GPU publishes nothing to claim. + +[dra-driver-cpu](https://github.com/kubernetes-sigs/dra-driver-cpu) publishes a +node's CPUs as DRA devices, under the driver and DeviceClass `dra.cpu`. In its +default grouped mode it publishes one device per NUMA node, with the node's +allocatable CPUs as consumable capacity (`dra.cpu/cpu`). A claimed device pins +the container that references the claim through `resources.claims` to a +cpuset, and Modelplane's engine containers already reference their claim that +way. A request that names no capacity consumes the whole device (KEP-5075), so +the engine gets every allocatable CPU in the NUMA node. + +The scheduler places an engine on such a device with no change. It evaluates a +member's CEL selectors against the devices a class declares whatever their +driver, and passes the matched device's `deviceClassName` through to the +replica's ResourceClaimTemplate. + +What's missing is a way to provision a node without a GPU, and to install the +driver on it. Every `spec.provisioning.` block requires an +`accelerator` with `count >= 1`, and on GKE its type is passed to GCP, so a +class can't describe a node without one. `compose-inference-cluster` also +composes every user pool with `role: GPU`, so even on EKS and AKS, where the +block is informational, every pool gets a GPU image, the `nvidia.com/gpu` +taint and a `modelplane.ai/gpu` label. + +## Goals + +- Provision a node pool with no GPU on every provider Modelplane supports, + from an InferenceClass that describes no accelerator. +- Make that pool's CPUs claimable with no step outside Modelplane. +- Keep system components off CPU pools, as they are kept off GPU pools. +- Change nothing for a class that has an accelerator. + +Fractional CPU claims, and packing several replicas onto one node, are a +non-goal for the first three changes. Arm CPU pools are out of scope too: the +EKS non-GPU AMI is `AL2023_x86_64_STANDARD`, and choosing an image by +architecture is a separate change. + +## Proposal + +### InferenceClass + +`accelerator` becomes optional in every `spec.provisioning.` block. +Its meaning is unchanged where it is set. A class with no accelerator must +still declare at least one device, since `spec.devices` keeps `minItems: 1`. +A CPU class declares its `dra.cpu` device: + +```yaml +apiVersion: modelplane.ai/v1alpha1 +kind: InferenceClass +metadata: + name: gke-cpu-c3-standard-8 +spec: + provisioning: + provider: GKE + gke: + machineType: c3-standard-8 + diskSizeGb: 100 + devices: + - name: cpu + claim: DRA + driver: dra.cpu + deviceClassName: dra.cpu + count: 1 + capacity: + cpu: { value: "6" } +``` + +The device's `capacity.cpu` is the node's vCPUs less those the driver reserves +(see [The driver](#the-driver)). It is what CEL selectors read, so a member can +require a floor: + +```yaml +nodeSelector: + devices: + - name: cpu + count: 1 + selectors: + - cel: device.capacity["dra.cpu"].cpu.compareTo(quantity("4")) >= 0 +``` + +### Node pool role + +The cloud cluster XRDs (EKSCluster, GKECluster, AKSCluster, NebiusCluster, +VultrCluster, CivoCluster) gain a third role, `CPU`, beside `System` and `GPU`. +`compose-inference-cluster` composes a pool as `GPU`, with its `gpu` block, +when its class has an accelerator, and as `CPU` when it has none. Today it +hard-codes `role="GPU"` for every user pool and reads `prov.accelerator` +unconditionally, so each of those reads needs a guard, including the Civo +NVLink projection. + +The cloud functions already compose a plain pool for any role other than +`GPU`: the standard AMI on EKS, COS without `guestAccelerator` on GKE, no +`gpuDriver` on AKS, and no `modelplane.ai/gpu` label or GPU taint anywhere. +Every existing `role == "GPU"` check keeps working. The new role exists so a +CPU pool is never mistaken for the injected system pool, which uses `System` +and is told apart from user pools only by its name. + +The `has(zones)` rule in the cluster XRDs applies only to `GPU` pools, but GKE +requires `minItems: 1` when zones are set, and `compose-inference-cluster` +passes an empty list today. It should set zones only when the pool has some. + +### Taint + +System components (the gateway, Prometheus, cert-manager and so on) set no +node selector. They stay off inference nodes only because GPU pools carry the +`nvidia.com/gpu` taint. An untainted CPU pool would take them. + +`CPU` pools carry the label `modelplane.ai/cpu=true` and the taint +`modelplane.ai/cpu=true:NoSchedule`. The label is what the CPU driver selects +on (see [The driver](#the-driver)). `place_pod` in +`compose-model-replica` already adds an `nvidia.com/gpu` toleration to every +engine pod; it adds this one beside it. Pinning system components to the +system pool would also work, but it touches every component in every stack, +while the taint touches one function per cloud and one in the replica. + +### The driver + +The serving stack function sees only its ServingStack, not the cluster's pools +or classes, so the cluster composition tells it which pools need the driver. +This follows the precedent of `spec.gpu.pools`, through which +`compose-inference-cluster` already projects per-pool NVLink settings for +Civo. + +ServingStack gains `spec.cpu.pools`, a list keyed by pool name. +`compose-inference-cluster` lists every pool whose class has a `claim: DRA` +device with `driver: dra.cpu`, for every source including `Existing`: + +```yaml +spec: + cpu: + pools: + - name: cpu-8vcpu +``` + +The list decides only whether the stack installs the driver. Where it runs is +decided by a label every `CPU` pool carries (see [Taint](#taint)), so the +driver needs no per-pool placement however many CPU pools a cluster has. + +`compose-serving-stack` applies a hand-written transform, beside the Civo one, +that appends a `Chart` when the list is non-empty: + +| Field | Value | +|---|---| +| key | `dra-driver-cpu` | +| release | `mp-dra-driver-cpu` | +| chart | `dra-driver-cpu` from `oci://registry.k8s.io/dra-driver-cpu/charts`, 0.3.0 | +| `nodeSelector` | `modelplane.ai/cpu: "true"` | +| `driverConfig.reservedCPUs` | `0-1` | + +The chart's other defaults suit Modelplane. It runs the driver in grouped mode, +and it installs the `dra.cpu` DeviceClass. `reservedCPUs` is set through +`driverConfig` because 0.3.0 deprecates the `args.*` form. Its default +tolerations accept every `NoSchedule` taint, including `modelplane.ai/cpu`, so +the `nodeSelector` is what keeps the driver off GPU pools and the system pool, +not their taints. + +0.3.0 is the oldest chart the design supports. Earlier charts expose neither +`nodeSelector` nor `affinity`, so they can't keep the driver to `CPU` pools. + +The driver is not part of the AICR-generated stacks. The generator only +classifies AICR's own components, and the `stacks-current` check would +overwrite a hand edit. + +Unlike the NVIDIA driver, which every stack installs whatever its pools, this +one is conditional. A cluster with no CPU pool runs nothing new. + +`reservedCPUs` decides what a class should declare as `capacity.cpu`: a node's +vCPUs less two. I propose a fixed value, documented with the InferenceClass, +over a per-pool one. A per-pool value would have to be kept in step with the +class by hand, which is the same problem in a less visible place. + +### Pool status + +InferenceCluster defines a `status.gpuPools` which declare the pools which can +be scheduled to. It lists every pool whose class declares a device, so it +will list CPU pools too, and nothing in an entry is specific to GPUs. I propose +renaming it to `status.schedulablePools`. `status.nodePools` would mirror +`spec.nodePools` more closely, but the schema generator already names the spec +item `NodePool`, and the status item would need a different name. + +### Capacity requests + +Today a member's device request has a name, a count and CEL selectors, and the +rendered ResourceClaimTemplate asks for whole devices. A `dra.cpu` claim +therefore takes the whole NUMA group, and one replica fills a node. + +Sharing a node needs a capacity request on the device: + +```yaml +nodeSelector: + devices: + - name: cpu + count: 1 + capacity: + requests: + cpu: "4" +``` + +That means adding `capacity.requests` to the ModelDeployment and ModelReplica +device requests, carrying it through the scheduler into the +ResourceClaimTemplate's `exactly.capacity.requests`, and checking it against +the device's published capacity when matching. + +That alone doesn't let replicas share a node, because the scheduler's ledger +charges a full node to every member that claims a device. It would need to +account for capacity on shared devices. The cluster also needs KEP-5075's +`DRAConsumableCapacity` feature, and I haven't checked which Kubernetes +versions enable it by default. That is enough work to deserve its own design. + +## Other effects on CPU pools + +These don't block serving, but behave differently on a CPU node: + +- Engine pods set no CPU or memory requests, so they are BestEffort. The + `/dev/shm` emptyDir is memory-backed with no size limit, which on a CPU node + competes with the model's KV cache for the same RAM. +- DCGM metrics are empty. vLLM's own metrics still apply. +- The endpoint picker assumes a KV block size of 16 when the engine doesn't + state one. vLLM's CPU backend may default to another size, which would + degrade prefix-cache routing; recipes should pass `--block-size`. + +## Testing + +`e2e/run.sh` already installs a second DRA driver, the vendored +dra-example-driver, on kind. If dra-driver-cpu runs on kind nodes, which I +haven't tried, an e2e case can run a CPU pool on an `Existing` cluster: a +`dra.cpu` class, the driver installed through the serving stack, and a +deployment whose member selects on `dra.cpu` capacity. + +Unit tests change in `compose-inference-cluster` (role and projection per +source, and the renamed status field), each cloud cluster function (the `CPU` +role and taint), `compose-serving-stack` (the transform, including that a +cluster with no CPU pool composes nothing new), `compose-model-replica` (the +toleration) and `compose-model-deployment` (the renamed status field). + +## Alternatives considered + +### Reusing the System role + +`System` already composes a plain pool, so a CPU pool could use it with no XRD +change. But the injected system pool is distinguished from user pools only by +its name, and nothing reserves that name. A distinct role keeps the two apart +in every function that reads `role`. + +### Synthetic CPU devices + +A class could declare a `claim: Synthetic` CPU device and skip the driver. The +scheduler rejects an engine whose only matches are synthetic, because it would +schedule a pod with nothing binding it to the node's hardware. Relaxing that +rule for CPUs would give the engine no cpuset, and two engines on one node +would compete for the same cores. + +## Open questions + +- What is the minimum Kubernetes version dra-driver-cpu 0.3.0 supports, given + it builds on 1.37, and does every provider Modelplane provisions offer it? +- Which Kubernetes versions on each provider enable `DRAConsumableCapacity`? + That decides whether capacity requests can be relied on.