From c3efff5250ae5b4e086fcd916adfffe180178eb4 Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Mon, 5 Oct 2026 10:56:48 -0700 Subject: [PATCH 1/9] Add the design for ModelCache sources and the mount contract A ModelCache reads from HuggingFace only, stages to a volume on each matched cluster, and publishes a phase. A consumer calculates the name of that volume, and two functions hold a copy of a calculation that one source makes true. This design adds an OCI source and an Existing source, and a mount contract on each cluster entry. The contract holds the volumes, the volume mounts and the environment that a consumer adds to a pod, in the Kubernetes types of those names. The consumer copies it and never learns the source. Section 4.4 gives the faults the current design permits, from the benchmark work: #495, where an engine starts while the cache stages and reads a 464 GB model a second time; #494, where a replaced node strands the staging volume; and #498, which asks for the cause on the control plane. Section 6 says what model delivery should measure, because the performance claim this design rests on has no series behind it today. Co-Authored-By: Claude Opus 5 Signed-off-by: Dennis Ramdass --- design/modelcache-sources.md | 422 +++++++++++++++++++++++++++++++++++ 1 file changed, 422 insertions(+) create mode 100644 design/modelcache-sources.md diff --git a/design/modelcache-sources.md b/design/modelcache-sources.md new file mode 100644 index 000000000..14101255a --- /dev/null +++ b/design/modelcache-sources.md @@ -0,0 +1,422 @@ +# ModelCache sources and the mount contract + +**Status:** draft +**Date:** 2026-10-05 +**Author:** Dennis Ramdass +**Supersedes:** [#362](https://github.com/modelplaneai/modelplane/pull/362) + +## 1. Introduction + +A `ModelCache` reads a model from one place. It is HuggingFace. The cache +writes the model to a volume on each cluster that it matches. A pod that uses +the model calculates the name of that volume. + +Each of these three statements causes a problem. + +A customer who keeps models in a private registry must send the model to +HuggingFace first. If the weights are already on a disk, there is no way to +tell Modelplane this. The calculation of the volume name is correct for one +source only. Two functions hold a copy of it. + +This document adds two sources and one status field. The status field tells a +consumer how to read the model on one cluster, so the consumer does not +calculate it. + +## 2. Terms + +This document uses these terms. Use each term only as this table tells. + +| term | meaning in this document | +|---|---| +| artifact | the bytes of one model, in a registry or on a volume | +| cache | one `ModelCache` object | +| source | the value of `spec.source`: `HuggingFace`, `OCI` or `Existing` | +| consumer | a function that adds the artifact to a pod, today `compose-model-replica` | +| cluster entry | one item of `status.clusters[]`, for one inference cluster | +| mount contract | the `mount` field of a cluster entry | +| image volume | a volume that the kubelet makes from a container image | +| model artifact | an OCI artifact that obeys the [model spec](https://modelpack.org/) | +| driver | the [model CSI driver](https://github.com/modelpack/model-csi-driver) | +| stage | to copy an artifact to a volume that Modelplane makes | +| mount | to make an artifact readable in a pod | +| resolve | to read the manifest of a reference and find which kind of artifact it is | +| collector | the OpenTelemetry collector that Modelplane composes onto each inference cluster | +| cold pull | a read of the artifact onto a node that does not hold it | +| warm pull | a read of the artifact onto a node that holds it already | + +## 3. Scope + +This design must do these four tasks: + +1. Read an artifact from an OCI registry, with no copy to a volume first. +2. Mount a volume that the customer filled. +3. Tell each consumer how to read the artifact on its own cluster. +4. Tell the user why a cluster cannot read the artifact. + +This design does not do these tasks: + +- It does not name a cache by the digest of the artifact. The name stays the + name that the user wrote. +- It does not find two caches that hold the same weights. +- It does not make a model catalog. +- It does not copy an artifact from one cluster to a different cluster. + +## 4. Background + +### 4.1 The cache reads from one place + +`spec.source` has one value, and that value is `HuggingFace`. A customer whose +models are in Artifact Registry or ECR must send them to HuggingFace to use +Modelplane. A fleet with no connection to the internet has no way to say that +the model is on a disk already. + +### 4.2 The volume is a ReadWriteMany volume + +More than one node of a gang reads the artifact together. Only a ReadWriteMany +volume permits this. The requirement selects Filestore on GKE and +EFS on EKS. +[modelcache.md](./modelcache.md) tells the reader that these volumes can make a +cache slower than no cache. + +Three reports give the cost of that choice. +[#204](https://github.com/modelplaneai/modelplane/issues/204) measures a slow +load from an EFS-backed cache. +[#414](https://github.com/modelplaneai/modelplane/issues/414) shows that +`sizeGiB` has no effect below 1 TiB on GKE, because the managed storage class +is Filestore Enterprise. +[#383](https://github.com/modelplaneai/modelplane/issues/383) asks for the +measurement that a ReadWriteMany backend must satisfy. That measurement does +not exist. + +An `OCI` source has none of these properties. The kubelet reads the artifact +from the image store of the node. + +### 4.3 The cache does not tell a consumer how to read the artifact + +A cluster entry has `phase` and `message`. It does not have the name of the +volume. The consumer calculates the name in `cache_pvc_name`. The function +`compose-model-cache` keeps a second copy of the same calculation. A comment +tells the reader to change both. + +This calculation is correct for a `HuggingFace` source. It is correct for no +other source. + +### 4.4 Faults that the current design permits + +Three reports from benchmark work show the effect. + +**The engine does not wait for the artifact.** +[#495](https://github.com/modelplaneai/modelplane/issues/495) shows an engine +that started while the cache was still staging. The engine downloaded the same +model again, in parallel with the staging. The two downloads of a 464 GB model +completed in the same second. + +Nothing in the code stops this. The function `compose-model-deployment` reads +`status.cache.storageClassName` from the cluster. This value tells that the +cluster has storage. It does not tell that the artifact is on the storage. The +cache publishes a phase for each cluster, and no function reads it. + +**A volume can stop the cache.** +[#494](https://github.com/modelplaneai/modelplane/issues/494) shows a volume +that stayed Pending. The cloud replaced the node that the scheduler chose. The +volume kept the name of the node that went away. The cache never staged, and +the status gave the reason `Hydrating` for 40 minutes. + +This class of fault is not new. +[#221](https://github.com/modelplaneai/modelplane/issues/221) is the same shape +on GKE: the cache volume never bound, because the Filestore CSI driver was not +enabled on the cluster. Each source that stages to a volume has this failure +mode. A source that mounts from a registry does not. + +**The user cannot see the cause.** +[#498](https://github.com/modelplaneai/modelplane/issues/498) shows that the +status of a deployment does not tell why a model does not serve. Both faults +above needed a person with access to the workload cluster. + +A source that does not use a volume removes the second fault. A contract that +each cluster publishes removes the first and helps with the third. + +## 5. Design + +### 5.1 Overview + +The design has two parts. + +1. `spec.source` gets two more values. `OCI` names a reference in a registry. + `Existing` names a claim that the customer filled. +2. Each cluster entry gets a `mount` field. The field holds the volumes, the + volume mounts and the environment variables that a consumer adds to a pod. + +The first part lets a user name an artifact where it is. The second part lets a +consumer read that artifact without knowledge of the source. + +### 5.2 The source field + +```yaml +spec: + source: # enum: HuggingFace, OCI, Existing + huggingFace: {} # no change + oci: + ref: # a tag or a digest + subPath: # the directory that holds the weights + existing: + claimName: # a claim that is bound on each matched cluster + subPath: +``` + +The kubelet does all the work that an `OCI` source needs. It pulls the +reference when the pod starts. It mounts the filesystem read-only. It gives the +second pod on that node the bytes that the first pod pulled. Modelplane makes +no object for each cluster. The source costs one field, not a second staging +procedure. + +Use a digest. A tag is resolved again at each pod start. If a person moves the +tag, the subsequent pod serves different weights. + +An `OCI` source names no credential on the cache. The pod of the engine spends +the credential, and that pod already takes `imagePullSecrets`. A `HuggingFace` +source is different: a Job that Modelplane makes spends that token, and there +is no pod of the user to hold it. + +The field `sizeGiB` applies to `HuggingFace` only. That source is the only +source that puts bytes on a volume that Modelplane makes. + +### 5.3 What each source needs from a cluster + +| source | what the user names | what mounts it | what the cluster needs | +|---|---|---|---| +| `HuggingFace` | a repository ID | a volume that Modelplane stages | a ReadWriteMany storage class | +| `OCI` | a container image that holds the weights | the kubelet, as an image volume | Kubernetes 1.36 | +| `OCI` | a model artifact | the driver | Kubernetes 1.25 | +| `Existing` | a claim that the user filled | the kubelet, as a claim | the claim bound on the cluster | + +Modelplane resolves the reference and finds which kind of artifact it has. The +user names a tag and does no more. An artifact of a third kind gives a `Failed` +cluster entry with the content of the manifest in the message. + +### 5.4 The mount contract + +Each cluster entry gets this field. Each part is a subset of the Kubernetes +type of the same name. + +```yaml +mount: + volumes: [] # corev1.Volume: persistentVolumeClaim, image or csi + volumeMounts: [] # corev1.VolumeMount: name, mountPath, subPath, readOnly + env: [] # corev1.EnvVar: name, value +``` + +The consumer copies these three lists into the pod. It does not read them. The +function that knows the source writes the contract. The function that makes the +pod does not learn the source. + +### 5.5 What each cluster reports + +A cluster entry gives these answers: + +- if the artifact is readable on this cluster +- how to read it +- why it is not readable, when it is not + +`InferenceCluster.status.cache` gets a boolean `imageVolumes`. It tells if the +cluster can mount an image volume. The field follows `storageClassName`, which +is in the same status for the same reason. + +The boolean gates one kind of artifact, not the source. A model artifact uses +the driver, and the version of Kubernetes does not change this. A container +image on a cluster below 1.36 gives a `Failed` entry that names the version. + +This distinction is necessary because the clouds do not agree. Modelplane +defaults to 1.36 for EKS and Vultr. The Regular channel of GKE gives 1.35. AKS +and Nebius give 1.34. + +One cluster that fails does not change a different cluster. Each entry comes +from the observed resources of its own cluster. + +### 5.6 How a consumer reads the contract + +The consumer finds the entry for its own cluster. It puts the `env` list first +and adds the two other lists. The `env` list is first because `native.py` puts +the entries of Modelplane first. This order lets a variable of the user be the +last word. + +The consumer acts on one field rather than copies it. Where a volume mount is +read-only, the consumer adds an `emptyDir` and points `HOME` at it. Engines +write tokenizer files, compile files and lock files beside the weights. `HOME` +sends these files to the `emptyDir`. The `emptyDir` has a size limit, because a +compile cache can be some gigabytes, and a volume with no limit removes the +pod. + +**The consumer stops when the entry is not ready.** This is the correction for +[#495](https://github.com/modelplaneai/modelplane/issues/495). If the cluster +has no entry, or the entry has no `mount`, the consumer writes a condition and +composes no workload. The engine does not start before the artifact is +readable. + +[#186](https://github.com/modelplaneai/modelplane/issues/186) made +`compose-model-deployment` intersect the two cluster selectors. Labels cannot +tell that a cluster inside that set has not completed its staging. The contract +tells this. + +## 6. Telemetry + +A `ModelCache` publishes no metric today. The surface in +[design/telemetry.md](./telemetry.md) names one, `modelplane_replica_cache_hit`, +and that metric reports only whether a replica used a cache. + +The faults in section 4.4 show the cost. The cache in +[#494](https://github.com/modelplaneai/modelplane/issues/494) reported +`Hydrating` for 40 minutes, and no series recorded it. The engine in +[#495](https://github.com/modelplaneai/modelplane/issues/495) read the model +twice, and the second read appeared in one engine log and nowhere else. + +### 6.1 How long a pod waits for the artifact + +This design says that an `OCI` source is faster than a volume. The study in +section 4.2 gives 11.7 seconds for a warm pull and 40.7 minutes over object +storage. Modelplane publishes no series that measures either number. + +Two metrics make the claim readable on a running fleet. + +``` +modelplane_model_pull_seconds histogram +modelplane_model_pull_bytes_total counter +``` + +Each carries three labels beyond the identity that every Modelplane series +has: + +| label | values | what it separates | +|---|---|---| +| `source` | `HuggingFace`, `OCI`, `Existing` | one source from another | +| `mechanism` | `image`, `csi`, `volume` | the kubelet from the driver | +| `cached` | `true`, `false` | a warm pull from a cold one | + +The `cached` label is the one that matters. A warm pull and a cold pull differ +by two orders of magnitude, so one histogram that holds both reports a number +that describes neither. + +The data exists on each node. The kubelet times an image pull, and the driver +times a model artifact. Neither is a pod, so the collector cannot read them +with the scrape jobs that it has: each job uses `role: pod`. A node-level +target is new work in `compose-serving-stack`, and it is the work that makes +this design measurable. + +### 6.2 What state each cache is in + +Four metrics report the cache itself. + +``` +modelplane_cache_cluster_ready gauge, one for each cache and cluster +modelplane_cache_ready_seconds histogram, created until readable +modelplane_cache_staged_bytes gauge +modelplane_cache_size_bytes gauge +``` + +`modelplane_cache_cluster_ready` carries the phase as a label, so a cache that +stays in one phase is visible as a series that does not change. The pair of +byte counts gives the progress that +[#495](https://github.com/modelplaneai/modelplane/issues/495) had to read from +a file listing. + +**These four are blocked by +[#476](https://github.com/modelplaneai/modelplane/issues/476).** Each one reads +`status.clusters[]`, which lives on the control plane, and Modelplane composes +a collector onto each inference cluster only. Nothing on the control plane +collects or exports. #476 holds that work, and these metrics are part of it. + +The contract in section 5.4 makes them possible. Before it, the state of a +cache on one cluster was a phase and a message. After it, each entry says +whether the artifact is readable and why it is not, which is what a metric +needs to report. + +## 7. Limitations + +**A cache reports what it can examine.** Resolution of the reference finds a +repository that is not there, a credential that does not work, and an artifact +of an unknown kind. Resolution does not find that the artifact holds a +different model. It does not find that an `Existing` volume is not complete. +Both of these give a Ready entry and fail when the engine starts. + +**An `OCI` mount uses the disk of the node.** The driver keeps artifacts under +`/var/lib/model-csi`. An image volume goes to the image store, where the +eviction limits of `imagefs` apply to it. The field `sizeGiB` does not control +either one. A model larger than the disk of the node fails at pod start. Make +the disk of the pool large enough for the largest model. + +**The driver is one more component.** A DaemonSet in the path of a pull is a +thing to install and to upgrade. The project is young, and it had no commit +after March 2026. Modelplane installs it on each cluster, because a composition +function sees one cluster and cannot know which artifacts come to it. Its +registry authentication is static configuration. The kubelet authenticates to +ECR, Artifact Registry and ACR with the identity of the node. A fleet that +publishes container images pays neither cost. + +**The driver must outlive the pods that mount through it.** A CSI driver is +necessary to unmount a volume, not only to mount one. If the driver goes while +a pod still holds a model artifact, that pod cannot release the volume and +stays in Terminating. + +[#499](https://github.com/modelplaneai/modelplane/issues/499) is this fault +with a different driver. Deleting an `InferenceCluster` removed the NVIDIA DRA +driver before the engine pods finished, so the pods could not release their +GPUs. The cluster stayed in Deleting for 35 minutes with an idle 8-GPU node. +Modelplane composes no ordering between the serving stack and the workloads of +a `ModelReplica`, so this design adds a second component with the same +exposure. The correction belongs to #499, and this design does not add one. + +## 8. Decisions + +### 8.1 Publish a contract, do not add a child resource + +[#210](https://github.com/modelplaneai/modelplane/issues/210) asks for a +`ModelCacheHydration` child for each cluster, and +[#362](https://github.com/modelplaneai/modelplane/pull/362) proposes one. I +wrote that proposal, and this document replaces it. + +The contract needs one place for each cluster. The field `status.clusters[]` is +that place, and it is there now. A child resource would compose one status +field for the `OCI` and `Existing` sources, and nothing more. The complaint +under the proposal is the organization of one function. A refactor corrects +that, and it adds no API to maintain. + +### 8.2 Use one `OCI` value, not one value for each kind of artifact + +A user who writes a reference frequently does not know if the publisher made a +container image or a model artifact. Modelplane resolves the reference and +finds this. Two enum values would move that work to the user and would give a +`Failed` entry when the user makes an incorrect choice. + +### 8.3 Install the driver on each cluster + +A composition function sees one cluster. It does not see the caches of the +fleet. It cannot know in advance if a model artifact comes to this cluster. A +DaemonSet that does no work uses few resources, where a missing DaemonSet gives +a pod that does not start. + +### 8.4 Use the driver on CRI-O + +CRI-O mounts model artifacts with no driver from v1.33. The function is behind +the flag `--oci-artifact-mount-support`, and OpenShift disables it by default. +An OpenShift cluster could use one path for both kinds of artifact, but this +design does not do that. A second path costs one more thing for a cluster to report and one more branch +to test. Modelplane also provisions no OpenShift clusters. The driver +stops being necessary everywhere on the day that containerd adds the same +function. + +## 9. Subsequent work + +**Materialization on the node.** An init container that copies the artifact to +the node, with a `hostPath` volume. This removes the limit that NFS puts on a +`HuggingFace` read. It also puts each model on each node that can serve it, so +it needs size limits and eviction first. + +**Acceleration of the HuggingFace fetch.** The Job downloads with HTTPS, not +through a container runtime. A proxy or a private CA needs environment, +annotations and volumes on that Job. + +**A cluster entry that names the pod fault.** +[#498](https://github.com/modelplaneai/modelplane/issues/498) asks for the +cause of a fault on the control plane. The contract gives the first part: a +cluster says that the artifact is not readable, and why. The exit code of a +container needs a different source of data. From e6c5ace452af2c3a06557584ea5d7b2475358f8a Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Mon, 5 Oct 2026 11:21:55 -0700 Subject: [PATCH 2/9] Make the introduction argue what the evidence now supports The introduction was written before the benchmark reports went into section 4.2, and it still argued the weaker case. It said a user cannot name their own registry, where the reports say the volume those users are forced through is slower than no cache (#204), does not take the size they asked for on GKE (#414), has no measurement it must satisfy (#383), and fails on its own (#494, #221). An OCI source removes that volume, which is most of why it is worth having. The introduction also named three problems where section 3 lists four tasks. Nothing set up the fourth, that a cluster never says why it cannot read the artifact, which is what #495 cost: an engine started while the cache staged and read a 464 GB model a second time. Each task in section 3 now traces to a problem here. Section 5.1 said the first part lets a user name an artifact where it is. That is true and incomplete, for the same reason. Co-Authored-By: Claude Opus 5 Signed-off-by: Dennis Ramdass --- design/modelcache-sources.md | 34 +++++++++++++++++++++------------- 1 file changed, 21 insertions(+), 13 deletions(-) diff --git a/design/modelcache-sources.md b/design/modelcache-sources.md index 14101255a..7e683bc03 100644 --- a/design/modelcache-sources.md +++ b/design/modelcache-sources.md @@ -7,20 +7,26 @@ ## 1. Introduction -A `ModelCache` reads a model from one place. It is HuggingFace. The cache -writes the model to a volume on each cluster that it matches. A pod that uses -the model calculates the name of that volume. +A `ModelCache` does three things, and each one causes a problem. -Each of these three statements causes a problem. +**It reads a model from HuggingFace, and from nowhere else.** A customer who +keeps models in a private registry must send them to HuggingFace first. A +customer whose weights are on a disk already has no way to say so. -A customer who keeps models in a private registry must send the model to -HuggingFace first. If the weights are already on a disk, there is no way to -tell Modelplane this. The calculation of the volume name is correct for one -source only. Two functions hold a copy of it. +**It stages what it reads onto a ReadWriteMany volume on each matched +cluster.** A read from that volume can be slower than no cache at all. The +volume does not take the size the user asked for on every cloud, and it can +fail on its own. Section 4.2 gives the reports. -This document adds two sources and one status field. The status field tells a -consumer how to read the model on one cluster, so the consumer does not -calculate it. +**It publishes a phase, and a consumer calculates the name of the volume.** The +calculation is correct for one source, and two functions hold a copy of it. +Nothing tells a consumer whether the artifact is readable yet, so an engine +starts before the cache is staged and reads the model a second time. + +This document makes two changes. `spec.source` gets two more values, and one of +them removes the volume rather than filling it. Each cluster entry gets a +`mount` field that says how to read the artifact there. A consumer copies that +answer instead of calculating one. Where there is no answer to copy, it waits. ## 2. Terms @@ -147,8 +153,10 @@ The design has two parts. 2. Each cluster entry gets a `mount` field. The field holds the volumes, the volume mounts and the environment variables that a consumer adds to a pod. -The first part lets a user name an artifact where it is. The second part lets a -consumer read that artifact without knowledge of the source. +The first part lets a user name an artifact where it is. For an `OCI` source it +also removes the volume from the path, with the cost and the failure mode that +section 4.2 gives. The second part lets a consumer read the artifact without +knowledge of the source, and tells it when it must wait. ### 5.2 The source field From 3e70293ed4a58621ada335bb6417853bf7fa7297 Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Mon, 5 Oct 2026 11:26:27 -0700 Subject: [PATCH 3/9] Drop the counting announcements from the introduction and overview "A ModelCache does three things, and each one causes a problem" counts the items before giving them and tells the reader they are problems. Both are the writer talking about the document rather than writing it. The same sentence appeared again in section 5.1 as "The design has two parts", followed by a list and then a paragraph restating the list. The introduction is now plain declarative prose with no bold scaffolding, which is what the rest of the document uses. Section 5.1 drops the list and says the two changes in the order a reader meets them. The vale ai-tells rules pass this construction, so it is one to catch by reading. I have added it to the skill. Co-Authored-By: Claude Opus 5 Signed-off-by: Dennis Ramdass --- design/modelcache-sources.md | 43 +++++++++++++++++------------------- 1 file changed, 20 insertions(+), 23 deletions(-) diff --git a/design/modelcache-sources.md b/design/modelcache-sources.md index 7e683bc03..d10bb1b0d 100644 --- a/design/modelcache-sources.md +++ b/design/modelcache-sources.md @@ -7,21 +7,20 @@ ## 1. Introduction -A `ModelCache` does three things, and each one causes a problem. +A `ModelCache` reads from HuggingFace and nowhere else. A customer whose models +are in Artifact Registry or ECR must send them to HuggingFace first. A customer +whose weights are on a disk already has no way to say so. -**It reads a model from HuggingFace, and from nowhere else.** A customer who -keeps models in a private registry must send them to HuggingFace first. A -customer whose weights are on a disk already has no way to say so. +What it reads, it stages onto a ReadWriteMany volume on each matched cluster. +A read from that volume can be slower than no cache at all. The volume ignores +the size the user asked for on some clouds, and it fails on its own. Section +4.2 gives the reports. -**It stages what it reads onto a ReadWriteMany volume on each matched -cluster.** A read from that volume can be slower than no cache at all. The -volume does not take the size the user asked for on every cloud, and it can -fail on its own. Section 4.2 gives the reports. - -**It publishes a phase, and a consumer calculates the name of the volume.** The -calculation is correct for one source, and two functions hold a copy of it. -Nothing tells a consumer whether the artifact is readable yet, so an engine -starts before the cache is staged and reads the model a second time. +The cache then publishes a phase, and a consumer calculates the name of the +volume. That calculation is correct for one source, and two functions hold a +copy of it. Nothing tells the consumer whether the artifact is readable, so an +engine starts while the cache is still staging and reads the model a second +time. This document makes two changes. `spec.source` gets two more values, and one of them removes the volume rather than filling it. Each cluster entry gets a @@ -146,17 +145,15 @@ each cluster publishes removes the first and helps with the third. ### 5.1 Overview -The design has two parts. - -1. `spec.source` gets two more values. `OCI` names a reference in a registry. - `Existing` names a claim that the customer filled. -2. Each cluster entry gets a `mount` field. The field holds the volumes, the - volume mounts and the environment variables that a consumer adds to a pod. +`spec.source` gets two more values. `OCI` names a reference in a registry, and +`Existing` names a claim that the customer filled. A user can then name an +artifact where it is. An `OCI` source also takes the volume out of the path, +with the cost and the failure mode that section 4.2 gives. -The first part lets a user name an artifact where it is. For an `OCI` source it -also removes the volume from the path, with the cost and the failure mode that -section 4.2 gives. The second part lets a consumer read the artifact without -knowledge of the source, and tells it when it must wait. +Each cluster entry gets a `mount` field, holding the volumes, the volume mounts +and the environment variables that a consumer adds to a pod. The consumer reads +the artifact without knowledge of the source, and learns from the same field +when it must wait. ### 5.2 The source field From 48ed735bcbefa0a5b5417866fc5641ce7c785d40 Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Mon, 12 Oct 2026 11:38:09 -0700 Subject: [PATCH 4/9] Take the ModelCache sources design to its reviewed state This brings in the wording that came out of review and cites the OCI delivery study the design draws on. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Dennis Ramdass --- design/modelcache-sources.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/design/modelcache-sources.md b/design/modelcache-sources.md index d10bb1b0d..d21f03590 100644 --- a/design/modelcache-sources.md +++ b/design/modelcache-sources.md @@ -211,9 +211,9 @@ mount: env: [] # corev1.EnvVar: name, value ``` -The consumer copies these three lists into the pod. It does not read them. The -function that knows the source writes the contract. The function that makes the -pod does not learn the source. +The consumer copies these three lists into the pod. It does not learn the source +from them. The function that knows the source writes the contract. The function +that makes the pod does not learn the source. ### 5.5 What each cluster reports @@ -277,8 +277,9 @@ twice, and the second read appeared in one engine log and nowhere else. ### 6.1 How long a pod waits for the artifact -This design says that an `OCI` source is faster than a volume. The study in -section 4.2 gives 11.7 seconds for a warm pull and 40.7 minutes over object +This design says that an `OCI` source is faster than a volume. A study of OCI +model delivery ([arXiv:2607.16596](https://arxiv.org/abs/2607.16596)) gives 11.7 +seconds for a warm pull and 40.7 minutes for a re-download over object storage. Modelplane publishes no series that measures either number. Two metrics make the claim readable on a running fleet. @@ -404,7 +405,8 @@ a pod that does not start. CRI-O mounts model artifacts with no driver from v1.33. The function is behind the flag `--oci-artifact-mount-support`, and OpenShift disables it by default. An OpenShift cluster could use one path for both kinds of artifact, but this -design does not do that. A second path costs one more thing for a cluster to report and one more branch +design does not do that. A second path costs one more thing for a cluster to +report and one more branch to test. Modelplane also provisions no OpenShift clusters. The driver stops being necessary everywhere on the day that containerd adds the same function. From fcd05a9f3bc1b4dd53ef5840061e2e9e6b7809a9 Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Mon, 12 Oct 2026 11:43:34 -0700 Subject: [PATCH 5/9] Clear the AI-tells findings in the ModelCache sources design The stricter rules flag counted lead-ins, negated subjects and figurative verbs throughout. Rewrite those sentences in plain terms without changing what the design says. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Dennis Ramdass --- design/modelcache-sources.md | 74 ++++++++++++++++++------------------ 1 file changed, 37 insertions(+), 37 deletions(-) diff --git a/design/modelcache-sources.md b/design/modelcache-sources.md index d21f03590..cdaad795a 100644 --- a/design/modelcache-sources.md +++ b/design/modelcache-sources.md @@ -18,9 +18,9 @@ the size the user asked for on some clouds, and it fails on its own. Section The cache then publishes a phase, and a consumer calculates the name of the volume. That calculation is correct for one source, and two functions hold a -copy of it. Nothing tells the consumer whether the artifact is readable, so an -engine starts while the cache is still staging and reads the model a second -time. +copy of it. An engine can start while the cache is still staging, because the +cache does not report whether the artifact is readable. The engine then reads +the model a second time. This document makes two changes. `spec.source` gets two more values, and one of them removes the volume rather than filling it. Each cluster entry gets a @@ -46,8 +46,8 @@ This document uses these terms. Use each term only as this table tells. | mount | to make an artifact readable in a pod | | resolve | to read the manifest of a reference and find which kind of artifact it is | | collector | the OpenTelemetry collector that Modelplane composes onto each inference cluster | -| cold pull | a read of the artifact onto a node that does not hold it | -| warm pull | a read of the artifact onto a node that holds it already | +| cold pull | a read of the artifact onto a node that does not have it | +| warm pull | a read of the artifact onto a node that has it already | ## 3. Scope @@ -60,9 +60,9 @@ This design must do these four tasks: This design does not do these tasks: -- It does not name a cache by the digest of the artifact. The name stays the +- It does not identify a cache by the digest of the artifact. The name stays the name that the user wrote. -- It does not find two caches that hold the same weights. +- It does not find two caches that contain the same weights. - It does not make a model catalog. - It does not copy an artifact from one cluster to a different cluster. @@ -83,7 +83,7 @@ EFS on EKS. [modelcache.md](./modelcache.md) tells the reader that these volumes can make a cache slower than no cache. -Three reports give the cost of that choice. +These reports give the cost of that choice. [#204](https://github.com/modelplaneai/modelplane/issues/204) measures a slow load from an EFS-backed cache. [#414](https://github.com/modelplaneai/modelplane/issues/414) shows that @@ -108,15 +108,15 @@ other source. ### 4.4 Faults that the current design permits -Three reports from benchmark work show the effect. +Reports from benchmark work show the effect. **The engine does not wait for the artifact.** [#495](https://github.com/modelplaneai/modelplane/issues/495) shows an engine that started while the cache was still staging. The engine downloaded the same -model again, in parallel with the staging. The two downloads of a 464 GB model +model again, in parallel with the staging. Both downloads of the 464 GB model completed in the same second. -Nothing in the code stops this. The function `compose-model-deployment` reads +The code does not stop this. The function `compose-model-deployment` reads `status.cache.storageClassName` from the cluster. This value tells that the cluster has storage. It does not tell that the artifact is on the storage. The cache publishes a phase for each cluster, and no function reads it. @@ -124,7 +124,7 @@ cache publishes a phase for each cluster, and no function reads it. **A volume can stop the cache.** [#494](https://github.com/modelplaneai/modelplane/issues/494) shows a volume that stayed Pending. The cloud replaced the node that the scheduler chose. The -volume kept the name of the node that went away. The cache never staged, and +volume kept the name of the node that the cloud removed. The cache never staged, and the status gave the reason `Hydrating` for 40 minutes. This class of fault is not new. @@ -145,9 +145,9 @@ each cluster publishes removes the first and helps with the third. ### 5.1 Overview -`spec.source` gets two more values. `OCI` names a reference in a registry, and -`Existing` names a claim that the customer filled. A user can then name an -artifact where it is. An `OCI` source also takes the volume out of the path, +`spec.source` gets two more values. `OCI` gives a reference in a registry, and +`Existing` gives a claim that the customer filled. A user can then point +Modelplane at an artifact where it is. An `OCI` source also takes the volume out of the path, with the cost and the failure mode that section 4.2 gives. Each cluster entry gets a `mount` field, holding the volumes, the volume mounts @@ -178,7 +178,7 @@ procedure. Use a digest. A tag is resolved again at each pod start. If a person moves the tag, the subsequent pod serves different weights. -An `OCI` source names no credential on the cache. The pod of the engine spends +An `OCI` source does not put a credential on the cache. The pod of the engine spends the credential, and that pod already takes `imagePullSecrets`. A `HuggingFace` source is different: a Job that Modelplane makes spends that token, and there is no pod of the user to hold it. @@ -186,7 +186,7 @@ is no pod of the user to hold it. The field `sizeGiB` applies to `HuggingFace` only. That source is the only source that puts bytes on a volume that Modelplane makes. -### 5.3 What each source needs from a cluster +### 5.3 Cluster requirements for each source | source | what the user names | what mounts it | what the cluster needs | |---|---|---|---| @@ -215,7 +215,7 @@ The consumer copies these three lists into the pod. It does not learn the source from them. The function that knows the source writes the contract. The function that makes the pod does not learn the source. -### 5.5 What each cluster reports +### 5.5 Status for each cluster A cluster entry gives these answers: @@ -229,16 +229,16 @@ is in the same status for the same reason. The boolean gates one kind of artifact, not the source. A model artifact uses the driver, and the version of Kubernetes does not change this. A container -image on a cluster below 1.36 gives a `Failed` entry that names the version. +image on a cluster below 1.36 gives a `Failed` entry that gives the version. -This distinction is necessary because the clouds do not agree. Modelplane +The version check is necessary because the clouds do not agree. Modelplane defaults to 1.36 for EKS and Vultr. The Regular channel of GKE gives 1.35. AKS and Nebius give 1.34. One cluster that fails does not change a different cluster. Each entry comes from the observed resources of its own cluster. -### 5.6 How a consumer reads the contract +### 5.6 The contract for a consumer The consumer finds the entry for its own cluster. It puts the `env` list first and adds the two other lists. The `env` list is first because `native.py` puts @@ -265,7 +265,7 @@ tells this. ## 6. Telemetry -A `ModelCache` publishes no metric today. The surface in +A `ModelCache` does not publish a metric today. The metric list in [design/telemetry.md](./telemetry.md) names one, `modelplane_replica_cache_hit`, and that metric reports only whether a replica used a cache. @@ -280,16 +280,16 @@ twice, and the second read appeared in one engine log and nowhere else. This design says that an `OCI` source is faster than a volume. A study of OCI model delivery ([arXiv:2607.16596](https://arxiv.org/abs/2607.16596)) gives 11.7 seconds for a warm pull and 40.7 minutes for a re-download over object -storage. Modelplane publishes no series that measures either number. +storage. Modelplane does not publish a series that measures either number. -Two metrics make the claim readable on a running fleet. +These metrics make the claim readable on a running fleet. ``` modelplane_model_pull_seconds histogram modelplane_model_pull_bytes_total counter ``` -Each carries three labels beyond the identity that every Modelplane series +Each has these labels beyond the identity that every Modelplane series has: | label | values | what it separates | @@ -299,7 +299,7 @@ has: | `cached` | `true`, `false` | a warm pull from a cold one | The `cached` label is the one that matters. A warm pull and a cold pull differ -by two orders of magnitude, so one histogram that holds both reports a number +by two orders of magnitude, so one histogram that contains both gives a number that describes neither. The data exists on each node. The kubelet times an image pull, and the driver @@ -310,7 +310,7 @@ this design measurable. ### 6.2 What state each cache is in -Four metrics report the cache itself. +These metrics report the cache itself. ``` modelplane_cache_cluster_ready gauge, one for each cache and cluster @@ -319,7 +319,7 @@ modelplane_cache_staged_bytes gauge modelplane_cache_size_bytes gauge ``` -`modelplane_cache_cluster_ready` carries the phase as a label, so a cache that +`modelplane_cache_cluster_ready` has the phase as a label, so a cache that stays in one phase is visible as a series that does not change. The pair of byte counts gives the progress that [#495](https://github.com/modelplaneai/modelplane/issues/495) had to read from @@ -358,10 +358,10 @@ registry authentication is static configuration. The kubelet authenticates to ECR, Artifact Registry and ACR with the identity of the node. A fleet that publishes container images pays neither cost. -**The driver must outlive the pods that mount through it.** A CSI driver is +**The driver must run as long as the pods that mount through it.** A CSI driver is necessary to unmount a volume, not only to mount one. If the driver goes while -a pod still holds a model artifact, that pod cannot release the volume and -stays in Terminating. +a pod still has a model artifact mounted, that pod cannot release the volume +and stays in the `Terminating` state. [#499](https://github.com/modelplaneai/modelplane/issues/499) is this fault with a different driver. Deleting an `InferenceCluster` removed the NVIDIA DRA @@ -382,22 +382,22 @@ wrote that proposal, and this document replaces it. The contract needs one place for each cluster. The field `status.clusters[]` is that place, and it is there now. A child resource would compose one status -field for the `OCI` and `Existing` sources, and nothing more. The complaint +field for the `OCI` and `Existing` sources, and no other field. The complaint under the proposal is the organization of one function. A refactor corrects -that, and it adds no API to maintain. +that, and it does not add an API to maintain. ### 8.2 Use one `OCI` value, not one value for each kind of artifact A user who writes a reference frequently does not know if the publisher made a container image or a model artifact. Modelplane resolves the reference and -finds this. Two enum values would move that work to the user and would give a +finds this. Separate enum values would move that work to the user and would give a `Failed` entry when the user makes an incorrect choice. ### 8.3 Install the driver on each cluster A composition function sees one cluster. It does not see the caches of the -fleet. It cannot know in advance if a model artifact comes to this cluster. A -DaemonSet that does no work uses few resources, where a missing DaemonSet gives +fleet. It cannot know in advance if a model artifact comes to this cluster. An +idle DaemonSet uses few resources, where a missing DaemonSet gives a pod that does not start. ### 8.4 Use the driver on CRI-O @@ -422,7 +422,7 @@ it needs size limits and eviction first. through a container runtime. A proxy or a private CA needs environment, annotations and volumes on that Job. -**A cluster entry that names the pod fault.** +**A cluster entry that gives the pod fault.** [#498](https://github.com/modelplaneai/modelplane/issues/498) asks for the cause of a fault on the control plane. The contract gives the first part: a cluster says that the artifact is not readable, and why. The exit code of a From bd9ae8961b865496ac53a7e647d57518eafac571 Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Tue, 13 Oct 2026 10:31:31 -0700 Subject: [PATCH 6/9] Move the ModelCache sources design to the house format The doc used numbered sections, a terms table, and scope and limitations sections that the other design docs don't have, and it buried examples and issue reports in long sentences. It now follows the house format, with a summary, background, goals, proposal and alternatives. The volume reports are a table, each source is a table row, and the proposal has whole OCI and Existing examples with the cluster entries they produce. The out-of-scope list is three items, and the prose uses shorter sentences that pass the ai-tells vale rules. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Dennis Ramdass --- design/modelcache-sources.md | 663 ++++++++++++++++------------------- 1 file changed, 298 insertions(+), 365 deletions(-) diff --git a/design/modelcache-sources.md b/design/modelcache-sources.md index cdaad795a..6f4be16d4 100644 --- a/design/modelcache-sources.md +++ b/design/modelcache-sources.md @@ -1,429 +1,362 @@ # ModelCache sources and the mount contract -**Status:** draft -**Date:** 2026-10-05 +**Status:** Draft +**Date:** October 2026 **Author:** Dennis Ramdass -**Supersedes:** [#362](https://github.com/modelplaneai/modelplane/pull/362) +**Issues:** [#512](https://github.com/modelplaneai/modelplane/issues/512), [#495](https://github.com/modelplaneai/modelplane/issues/495), [#210](https://github.com/modelplaneai/modelplane/issues/210) -## 1. Introduction +## Summary -A `ModelCache` reads from HuggingFace and nowhere else. A customer whose models -are in Artifact Registry or ECR must send them to HuggingFace first. A customer -whose weights are on a disk already has no way to say so. +A `ModelCache` reads only from HuggingFace, and it stages what it reads onto a +ReadWriteMany volume on each cluster. That volume can be slower than no cache, +and it can fail on its own. A consumer works out the volume's name itself, and +it doesn't wait for the cache. An engine can start before the weights are +there, and download them a second time. -What it reads, it stages onto a ReadWriteMany volume on each matched cluster. -A read from that volume can be slower than no cache at all. The volume ignores -the size the user asked for on some clouds, and it fails on its own. Section -4.2 gives the reports. +I propose two changes: -The cache then publishes a phase, and a consumer calculates the name of the -volume. That calculation is correct for one source, and two functions hold a -copy of it. An engine can start while the cache is still staging, because the -cache does not report whether the artifact is readable. The engine then reads -the model a second time. +- `spec.source` gets two values. `OCI` reads the model from a registry with no + volume. `Existing` mounts a claim the user already filled. +- Each cluster entry in `status.clusters[]` gets a `mount` field. It says how to + read the model on that cluster. A consumer copies it into the pod, and waits + until it's there. -This document makes two changes. `spec.source` gets two more values, and one of -them removes the volume rather than filling it. Each cluster entry gets a -`mount` field that says how to read the artifact there. A consumer copies that -answer instead of calculating one. Where there is no answer to copy, it waits. +```yaml +apiVersion: modelplane.ai/v1alpha1 +kind: ModelCache +metadata: + name: qwen3-8b + namespace: ml-team +spec: + source: OCI + oci: + ref: us-docker.pkg.dev/ml-team/models/qwen3-8b@sha256: +``` + +## Background + +### One source + +`spec.source` has one value, `HuggingFace`. This forces two kinds of user to +work around Modelplane: -## 2. Terms +- A user whose models are in a registry, like Artifact Registry or ECR, must + copy them to HuggingFace first. +- A user whose weights are already on a volume can't say so. A fleet with no + internet connection has no other option. -This document uses these terms. Use each term only as this table tells. +### The ReadWriteMany volume -| term | meaning in this document | +Every pod of a multi-node gang reads the same weights, so the volume must be +ReadWriteMany. On GKE that's Filestore, and on EKS it's EFS. These reports show +the cost: + +| Issue | What it shows | |---|---| -| artifact | the bytes of one model, in a registry or on a volume | -| cache | one `ModelCache` object | -| source | the value of `spec.source`: `HuggingFace`, `OCI` or `Existing` | -| consumer | a function that adds the artifact to a pod, today `compose-model-replica` | -| cluster entry | one item of `status.clusters[]`, for one inference cluster | -| mount contract | the `mount` field of a cluster entry | -| image volume | a volume that the kubelet makes from a container image | -| model artifact | an OCI artifact that obeys the [model spec](https://modelpack.org/) | -| driver | the [model CSI driver](https://github.com/modelpack/model-csi-driver) | -| stage | to copy an artifact to a volume that Modelplane makes | -| mount | to make an artifact readable in a pod | -| resolve | to read the manifest of a reference and find which kind of artifact it is | -| collector | the OpenTelemetry collector that Modelplane composes onto each inference cluster | -| cold pull | a read of the artifact onto a node that does not have it | -| warm pull | a read of the artifact onto a node that has it already | - -## 3. Scope - -This design must do these four tasks: - -1. Read an artifact from an OCI registry, with no copy to a volume first. -2. Mount a volume that the customer filled. -3. Tell each consumer how to read the artifact on its own cluster. -4. Tell the user why a cluster cannot read the artifact. - -This design does not do these tasks: - -- It does not identify a cache by the digest of the artifact. The name stays the - name that the user wrote. -- It does not find two caches that contain the same weights. -- It does not make a model catalog. -- It does not copy an artifact from one cluster to a different cluster. - -## 4. Background - -### 4.1 The cache reads from one place - -`spec.source` has one value, and that value is `HuggingFace`. A customer whose -models are in Artifact Registry or ECR must send them to HuggingFace to use -Modelplane. A fleet with no connection to the internet has no way to say that -the model is on a disk already. - -### 4.2 The volume is a ReadWriteMany volume - -More than one node of a gang reads the artifact together. Only a ReadWriteMany -volume permits this. The requirement selects Filestore on GKE and -EFS on EKS. -[modelcache.md](./modelcache.md) tells the reader that these volumes can make a -cache slower than no cache. - -These reports give the cost of that choice. -[#204](https://github.com/modelplaneai/modelplane/issues/204) measures a slow -load from an EFS-backed cache. -[#414](https://github.com/modelplaneai/modelplane/issues/414) shows that -`sizeGiB` has no effect below 1 TiB on GKE, because the managed storage class -is Filestore Enterprise. -[#383](https://github.com/modelplaneai/modelplane/issues/383) asks for the -measurement that a ReadWriteMany backend must satisfy. That measurement does -not exist. - -An `OCI` source has none of these properties. The kubelet reads the artifact -from the image store of the node. - -### 4.3 The cache does not tell a consumer how to read the artifact - -A cluster entry has `phase` and `message`. It does not have the name of the -volume. The consumer calculates the name in `cache_pvc_name`. The function -`compose-model-cache` keeps a second copy of the same calculation. A comment -tells the reader to change both. - -This calculation is correct for a `HuggingFace` source. It is correct for no -other source. - -### 4.4 Faults that the current design permits - -Reports from benchmark work show the effect. - -**The engine does not wait for the artifact.** -[#495](https://github.com/modelplaneai/modelplane/issues/495) shows an engine -that started while the cache was still staging. The engine downloaded the same -model again, in parallel with the staging. Both downloads of the 464 GB model -completed in the same second. - -The code does not stop this. The function `compose-model-deployment` reads -`status.cache.storageClassName` from the cluster. This value tells that the -cluster has storage. It does not tell that the artifact is on the storage. The -cache publishes a phase for each cluster, and no function reads it. - -**A volume can stop the cache.** -[#494](https://github.com/modelplaneai/modelplane/issues/494) shows a volume -that stayed Pending. The cloud replaced the node that the scheduler chose. The -volume kept the name of the node that the cloud removed. The cache never staged, and -the status gave the reason `Hydrating` for 40 minutes. - -This class of fault is not new. -[#221](https://github.com/modelplaneai/modelplane/issues/221) is the same shape -on GKE: the cache volume never bound, because the Filestore CSI driver was not -enabled on the cluster. Each source that stages to a volume has this failure -mode. A source that mounts from a registry does not. - -**The user cannot see the cause.** -[#498](https://github.com/modelplaneai/modelplane/issues/498) shows that the -status of a deployment does not tell why a model does not serve. Both faults -above needed a person with access to the workload cluster. - -A source that does not use a volume removes the second fault. A contract that -each cluster publishes removes the first and helps with the third. +| [#204](https://github.com/modelplaneai/modelplane/issues/204) | A slow load from an EFS-backed cache. | +| [#414](https://github.com/modelplaneai/modelplane/issues/414) | `sizeGiB` has no effect below 1 TiB on GKE, because the storage class is Filestore Enterprise. | +| [#383](https://github.com/modelplaneai/modelplane/issues/383) | No measurement says what a ReadWriteMany backend must deliver. | +| [#494](https://github.com/modelplaneai/modelplane/issues/494) | A volume was Pending after the cloud replaced its node. The cache said `Hydrating` for 40 minutes. | +| [#221](https://github.com/modelplaneai/modelplane/issues/221) | A volume never bound, because the cluster didn't enable the Filestore CSI driver. | -## 5. Design +An `OCI` source has none of these problems. The kubelet reads the model from the +node's image store. -### 5.1 Overview +### Consumers -`spec.source` gets two more values. `OCI` gives a reference in a registry, and -`Existing` gives a claim that the customer filled. A user can then point -Modelplane at an artifact where it is. An `OCI` source also takes the volume out of the path, -with the cost and the failure mode that section 4.2 gives. +A cluster entry has a `phase` and a `message`, but not the volume's name. +`compose-model-replica` computes the name in `cache_pvc_name`, and +`compose-model-cache` keeps a second copy of the same code. The name is only +right for a `HuggingFace` source. -Each cluster entry gets a `mount` field, holding the volumes, the volume mounts -and the environment variables that a consumer adds to a pod. The consumer reads -the artifact without knowledge of the source, and learns from the same field -when it must wait. +The consumer also doesn't check whether the model is readable. +`compose-model-deployment` checks that a cluster has a storage class, not that +the cache staged onto it. This causes these faults: -### 5.2 The source field +- [#495](https://github.com/modelplaneai/modelplane/issues/495): an engine + started while its 464 GB cache was staging. It downloaded the model again, in + parallel. +- [#498](https://github.com/modelplaneai/modelplane/issues/498): a deployment's + status doesn't say why a model doesn't serve. The faults above each needed a + person with access to the workload cluster. -```yaml -spec: - source: # enum: HuggingFace, OCI, Existing - huggingFace: {} # no change - oci: - ref: # a tag or a digest - subPath: # the directory that holds the weights - existing: - claimName: # a claim that is bound on each matched cluster - subPath: -``` +## Goals -The kubelet does all the work that an `OCI` source needs. It pulls the -reference when the pod starts. It mounts the filesystem read-only. It gives the -second pod on that node the bytes that the first pod pulled. Modelplane makes -no object for each cluster. The source costs one field, not a second staging -procedure. +**Read the model where it is.** A user can serve a model from an OCI registry or +from a volume they filled. -Use a digest. A tag is resolved again at each pod start. If a person moves the -tag, the subsequent pod serves different weights. +**Don't stage what a node can pull.** An `OCI` source doesn't use a Modelplane +volume. -An `OCI` source does not put a credential on the cache. The pod of the engine spends -the credential, and that pod already takes `imagePullSecrets`. A `HuggingFace` -source is different: a Job that Modelplane makes spends that token, and there -is no pod of the user to hold it. +**Give each cluster one answer.** A cluster entry says how to read the model +there, and the consumer copies that answer instead of computing it. -The field `sizeGiB` applies to `HuggingFace` only. That source is the only -source that puts bytes on a volume that Modelplane makes. +**Wait for the model.** An engine doesn't start before its model is readable. -### 5.3 Cluster requirements for each source +**Say why a cluster can't read the model.** The cluster entry gives the reason. -| source | what the user names | what mounts it | what the cluster needs | -|---|---|---|---| -| `HuggingFace` | a repository ID | a volume that Modelplane stages | a ReadWriteMany storage class | -| `OCI` | a container image that holds the weights | the kubelet, as an image volume | Kubernetes 1.36 | -| `OCI` | a model artifact | the driver | Kubernetes 1.25 | -| `Existing` | a claim that the user filled | the kubelet, as a claim | the claim bound on the cluster | +This design doesn't cover: -Modelplane resolves the reference and finds which kind of artifact it has. The -user names a tag and does no more. An artifact of a third kind gives a `Failed` -cluster entry with the content of the manifest in the message. +- Identifying a cache by the model's digest. The cache's name stays the name the + user wrote. +- Finding two caches with the same model. +- Copying a model from one cluster to another. -### 5.4 The mount contract +## Proposal -Each cluster entry gets this field. Each part is a subset of the Kubernetes -type of the same name. +### Sources ```yaml -mount: - volumes: [] # corev1.Volume: persistentVolumeClaim, image or csi - volumeMounts: [] # corev1.VolumeMount: name, mountPath, subPath, readOnly - env: [] # corev1.EnvVar: name, value +spec: + source: OCI # HuggingFace, OCI, or Existing + huggingFace: {} # unchanged + oci: + ref: "" # a tag or a digest + subPath: "" # the directory that holds the weights + existing: + claimName: "" # a claim bound on each matched cluster + subPath: "" ``` -The consumer copies these three lists into the pod. It does not learn the source -from them. The function that knows the source writes the contract. The function -that makes the pod does not learn the source. - -### 5.5 Status for each cluster - -A cluster entry gives these answers: +| Source | The user gives | What mounts it | What the cluster needs | +|---|---|---|---| +| `HuggingFace` | A repository ID | A volume that Modelplane stages | A ReadWriteMany storage class | +| `OCI` | A container image that holds the weights | The kubelet, as an image volume | Kubernetes 1.36 | +| `OCI` | A [model artifact](https://modelpack.org/) | The [model CSI driver](https://github.com/modelpack/model-csi-driver) | Kubernetes 1.25 | +| `Existing` | A claim the user filled | The kubelet, as a claim | The claim, bound on the cluster | -- if the artifact is readable on this cluster -- how to read it -- why it is not readable, when it is not +For an `OCI` source: -`InferenceCluster.status.cache` gets a boolean `imageVolumes`. It tells if the -cluster can mount an image volume. The field follows `storageClassName`, which -is in the same status for the same reason. +- Modelplane reads the reference's manifest to find which kind of artifact it + is. The user doesn't say. Any other kind of artifact fails the cluster entry, + and the message gives the manifest's media type. +- The kubelet pulls the reference when the pod starts, and shares the pulled + bytes with later pods on the node. Modelplane composes nothing on the cluster + for the cache. +- The user should use a digest. The kubelet resolves a tag at each pod start, so + a moved tag serves different weights. +- The cache has no credential. The engine pod pulls the artifact, and it already + takes `imagePullSecrets`. -The boolean gates one kind of artifact, not the source. A model artifact uses -the driver, and the version of Kubernetes does not change this. A container -image on a cluster below 1.36 gives a `Failed` entry that gives the version. +`sizeGiB` applies only to `HuggingFace`, the one source that stages onto a +Modelplane volume. -The version check is necessary because the clouds do not agree. Modelplane -defaults to 1.36 for EKS and Vultr. The Regular channel of GKE gives 1.35. AKS -and Nebius give 1.34. +### The mount contract -One cluster that fails does not change a different cluster. Each entry comes -from the observed resources of its own cluster. +Each cluster entry gets a `mount` field. Each list is a subset of the +Kubernetes type with the same name: -### 5.6 The contract for a consumer +```yaml +mount: + volumes: [] # corev1.Volume: persistentVolumeClaim, image, or csi + volumeMounts: [] # corev1.VolumeMount: name, mountPath, subPath, readOnly + env: [] # corev1.EnvVar: name, value +``` -The consumer finds the entry for its own cluster. It puts the `env` list first -and adds the two other lists. The `env` list is first because `native.py` puts -the entries of Modelplane first. This order lets a variable of the user be the -last word. +`compose-model-cache` knows the source, so it writes the contract. +`compose-model-replica` builds the pod, and it doesn't need to know the source. +It follows these rules: -The consumer acts on one field rather than copies it. Where a volume mount is -read-only, the consumer adds an `emptyDir` and points `HOME` at it. Engines -write tokenizer files, compile files and lock files beside the weights. `HOME` -sends these files to the `emptyDir`. The `emptyDir` has a size limit, because a -compile cache can be some gigabytes, and a volume with no limit removes the -pod. +- It finds the entry for its own cluster. +- It adds `env` before the user's environment. A user's variable with the same + name then takes effect. It adds `volumes` and `volumeMounts` as they are. +- If a mount is read-only, it adds an `emptyDir` with a size limit and points + `HOME` at it. Engines write tokenizer, compile, and lock files next to the + weights. Without a size limit, a large compile cache can get the pod evicted. +- If the cluster has no entry, or the entry has no `mount`, it sets a condition + and composes no workload. This fixes #495. -**The consumer stops when the entry is not ready.** This is the correction for -[#495](https://github.com/modelplaneai/modelplane/issues/495). If the cluster -has no entry, or the entry has no `mount`, the consumer writes a condition and -composes no workload. The engine does not start before the artifact is -readable. +### Cluster status -[#186](https://github.com/modelplaneai/modelplane/issues/186) made -`compose-model-deployment` intersect the two cluster selectors. Labels cannot -tell that a cluster inside that set has not completed its staging. The contract -tells this. +`InferenceCluster.status.cache` gets an `imageVolumes` boolean, next to +`storageClassName`. It says whether the cluster can mount an image volume. -## 6. Telemetry +It only matters for container images. A model artifact uses the CSI driver on +any version. A container image on a cluster below 1.36 fails the cluster entry, +and the message gives the version. Clouds disagree on the version Modelplane +provisions today: -A `ModelCache` does not publish a metric today. The metric list in -[design/telemetry.md](./telemetry.md) names one, `modelplane_replica_cache_hit`, -and that metric reports only whether a replica used a cache. +| Cloud | Kubernetes version | +|---|---| +| EKS, Vultr | 1.36 | +| GKE (Regular channel) | 1.35 | +| AKS, Nebius | 1.34 | -The faults in section 4.4 show the cost. The cache in -[#494](https://github.com/modelplaneai/modelplane/issues/494) reported -`Hydrating` for 40 minutes, and no series recorded it. The engine in -[#495](https://github.com/modelplaneai/modelplane/issues/495) read the model -twice, and the second read appeared in one engine log and nowhere else. +Each cluster entry comes from that cluster's own observed resources. A failed +cluster doesn't change another cluster's entry. -### 6.1 How long a pod waits for the artifact +### Metrics -This design says that an `OCI` source is faster than a volume. A study of OCI -model delivery ([arXiv:2607.16596](https://arxiv.org/abs/2607.16596)) gives 11.7 -seconds for a warm pull and 40.7 minutes for a re-download over object -storage. Modelplane does not publish a series that measures either number. +A `ModelCache` doesn't publish metrics today. In #494 the cache said `Hydrating` +for 40 minutes, and no series recorded it. In #495 the second download showed +up in one engine log and nowhere else. -These metrics make the claim readable on a running fleet. +**Pull time.** A [study of OCI model +delivery](https://arxiv.org/abs/2607.16596) measured 11.7 seconds for a warm +pull and 40.7 minutes for a re-download from object storage. These metrics +measure the same thing on a running fleet: ``` modelplane_model_pull_seconds histogram modelplane_model_pull_bytes_total counter ``` -Each has these labels beyond the identity that every Modelplane series -has: - -| label | values | what it separates | +| Label | Values | What it separates | |---|---|---| -| `source` | `HuggingFace`, `OCI`, `Existing` | one source from another | -| `mechanism` | `image`, `csi`, `volume` | the kubelet from the driver | -| `cached` | `true`, `false` | a warm pull from a cold one | - -The `cached` label is the one that matters. A warm pull and a cold pull differ -by two orders of magnitude, so one histogram that contains both gives a number -that describes neither. +| `source` | `HuggingFace`, `OCI`, `Existing` | One source from another | +| `mechanism` | `image`, `csi`, `volume` | The kubelet from the CSI driver | +| `cached` | `true`, `false` | A warm pull from a cold one | -The data exists on each node. The kubelet times an image pull, and the driver -times a model artifact. Neither is a pod, so the collector cannot read them -with the scrape jobs that it has: each job uses `role: pod`. A node-level -target is new work in `compose-serving-stack`, and it is the work that makes -this design measurable. +`cached` matters most. A warm and a cold pull differ by two orders of +magnitude, so one histogram that mixes them describes neither. -### 6.2 What state each cache is in +The kubelet measures how long an image pull takes, and the CSI driver measures +an artifact pull. Neither +is a pod, and every scrape job in the collector uses `role: pod`. These metrics +need a node-level scrape target in `compose-serving-stack`. -These metrics report the cache itself. +**Cache state.** These metrics report each cache: ``` modelplane_cache_cluster_ready gauge, one for each cache and cluster -modelplane_cache_ready_seconds histogram, created until readable +modelplane_cache_ready_seconds histogram, from creation until readable modelplane_cache_staged_bytes gauge modelplane_cache_size_bytes gauge ``` -`modelplane_cache_cluster_ready` has the phase as a label, so a cache that -stays in one phase is visible as a series that does not change. The pair of -byte counts gives the progress that -[#495](https://github.com/modelplaneai/modelplane/issues/495) had to read from -a file listing. - -**These four are blocked by -[#476](https://github.com/modelplaneai/modelplane/issues/476).** Each one reads -`status.clusters[]`, which lives on the control plane, and Modelplane composes -a collector onto each inference cluster only. Nothing on the control plane -collects or exports. #476 holds that work, and these metrics are part of it. - -The contract in section 5.4 makes them possible. Before it, the state of a -cache on one cluster was a phase and a message. After it, each entry says -whether the artifact is readable and why it is not, which is what a metric -needs to report. - -## 7. Limitations - -**A cache reports what it can examine.** Resolution of the reference finds a -repository that is not there, a credential that does not work, and an artifact -of an unknown kind. Resolution does not find that the artifact holds a -different model. It does not find that an `Existing` volume is not complete. -Both of these give a Ready entry and fail when the engine starts. - -**An `OCI` mount uses the disk of the node.** The driver keeps artifacts under -`/var/lib/model-csi`. An image volume goes to the image store, where the -eviction limits of `imagefs` apply to it. The field `sizeGiB` does not control -either one. A model larger than the disk of the node fails at pod start. Make -the disk of the pool large enough for the largest model. - -**The driver is one more component.** A DaemonSet in the path of a pull is a -thing to install and to upgrade. The project is young, and it had no commit -after March 2026. Modelplane installs it on each cluster, because a composition -function sees one cluster and cannot know which artifacts come to it. Its -registry authentication is static configuration. The kubelet authenticates to -ECR, Artifact Registry and ACR with the identity of the node. A fleet that -publishes container images pays neither cost. - -**The driver must run as long as the pods that mount through it.** A CSI driver is -necessary to unmount a volume, not only to mount one. If the driver goes while -a pod still has a model artifact mounted, that pod cannot release the volume -and stays in the `Terminating` state. - -[#499](https://github.com/modelplaneai/modelplane/issues/499) is this fault -with a different driver. Deleting an `InferenceCluster` removed the NVIDIA DRA -driver before the engine pods finished, so the pods could not release their -GPUs. The cluster stayed in Deleting for 35 minutes with an idle 8-GPU node. -Modelplane composes no ordering between the serving stack and the workloads of -a `ModelReplica`, so this design adds a second component with the same -exposure. The correction belongs to #499, and this design does not add one. - -## 8. Decisions - -### 8.1 Publish a contract, do not add a child resource - -[#210](https://github.com/modelplaneai/modelplane/issues/210) asks for a -`ModelCacheHydration` child for each cluster, and -[#362](https://github.com/modelplaneai/modelplane/pull/362) proposes one. I -wrote that proposal, and this document replaces it. - -The contract needs one place for each cluster. The field `status.clusters[]` is -that place, and it is there now. A child resource would compose one status -field for the `OCI` and `Existing` sources, and no other field. The complaint -under the proposal is the organization of one function. A refactor corrects -that, and it does not add an API to maintain. - -### 8.2 Use one `OCI` value, not one value for each kind of artifact - -A user who writes a reference frequently does not know if the publisher made a -container image or a model artifact. Modelplane resolves the reference and -finds this. Separate enum values would move that work to the user and would give a -`Failed` entry when the user makes an incorrect choice. - -### 8.3 Install the driver on each cluster - -A composition function sees one cluster. It does not see the caches of the -fleet. It cannot know in advance if a model artifact comes to this cluster. An -idle DaemonSet uses few resources, where a missing DaemonSet gives -a pod that does not start. - -### 8.4 Use the driver on CRI-O - -CRI-O mounts model artifacts with no driver from v1.33. The function is behind -the flag `--oci-artifact-mount-support`, and OpenShift disables it by default. -An OpenShift cluster could use one path for both kinds of artifact, but this -design does not do that. A second path costs one more thing for a cluster to -report and one more branch -to test. Modelplane also provisions no OpenShift clusters. The driver -stops being necessary everywhere on the day that containerd adds the same -function. - -## 9. Subsequent work - -**Materialization on the node.** An init container that copies the artifact to -the node, with a `hostPath` volume. This removes the limit that NFS puts on a -`HuggingFace` read. It also puts each model on each node that can serve it, so -it needs size limits and eviction first. - -**Acceleration of the HuggingFace fetch.** The Job downloads with HTTPS, not -through a container runtime. A proxy or a private CA needs environment, -annotations and volumes on that Job. - -**A cluster entry that gives the pod fault.** -[#498](https://github.com/modelplaneai/modelplane/issues/498) asks for the -cause of a fault on the control plane. The contract gives the first part: a -cluster says that the artifact is not readable, and why. The exit code of a -container needs a different source of data. +`modelplane_cache_cluster_ready` has the phase as a label, so a cache stuck in +one phase shows up as a flat series. The staged and total byte gauges give the +staging progress that #495 had to read from a file listing. + +They read `status.clusters[]`, which is on the control plane. Modelplane +composes collectors only on inference clusters, so these metrics wait on #476. + +### Examples + +#### OCI + +The `qwen3-8b` cache from the summary gives this cluster entry on a 1.36 +cluster: + +```yaml +status: + clusters: + - name: gke-us-central1 + phase: Ready + mount: + volumes: + - name: model-cache + image: + reference: us-docker.pkg.dev/ml-team/models/qwen3-8b@sha256: + pullPolicy: IfNotPresent + volumeMounts: + - name: model-cache + mountPath: /mnt/models + readOnly: true + env: [] +``` + +The mount is read-only. The consumer also adds an `emptyDir` for `HOME`. + +#### Existing + +```yaml +apiVersion: modelplane.ai/v1alpha1 +kind: ModelCache +metadata: + name: kimi-k2 + namespace: ml-team +spec: + source: Existing + existing: + claimName: kimi-k2-weights + subPath: Kimi-K2-Instruct +``` + +```yaml +status: + clusters: + - name: nebius-eu-north1 + phase: Ready + mount: + volumes: + - name: model-cache + persistentVolumeClaim: + claimName: kimi-k2-weights + volumeMounts: + - name: model-cache + mountPath: /mnt/models + subPath: Kimi-K2-Instruct + env: [] + - name: gke-us-central1 + phase: Failed + message: persistentvolumeclaim "kimi-k2-weights" not found +``` + +The second cluster has no `mount`, so no replica of a deployment that uses this +cache starts there. + +### Limits + +- **A Ready cache can still hold the wrong model.** Resolving the reference + finds a missing repository, a bad credential, or an unknown artifact kind. It + can't tell that the artifact is a different model, or that an `Existing` + volume is incomplete. Those fail when the engine starts. +- **An `OCI` source uses the node's disk.** The CSI driver keeps artifacts under + `/var/lib/model-csi`. Image volumes go to the image store, under the kubelet's + `imagefs` eviction limits. `sizeGiB` controls neither. A pool's disk must fit + its largest model. +- **The CSI driver is another component.** It's a DaemonSet to install and + upgrade on every cluster, and its registry credentials are static. The kubelet + uses the node's identity for ECR, Artifact Registry, and ACR. Container images + avoid both costs. +- **The CSI driver must keep running until its pods stop.** A pod needs the + driver to unmount a model artifact. If the driver goes first, the pod stays + `Terminating`. #499 is the same fault with the NVIDIA DRA driver, and its fix + covers both. + +## Future improvements + +**Copy to the node.** An init container could copy the model to the node's disk +through a `hostPath` volume. That avoids the NFS read for `HuggingFace`, but it +puts each model on each node, so it needs size limits and eviction first. + +**Faster HuggingFace downloads.** The staging Job downloads over HTTPS, not +through the container runtime. A proxy or a private CA needs environment, +annotations, and volumes on that Job. + +**Pod faults in the cluster entry.** #498 asks the control plane to show why a +model doesn't serve. The contract covers the model not being readable. A +container's exit code needs another data source. + +## Alternatives considered + +### A child resource for each cluster + +#210 asks for a `ModelCacheHydration` child for each cluster, and #362 +proposed one. The contract needs one place for each cluster, and +`status.clusters[]` already is that place. A child resource would contain one +status field for `OCI` and `Existing` and nothing else. The real complaint +behind #210 is how one function is organized, and a refactor fixes that +without a new API. + +### One enum value for each kind of artifact + +`OCIImage` and `OCIArtifact` would make the user say which kind the publisher +made. Users often don't know. Modelplane can read the manifest and find out, +and a wrong guess would only fail the cluster entry. + +### Install the CSI driver only where it's needed + +A composition function sees one cluster, not the fleet's caches, so it can't +know whether a model artifact will come to that cluster. An idle DaemonSet uses +few resources. A missing one means a pod that never starts. Modelplane installs +it on every cluster. + +### CRI-O's built-in artifact mounts + +CRI-O mounts model artifacts with no driver from v1.33, behind +`--oci-artifact-mount-support`, which OpenShift disables by default. A second +mount path would add another thing for a cluster to report and another branch +to test, and Modelplane provisions no OpenShift clusters. The driver stops +being needed once containerd adds the same feature. From de8b62526d138ccec5a7371258eb7178a116ba42 Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Tue, 13 Oct 2026 10:53:23 -0700 Subject: [PATCH 7/9] Settle the credential and wait rules in the ModelCache sources design The design said an OCI cache has no credential, yet Modelplane reads the reference's manifest, which a private registry won't serve without one. It also promised an engine never starts before its model is readable, which a Ready cache holding the wrong model breaks, and it gave metrics a full section that no goal asked for. oci.imagePullSecrets now names the secrets Modelplane reads the manifest with, and mount lists them for the engine pod. A tag resolves to the digest written into mount. The wait goal is that no engine pod is composed until the cluster entry has a mount. Metrics move to future improvements, the CSI driver install moves into the proposal, and the container image and model artifact forms are explained up front. Claim replaces volume where a PersistentVolumeClaim is meant. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Dennis Ramdass --- design/modelcache-sources.md | 219 ++++++++++++++--------------------- 1 file changed, 90 insertions(+), 129 deletions(-) diff --git a/design/modelcache-sources.md b/design/modelcache-sources.md index 6f4be16d4..4ca95bb4c 100644 --- a/design/modelcache-sources.md +++ b/design/modelcache-sources.md @@ -8,18 +8,19 @@ ## Summary A `ModelCache` reads only from HuggingFace, and it stages what it reads onto a -ReadWriteMany volume on each cluster. That volume can be slower than no cache, -and it can fail on its own. A consumer works out the volume's name itself, and -it doesn't wait for the cache. An engine can start before the weights are +ReadWriteMany claim on each cluster. That claim can be slower than no cache, and +it can fail on its own. `compose-model-replica`, the function that mounts the +cache into engine pods, works out the claim's name itself, and it doesn't wait +for the cache. An engine can start before the weights are there, and download them a second time. I propose two changes: -- `spec.source` gets two values. `OCI` reads the model from a registry with no - volume. `Existing` mounts a claim the user already filled. +- `spec.source` gets two new values. `OCI` reads the model from a registry with + no claim. `Existing` mounts a claim the user already filled. - Each cluster entry in `status.clusters[]` gets a `mount` field. It says how to - read the model on that cluster. A consumer copies it into the pod, and waits - until it's there. + read the model on that cluster. `compose-model-replica` copies it into the + pod, and waits until it's there. ```yaml apiVersion: modelplane.ai/v1alpha1 @@ -42,12 +43,19 @@ work around Modelplane: - A user whose models are in a registry, like Artifact Registry or ECR, must copy them to HuggingFace first. -- A user whose weights are already on a volume can't say so. A fleet with no +- A user whose weights are already on a claim can't say so. A fleet with no internet connection has no other option. -### The ReadWriteMany volume +A registry can hold weights in two forms: -Every pod of a multi-node gang reads the same weights, so the volume must be +- A **container image**. The kubelet can mount it as an image volume. +- A **model artifact**, in the [ModelPack](https://modelpack.org/) format. The + kubelet can't mount it, so the + [model CSI driver](https://github.com/modelpack/model-csi-driver) mounts it. + +### The ReadWriteMany claim + +Every pod of a multi-node gang reads the same weights, so the claim must be ReadWriteMany. On GKE that's Filestore, and on EKS it's EFS. These reports show the cost: @@ -56,20 +64,20 @@ the cost: | [#204](https://github.com/modelplaneai/modelplane/issues/204) | A slow load from an EFS-backed cache. | | [#414](https://github.com/modelplaneai/modelplane/issues/414) | `sizeGiB` has no effect below 1 TiB on GKE, because the storage class is Filestore Enterprise. | | [#383](https://github.com/modelplaneai/modelplane/issues/383) | No measurement says what a ReadWriteMany backend must deliver. | -| [#494](https://github.com/modelplaneai/modelplane/issues/494) | A volume was Pending after the cloud replaced its node. The cache said `Hydrating` for 40 minutes. | -| [#221](https://github.com/modelplaneai/modelplane/issues/221) | A volume never bound, because the cluster didn't enable the Filestore CSI driver. | +| [#494](https://github.com/modelplaneai/modelplane/issues/494) | A claim was Pending after the cloud replaced its node. The cache said `Hydrating` for 40 minutes. | +| [#221](https://github.com/modelplaneai/modelplane/issues/221) | A claim never bound, because the cluster didn't enable the Filestore CSI driver. | -An `OCI` source has none of these problems. The kubelet reads the model from the -node's image store. +An `OCI` source doesn't use a ReadWriteMany claim. The node pulls the model to +its own disk. -### Consumers +### Mounting the cache -A cluster entry has a `phase` and a `message`, but not the volume's name. +A cluster entry has a `phase` and a `message`, but not the claim's name. `compose-model-replica` computes the name in `cache_pvc_name`, and `compose-model-cache` keeps a second copy of the same code. The name is only right for a `HuggingFace` source. -The consumer also doesn't check whether the model is readable. +No function checks whether the model is readable. `compose-model-deployment` checks that a cluster has a storage class, not that the cache staged onto it. This causes these faults: @@ -83,22 +91,21 @@ the cache staged onto it. This causes these faults: ## Goals **Read the model where it is.** A user can serve a model from an OCI registry or -from a volume they filled. +from a claim they filled. **Don't stage what a node can pull.** An `OCI` source doesn't use a Modelplane -volume. +claim. **Give each cluster one answer.** A cluster entry says how to read the model -there, and the consumer copies that answer instead of computing it. +there, and `compose-model-replica` copies that answer instead of computing it. -**Wait for the model.** An engine doesn't start before its model is readable. +**Wait for the model.** Modelplane composes no engine pod on a cluster until +that cluster's entry has a `mount`. **Say why a cluster can't read the model.** The cluster entry gives the reason. This design doesn't cover: -- Identifying a cache by the model's digest. The cache's name stays the name the - user wrote. - Finding two caches with the same model. - Copying a model from one cluster to another. @@ -113,6 +120,7 @@ spec: oci: ref: "" # a tag or a digest subPath: "" # the directory that holds the weights + imagePullSecrets: [] # secrets on each cluster, for a private registry existing: claimName: "" # a claim bound on each matched cluster subPath: "" @@ -120,44 +128,56 @@ spec: | Source | The user gives | What mounts it | What the cluster needs | |---|---|---|---| -| `HuggingFace` | A repository ID | A volume that Modelplane stages | A ReadWriteMany storage class | -| `OCI` | A container image that holds the weights | The kubelet, as an image volume | Kubernetes 1.36 | -| `OCI` | A [model artifact](https://modelpack.org/) | The [model CSI driver](https://github.com/modelpack/model-csi-driver) | Kubernetes 1.25 | -| `Existing` | A claim the user filled | The kubelet, as a claim | The claim, bound on the cluster | +| `HuggingFace` | A repository ID | A claim that Modelplane stages | A ReadWriteMany storage class | +| `OCI` | A container image that holds the weights | An `image` volume | Kubernetes 1.36 | +| `OCI` | A model artifact | A `csi` volume from the model CSI driver | The driver, which Modelplane installs on every cluster | +| `Existing` | A claim the user filled | A `persistentVolumeClaim` volume | The claim, bound on the cluster | For an `OCI` source: -- Modelplane reads the reference's manifest to find which kind of artifact it - is. The user doesn't say. Any other kind of artifact fails the cluster entry, - and the message gives the manifest's media type. +- Modelplane reads the reference's manifest from each cluster to find which kind + of artifact it is. The user doesn't say. Any other kind of artifact fails the + cluster entry, and the message gives the manifest's media type. +- Modelplane resolves a tag to its digest and writes the digest into `mount`. A + tag that moves later doesn't change what a running deployment serves. +- A private registry needs `oci.imagePullSecrets`. Modelplane reads the manifest + with them, and `mount` lists them for the engine pod's pull. - The kubelet pulls the reference when the pod starts, and shares the pulled - bytes with later pods on the node. Modelplane composes nothing on the cluster - for the cache. -- The user should use a digest. The kubelet resolves a tag at each pod start, so - a moved tag serves different weights. -- The cache has no credential. The engine pod pulls the artifact, and it already - takes `imagePullSecrets`. + bytes with later pods on the node. `sizeGiB` applies only to `HuggingFace`, the one source that stages onto a -Modelplane volume. +Modelplane claim. + +Modelplane installs the model CSI driver on every cluster. A composition +function sees one cluster, not the fleet's caches, so it can't know whether a +model artifact will come to that cluster. ### The mount contract -Each cluster entry gets a `mount` field. Each list is a subset of the -Kubernetes type with the same name: +Each cluster entry gets a `mount` field. Each item uses some of the fields of +the Kubernetes type in its comment: ```yaml mount: - volumes: [] # corev1.Volume: persistentVolumeClaim, image, or csi - volumeMounts: [] # corev1.VolumeMount: name, mountPath, subPath, readOnly - env: [] # corev1.EnvVar: name, value + volumes: [] # corev1.Volume: persistentVolumeClaim, image, or csi + volumeMounts: [] # corev1.VolumeMount: name, mountPath, subPath, readOnly + env: [] # corev1.EnvVar: name, value + imagePullSecrets: [] # corev1.LocalObjectReference ``` -`compose-model-cache` knows the source, so it writes the contract. +The model is always at `/mnt/models`. `env` is empty for `OCI` and `Existing`. A +`HuggingFace` entry sets `HF_HUB_CACHE`. The engine then finds the staged +snapshot by repository ID. + +`compose-model-cache` knows the source, so it writes the contract. Each cluster +entry comes from that cluster's own observed resources, so a failed cluster +doesn't change another cluster's entry. + `compose-model-replica` builds the pod, and it doesn't need to know the source. It follows these rules: - It finds the entry for its own cluster. +- It adds `imagePullSecrets` to the pod. - It adds `env` before the user's environment. A user's variable with the same name then takes effect. It adds `volumes` and `volumeMounts` as they are. - If a mount is read-only, it adds an `emptyDir` with a size limit and points @@ -169,67 +189,13 @@ It follows these rules: ### Cluster status `InferenceCluster.status.cache` gets an `imageVolumes` boolean, next to -`storageClassName`. It says whether the cluster can mount an image volume. +`storageClassName`. Modelplane sets it from the cluster's Kubernetes version. It +says whether the cluster can mount an image volume. It only matters for container images. A model artifact uses the CSI driver on any version. A container image on a cluster below 1.36 fails the cluster entry, -and the message gives the version. Clouds disagree on the version Modelplane -provisions today: - -| Cloud | Kubernetes version | -|---|---| -| EKS, Vultr | 1.36 | -| GKE (Regular channel) | 1.35 | -| AKS, Nebius | 1.34 | - -Each cluster entry comes from that cluster's own observed resources. A failed -cluster doesn't change another cluster's entry. - -### Metrics - -A `ModelCache` doesn't publish metrics today. In #494 the cache said `Hydrating` -for 40 minutes, and no series recorded it. In #495 the second download showed -up in one engine log and nowhere else. - -**Pull time.** A [study of OCI model -delivery](https://arxiv.org/abs/2607.16596) measured 11.7 seconds for a warm -pull and 40.7 minutes for a re-download from object storage. These metrics -measure the same thing on a running fleet: - -``` -modelplane_model_pull_seconds histogram -modelplane_model_pull_bytes_total counter -``` - -| Label | Values | What it separates | -|---|---|---| -| `source` | `HuggingFace`, `OCI`, `Existing` | One source from another | -| `mechanism` | `image`, `csi`, `volume` | The kubelet from the CSI driver | -| `cached` | `true`, `false` | A warm pull from a cold one | - -`cached` matters most. A warm and a cold pull differ by two orders of -magnitude, so one histogram that mixes them describes neither. - -The kubelet measures how long an image pull takes, and the CSI driver measures -an artifact pull. Neither -is a pod, and every scrape job in the collector uses `role: pod`. These metrics -need a node-level scrape target in `compose-serving-stack`. - -**Cache state.** These metrics report each cache: - -``` -modelplane_cache_cluster_ready gauge, one for each cache and cluster -modelplane_cache_ready_seconds histogram, from creation until readable -modelplane_cache_staged_bytes gauge -modelplane_cache_size_bytes gauge -``` - -`modelplane_cache_cluster_ready` has the phase as a label, so a cache stuck in -one phase shows up as a flat series. The staged and total byte gauges give the -staging progress that #495 had to read from a file listing. - -They read `status.clusters[]`, which is on the control plane. Modelplane -composes collectors only on inference clusters, so these metrics wait on #476. +and the message gives the version. Of the clouds Modelplane provisions, only EKS +and Vultr run 1.36 today. ### Examples @@ -253,11 +219,8 @@ status: - name: model-cache mountPath: /mnt/models readOnly: true - env: [] ``` -The mount is read-only. The consumer also adds an `emptyDir` for `HOME`. - #### Existing ```yaml @@ -287,21 +250,19 @@ status: - name: model-cache mountPath: /mnt/models subPath: Kimi-K2-Instruct - env: [] - name: gke-us-central1 phase: Failed message: persistentvolumeclaim "kimi-k2-weights" not found ``` -The second cluster has no `mount`, so no replica of a deployment that uses this -cache starts there. +The second cluster has no `mount`. ### Limits - **A Ready cache can still hold the wrong model.** Resolving the reference - finds a missing repository, a bad credential, or an unknown artifact kind. It + finds a missing repository, a bad pull secret, or an unknown artifact kind. It can't tell that the artifact is a different model, or that an `Existing` - volume is incomplete. Those fail when the engine starts. + claim is incomplete. Those fail when the engine starts. - **An `OCI` source uses the node's disk.** The CSI driver keeps artifacts under `/var/lib/model-csi`. Image volumes go to the image store, under the kubelet's `imagefs` eviction limits. `sizeGiB` controls neither. A pool's disk must fit @@ -310,21 +271,26 @@ cache starts there. upgrade on every cluster, and its registry credentials are static. The kubelet uses the node's identity for ECR, Artifact Registry, and ACR. Container images avoid both costs. -- **The CSI driver must keep running until its pods stop.** A pod needs the - driver to unmount a model artifact. If the driver goes first, the pod stays - `Terminating`. #499 is the same fault with the NVIDIA DRA driver, and its fix - covers both. +- **The CSI driver must stop after its pods.** Otherwise a pod stays + `Terminating`, as in #499. ## Future improvements **Copy to the node.** An init container could copy the model to the node's disk -through a `hostPath` volume. That avoids the NFS read for `HuggingFace`, but it +through a `hostPath` volume. That avoids the ReadWriteMany read for +`HuggingFace`, but it puts each model on each node, so it needs size limits and eviction first. -**Faster HuggingFace downloads.** The staging Job downloads over HTTPS, not +**HuggingFace through a proxy.** The staging Job downloads over HTTPS, not through the container runtime. A proxy or a private CA needs environment, annotations, and volumes on that Job. +**Cache metrics.** A `ModelCache` doesn't publish metrics. In #494 the cache +said `Hydrating` for 40 minutes, and no series recorded it. Pull time, split by +source and by warm or cold pull, and each cluster entry's phase would show both. +Pull time needs a node-level scrape, because the kubelet and the CSI driver +measure it. Cache state is on the control plane, so it waits on #476. + **Pod faults in the cluster entry.** #498 asks the control plane to show why a model doesn't serve. The contract covers the model not being readable. A container's exit code needs another data source. @@ -333,12 +299,9 @@ container's exit code needs another data source. ### A child resource for each cluster -#210 asks for a `ModelCacheHydration` child for each cluster, and #362 -proposed one. The contract needs one place for each cluster, and -`status.clusters[]` already is that place. A child resource would contain one -status field for `OCI` and `Existing` and nothing else. The real complaint -behind #210 is how one function is organized, and a refactor fixes that -without a new API. +#210 asks for a `ModelCacheHydration` child for each cluster. The contract +needs one place for each cluster, and `status.clusters[]` already gives it. A +child resource would contain only one status field for `OCI` and `Existing`. ### One enum value for each kind of artifact @@ -348,15 +311,13 @@ and a wrong guess would only fail the cluster entry. ### Install the CSI driver only where it's needed -A composition function sees one cluster, not the fleet's caches, so it can't -know whether a model artifact will come to that cluster. An idle DaemonSet uses -few resources. A missing one means a pod that never starts. Modelplane installs -it on every cluster. +Installing the driver only on clusters that serve a model artifact would save an +idle DaemonSet. But a composition function can't see which clusters those are, +and a missing driver means a pod that never starts. ### CRI-O's built-in artifact mounts -CRI-O mounts model artifacts with no driver from v1.33, behind -`--oci-artifact-mount-support`, which OpenShift disables by default. A second -mount path would add another thing for a cluster to report and another branch -to test, and Modelplane provisions no OpenShift clusters. The driver stops +CRI-O can mount model artifacts with no driver from v1.33, behind a flag. +Modelplane provisions no CRI-O clusters, and a second mount path adds another +branch to test. The driver stops being needed once containerd adds the same feature. From c53aa597383d7724e3861e2578cf2d5e38afe155 Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Tue, 13 Oct 2026 11:27:16 -0700 Subject: [PATCH 8/9] Close the gaps a fresh review found in the ModelCache sources design The design required a pull secret for any private registry while saying the kubelet uses the node's identity, put every source's model at the same path, and still let placement require a ReadWriteMany storage class for sources that never stage. It also claimed image volumes need Kubernetes 1.36, said nothing about the claim's namespace or access mode for Existing, and failed a cluster for a claim the user may create later. Pull secrets are now optional, the HuggingFace path differs from OCI and Existing, and only HuggingFace needs a storage class. Image volumes need Kubernetes 1.35 or later with containerd 2.1 or CRI-O. An Existing claim must be in the replica's namespace, needs ReadWriteMany or ReadOnlyMany for a gang, and a missing one is Pending. A tag resolves once per cluster, compose-model-replica reads the ModelCache, and ModelExpress stays HuggingFace only. The proxy future improvement and repeated sentences are cut. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Dennis Ramdass --- design/modelcache-sources.md | 114 +++++++++++++++++------------------ 1 file changed, 56 insertions(+), 58 deletions(-) diff --git a/design/modelcache-sources.md b/design/modelcache-sources.md index 4ca95bb4c..adc49ce8e 100644 --- a/design/modelcache-sources.md +++ b/design/modelcache-sources.md @@ -8,11 +8,11 @@ ## Summary A `ModelCache` reads only from HuggingFace, and it stages what it reads onto a -ReadWriteMany claim on each cluster. That claim can be slower than no cache, and -it can fail on its own. `compose-model-replica`, the function that mounts the -cache into engine pods, works out the claim's name itself, and it doesn't wait -for the cache. An engine can start before the weights are -there, and download them a second time. +ReadWriteMany claim on each cluster. On some clouds that claim is slow, costly +at small sizes, or fails to bind. `compose-model-replica`, the function that +mounts the cache into engine pods, works out the claim's name itself, and it +doesn't wait for the cache. An engine can start before the weights are there, +and download them a second time. I propose two changes: @@ -120,37 +120,44 @@ spec: oci: ref: "" # a tag or a digest subPath: "" # the directory that holds the weights - imagePullSecrets: [] # secrets on each cluster, for a private registry + imagePullSecrets: [] # optional secrets on each cluster existing: - claimName: "" # a claim bound on each matched cluster + claimName: "" # a claim in the replica's namespace on each cluster subPath: "" ``` | Source | The user gives | What mounts it | What the cluster needs | |---|---|---|---| | `HuggingFace` | A repository ID | A claim that Modelplane stages | A ReadWriteMany storage class | -| `OCI` | A container image that holds the weights | An `image` volume | Kubernetes 1.36 | +| `OCI` | A container image that holds the weights | An `image` volume | Kubernetes 1.35 or later, and containerd 2.1 or later or CRI-O | | `OCI` | A model artifact | A `csi` volume from the model CSI driver | The driver, which Modelplane installs on every cluster | -| `Existing` | A claim the user filled | A `persistentVolumeClaim` volume | The claim, bound on the cluster | +| `Existing` | A claim the user filled | A `persistentVolumeClaim` volume | The claim, bound on the cluster. A multi-node gang needs ReadWriteMany or ReadOnlyMany. | For an `OCI` source: - Modelplane reads the reference's manifest from each cluster to find which kind of artifact it is. The user doesn't say. Any other kind of artifact fails the cluster entry, and the message gives the manifest's media type. -- Modelplane resolves a tag to its digest and writes the digest into `mount`. A - tag that moves later doesn't change what a running deployment serves. -- A private registry needs `oci.imagePullSecrets`. Modelplane reads the manifest - with them, and `mount` lists them for the engine pod's pull. +- - Modelplane resolves a tag to its digest once, stores the digest in the + cluster entry, and writes it into `mount`. It resolves again only when the + spec changes. Each cluster resolves a moving tag on its own, possibly to a + different digest. A digest in `ref` pins every cluster to one model. +- `oci.imagePullSecrets` is optional. When the cache names them, `mount` lists + them for the engine pod. Without them, the kubelet pulls with the node's + credential provider. - The kubelet pulls the reference when the pod starts, and shares the pulled bytes with later pods on the node. -`sizeGiB` applies only to `HuggingFace`, the one source that stages onto a -Modelplane claim. +For an `Existing` source, the claim must be in the namespace where the +replica's pods run. A claim that isn't there yet leaves the cluster entry +`Pending`, with a message. -Modelplane installs the model CSI driver on every cluster. A composition -function sees one cluster, not the fleet's caches, so it can't know whether a -model artifact will come to that cluster. +Only `HuggingFace` stages onto a Modelplane claim, so only `HuggingFace` uses +`sizeGiB` and a cluster's `status.cache.storageClassName`. `compose-model-cache` +and `compose-model-deployment` don't require a storage class for `OCI` and +`Existing`. + +Modelplane installs the model CSI driver on every cluster. ### The mount contract @@ -165,44 +172,47 @@ mount: imagePullSecrets: [] # corev1.LocalObjectReference ``` -The model is always at `/mnt/models`. `env` is empty for `OCI` and `Existing`. A -`HuggingFace` entry sets `HF_HUB_CACHE`. The engine then finds the staged -snapshot by repository ID. +`OCI` and `Existing` put the model at `/mnt/models`, and `env` is empty. A +`HuggingFace` entry puts the hub cache root at `/mnt/models` and sets +`HF_HUB_CACHE`. The engine then finds the staged snapshot by repository ID. `compose-model-cache` knows the source, so it writes the contract. Each cluster entry comes from that cluster's own observed resources, so a failed cluster doesn't change another cluster's entry. -`compose-model-replica` builds the pod, and it doesn't need to know the source. -It follows these rules: +`compose-model-replica` reads the `ModelCache` as a required resource, and +builds the pod without knowing the source. It follows these rules: - It finds the entry for its own cluster. - It adds `imagePullSecrets` to the pod. - It adds `env` before the user's environment. A user's variable with the same name then takes effect. It adds `volumes` and `volumeMounts` as they are. - If a mount is read-only, it adds an `emptyDir` with a size limit and points - `HOME` at it. Engines write tokenizer, compile, and lock files next to the - weights. Without a size limit, a large compile cache can get the pod evicted. + `HOME` at it. Engines write compile and download caches under `HOME`, and the + size limit stops a large one from getting the pod evicted. - If the cluster has no entry, or the entry has no `mount`, it sets a condition and composes no workload. This fixes #495. ### Cluster status `InferenceCluster.status.cache` gets an `imageVolumes` boolean, next to -`storageClassName`. Modelplane sets it from the cluster's Kubernetes version. It -says whether the cluster can mount an image volume. +`storageClassName`. Modelplane sets it from the cluster's observed server +version and its nodes' container runtime. It says whether the cluster can mount +an image volume. It only matters for container images. A model artifact uses the CSI driver on -any version. A container image on a cluster below 1.36 fails the cluster entry, -and the message gives the version. Of the clouds Modelplane provisions, only EKS -and Vultr run 1.36 today. +any version. A container image on a cluster that can't mount it fails the +cluster entry, and the message gives the reason. + +ModelExpress applies only to `HuggingFace` caches. An `OCI` pull is already +local to each node. ### Examples #### OCI -The `qwen3-8b` cache from the summary gives this cluster entry on a 1.36 -cluster: +The `qwen3-8b` cache from the summary gives this cluster entry on a cluster +that can mount image volumes. An image volume is always read-only: ```yaml status: @@ -251,12 +261,10 @@ status: mountPath: /mnt/models subPath: Kimi-K2-Instruct - name: gke-us-central1 - phase: Failed + phase: Pending message: persistentvolumeclaim "kimi-k2-weights" not found ``` -The second cluster has no `mount`. - ### Limits - **A Ready cache can still hold the wrong model.** Resolving the reference @@ -269,27 +277,19 @@ The second cluster has no `mount`. its largest model. - **The CSI driver is another component.** It's a DaemonSet to install and upgrade on every cluster, and its registry credentials are static. The kubelet - uses the node's identity for ECR, Artifact Registry, and ACR. Container images - avoid both costs. -- **The CSI driver must stop after its pods.** Otherwise a pod stays - `Terminating`, as in #499. + uses the node's identity for ECR, Artifact Registry, and ACR. The driver must + also stop after its pods, or a pod stays `Terminating`, as in #499. Container + images avoid these costs. ## Future improvements **Copy to the node.** An init container could copy the model to the node's disk through a `hostPath` volume. That avoids the ReadWriteMany read for -`HuggingFace`, but it -puts each model on each node, so it needs size limits and eviction first. - -**HuggingFace through a proxy.** The staging Job downloads over HTTPS, not -through the container runtime. A proxy or a private CA needs environment, -annotations, and volumes on that Job. +`HuggingFace`, but it puts each model on each node, so it needs size limits and +eviction first. -**Cache metrics.** A `ModelCache` doesn't publish metrics. In #494 the cache -said `Hydrating` for 40 minutes, and no series recorded it. Pull time, split by -source and by warm or cold pull, and each cluster entry's phase would show both. -Pull time needs a node-level scrape, because the kubelet and the CSI driver -measure it. Cache state is on the control plane, so it waits on #476. +**Cache metrics.** Pull time and each cluster entry's phase would have shown the +stuck cache in #494. Cache state is on the control plane, so it waits on #476. **Pod faults in the cluster entry.** #498 asks the control plane to show why a model doesn't serve. The contract covers the model not being readable. A @@ -300,8 +300,7 @@ container's exit code needs another data source. ### A child resource for each cluster #210 asks for a `ModelCacheHydration` child for each cluster. The contract -needs one place for each cluster, and `status.clusters[]` already gives it. A -child resource would contain only one status field for `OCI` and `Existing`. +needs one place for each cluster, and `status.clusters[]` already gives it. ### One enum value for each kind of artifact @@ -312,12 +311,11 @@ and a wrong guess would only fail the cluster entry. ### Install the CSI driver only where it's needed Installing the driver only on clusters that serve a model artifact would save an -idle DaemonSet. But a composition function can't see which clusters those are, -and a missing driver means a pod that never starts. +idle DaemonSet. But a composition function sees one cluster, not the fleet's +caches, and a missing driver means a pod that never starts. ### CRI-O's built-in artifact mounts -CRI-O can mount model artifacts with no driver from v1.33, behind a flag. -Modelplane provisions no CRI-O clusters, and a second mount path adds another -branch to test. The driver stops -being needed once containerd adds the same feature. +CRI-O can mount OCI artifacts as image volumes, with no driver. Modelplane +provisions no CRI-O clusters, and a second mount path adds another branch to +test. The driver stops being needed once containerd adds the same feature. From 6db92f761e6a9a02737b277397d1f884ef956aa8 Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Tue, 13 Oct 2026 11:52:46 -0700 Subject: [PATCH 9/9] Name the OCI resolver and the PersistentVolumeClaim source The design said Modelplane reads an OCI manifest from each cluster without saying what reads it or with which credential, and it named the user-filled claim source Existing, a word InferenceCluster already uses for something else. A resolver Job on each cluster reads the manifest with oci.imagePullSecrets and writes the digest and media type to a ConfigMap Modelplane observes, as it observes the HuggingFace staging Job. A private registry therefore needs a pull secret even where the kubelet could use the node's identity, which the limits now say. The source is now PersistentVolumeClaim, with a persistentVolumeClaim block. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Dennis Ramdass --- design/modelcache-sources.md | 64 ++++++++++++++++++++---------------- 1 file changed, 35 insertions(+), 29 deletions(-) diff --git a/design/modelcache-sources.md b/design/modelcache-sources.md index adc49ce8e..2d1171762 100644 --- a/design/modelcache-sources.md +++ b/design/modelcache-sources.md @@ -17,7 +17,7 @@ and download them a second time. I propose two changes: - `spec.source` gets two new values. `OCI` reads the model from a registry with - no claim. `Existing` mounts a claim the user already filled. + no claim. `PersistentVolumeClaim` mounts a claim the user already filled. - Each cluster entry in `status.clusters[]` gets a `mount` field. It says how to read the model on that cluster. `compose-model-replica` copies it into the pod, and waits until it's there. @@ -115,13 +115,13 @@ This design doesn't cover: ```yaml spec: - source: OCI # HuggingFace, OCI, or Existing + source: OCI # HuggingFace, OCI, or PersistentVolumeClaim huggingFace: {} # unchanged oci: ref: "" # a tag or a digest subPath: "" # the directory that holds the weights - imagePullSecrets: [] # optional secrets on each cluster - existing: + imagePullSecrets: [] # secrets on each cluster, for a private registry + persistentVolumeClaim: claimName: "" # a claim in the replica's namespace on each cluster subPath: "" ``` @@ -131,31 +131,34 @@ spec: | `HuggingFace` | A repository ID | A claim that Modelplane stages | A ReadWriteMany storage class | | `OCI` | A container image that holds the weights | An `image` volume | Kubernetes 1.35 or later, and containerd 2.1 or later or CRI-O | | `OCI` | A model artifact | A `csi` volume from the model CSI driver | The driver, which Modelplane installs on every cluster | -| `Existing` | A claim the user filled | A `persistentVolumeClaim` volume | The claim, bound on the cluster. A multi-node gang needs ReadWriteMany or ReadOnlyMany. | +| `PersistentVolumeClaim` | A claim the user filled | A `persistentVolumeClaim` volume | The claim, bound on the cluster. A multi-node gang needs ReadWriteMany or ReadOnlyMany. | For an `OCI` source: -- Modelplane reads the reference's manifest from each cluster to find which kind - of artifact it is. The user doesn't say. Any other kind of artifact fails the - cluster entry, and the message gives the manifest's media type. -- - Modelplane resolves a tag to its digest once, stores the digest in the - cluster entry, and writes it into `mount`. It resolves again only when the - spec changes. Each cluster resolves a moving tag on its own, possibly to a - different digest. A digest in `ref` pins every cluster to one model. -- `oci.imagePullSecrets` is optional. When the cache names them, `mount` lists - them for the engine pod. Without them, the kubelet pulls with the node's - credential provider. +- On each cluster, a resolver Job reads the reference's manifest. The Job writes + the digest and the manifest's media type to a ConfigMap, and Modelplane + observes the ConfigMap, as it observes the HuggingFace staging Job. +- The media type tells Modelplane which kind of artifact it is. The user doesn't + say. Any other kind of artifact fails the cluster entry, and the message gives + the media type. +- Modelplane resolves a tag once and stores the digest in the cluster entry. It + writes the digest into `mount`, and resolves again only when the spec changes. + Each cluster resolves a moving tag on its own, possibly to a different digest. + A digest in `ref` pins every cluster to one model. +- A private registry needs `oci.imagePullSecrets`. The resolver Job reads the + manifest with them, and `mount` lists them for the engine pod. A public + registry needs none. - The kubelet pulls the reference when the pod starts, and shares the pulled bytes with later pods on the node. -For an `Existing` source, the claim must be in the namespace where the -replica's pods run. A claim that isn't there yet leaves the cluster entry +For a `PersistentVolumeClaim` source, the claim must be in the namespace where +the replica's pods run. A claim that isn't there yet leaves the cluster entry `Pending`, with a message. Only `HuggingFace` stages onto a Modelplane claim, so only `HuggingFace` uses `sizeGiB` and a cluster's `status.cache.storageClassName`. `compose-model-cache` and `compose-model-deployment` don't require a storage class for `OCI` and -`Existing`. +`PersistentVolumeClaim`. Modelplane installs the model CSI driver on every cluster. @@ -172,8 +175,8 @@ mount: imagePullSecrets: [] # corev1.LocalObjectReference ``` -`OCI` and `Existing` put the model at `/mnt/models`, and `env` is empty. A -`HuggingFace` entry puts the hub cache root at `/mnt/models` and sets +`OCI` and `PersistentVolumeClaim` put the model at `/mnt/models`, and `env` is +empty. A `HuggingFace` entry puts the hub cache root at `/mnt/models` and sets `HF_HUB_CACHE`. The engine then finds the staged snapshot by repository ID. `compose-model-cache` knows the source, so it writes the contract. Each cluster @@ -231,7 +234,7 @@ status: readOnly: true ``` -#### Existing +#### PersistentVolumeClaim ```yaml apiVersion: modelplane.ai/v1alpha1 @@ -240,8 +243,8 @@ metadata: name: kimi-k2 namespace: ml-team spec: - source: Existing - existing: + source: PersistentVolumeClaim + persistentVolumeClaim: claimName: kimi-k2-weights subPath: Kimi-K2-Instruct ``` @@ -269,17 +272,20 @@ status: - **A Ready cache can still hold the wrong model.** Resolving the reference finds a missing repository, a bad pull secret, or an unknown artifact kind. It - can't tell that the artifact is a different model, or that an `Existing` - claim is incomplete. Those fail when the engine starts. + can't tell that the artifact is a different model, or that the claim behind a + `PersistentVolumeClaim` source is incomplete. Those fail when the engine + starts. - **An `OCI` source uses the node's disk.** The CSI driver keeps artifacts under `/var/lib/model-csi`. Image volumes go to the image store, under the kubelet's `imagefs` eviction limits. `sizeGiB` controls neither. A pool's disk must fit its largest model. +- **A private registry needs a pull secret.** The resolver Job can't use the + node's identity, so a private ECR, Artifact Registry, or ACR repository needs + `oci.imagePullSecrets` too. - **The CSI driver is another component.** It's a DaemonSet to install and - upgrade on every cluster, and its registry credentials are static. The kubelet - uses the node's identity for ECR, Artifact Registry, and ACR. The driver must - also stop after its pods, or a pod stays `Terminating`, as in #499. Container - images avoid these costs. + upgrade on every cluster, and its registry credentials are static. The driver + must also stop after its pods, or a pod stays `Terminating`, as in #499. + Container images avoid these costs. ## Future improvements