Repository navigation
Propose ModelCache sources and a mount contract - #505
Draft
dennis-upbound wants to merge 5 commits into
Draft
dennis-upbound wants to merge 5 commits into
dennis-upbound wants to merge 5 commits into
Conversation
Collaborator
Author
|
Closing until the design is reviewed in upbound/inference#7, which holds this document and the Upbound Inference one together. The split between them is the thing worth a reviewer's judgement, and it is only visible with both side by side. This reopens, or comes back as a fresh PR, once that review settles. The branch stays at Nothing here is blocked: #504 (the telemetry guide correction) and the research on #476 are independent. |
A ModelCache reads from HuggingFace only, stages to a volume on each matched cluster, and publishes a phase. A consumer calculates the name of that volume, and two functions hold a copy of a calculation that one source makes true. This design adds an OCI source and an Existing source, and a mount contract on each cluster entry. The contract holds the volumes, the volume mounts and the environment that a consumer adds to a pod, in the Kubernetes types of those names. The consumer copies it and never learns the source. Section 4.4 gives the faults the current design permits, from the benchmark work: modelplaneai#495, where an engine starts while the cache stages and reads a 464 GB model a second time; modelplaneai#494, where a replaced node strands the staging volume; and modelplaneai#498, which asks for the cause on the control plane. Section 6 says what model delivery should measure, because the performance claim this design rests on has no series behind it today. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Dennis Ramdass <dennis@upbound.io>
The introduction was written before the benchmark reports went into section 4.2, and it still argued the weaker case. It said a user cannot name their own registry, where the reports say the volume those users are forced through is slower than no cache (modelplaneai#204), does not take the size they asked for on GKE (modelplaneai#414), has no measurement it must satisfy (modelplaneai#383), and fails on its own (modelplaneai#494, modelplaneai#221). An OCI source removes that volume, which is most of why it is worth having. The introduction also named three problems where section 3 lists four tasks. Nothing set up the fourth, that a cluster never says why it cannot read the artifact, which is what modelplaneai#495 cost: an engine started while the cache staged and read a 464 GB model a second time. Each task in section 3 now traces to a problem here. Section 5.1 said the first part lets a user name an artifact where it is. That is true and incomplete, for the same reason. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Dennis Ramdass <dennis@upbound.io>
"A ModelCache does three things, and each one causes a problem" counts the items before giving them and tells the reader they are problems. Both are the writer talking about the document rather than writing it. The same sentence appeared again in section 5.1 as "The design has two parts", followed by a list and then a paragraph restating the list. The introduction is now plain declarative prose with no bold scaffolding, which is what the rest of the document uses. Section 5.1 drops the list and says the two changes in the order a reader meets them. The vale ai-tells rules pass this construction, so it is one to catch by reading. I have added it to the skill. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Dennis Ramdass <dennis@upbound.io>
The design was reviewed alongside the Upbound Inference one. This brings in the wording that came out of that review and cites the OCI delivery study it draws on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Dennis Ramdass <dennis@upbound.io>
dennis-upbound
force-pushed
the
dennis/design-modelcache-sources
branch
from
October 12, 2026 18:38
b14c48d to
a5deceb
Compare
dennis-upbound
marked this pull request as draft
October 12, 2026 18:38
The stricter rules flag counted lead-ins, negated subjects and figurative verbs throughout. Rewrite those sentences in plain terms without changing what the design says. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Dennis Ramdass <dennis@upbound.io>
dennis-upbound
changed the title
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of your changes
A
ModelCachereads from HuggingFace and nowhere else, and stages what it reads onto a ReadWriteMany volume on every matched cluster. A consumer reads the artifact by deriving the volume's name, which is right for one source, and nothing tells it whether the artifact is ready yet (#495).This proposes two more sources and a mount contract:
source: OCIreads an artifact from a registry. The kubelet mounts it read-only at pod start, as an image volume for a container image or through a CSI driver for a model artifact. Modelplane composes nothing per cluster for it.source: Existingmounts a claim the user already filled.status.clusters[].mountgives the volumes, mounts and environment that read the artifact on each cluster. A consumer copies them into its pod and doesn't need to know the source.It also proposes metrics that show whether a cache helps (section 6). It supersedes #362 and #210, and revises
design/modelcache.md.Refs #512, #414, #457
I have:
Runnix flake check(or./nix.sh flake check) and made sure it passes.Added or updated tests covering any composition function changes.git commit -s.🤖 Generated with Claude Code