Repository navigation
Bump the e2e workload node image past the DRA idle deadlock - #421
Merged
haarchri merged 1 commit intoSep 17, 2026
Merged
Conversation
Collaborator
|
thanks for the bump, can you please add the checklist and rebase ? i will add the |
The workload cluster runs kindest/node:v1.34.0, whose kubelet deadlocks once its gRPC connection to a DRA driver has been idle for 30 minutes. The next pod referencing a ResourceClaim then blocks in ContainerCreating with a single Scheduled event, no kubelet error, and NodePrepareResources never reaching the driver. Fixed upstream in v1.34.2 (k/k#133934). CI never sees it, since those clusters are minutes old. A cluster left up overnight hits it on the first ModelDeployment of the morning, and the symptom carries nothing that points at the kubelet. The pin exists for the DRA APIs, GA in 1.34, so any 1.34 image satisfies it. v1.34.8 is the oldest published image that also clears the deadlock, and it carries containerd 2.3.1, past the 2.2.0 that installation.md pins the getting-started cluster away from. Whether Modelplane runs on containerd 2.3.x is untested here; that docs pin is a separate constraint and is left alone, since its cluster runs no DRA driver. The reuse guard now rejects a v1.34.0 or v1.34.1 cluster rather than reusing it. Long-lived dev clusters are the ones that hit this, so bumping only the image would leave them deadlocking. Fixes modelplaneai#419. Signed-off-by: Nick Nikolakakis <nonicked@protonmail.com>
nicknikolakakis
force-pushed
the
bump-workload-node-image
branch
from
September 16, 2026 03:52
94fa6d2 to
5c0c6f1
Compare
Contributor
Author
|
Rebased onto main, no conflicts, the diff is unchanged. The checklist is already at the bottom of the description: three checked, and the composition-function one struck through since this only touches |
haarchri
approved these changes
Sep 17, 2026
github-project-automation
Bot
moved this from Backlog
to In Review
in Modelplane Roadmap
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #419.
The e2e workload cluster runs
kindest/node:v1.34.0, whose kubelet deadlocks once its connection to a DRA driver has been idle for 30 minutes: the next pod referencing a ResourceClaim sits inContainerCreatingwith oneScheduledevent and nothing else, andNodePrepareResourcesnever reaches the driver. The goroutine dump and the reproduction are in the issue. Fixed upstream by kubernetes/kubernetes#133934, whichCHANGELOG-1.34.mdlists underv1.34.2.The pin's stated reason is the DRA APIs, GA in 1.34, so any 1.34 image satisfies it. The containerd constraint you may remember is a different pin: #315 changed only
docs/content/getting-started/installation.md, where the comment says kind v0.31+ ships containerd 2.2.0.e2e/run.shgot its pin from the e2e PR instead and never carried that constraint. containerd 2.2.0 was also one release: v1.34.3 has 2.2.0, v1.34.8 has 2.3.1, v1.34.11 has 2.3.4, each checked withdocker run --rm --entrypoint containerd <image> --version.So this takes v1.34.8, the oldest published image clearing both the deadlock and 2.2.0. v1.34.11 is there if you would rather track kind's latest. I have not run the full e2e locally, so whether Modelplane is happy on containerd 2.3.x is what the
test-e2elabel would settle. The docs pin is untouched: that cluster runs no DRA driver, so the deadlock doesn't reach it.The reuse guard rejects a v1.34.0 or v1.34.1 cluster now instead of reusing it, since long-lived dev clusters are the population that hits this and bumping only the image would leave them deadlocking.
Touches
e2e/run.shlike #420 does, on different lines.I have:
nix flake check(or./nix.sh flake check) and made sure it passes.Added or updated tests covering any composition function changes.This PR changes onlye2e/run.sh, no composition functions.git commit -s.