feat(k8s): always re-pull remote game server images (v1.0.2)#4
Conversation
Game server Pods and installation Jobs hard-coded ImagePullPolicy: IfNotPresent, so once an image was cached on a node, restarting or redeploying a server reused the stale local copy and never picked up updated tags. This mirrors the Docker backend instead, which always attempts a pull before (re)creating a container. - Add resolveImagePullPolicy: remote images default to Always so updated tags are re-pulled; ~-prefixed local images stay IfNotPresent. - Add optional kubernetes.image_pull_policy override (Always / IfNotPresent / Never) for air-gapped clusters. - Surface image_pull_policy in the Helm chart (values + ConfigMap) and document the behavior in the chart and RBAC READMEs. - Bump chart version/appVersion to 1.0.2.
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
| if strings.HasPrefix(image, "~") { | ||
| return cleaned, corev1.PullIfNotPresent | ||
| } | ||
| return cleaned, corev1.PullAlways |
There was a problem hiding this comment.
🚩 Behavioral change: default pull policy shifted from IfNotPresent to Always for remote images
Before this PR, both game server Pods (container.go:151 old) and installer Jobs (installer.go:124 old) used corev1.PullIfNotPresent unconditionally. After this PR, remote (non-~) images default to corev1.PullAlways. This is an intentional change to match Docker backend behavior (which always attempts a pull for remote images, see environment/docker/container.go:381-382), but it's a meaningful behavioral difference for existing deployments: Kubernetes will now attempt to pull on every Pod/Job creation, adding latency and requiring registry access. Operators relying on pre-cached images without registry access will see failures after upgrading unless they set image_pull_policy: IfNotPresent.
Was this helpful? React with 👍 or 👎 to provide feedback.
Address three robustness issues in the Kubernetes backend: - api.go: Client() stored the init error in a function-local variable, so after a failed first call the sync.Once never re-ran and subsequent calls returned (nil, nil), masking the failure. Persist the error in a package variable and return it on every call. - container.go: SendCommand() checked IsAttached() before taking the lock that guards e.stream, so the stream could be cleared in between and cause a nil-pointer write. Nil-check e.stream under the read lock instead. - quota.go: buildResourceQuota/buildLimitRange used resource.MustParse on user-configurable quantity strings, panicking the process on invalid values. Parse with resource.ParseQuantity and propagate errors through EnsureResourceQuota/EnsureLimitRange.
ff5bd82 to
9474fa8
Compare
Harden the Kubernetes environment and Helm chart per the full CodeRabbit review: environment/kubernetes: - container.go: surface Exists() errors in Create(), order Destroy() offline state after the delete check, use apierrors.IsNotFound, and deduplicate container ports shared across allocation IPs. - power.go: handle delete/wait errors in OnBeforeStart, give signal-based stops a graceful SIGTERM grace period (immediate for SIGKILL), and treat a terminal (non-running) Pod as stopped so restart-while-off does not hang. - installer.go: propagate caller context into WriteInstallScript, use foreground propagation + wait when recreating the installer Job, and reject SubPath values that escape the root directory. - quota.go: fail fast on non-NotFound Get errors instead of falling through to Create. - storage.go: ResizePVC is a no-op when a shared DataPVC is configured. - network.go: reconcile Service type and annotations on a networkMode switch. - stats.go: bound each metrics/stats request with a timeout and retry the metrics API every tick instead of sticking to the kubelet fallback. server/router/system: - update.go: sync image and stop configuration to the Kubernetes environment. - router_system.go: return an error when no assignable IPs are discovered. - system.go: treat a missing os-release as best-effort in Kubernetes mode. chart/pelican-wings: - Render credentials into a Secret instead of a ConfigMap. - Guard wings.kubernetes.namespace against gameNamespace drift. - Fail rendering when ResourceQuota/LimitRange are enabled but empty. - Require an explicit serviceAccount.name when not creating one. - Gate the broad nodes/proxy kubelet fallback behind rbac.kubeletMetricsFallback. docs/manifests: - Split the optional nodes/proxy ClusterRole into clusterrole-metrics-kubelet.yaml. - Expand the chart and kubernetes RBAC READMEs (credentials, networkMode, multi-instance guidance, RBAC tables). Add regression tests for port deduplication, Service type reconciliation, ResizePVC with a shared DataPVC, and quota Get-failure fail-fast.
… test - server/manager.go: initialize Kubernetes Metadata with the server stop configuration so command/signal stop behavior is correct from first boot. - kubernetes/README.md: sync the namespace-scoped RBAC verb table with role.yaml (pods watch, services list, configmaps update). - chart README: use a local values file for credentials in the quick-start instead of leaking them via --set. - installer_test.go: assert the Run cancellation path returns context.Canceled.
Summary
Game server Pods and installation Jobs hard-coded
ImagePullPolicy: IfNotPresent, so once an image was cached on a node, restarting or redeploying a server reused the stale local copy and never picked up an updated tag — with no config knob to change it. This aligns the Kubernetes backend with the Docker backend, which always attempts a pull before (re)creating a container.New
resolveImagePullPolicycentralizes the decision and is used by both the server Pod (container.go) and the install Job (installer.go):kubernetes.image_pull_policyconfig (Always/IfNotPresent/Never) for air-gapped clusters; empty = the smart default above.wings.kubernetes.imagePullPolicy(values) → rendered into the ConfigMap asimage_pull_policy(omitted when empty). This is independent ofimage.pullPolicy, which is for the Wings daemon image itself.kubernetes/README.md.version/appVersionbumped to1.0.2.Note: the power-lock restart-while-off fix (#3) is already merged to
main, so this is the only remaining change since v1.0.1. Taggingv1.0.2after merge will trigger the release + helm/container publish workflows.Test plan
go build ./...,go vet ./environment/kubernetes/...— cleango test ./environment/kubernetes/...— newTestResolveImagePullPolicycovers remote→Always,~-local→IfNotPresent, and override→configured valuehelm lint chart/pelican-wings— passesgofmt— cleanLink to Devin session: https://app.devin.ai/sessions/c6e2874ef8924e68a929e491c2018270
Requested by: @Exonical