fix(chart): harden Kubernetes persistence - #5088
Open
TheSentinel454 wants to merge 1 commit into
Open
Conversation
Signed-off-by: Luke Tornquist <tornquist@squareup.com>
TheSentinel454
marked this pull request as ready for review
August 6, 2026 18:09
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A real multi-node Kubernetes deployment exposed two storage assumptions that are not portable across CSI drivers. A
RollingUpdatecan start the replacement relay pod before the existing pod releases itsReadWriteOnceGit scratch volume, while a multi-replica Deployment currently points every replica at the same chart-managed claim. Drivers that enforce single attachment can leave upgrades wedged or replicas permanently pending.The Git volume is working space, not the durable source of truth. Git objects, manifests, and refs live in S3-compatible object storage, while repository-name metadata lives in Postgres. Multi-replica deployments should therefore use a per-pod
emptyDir; single-replica deployments that retain a single-writer claim should useRecreateso Kubernetes releases the old mount before starting its replacement.Setting
persistence.git.enabled=falsedisables only the Git scratch PVC; it does not disable Git or remove durable Git storage. Each request hydrates a temporary repository from object storage, and an accepted push is durable before the relay reports success.What
Recreatefor relay deployments mountingReadWriteOnceorReadWriteOncePodGit scratch claimsRollingUpdatefor per-podemptyDirand multi-writer claimsRisk Assessment
Medium — installations using the default single-writer Git PVC will have brief relay downtime during upgrades because
Recreatestops the existing pod before starting its replacement. This trades rolling availability for deterministic volume handoff and avoids an upgrade becoming stuck indefinitely.Multi-replica configurations that share one
ReadWriteOnceorReadWriteOncePodclaim will now stop during chart rendering. Operators must usepersistence.git.enabled=falsefor per-pod scratch space or provide a multi-writer claim. This does not discard durable Git state because S3-compatible storage and Postgres remain the sources of truth.Verification
just ciRuntime integration/E2E tests were not run because this PR changes only Helm templates, values, tests, examples, and documentation; it does not change relay, database, or authentication implementation. The chart-specific unit, lint, and render tests exercise the changed behavior directly.
Manual test
Render a single-replica deployment with the default
ReadWriteOnceGit PVC and verify that its strategy isRecreate. Then render a multi-replica deployment with Redis enabled andpersistence.git.enabled=false; verify that the strategy isRollingUpdateand every pod receives its ownemptyDir. A multi-replica render using the chart-managedReadWriteOnceclaim should fail with an actionable validation message.References
Generated with Codex