Skip to content

fix(chart): harden Kubernetes persistence - #5088

Open
TheSentinel454 wants to merge 1 commit into
block:mainfrom
TheSentinel454:codex/kubernetes-relay-docs
Open

fix(chart): harden Kubernetes persistence#5088
TheSentinel454 wants to merge 1 commit into
block:mainfrom
TheSentinel454:codex/kubernetes-relay-docs

Conversation

@TheSentinel454

@TheSentinel454 TheSentinel454 commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Why

A real multi-node Kubernetes deployment exposed two storage assumptions that are not portable across CSI drivers. A RollingUpdate can start the replacement relay pod before the existing pod releases its ReadWriteOnce Git scratch volume, while a multi-replica Deployment currently points every replica at the same chart-managed claim. Drivers that enforce single attachment can leave upgrades wedged or replicas permanently pending.

The Git volume is working space, not the durable source of truth. Git objects, manifests, and refs live in S3-compatible object storage, while repository-name metadata lives in Postgres. Multi-replica deployments should therefore use a per-pod emptyDir; single-replica deployments that retain a single-writer claim should use Recreate so Kubernetes releases the old mount before starting its replacement.

Setting persistence.git.enabled=false disables only the Git scratch PVC; it does not disable Git or remove durable Git storage. Each request hydrates a temporary repository from object storage, and an accepted push is durable before the relay reports success.

What

  • Use Recreate for relay deployments mounting ReadWriteOnce or ReadWriteOncePod Git scratch claims
  • Preserve RollingUpdate for per-pod emptyDir and multi-writer claims
  • Reject multi-replica configurations that share one chart-managed single-writer claim
  • Update HA examples to use per-pod Git scratch storage
  • Add an explicit storage-class setting for bundled MinIO
  • Document relay and owner identities, onboarding paths, storage migration constraints, and the current backup model
  • Update the Flux example to the latest published chart version

Risk Assessment

Medium — installations using the default single-writer Git PVC will have brief relay downtime during upgrades because Recreate stops the existing pod before starting its replacement. This trades rolling availability for deterministic volume handoff and avoids an upgrade becoming stuck indefinitely.

Multi-replica configurations that share one ReadWriteOnce or ReadWriteOncePod claim will now stop during chart rendering. Operators must use persistence.git.enabled=false for per-pod scratch space or provide a multi-writer claim. This does not discard durable Git state because S3-compatible storage and Postgres remain the sources of truth.

Verification

  • just ci
  • Helm unit tests: 49/49 passed
  • All chart render fixtures passed
  • Helm lint completed with no chart failures
  • Verified explicit storage-class rendering for Git scratch, Postgres, Redis, and MinIO
  • Verified that unsafe multi-replica/single-writer configurations are rejected before deployment

Runtime integration/E2E tests were not run because this PR changes only Helm templates, values, tests, examples, and documentation; it does not change relay, database, or authentication implementation. The chart-specific unit, lint, and render tests exercise the changed behavior directly.

Manual test

Render a single-replica deployment with the default ReadWriteOnce Git PVC and verify that its strategy is Recreate. Then render a multi-replica deployment with Redis enabled and persistence.git.enabled=false; verify that the strategy is RollingUpdate and every pod receives its own emptyDir. A multi-replica render using the chart-managed ReadWriteOnce claim should fail with an actionable validation message.

References

  • Related open issue or PR: none found
  • Follow-up work: none identified

Generated with Codex

Signed-off-by: Luke Tornquist <tornquist@squareup.com>
@TheSentinel454 TheSentinel454 changed the title Harden Kubernetes persistence and deployment guidance fix(chart): harden Kubernetes persistence Aug 6, 2026
@TheSentinel454
TheSentinel454 marked this pull request as ready for review August 6, 2026 18:09
@TheSentinel454
TheSentinel454 requested a review from a team as a code owner August 6, 2026 18:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant