Repository navigation
Restart each pod once per rolling update (v0.3.3) - #19
Merged
Merged
Conversation
A one-node pool restarted its pod twice in one rolling update. With the proxy enabled the StatefulSet kept RollingUpdate with partition 0, so the StatefulSet controller replaced each pod as soon as the template changed, before the proxy had excluded it; the operator then excluded, drained and deleted the replacement again. Larger pools had the same race: most pods were replaced without the exclude, some twice. - With the proxy enabled the partition is pinned at the replica count, so a template change replaces nothing by itself. - The operator walks ordinals from the highest down. After the exclude and drain it lowers the partition to that ordinal and the StatefulSet controller replaces the pod, once; it waits for the pod on the update revision, Ready, then includes it. - A pod already on the update revision is not restarted. - An exclude or include a running proxy keeps refusing is given up after 2 minutes (plus 10 for the restart, for include) with a Warning event, so a broken proxy cannot hold an update. - Without the proxy nothing changes. Chart 0.3.3.
austin-barrington
added a commit
that referenced
this pull request
Oct 24, 2026
Newest known HyperbyteDB: 0.9.3-beta, so a 0.9.3-beta cluster no longer reads NewerThanKnown; every capability needs 0.9.1-beta or older, so nothing renders differently. docs/compatibility.md: the v0.3.3 row and the backup, Go and CA changes (#20) in the v0.3.2 -> v0.3.3 notes. v0.3.3: each pod restarts once per rolling update (#19); backups use the AWS CLI since minio/mc can no longer be pulled, and the image is built with Go 1.27.3 on a base without the compromised eMudhra root (#20). Checked on kind with HyperbyteDB 0.9.3 (3-node sync_quorum and 1-node pools behind the proxy, 10 writes/s): one restart per pod, every acknowledged write on every pod, in three rolling updates.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A one-node pool restarted its pod twice in one rolling update (reported from kind, operator 0.3.2).
Cause. With the proxy enabled the StatefulSet kept
RollingUpdatewith partition 0. The StatefulSet controller replaced each pod as soon as the template changed, before the proxy had excluded it. The operator's own coordination then excluded, drained (10 s) and deleted the pod again, though it was already on the new revision. Larger pools had the same race: the StatefulSet controller worked down from the top while the operator worked up from 0, so most pods were replaced without the exclude, and some twice. Those un-excluded replacements are also part of why the proxy answered 503 during restarts.Fix.
rolloutPartition), written together with any template change, so a template change replaces nothing by itself.releaseOrdinal). The StatefulSet controller replaces that pod, once. The operator waits for the pod on the update revision, Ready, then includes it. When the update completes the partition goes back up.spec.proxy.enabled: false) nothing changes.Upgrade. The partition is applied on the first reconcile after the operator upgrade, which restarts nothing. A restart already in flight from v0.3.2 continues: its pods that are already updated are skipped. The proxy is on by default, so this changes how every such pool rolls; there are no pod template changes.
Tests.
rollout_partition_test.go. Envtest runs no StatefulSet controller, so the test plays it: it replaces the highest released pod not on the update revision.WaitingReady(after exclude and drain); the partition is pinned again afterwards.Not yet run on a real cluster.