Repository navigation
fix(operator): a quorum whose members cannot run one is rolled all at once - #678
Merged
Merged
Conversation
… once The roll replaced one member at a time, and only under a settled leader. A Kates whose members never elect one (crashing on a bad image, or on an image older than the quorum) could then never be rolled to a fixed version: the status stayed "the next member is replaced once the quorum has no leader". When no member may run a quorum, every stale member is replaced together: there is no leader to keep and nothing serving, and a replaced pod keeps its volume. A member may run one when it is Ready and either reports a quorum or does not answer, so a Ready member the operator cannot reach keeps the leader rule. One at a time would not do: the first member back would wait for a majority the others cannot give.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found while testing #677 (
kates deploy --kates-mode operator).The deadlock
quorum::nextreplaced one stale member at a time, and only under a settled leader. AKateswhose members never elect a leader could never be rolled. Two cases hit it:/api/healthhas noquorumblock.Changing
spec.versionto a working image then leftReady=False RollingOut: the next member is replaced once the quorum has no leaderfor ever, and the only way out was deleting the pods by hand.The rule
When no member may run a quorum, every stale member is replaced at once (
Next::ReplaceAll).quorum.enabledor doesn't answer.Proof
Unit test
members_that_cannot_run_a_quorum_are_replaced_together_and_only_then:cargo fmt,clippy -D warningsandcargo testare green.On a three-node Kind cluster with this operator: a three-member
Kateswith all members crash-looping (kates:native-local, refused by a database the newer image had migrated). Afterspec.versionwas set to a working image, the operator recordedreplacing kates-0, kates-1, kates-2 with j4d, and the resource was Ready with leaderkates-214 s after the change. The operator before this change stayed at the message above.Not covered: rolling a working quorum down to a pre-quorum image still stalls after the second member, because one member that runs a quorum remains. Downgrading below the quorum isn't supported.