Repository navigation
fix(quorum): retry a failed hand-over, and catch up a member rebuilt on an empty disk - #679
Merged
Merged
Conversation
…other member A leader cut off from PostgreSQL handed over to the next member in kates.quorum.peers, and a failed transfer counted as a hand-over: the next try waited 2 minutes. In the e2e that member was the one fenced a step before; its outbound connections had not recovered, its election got no votes, and the old leader won the next one. Main has failed the check on every Operator run since. A transfer that fails, or after which this member still leads, no longer starts the 2-minute rest, and the next check, 5 s on, asks the member after the one that failed. e2e-node-loss.sh: on two nodes, a follower sharing one gets a new volume, which the scheduler places on the third, instead of failing. The anti-affinity is preferred, and a local volume pins a member.
Kind keeps the control plane's NoSchedule taint when the cluster has workers, so the three members ran on two nodes and the node-loss test had no third to spread them over.
…n it has not caught up
…tabase The test PostgreSQL could share the lost node: every member then lost it, and the leader handed over by design while the test checked that it led throughout. The database now prefers the control plane, which the test never takes, and the lost node is never the database's.
Ratis 3.3.1's leader keeps the index a member matched before it lost its disk: it answers the member's INCONSISTENCY nextIndex=0 with heartbeats from that index, and the member stays at applied index -1 until the leader changes. A follower whose log is still empty 15 s after it knows a leader now asks that leader to step down, at most once a minute; the next leader starts replication over and sends the whole log. The node-loss e2e checks the leader only while the member is gone, and gives the catch-up 90 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…etry # Conflicts: # kates/src/main/java/com/bmscomp/kates/quorum/DatabaseHandover.java # kates/src/main/java/com/bmscomp/kates/quorum/Quorum.java # operator/e2e-quorum.sh
This was referenced Nov 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
main's CI · Operator E2E has failed on every run since #674/#675. With #676, this fixes the two quorum bugs behind it and the nightly test setup.
Hand-over without PostgreSQL (J4d)
#676 now tries each other member in turn and retries a failed hand-over at the next check. On top of it, this PR also counts a transfer as failed when this member leads again afterwards (its target lost the election), so the next check retries instead of resting 2 minutes.
A member rebuilt on an empty disk (O6, #672)
Ratis 3.3.1's leader keeps the index a member matched before it lost its disk. It answers the member's
INCONSISTENCY nextIndex=0with heartbeats only, so the member stays at applied index -1 until the leader changes. With the newEmptyLogCatchUpcheck, a follower whose log is still empty 15 s after it knows a leader asks that leader to step down, at most once a minute. The next leader sends it the whole log.QuorumMemberTestreproduces the stuck state, then the catch-up.Nightly node-loss test
alpha, so the three members can land on three nodes.Tests