Repository navigation
fix(quorum): a leader cut off from PostgreSQL hands leadership on - #674
Merged
Merged
Conversation
J4d and J4e of plans/j4-resilient-quorum.md. J4d: the leader checks PostgreSQL every 5 s. After 15 s without it, once nothing is live or after 2 minutes, it hands leadership to the next member in kates.quorum.peers with Ratis' transferLeadership, at most once per 2 minutes, so members that all lack the database do not pass it back and forth. Before, it kept leading and every write failed while a member that could reach the database stood by (F5). J4e: e2e-quorum.sh cuts the leader from PostgreSQL with iptables and checks that another member leads and takes writes. The operator workflow runs nightly, and then also e2e-node-loss.sh on CI's three-node Kind. Changes under kates/.../quorum/ start the e2e.
bmscomp
added a commit
that referenced
this pull request
Nov 16, 2026
…on an empty disk (#679) * test(operator): print the quorum members' logs when the PostgreSQL handover check fails The resource is deleted before the workflow's Show the stack step, so a failed handover left nothing to read. main fails this check on every run since #675 merged on top of #674. * fix(quorum): a failed hand-over is retried at the next check, with another member A leader cut off from PostgreSQL handed over to the next member in kates.quorum.peers, and a failed transfer counted as a hand-over: the next try waited 2 minutes. In the e2e that member was the one fenced a step before; its outbound connections had not recovered, its election got no votes, and the old leader won the next one. Main has failed the check on every Operator run since. A transfer that fails, or after which this member still leads, no longer starts the 2-minute rest, and the next check, 5 s on, asks the member after the one that failed. e2e-node-loss.sh: on two nodes, a follower sharing one gets a new volume, which the scheduler places on the third, instead of failing. The anti-affinity is preferred, and a local volume pins a member. * ci(operator): untaint Kind's control plane for the node-loss test Kind keeps the control plane's NoSchedule taint when the cluster has workers, so the three members ran on two nodes and the node-loss test had no third to spread them over. * test(operator): print the leader's and the replaced member's logs when it has not caught up * test(operator): the node-loss test takes a worker without the test database The test PostgreSQL could share the lost node: every member then lost it, and the leader handed over by design while the test checked that it led throughout. The database now prefers the control plane, which the test never takes, and the lost node is never the database's. * fix(quorum): a member rebuilt on an empty disk has the leader step down Ratis 3.3.1's leader keeps the index a member matched before it lost its disk: it answers the member's INCONSISTENCY nextIndex=0 with heartbeats from that index, and the member stays at applied index -1 until the leader changes. A follower whose log is still empty 15 s after it knows a leader now asks that leader to step down, at most once a minute; the next leader starts replication over and sends the whole log. The node-loss e2e checks the leader only while the member is gone, and gives the catch-up 90 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
J4d and J4e of
plans/j4-resilient-quorum.md. They're in one PR because both editoperator/e2e-quorum.shandci-operator.yml.J4d (F5)
Before this, a leader cut off from PostgreSQL kept leading and every write failed, while a member that could reach the database stood by.
DatabaseHandoverchecks the database on the leader every 5 s.liveRuns()plus faults in flight) or after 2 minutes, it hands leadership to the next member inkates.quorum.peers(QuorumMember.handOver, RatistransferLeadershipon its own client and timeout).J4e
e2e-quorum.shgains a stage that cuts the leader from PostgreSQL with iptables, then checks that another member leads and answers a write 400, not 503.ci-operator.ymlruns nightly (03:17 UTC) and onworkflow_dispatch. Those runs also calle2e-node-loss.sh: CI's Kind cluster has three schedulable nodes.kates/src/main/java/com/bmscomp/kates/quorum/now start the e2e. The failure dump reads every operator pod's log.Proof
DatabaseHandoverTest: 15 s with nothing live; a live run waits up to 2 minutes; at most once in 2 minutes; the database coming back resets the clock.QuorumMemberTest: three real Ratis members, and the leader hands over to the member it names, in a later term../mvnw test: 1938 tests, 0 failures (run with-Dquarkus.grpc.codegen.skip=true).TestMCPCaveatRefAnchorspasses.Leading without PostgreSQL; handing over after PT15S, thenleadership handed to j4-0.workflow_dispatch.