Skip to content

fix(quorum): a leader cut off from PostgreSQL hands leadership on - #674

Merged
bmscomp merged 1 commit into
mainfrom
fix/quorum-hand-over-without-db
Nov 15, 2026
Merged

bmscomp merged 1 commit into
mainfrom
fix/quorum-hand-over-without-db

Conversation

@bmscomp

@bmscomp bmscomp commented Nov 15, 2026

Copy link
Copy Markdown
Owner

J4d and J4e of plans/j4-resilient-quorum.md. They're in one PR because both edit operator/e2e-quorum.sh and ci-operator.yml.

J4d (F5)

Before this, a leader cut off from PostgreSQL kept leading and every write failed, while a member that could reach the database stood by.

  • DatabaseHandover checks the database on the leader every 5 s.
  • After 15 s without it, once nothing is live (liveRuns() plus faults in flight) or after 2 minutes, it hands leadership to the next member in kates.quorum.peers (QuorumMember.handOver, Ratis transferLeadership on its own client and timeout).
  • It hands over at most once per 2 minutes, so members that all lack the database don't pass leadership back and forth. As the design says, there's no heartbeat between members.

J4e

  • e2e-quorum.sh gains a stage that cuts the leader from PostgreSQL with iptables, then checks that another member leads and answers a write 400, not 503.
  • ci-operator.yml runs nightly (03:17 UTC) and on workflow_dispatch. Those runs also call e2e-node-loss.sh: CI's Kind cluster has three schedulable nodes.
  • Changes under kates/src/main/java/com/bmscomp/kates/quorum/ now start the e2e. The failure dump reads every operator pod's log.

Proof

  • Unit tests:
    • DatabaseHandoverTest: 15 s with nothing live; a live run waits up to 2 minutes; at most once in 2 minutes; the database coming back resets the clock.
    • QuorumMemberTest: three real Ratis members, and the leader hands over to the member it names, in a later term.
  • ./mvnw test: 1938 tests, 0 failures (run with -Dquarkus.grpc.codegen.skip=true). TestMCPCaveatRefAnchors passes.
  • On a three-node Kind cluster, running this image and the new stage taken from the script: "j4-0 leads 27 s after j4-2 lost PostgreSQL" (target 30 s), and the new leader takes writes. The cut leader logged Leading without PostgreSQL; handing over after PT15S, then leadership handed to j4-0.
  • The nightly path (node loss inside CI) hasn't run yet: it first runs after merge, or by hand with workflow_dispatch.

J4d and J4e of plans/j4-resilient-quorum.md.

J4d: the leader checks PostgreSQL every 5 s. After 15 s without it, once
nothing is live or after 2 minutes, it hands leadership to the next member
in kates.quorum.peers with Ratis' transferLeadership, at most once per 2
minutes, so members that all lack the database do not pass it back and
forth. Before, it kept leading and every write failed while a member that
could reach the database stood by (F5).

J4e: e2e-quorum.sh cuts the leader from PostgreSQL with iptables and checks
that another member leads and takes writes. The operator workflow runs
nightly, and then also e2e-node-loss.sh on CI's three-node Kind. Changes
under kates/.../quorum/ start the e2e.
@bmscomp
bmscomp merged commit cb11c41 into main Nov 15, 2026
66 checks passed
bmscomp added a commit that referenced this pull request Nov 16, 2026
…on an empty disk (#679)

* test(operator): print the quorum members' logs when the PostgreSQL handover check fails

The resource is deleted before the workflow's Show the stack step, so a
failed handover left nothing to read. main fails this check on every
run since #675 merged on top of #674.

* fix(quorum): a failed hand-over is retried at the next check, with another member

A leader cut off from PostgreSQL handed over to the next member in
kates.quorum.peers, and a failed transfer counted as a hand-over: the
next try waited 2 minutes. In the e2e that member was the one fenced a
step before; its outbound connections had not recovered, its election
got no votes, and the old leader won the next one. Main has failed the
check on every Operator run since.

A transfer that fails, or after which this member still leads, no
longer starts the 2-minute rest, and the next check, 5 s on, asks the
member after the one that failed.

e2e-node-loss.sh: on two nodes, a follower sharing one gets a new
volume, which the scheduler places on the third, instead of failing.
The anti-affinity is preferred, and a local volume pins a member.

* ci(operator): untaint Kind's control plane for the node-loss test

Kind keeps the control plane's NoSchedule taint when the cluster has
workers, so the three members ran on two nodes and the node-loss test
had no third to spread them over.

* test(operator): print the leader's and the replaced member's logs when it has not caught up

* test(operator): the node-loss test takes a worker without the test database

The test PostgreSQL could share the lost node: every member then lost
it, and the leader handed over by design while the test checked that it
led throughout. The database now prefers the control plane, which the
test never takes, and the lost node is never the database's.

* fix(quorum): a member rebuilt on an empty disk has the leader step down

Ratis 3.3.1's leader keeps the index a member matched before it lost its
disk: it answers the member's INCONSISTENCY nextIndex=0 with heartbeats
from that index, and the member stays at applied index -1 until the
leader changes. A follower whose log is still empty 15 s after it knows
a leader now asks that leader to step down, at most once a minute; the
next leader starts replication over and sends the whole log.

The node-loss e2e checks the leader only while the member is gone, and
gives the catch-up 90 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant