Skip to content

fix(quorum): retry a failed hand-over, and catch up a member rebuilt on an empty disk - #679

Merged
bmscomp merged 7 commits into
mainfrom
fix/quorum-handover-retry
Nov 16, 2026
Merged

bmscomp merged 7 commits into
mainfrom
fix/quorum-handover-retry

Conversation

@bmscomp

@bmscomp bmscomp commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

main's CI · Operator E2E has failed on every run since #674/#675. With #676, this fixes the two quorum bugs behind it and the nightly test setup.

Hand-over without PostgreSQL (J4d)

#676 now tries each other member in turn and retries a failed hand-over at the next check. On top of it, this PR also counts a transfer as failed when this member leads again afterwards (its target lost the election), so the next check retries instead of resting 2 minutes.

A member rebuilt on an empty disk (O6, #672)

Ratis 3.3.1's leader keeps the index a member matched before it lost its disk. It answers the member's INCONSISTENCY nextIndex=0 with heartbeats only, so the member stays at applied index -1 until the leader changes. With the new EmptyLogCatchUp check, a follower whose log is still empty 15 s after it knows a leader asks that leader to step down, at most once a minute. The next leader sends it the whole log. QuorumMemberTest reproduces the stuck state, then the catch-up.

Nightly node-loss test

  • The workflow removes the NoSchedule taint from Kind's control plane alpha, so the three members can land on three nodes.
  • The test database prefers the control plane. The node-loss victim avoids the database's node: losing PostgreSQL makes the leader hand over by design.
  • The leader is checked only while the victim is gone, and the catch-up gets 90 s.
  • The handover and catch-up checks print the members' logs when they fail.

Tests

bmscomp and others added 7 commits November 16, 2026 01:09
…ndover check fails

The resource is deleted before the workflow's Show the stack step, so a
failed handover left nothing to read. main fails this check on every
run since #675 merged on top of #674.
…other member

A leader cut off from PostgreSQL handed over to the next member in
kates.quorum.peers, and a failed transfer counted as a hand-over: the
next try waited 2 minutes. In the e2e that member was the one fenced a
step before; its outbound connections had not recovered, its election
got no votes, and the old leader won the next one. Main has failed the
check on every Operator run since.

A transfer that fails, or after which this member still leads, no
longer starts the 2-minute rest, and the next check, 5 s on, asks the
member after the one that failed.

e2e-node-loss.sh: on two nodes, a follower sharing one gets a new
volume, which the scheduler places on the third, instead of failing.
The anti-affinity is preferred, and a local volume pins a member.
Kind keeps the control plane's NoSchedule taint when the cluster has
workers, so the three members ran on two nodes and the node-loss test
had no third to spread them over.
…tabase

The test PostgreSQL could share the lost node: every member then lost
it, and the leader handed over by design while the test checked that it
led throughout. The database now prefers the control plane, which the
test never takes, and the lost node is never the database's.
Ratis 3.3.1's leader keeps the index a member matched before it lost its
disk: it answers the member's INCONSISTENCY nextIndex=0 with heartbeats
from that index, and the member stays at applied index -1 until the
leader changes. A follower whose log is still empty 15 s after it knows
a leader now asks that leader to step down, at most once a minute; the
next leader starts replication over and sends the whole log.

The node-loss e2e checks the leader only while the member is gone, and
gives the catch-up 90 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…etry

# Conflicts:
#	kates/src/main/java/com/bmscomp/kates/quorum/DatabaseHandover.java
#	kates/src/main/java/com/bmscomp/kates/quorum/Quorum.java
#	operator/e2e-quorum.sh
@bmscomp
bmscomp merged commit 360f5f9 into main Nov 16, 2026
66 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant