Skip to content

feat(operator): a quorum member whose node is gone gets a new volume on another node - #672

Merged
bmscomp merged 1 commit into
mainfrom
feat/operator-replace-member
Nov 15, 2026
Merged

bmscomp merged 1 commit into
mainfrom
feat/operator-replace-member

Conversation

@bmscomp

@bmscomp bmscomp commented Nov 15, 2026

Copy link
Copy Markdown
Owner

O6 of plans/operator-kates-kind.md: replace a member whose node is gone.

Why no membership API

Members are identified by pod name, and each is reached through a Service of its own, so a member that comes back on a new disk has the same id and the same address. Ratis brings it up to date from the leader. Scaling (O5) is the only thing that needs membership changes, and it is not part of this PR.

What changes

  • Node lost, member replaced. A member pod that has been Pending for 2 minutes, because its volume can only be on another node, has its claim and its pod deleted. The StatefulSet then makes both again on a node that exists.
    • One member at a time, and only while every other member follows a leader and has caught up. The new member starts with an empty log and does not vote until it catches up.
    • With no leader, nothing is replaced, and Ready=False with reason MemberLost says why.
  • Pod waiting on a deleted claim. A pending pod whose claim is gone or being deleted is itself deleted, so it doesn't wait on the old claim.
  • Spread. Members prefer one node each (preferred anti-affinity on kubernetes.io/hostname). This only places new volumes: members that already share a node with node-local storage stay there until their volume is replaced.
  • RBAC. The operator can now get and delete persistentvolumeclaims, in both RBAC scopes.
  • README.
    • Describes node loss, including that a drained node holding a member, unschedulable for 2 minutes, also gets that member moved.
    • Gives the manual recovery for a quorum that lost its majority: delete every member's claim, then their pods. After that, faults the old quorum left are removed only when their deadline passes, because terms count from the start again.

The lost member is detected from the scheduler's message, in both its old and new wording (marked shortcut:). The operator has no rights to read Nodes or PersistentVolumes, and this PR doesn't add any.

Proof

  • Unit tests:
    • lost(): replaced only under a settled leader; two lost of five handled one at a time; waits while another member catches up; no leader means nothing is done.
    • Both scheduler wordings are recognised.
    • The spread rule is set.
  • operator/e2e-node-loss.sh on a Kind cluster with three nodes: a worker node holding a follower and its volume was stopped and deleted. The member ran on another node with a new volume 184 s later, the leader never changed, the quorum was Ready with three members, and the new member caught up (applied index 162).
  • In an earlier run, two members shared the deleted node, so the majority was lost. The operator replaced nothing and reported the quorum has no leader to replace it under.
  • The script isn't in CI, because CI's Kind cluster has one node. It belongs in J4e's nightly suite.

…on another node

O6 of plans/operator-kates-kind.md. Members are pod names, reached through a
Service each, so a replacement needs no membership change: the same member
comes back on an empty disk and catches up from the leader.

- A member pod Pending because its volume can only be on another node, for
  two minutes, has its claim and its pod deleted; the StatefulSet makes both
  again. One at a time, and only while every other member follows a leader
  and has caught up. With no leader nothing is replaced; Ready=False reason
  MemberLost says why.
- A pending pod whose claim is gone or going is deleted, so it does not wait
  on the old claim.
- Members prefer one node each (preferred pod anti-affinity on hostname), so
  a lost node takes one member, not the majority.
- The chart lets the operator get and delete PersistentVolumeClaims in the
  namespaces it watches, in both RBAC scopes.
- operator/e2e-node-loss.sh deletes a Kind worker node holding a follower and
  its volume. Not in CI, whose Kind has one node.
@bmscomp
bmscomp merged commit 6ccddc0 into main Nov 15, 2026
62 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant