Skip to content

fix(operator): a quorum whose members cannot run one is rolled all at once - #678

Merged
bmscomp merged 1 commit into
mainfrom
fix/operator-roll-without-quorum
Nov 16, 2026
Merged

bmscomp merged 1 commit into
mainfrom
fix/operator-roll-without-quorum

Conversation

@bmscomp

@bmscomp bmscomp commented Nov 16, 2026

Copy link
Copy Markdown
Owner

Found while testing #677 (kates deploy --kates-mode operator).

The deadlock

quorum::next replaced one stale member at a time, and only under a settled leader. A Kates whose members never elect a leader could never be rolled. Two cases hit it:

  • members crash-looping on a bad image;
  • members on an image older than the quorum, whose /api/health has no quorum block.

Changing spec.version to a working image then left Ready=False RollingOut: the next member is replaced once the quorum has no leader for ever, and the only way out was deleting the pods by hand.

The rule

When no member may run a quorum, every stale member is replaced at once (Next::ReplaceAll).

  • "May run a quorum" means the member is Ready and either reports quorum.enabled or doesn't answer.
  • Why it's safe: there's no leader to keep and nothing serving, and a replaced pod keeps its volume, so terms and votes survive.
  • Why not one at a time: the first member back would wait for a majority the others can't give.
  • What keeps the strict rule: a Ready member the operator can't reach may be leading, so the leader rule still applies then. A normal roll (Ready members running a quorum, an election in progress) also keeps the strict rule, unchanged.

Proof

Unit test members_that_cannot_run_a_quorum_are_replaced_together_and_only_then:

  • Pre-quorum members: replaced together.
  • Members that are crashing (not Ready, no answer): replaced together.
  • A Ready member that doesn't answer: the leader rule holds.
  • One member running a quorum: the leader rule holds.

cargo fmt, clippy -D warnings and cargo test are green.

On a three-node Kind cluster with this operator: a three-member Kates with all members crash-looping (kates:native-local, refused by a database the newer image had migrated). After spec.version was set to a working image, the operator recorded replacing kates-0, kates-1, kates-2 with j4d, and the resource was Ready with leader kates-2 14 s after the change. The operator before this change stayed at the message above.

Not covered: rolling a working quorum down to a pre-quorum image still stalls after the second member, because one member that runs a quorum remains. Downgrading below the quorum isn't supported.

… once

The roll replaced one member at a time, and only under a settled leader. A
Kates whose members never elect one (crashing on a bad image, or on an image
older than the quorum) could then never be rolled to a fixed version: the
status stayed "the next member is replaced once the quorum has no leader".

When no member may run a quorum, every stale member is replaced together:
there is no leader to keep and nothing serving, and a replaced pod keeps its
volume. A member may run one when it is Ready and either reports a quorum or
does not answer, so a Ready member the operator cannot reach keeps the leader
rule. One at a time would not do: the first member back would wait for a
majority the others cannot give.
@bmscomp
bmscomp merged commit 9e5d697 into main Nov 16, 2026
62 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant