Repository navigation
Transfer Keeper leadership before restarting the leader during rolling updates #346
Description
Company or project name
Gen-AX
Use case
We run a 3-node KeeperCluster and use the operator to perform rolling updates for configuration and image changes.
When the current leader is restarted, a new leader is elected only after the old leader stops. This creates a short period without a leader.
In our test, the operator reported No leader for about 5–6 seconds. During the same period, ClickHouse logged Keeper connection loss and session expired errors, and INSERT p95 latency increased from about 112–167 ms to 6.3 s.
Describe the solution you'd like
Before restarting a Keeper replica that is currently the leader, transfer or yield leadership to another healthy replica and wait until a new leader is confirmed.
Keeper provides commands such as ydld and rqld, which may be usable for this.
Describe alternatives you've considered
We considered adding a preStop hook that yields leadership before the Keeper process stops.
However, KeeperCluster v0.0.7 does not expose lifecycle / preStop through podTemplate or containerTemplate, so this cannot currently be configured directly.
A mutating admission webhook could inject the hook, but handling this in the operator would avoid requiring an additional component.
Additional context
No response
Activity
melancholictheory commentedon Sep 24, 2026
Update: #347 is closed in favour of #351 from @mu-soliman, which is the fuller shape of the same idea: a
preStopscript that sendsydldand then waits for/readyto stop reporting the replica as leader, plus acontainerTemplate.lifecycleoverride for distroless images.Two things from measuring this on a 3-node Keeper 26.7.5.10 that are worth knowing regardless of which PR lands. Under a heavy write load, sending
ydldand stopping the container at once lost the race in one of two runs and fell back to a timeout election (1.5 s); waiting for the handover first costs 60 to 110 ms and kept the switch under 0.25 s every time. Without any hook, the leaderless window under the same load reached 9 s. Details and the NuRaft reasoning are in the #347 thread.One thing worth knowing if you script this yourself: the reply
Sent yield leadership request to leader.is returned even when you send the command to a follower, so it cannot be used as confirmation;/readyormntrcan.feat(keeper): hand leadership over before restarting the leader
feat(keeper): hand leadership over before a leader pod stops
feat(keeper): hand leadership over before a leader pod stops
feat(keeper): hand leadership over before a leader pod stops