[austin-barrington]

A cluster formed through the deprecated cluster.peers left every node but the first as a Raft learner, stuck in Syncing, with /health returning 503 indefinitely. There were two causes in raft_formation.rs:

• Promotion waited for every peer. The bootstrap node promoted learners only once every configured peer was reachable and caught up. With ordered startup (a StatefulSet's OrderedReady, or a script), the next node starts only once this one is Ready, and a learner never is. Each learner is now promoted as soon as it has caught up, up to five voters.
• Learners never counted as caught up. Every pass re-proposed each known peer as a learner. That appended a membership entry just before the check that learners had matched the log tail, so none ever had. A peer is now proposed once.

A node's first writes after its /health turned 200 could get an empty 503. Two causes:

• /health reports a node that Raft serves as a voter as Active, even while its own record still says Syncing. /write refused based on the record; it now follows /health.
• A node that had already synced but held no data (a new, empty cluster) was treated as new by the leader's sync trigger and re-joined, which flipped it back to Syncing. It now reconnects, which stays Active for a small gap.

Results, 3 nodes, ordered start:

• Node 2 is healthy after 7 s and node 3 after 11 s, all voters. Before, they stayed learners answering 503 indefinitely.
• Writing straight after /health returns 200, on every node: fresh clusters with a 503 went from 4 of 6 to 1 of 12. The remaining case is a write to a node in the second or two before the leader adds it as a learner; that 503 is retryable, and the proxy retries it.

Unit tests cover the formation planning: a caught-up learner is promoted without waiting for every peer, a lagging one waits, a known learner isn't re-added, and voters stop at five. The CI-equivalent run (libchdb 26.9.0, -D warnings) passes.

[cursor]

PR Summary

Medium Risk
Changes Raft membership promotion, health/write gating, and sync-trigger behavior during cluster bootstrap—core availability paths, though bounded by existing Raft and membership safeguards.

Overview
Fixes Raft cluster formation for ordered startup (cluster.peers): learners are added once and promoted individually when replication catches up (cap 5 voters), instead of waiting for every peer and re-adding learners each loop—which blocked promotion and left nodes stuck as learners with perpetual 503 on /health.

Aligns the data plane with that readiness model: /write no longer returns 503 for Syncing/Joining when Raft already serves the node as Leader/Follower (same rule as /health, via shared raft_is_serving). The sync trigger uses reconnect_sync for nodes that were already Active without needs_sync on an empty cluster, so they are not flipped back to Syncing via join_and_sync.

Adds unit tests for plan_formation (per-learner promotion, no duplicate learner adds, voter cap).

^{Reviewed by Cursor Bugbot for commit 853e6ee. Configure here.}

[cursor]

Approved. Cursor Bugbot completed successfully and reported no findings that require human review. No approval policy requires additional human review on this head.

  

_{Sent by Cursor Approval Agent: Pull Request Router and Approver}