Skip to content

fix(backend): a quorum member whose Ratis server closed comes back, and the database fails fast - #666

Merged
bmscomp merged 6 commits into
mainfrom
fix/quorum-member-revives
Nov 15, 2026
Merged

bmscomp merged 6 commits into
mainfrom
fix/quorum-member-revives

Conversation

@bmscomp

@bmscomp bmscomp commented Nov 15, 2026

Copy link
Copy Markdown
Owner

J4a of the resilient-quorum plan. Two findings from a failure study run on a three-member Kind quorum against main (d464571).

F2: a member paused for more than 60 s was dead for good, and looked alive

Ratis closes its own server after a JVM pause longer than raft.server.close.threshold (60 s). The Kates process lived on, /api/health kept answering the old term and leader, and liveness passed. With a second member frozen as well, the quorum had no leader for about 80 s, and once for about 8 minutes.

  • Quorum's 2 s schedule starts a closed member's server again from its disk. /api/health quorum.state names the life cycle, and the stale fields are left out while the server is down.
  • Liveness goes DOWN only if that restart keeps failing for kates.quorum.liveness-after (30 s).
  • A start that fails closes what it started. Ratis locks the storage directory before it binds, so a failed start used to block every later one.
  • Ratis' System.exit(1) on a bind failure is turned off (ExitUtils.disableSystemExit), so a failed restart is retried instead of killing live runs. In Ratis 3.3.1 that bind failure is the only exit path.

On Kind: before, no leader for 80 s, until liveness killed the frozen leader. After, a leader within 5 s of the thaw, with no pod restart.

F5 (first half): writes hung against an unreachable database

With no JDBC socket timeout, a query on a dead connection waited on TCP, and every write hung.

  • %prod: PgJDBC socketTimeout=30, connectTimeout=5, tcpKeepAlive=true. Agroal: acquisition-timeout=5S, background-validation-interval=10S. The ITs get the same.
  • A database that cannot be reached answers 503 with Retry-After: 5, from GlobalExceptionMapper and from POST /api/tests. That means SQL state 08 or 57P01–57P03, a JDBC connection exception, or Agroal's acquisition timeout.
  • The audit filter skips its row for that one answer and logs it instead. Its retries held the 503 back by another 30 s.
  • The chart's CNPG URL now sets socketTimeout=30 instead of 60, since a URL parameter overrides the backend's own. kates chart 0.19.5.

On Kind, a create against a leader cut from PostgreSQL: before, no answer within the 10 s the test waited (a hang). After, 503 after 35.1 s the first time and 10 s after that. The bound is about 35 s. The follow-up J4f makes the leader hand over when it cannot reach the database.

Tests

  • New: QuorumMemberTest (aClosedServerSaysSo, aMemberWhoseServerClosesRejoinsAsAnotherTerm), QuorumTest (restart by schedule, health while closed, a failed restart that succeeds once the port is free, no restart after shutdown), KatesLivenessCheckTest, GlobalExceptionMapperTest, AuditResponseFilterTest, and DatabaseOutageIT, which pauses the PostgreSQL container and expects the 503 within 35 s.
  • Run locally: the unit suite (1876, 0 failures), all 101 failsafe ITs, CLI go test ./..., the native image build and scripts/native-smoke-test.sh (quorum stage included), Spotless on the changed files, helm lint, and the CNPG template assertions.
  • check-chart-policies.sh and check-rbac-scope.sh fail locally on main too: the local Kyverno v1.20 refuses legacy policies. check-chart-matrix.py needs PyYAML, which is not installed locally. CI runs all three.

Not in this PR

  • J4b–J4g: the Ratis ledger, fencing each fault, the reconciler, schedules firing once, and stepping down when cut off from the database.
  • Deferred minors: one request can answer 500 if Ratis closes the server between the serving check and the call; a shutdown during a restart can leave the new member unclosed as the JVM exits; the 30 s socket timeout also bounds Flyway statements.

…in from its disk

Ratis closes its own server after a JVM pause longer than
raft.server.close.threshold (60 s). The Kates process lived on and kept
answering /api/health with its old term and leader, so the quorum was a
member short until the pod restarted. Quorum's 2 s schedule now starts a
new member on the same disk when the server is no longer running, and
health shows the server's state without the stale fields.

Ratis exits the JVM when a server it starts cannot bind; that is now an
ExitException the next tick retries.
…s closed

The member starts a closed server again itself within seconds; if that
keeps failing for kates.quorum.liveness-after (30 s), liveness goes DOWN
so the kubelet restarts the pod. Liveness reports quorumState and
quorumNotServingSec.
…ead of hanging

With no JDBC socket timeout, a query on a dead connection waited on TCP
for minutes and every write hung with it: a leader cut off from
PostgreSQL answered nothing (s07 of the J4 failure study). PostgreSQL's
socketTimeout 30 s, connectTimeout 5 s and TCP keepalive (prod, and the
ITs), with Agroal's acquisition timeout 5 s and validation every 10 s.

A failure whose cause is the database being unreachable (SQL state 08,
JDBCConnectionException, SQLTransientConnectionException, Agroal's
acquisition timeout) answers 503 with Retry-After 5, from
GlobalExceptionMapper and from POST /api/tests. The audit filter writes
no row for that answer, whose retries held it back another 30 s; it logs
the attempt instead. DatabaseOutageIT pauses PostgreSQL and expects the
503 within 35 s.
…e outage answers 503

The /api/health quorum object's state, what a member reports while its
Ratis server is down, the liveness backstop, and the 503 with Retry-After
for an unreachable database. Three CLI caveat refs re-pointed to the
lines the backend edits moved.
…G outages answer 503 in 30 s

From the branch review:
- QuorumMember.start closes the server when its start fails. Ratis starts
  the division, which locks the storage directory, before it binds, so a
  failed start left a running division and every later start found the
  directory locked.
- SQL states 57P01 to 57P03 (the server shutting down or starting, what a
  CloudNativePG switchover leaves) count as the database being unreachable.
- The chart's CNPG JDBC URL sets socketTimeout=30, the backend's own: a URL
  parameter overrides it, and 60 doubled the 503 bound on the HA path.
  Chart 0.19.5; ci.yml asserts it.
- The MCP audit caveat says a database-outage 503 leaves a log line, not a
  row.
The chart-versions gate: an umbrella that packs a chart whose version
moved carries a new version too.
@bmscomp
bmscomp merged commit 7bbf742 into main Nov 15, 2026
73 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant