Repository navigation
fix(backend): a quorum member whose Ratis server closed comes back, and the database fails fast - #666
Merged
Merged
Conversation
…in from its disk Ratis closes its own server after a JVM pause longer than raft.server.close.threshold (60 s). The Kates process lived on and kept answering /api/health with its old term and leader, so the quorum was a member short until the pod restarted. Quorum's 2 s schedule now starts a new member on the same disk when the server is no longer running, and health shows the server's state without the stale fields. Ratis exits the JVM when a server it starts cannot bind; that is now an ExitException the next tick retries.
…s closed The member starts a closed server again itself within seconds; if that keeps failing for kates.quorum.liveness-after (30 s), liveness goes DOWN so the kubelet restarts the pod. Liveness reports quorumState and quorumNotServingSec.
…ead of hanging With no JDBC socket timeout, a query on a dead connection waited on TCP for minutes and every write hung with it: a leader cut off from PostgreSQL answered nothing (s07 of the J4 failure study). PostgreSQL's socketTimeout 30 s, connectTimeout 5 s and TCP keepalive (prod, and the ITs), with Agroal's acquisition timeout 5 s and validation every 10 s. A failure whose cause is the database being unreachable (SQL state 08, JDBCConnectionException, SQLTransientConnectionException, Agroal's acquisition timeout) answers 503 with Retry-After 5, from GlobalExceptionMapper and from POST /api/tests. The audit filter writes no row for that answer, whose retries held it back another 30 s; it logs the attempt instead. DatabaseOutageIT pauses PostgreSQL and expects the 503 within 35 s.
…e outage answers 503 The /api/health quorum object's state, what a member reports while its Ratis server is down, the liveness backstop, and the 503 with Retry-After for an unreachable database. Three CLI caveat refs re-pointed to the lines the backend edits moved.
…G outages answer 503 in 30 s From the branch review: - QuorumMember.start closes the server when its start fails. Ratis starts the division, which locks the storage directory, before it binds, so a failed start left a running division and every later start found the directory locked. - SQL states 57P01 to 57P03 (the server shutting down or starting, what a CloudNativePG switchover leaves) count as the database being unreachable. - The chart's CNPG JDBC URL sets socketTimeout=30, the backend's own: a URL parameter overrides it, and 60 doubled the 503 bound on the HA path. Chart 0.19.5; ci.yml asserts it. - The MCP audit caveat says a database-outage 503 leaves a log line, not a row.
The chart-versions gate: an umbrella that packs a chart whose version moved carries a new version too.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
J4a of the resilient-quorum plan. Two findings from a failure study run on a three-member Kind quorum against
main(d464571).F2: a member paused for more than 60 s was dead for good, and looked alive
Ratis closes its own server after a JVM pause longer than
raft.server.close.threshold(60 s). The Kates process lived on,/api/healthkept answering the old term and leader, and liveness passed. With a second member frozen as well, the quorum had no leader for about 80 s, and once for about 8 minutes.Quorum's 2 s schedule starts a closed member's server again from its disk./api/healthquorum.statenames the life cycle, and the stale fields are left out while the server is down.kates.quorum.liveness-after(30 s).System.exit(1)on a bind failure is turned off (ExitUtils.disableSystemExit), so a failed restart is retried instead of killing live runs. In Ratis 3.3.1 that bind failure is the only exit path.On Kind: before, no leader for 80 s, until liveness killed the frozen leader. After, a leader within 5 s of the thaw, with no pod restart.
F5 (first half): writes hung against an unreachable database
With no JDBC socket timeout, a query on a dead connection waited on TCP, and every write hung.
%prod: PgJDBCsocketTimeout=30,connectTimeout=5,tcpKeepAlive=true. Agroal:acquisition-timeout=5S,background-validation-interval=10S. The ITs get the same.Retry-After: 5, fromGlobalExceptionMapperand fromPOST /api/tests. That means SQL state 08 or 57P01–57P03, a JDBC connection exception, or Agroal's acquisition timeout.socketTimeout=30instead of 60, since a URL parameter overrides the backend's own.kateschart 0.19.5.On Kind, a create against a leader cut from PostgreSQL: before, no answer within the 10 s the test waited (a hang). After, 503 after 35.1 s the first time and 10 s after that. The bound is about 35 s. The follow-up J4f makes the leader hand over when it cannot reach the database.
Tests
QuorumMemberTest(aClosedServerSaysSo,aMemberWhoseServerClosesRejoinsAsAnotherTerm),QuorumTest(restart by schedule, health while closed, a failed restart that succeeds once the port is free, no restart after shutdown),KatesLivenessCheckTest,GlobalExceptionMapperTest,AuditResponseFilterTest, andDatabaseOutageIT, which pauses the PostgreSQL container and expects the 503 within 35 s.go test ./..., the native image build andscripts/native-smoke-test.sh(quorum stage included), Spotless on the changed files,helm lint, and the CNPG template assertions.check-chart-policies.shandcheck-rbac-scope.shfail locally onmaintoo: the local Kyverno v1.20 refuses legacy policies.check-chart-matrix.pyneeds PyYAML, which is not installed locally. CI runs all three.Not in this PR