Repository navigation
fix(backend): a member that stopped leading injects nothing, and the leader cleans up every 10 s - #668
Merged
Conversation
…ult carries its term and deadline J4b of the resilient-quorum plan. A paused leader that resumed after another member took over injected its plan's partition anyway (s09 of the failure study): leadership was checked when the plan started, not before the fault went in. Both providers now call Quorum.requireWrites() right after the fault's delay, before any write, and refuse with the leader's name. The NetworkPolicy, the ChaosEngine and the scaled StatefulSet or KafkaNodePool carry kates.io/term (the writer's Ratis term) and kates.io/deadline (now plus kates.engine.max-duration-ms), which the reconciler reads next. A scale-down writes them each time, since an earlier plan's snapshot stays on the object with an older term.
…their deadline every 10 s DisruptionOrphanReconciler ran once, at start or takeover, judged by creation time and swallowed its errors. A paused leader's late partition, made after the takeover, was never removed, and nor was one a new leader could not reach the Kubernetes API to delete (s09b, s08b of the failure study). - A pass runs every kates.quorum.reconcile-interval (10 s) on the member that leads, and undoes a Kates fault past its kates.io/deadline or of an older kates.io/term. The age rule stays for the start or takeover pass only. - Kates ChaosEngines are reconciled too, and a node-drain engine's TARGET_NODE is uncordoned. - Scale-downs are restored in every namespace, falling back to the Kafka namespace where listing across namespaces is refused. - Reports are marked INTERRUPTED only in a pass whose deletes all succeeded.
…sed member stops its runs TestOrchestrator's takeover recovery threw when PostgreSQL was down, so a new leader took no write until the database came back, and whether fault cleanup ran depended on CDI observer order (F6 of the failure study). It now logs the failure, and the reconciler retries the recovery on each pass. A member that stops leading is told by Ratis (notifyLeaderChanged on its state machine) and fires LeadershipLost; the orchestrator settles its live runs, so their load stops and their permits come back. The new leader stores them FAILED.
…ciles every 10 s The /api/health quorum paragraph says what J4b changed: the check right before a fault goes in, kates.io/term and kates.io/deadline, the 10 s reconcile pass and what it removes, and a deposed member stopping its runs. The stale-report caveat says when a quorum's leader marks a report INTERRUPTED. 18 CLI refs re-pointed to the lines the backend edits moved.
…n stops when its term ends From the branch review: - requireWrites() returns the term it confirmed, and every object of that fault carries it: read again per object, a paused leader could stamp a fault with the new leader's term and keep it for 2 h. - A plan stops before its next step once the term it started in ends. A member that no longer leads skips the auto-rollback and gives back no replicas at release: the rollback deletes every Kates partition in the namespace, the next leader's too, which undoes this one's by term. - A member that leads again fails its own runs it settled when it lost the lead; a revived member that led says it lost the lead. - A refused NetworkPolicy or ChaosEngine list holds reports back, and the database step runs after the Kubernetes undos in each pass. - The node uncordon is removed: the shipped RBAC cannot patch nodes, and the cordon is Litmus'.
The earlier spotless:apply ran with paths that matched nothing. Six CLI caveat refs follow the lines the formatting moved.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
J4b of the resilient-quorum plan (
plans/j4-resilient-quorum.md, with the Ratis ledger dropped). It fixes six findings from the failure study run on a three-member Kind quorum.What changes
Quorum.requireWrites()after the fault's delay and before any write. It refuses with the leader's name, and returns the term it confirmed.kates.io/term, andkates.io/deadline(now pluskates.engine.max-duration-ms). That covers the NetworkPolicy, the ChaosEngine, and the scaled StatefulSet or KafkaNodePool.DisruptionOrphanReconcilerruns everykates.quorum.reconcile-interval(10 s) on the leader.notifyLeaderChanged, or its own server being restarted after a long pause) and settles its runs; the new leader stores them FAILED.Evidence on Kind (branch image, before the review fixes)
leader:falseThe review fixes (the term returned by the fence, plans stopped when their term ends, deposed rollbacks skipped, regained runs recovered, a failed list holding reports back) are covered by unit tests, each failing before its fix. They were not re-run on Kind.
Cut on purpose
kates.io/fault-id: nothing reads it.Tests
go test ./...;scripts/native-smoke-test.sh, including the quorum stage;DisruptionOrchestratorQuorumTest;strayRuletable and seven behaviour tests inDisruptionOrphanReconcilerTest;TestOrchestratorQuorumTest(takeover with the database down, a deposed member, a regained member);QuorumTest(requireWrites, a revived leader);QuorumMemberTest(the leader-change signal);DisruptionConcurrencyGuardScaleDownTest.Not in this PR