Skip to content

fix(backend): a member that stopped leading injects nothing, and the leader cleans up every 10 s - #668

Merged
bmscomp merged 6 commits into
mainfrom
fix/quorum-fence-faults
Nov 15, 2026
Merged

bmscomp merged 6 commits into
mainfrom
fix/quorum-fence-faults

Conversation

@bmscomp

@bmscomp bmscomp commented Nov 15, 2026

Copy link
Copy Markdown
Owner

J4b of the resilient-quorum plan (plans/j4-resilient-quorum.md, with the Ratis ledger dropped). It fixes six findings from the failure study run on a three-member Kind quorum.

What changes

  • No fault from a member that does not lead (F1, F1b).
    • Both chaos providers call Quorum.requireWrites() after the fault's delay and before any write. It refuses with the leader's name, and returns the term it confirmed.
    • Every object the fault writes carries that term as kates.io/term, and kates.io/deadline (now plus kates.engine.max-duration-ms). That covers the NetworkPolicy, the ChaosEngine, and the scaled StatefulSet or KafkaNodePool.
    • A scale-down writes both each time, since an earlier plan's snapshot stays on the object with an older term.
  • A plan belongs to the term it started in. It stops before its next step once that term ends. A member that no longer leads skips the auto-rollback and gives back no replicas when it releases the lock, because that rollback deletes every Kates partition in the namespace, the next leader's included.
  • The leader keeps reconciling (F4, F7, F8).
    • DisruptionOrphanReconciler runs every kates.quorum.reconcile-interval (10 s) on the leader.
    • It removes any Kates fault of an older term or past its deadline: NetworkPolicies, ChaosEngines, and scale-downs in every namespace, falling back to the Kafka namespace on a 403. The age rule stays for the start or takeover pass.
    • A report becomes INTERRUPTED only in a pass whose lists and deletes all succeeded.
    • The database step runs after the Kubernetes undos.
  • Takeover no longer waits on PostgreSQL (F6).
    • A failure of the run recovery is logged, and retried every pass.
    • A member that stops leading learns it from Ratis (notifyLeaderChanged, or its own server being restarted after a long pause) and settles its runs; the new leader stores them FAILED.
    • A member that leads again also fails its own runs it no longer runs.

Evidence on Kind (branch image, before the review fixes)

Scenario Before After
s09b: leader frozen before its fault, thawed after a new leader the partition went in from a member at leader:false nothing went in
s08b: new leader cut from the Kubernetes API at takeover the partition was still there 3 min after the API returned gone 10 s after
s02: PostgreSQL down when the leader dies the report stayed RUNNING for good INTERRUPTED about 20 s after PostgreSQL returned
s10, s11, s15: leader kills, and a follower restart nothing left behind unchanged, and the leader's live partition stays

The review fixes (the term returned by the fence, plans stopped when their term ends, deposed rollbacks skipped, regained runs recovered, a failed list holding reports back) are covered by unit tests, each failing before its fix. They were not re-run on Kind.

Cut on purpose

  • No kates.io/fault-id: nothing reads it.
  • No per-plan deadline: the providers never see the plan, and the engine's run bound is used instead.
  • No new reconciler class.
  • No uncordon of a node-drain's node: the shipped RBAC cannot patch nodes, and the cordon is Litmus'.

Tests

  • Local:
    • the unit suite: 1910 run, 0 failures;
    • all 101 failsafe ITs;
    • CLI go test ./...;
    • the native image build and scripts/native-smoke-test.sh, including the quorum stage;
    • Spotless on the whole project.
  • New:
    • DisruptionOrchestratorQuorumTest;
    • the strayRule table and seven behaviour tests in DisruptionOrphanReconcilerTest;
    • the fence and marks tests in both provider tests;
    • TestOrchestratorQuorumTest (takeover with the database down, a deposed member, a regained member);
    • QuorumTest (requireWrites, a revived leader);
    • QuorumMemberTest (the leader-change signal);
    • DisruptionConcurrencyGuardScaleDownTest.
  • CLI: 40 caveat refs re-pointed to the lines that moved.

Not in this PR

  • J4c: schedules fire once per due time.
  • J4d: the leader hands over when cut off from the database.
  • J4e: the nightly failure suite.
  • Deferred minors:
    • the deadline is based on the engine bound, not the fault's own delay and duration;
    • a stack trace is logged every 10 s while PostgreSQL is down;
    • a log line says "in null" for the every-namespace list.

…ult carries its term and deadline

J4b of the resilient-quorum plan. A paused leader that resumed after
another member took over injected its plan's partition anyway (s09 of
the failure study): leadership was checked when the plan started, not
before the fault went in. Both providers now call Quorum.requireWrites()
right after the fault's delay, before any write, and refuse with the
leader's name.

The NetworkPolicy, the ChaosEngine and the scaled StatefulSet or
KafkaNodePool carry kates.io/term (the writer's Ratis term) and
kates.io/deadline (now plus kates.engine.max-duration-ms), which the
reconciler reads next. A scale-down writes them each time, since an
earlier plan's snapshot stays on the object with an older term.
…their deadline every 10 s

DisruptionOrphanReconciler ran once, at start or takeover, judged by
creation time and swallowed its errors. A paused leader's late partition,
made after the takeover, was never removed, and nor was one a new leader
could not reach the Kubernetes API to delete (s09b, s08b of the failure
study).

- A pass runs every kates.quorum.reconcile-interval (10 s) on the member
  that leads, and undoes a Kates fault past its kates.io/deadline or of
  an older kates.io/term. The age rule stays for the start or takeover
  pass only.
- Kates ChaosEngines are reconciled too, and a node-drain engine's
  TARGET_NODE is uncordoned.
- Scale-downs are restored in every namespace, falling back to the Kafka
  namespace where listing across namespaces is refused.
- Reports are marked INTERRUPTED only in a pass whose deletes all
  succeeded.
…sed member stops its runs

TestOrchestrator's takeover recovery threw when PostgreSQL was down, so
a new leader took no write until the database came back, and whether
fault cleanup ran depended on CDI observer order (F6 of the failure
study). It now logs the failure, and the reconciler retries the
recovery on each pass.

A member that stops leading is told by Ratis (notifyLeaderChanged on
its state machine) and fires LeadershipLost; the orchestrator settles
its live runs, so their load stops and their permits come back. The new
leader stores them FAILED.
…ciles every 10 s

The /api/health quorum paragraph says what J4b changed: the check right
before a fault goes in, kates.io/term and kates.io/deadline, the 10 s
reconcile pass and what it removes, and a deposed member stopping its
runs. The stale-report caveat says when a quorum's leader marks a report
INTERRUPTED. 18 CLI refs re-pointed to the lines the backend edits moved.
…n stops when its term ends

From the branch review:
- requireWrites() returns the term it confirmed, and every object of
  that fault carries it: read again per object, a paused leader could
  stamp a fault with the new leader's term and keep it for 2 h.
- A plan stops before its next step once the term it started in ends.
  A member that no longer leads skips the auto-rollback and gives back
  no replicas at release: the rollback deletes every Kates partition in
  the namespace, the next leader's too, which undoes this one's by term.
- A member that leads again fails its own runs it settled when it lost
  the lead; a revived member that led says it lost the lead.
- A refused NetworkPolicy or ChaosEngine list holds reports back, and
  the database step runs after the Kubernetes undos in each pass.
- The node uncordon is removed: the shipped RBAC cannot patch nodes,
  and the cordon is Litmus'.
The earlier spotless:apply ran with paths that matched nothing. Six CLI
caveat refs follow the lines the formatting moved.
@bmscomp
bmscomp merged commit 82709ac into main Nov 15, 2026
69 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant