Skip to content

fix(persist): bound configuration consensus leader-route rediscovery - #1467

Merged
VerifiedOrganic merged 1 commit into
backlogfrom
fix/1049-config-leader-rediscovery
Oct 25, 2026
Merged

VerifiedOrganic merged 1 commit into
backlogfrom
fix/1049-config-leader-rediscovery

Conversation

@VerifiedOrganic

@VerifiedOrganic VerifiedOrganic commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A configuration-consensus call routed to a lost leader waited for the complete
operation deadline even after this replica's own engine had already named the
successor. A black-holed former leader could therefore hold a configuration
write or linearizable read past the documented stall bound (#1049).

Change

Every leader-routed forwarded mutation (append_commit_idempotent and the other
ConfigStore mutation paths) and linearizable read barrier is now bounded by
this replica's own replicated leader view:

  • Superseded before send. When the local engine already names a different
    member as leader, the call is not transmitted; the caller retries against the
    observed leader immediately.
  • Route retirement. When the engine names a successor while a call to the
    attempted leader is outstanding, the unanswered call is abandoned after
    DURABLE_CONSENSUS_STALE_LEADER_ROUTE_GRACE (200 ms) and re-offered through the
    observed successor. A view change during the grace re-evaluates the route
    instead of abandoning it blindly.
  • Fail closed at the unchanged bound. Without an observed successor the
    existing complete-operation deadline still applies, and mutations return the
    existing typed OutcomeUnknown instead of hanging.
  • Exactly-once retry. An abandoned forwarded mutation is classified as
    possibly transmitted; the retry reuses the same request identity and the
    configuration state machine's config_raft_request_outcomes idempotency
    returns the already-committed response instead of applying a second effect.
  • No timing changes. Election, heartbeat, forwarded-mutation and operation
    budgets are untouched. Engine-owned handoff and member-repair protocols keep
    their exact-target unbounded-route call.

The 200 ms grace is a new standalone shared constant in
opc-consensus::profile (DURABLE_CONSENSUS_STALE_LEADER_ROUTE_GRACE), not a
field of DURABLE_CONSENSUS_TIMING_PROFILE, so every existing profile
validation and qualification envelope is unchanged. opc-consensus README and
CHANGELOG.md document the bound.

Tests

New serial all-features consensus e2e tests
(crates/opc-persist/tests/consensus_openraft/route_retirement.rs):

  • routed_write_rediscovers_the_successor_after_a_black_holed_lost_leader —
    black-holes the follower-to-leader forward route, confirms the forwarded
    mutation was transmitted, then loses the leader and elects a successor while
    the call is outstanding. Before the write, the paused clock is advanced
    through the survivors' exact committed-vote leases, so the deterministic
    replacement campaign has no randomized wait and fits the fixture's 3 s
    declared operation budget. Asserts the write commits exactly once through the
    successor within grace + one forwarded round trip, that it stays inside the
    declared operation budget, that the abandoned lost-leader route is not
    retried, and that replaying the same request identity returns the same single
    revision.
  • unanswered_forward_without_a_successor_fails_closed_at_the_operation_bound —
    an unanswered forward with a healthy leader and no observed successor keeps the
    unchanged complete operation bound and returns typed OutcomeUnknown.

Existing tests cover the normal forwarded-mutation and lost-response replay
paths and remain unchanged.

Review round 1 (V4-Pro C3) fixes

  • HIGH — wrong operation budget in the no-successor regression: the fixture
    builds stores with a declared 3 s operation timeout, not the 10 s profile
    constant. The test now asserts against the declared budget (mirroring the
    existing DECLARED/TEST_CEILING convention in consensus_openraft.rs), and
    its outer timeout is tightened from 11 s to the declared budget plus one
    re-offer allowance. The leader-rediscovery regression was also reworked
    against the same budget: the paused clock is advanced through the survivors'
    committed-vote leases before the write, so the triggered replacement
    campaign wins immediately instead of waiting a randomized election timeout
    that no longer fits the fixture budget.
  • MEDIUM — soft 200 ms grace under metrics churn: the grace deadline is now
    absolute per observed successor. metrics.changed() wakes re-evaluate the
    leader view but do not re-arm the grace; only an actual current_leader
    change re-arms it. The hard "abandoned after the grace" claim now holds under
    busy replication.
  • LOW — upper-bound slack: the re-offer allowance is tightened from 1 s to
    500 ms (one decoded forwarded round trip plus at most one stale-route refresh
    backoff). It is still an allowance, not an exact equality, because the
    re-offer path is not guaranteed to consume zero protocol time.

Verification (macOS analysis host)

  • cargo fmt --all -- --check — pass.
  • cargo clippy --locked -p opc-persist --all-targets --all-features -- -D warnings -A clippy::nonminimal_bool — pass (the allowed lint is a pre-existing nightly-only finding in untouched consensus/types.rs; CI's stable toolchain does not raise it).
  • cargo clippy --locked -p opc-persist --all-targets --no-default-features -- -D warnings -A clippy::nonminimal_bool — pass (covers the CI "opc-persist default test contract" clippy lane; the new test module is feature-gated so it compiles out there).
  • cargo test --locked -p opc-persist --no-run — pass (default-feature contract lane compiles).
  • cargo test -p opc-persist --all-features --test consensus_openraft --no-run — pass.
  • cargo test -p opc-consensus --lib — 43 passed (includes the new exact grace/profile assertions).
  • Focused E2E execution is Linux-only: running
    cargo test -p opc-persist --all-features --test consensus_openraft route_retirement::
    on this macOS host fails in fixture setup (consensus_openraft.rs:562
    .expect("store"), store: StorageUnavailable) before any assertion. The
    same pre-existing failure occurs for unmodified existing fixtures in this
    binary (for example lost_commit_ack_is_outcome_unknown_and_same_id_replays_after_leader_loss),
    because consensus fixture storage opening is Linux-host-only. Hosted Linux CI
    is therefore the execution gate for these two regressions.

Limitations / merge gate

  • The functional cluster fixtures cannot run on this macOS host. A pre-existing environment limitation fails consensus store open with StorageUnavailable for every existing cluster fixture in this test target as well; it is not caused by this change. Hosted Linux CI (cargo test --locked -p opc-persist --all-features --quiet -- --test-threads=1) is the merge gate for the two new tests.
  • The grace constant is intentionally shared in opc-consensus, but only the configuration-consensus adapter consumes it in this PR. Session-store route retirement remains part of Durable consensus: faster, predictable failover on an unplanned leader loss #1037/fix(consensus): fail over within one operation after an unplanned leader loss #1046 and is not changed here.
  • The bounded retry relies on the existing configuration request-identity idempotency; no new durable format or migration is introduced.
  • C3-class change: independent review should focus on the abandonment/ambiguity classification and the interaction with the existing leader-hint retry loop.

Refs #1049

@VerifiedOrganic
VerifiedOrganic force-pushed the fix/1049-config-leader-rediscovery branch 2 times, most recently from 0eec1ff to 491f20a Compare October 25, 2026 05:38
A configuration-consensus call routed to a lost leader waited for the
complete operation deadline even after this replica's own engine had
already named the successor, so a black-holed former leader held the
write past the documented stall bound.

Callers now bound each leader-routed forwarded mutation and linearizable
read barrier by their own replicated leader view:

- A route this replica has already superseded is not transmitted.
- Once the engine names a successor, an unanswered call is abandoned
  after DURABLE_CONSENSUS_STALE_LEADER_ROUTE_GRACE (200 ms) and
  re-offered through the observed successor.
- The grace deadline is absolute per observed successor: unrelated
  metrics churn (replication progress, applied index, term) never
  re-arms it, and only an actual current_leader change does.
- A route with no observed successor keeps the existing complete
  operation deadline and returns the existing typed ambiguous outcome.
- An abandoned forwarded mutation is classified as possibly transmitted;
  the retry reuses the same request identity, so the configuration
  state machine's request-outcome idempotency keeps it exactly-once.

Election, heartbeat and forwarded-mutation timing budgets are unchanged;
the grace is a separate route-retirement bound in the shared timing
crate. Engine-owned handoff and member-repair calls keep the unbounded
exact-target route.

Tests (serial all-features consensus e2e lane, 3 s fixture operation
budget):
- routed write black-holes the lost leader's forward route, loses the
  leader while the call is outstanding, and asserts the write commits
  once through the successor within grace + one forwarded re-offer while
  staying inside the fixture's declared operation budget. The paused
  clock is advanced through the survivors' committed-vote leases first,
  so the deterministic replacement campaign has no randomized wait.
- an unanswered forward with no observed successor fails closed with
  OutcomeUnknown at the fixture's declared operation budget.

Review round (V4-Pro C3): assert against the fixture's declared 3 s
operation budget instead of the 10 s profile constant and tighten both
outer timeouts; make the stale-leader grace absolute across metrics
churn; tighten the re-offer allowance to one forwarded round trip plus
one route-refresh backoff.

The functional cluster fixtures cannot run on the macOS analysis host
(pre-existing consensus store-open StorageUnavailable); Linux CI is the
merge gate.

Refs #1049
Signed-off-by: VerifiedOrganic <verifiedorganic@sent.com>
@VerifiedOrganic
VerifiedOrganic force-pushed the fix/1049-config-leader-rediscovery branch from 491f20a to eba7469 Compare October 25, 2026 05:47
@VerifiedOrganic
VerifiedOrganic merged commit f43f2bc into backlog Oct 25, 2026
7 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant