Repository navigation
fix(persist): bound configuration consensus leader-route rediscovery - #1467
Merged
Merged
Conversation
VerifiedOrganic
force-pushed
the
fix/1049-config-leader-rediscovery
branch
2 times, most recently
from
October 25, 2026 05:38
0eec1ff to
491f20a
Compare
A configuration-consensus call routed to a lost leader waited for the complete operation deadline even after this replica's own engine had already named the successor, so a black-holed former leader held the write past the documented stall bound. Callers now bound each leader-routed forwarded mutation and linearizable read barrier by their own replicated leader view: - A route this replica has already superseded is not transmitted. - Once the engine names a successor, an unanswered call is abandoned after DURABLE_CONSENSUS_STALE_LEADER_ROUTE_GRACE (200 ms) and re-offered through the observed successor. - The grace deadline is absolute per observed successor: unrelated metrics churn (replication progress, applied index, term) never re-arms it, and only an actual current_leader change does. - A route with no observed successor keeps the existing complete operation deadline and returns the existing typed ambiguous outcome. - An abandoned forwarded mutation is classified as possibly transmitted; the retry reuses the same request identity, so the configuration state machine's request-outcome idempotency keeps it exactly-once. Election, heartbeat and forwarded-mutation timing budgets are unchanged; the grace is a separate route-retirement bound in the shared timing crate. Engine-owned handoff and member-repair calls keep the unbounded exact-target route. Tests (serial all-features consensus e2e lane, 3 s fixture operation budget): - routed write black-holes the lost leader's forward route, loses the leader while the call is outstanding, and asserts the write commits once through the successor within grace + one forwarded re-offer while staying inside the fixture's declared operation budget. The paused clock is advanced through the survivors' committed-vote leases first, so the deterministic replacement campaign has no randomized wait. - an unanswered forward with no observed successor fails closed with OutcomeUnknown at the fixture's declared operation budget. Review round (V4-Pro C3): assert against the fixture's declared 3 s operation budget instead of the 10 s profile constant and tighten both outer timeouts; make the stale-leader grace absolute across metrics churn; tighten the re-offer allowance to one forwarded round trip plus one route-refresh backoff. The functional cluster fixtures cannot run on the macOS analysis host (pre-existing consensus store-open StorageUnavailable); Linux CI is the merge gate. Refs #1049 Signed-off-by: VerifiedOrganic <verifiedorganic@sent.com>
VerifiedOrganic
force-pushed
the
fix/1049-config-leader-rediscovery
branch
from
October 25, 2026 05:47
491f20a to
eba7469
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A configuration-consensus call routed to a lost leader waited for the complete
operation deadline even after this replica's own engine had already named the
successor. A black-holed former leader could therefore hold a configuration
write or linearizable read past the documented stall bound (#1049).
Change
Every leader-routed forwarded mutation (
append_commit_idempotentand the otherConfigStoremutation paths) and linearizable read barrier is now bounded bythis replica's own replicated leader view:
member as leader, the call is not transmitted; the caller retries against the
observed leader immediately.
attempted leader is outstanding, the unanswered call is abandoned after
DURABLE_CONSENSUS_STALE_LEADER_ROUTE_GRACE(200 ms) and re-offered through theobserved successor. A view change during the grace re-evaluates the route
instead of abandoning it blindly.
existing complete-operation deadline still applies, and mutations return the
existing typed
OutcomeUnknowninstead of hanging.possibly transmitted; the retry reuses the same request identity and the
configuration state machine's
config_raft_request_outcomesidempotencyreturns the already-committed response instead of applying a second effect.
budgets are untouched. Engine-owned handoff and member-repair protocols keep
their exact-target unbounded-route call.
The 200 ms grace is a new standalone shared constant in
opc-consensus::profile(DURABLE_CONSENSUS_STALE_LEADER_ROUTE_GRACE), not afield of
DURABLE_CONSENSUS_TIMING_PROFILE, so every existing profilevalidation and qualification envelope is unchanged.
opc-consensusREADME andCHANGELOG.mddocument the bound.Tests
New serial all-features consensus e2e tests
(
crates/opc-persist/tests/consensus_openraft/route_retirement.rs):routed_write_rediscovers_the_successor_after_a_black_holed_lost_leader—black-holes the follower-to-leader forward route, confirms the forwarded
mutation was transmitted, then loses the leader and elects a successor while
the call is outstanding. Before the write, the paused clock is advanced
through the survivors' exact committed-vote leases, so the deterministic
replacement campaign has no randomized wait and fits the fixture's 3 s
declared operation budget. Asserts the write commits exactly once through the
successor within
grace + one forwarded round trip, that it stays inside thedeclared operation budget, that the abandoned lost-leader route is not
retried, and that replaying the same request identity returns the same single
revision.
unanswered_forward_without_a_successor_fails_closed_at_the_operation_bound—an unanswered forward with a healthy leader and no observed successor keeps the
unchanged complete operation bound and returns typed
OutcomeUnknown.Existing tests cover the normal forwarded-mutation and lost-response replay
paths and remain unchanged.
Review round 1 (V4-Pro C3) fixes
builds stores with a declared 3 s operation timeout, not the 10 s profile
constant. The test now asserts against the declared budget (mirroring the
existing
DECLARED/TEST_CEILINGconvention inconsensus_openraft.rs), andits outer timeout is tightened from 11 s to the declared budget plus one
re-offer allowance. The leader-rediscovery regression was also reworked
against the same budget: the paused clock is advanced through the survivors'
committed-vote leases before the write, so the triggered replacement
campaign wins immediately instead of waiting a randomized election timeout
that no longer fits the fixture budget.
absolute per observed successor.
metrics.changed()wakes re-evaluate theleader view but do not re-arm the grace; only an actual
current_leaderchange re-arms it. The hard "abandoned after the grace" claim now holds under
busy replication.
500 ms (one decoded forwarded round trip plus at most one stale-route refresh
backoff). It is still an allowance, not an exact equality, because the
re-offer path is not guaranteed to consume zero protocol time.
Verification (macOS analysis host)
cargo fmt --all -- --check— pass.cargo clippy --locked -p opc-persist --all-targets --all-features -- -D warnings -A clippy::nonminimal_bool— pass (the allowed lint is a pre-existing nightly-only finding in untouchedconsensus/types.rs; CI's stable toolchain does not raise it).cargo clippy --locked -p opc-persist --all-targets --no-default-features -- -D warnings -A clippy::nonminimal_bool— pass (covers the CI "opc-persist default test contract" clippy lane; the new test module is feature-gated so it compiles out there).cargo test --locked -p opc-persist --no-run— pass (default-feature contract lane compiles).cargo test -p opc-persist --all-features --test consensus_openraft --no-run— pass.cargo test -p opc-consensus --lib— 43 passed (includes the new exact grace/profile assertions).cargo test -p opc-persist --all-features --test consensus_openraft route_retirement::on this macOS host fails in fixture setup (
consensus_openraft.rs:562.expect("store"),store: StorageUnavailable) before any assertion. Thesame pre-existing failure occurs for unmodified existing fixtures in this
binary (for example
lost_commit_ack_is_outcome_unknown_and_same_id_replays_after_leader_loss),because consensus fixture storage opening is Linux-host-only. Hosted Linux CI
is therefore the execution gate for these two regressions.
Limitations / merge gate
StorageUnavailablefor every existing cluster fixture in this test target as well; it is not caused by this change. Hosted Linux CI (cargo test --locked -p opc-persist --all-features --quiet -- --test-threads=1) is the merge gate for the two new tests.opc-consensus, but only the configuration-consensus adapter consumes it in this PR. Session-store route retirement remains part of Durable consensus: faster, predictable failover on an unplanned leader loss #1037/fix(consensus): fail over within one operation after an unplanned leader loss #1046 and is not changed here.Refs #1049