Skip to content

docs: add chain_split_halt polish design - #4812

Draft
OisinKyne wants to merge 1 commit into
mainfrom
oisin/chain-split-halt-design
Draft

OisinKyne wants to merge 1 commit into
mainfrom
oisin/chain-split-halt-design

Conversation

@OisinKyne

Copy link
Copy Markdown
Contributor

Adds docs/design/chain-split-halt.md, a design doc and plan for polishing the chain_split_halt feature. It was prompted by its first real chain split: dvt-c3 split 2/2 during the cancelled Hoodi Gloas fork on 2026-10-26.

Safety held. The cluster reached no decision while the split lasted, so no key share signed on the dead branch, and it recovered once the clients were upgraded. The operator experience did not hold up:

  • readyz stayed 1;
  • the only health check that fired was a misleading insufficient_round_changes;
  • duties failed as generic no_consensus or as bug_fetch_error;
  • no log said "chain split".

The doc lays out the incident timeline from metrics and proposes the following changes for discussion:

  • P0
    • exchange fork schedules over peerinfo, to warn before a fork;
    • add a core_consensus_compare_failed_total{duty,field} metric and replace the per-round WARN with one actionable ERROR;
    • add a chain_split_halt tracker reason.
  • P1
    • add a readyz state;
    • add a health check;
    • diagnose a VC signing with the wrong fork domain;
    • fix the bug_fetch_error misclassification;
    • add a "resolved" signal.
  • P2
    • add an operator runbook;
    • add real-time alert guidance;
    • decide the graduation path to beta and stable.

Comparing head votes is explicitly a non-goal. Head votes aren't slashable, and halting on them would turn ordinary propagation jitter into missed attestations.

The aim is to agree here on what we implement and why. Implementation PRs will follow separately.

category: docs
ticket: none
feature_flag: chain_split_halt

Design doc for making chain_split_halt halts legible, actionable and
pre-emptive, following dvt-c3's 2/2 split during the cancelled Hoodi
Gloas fork.

category: docs
ticket: none
feature_flag: chain_split_halt
@sonarqubecloud

Copy link
Copy Markdown

@codecov

codecov Bot commented Oct 28, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 67.68%. Comparing base (4e4d996) to head (ece1b9c).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4812      +/-   ##
==========================================
+ Coverage   67.61%   67.68%   +0.07%     
==========================================
  Files         254      254              
  Lines       33609    33685      +76     
==========================================
+ Hits        22725    22800      +75     
- Misses      10883    10884       +1     
  Partials        1        1              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant