Skip to content

fix: WAL persist failure silently ignored — node must exit on IO error #404

Closed

Description

@JoshuaChi

When persist_entries fails in the IO thread (buffered_raft_log.rs:892),
d-engine only logs an error. The Raft loop receives no signal, the node
continues serving, and in Level 2 (MemFirst) mode clients may have already
received ACKs for entries that were never persisted — silent data loss.

Fix: on persist_entries failure, send a fatal/shutdown signal to the Raft
loop so the node exits cleanly, consistent with etcd's behavior.

Affected: buffered_raft_log.rs IO thread, both durable and MemFirst modes.

Activity

  1. added this to the v0.2.5 milestone on Jun 6, 2026
  2. added theissue type on Jun 6, 2026
  3. added
    component:raft-statemachineState machine semantics, apply pipeline, RocksDB/File state machine implementations.
    concern:correctnessLinearizability, serializability, consistency guarantees across replicas.
    on Jun 6, 2026
  4. JoshuaChi commented on Oct 2, 2026

    @JoshuaChi
    ContributorAuthor

    fixed in #447 (1d65aae)
    persist/fsync failure → mark_poisoned_and_notify → InternalEvent::FatalError → Raft loop exits
    tests: drain_fsync_test.rs (persist / fsync failure poisons)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    component:raft-statemachineState machine semantics, apply pipeline, RocksDB/File state machine implementations.concern:correctnessLinearizability, serializability, consistency guarantees across replicas.

    Type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions