diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 4c4da247..e240f9aa 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -32,8 +32,12 @@ jobs: toolchain: 1.89.0 components: rustfmt, clippy + - name: Install nightly rustfmt + run: rustup toolchain install nightly --profile minimal --component rustfmt + + # rustfmt.toml uses nightly-only options, so formatting must be checked with nightly. - name: Check Formatting - run: cargo fmt --all -- --check + run: cargo +nightly fmt --all -- --check - name: Run Clippy run: cargo clippy --all-targets --all-features -- -D warnings diff --git a/CHANGELOG.md b/CHANGELOG.md index a4eebbac..4847a933 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,6 +22,10 @@ All notable changes to this project will be documented in this file. - **Write admission and CheckQuorum** (#423): A leader without a recent quorum ACK rejects new client writes with `NotLeader` after `election_timeout_min`, before anything is appended, so clients can retry safely. After `election_timeout_max x write_admission_election_timeout_multiple` (new key in `[raft.backpressure]`, default `2`) without a quorum ACK it steps down. A new leader gets one window for its first ACK; single-voter clusters are exempt. +- **Queue-wait metrics** (#455): `core.raft.client_cmd.issue_to_push_ms`, `core.raft.sm_worker.queue_wait_ms` and `core.raft.central_loop.apply_completed_queue_wait_ms` show where a write waits between the client, the Raft loop and the state machine worker. + +- **`raft.state_machine.shutdown_apply_timeout_ms`** (#455, default `5000`): how long shutdown waits for the state machine worker to apply the entries already committed. On timeout it logs an error and shuts down anyway; a restart replays what was not applied. + ### Fixed - **fix(ttl) #398**: Removed `lease.enabled` flag — TTL expiration is always active. Fixes fatal crash when calling `put_with_ttl` without setting the (now-removed) `lease.enabled = true`. @@ -54,6 +58,14 @@ All notable changes to this project will be documented in this file. ### Changed +- **⚠️ `StateMachine::stop()` removed; `StateMachine::close_storage()` renamed to `close()`** (#455): `stop()` had no caller in d-engine. `close()` is now the single, permanent, idempotent end-of-life hook and also persists what must survive a restart (lease state). It returns `Result<(), Error>`: release your resources even if persisting fails, and return the first error; the node logs it at ERROR level. Custom implementations: delete `stop()`, rename `close_storage()` to `close()`, and move any lease persistence from `stop()` into `close()`. See `MIGRATION_GUIDE.md`. + +- **State machine worker lifecycle** (#455): the `NodeBuilder` only assembles the worker; `Node::run()` starts it and stops it. `StateMachineWorker::new` no longer takes a shutdown signal. + +- **Known limitation: re-opening a RocksDB directory while you still hold the storage engine** (#455): the state machine now releases its RocksDB handle at shutdown, but the log store keeps its own handle until the `RocksDBStorageEngine` is dropped. If you keep your own clone of the storage engine after `stop()`, opening the same directory again fails with a lock error until that clone is dropped. Passing the engine to `EmbeddedEngine` without keeping a clone is unaffected. + +- **Internal commit path** (#455): the separate commit handler task is merged into the Raft loop (`dispatch_committed`). `ClientCmd::Propose`, `StateMachineCommand::ApplyEntries` and `InternalEvent::ApplyCompleted` each carry one more timestamp field (internal types). + - **MSRV raised to Rust 1.89**: The `data_dir` startup lock (prevents two node processes from sharing the same directory) uses `std::fs::File::try_lock()`, stable since 1.89.0. CI and `rust-toolchain.toml` were already pinned to 1.89.0 — only the `rust-version` field in @@ -99,7 +111,7 @@ All notable changes to this project will be documented in this file. - **⚠️ `[raft.persistence] strategy` removed** (#446): `PersistenceStrategy` was a single-variant enum (`MemFirst`) left over from #268; its only meaning now lives in whether an entry has reached `durable_index`, which is no longer a configurable choice. Existing configs setting `strategy = - "MemFirst"` or `"DiskFirst"` are silently ignored, not an error — remove the field, `flush_policy` +"MemFirst"` or `"DiskFirst"` are silently ignored, not an error — remove the field, `flush_policy` is the only persistence knob now. --- diff --git a/MIGRATION_GUIDE.md b/MIGRATION_GUIDE.md index b7957a93..97480ea8 100644 --- a/MIGRATION_GUIDE.md +++ b/MIGRATION_GUIDE.md @@ -352,6 +352,45 @@ fn apply_chunk(&mut self, entries: &[ApplyEntry]) -> Result<(), ...> --- +## 🚨 For v0.2.4 Users: `StateMachine` Lifecycle Changes in v0.2.5 (#455) + +### What Changed + +| Before (v0.2.4) | After (v0.2.5) | +| ---------------------------------------------------------------- | ----------------------------------------------------------------- | +| `fn stop(&self) -> Result<(), Error>` (required) | **Removed** | +| `fn close_storage(&self)` (default no-op) | **Renamed** `fn close(&self) -> Result<(), Error>` (default `Ok`) | +| `[raft.storage]` accepted as an alias of `[raft.state_machine]` | Use `[raft.state_machine]` | + +`stop()` was never called by d-engine. `close()` is now the one end-of-life hook. d-engine calls it once, as the last step of the +state machine worker. When the worker drains cleanly, every committed entry has been applied by then; if +`shutdown_apply_timeout_ms` expires first, shutdown continues and committed entries may remain unapplied until the next start. +Because you may keep your own `Arc` to the state machine, do not rely on `Drop` alone. `close()` returns an error if persisting +fails: release your resources anyway and return the first error. The node logs it at ERROR level. + +### Migration Steps + +1. Delete your `fn stop(...)`. +2. Rename `fn close_storage(&self)` to `fn close(&self) -> Result<(), Error>`. +3. If your `stop()` persisted something that must survive a restart (for example lease state), do it in `close()`. +4. `close()` must be synchronous and idempotent: after it returns, every operation should return a "not serving" error. +5. In config files, rename `[raft.storage]` to `[raft.state_machine]`. + +```rust +// Before +fn stop(&self) -> Result<(), Error> { self.persist_ttl()?; Ok(()) } +fn close_storage(&self) { self.db.flush(); } + +// After +fn close(&self) -> Result<(), Error> { + let persisted = self.persist_ttl(); + self.db.flush(); + persisted +} +``` + +--- + ## 🚨 For v0.2.3 Users: API Surface Changes in v0.2.4 (#326) ### What Changed diff --git a/Makefile b/Makefile index 74493e61..fcb35f83 100644 --- a/Makefile +++ b/Makefile @@ -26,6 +26,8 @@ # Cargo executable CARGO ?= cargo +# rustfmt.toml uses nightly-only options (group_imports, imports_granularity, ...); stable rustfmt ignores them. +CARGO_FMT ?= cargo +nightly fmt # On macOS with Homebrew: auto-detect compression lib paths to skip bundled C++ # compilation of RocksDB dependencies, which fails under macOS 26 + Xcode 26 @@ -127,6 +129,7 @@ check-env: install-tools: check-env @echo "$(BLUE)Installing Rust development components...$(NC)" @rustup component add rustfmt clippy rust-src rust-analyzer 2>/dev/null || true + @rustup toolchain install nightly --profile minimal --component rustfmt 2>/dev/null || true @if ! command -v cargo-nextest >/dev/null 2>&1; then \ echo "$(YELLOW)Installing cargo-nextest for fast parallel testing...$(NC)"; \ cargo install cargo-nextest --locked; \ @@ -163,14 +166,14 @@ check-workspace: check-env ## fmt-check Check code formatting without modifying files (dry-run) fmt-check: check-workspace @echo "$(BLUE)Checking code formatting...$(NC)" - @$(CARGO) fmt --all -- --check || \ + @$(CARGO_FMT) --all -- --check || \ { echo "$(RED)✗ Formatting issues found. Run 'make fmt-fix' to fix them.$(NC)"; exit 1; } @echo "$(GREEN)✓ Code formatting check passed$(NC)" ## fmt fmt-fix Fix all code formatting issues automatically fmt fmt-fix: install-tools check-workspace @echo "$(BLUE)Applying code formatting fixes...$(NC)" - @$(CARGO) fmt --all + @$(CARGO_FMT) --all @echo "$(GREEN)✓ Code formatting fixed in all crates$(NC)" # ============================================================================ @@ -232,7 +235,7 @@ check: fmt-check clippy deny quick-check: check-workspace @echo "$(BLUE)Running quick validation...$(NC)" @$(BREW_ROCKSDB_ENV) $(CARGO) check --workspace --lib --tests --all-features - @$(CARGO) fmt --all -- --check || exit 1 + @$(CARGO_FMT) --all -- --check || exit 1 @echo "$(GREEN)✓ Quick validation passed$(NC)" # ============================================================================ @@ -257,7 +260,7 @@ build-release: check check-workspace # ============================================================================ ## test Run all tests with nextest (fast, parallel) -test: install-tools check-workspace check-all-projects +test: install-tools check-workspace fmt-check check-all-projects @echo "$(BLUE)Running all tests with nextest...$(NC)" @CI=1 RUST_LOG=$(RUST_LOG_LEVEL) RUST_BACKTRACE=$(RUST_BACKTRACE) \ $(BREW_ROCKSDB_ENV) $(CARGO) nextest run --all-features --workspace --no-fail-fast @@ -378,7 +381,7 @@ clean-deps: clean # ============================================================================ ## pre-release Full pre-release validation (comprehensive checks) -pre-release: install-tools check-workspace check docs-check test audit build-release +pre-release: install-tools check-workspace check docs-check fmt-check test audit build-release @echo "$(BLUE)Checking version consistency across workspace...$(NC)" @version=$$(grep '^version' d-engine/Cargo.toml | head -n1 | cut -d'"' -f2); \ for crate in d-engine-proto d-engine-core d-engine-client d-engine-server d-engine; do \ diff --git a/d-engine-client/src/grpc_client.rs b/d-engine-client/src/grpc_client.rs index 9d63c34a..9d351998 100644 --- a/d-engine-client/src/grpc_client.rs +++ b/d-engine-client/src/grpc_client.rs @@ -1,13 +1,12 @@ -use super::ClientInner; -use crate::ClientApiError; -use crate::ClientResponseExt; -use crate::scoped_timer::ScopedTimer; +use std::sync::Arc; + use arc_swap::ArcSwap; use bytes::Bytes; use d_engine_core::BatchOp; +use d_engine_core::client::ClientApi; +use d_engine_core::client::ClientApiResult; use d_engine_core::client::ErrorCode; use d_engine_core::client::KvEntry; -use d_engine_core::client::{ClientApi, ClientApiResult}; use d_engine_core::config::ReadConsistencyPolicy; use d_engine_core::types::ScanResults; use d_engine_proto::client::ClientReadRequest; @@ -26,13 +25,17 @@ use d_engine_proto::client::write_command::batch_op; use rand::Rng; use rand::SeedableRng; use rand::rngs::StdRng; -use std::sync::Arc; use tonic::codec::CompressionEncoding; use tonic::transport::Channel; use tracing::debug; use tracing::error; use tracing::warn; +use super::ClientInner; +use crate::ClientApiError; +use crate::ClientResponseExt; +use crate::scoped_timer::ScopedTimer; + /// gRPC-based KV client for standalone mode. Obtained via [`crate::ClientBuilder`]. /// /// For embedded mode use `EmbeddedClient` instead. Both implement `ClientApi`. diff --git a/d-engine-client/src/grpc_client_test.rs b/d-engine-client/src/grpc_client_test.rs index 4a5ee5c7..f3a10cea 100644 --- a/d-engine-client/src/grpc_client_test.rs +++ b/d-engine-client/src/grpc_client_test.rs @@ -18,7 +18,6 @@ use tracing_test::traced_test; use crate::Client; use crate::ClientConfig; use crate::ClientInner; - use crate::ConnectionPool; use crate::GrpcClient; use crate::mock_rpc_service::MockNode; @@ -1435,10 +1434,11 @@ mod cas_operations { // ============================================================================= mod watch_membership_tests { - use super::*; use d_engine_proto::client::MembershipSnapshot; use tokio_stream::StreamExt; + use super::*; + async fn make_client(port: u16) -> GrpcClient { let endpoints = vec![format!("http://localhost:{}", port)]; let config = ClientConfig::default(); @@ -1547,9 +1547,10 @@ mod watch_membership_tests { // ============================================================================= mod watch_tests { - use super::*; use tokio_stream::StreamExt; + use super::*; + async fn make_client(port: u16) -> GrpcClient { let endpoints = vec![format!("http://localhost:{}", port)]; let config = ClientConfig::default(); @@ -1664,8 +1665,11 @@ mod scan_tests { use tokio::sync::oneshot; use tracing_test::traced_test; + use crate::ClientConfig; + use crate::ClientInner; + use crate::ConnectionPool; + use crate::GrpcClient; use crate::mock_rpc_service::MockNode; - use crate::{ClientConfig, ClientInner, ConnectionPool, GrpcClient}; async fn make_client(port: u16) -> GrpcClient { let endpoints = vec![format!("http://localhost:{port}")]; diff --git a/d-engine-client/src/lib.rs b/d-engine-client/src/lib.rs index a8c8599c..38a5756f 100644 --- a/d-engine-client/src/lib.rs +++ b/d-engine-client/src/lib.rs @@ -73,9 +73,11 @@ mod utils; pub use builder::*; pub use config::*; +pub use d_engine_core::client::ClientApi; +pub use d_engine_core::client::ClientApiError; +pub use d_engine_core::client::ClientApiResult; pub use d_engine_core::client::ErrorCode; pub use d_engine_core::client::ScanResults; -pub use d_engine_core::client::{ClientApi, ClientApiError, ClientApiResult}; pub use grpc_client::*; pub(crate) use pool::*; @@ -170,21 +172,21 @@ impl Client { /// or `ClientConfig::cluster_ready_timeout` elapses. /// /// **What this does:** - /// - Probes all endpoints in round-robin until one reports a ready leader - /// (`current_leader_id` is `Some` and present in the member list) + /// - Probes all endpoints in round-robin until one reports a ready leader (`current_leader_id` + /// is `Some` and present in the member list) /// - Atomically replaces the cached leader/follower connections - /// - After `Ok(())`, `get_leader_id()` returns the new leader and all - /// write/read operations are routed to the correct node + /// - After `Ok(())`, `get_leader_id()` returns the new leader and all write/read operations are + /// routed to the correct node /// /// **What this does NOT do:** - /// - Does not guarantee that the caller's in-flight requests succeeded; - /// requests sent before `refresh()` may have failed and need to be retried - /// - Writes are at-least-once across a failover: a write that was already proposed when - /// the leader stepped down may still be committed by the new leader even though the - /// caller received an error. Re-issuing a non-idempotent command can apply it twice - /// (the engine has no request-id dedup yet). - /// - Does not update endpoints permanently; pass `new_endpoints` to change - /// the bootstrap list for this and future refreshes + /// - Does not guarantee that the caller's in-flight requests succeeded; requests sent before + /// `refresh()` may have failed and need to be retried + /// - Writes are at-least-once across a failover: a write that was already proposed when the + /// leader stepped down may still be committed by the new leader even though the caller + /// received an error. Re-issuing a non-idempotent command can apply it twice (the engine has + /// no request-id dedup yet). + /// - Does not update endpoints permanently; pass `new_endpoints` to change the bootstrap list + /// for this and future refreshes /// /// **Typical usage after leader failover:** /// ```ignore diff --git a/d-engine-client/src/mock_rpc.rs b/d-engine-client/src/mock_rpc.rs index b4f31720..7c53fefa 100644 --- a/d-engine-client/src/mock_rpc.rs +++ b/d-engine-client/src/mock_rpc.rs @@ -66,11 +66,14 @@ impl MockRpcService { /// Convert core `ClientResponse` to the proto wire type for test mock gRPC handlers. /// Both `ErrorCode` enums are `#[repr(i32)]` with identical discriminants. fn core_to_proto_response(r: CoreClientResponse) -> ClientResponse { - use d_engine_core::client::{ClientResponsePayload, WriteResult}; - use d_engine_proto::client::{ - ClientResult, KvEntry, ReadResults as ProtoReadResults, ScanResults as ProtoScanResults, - WriteResult as ProtoWriteResult, client_response::SuccessResult, - }; + use d_engine_core::client::ClientResponsePayload; + use d_engine_core::client::WriteResult; + use d_engine_proto::client::ClientResult; + use d_engine_proto::client::KvEntry; + use d_engine_proto::client::ReadResults as ProtoReadResults; + use d_engine_proto::client::ScanResults as ProtoScanResults; + use d_engine_proto::client::WriteResult as ProtoWriteResult; + use d_engine_proto::client::client_response::SuccessResult; use d_engine_proto::error::ErrorMetadata; let error_code = r.error as i32; diff --git a/d-engine-client/src/mock_rpc_service.rs b/d-engine-client/src/mock_rpc_service.rs index 485d7169..84f1586b 100644 --- a/d-engine-client/src/mock_rpc_service.rs +++ b/d-engine-client/src/mock_rpc_service.rs @@ -187,7 +187,8 @@ impl MockNode { /// /// # Parameters /// - `rx`: Shutdown signal receiver - /// - `cas_should_succeed`: true = CAS succeeds (comparison matched), false = CAS fails (mismatch) + /// - `cas_should_succeed`: true = CAS succeeds (comparison matched), false = CAS fails + /// (mismatch) /// /// # Returns /// - Channel and port for the mock server diff --git a/d-engine-client/src/pool.rs b/d-engine-client/src/pool.rs index 022d670e..ea520c6b 100644 --- a/d-engine-client/src/pool.rs +++ b/d-engine-client/src/pool.rs @@ -85,8 +85,8 @@ impl ConnectionPool { config: &ClientConfig, ) -> std::result::Result<(Channel, Vec, Vec, Option), ClientApiError> { - // 1. Load cluster metadata + establish verified leader channel atomically. - // Returns only when a ready leader is confirmed AND its channel is connectable. + // 1. Load cluster metadata + establish verified leader channel atomically. Returns only + // when a ready leader is confirmed AND its channel is connectable. let (membership, leader_conn) = Self::load_cluster_metadata(endpoints, config).await?; info!("Cluster members discovered: {:?}", membership.nodes); @@ -151,8 +151,8 @@ impl ConnectionPool { /// /// Returns: /// - `Some(Ok(membership))` — leader is known and present in member list (ready) - /// - `Some(Err(()))` — node responded but cluster not yet ready (election in progress - /// or stale leader ID); caller should try next node + /// - `Some(Err(()))` — node responded but cluster not yet ready (election in progress or + /// stale leader ID); caller should try next node /// - `None` — node unreachable; caller should try next node pub(super) async fn probe_endpoint( addr: &str, diff --git a/d-engine-client/src/proto/client_ext_test.rs b/d-engine-client/src/proto/client_ext_test.rs index b7090cd1..9aa481dd 100644 --- a/d-engine-client/src/proto/client_ext_test.rs +++ b/d-engine-client/src/proto/client_ext_test.rs @@ -286,10 +286,10 @@ fn test_into_scan_results_success() { let scan = response.into_scan_results().unwrap(); assert_eq!(scan.revision, 7); - assert_eq!( - scan.entries, - vec![(Bytes::from("/services/node1"), Bytes::from("10.0.0.1"))] - ); + assert_eq!(scan.entries, vec![( + Bytes::from("/services/node1"), + Bytes::from("10.0.0.1") + )]); } #[test] diff --git a/d-engine-core/benches/leader_state_bench.rs b/d-engine-core/benches/leader_state_bench.rs index 1a8fc0a0..39160a99 100644 --- a/d-engine-core/benches/leader_state_bench.rs +++ b/d-engine-core/benches/leader_state_bench.rs @@ -9,15 +9,16 @@ //! //! 1. **LeaderState Creation**: Measures instantiation overhead //! 2. **Pending Promotion Processing**: Measures learner-to-voter promotion latency -//! 3. **Batch Promotion Scaling**: Validates performance characteristics across different batch sizes +//! 3. **Batch Promotion Scaling**: Validates performance characteristics across different batch +//! sizes //! //! ## Performance Acceptance Criteria //! //! ### Benchmark 1: LeaderState Creation //! - **Target**: < 500 ns per operation //! - **Failure Criteria**: > 1000 ns indicates memory allocation issues or lock contention -//! - **Why It Matters**: Leader election happens during network partitions; slow creation -//! delays cluster recovery +//! - **Why It Matters**: Leader election happens during network partitions; slow creation delays +//! cluster recovery //! //! ### Benchmark 2: Process Pending Promotions (2 nodes) //! - **Target**: < 20 ms per operation @@ -32,8 +33,8 @@ //! - batch_2: ~20 ms (2x baseline) //! - batch_5: ~50 ms (5x baseline) //! - batch_10: ~100 ms (10x baseline) -//! - **Why It Matters**: Non-linear scaling would cause cluster instability during -//! large-scale node additions +//! - **Why It Matters**: Non-linear scaling would cause cluster instability during large-scale node +//! additions //! //! ## How to Identify Failures //! @@ -104,14 +105,18 @@ //! ## Historical Context //! //! - **v0.1.0**: Initial benchmark suite -//! - **v0.2.0** (#236): Added after ReadIndex batching refactor to ensure no regression -//! in promotion path +//! - **v0.2.0** (#236): Added after ReadIndex batching refactor to ensure no regression in +//! promotion path use std::sync::Arc; -use criterion::{Criterion, black_box, criterion_group, criterion_main}; +use criterion::Criterion; +use criterion::black_box; +use criterion::criterion_group; +use criterion::criterion_main; +use d_engine_core::MockTypeConfig; +use d_engine_core::RaftNodeConfig; use d_engine_core::leader_state::LeaderState; -use d_engine_core::{MockTypeConfig, RaftNodeConfig}; /// Benchmark 1: LeaderState Creation Performance /// diff --git a/d-engine-core/src/client/client_api.rs b/d-engine-core/src/client/client_api.rs index 7f55f9b5..9cabba44 100644 --- a/d-engine-core/src/client/client_api.rs +++ b/d-engine-core/src/client/client_api.rs @@ -25,11 +25,12 @@ //! } //! ``` +use bytes::Bytes; + use crate::BatchOp; use crate::client::client_api_error::ClientApiResult; use crate::config::ReadConsistencyPolicy; use crate::types::ScanResults; -use bytes::Bytes; /// Unified key-value store interface implemented by `EmbeddedClient` and `GrpcClient`. /// @@ -57,8 +58,8 @@ pub trait ClientApi: Send + Sync { /// /// # Errors /// - /// - [`crate::client::client_api_error::ClientApiError::Network`] if node is shutting down, timeout - /// occurs, or the node is not leader (carries `leader_hint` for redirect) + /// - [`crate::client::client_api_error::ClientApiError::Network`] if node is shutting down, + /// timeout occurs, or the node is not leader (carries `leader_hint` for redirect) /// - [`crate::client::client_api_error::ClientApiError::Business`] for other server-side errors /// /// # Example @@ -147,7 +148,8 @@ pub trait ClientApi: Send + Sync { /// /// # Errors /// - /// - [`crate::client::client_api_error::ClientApiError::Network`] if node is shutting down or timeout occurs + /// - [`crate::client::client_api_error::ClientApiError::Network`] if node is shutting down or + /// timeout occurs /// - [`crate::client::client_api_error::ClientApiError::Business`] for server-side errors /// /// # Example @@ -206,7 +208,8 @@ pub trait ClientApi: Send + Sync { /// /// # Errors /// - /// - [`crate::client::client_api_error::ClientApiError::Network`] if node is shutting down or timeout occurs + /// - [`crate::client::client_api_error::ClientApiError::Network`] if node is shutting down or + /// timeout occurs /// - [`crate::client::client_api_error::ClientApiError::Business`] for server-side errors /// /// # Example diff --git a/d-engine-core/src/client/client_api_error.rs b/d-engine-core/src/client/client_api_error.rs index 48fc5b2d..7352dfae 100644 --- a/d-engine-core/src/client/client_api_error.rs +++ b/d-engine-core/src/client/client_api_error.rs @@ -1,11 +1,13 @@ -use crate::client::types::ErrorCode; +use std::error::Error; + use serde::Deserialize; use serde::Serialize; -use std::error::Error; use tokio::task::JoinError; use tonic::Code; use tonic::Status; +use crate::client::types::ErrorCode; + /// Result type for KV operations pub type ClientApiResult = std::result::Result; @@ -272,7 +274,8 @@ impl From for ClientApiError { code, message: "Not leader".to_string(), retry_after_ms: Some(100), // suggest quick retry - leader_hint: None, // populated by ClientResponseExt::validate_error() when ErrorMetadata is available + leader_hint: None, /* populated by ClientResponseExt::validate_error() + * when ErrorMetadata is available */ }, ErrorCode::StaleOperation => ClientApiError::Business { code, diff --git a/d-engine-core/src/client/client_api_error_test.rs b/d-engine-core/src/client/client_api_error_test.rs index 464f07e7..326dd147 100644 --- a/d-engine-core/src/client/client_api_error_test.rs +++ b/d-engine-core/src/client/client_api_error_test.rs @@ -2,7 +2,8 @@ #[cfg(test)] mod client_api_error_tests { - use crate::client::client_api_error::{ClientApiError, ClientApiResult}; + use crate::client::client_api_error::ClientApiError; + use crate::client::client_api_error::ClientApiResult; use crate::client::types::ErrorCode; /// Test: NotLeader error construction and code mapping diff --git a/d-engine-core/src/client/mod.rs b/d-engine-core/src/client/mod.rs index 4a47247e..de948c27 100644 --- a/d-engine-core/src/client/mod.rs +++ b/d-engine-core/src/client/mod.rs @@ -8,11 +8,19 @@ pub mod client_api_error; pub mod types; pub use client_api::ClientApi; -pub use client_api_error::{ClientApiError, ClientApiResult}; -pub use types::{ - ClientReadRequest, ClientResponse, ClientResponsePayload, ClientWriteRequest, ErrorCode, - KvEntry, LeaderHint, ReadResults, ScanResults, WriteResult, -}; +pub use client_api_error::ClientApiError; +pub use client_api_error::ClientApiResult; +pub use types::ClientReadRequest; +pub use types::ClientResponse; +pub use types::ClientResponsePayload; +pub use types::ClientWriteRequest; +pub use types::ErrorCode; +pub use types::KvEntry; +pub use types::LeaderHint; +pub use types::ReadResults; +pub use types::ScanResults; +pub use types::WriteResult; + // ReadConsistencyPolicy lives in config but is part of the public client API surface. pub use crate::config::ReadConsistencyPolicy; diff --git a/d-engine-core/src/command.rs b/d-engine-core/src/command.rs index a9eb45c3..6ccaa546 100644 --- a/d-engine-core/src/command.rs +++ b/d-engine-core/src/command.rs @@ -151,9 +151,9 @@ impl TryFrom for Command { /// Decode a batch of raw Raft `Entry` values into `ApplyEntry` values. /// -/// - `Config` entries become `Command::Noop` — membership is handled by the Raft layer, but -/// the index must still reach SM so `last_applied` stays contiguous; dropping them would -/// leave `last_applied` stuck below the config index, breaking ReadIndex drain. +/// - `Config` entries become `Command::Noop` — membership is handled by the Raft layer, but the +/// index must still reach SM so `last_applied` stays contiguous; dropping them would leave +/// `last_applied` stuck below the config index, breaking ReadIndex drain. /// - `Noop` entries become `Command::Noop`. /// - `Command` bytes are decoded via `TryFrom`. /// - Any decode failure returns `Err` immediately. diff --git a/d-engine-core/src/command_test.rs b/d-engine-core/src/command_test.rs index 3734ed0d..173cba60 100644 --- a/d-engine-core/src/command_test.rs +++ b/d-engine-core/src/command_test.rs @@ -1,16 +1,22 @@ use bytes::Bytes; use d_engine_proto::client::WriteCommand; +use d_engine_proto::client::write_command::Batch; +use d_engine_proto::client::write_command::BatchOp; +use d_engine_proto::client::write_command::CompareAndSwap; +use d_engine_proto::client::write_command::Delete; +use d_engine_proto::client::write_command::Insert; +use d_engine_proto::client::write_command::Operation; use d_engine_proto::client::write_command::batch_op; -use d_engine_proto::client::write_command::{ - Batch, BatchOp, CompareAndSwap, Delete, Insert, Operation, -}; use d_engine_proto::common::AddNode; use d_engine_proto::common::Entry; use d_engine_proto::common::EntryPayload; use d_engine_proto::common::membership_change::Change; use prost::Message; -use crate::command::{ApplyEntry, BatchOp as CommandBatchOp, Command, decode_entries}; +use crate::command::ApplyEntry; +use crate::command::BatchOp as CommandBatchOp; +use crate::command::Command; +use crate::command::decode_entries; // ── helpers ──────────────────────────────────────────────────────────────── @@ -170,14 +176,11 @@ fn test_decode_noop_entry_produces_noop_command() { let entries = vec![noop_entry(1, 1)]; let result = decode_entries(entries).unwrap(); assert_eq!(result.len(), 1); - assert_eq!( - result[0], - ApplyEntry { - index: 1, - term: 1, - command: Command::Noop - } - ); + assert_eq!(result[0], ApplyEntry { + index: 1, + term: 1, + command: Command::Noop + }); } #[test] @@ -196,28 +199,22 @@ fn test_decode_insert_without_ttl() { let entries = vec![insert_entry(2, 1, b"k", b"v", 0)]; let result = decode_entries(entries).unwrap(); assert_eq!(result.len(), 1); - assert_eq!( - result[0].command, - Command::Insert { - key: Bytes::from_static(b"k"), - value: Bytes::from_static(b"v"), - ttl_secs: None, - } - ); + assert_eq!(result[0].command, Command::Insert { + key: Bytes::from_static(b"k"), + value: Bytes::from_static(b"v"), + ttl_secs: None, + }); } #[test] fn test_decode_insert_with_ttl_converts_to_some() { let entries = vec![insert_entry(3, 1, b"key", b"val", 60)]; let result = decode_entries(entries).unwrap(); - assert_eq!( - result[0].command, - Command::Insert { - key: Bytes::from_static(b"key"), - value: Bytes::from_static(b"val"), - ttl_secs: Some(60), - } - ); + assert_eq!(result[0].command, Command::Insert { + key: Bytes::from_static(b"key"), + value: Bytes::from_static(b"val"), + ttl_secs: Some(60), + }); } #[test] @@ -225,40 +222,31 @@ fn test_decode_delete_entry() { let entries = vec![delete_entry(4, 2, b"gone")]; let result = decode_entries(entries).unwrap(); assert_eq!(result.len(), 1); - assert_eq!( - result[0].command, - Command::Delete { - key: Bytes::from_static(b"gone") - } - ); + assert_eq!(result[0].command, Command::Delete { + key: Bytes::from_static(b"gone") + }); } #[test] fn test_decode_cas_with_expected_value() { let entries = vec![cas_entry(5, 1, b"k", Some(b"old"), b"new")]; let result = decode_entries(entries).unwrap(); - assert_eq!( - result[0].command, - Command::CompareAndSwap { - key: Bytes::from_static(b"k"), - expected: Some(Bytes::from_static(b"old")), - value: Bytes::from_static(b"new"), - } - ); + assert_eq!(result[0].command, Command::CompareAndSwap { + key: Bytes::from_static(b"k"), + expected: Some(Bytes::from_static(b"old")), + value: Bytes::from_static(b"new"), + }); } #[test] fn test_decode_cas_with_no_expected_value_means_key_must_not_exist() { let entries = vec![cas_entry(6, 1, b"k", None, b"new")]; let result = decode_entries(entries).unwrap(); - assert_eq!( - result[0].command, - Command::CompareAndSwap { - key: Bytes::from_static(b"k"), - expected: None, - value: Bytes::from_static(b"new"), - } - ); + assert_eq!(result[0].command, Command::CompareAndSwap { + key: Bytes::from_static(b"k"), + expected: None, + value: Bytes::from_static(b"new"), + }); } // ── decode: index and term preservation ──────────────────────────────────── @@ -319,21 +307,18 @@ fn test_decode_batch_inserts_only() { let entries = vec![batch_entry(7, 2, ops)]; let result = decode_entries(entries).unwrap(); assert_eq!(result.len(), 1); - assert_eq!( - result[0].command, - Command::Batch { - ops: vec![ - CommandBatchOp::Insert { - key: Bytes::from_static(b"k1"), - value: Bytes::from_static(b"v1"), - }, - CommandBatchOp::Insert { - key: Bytes::from_static(b"k2"), - value: Bytes::from_static(b"v2"), - }, - ] - } - ); + assert_eq!(result[0].command, Command::Batch { + ops: vec![ + CommandBatchOp::Insert { + key: Bytes::from_static(b"k1"), + value: Bytes::from_static(b"v1"), + }, + CommandBatchOp::Insert { + key: Bytes::from_static(b"k2"), + value: Bytes::from_static(b"v2"), + }, + ] + }); } #[test] @@ -343,19 +328,16 @@ fn test_decode_batch_deletes_only() { let entries = vec![batch_entry(8, 2, ops)]; let result = decode_entries(entries).unwrap(); assert_eq!(result.len(), 1); - assert_eq!( - result[0].command, - Command::Batch { - ops: vec![ - CommandBatchOp::Delete { - key: Bytes::from_static(b"k1"), - }, - CommandBatchOp::Delete { - key: Bytes::from_static(b"k2"), - }, - ] - } - ); + assert_eq!(result[0].command, Command::Batch { + ops: vec![ + CommandBatchOp::Delete { + key: Bytes::from_static(b"k1"), + }, + CommandBatchOp::Delete { + key: Bytes::from_static(b"k2"), + }, + ] + }); } #[test] @@ -369,24 +351,21 @@ fn test_decode_batch_mixed_ops() { let entries = vec![batch_entry(9, 2, ops)]; let result = decode_entries(entries).unwrap(); assert_eq!(result.len(), 1); - assert_eq!( - result[0].command, - Command::Batch { - ops: vec![ - CommandBatchOp::Insert { - key: Bytes::from_static(b"k1"), - value: Bytes::from_static(b"v1"), - }, - CommandBatchOp::Delete { - key: Bytes::from_static(b"k2"), - }, - CommandBatchOp::Insert { - key: Bytes::from_static(b"k3"), - value: Bytes::from_static(b"v3"), - }, - ] - } - ); + assert_eq!(result[0].command, Command::Batch { + ops: vec![ + CommandBatchOp::Insert { + key: Bytes::from_static(b"k1"), + value: Bytes::from_static(b"v1"), + }, + CommandBatchOp::Delete { + key: Bytes::from_static(b"k2"), + }, + CommandBatchOp::Insert { + key: Bytes::from_static(b"k3"), + value: Bytes::from_static(b"v3"), + }, + ] + }); } #[test] diff --git a/d-engine-core/src/commit_handler/default_commit_handler.rs b/d-engine-core/src/commit_handler/default_commit_handler.rs deleted file mode 100644 index bd672331..00000000 --- a/d-engine-core/src/commit_handler/default_commit_handler.rs +++ /dev/null @@ -1,306 +0,0 @@ -use async_trait::async_trait; -use d_engine_proto::common::Entry; -use d_engine_proto::common::entry_payload::Payload; -use std::sync::Arc; -use tokio::sync::mpsc; -use tokio::sync::watch; -use tracing::debug; -use tracing::error; -use tracing::info; -use tracing::trace; -use tracing::warn; - -use super::CommitHandler; -use crate::InternalEvent; -use crate::Membership; -use crate::NewCommitData; -use crate::RaftLog; -use crate::Result; -use crate::StateMachineCommandSender; -use crate::StateMachineHandler; -use crate::TypeConfig; -use crate::alias::MOF; -use crate::alias::ROF; -use crate::alias::SMHOF; -use crate::scoped_timer::ScopedTimer; - -// Dependencies container -pub struct CommitHandlerDependencies { - pub state_machine_handler: Arc>, - pub raft_log: Arc>, - pub membership: Arc>, - pub internal_event_tx: mpsc::UnboundedSender, - pub sm_apply_tx: StateMachineCommandSender, - pub shutdown_signal: watch::Receiver<()>, - pub max_batch_size: usize, -} - -#[derive(Debug)] -pub struct DefaultCommitHandler -where - T: TypeConfig, -{ - my_id: u32, - my_role: i32, - my_current_term: u64, - state_machine_handler: Arc>, - raft_log: Arc>, - new_commit_rx: Option>, - membership: Arc>, - - internal_event_tx: mpsc::UnboundedSender, // Cloned from Raft - sm_apply_tx: StateMachineCommandSender, // Send entries to SM Worker - - // Shutdown signal - shutdown_signal: watch::Receiver<()>, - - // Batch size for draining commit notifications - max_batch_size: usize, -} - -#[async_trait] -impl CommitHandler for DefaultCommitHandler -where - T: TypeConfig, -{ - async fn run(&mut self) -> Result<()> { - info!("[Node-{}] Commit handler started", self.my_id); - - let mut new_commit_rx = - self.new_commit_rx.take().expect("Expected a commit recv but found None"); - let mut shutdown_signal = self.shutdown_signal.clone(); - - loop { - tokio::select! { - // P0: shutdown received; - _ = shutdown_signal.changed() => { - info!("[CommitHandler] shutdown signal received."); - return Ok(()); - } - - // Submit events in real time - Some(new_commit_data) = new_commit_rx.recv() => { - trace!("[Node-{}] new commit index = {:?} committed..", self.my_id, new_commit_data.new_commit_index); - self.state_machine_handler.update_pending(new_commit_data.new_commit_index); - - // Keep sync my current term and role from new commit data - self.my_current_term = new_commit_data.current_term; - self.my_role = new_commit_data.role; - - // Drain all pending commit notifications (max_batch_size limit) - // This batches multiple committed entries into a single process_batch() call - let mut count = 1; - while count < self.max_batch_size { - match new_commit_rx.try_recv() { - Ok(next_data) => { - self.state_machine_handler.update_pending(next_data.new_commit_index); - self.my_current_term = next_data.current_term; - self.my_role = next_data.role; - count += 1; - } - Err(_) => break, - } - } - - trace!("[Node-{}] Processing batch with {} commit notifications", self.my_id, count); - if let Err(e) = self.process_batch().await { - error!("Failed to process batch: {}", e); - } - } - } - } - } -} - -impl DefaultCommitHandler -where - T: TypeConfig, -{ - pub fn new( - my_id: u32, - my_role: i32, - my_current_term: u64, - deps: CommitHandlerDependencies, - new_commit_rx: mpsc::UnboundedReceiver, - ) -> Self { - Self { - my_id, - my_role, - my_current_term, - state_machine_handler: deps.state_machine_handler, - raft_log: deps.raft_log, - membership: deps.membership, - new_commit_rx: Some(new_commit_rx), - internal_event_tx: deps.internal_event_tx, - sm_apply_tx: deps.sm_apply_tx, - shutdown_signal: deps.shutdown_signal, - max_batch_size: deps.max_batch_size, - } - } - - /// Process batch logs - /// - Separates config changes from application commands - /// - Applies config changes first via membership module - /// - Then applies application commands to state machine - /// - Error handling for config changes - /// - /// # Note:Sequential Integrity - /// Consider this sequence in a single batch: [ConfigRemove(A), ConfigAdd(B), EntryNormal(X)] - pub(crate) async fn process_batch(&self) -> Result<()> { - let _timer = ScopedTimer::new("CommitHandler::process_batch"); - let pending_range = self.state_machine_handler.pending_range(); - trace!("[Node-{}] Pending range: {:?}", self.my_id, pending_range); - - let Some(range) = pending_range else { - return Ok(()); - }; - let entries = self.raft_log.get_entries_range(range)?; - - debug!( - "[Node-{}] commit handler process batch, length = {}", - self.my_id, - entries.len() - ); - // Merge consecutive normal commands - let mut command_batch = vec![]; - - let mut last_error = None; - for entry in entries { - // In exact log order - if let Some(ref entry_payload) = entry.payload { - match entry_payload.payload { - Some(Payload::Command(_)) => command_batch.push(entry), - Some(Payload::Config(_)) => { - command_batch.push(entry.clone()); - - self.send_to_sm_worker(&mut command_batch).await?; - - if last_error.is_none() - && let Err(e) = self.apply_config_change(entry).await - { - last_error = Some(e); - } - } - Some(Payload::Noop(_)) => { - command_batch.push(entry); - - self.send_to_sm_worker(&mut command_batch).await?; - } - None => unreachable!(), - } - } - } - - debug!(?last_error, "flush complete"); - // Finally send remaining commands to SM Worker - if let Some(e) = last_error { - return Err(e); - } else { - self.send_to_sm_worker(&mut command_batch).await?; - } - - // Snapshot check moved to ApplyCompleted handler in inbound event loop. - // SM Worker applies entries asynchronously, so last_applied is stale here. - - Ok(()) - } - - /// Check if configuration change is a self-removal - /// - /// Returns true if the change is RemoveNode(my_id), indicating - /// that this node is removing itself from the cluster. - pub(crate) fn is_self_removal_config( - my_id: u32, - change: &d_engine_proto::common::MembershipChange, - ) -> bool { - matches!( - &change.change, - Some(d_engine_proto::common::membership_change::Change::RemoveNode(remove)) - if remove.node_id == my_id - ) - } - - /// Apply configuration change and detect self-removal - /// - /// If the first configure been applied failed, then all the following commands will be - /// rejected. (Consistency) - /// - /// Per Raft protocol: Leader can remove itself. After applying the removal, - /// leader must step down immediately. - async fn apply_config_change( - &self, - entry: Entry, - ) -> Result<()> { - let _timer = ScopedTimer::new("apply_config_change"); - debug!("Received config change:{:?}", &entry); - - if let Some(payload) = entry.payload - && let Some(Payload::Config(change)) = payload.payload - { - // Check if this is a self-removal (check BEFORE applying) - let is_self_removal = Self::is_self_removal_config(self.my_id, &change); - - // 1. Apply to membership state - if let Err(e) = self.membership.apply_config_change(change).await { - error!( - "[{}] Failed to apply config change at index {}: {:?}", - self.my_id, entry.index, e - ); - // Critical error - should panic or handle carefully - return Err(e); - } - - // 2. CRITICAL: Barrier point - self.membership.notify_config_applied(entry.index); - - // 2.5. Notify leader to refresh cluster metadata cache - // This must happen AFTER membership is applied - if let Err(e) = self.internal_event_tx.send(InternalEvent::MembershipApplied) { - warn!("Failed to send MembershipApplied event: {:?}", e); - } - - // 3. Leader self-removal: Step down immediately per Raft protocol - if is_self_removal { - warn!( - "[{}] Node removed from cluster membership, triggering step down (index {})", - self.my_id, entry.index - ); - // Signal step down - error is non-fatal as removal is already committed - if let Err(e) = self.internal_event_tx.send(InternalEvent::StepDownSelfRemoved) { - error!( - "[{}] Failed to send StepDownSelfRemoved event: {:?}", - self.my_id, e - ); - } - } - } - - Ok(()) - } - - // Define flush as an async function - async fn send_to_sm_worker( - &self, - batch: &mut Vec, - ) -> Result<()> { - if !batch.is_empty() { - let entries = std::mem::take(batch); - trace!( - "[Node-{}] Sending batch to SM Worker: {} entries", - self.my_id, - entries.len() - ); - - // Send entries to SM Worker without waiting for apply - self.sm_apply_tx.apply_entries(entries).map_err(|e| { - error!("[Node-{}] SM Worker channel closed: {:?}", self.my_id, e); - e - })?; - } - Ok(()) - } -} - -#[cfg(test)] -#[path = "default_commit_handler_test.rs"] -mod default_commit_handler_test; diff --git a/d-engine-core/src/commit_handler/default_commit_handler_test.rs b/d-engine-core/src/commit_handler/default_commit_handler_test.rs deleted file mode 100644 index 98d060b8..00000000 --- a/d-engine-core/src/commit_handler/default_commit_handler_test.rs +++ /dev/null @@ -1,1286 +0,0 @@ -use bytes::Bytes; -use d_engine_proto::common::Entry; -use d_engine_proto::common::EntryPayload; -use d_engine_proto::common::NodeRole::Leader; -use d_engine_proto::common::membership_change::Change; -use std::sync::Arc; -use std::time::Duration; -use tokio::sync::mpsc::{self}; -use tokio::sync::watch; -use tokio::task::JoinHandle; -use tokio::time::{self}; - -use super::CommitHandler; -use super::CommitHandlerDependencies; -use super::DefaultCommitHandler; -use crate::Error; -use crate::InternalEvent; -use crate::MockMembership; -use crate::MockRaftLog; -use crate::MockStateMachineHandler; -use crate::MockTypeConfig; -use crate::NewCommitData; -use crate::Result; - -const TEST_TERM: u64 = 1; - -pub enum CommandType { - Command(Bytes), - Configuration(Change), - Noop, -} - -pub fn build_entries( - commands: Vec, - term: u64, -) -> Vec { - let mut r = vec![]; - let mut index = 1; - for c in commands { - let entry = match c { - CommandType::Command(data) => Entry { - index, - term, - payload: Some(EntryPayload::command(data)), - }, - CommandType::Configuration(change) => Entry { - index, - term, - payload: Some(EntryPayload::config(change)), - }, - CommandType::Noop => Entry { - index, - term, - payload: Some(EntryPayload::noop()), - }, - }; - r.push(entry); - index += 1; - } - r -} - -pub struct TestHarness { - role: i32, - term: u64, - mock_smh: Arc>, - mock_log: Arc, - mock_membership: Arc>, - commit_tx: mpsc::UnboundedSender, - commit_rx: Option>, - internal_event_tx: mpsc::UnboundedSender, - internal_event_rx: mpsc::UnboundedReceiver, - shutdown_tx: watch::Sender<()>, - shutdown_rx: Option>, - handle: Option>, -} -#[allow(clippy::too_many_arguments)] -fn setup_harness( - role: i32, - term: u64, - entries: Vec, - last_applied: u64, - config_hook: F, - snapshot_condition: Option, -) -> TestHarness -where - F: Fn() -> bool + 'static + Send + Sync, -{ - let (commit_tx, commit_rx) = mpsc::unbounded_channel(); - let (shutdown_tx, shutdown_rx) = watch::channel(()); - let (internal_event_tx, internal_event_rx) = mpsc::unbounded_channel(); // Large capacity to prevent blocking in tests - - // Mock state machine - let mut mock_smh = MockStateMachineHandler::new(); - let cloned_entries = entries.clone(); - mock_smh - .expect_pending_range() - .returning(move || Some(1..=cloned_entries.last().map(|e| e.index).unwrap_or(1))); - mock_smh.expect_update_pending().returning(|_| {}); - mock_smh.expect_last_applied().returning(move || last_applied); - mock_smh - .expect_should_snapshot() - .returning(move |data| snapshot_condition.is_some_and(|idx| data.new_commit_index >= idx)); - - // Mock raft log - let mut mock_log = MockRaftLog::new(); - mock_log.expect_get_entries_range().returning(move |_| Ok(entries.clone())); - - // Mock membership - let mut mock_membership = MockMembership::new(); - mock_membership.expect_notify_config_applied().returning(|_| {}); - mock_membership.expect_apply_config_change().returning(move |_| { - if config_hook() { - Err(Error::Fatal("Command execution failed".to_string())) - } else { - Ok(()) - } - }); - - TestHarness { - role, - term, - mock_smh: Arc::new(mock_smh), - mock_log: Arc::new(mock_log), - mock_membership: Arc::new(mock_membership), - commit_rx: Some(commit_rx), - commit_tx, - internal_event_tx, - internal_event_rx, - shutdown_tx, - shutdown_rx: Some(shutdown_rx), - handle: None, - } -} - -impl TestHarness { - async fn run_handler(&mut self) { - let (sm_apply_tx, mut sm_apply_rx) = tokio::sync::mpsc::unbounded_channel(); - let sm_apply_tx = crate::StateMachineCommandSender::new(sm_apply_tx); - - let deps = CommitHandlerDependencies { - state_machine_handler: self.mock_smh.clone(), - raft_log: self.mock_log.clone(), - membership: self.mock_membership.clone(), - internal_event_tx: self.internal_event_tx.clone(), - sm_apply_tx, - shutdown_signal: self.shutdown_rx.take().unwrap(), - max_batch_size: 10, - }; - - let mut handler = DefaultCommitHandler::::new( - 1, - self.role, - self.term, - deps, - self.commit_rx.take().unwrap(), - ); - - // Spawn SM Worker to process entries - let _sm_worker_handle = tokio::spawn(async move { - while let Some(_entries) = sm_apply_rx.recv().await { - // SM Worker simply consumes entries from the channel - // In a real system, this would apply them to the state machine - // For testing, we just drain the channel to keep it open - } - }); - - self.handle = Some(tokio::spawn(async move { - let _ = handler.run().await; - })); - } - - async fn process_batch_handler(&mut self) -> Result<()> { - let (sm_apply_tx, mut sm_apply_rx) = tokio::sync::mpsc::unbounded_channel(); - let sm_apply_tx = crate::StateMachineCommandSender::new(sm_apply_tx); - - let deps = CommitHandlerDependencies { - state_machine_handler: self.mock_smh.clone(), - raft_log: self.mock_log.clone(), - membership: self.mock_membership.clone(), - internal_event_tx: self.internal_event_tx.clone(), - sm_apply_tx, - shutdown_signal: self.shutdown_rx.take().unwrap(), - max_batch_size: 10, - }; - - let handler = DefaultCommitHandler::::new( - 1, - self.role, - self.term, - deps, - self.commit_rx.take().unwrap(), - ); - - // Spawn SM Worker to process entries - let _sm_worker_handle = tokio::spawn(async move { - while let Some(_entries) = sm_apply_rx.recv().await { - // SM Worker simply consumes entries from the channel - } - }); - - handler.process_batch().await - } - - /// Same as `process_batch_handler`, but the SM Worker's receiving end is dropped - /// immediately instead of being drained — simulates the Worker task having - /// already exited, so `send_to_sm_worker`'s send must fail. - async fn process_batch_handler_with_closed_worker_channel(&mut self) -> Result<()> { - let (sm_apply_tx, sm_apply_rx) = tokio::sync::mpsc::unbounded_channel(); - drop(sm_apply_rx); - let sm_apply_tx = crate::StateMachineCommandSender::new(sm_apply_tx); - - let deps = CommitHandlerDependencies { - state_machine_handler: self.mock_smh.clone(), - raft_log: self.mock_log.clone(), - membership: self.mock_membership.clone(), - internal_event_tx: self.internal_event_tx.clone(), - sm_apply_tx, - shutdown_signal: self.shutdown_rx.take().unwrap(), - max_batch_size: 10, - }; - - let handler = DefaultCommitHandler::::new( - 1, - self.role, - self.term, - deps, - self.commit_rx.take().unwrap(), - ); - - handler.process_batch().await - } - - async fn send_commit( - &self, - index: u64, - role: i32, - ) { - self.commit_tx - .send(NewCommitData { - new_commit_index: index, - role, - current_term: TEST_TERM, - }) - .unwrap(); - } - - async fn expect_snapshot_trigger(&mut self) -> bool { - match time::timeout(Duration::from_millis(50), self.internal_event_rx.recv()).await { - Ok(Some(_)) => true, // Event received normally - Ok(None) => false, // Channel closed - Err(_) => false, // Timeout - } - } -} - -#[cfg(test)] -mod run_test { - use d_engine_proto::common::AddNode; - use d_engine_proto::common::NodeRole::Follower; - use d_engine_proto::common::NodeRole::Leader; - use d_engine_proto::common::RemoveNode; - use d_engine_proto::common::membership_change::Change; - use tokio::time; - - use super::*; - - /// 1. Test happy path with all entry types - #[tokio::test] - async fn test_full_processing_flow() { - let entries = build_entries( - vec![ - CommandType::Command(Bytes::from(b"cmd1".to_vec())), - CommandType::Configuration(Change::AddNode(AddNode { - node_id: 1, - address: "addr".into(), - status: d_engine_proto::common::NodeStatus::Promotable as i32, - })), - CommandType::Noop, - CommandType::Command(Bytes::from(b"cmd2".to_vec())), - ], - 1, - ); - - let last_applied = entries.len(); - let mut harness = setup_harness( - Leader as i32, - 1, - entries, - last_applied as u64, - move || false, - Some(4), - ); - harness.run_handler().await; - - // Send commits to trigger processing - for i in 1..=4 { - harness.send_commit(i, Leader as i32).await; - } - - // Verify snapshot triggered - assert!(harness.expect_snapshot_trigger().await); - - // Clean shutdown - harness.shutdown_tx.send(()).unwrap(); - harness.handle.unwrap().await.unwrap(); - } - - /// 2. Test leadership change during batch processing - #[tokio::test] - async fn test_leadership_loss_during_batch() { - let entries = build_entries( - vec![ - CommandType::Command(Bytes::from(b"cmd1".to_vec())), - CommandType::Command(Bytes::from(b"cmd2".to_vec())), - ], - 1, - ); - let last_applied = entries.len(); - let mut harness = setup_harness( - Leader as i32, - 1, - entries, - last_applied as u64, - move || false, - None, - ); - - harness.run_handler().await; - harness.send_commit(1, Leader as i32).await; - harness.send_commit(2, Follower as i32).await; - - // Should not process second command - time::sleep(Duration::from_millis(50)).await; - harness.shutdown_tx.send(()).unwrap(); - harness.handle.unwrap().await.unwrap(); - } - - /// 3. Test config change with node removal (non-self) - #[tokio::test] - async fn test_config_remove_node() { - let entries = build_entries( - vec![CommandType::Configuration(Change::RemoveNode(RemoveNode { - node_id: 2, // Remove different node (not self) - }))], - 1, - ); - - let last_applied = entries.len(); - let mut harness = setup_harness( - Leader as i32, - 1, - entries, - last_applied as u64, - move || false, - Some(4), - ); - - harness.run_handler().await; - harness.send_commit(1, Leader as i32).await; - time::sleep(Duration::from_millis(50)).await; - harness.shutdown_tx.send(()).unwrap(); - harness.handle.unwrap().await.unwrap(); - } - - /// 4. Test self-removal detection logic - /// - /// Lightweight unit test verifying is_self_removal_config() correctly - /// identifies when a node is removing itself from cluster. - /// - /// Related: Issue #200 - #[test] - fn test_is_self_removal_config() { - use d_engine_proto::common::AddNode; - use d_engine_proto::common::MembershipChange; - use d_engine_proto::common::RemoveNode; - use d_engine_proto::common::membership_change::Change; - - // Case 1: Self-removal (my_id matches remove node_id) - let self_removal = MembershipChange { - change: Some(Change::RemoveNode(RemoveNode { node_id: 1 })), - }; - assert!( - DefaultCommitHandler::::is_self_removal_config(1, &self_removal), - "Should detect self-removal when my_id == removed node_id" - ); - - // Case 2: Removing other node (my_id != remove node_id) - let other_removal = MembershipChange { - change: Some(Change::RemoveNode(RemoveNode { node_id: 2 })), - }; - assert!( - !DefaultCommitHandler::::is_self_removal_config(1, &other_removal), - "Should NOT detect self-removal when removing different node" - ); - - // Case 3: AddNode (not a removal) - let add_node = MembershipChange { - change: Some(Change::AddNode(AddNode { - node_id: 1, - address: "127.0.0.1:8080".to_string(), - status: 1, - })), - }; - assert!( - !DefaultCommitHandler::::is_self_removal_config(1, &add_node), - "Should NOT detect self-removal for AddNode changes" - ); - - // Case 4: No change - let no_change = MembershipChange { change: None }; - assert!( - !DefaultCommitHandler::::is_self_removal_config(1, &no_change), - "Should NOT detect self-removal when change is None" - ); - } - - /// 5. Test batch processing under load - #[tokio::test] - async fn test_high_throughput_processing() { - let mut entries = Vec::new(); - for i in 1..=1000 { - entries.push(CommandType::Command(Bytes::from(format!("cmd{i}")))); - } - let entries = build_entries(entries, 1); - - let last_applied = entries.len(); - let mut harness = setup_harness( - Leader as i32, - 1, - entries, - last_applied as u64, - move || false, - None, - ); - harness.run_handler().await; - - // Send all commits at once - for i in 1..=1000 { - harness.send_commit(i, Leader as i32).await; - } - - println!("Sent all commits"); - // Verify snapshot at end - assert!(!harness.expect_snapshot_trigger().await); - harness.shutdown_tx.send(()).unwrap(); - println!("Shutdown sent"); - println!("Waiting for handle to finish"); - harness.handle.unwrap().await.unwrap(); - println!("Handle finished"); - } - - // Case 1: test if process_batch been triggered if - // interval ticks - // - // ## Setup: - // - process_idle_flush_interval_ms = 1ms - - // ## Criterias: - // - apply_batch been triggered twice - // - #[tokio::test] - async fn test_run_case1() { - tokio::time::pause(); - - let entries = build_entries( - vec![ - CommandType::Command(Bytes::from(b"cmd1".to_vec())), - CommandType::Command(Bytes::from(b"cmd2".to_vec())), - ], - 1, - ); - - let last_applied = entries.len(); - let mut harness = setup_harness( - Leader as i32, - 1, - entries, - last_applied as u64, - move || false, - Some(4), - ); - harness.run_handler().await; - - // Send commits to trigger processing - for i in 1..=2 { - harness.send_commit(i, Leader as i32).await; - } - tokio::time::advance(Duration::from_millis(3)).await; - // Clean shutdown - harness.shutdown_tx.send(()).unwrap(); - // Ensure the task completes - harness.handle.unwrap().await.unwrap(); - } - - // Case 2: test if process_batch been triggered if - // batch exceeds threshold - // - // ## Setup: - // - commit_handle_interval_in_ms = 1s - // - send 10 new commit ids - // - batch threshold = 2 - // - // ## Criterias: - // - apply_batch been triggered twice - // - #[tokio::test] - async fn test_run_case2() { - tokio::time::pause(); - - // prepare commit channel - // let (new_commit_tx, new_commit_rx) = mpsc::unbounded_channel::(); - // let (_graceful_tx, graceful_rx) = watch::channel(()); - // let mut handler = setup(batch_thresold, 1000, 2, new_commit_rx, graceful_rx); - let batch_thresold = 10; - let entries = build_entries( - vec![ - CommandType::Command(Bytes::from(b"cmd1".to_vec())), - CommandType::Command(Bytes::from(b"cmd2".to_vec())), - ], - 1, - ); - - let last_applied = entries.len(); - let mut harness = setup_harness( - Leader as i32, - 1, - entries, - last_applied as u64, - move || false, - None, - ); - harness.run_handler().await; - - for i in 1..=batch_thresold { - harness.send_commit(i, Leader as i32).await; - } - tokio::time::advance(Duration::from_millis(2)).await; - tokio::time::sleep(Duration::from_millis(2)).await; - // Clean shutdown - harness.shutdown_tx.send(()).unwrap(); - // Ensure the task completes - harness.handle.unwrap().await.unwrap(); - } - - // Case 3: test if process_batch will not be triggered if - // batch not exceeds threshold, duration not exceeds interval - // - // ## Setup: - // - commit_handle_interval_in_ms = 1s - // - send 10 new commit ids - // - batch threshold = 1000 - // - // ## Criterias: - // - apply_batch been triggered only 1 time - // - #[tokio::test] - async fn test_run_case3() { - tokio::time::pause(); - - // prepare commit channel - // let (new_commit_tx, new_commit_rx) = mpsc::unbounded_channel::(); - // let (_graceful_tx, graceful_rx) = watch::channel(()); - // let mut handler = setup(batch_thresold, 1000, 1, new_commit_rx, graceful_rx); - let batch_thresold = 1000; - let entries = build_entries( - vec![ - CommandType::Command(Bytes::from(b"cmd1".to_vec())), - CommandType::Command(Bytes::from(b"cmd2".to_vec())), - ], - 1, - ); - - let last_applied = entries.len(); - let mut harness = setup_harness( - Leader as i32, - 1, - entries, - last_applied as u64, - move || false, - None, - ); - harness.run_handler().await; - - for i in 1..=(batch_thresold - 10) { - harness.send_commit(i, Leader as i32).await; - } - tokio::time::advance(Duration::from_millis(2)).await; - tokio::time::sleep(Duration::from_millis(2)).await; - // Clean shutdown - harness.shutdown_tx.send(()).unwrap(); - // Ensure the task completes - harness.handle.unwrap().await.unwrap(); - } - - // Case 4: test if process_batch will be triggered if both - // batch exceeds threshold and duration exceeds interval - // - // ## Setup: - // - commit_handle_interval_in_ms = 2ms - // - run for 3ms - // - send 10 new commit ids - // - batch threshold = 10 - // - // ## Criterias: - // - apply_batch been triggered only 3 time - // - #[tokio::test] - async fn test_run_case4() { - tokio::time::pause(); - - // prepare commit channel - // let (new_commit_tx, new_commit_rx) = mpsc::unbounded_channel::(); - // let (_graceful_tx, graceful_rx) = watch::channel(()); - let batch_thresold = 10; - let entries = build_entries( - vec![ - CommandType::Command(Bytes::from(b"cmd1".to_vec())), - CommandType::Command(Bytes::from(b"cmd2".to_vec())), - ], - 1, - ); - - let last_applied = entries.len(); - let mut harness = setup_harness( - Leader as i32, - 1, - entries, - last_applied as u64, - move || false, - None, - ); - harness.run_handler().await; - - for i in 1..=batch_thresold { - harness.send_commit(i, Leader as i32).await; - } - tokio::time::advance(Duration::from_millis(2)).await; - tokio::time::sleep(Duration::from_millis(2)).await; - // Clean shutdown - harness.shutdown_tx.send(()).unwrap(); - // Ensure the task completes - harness.handle.unwrap().await.unwrap(); - } -} - -#[cfg(test)] -mod process_batch_test { - use d_engine_proto::common::AddNode; - use d_engine_proto::common::RemoveNode; - use d_engine_proto::common::membership_change::Change; - use parking_lot::Mutex; - - use super::*; - - // Test helper setup with configurable mocks - // fn setup_test_handler( - // role: i32, - // term: u64, - // entries: Vec, - // config_hook: F, - // command_hook: G, - // snapshot_condition: Option, - // ) -> ( - // DefaultCommitHandler, - // mpsc::Receiver, - // watch::Sender<()>, - // ) where - // F: Fn() -> bool + 'static + Send + Sync, - // G: Fn() -> bool + 'static + Send + Sync - // { - // let (_new_commit_tx, new_commit_rx) = mpsc::unbounded_channel(); - // let (shutdown_tx, shutdown_rx) = watch::channel(()); - // let (event_tx, event_rx) = mpsc::channel(10); - - // // Mock state machine - // let mut mock_smh = MockStateMachineHandler::new(); - // let cloned_entries = entries.clone(); - // mock_smh.expect_pending_range() - // .returning(move || Some(1..=cloned_entries.last().map(|e| e.index).unwrap_or(1))); - // mock_smh.expect_apply_chunk() - // .returning(move |_| { - // if command_hook() { - // Err(Error::Fatal("Command execution failed".to_string())) - // } else { - // Ok(()) - // } - // }); - // mock_smh.expect_should_snapshot() - // .returning(move |data| snapshot_condition.map_or(false, |idx| data.new_commit_index - // >= idx)); - - // // Mock raft log - // let mut mock_log = MockRaftLog::new(); - // mock_log.expect_get_entries_range() - // .return_once(move |_| entries.clone()); - - // // Mock membership - // let mut mock_membership = MockMembership::new(); - // mock_membership.expect_apply_config_change() - // .returning(move |_| if config_hook() { - // Err(Error::Fatal("Command execution failed".to_string())) - // } else { - // Ok(()) - // }); - - // let handler = DefaultCommitHandler::new( - // role, - // term, - // Arc::new(mock_smh), - // Arc::new(mock_log), - // Arc::new(mock_membership), - // new_commit_rx, - // event_tx.clone(), - // 100, // batch_threshold - // 100, // process_interval - // shutdown_rx, - // ); - - // (handler, event_rx, shutdown_tx) - // } - - // Generic entry creation helpers - #[tokio::test] - async fn processes_empty_batch_successfully() { - let last_applied = 0; - let mut harness = setup_harness( - Leader as i32, - 1, - vec![], - last_applied as u64, - move || false, - None, - ); - let result = harness.process_batch_handler().await; - assert!(result.is_ok()); - } - - /// `CommitHandler` sends `ApplyEntries` on its own raw channel handle rather than - /// through `StateMachineCommandSender` (the wrapper every other caller of - /// `StateMachineWorker` goes through) — so a closed-channel failure here is - /// currently classified as `Error::Fatal`, not `SnapshotError::WorkerUnavailable` - /// like every other Worker-unavailable error in the codebase. - /// - /// `spawn_state_machine_commit_listener` (builder.rs) doesn't branch on - /// `is_fatal()` — any `Err` from `run()` just logs and lets the task exit either - /// way — so this discrepancy has no runtime consequence today. It's still the - /// wrong classification: a `StateMachineWorker unavailable` should carry the same - /// error type everywhere, not be Fatal only when CommitHandler happens to hit it. - #[tokio::test] - async fn test_process_batch_reports_worker_unavailable_not_fatal_when_channel_closed() { - let entries = build_entries(vec![CommandType::Command(Bytes::from(b"cmd1".to_vec()))], 1); - let last_applied = entries.len(); - let mut harness = setup_harness( - Leader as i32, - 1, - entries, - last_applied as u64, - move || false, - None, - ); - - let result = harness.process_batch_handler_with_closed_worker_channel().await; - - let err = result.expect_err("process_batch must fail when the SM Worker channel is closed"); - assert!( - !err.is_fatal(), - "a closed SM Worker channel must be reported the same way every other \ - Worker-unavailable error is (non-fatal) — not as Error::Fatal, which is \ - reserved for conditions that require node shutdown" - ); - } - - #[tokio::test] - async fn batches_consecutive_commands() { - let entries = build_entries( - vec![ - CommandType::Command(Bytes::from(b"cmd1".to_vec())), - CommandType::Command(Bytes::from(b"cmd2".to_vec())), - CommandType::Command(Bytes::from(b"cmd3".to_vec())), - ], - 1, - ); - - let last_applied = entries.len(); - let mut harness = setup_harness( - Leader as i32, - 1, - entries, - last_applied as u64, - move || false, - None, - ); - - // Expect single apply_chunk call with all 3 commands - let result = harness.process_batch_handler().await; - assert!(result.is_ok()); - } - - #[tokio::test] - async fn flushes_commands_at_config_change() { - let entries = build_entries( - vec![ - CommandType::Command(Bytes::from(b"cmd1".to_vec())), - CommandType::Configuration(Change::AddNode(AddNode { - node_id: 1, - address: "addr".into(), - status: d_engine_proto::common::NodeStatus::Promotable as i32, - })), - CommandType::Command(Bytes::from(b"cmd2".to_vec())), - ], - 1, - ); - - let last_applied = entries.len(); - let mut harness = setup_harness( - Leader as i32, - 1, - entries, - last_applied as u64, - move || false, - None, - ); - - // Expect single apply_chunk call with all 3 commands - let result = harness.process_batch_handler().await; - assert!(result.is_ok()); - - // Verification: - // - apply_chunk called twice: [cmd1] and [cmd2] - // - apply_config_change called once for entry 2 - } - - #[tokio::test] - async fn flushes_commands_at_noop() { - let entries = build_entries( - vec![ - CommandType::Command(Bytes::from(b"cmd1".to_vec())), - CommandType::Noop, - CommandType::Command(Bytes::from(b"cmd2".to_vec())), - ], - 1, - ); - - let last_applied = entries.len(); - let mut harness = setup_harness( - Leader as i32, - 1, - entries, - last_applied as u64, - move || false, - None, - ); - - // Expect single apply_chunk call with all 3 commands - let result = harness.process_batch_handler().await; - assert!(result.is_ok()); - } - - #[tokio::test] - async fn handles_config_failure_properly() { - let entries = build_entries( - vec![ - CommandType::Configuration(Change::AddNode(AddNode { - node_id: 1, - address: "addr".into(), - status: d_engine_proto::common::NodeStatus::Promotable as i32, - })), - CommandType::Command(Bytes::from(b"cmd1".to_vec())), - ], - 1, - ); - - let last_applied = entries.len(); - let mut harness = setup_harness( - Leader as i32, - 1, - entries, - last_applied as u64, - move || true, // Config will fail - None, - ); - let result = harness.process_batch_handler().await; - assert!(result.is_err()); - - // Verify command was NOT applied - } - - #[tokio::test] - async fn processes_mixed_entries_correctly() { - let entries = build_entries( - vec![ - CommandType::Command(Bytes::from(b"cmd1".to_vec())), - CommandType::Configuration(Change::AddNode(AddNode { - node_id: 1, - address: "addr".into(), - status: d_engine_proto::common::NodeStatus::Promotable as i32, - })), - CommandType::Noop, - CommandType::Command(Bytes::from(b"cmd2".to_vec())), - CommandType::Command(Bytes::from(b"cmd3".to_vec())), - ], - 1, - ); - - let last_applied = entries.len(); - let mut harness = setup_harness( - Leader as i32, - 1, - entries, - last_applied as u64, - move || false, - None, - ); - let result = harness.process_batch_handler().await; - assert!(result.is_ok()); - // Verification: - // - apply_config_change called once for entry 2 - // - apply_chunk called twice: [cmd1] and [cmd2, cmd3] - } - - #[tokio::test] - async fn handles_large_command_batches() { - let mut r = vec![]; - for _i in 1..=1000 { - r.push(CommandType::Command(Bytes::from(b"cmd".to_vec()))) - } - let entries = build_entries(r, 1); - let last_applied = entries.len(); - let mut harness = setup_harness( - Leader as i32, - 1, - entries, - last_applied as u64, - move || false, - None, - ); - let result = harness.process_batch_handler().await; - assert!(result.is_ok()); - - // Verification: apply_chunk called once with 1000 commands - } - - /// Test 1: MembershipApplied event MUST be sent after successful config change - /// - /// Verifies that when apply_config_change() succeeds, the commit handler - /// sends InternalEvent::MembershipApplied to notify Leader to refresh cache. - /// - /// Related: Bug fix #209 - cluster metadata cache timing issue - #[tokio::test] - async fn membership_applied_event_sent_on_success() { - let entries = build_entries( - vec![CommandType::Configuration(Change::AddNode(AddNode { - node_id: 2, - address: "127.0.0.1:8080".into(), - status: d_engine_proto::common::NodeStatus::Promotable as i32, - }))], - 1, - ); - - let last_applied = entries.len(); - let mut harness = setup_harness( - Leader as i32, - 1, - entries, - last_applied as u64, - || false, // Config succeeds - None, - ); - - let result = harness.process_batch_handler().await; - assert!(result.is_ok(), "Config change should succeed"); - - // Verify MembershipApplied event was sent - match tokio::time::timeout(Duration::from_millis(50), harness.internal_event_rx.recv()) - .await - { - Ok(Some(InternalEvent::MembershipApplied)) => { - // Success - event received - } - Ok(Some(other)) => panic!("Expected MembershipApplied, got {other:?}"), - Ok(None) => panic!("Event channel closed unexpectedly"), - Err(_) => panic!("Timeout waiting for MembershipApplied event"), - } - } - - /// Test 2: Event order - MembershipApplied sent AFTER apply_config_change - /// - /// Verifies the critical ordering guarantee: - /// 1. apply_config_change() updates membership state - /// 2. notify_config_applied() barrier - /// 3. MembershipApplied event sent - /// - /// This ensures Leader refreshes cache with up-to-date membership. - /// - /// Related: Bug fix #209 - #[tokio::test] - async fn membership_applied_event_sent_after_apply() { - let apply_order = Arc::new(Mutex::new(Vec::new())); - let order_clone = apply_order.clone(); - - let entries = build_entries( - vec![CommandType::Configuration(Change::AddNode(AddNode { - node_id: 2, - address: "127.0.0.1:8080".into(), - status: d_engine_proto::common::NodeStatus::Promotable as i32, - }))], - 1, - ); - - let last_applied = entries.len(); - let mut harness = setup_harness( - Leader as i32, - 1, - entries, - last_applied as u64, - { - let order = order_clone.clone(); - move || { - order.lock().push("apply_config_change"); - false - } - }, - None, - ); - - harness.process_batch_handler().await.unwrap(); - - // Verify event received - match tokio::time::timeout(Duration::from_millis(50), harness.internal_event_rx.recv()) - .await - { - Ok(Some(InternalEvent::MembershipApplied)) => { - // Record event reception - apply_order.lock().push("MembershipApplied_event"); - } - _ => panic!("Expected MembershipApplied event"), - } - - // Verify ordering: apply first, then event - let order = apply_order.lock(); - assert_eq!( - *order, - vec!["apply_config_change", "MembershipApplied_event"], - "Event must be sent AFTER apply_config_change" - ); - } - - /// Test 3: NO event sent when config change fails - /// - /// Verifies that when apply_config_change() returns an error, - /// MembershipApplied event is NOT sent (membership state is unchanged). - /// - /// Related: Bug fix #209 - #[tokio::test] - async fn no_membership_applied_event_on_config_failure() { - let entries = build_entries( - vec![CommandType::Configuration(Change::AddNode(AddNode { - node_id: 2, - address: "127.0.0.1:8080".into(), - status: d_engine_proto::common::NodeStatus::Promotable as i32, - }))], - 1, - ); - - let last_applied = entries.len(); - let mut harness = setup_harness( - Leader as i32, - 1, - entries, - last_applied as u64, - || true, // Config fails - None, - ); - - let result = harness.process_batch_handler().await; - assert!(result.is_err(), "Config change should fail"); - - // Verify NO event was sent - match tokio::time::timeout(Duration::from_millis(50), harness.internal_event_rx.recv()) - .await - { - Err(_) => { - // Timeout is expected - no event sent - } - Ok(Some(event)) => panic!("Expected no event, but received {event:?}"), - Ok(None) => { - // Channel closed is also acceptable - } - } - } - - /// Test 5: All config change types trigger MembershipApplied event - /// - /// Verifies that AddNode, RemoveNode, and Promote all send the event. - /// - /// Related: Bug fix #209 - #[tokio::test] - async fn all_config_change_types_send_membership_applied() { - use d_engine_proto::common::NodeStatus; - use d_engine_proto::common::PromoteLearner; - - // Test AddNode - let entries = build_entries( - vec![CommandType::Configuration(Change::AddNode(AddNode { - node_id: 2, - address: "127.0.0.1:8080".into(), - status: d_engine_proto::common::NodeStatus::Promotable as i32, - }))], - 1, - ); - let mut harness = setup_harness(Leader as i32, 1, entries, 1, || false, None); - harness.process_batch_handler().await.unwrap(); - assert!( - matches!( - tokio::time::timeout(Duration::from_millis(50), harness.internal_event_rx.recv()) - .await, - Ok(Some(InternalEvent::MembershipApplied)) - ), - "AddNode should send MembershipApplied" - ); - - // Test RemoveNode - let entries = build_entries( - vec![CommandType::Configuration(Change::RemoveNode(RemoveNode { - node_id: 2, - }))], - 1, - ); - let mut harness = setup_harness(Leader as i32, 1, entries, 1, || false, None); - harness.process_batch_handler().await.unwrap(); - assert!( - matches!( - tokio::time::timeout(Duration::from_millis(50), harness.internal_event_rx.recv()) - .await, - Ok(Some(InternalEvent::MembershipApplied)) - ), - "RemoveNode should send MembershipApplied" - ); - - // Test PromoteLearner - let entries = build_entries( - vec![CommandType::Configuration(Change::Promote( - PromoteLearner { - node_id: 2, - status: NodeStatus::Active as i32, - }, - ))], - 1, - ); - let mut harness = setup_harness(Leader as i32, 1, entries, 1, || false, None); - harness.process_batch_handler().await.unwrap(); - assert!( - matches!( - tokio::time::timeout(Duration::from_millis(50), harness.internal_event_rx.recv()) - .await, - Ok(Some(InternalEvent::MembershipApplied)) - ), - "PromoteLearner should send MembershipApplied" - ); - } - - /// Dropping the event receiver makes event_tx.send() fail. - /// The MembershipApplied warn path must be executed without panicking or returning Err. - #[tokio::test] - async fn membership_applied_send_fails_gracefully_when_receiver_dropped() { - let entries = build_entries( - vec![CommandType::Configuration(Change::AddNode(AddNode { - node_id: 2, - address: "127.0.0.1:8080".into(), - status: d_engine_proto::common::NodeStatus::Promotable as i32, - }))], - 1, - ); - - let last_applied = entries.len(); - let mut harness = setup_harness( - Leader as i32, - 1, - entries, - last_applied as u64, - || false, - None, - ); - - // Replace the receiver with a dummy so the original event_tx sends will fail. - // Swap old_rx out of the harness so harness is still usable, then drop old_rx. - let (_dummy_tx, dummy_rx) = mpsc::unbounded_channel::(); - let old_rx = std::mem::replace(&mut harness.internal_event_rx, dummy_rx); - drop(old_rx); // Now harness.event_tx's peer receiver is gone → sends fail. - - let result = harness.process_batch_handler().await; - // Config change itself succeeds; send failure is non-fatal (warn only). - assert!( - result.is_ok(), - "send failure on MembershipApplied must not propagate as Err" - ); - } - - /// Dropping the event receiver makes StepDownSelfRemoved send fail. - /// The error! path inside the self-removal branch must be executed without panicking. - #[tokio::test] - async fn step_down_send_fails_gracefully_when_receiver_dropped() { - let entries = build_entries( - vec![CommandType::Configuration(Change::RemoveNode(RemoveNode { - node_id: 1, // Self-removal (node_id = 1 matches my_id = 1) - }))], - 1, - ); - - let last_applied = entries.len(); - let mut harness = setup_harness( - Leader as i32, - 1, - entries, - last_applied as u64, - || false, - None, - ); - - // Replace receiver with a dummy so both sends fail. - let (_dummy_tx, dummy_rx) = mpsc::unbounded_channel::(); - let old_rx = std::mem::replace(&mut harness.internal_event_rx, dummy_rx); - drop(old_rx); - - let result = harness.process_batch_handler().await; - // Self-removal is committed; send failure is non-fatal (error! logged, not returned). - assert!( - result.is_ok(), - "send failure on StepDownSelfRemoved must not propagate as Err" - ); - } - - /// Test 6: Self-removal sends both MembershipApplied AND StepDownSelfRemoved - /// - /// Verifies that when Leader removes itself: - /// 1. MembershipApplied is sent (to refresh cache) - /// 2. StepDownSelfRemoved is sent (to trigger step down) - /// - /// Order: MembershipApplied should be sent before StepDownSelfRemoved - /// - /// Related: Bug fix #209, Issue #200 (self-removal) - #[tokio::test] - async fn self_removal_sends_both_events() { - let entries = build_entries( - vec![CommandType::Configuration(Change::RemoveNode(RemoveNode { - node_id: 1, // Remove self (node_id = 1) - }))], - 1, - ); - - let last_applied = entries.len(); - let mut harness = setup_harness( - Leader as i32, - 1, - entries, - last_applied as u64, - || false, - None, - ); - - harness.process_batch_handler().await.unwrap(); - - // First event should be MembershipApplied - match tokio::time::timeout(Duration::from_millis(50), harness.internal_event_rx.recv()) - .await - { - Ok(Some(InternalEvent::MembershipApplied)) => { - // Correct first event - } - Ok(Some(other)) => panic!("Expected MembershipApplied first, got {other:?}"), - _ => panic!("Expected MembershipApplied event"), - } - - // Second event should be StepDownSelfRemoved - match tokio::time::timeout(Duration::from_millis(50), harness.internal_event_rx.recv()) - .await - { - Ok(Some(InternalEvent::StepDownSelfRemoved)) => { - // Correct second event - } - Ok(Some(other)) => panic!("Expected StepDownSelfRemoved second, got {other:?}"), - _ => panic!("Expected StepDownSelfRemoved event"), - } - } -} diff --git a/d-engine-core/src/commit_handler/mod.rs b/d-engine-core/src/commit_handler/mod.rs deleted file mode 100644 index f7bc1f0a..00000000 --- a/d-engine-core/src/commit_handler/mod.rs +++ /dev/null @@ -1,60 +0,0 @@ -//! # Commit Handler Module -//! -//! The `commit_handler` module is responsible for processing newly committed -//! log entries in a Raft cluster. It acts as a bridge between the Raft -//! consensus protocol and the application-specific business logic. When a new -//! log entry is committed by the Raft leader and replicated to a majority of -//! nodes, the commit handler ensures that the entry is applied to the state -//! machine in a consistent and thread-safe manner. -//! -//! ## Key Responsibilities -//! 1. **Receiving Commits**: Listens for newly committed log entries from the Raft log. -//! 2. **Batching**: Groups multiple log entries into batches for efficient processing. -//! 3. **Applying to State Machine**: Passes the committed entries to the state machine for -//! application-specific processing. -//! 4. **Snapshotting**: Periodically triggers snapshots to reduce the size of the Raft log and -//! improve recovery performance. -//! -//! ## Raft Protocol Integration -//! - The commit handler adheres to the Raft protocol by ensuring that committed entries are applied -//! in the exact order they were committed. -//! - It guarantees linearizability by applying entries only after they are committed and replicated -//! to a majority of nodes. -//! -//! ## Best Practices -//! - **Batching**: Use batching to reduce the overhead of applying individual log entries. -//! - **Concurrency**: Apply log entries concurrently (if safe) to improve throughput. -//! - **Snapshotting**: Regularly take snapshots to prevent the Raft log from growing indefinitely. -//! - **Error Handling**: Handle errors gracefully and ensure the state machine remains consistent -//! even in the face of failures. -//! -//! ## Customization for Business Logic -//! - The actual implementation of how to use the committed log entries is left to the developer -//! (business layer). The commit handler provides a framework for processing entries, but the -//! business logic must define how to interpret and apply them. -//! - Developers should implement the `StateMachine` trait to define how log entries are applied to -//! the application state. -//! -//! ## Example Workflow -//! 1. The Raft leader commits a new log entry and replicates it to a majority of nodes. -//! 2. The commit handler receives the committed entry and adds it to a batch. -//! 3. Once the batch reaches a certain size or a timeout occurs, the handler applies the batch to -//! the state machine. -//! 4. The state machine processes the entries and updates the application state. -//! 5. Periodically, the commit handler triggers a snapshot(TODO: in future release) to compact the -//! Raft log. -//! -//! ## Notes -//! - The commit handler is designed to be thread-safe and can be used in a multi-threaded or -//! asynchronous environment. -//! - Developers must ensure that the state machine implementation is idempotent to handle potential -//! retries or re-applications of log entries. -mod default_commit_handler; -pub use default_commit_handler::*; - -#[cfg_attr(any(test, feature = "__test_support"), mockall::automock)] -#[doc(hidden)] -#[async_trait::async_trait] -pub trait CommitHandler: Send + Sync + 'static { - async fn run(&mut self) -> crate::Result<()>; -} diff --git a/d-engine-core/src/config/config_test.rs b/d-engine-core/src/config/config_test.rs index 76cc38d3..334956e5 100644 --- a/d-engine-core/src/config/config_test.rs +++ b/d-engine-core/src/config/config_test.rs @@ -1,9 +1,11 @@ -use super::*; +use std::io::Write; + use config::ConfigError; use serial_test::serial; -use std::io::Write; use temp_env::with_vars; +use super::*; + fn cleanup_all_raft_env_vars() { for (key, _) in std::env::vars() { if key.starts_with("RAFT__") || key == "CONFIG_PATH" { diff --git a/d-engine-core/src/config/mod.rs b/d-engine-core/src/config/mod.rs index b8a3db5e..b9ecbbb8 100644 --- a/d-engine-core/src/config/mod.rs +++ b/d-engine-core/src/config/mod.rs @@ -30,14 +30,16 @@ mod raft_test; #[cfg(test)] mod tls_test; -use crate::Result; +use std::env; +use std::fmt::Debug; + use config::Config; use config::Environment; use config::File; use serde::Deserialize; use serde::Serialize; -use std::env; -use std::fmt::Debug; + +use crate::Result; /// Main configuration container for Raft consensus engine components /// diff --git a/d-engine-core/src/config/raft.rs b/d-engine-core/src/config/raft.rs index badd011c..67a8e661 100644 --- a/d-engine-core/src/config/raft.rs +++ b/d-engine-core/src/config/raft.rs @@ -39,7 +39,7 @@ pub struct RaftConfig { /// Configuration settings for state machine behavior /// Controls state machine operations like lease management, compaction, etc. /// For backward compatibility, can also be configured via `storage` in TOML files. - #[serde(default, alias = "storage")] + #[serde(default)] pub state_machine: StateMachineConfig, /// Configuration settings for snapshot feature @@ -353,8 +353,7 @@ impl ReplicationConfig { /// A single value intentionally covers all drain loops because they all operate /// within the same heartbeat period and share the same order-of-magnitude concurrency: /// - `raft.rs` cmd_rx drain (client propose ingestion) -/// - `raft.rs` internal_event_rx drain (commit index event coalescing) -/// - `DefaultCommitHandler` new_commit_rx drain (state machine apply batching) +/// - `raft.rs` internal_event_rx drain /// /// Also used as the initial Vec capacity for propose and linearizable-read buffers /// (construction-time hint only; the propose buffer retains capacity across flushes @@ -495,7 +494,8 @@ pub struct MembershipConfig { #[serde(default)] pub promotion: PromotionConfig, - /// Bound on how long a voter waits at startup for quorum to become reachable before failing fast; independent of retry.membership. + /// Bound on how long a voter waits at startup for quorum to become reachable before failing + /// fast; independent of retry.membership. #[serde(default = "default_startup_quorum_timeout")] pub startup_quorum_timeout: Duration, } @@ -524,7 +524,8 @@ fn default_verify_leadership_persistent_timeout() -> Duration { Duration::from_secs(3600) } -/// Default 30s — enough for peers to finish their own bootstrap race, short enough to fail fast if truly unreachable. +/// Default 30s — enough for peers to finish their own bootstrap race, short enough to fail fast if +/// truly unreachable. fn default_startup_quorum_timeout() -> Duration { Duration::from_secs(30) } @@ -551,22 +552,46 @@ impl MembershipConfig { /// and other data lifecycle features. This configuration affects how the state machine /// processes applied log entries and manages data. #[derive(Serialize, Deserialize, Clone, Debug)] -#[derive(Default)] pub struct StateMachineConfig { /// Lease (time-based expiration) configuration /// /// For backward compatibility, can also be configured via `ttl` in TOML files. #[serde(alias = "ttl")] pub lease: LeaseConfig, + + /// Maximum time to wait, on shutdown, for the state machine worker to finish + /// applying the entries already handed to it. On timeout an error is logged and + /// shutdown proceeds: entries the node knew as committed may then be unapplied + /// until the next start. Raise it for state machines with slow batch applies. + #[serde(default = "default_shutdown_apply_timeout_ms")] + pub shutdown_apply_timeout_ms: u64, } impl StateMachineConfig { pub fn validate(&self) -> Result<()> { self.lease.validate()?; + if self.shutdown_apply_timeout_ms == 0 { + return Err(Error::Config(ConfigError::Message( + "shutdown_apply_timeout_ms must be greater than 0".into(), + ))); + } Ok(()) } } +fn default_shutdown_apply_timeout_ms() -> u64 { + 5_000 +} + +impl Default for StateMachineConfig { + fn default() -> Self { + Self { + lease: LeaseConfig::default(), + shutdown_apply_timeout_ms: default_shutdown_apply_timeout_ms(), + } + } +} + /// Submit processor-specific configuration #[derive(Debug, Serialize, Deserialize, Clone)] pub struct SnapshotConfig { @@ -1092,8 +1117,8 @@ pub struct ReadConsistencyConfig { /// The lease deadline is anchored to heartbeat *send* time, but the follower's /// election timer resets at *receive* time (~rtt/2 later). Without accounting for /// rtt/2 the safety invariant is violated under network latency — confirmed by - /// Jepsen set workload (19 elements lost). etcd absorbs this via `electionTimeout × 2/3`; - /// d-engine makes it explicit via `network_rtt_p99_ms`. + /// Jepsen set workload (19 elements lost). The follower timer already tolerates part of that + /// delay; this setting makes the allowance explicit via `network_rtt_p99_ms`. #[serde(default = "default_lease_duration_ms")] pub lease_duration_ms: u64, diff --git a/d-engine-core/src/election/election_handler.rs b/d-engine-core/src/election/election_handler.rs index 3b6c084b..a8495f26 100644 --- a/d-engine-core/src/election/election_handler.rs +++ b/d-engine-core/src/election/election_handler.rs @@ -1,12 +1,13 @@ +use std::collections::HashSet; +use std::fmt::Debug; +use std::marker::PhantomData; +use std::sync::Arc; + use async_trait::async_trait; use d_engine_proto::common::LogId; use d_engine_proto::server::election::VoteRequest; use d_engine_proto::server::election::VoteResponse; use d_engine_proto::server::election::VotedFor; -use std::collections::HashSet; -use std::fmt::Debug; -use std::marker::PhantomData; -use std::sync::Arc; use tokio::task::JoinSet; use tracing::debug; use tracing::error; @@ -433,8 +434,7 @@ where } // A denial is just a missing vote: keep counting. A peer with a - // strictly newer log is only worth noting (openraft records it to - // delay the next election; raft-rs ignores it). + // strictly newer log is only worth noting. if is_target_log_strictly_more_recent( last_log_index, last_log_term, @@ -457,7 +457,7 @@ where // Even if every peer still outstanding granted, could a majority be reached? // If not the round is lost: return now instead of waiting for silent peers - // (raft-rs reports `Lost` at this point). + // (the round is reported as `Lost` at this point). if !is_majority(succeed + pending, required) { return Err(ElectionError::QuorumFailure { required, succeed }.into()); } diff --git a/d-engine-core/src/election/election_handler_test.rs b/d-engine-core/src/election/election_handler_test.rs index c91ded5a..b9a69aae 100644 --- a/d-engine-core/src/election/election_handler_test.rs +++ b/d-engine-core/src/election/election_handler_test.rs @@ -812,9 +812,9 @@ mod single_node_election_tests { /// - Peer denies the vote and reports last_log_term=4 (higher than candidate's) /// /// Expected: - /// - Returns ElectionError::QuorumFailure with only the candidate's own vote: a peer with - /// a more recent log term denies, so the candidate must not win this election. - /// (A denial is just a missing vote; there is no separate log-conflict outcome.) + /// - Returns ElectionError::QuorumFailure with only the candidate's own vote: a peer with a + /// more recent log term denies, so the candidate must not win this election. (A denial is + /// just a missing vote; there is no separate log-conflict outcome.) #[tokio::test] async fn test_broadcast_vote_requests_rejects_on_peer_log_higher_term() { let election_handler = ElectionHandler::::new(1); @@ -901,7 +901,8 @@ mod single_node_election_tests { /// successfully obtains majority votes and can transition to Leader role. /// /// Original test location: - /// `d-engine-server/tests/components/election/election_handler_test.rs::test_broadcast_vote_requests_case3` + /// `d-engine-server/tests/components/election/election_handler_test. + /// rs::test_broadcast_vote_requests_case3` #[tokio::test] async fn test_broadcast_vote_requests_wins_election_with_majority_votes() { let election_handler = ElectionHandler::::new(1); @@ -1091,7 +1092,6 @@ mod single_node_election_tests { type SM = crate::MockStateMachine; type M = MockMembership; type REP = crate::MockReplicationCore; - type C = crate::MockCommitHandler; type SMH = crate::MockStateMachineHandler; type SMW = crate::MockStateMachineWriterOps; type SNP = crate::MockSnapshotPolicy; @@ -1171,10 +1171,10 @@ mod single_node_election_tests { /// - Own vote + peer 3 is at most 2 of 5, so the round is already lost. /// /// Expected: - /// - QuorumFailure right after the third denial, without waiting for peer 3. The caller - /// awaits this inside the Raft loop; waiting for a dead peer keeps the node from - /// handling the heartbeat of whoever won, and (with PreVote) delays the next attempt. - /// raft-rs reports `Lost` as soon as the rejections make a majority impossible. + /// - QuorumFailure right after the third denial, without waiting for peer 3. The caller awaits + /// this inside the Raft loop; waiting for a dead peer keeps the node from handling the + /// heartbeat of whoever won, and (with PreVote) delays the next attempt. The round must + /// report `Lost` as soon as the rejections make a majority impossible. #[tokio::test(start_paused = true)] async fn test_round_returns_quorum_failure_once_majority_is_impossible() { for pre_vote in [false, true] { @@ -1253,8 +1253,8 @@ mod single_node_election_tests { /// - Peer denies the vote and reports term=2 (higher than the candidate's) /// /// Expected: - /// - Returns ElectionError::HigherTerm(2), signaling the candidate must - /// step down rather than continue the election (Raft §5.1) + /// - Returns ElectionError::HigherTerm(2), signaling the candidate must step down rather than + /// continue the election (Raft §5.1) #[tokio::test] async fn test_broadcast_vote_requests_steps_down_on_peer_higher_term() { let election_handler = ElectionHandler::::new(1); @@ -1406,12 +1406,12 @@ mod single_node_election_tests { /// /// Scenario: /// - Two-node cluster (candidate + 1 peer) - /// - Peer denies the vote with the same term and a strictly older log — - /// neither HigherTerm nor LogConflict applies + /// - Peer denies the vote with the same term and a strictly older log — neither HigherTerm nor + /// LogConflict applies /// /// Expected: - /// - Returns ElectionError::QuorumFailure { required: 2, succeed: 1 } — - /// only the candidate's own vote counts, 1/2 is not a majority + /// - Returns ElectionError::QuorumFailure { required: 2, succeed: 1 } — only the candidate's + /// own vote counts, 1/2 is not a majority #[tokio::test] async fn test_broadcast_vote_requests_fails_quorum_on_plain_denial() { let election_handler = ElectionHandler::::new(1); @@ -1504,7 +1504,8 @@ mod single_node_election_tests { /// with higher term and at-least-as-up-to-date log. /// /// Original test location: - /// `d-engine-server/tests/components/election/election_handler_test.rs::test_handle_vote_request_case1` + /// `d-engine-server/tests/components/election/election_handler_test. + /// rs::test_handle_vote_request_case1` #[tokio::test] async fn test_handle_vote_request_grants_vote_for_valid_higher_term() { let election_handler = ElectionHandler::::new(1); @@ -1573,7 +1574,8 @@ mod single_node_election_tests { /// preventing stale candidates from disrupting the cluster. /// /// Original test location: - /// `d-engine-server/tests/components/election/election_handler_test.rs::test_handle_vote_request_case2` + /// `d-engine-server/tests/components/election/election_handler_test. + /// rs::test_handle_vote_request_case2` #[tokio::test] async fn test_handle_vote_request_rejects_vote_for_lower_term() { let election_handler = ElectionHandler::::new(1); @@ -1646,10 +1648,12 @@ mod single_node_election_tests { /// - Reason: Current term is not less than request term /// /// Original test location: - /// `d-engine-server/tests/components/election/election_handler_test.rs::test_check_vote_request_is_legal_case_1_1` + /// `d-engine-server/tests/components/election/election_handler_test. + /// rs::test_check_vote_request_is_legal_case_1_1` #[tokio::test] async fn test_check_vote_request_is_legal_rejects_when_current_term_not_lower() { - // TODO: Simplify to just `let election_handler = ElectionHandler::::new(1);` + // TODO: Simplify to just `let election_handler = + // ElectionHandler::::new(1);` let _temp_dir = tempfile::tempdir().expect("Failed to create temp dir"); let node_config = RaftNodeConfig::new().expect("Should create default config"); let _node_config = node_config.validate().expect("Should validate config"); @@ -1716,7 +1720,8 @@ mod single_node_election_tests { /// - Reason: Request log term is not more recent than local /// /// Original test location: - /// `d-engine-server/tests/components/election/election_handler_test.rs::test_check_vote_request_is_legal_case_1_2` + /// `d-engine-server/tests/components/election/election_handler_test. + /// rs::test_check_vote_request_is_legal_case_1_2` #[tokio::test] async fn test_check_vote_request_is_legal_rejects_when_request_log_not_more_recent() { let _temp_dir = tempfile::tempdir().expect("Failed to create temp dir"); @@ -1785,7 +1790,8 @@ mod single_node_election_tests { /// - Reason: Request has same term but higher index (more up-to-date) /// /// Original test location: - /// `d-engine-server/tests/components/election/election_handler_test.rs::test_check_vote_request_is_legal_case_1_3` + /// `d-engine-server/tests/components/election/election_handler_test. + /// rs::test_check_vote_request_is_legal_case_1_3` #[tokio::test] async fn test_check_vote_request_is_legal_accepts_when_request_log_more_recent() { let _temp_dir = tempfile::tempdir().expect("Failed to create temp dir"); @@ -1831,7 +1837,8 @@ mod single_node_election_tests { /// - Reason: Local log has more entries (higher index) in same term /// /// Original test location: - /// `d-engine-server/tests/components/election/election_handler_test.rs::test_check_vote_request_is_legal_case_1_4` + /// `d-engine-server/tests/components/election/election_handler_test. + /// rs::test_check_vote_request_is_legal_case_1_4` #[tokio::test] async fn test_check_vote_request_is_legal_rejects_when_local_log_more_recent() { let _temp_dir = tempfile::tempdir().expect("Failed to create temp dir"); @@ -1884,7 +1891,8 @@ mod single_node_election_tests { /// This validates the Raft rule: at most one vote per term. /// /// Original test location: - /// `d-engine-server/tests/components/election/election_handler_test.rs::test_check_vote_request_is_legal_case_2_1` + /// `d-engine-server/tests/components/election/election_handler_test. + /// rs::test_check_vote_request_is_legal_case_2_1` #[tokio::test] async fn test_check_vote_request_is_legal_rejects_when_already_voted_for_different_candidate() { let _temp_dir = tempfile::tempdir().expect("Failed to create temp dir"); @@ -1936,7 +1944,8 @@ mod single_node_election_tests { /// - Reason: Already voted in a higher term (should not happen in normal operation) /// /// Original test location: - /// `d-engine-server/tests/components/election/election_handler_test.rs::test_check_vote_request_is_legal_case_2_2` + /// `d-engine-server/tests/components/election/election_handler_test. + /// rs::test_check_vote_request_is_legal_case_2_2` #[tokio::test] async fn test_check_vote_request_is_legal_rejects_when_voted_in_higher_term() { let _temp_dir = tempfile::tempdir().expect("Failed to create temp dir"); @@ -1991,7 +2000,8 @@ mod single_node_election_tests { /// again in a new term. /// /// Original test location: - /// `d-engine-server/tests/components/election/election_handler_test.rs::test_check_vote_request_is_legal_case_2_3` + /// `d-engine-server/tests/components/election/election_handler_test. + /// rs::test_check_vote_request_is_legal_case_2_3` #[tokio::test] async fn test_check_vote_request_is_legal_accepts_revote_for_same_candidate_new_term() { let _temp_dir = tempfile::tempdir().expect("Failed to create temp dir"); @@ -2150,8 +2160,8 @@ mod single_node_election_tests { /// - Three nodes, both peers unreachable (the node is isolated). /// /// Expected: - /// - QuorumFailure. If errors counted as grants, an isolated node would pass PreVote on - /// its own, bump its term every round, and PreVote would protect nothing. + /// - QuorumFailure. If errors counted as grants, an isolated node would pass PreVote on its + /// own, bump its term every round, and PreVote would protect nothing. #[tokio::test] async fn test_broadcast_pre_vote_requests_does_not_count_unreachable_peers_as_grants() { let mut transport = MockTransport::new(); @@ -2233,14 +2243,14 @@ mod single_node_election_tests { /// /// Scenario: /// - The node's real term is 1, so the PreVote asks about term 2. - /// - The peer is already at term 2 (an election happened while this node was away) and - /// denies, reporting term 2. + /// - The peer is already at term 2 (an election happened while this node was away) and denies, + /// reporting term 2. /// /// Expected: - /// - HigherTerm(2). Comparing the peer's term with the hypothetical term (2) using a strict - /// `>` would call this a plain denial: the node would never learn it is behind and would - /// keep retrying PreVote at a term the cluster has already passed. The peer's term must - /// be compared with the node's real term (the PreVote term minus 1). + /// - HigherTerm(2). Comparing the peer's term with the hypothetical term (2) using a strict `>` + /// would call this a plain denial: the node would never learn it is behind and would keep + /// retrying PreVote at a term the cluster has already passed. The peer's term must be + /// compared with the node's real term (the PreVote term minus 1). #[tokio::test] async fn test_broadcast_pre_vote_requests_reports_peer_at_pre_vote_term_as_higher() { let mut transport = MockTransport::new(); @@ -2263,8 +2273,8 @@ mod single_node_election_tests { /// Test: a peer at the node's real term is a plain denial, not a reason to step down. /// /// Scenario: - /// - The node's real term is 1 (PreVote term 2); the peer is also at term 1 and denies - /// because it still hears from a leader. + /// - The node's real term is 1 (PreVote term 2); the peer is also at term 1 and denies because + /// it still hears from a leader. /// /// Expected: /// - QuorumFailure, so the node keeps its term and simply retries after the next timeout. @@ -2295,8 +2305,6 @@ mod single_node_election_tests { // Both rounds use the same counting loop, so every scenario runs against both. A denial is // just a missing vote: the round only ends when a majority is reached, when a peer reveals // a higher term (the node must catch up at once), or when every peer has answered. - // Reference scenarios: raft-rs `test_dueling_candidates`, `test_prevote_with_split_vote`; - // openraft `handle_vote_resp_test` / `handle_pre_vote_resp_test` ("reject keeps waiting"). // ======================================================================== #[derive(Clone, Copy, Debug)] @@ -2463,12 +2471,12 @@ mod single_node_election_tests { /// Test: a denial from a peer whose log equals ours is just a missing vote. /// /// Scenario (3 nodes, steady state, all logs equal): - /// - Peer 2 denies (it already voted for someone else in this term, or still hears a - /// leader); peer 3 grants. + /// - Peer 2 denies (it already voted for someone else in this term, or still hears a leader); + /// peer 3 grants. /// /// Expected: - /// - Ok(()): own vote + peer 3 is a majority. An equal log is not a "more recent" log, so - /// the denial must not end the round and discard peer 3's grant. + /// - Ok(()): own vote + peer 3 is a majority. An equal log is not a "more recent" log, so the + /// denial must not end the round and discard peer 3's grant. #[tokio::test] async fn test_round_denial_with_equal_log_is_just_a_missing_vote() { for round in BOTH_ROUNDS { @@ -2480,9 +2488,8 @@ mod single_node_election_tests { /// Test: a denial from a peer whose log is strictly newer is also just a missing vote. /// /// Expected: - /// - Ok(()) when another peer grants. A peer's newer log does not stop a majority of - /// other peers (each already checked our log against theirs) from electing us. openraft - /// and raft-rs never end the round on a log comparison. + /// - Ok(()) when another peer grants. A peer's newer log does not stop a majority of other + /// peers (each already checked our log against theirs) from electing us. #[tokio::test] async fn test_round_denial_with_newer_log_is_just_a_missing_vote() { for round in BOTH_ROUNDS { @@ -2515,20 +2522,17 @@ mod single_node_election_tests { /// Test: five nodes, two denials (equal and newer log) and two grants still win. /// - /// Scenario (raft-rs `test_dueling_candidates`: "a rejection is not the end of the round"): + /// Scenario ("a rejection is not the end of the round"): /// - Own vote + 2 grants = 3 of 5. #[tokio::test] async fn test_round_five_nodes_two_denials_and_two_grants_wins() { for round in BOTH_ROUNDS { - let result = run_round( - round, - vec![ - (2, deny_equal_log()), - (3, deny_newer_log()), - (4, Reply::Grant), - (5, Reply::Grant), - ], - ) + let result = run_round(round, vec![ + (2, deny_equal_log()), + (3, deny_newer_log()), + (4, Reply::Grant), + (5, Reply::Grant), + ]) .await; assert!(result.is_ok(), "{round:?}: got {result:?}"); } @@ -2542,15 +2546,12 @@ mod single_node_election_tests { #[tokio::test] async fn test_round_five_nodes_three_denials_fail_quorum() { for round in BOTH_ROUNDS { - let result = run_round( - round, - vec![ - (2, deny_equal_log()), - (3, deny_newer_log()), - (4, deny_older_log()), - (5, Reply::Grant), - ], - ) + let result = run_round(round, vec![ + (2, deny_equal_log()), + (3, deny_newer_log()), + (4, deny_older_log()), + (5, Reply::Grant), + ]) .await; assert!(is_quorum_failure(&result), "{round:?}: got {result:?}"); } @@ -2562,8 +2563,8 @@ mod single_node_election_tests { /// - The peer's log is equal, newer, or older than ours. /// /// Expected: - /// - QuorumFailure in all three cases. The node could not win, which is all the caller - /// needs to know; there is no separate "log conflict" outcome. + /// - QuorumFailure in all three cases. The node could not win, which is all the caller needs to + /// know; there is no separate "log conflict" outcome. #[tokio::test] async fn test_round_single_peer_denial_is_quorum_failure_for_any_log() { for round in BOTH_ROUNDS { @@ -2580,24 +2581,19 @@ mod single_node_election_tests { /// Test: a peer revealing a higher term ends the round at once, even after a grant. /// /// Scenario (5 nodes, so one grant is not yet a majority): - /// - Peer 2 grants, peer 3 denies and reports a term far above ours, peers 4 and 5 would - /// grant. + /// - Peer 2 grants, peer 3 denies and reports a term far above ours, peers 4 and 5 would grant. /// /// Expected: - /// - HigherTerm(peer's term): the node must catch up immediately (raft-rs and openraft both - /// adopt a higher term as soon as they see it), not carry on counting. + /// - HigherTerm(peer's term): the node must catch up immediately, not carry on counting. #[tokio::test] async fn test_round_higher_term_denial_ends_round_even_after_a_grant() { for round in BOTH_ROUNDS { - let result = run_round( - round, - vec![ - (2, Reply::Grant), - (3, Reply::DenyHigherTerm { peer_term: 9 }), - (4, Reply::Grant), - (5, Reply::Grant), - ], - ) + let result = run_round(round, vec![ + (2, Reply::Grant), + (3, Reply::DenyHigherTerm { peer_term: 9 }), + (4, Reply::Grant), + (5, Reply::Grant), + ]) .await; assert!( matches!( @@ -2620,15 +2616,12 @@ mod single_node_election_tests { #[tokio::test] async fn test_round_unreachable_peer_and_denial_do_not_block_majority() { for round in BOTH_ROUNDS { - let result = run_round( - round, - vec![ - (2, Reply::Down), - (3, deny_equal_log()), - (4, Reply::Grant), - (5, Reply::Grant), - ], - ) + let result = run_round(round, vec![ + (2, Reply::Down), + (3, deny_equal_log()), + (4, Reply::Grant), + (5, Reply::Grant), + ]) .await; assert!(result.is_ok(), "{round:?}: got {result:?}"); } @@ -2649,8 +2642,8 @@ mod single_node_election_tests { /// `>=` for PreVote. /// /// - Real vote: asked_term is this node's own term; a peer must be strictly above it. - /// - PreVote: asked_term is the term this node WANTS; a peer already at it means that - /// term is taken. + /// - PreVote: asked_term is the term this node WANTS; a peer already at it means that term is + /// taken. #[test] fn test_vote_rpc_peer_term_shows_behind_boundaries() { use crate::election::election_handler::VoteRpc; @@ -2667,8 +2660,8 @@ mod single_node_election_tests { /// Test: a peer whose log is more recent than the candidate's fails the PreVote. /// /// Expected: - /// - ElectionError::QuorumFailure with only the candidate's own vote, same as the real - /// vote: the candidate could not win, so it must not start a real election. + /// - ElectionError::QuorumFailure with only the candidate's own vote, same as the real vote: + /// the candidate could not win, so it must not start a real election. #[tokio::test] async fn test_broadcast_pre_vote_requests_rejects_when_peer_log_is_more_recent() { let mut transport = MockTransport::new(); @@ -2775,8 +2768,7 @@ mod single_node_election_tests { /// - A node returning from a partition asks about a free term with a perfectly good log. /// /// Expected: - /// - Denied. This is what stops it from becoming a candidate and disrupting a healthy - /// leader. + /// - Denied. This is what stops it from becoming a candidate and disrupting a healthy leader. #[test] fn test_handle_pre_vote_request_denies_while_leader_is_active() { let response = ask(RECEIVER_TERM + 1, (9, 9), true); @@ -2837,8 +2829,6 @@ mod single_node_election_tests { // ======================================================================== // test_handle_vote_request_stale_vote_* - a vote cast in an OLDER term must not block // a vote in the current term (Raft: on a new term, votedFor is null). - // raft-rs `reset(term)` clears the vote whenever the term changes; openraft replaces the - // whole (term, node) vote. // ======================================================================== /// Test: a stale vote from an older term does not block a legitimate candidate in the @@ -2920,8 +2910,8 @@ mod single_node_election_tests { /// /// - A PreVote that grants what the real vote denies sends the node into a real election it /// cannot win, bumping terms for nothing (the thing PreVote exists to prevent). - /// - A PreVote that denies what the real vote would grant makes a node that could win - /// unable to start an election: a liveness hole after a leader failure. + /// - A PreVote that denies what the real vote would grant makes a node that could win unable to + /// start an election: a liveness hole after a leader failure. #[tokio::test] async fn test_pre_vote_grants_exactly_when_the_real_vote_would() { let handler = ElectionHandler::::new(1); @@ -2963,7 +2953,7 @@ mod single_node_election_tests { } // ======================================================================== - // vote_from_any_state (raft-rs): a request with a higher term is adopted whether or not the + // A request with a higher term is adopted whether or not the // vote can be granted; a request with the same or a lower term is never adopted. // ======================================================================== diff --git a/d-engine-core/src/election/mod.rs b/d-engine-core/src/election/mod.rs index 94d91d75..2f0d35d5 100644 --- a/d-engine-core/src/election/mod.rs +++ b/d-engine-core/src/election/mod.rs @@ -5,14 +5,14 @@ //! - Candidate vote collection //! - Voter eligibility validation mod election_handler; -pub use election_handler::*; +use std::sync::Arc; use async_trait::async_trait; use d_engine_proto::server::election::VoteRequest; use d_engine_proto::server::election::VotedFor; +pub use election_handler::*; #[cfg(any(test, feature = "__test_support"))] use mockall::automock; -use std::sync::Arc; use crate::RaftNodeConfig; use crate::Result; @@ -71,8 +71,8 @@ where /// no term, vote or timer can be touched through this call. /// /// Denies when any of these holds: - /// - `leader_active`: the caller still has evidence of a live leader (Follower: recent - /// leader contact; Leader: recent quorum ACK; Candidate: false). + /// - `leader_active`: the caller still has evidence of a live leader (Follower: recent leader + /// contact; Leader: recent quorum ACK; Candidate: false). /// - `request.term <= current_term`: that term is already taken. /// - The requester's log is not at least as up-to-date as ours (Raft §5.4.1). /// diff --git a/d-engine-core/src/event.rs b/d-engine-core/src/event.rs index 6b446f9e..8343b872 100644 --- a/d-engine-core/src/event.rs +++ b/d-engine-core/src/event.rs @@ -1,6 +1,3 @@ -use crate::client::ClientReadRequest; -use crate::client::ClientResponse; -use crate::client::ClientWriteRequest; use d_engine_proto::common::LogId; use d_engine_proto::server::cluster::ClusterConfChangeRequest; use d_engine_proto::server::cluster::ClusterConfUpdateResponse; @@ -22,6 +19,9 @@ use tonic::Status; use crate::ApplyResult; use crate::MaybeCloneOneshotSender; use crate::Result; +use crate::client::ClientReadRequest; +use crate::client::ClientResponse; +use crate::client::ClientWriteRequest; #[derive(Debug, Clone, PartialEq)] pub struct NewCommitData { @@ -36,6 +36,7 @@ pub struct NewCommitData { pub enum ClientCmd { Propose( ClientWriteRequest, + tokio::time::Instant, MaybeCloneOneshotSender>, ), Read( @@ -47,6 +48,16 @@ pub enum ClientCmd { MaybeCloneOneshotSender>, ), } +impl ClientCmd { + /// Stamps `Instant::now()` so every write carries its issue time. One variant, no + /// timestamp-less twin to forget handling. + pub fn propose( + request: ClientWriteRequest, + sender: MaybeCloneOneshotSender>, + ) -> Self { + Self::Propose(request, tokio::time::Instant::now(), sender) + } +} #[derive(Debug)] pub enum InternalEvent { @@ -55,8 +66,6 @@ pub enum InternalEvent { BecomeLeader, BecomeLearner, - NotifyNewCommitIndex(NewCommitData), - /// Notify when follower/learner confirms leader via committed vote /// Triggered when committed vote changes from false to true /// No state transition - pure notification for watch channel @@ -123,6 +132,9 @@ pub enum InternalEvent { ApplyCompleted { last_index: u64, results: Vec, + /// When the SM Worker sent this; the central loop records the queue wait + /// (`core.raft.central_loop.apply_completed_queue_wait_ms`) when it handles it. + sent_at: tokio::time::Instant, }, /// Trigger state machine snapshot creation — routed via P2 (unbounded internal_event_tx) to @@ -214,7 +226,8 @@ pub enum InboundEvent { ), /// State machine apply failed - node must shutdown. - /// Conservative: treat all SM errors as fatal (future: distinguish fatal vs application errors). + /// Conservative: treat all SM errors as fatal (future: distinguish fatal vs application + /// errors). FatalError { source: String, // Error source error: String, // Error message diff --git a/d-engine-core/src/lib.rs b/d-engine-core/src/lib.rs index 07faf6e3..41563c59 100644 --- a/d-engine-core/src/lib.rs +++ b/d-engine-core/src/lib.rs @@ -2,7 +2,7 @@ //! //! Pure Raft consensus algorithm - for building custom Raft-based systems //! -//! ## ⚠️ Internal Crate - Not Ready for Standalone Use +//! ## Internal Crate - Not Ready for Standalone Use //! //! **Use [`d-engine`](https://crates.io/crates/d-engine) instead.** //! @@ -14,7 +14,7 @@ //! [dependencies] //! d-engine-core = "0.2" //! -//! # ✅ Use this instead +//! # Use this instead //! [dependencies] //! d-engine = "0.2" //! ``` @@ -55,7 +55,6 @@ pub mod client; pub mod command; -mod commit_handler; pub mod config; mod election; mod errors; @@ -82,51 +81,49 @@ pub mod watch; pub use client::*; pub use command::*; pub use config::*; -pub use errors::*; -pub use storage::*; -#[cfg(feature = "watch")] -pub use watch::*; - -// Stable extension points — types developers need when implementing custom -// storage engines, state machines, or transport layers. -pub use membership::Membership; -pub use network::Transport; -pub use purge::PurgeExecutor; -pub use raft::{LeaderInfo, Raft, SignalParams}; -pub use state_machine_handler::{SnapshotPolicy, StateMachineHandler}; -pub use type_config::TypeConfig; - // ── Internal implementation details (not part of public API) ─────────────── // These remain accessible to d-engine-server but are hidden from cargo doc. // Do not depend on these from external crates. #[doc(hidden)] -pub use commit_handler::*; -#[doc(hidden)] pub use election::*; +pub use errors::*; #[doc(hidden)] pub use event::*; #[doc(hidden)] pub use maybe_clone_oneshot::*; +// Stable extension points — types developers need when implementing custom +// storage engines, state machines, or transport layers. +pub use membership::Membership; #[doc(hidden)] pub use membership::*; +pub use network::Transport; #[doc(hidden)] pub use network::*; +pub use purge::PurgeExecutor; #[doc(hidden)] pub use purge::*; +pub use raft::LeaderInfo; +pub use raft::Raft; +pub use raft::SignalParams; #[doc(hidden)] pub use raft_context::*; #[doc(hidden)] pub use raft_role::*; #[doc(hidden)] pub use replication::*; +pub use state_machine_handler::SnapshotPolicy; +pub use state_machine_handler::StateMachineHandler; #[doc(hidden)] pub use state_machine_handler::*; +pub use storage::*; +pub(crate) use timer::*; +pub use type_config::TypeConfig; #[doc(hidden)] pub use type_config::*; #[doc(hidden)] pub use utils::*; - -pub(crate) use timer::*; +#[cfg(feature = "watch")] +pub use watch::*; #[cfg(test)] mod maybe_clone_oneshot_test; diff --git a/d-engine-core/src/membership.rs b/d-engine-core/src/membership.rs index 8960258a..ea157018 100644 --- a/d-engine-core/src/membership.rs +++ b/d-engine-core/src/membership.rs @@ -224,10 +224,10 @@ pub fn ensure_safe_join( Ok(()) } else { // metrics::counter!("core.cluster.unsafe_join_attempts", 1); - metrics::counter!( - "core.cluster.unsafe_join_attempts", - &[("node_id", node_id.to_string())] - ) + metrics::counter!("core.cluster.unsafe_join_attempts", &[( + "node_id", + node_id.to_string() + )]) .increment(1); warn!( diff --git a/d-engine-core/src/network/background_snapshot_transfer.rs b/d-engine-core/src/network/background_snapshot_transfer.rs index d9cd0dbf..5a9ad900 100644 --- a/d-engine-core/src/network/background_snapshot_transfer.rs +++ b/d-engine-core/src/network/background_snapshot_transfer.rs @@ -1,12 +1,13 @@ +use std::marker::PhantomData; +use std::pin::Pin; +use std::sync::Arc; +use std::time::Duration; + use d_engine_proto::server::storage::SnapshotChunk; use d_engine_proto::server::storage::snapshot_service_client::SnapshotServiceClient; use futures::Stream; use futures::StreamExt; use futures::stream::BoxStream; -use std::marker::PhantomData; -use std::pin::Pin; -use std::sync::Arc; -use std::time::Duration; use tokio::sync::mpsc; use tokio::sync::mpsc::error::TrySendError; use tokio::time::sleep; diff --git a/d-engine-core/src/network/mod.rs b/d-engine-core/src/network/mod.rs index 3b85d8a3..e1884f70 100644 --- a/d-engine-core/src/network/mod.rs +++ b/d-engine-core/src/network/mod.rs @@ -230,8 +230,8 @@ where /// # Parameters /// - `peer_id`: Target follower node ID /// - `metadata`: Snapshot metadata (term, index, size) - /// - `leader_term`: Sender's own current term, not `metadata`'s (the snapshotted - /// data's boundary term can be stale after a leader change — #436) + /// - `leader_term`: Sender's own current term, not `metadata`'s (the snapshotted data's + /// boundary term can be stale after a leader change — #436) /// - `state_machine_handler`: Used to load the snapshot data stream /// - `membership`: Cluster membership for bulk-channel resolution /// - `config`: Snapshot transfer configuration (chunk size, timeout, etc.) diff --git a/d-engine-core/src/raft.rs b/d-engine-core/src/raft.rs index 2d5a1c19..2bcaf118 100644 --- a/d-engine-core/src/raft.rs +++ b/d-engine-core/src/raft.rs @@ -1,6 +1,9 @@ // Re-export LeaderInfo from proto (application layer use) use std::collections::VecDeque; use std::sync::Arc; + +pub use d_engine_proto::common::LeaderInfo; +use d_engine_proto::server::election::VotedFor; use tokio::sync::mpsc; use tokio::sync::watch; use tokio::time::sleep_until; @@ -12,7 +15,6 @@ use tracing::warn; use super::InboundEvent; use super::InternalEvent; -use super::NewCommitData; use super::RaftContext; use super::RaftCoreHandlers; use super::RaftRole; @@ -28,8 +30,6 @@ use crate::TypeConfig; use crate::alias::MOF; use crate::alias::TROF; use crate::role_state::PeerReplicationState; -pub use d_engine_proto::common::LeaderInfo; -use d_engine_proto::server::election::VotedFor; pub struct Raft where @@ -58,9 +58,6 @@ where internal_event_rx: mpsc::UnboundedReceiver, buffered_internal_event: VecDeque, - // For business logic to apply logs into state machine - new_commit_listener: Vec>, - // Leader change notification // Uses watch::Sender for efficient multi-subscriber pattern leader_change_listener: Option>>, @@ -153,8 +150,6 @@ where internal_event_rx: signal_params.internal_event_rx, buffered_internal_event: VecDeque::new(), - new_commit_listener: Vec::new(), - shutdown_signal: signal_params.shutdown_signal, leader_change_listener: None, @@ -230,7 +225,24 @@ where self.role.join_cluster(&self.ctx).await } + /// Runs the Raft loop until shutdown or a fatal error, then closes the log. pub async fn run(&mut self) -> Result<()> { + let result = self.run_loop().await; + + // `Raft` is the log's only writer: wait for its in-flight writes before returning, + // on every exit path, not only on a clean shutdown. + self.ctx.storage.raft_log.close().await; + + // Nobody consumes events any more, however the loop ended: release the senders blocked + // in `event_tx.send().await` (e.g. gRPC stream handlers). Without this, + // `serve_with_shutdown` never completes and `Arc` is never released, keeping + // `Arc` alive and the RocksDB LOCK held indefinitely. + self.event_rx.close(); + + result + } + + async fn run_loop(&mut self) -> Result<()> { info!("Node is running"); if self.role.is_timer_expired() { @@ -248,7 +260,15 @@ where // P0: shutdown received; _ = self.shutdown_signal.changed() => { - return self.handle_shutdown().await; + info!("[Raft:{}] shutdown signal received.", self.node_id); + // Hand over the commits this node already knows about, so the state machine + // worker applies them before it stops. Only a clean shutdown does this: after + // a fatal error nothing more is delivered (a restart replays from + // `last_applied`). + if let Err(e) = self.process_committed().await { + error!(%self.node_id, ?e, "final commit delivery on shutdown failed"); + } + return Ok(()); } @@ -296,6 +316,7 @@ where self.process_internal_events().await?; self.process_client_cmds().await?; self.process_inbound_events().await?; + self.process_committed().await?; let new_deadline = self.role.next_deadline(); if new_deadline != tick.deadline() { @@ -304,18 +325,6 @@ where } } - async fn handle_shutdown(&mut self) -> Result<()> { - info!("[Raft:{}] shutdown signal received.", self.node_id); - // Close the raft log (before runtime shutdown) - // This ensures RocksDB file lock is released before tokio runtime shuts down - self.ctx.storage.raft_log.close().await; - // Unblock any tasks stuck in event_tx.send().await (e.g. gRPC stream handlers). - // Without this, serve_with_shutdown never completes and Arc is never - // released, keeping Arc alive and the RocksDB LOCK held indefinitely. - self.event_rx.close(); - Ok(()) - } - /// Drain all pending inbound events (up to max_batch_size). async fn drain_inbound_events(&mut self) -> Result<()> { let max = self.ctx.node_config.raft.batching.max_batch_size; @@ -423,6 +432,14 @@ where Ok(()) } + /// Last stage of an iteration: commit_index may have advanced in any of the stages above + /// (leader: flush / AppendResult; follower: AppendEntries; any role: snapshot install), so + /// hand the newly committed entries over before waiting for the next event. + /// Any error means committed entries can no longer be applied: the loop stops. + async fn process_committed(&mut self) -> Result<()> { + self.role.dispatch_committed(&self.ctx, &self.internal_event_tx).await + } + async fn process_client_cmds(&mut self) -> Result<()> { // Always flush: the P3 select arm may have pushed a command before this drain ran. // flush_cmd_buffers checks is_empty() internally and is a no-op when nothing is buffered. @@ -586,39 +603,6 @@ where #[cfg(test)] self.notify_role_transition(); } - InternalEvent::NotifyNewCommitIndex(mut new_commit_data) => { - // Drain all pending NotifyNewCommitIndex events (max_batch_size limit) - // This batches multiple committed entries into a single notification - let max_batch = self.ctx.node_config.raft.batching.max_batch_size; - let mut count = 1; - - while count < max_batch { - match self.internal_event_rx.try_recv() { - Ok(InternalEvent::NotifyNewCommitIndex(next)) => { - // Only keep the largest commit_index - if next.new_commit_index > new_commit_data.new_commit_index { - new_commit_data = next; - } - count += 1; - } - Ok(other) => { - self.internal_event_tx.send(other).map_err(|e| { - error!("Failed to resend internal event: {:?}", e); - crate::Error::Fatal(e.to_string()) - })?; - break; - } - Err(_) => break, - } - } - - debug!( - "[{}] NotifyNewCommitIndex drained: {} events, max_commit_index={}", - self.node_id, count, new_commit_data.new_commit_index - ); - - self.notify_new_commit(new_commit_data); - } InternalEvent::LeaderDiscovered(leader_id, term) => { debug!("LeaderDiscovered: leader_id={}, term={}", leader_id, term); // Notify leader change listeners - no state transition @@ -675,8 +659,12 @@ where InternalEvent::ApplyCompleted { last_index, results, + sent_at, } => { - // Routed via internal_event_tx (P2) to avoid priority inversion against AppendEntries at P4. + metrics::histogram!("core.raft.central_loop.apply_completed_queue_wait_ms") + .record(sent_at.elapsed().as_secs_f64() * 1_000.0); + // Routed via internal_event_tx (P2) to avoid priority inversion against + // AppendEntries at P4. if let Err(e) = self .role .handle_apply_completed(last_index, results, &self.ctx, &self.internal_event_tx) @@ -749,10 +737,11 @@ where // received, silently skipping them forever. match last_included_index { Some(last_included_index) => { - let _ = self.role.init_peers_next_index_and_match_index( - last_included_index, - vec![peer_id], - ); + let _ = self + .role + .init_peers_next_index_and_match_index(last_included_index, vec![ + peer_id, + ]); } None => { error!( @@ -833,26 +822,6 @@ where Ok(()) } - pub fn register_new_commit_listener( - &mut self, - tx: mpsc::UnboundedSender, - ) { - self.new_commit_listener.push(tx); - } - - pub fn notify_new_commit( - &self, - new_commit_data: NewCommitData, - ) { - debug!(?new_commit_data, "notify_new_commit",); - - for tx in &self.new_commit_listener { - if let Err(e) = tx.send(new_commit_data.clone()) { - error!("notify_new_commit failed: {:?}", e); - } - } - } - #[cfg(test)] pub fn register_role_transition_listener( &mut self, @@ -953,6 +922,9 @@ where } } +#[cfg(test)] +#[path = "raft_test/commit_path_tests.rs"] +mod commit_path_tests; #[cfg(test)] #[path = "raft_test/leader_change_tests.rs"] mod leader_change_tests; diff --git a/d-engine-core/src/raft_role/buffers/batch_buffer_test.rs b/d-engine-core/src/raft_role/buffers/batch_buffer_test.rs index 18f6b272..3ba8d4bc 100644 --- a/d-engine-core/src/raft_role/buffers/batch_buffer_test.rs +++ b/d-engine-core/src/raft_role/buffers/batch_buffer_test.rs @@ -1,8 +1,10 @@ -use super::*; -use batch_buffer::BatchBuffer; use std::time::Duration; + +use batch_buffer::BatchBuffer; use tokio::time::Instant; +use super::*; + #[derive(Debug, Clone, PartialEq)] struct TestRequest; diff --git a/d-engine-core/src/raft_role/buffers/mod.rs b/d-engine-core/src/raft_role/buffers/mod.rs index 751588e0..f4525429 100644 --- a/d-engine-core/src/raft_role/buffers/mod.rs +++ b/d-engine-core/src/raft_role/buffers/mod.rs @@ -1,10 +1,9 @@ //! Batching buffers for the Leader's client request hot path. //! -//! - [`BatchBuffer`] — generic single-type buffer backed by a contiguous `Vec`. -//! Used for linearizable read requests. -//! - [`ProposeBatchBuffer`] — SoA (Struct-of-Arrays) buffer for write proposals. -//! Stores payloads and senders in separate `Vec`s; -//! `flush()` is O(1) via `mem::take`, no unzip scan. +//! - [`BatchBuffer`] — generic single-type buffer backed by a contiguous `Vec`. Used for +//! linearizable read requests. +//! - [`ProposeBatchBuffer`] — SoA (Struct-of-Arrays) buffer for write proposals. Stores payloads +//! and senders in separate `Vec`s; `flush()` is O(1) via `mem::take`, no unzip scan. pub mod batch_buffer; pub mod propose_batch_buffer; diff --git a/d-engine-core/src/raft_role/buffers/propose_batch_buffer.rs b/d-engine-core/src/raft_role/buffers/propose_batch_buffer.rs index 389773c5..f87520eb 100644 --- a/d-engine-core/src/raft_role/buffers/propose_batch_buffer.rs +++ b/d-engine-core/src/raft_role/buffers/propose_batch_buffer.rs @@ -20,16 +20,16 @@ //! `payloads.len() == senders.len()` at all times. //! Enforced by routing all mutations through `push()` and `flush()`. -use rand::distr::SampleString; use std::sync::Arc; -use crate::client::ClientResponse; use d_engine_proto::common::EntryPayload; +use rand::distr::SampleString; use tokio::time::Instant; use tonic::Status; use crate::MaybeCloneOneshotSender; use crate::RaftRequestWithSignal; +use crate::client::ClientResponse; /// Sender type alias for readability. type ProposeSender = MaybeCloneOneshotSender>; @@ -48,7 +48,8 @@ pub struct ProposeBatchBuffer { pub last_flush: Instant, /// Pre-allocated metric labels for zero-allocation hot path. metrics_labels: Option>, - /// Set when the buffer receives the first payload of a new batch, cleared on flush — measures how long the first request waited before the batch got drained (#446). + /// Set when the buffer receives the first payload of a new batch, cleared on flush — measures + /// how long the first request waited before the batch got drained (#446). oldest_pending_since: Option, } diff --git a/d-engine-core/src/raft_role/buffers/propose_batch_buffer_test.rs b/d-engine-core/src/raft_role/buffers/propose_batch_buffer_test.rs index 8094ef72..56506d0f 100644 --- a/d-engine-core/src/raft_role/buffers/propose_batch_buffer_test.rs +++ b/d-engine-core/src/raft_role/buffers/propose_batch_buffer_test.rs @@ -1,11 +1,10 @@ -use crate::client::ClientResponse; use d_engine_proto::common::EntryPayload; use tonic::Status; +use super::propose_batch_buffer::ProposeBatchBuffer; use crate::MaybeCloneOneshot; use crate::RaftOneshot; - -use super::propose_batch_buffer::ProposeBatchBuffer; +use crate::client::ClientResponse; // Helper: create a dummy sender, discarding the receiver. fn make_sender() -> crate::MaybeCloneOneshotSender> { diff --git a/d-engine-core/src/raft_role/candidate_state.rs b/d-engine-core/src/raft_role/candidate_state.rs index b6329f44..164067a9 100644 --- a/d-engine-core/src/raft_role/candidate_state.rs +++ b/d-engine-core/src/raft_role/candidate_state.rs @@ -1,3 +1,7 @@ +use std::fmt::Debug; +use std::marker::PhantomData; +use std::sync::Arc; + use async_trait::async_trait; use d_engine_proto::common::LogId; use d_engine_proto::common::NodeRole::Candidate; @@ -7,9 +11,6 @@ use d_engine_proto::common::NodeRole::Learner; use d_engine_proto::server::cluster::ClusterConfUpdateResponse; use d_engine_proto::server::election::VoteResponse; use d_engine_proto::server::election::VotedFor; -use std::fmt::Debug; -use std::marker::PhantomData; -use std::sync::Arc; use tokio::sync::mpsc; use tokio::time::Instant; use tonic::Status; @@ -433,7 +434,8 @@ impl RaftRoleState for CandidateState { debug!("Rejecting AppendEntries from stale leader: {:?}", &response); for sender in senders { if let Err(e) = sender.send(Ok(response)) { - // Receiver timed out and dropped — this is normal, do not crash the node + // Receiver timed out and dropped — this is normal, do not crash the + // node error!( "Failed to send AppendEntries rejection (receiver dropped): {:?}", e diff --git a/d-engine-core/src/raft_role/candidate_state_test.rs b/d-engine-core/src/raft_role/candidate_state_test.rs index 4d449e77..037b1288 100644 --- a/d-engine-core/src/raft_role/candidate_state_test.rs +++ b/d-engine-core/src/raft_role/candidate_state_test.rs @@ -1,12 +1,6 @@ use std::sync::Arc; -use crate::MockRaftLog; -use crate::client::ClientReadRequest; -use crate::client::ClientResponse; -use crate::client::ClientWriteRequest; -use crate::client::ErrorCode; -use crate::config::ReadConsistencyPolicy; - +use d_engine_proto::client::WriteCommand; use d_engine_proto::common::LogId; use d_engine_proto::server::cluster::ClusterConfChangeRequest; use d_engine_proto::server::cluster::ClusterConfUpdateResponse; @@ -18,6 +12,9 @@ use d_engine_proto::server::election::VoteRequest; use d_engine_proto::server::election::VotedFor; use d_engine_proto::server::replication::AppendEntriesRequest; use d_engine_proto::server::replication::AppendEntriesResponse; +use prost::Message; +use tokio::sync::mpsc; +use tokio::sync::watch; use tonic::Code; use crate::ClientCmd; @@ -30,9 +27,15 @@ use crate::MaybeCloneOneshot; use crate::MockBuilder; use crate::MockElectionCore; use crate::MockMembership; +use crate::MockRaftLog; use crate::MockReplicationCore; use crate::MockStateMachineHandler; use crate::RaftOneshot; +use crate::client::ClientReadRequest; +use crate::client::ClientResponse; +use crate::client::ClientWriteRequest; +use crate::client::ErrorCode; +use crate::config::ReadConsistencyPolicy; use crate::raft_role::candidate_state::CandidateState; use crate::raft_role::follower_state::FollowerState; use crate::raft_role::role_state::RaftRoleState; @@ -41,9 +44,6 @@ use crate::test_utils::mock::MockTypeConfig; use crate::test_utils::mock::mock_election_core; use crate::test_utils::mock::mock_raft_context; use crate::test_utils::node_config; -use d_engine_proto::client::WriteCommand; -use prost::Message; -use tokio::sync::{mpsc, watch}; /// Test: CandidateState can_vote_myself returns true for new candidate /// @@ -218,14 +218,14 @@ async fn test_tick_triggers_new_election_round_on_success() { /// /// Scenario: /// - Candidate ticks (election timeout): term increments (1 → 2), votes for itself. -/// - `RaftLog::save_hard_state` is documented (raft_log.rs "When to Call") as required -/// "When becoming candidate (incrementing term)". +/// - `RaftLog::save_hard_state` is documented (raft_log.rs "When to Call") as required "When +/// becoming candidate (incrementing term)". /// /// Expected: -/// - `save_hard_state` is called exactly once, with the new term (2) and the -/// self-vote (`voted_for_id: 1, voted_for_term: 2`). -/// - The call happens BEFORE `broadcast_vote_requests` is invoked — a candidate must -/// not ask other nodes to vote for a term it hasn't durably committed to itself +/// - `save_hard_state` is called exactly once, with the new term (2) and the self-vote +/// (`voted_for_id: 1, voted_for_term: 2`). +/// - The call happens BEFORE `broadcast_vote_requests` is invoked — a candidate must not ask other +/// nodes to vote for a term it hasn't durably committed to itself #[tokio::test(start_paused = true)] async fn test_tick_election_timeout_persists_hard_state_before_broadcasting_votes() { let (internal_event_tx, mut internal_event_rx) = mpsc::unbounded_channel(); @@ -362,13 +362,12 @@ async fn test_tick_discovers_higher_term_and_steps_down() { /// - `RaftLog::save_hard_state` is documented as required "When discovering higher term". /// /// Expected: -/// - `save_hard_state` is called exactly once with term=100 (the discovered higher term, -/// not the candidate's own pre-election term). -/// - The call happens BEFORE `InternalEvent::BecomeFollower` is sent — per the Raft -/// safety invariant, a node must not act on (or announce) a term change until it is -/// durable, otherwise a crash between "step down" and "persist" resurrects the stale -/// term on restart (this is exactly the class of bug the sm_wal_disabled_crash_recovery -/// integration test surfaced). +/// - `save_hard_state` is called exactly once with term=100 (the discovered higher term, not the +/// candidate's own pre-election term). +/// - The call happens BEFORE `InternalEvent::BecomeFollower` is sent — per the Raft safety +/// invariant, a node must not act on (or announce) a term change until it is durable, otherwise a +/// crash between "step down" and "persist" resurrects the stale term on restart (this is exactly +/// the class of bug the sm_wal_disabled_crash_recovery integration test surfaced). #[tokio::test(start_paused = true)] async fn test_tick_discovers_higher_term_via_election_persists_hard_state() { let (internal_event_tx, internal_event_rx) = mpsc::unbounded_channel(); @@ -467,7 +466,8 @@ async fn test_tick_discovers_higher_term_via_election_persists_hard_state() { /// - Term remains unchanged /// /// Original test location: -/// `d-engine-server/tests/components/raft_role/candidate_state_test.rs::test_handle_inbound_event_case1_1` +/// `d-engine-server/tests/components/raft_role/candidate_state_test. +/// rs::test_handle_inbound_event_case1_1` #[tokio::test] async fn test_handle_vote_request_rejects_illegal_request() { let (_graceful_tx, graceful_rx) = watch::channel(()); @@ -537,7 +537,8 @@ async fn test_handle_vote_request_rejects_illegal_request() { /// step down and let the new role process it. /// /// Original test location: -/// `d-engine-server/tests/components/raft_role/candidate_state_test.rs::test_handle_inbound_event_case1_2` +/// `d-engine-server/tests/components/raft_role/candidate_state_test. +/// rs::test_handle_inbound_event_case1_2` #[tokio::test] async fn test_handle_vote_request_grants_and_steps_down_when_legal() { let (_graceful_tx, graceful_rx) = watch::channel(()); @@ -620,7 +621,8 @@ async fn test_handle_vote_request_grants_and_steps_down_when_legal() { /// This validates that Candidate can serve read-only cluster metadata queries. /// /// Original test location: -/// `d-engine-server/tests/components/raft_role/candidate_state_test.rs::test_handle_inbound_event_case2` +/// `d-engine-server/tests/components/raft_role/candidate_state_test. +/// rs::test_handle_inbound_event_case2` #[tokio::test] async fn test_handle_cluster_conf_metadata_request() { let (_graceful_tx, graceful_rx) = watch::channel(()); @@ -675,7 +677,8 @@ async fn test_handle_cluster_conf_metadata_request() { /// This validates that Candidate can apply configuration changes from leader. /// /// Original test location: -/// `d-engine-server/tests/components/raft_role/candidate_state_test.rs::test_handle_inbound_event_case3` +/// `d-engine-server/tests/components/raft_role/candidate_state_test. +/// rs::test_handle_inbound_event_case3` #[tokio::test] async fn test_handle_cluster_conf_update_success() { let (_graceful_tx, graceful_rx) = watch::channel(()); @@ -980,8 +983,8 @@ async fn test_handle_append_entries_rejects_stale_term_does_not_reset_timer() { /// /// Scenario: /// - Candidate is in term 3 -/// - Receives AppendEntries with term=3 (legitimate leader, same term) where prev_log_index -/// exceeds the candidate's log length — a log conflict would occur +/// - Receives AppendEntries with term=3 (legitimate leader, same term) where prev_log_index exceeds +/// the candidate's log length — a log conflict would occur /// /// Expected (Raft §5.2): /// - Candidate must step down immediately (BecomeFollower + ReprocessEvent) @@ -1062,8 +1065,8 @@ async fn test_handle_append_entries_same_term_log_conflict_steps_down() { /// /// Scenario: /// - Candidate is in term 2 -/// - Receives AppendEntries with term=3 (new leader, higher term) where prev_log_index -/// exceeds the candidate's log — log conflict would occur +/// - Receives AppendEntries with term=3 (new leader, higher term) where prev_log_index exceeds the +/// candidate's log — log conflict would occur /// /// Expected (Raft §5.2): /// - Candidate must step down immediately (BecomeFollower + ReprocessEvent) @@ -1148,14 +1151,14 @@ async fn test_handle_append_entries_higher_term_log_conflict_steps_down_and_upda /// /// Scenario: /// - Candidate is in term 2, receives AppendEntries with term=3 (new leader). -/// - `RaftLog::save_hard_state` is documented as required "When discovering higher term" -/// — this is a distinct code path from the election-response case above (candidate_state.rs -/// handles AppendEntries and VoteRequest term-updates separately), so it needs its own test. +/// - `RaftLog::save_hard_state` is documented as required "When discovering higher term" — this is +/// a distinct code path from the election-response case above (candidate_state.rs handles +/// AppendEntries and VoteRequest term-updates separately), so it needs its own test. /// /// Expected: /// - `save_hard_state` is called exactly once with term=3. -/// - The call happens BEFORE `InternalEvent::BecomeFollower` is sent, for the same -/// crash-safety reason as the election-response case. +/// - The call happens BEFORE `InternalEvent::BecomeFollower` is sent, for the same crash-safety +/// reason as the election-response case. #[tokio::test] async fn test_handle_append_entries_higher_term_persists_hard_state_before_stepping_down() { let (_graceful_tx, graceful_rx) = watch::channel(()); @@ -1251,7 +1254,7 @@ async fn test_handle_client_write_returns_not_leader() { let mut state = CandidateState::::new(1, context.node_config.clone()); let (resp_tx, mut resp_rx) = MaybeCloneOneshot::new(); - let cmd = ClientCmd::Propose( + let cmd = ClientCmd::propose( ClientWriteRequest { client_id: 1, command: Some(bytes::Bytes::from( @@ -1455,7 +1458,8 @@ mod snapshot_tests { ); // VERIFY 2: No CreateSnapshotEvent is sent (channel should be empty). - // internal_event_tx is kept alive so that Disconnected cannot be confused with "no event sent". + // internal_event_tx is kept alive so that Disconnected cannot be confused with "no event + // sent". match internal_event_rx.try_recv() { Err(mpsc::error::TryRecvError::Empty) => { // Expected: no event sent — correct behavior @@ -1472,7 +1476,8 @@ mod snapshot_tests { drop(internal_event_tx); // VERIFY 3: Candidate state remains unchanged (no snapshot in progress) - // Note: CandidateState doesn't have snapshot_in_progress field because it doesn't handle snapshots + // Note: CandidateState doesn't have snapshot_in_progress field because it doesn't handle + // snapshots } } @@ -1547,8 +1552,8 @@ mod handle_client_read_request { /// it returns Error::Fatal and stops further processing. /// /// # Test Scenario -/// Candidate receives FatalError event from state machine while in candidate role (during election). -/// Candidate should recognize the fatal error and return Error::Fatal. +/// Candidate receives FatalError event from state machine while in candidate role (during +/// election). Candidate should recognize the fatal error and return Error::Fatal. /// /// # Given /// - Candidate in normal state (in election) @@ -1679,7 +1684,7 @@ async fn test_new_leader_initializes_empty_buffers() { )), }; - let cmd = crate::ClientCmd::Propose(write_req, response_tx); + let cmd = crate::ClientCmd::propose(write_req, response_tx); // Push command to buffer (should work on new Leader) leader.push_client_cmd(cmd, &raft.ctx); @@ -1816,8 +1821,8 @@ async fn expired_candidate( /// Test: a failed PreVote keeps the term unchanged and starts no real election. /// /// Scenario: -/// - Candidate (term 1) times out; the PreVote round does not reach a majority -/// (e.g. the node is isolated). +/// - Candidate (term 1) times out; the PreVote round does not reach a majority (e.g. the node is +/// isolated). /// /// Expected: /// - Term stays 1, nothing is persisted (a MockRaftLog without expectations panics on any @@ -1864,7 +1869,8 @@ async fn test_tick_pre_vote_failure_keeps_term_and_skips_real_election() { /// - Then the real election runs for term 2 and the node becomes leader. #[tokio::test(start_paused = true)] async fn test_tick_pre_vote_runs_for_next_term_before_persisting_it() { - use std::sync::atomic::{AtomicBool, Ordering}; + use std::sync::atomic::AtomicBool; + use std::sync::atomic::Ordering; let pre_vote_done = Arc::new(AtomicBool::new(false)); let mut election_core = MockElectionCore::::new(); @@ -1916,8 +1922,8 @@ async fn test_tick_pre_vote_runs_for_next_term_before_persisting_it() { /// - The peers answer the PreVote with term 100 (the cluster moved on while this node was away). /// /// Expected: -/// - The higher term is persisted BEFORE `BecomeFollower` is sent, the node ends in term 100, -/// and no RequestVote is broadcast. +/// - The higher term is persisted BEFORE `BecomeFollower` is sent, the node ends in term 100, and +/// no RequestVote is broadcast. #[tokio::test(start_paused = true)] async fn test_tick_pre_vote_higher_term_catches_up_without_real_election() { let mut election_core = MockElectionCore::::new(); @@ -2038,17 +2044,17 @@ async fn test_candidate_pre_vote_reports_no_active_leader_and_changes_nothing() /// even when it cannot grant the vote. /// /// Scenario: -/// - Candidate (term 1) receives a VoteRequest for term 5 from a node whose log is behind, so -/// the request is not legal and the vote cannot be granted. +/// - Candidate (term 1) receives a VoteRequest for term 5 from a node whose log is behind, so the +/// request is not legal and the vote cannot be granted. /// /// Expected: -/// - The term becomes 5 (persisted), `BecomeFollower` is sent, and the request is replayed to -/// the new Follower, which answers it (here: a denial because of the log). +/// - The term becomes 5 (persisted), `BecomeFollower` is sent, and the request is replayed to the +/// new Follower, which answers it (here: a denial because of the log). /// /// Raft §5.1: "If RPC request or response contains term T > currentTerm: set currentTerm = T, -/// convert to follower", whatever the RPC and whether or not the vote is granted (raft-rs -/// `test_vote_from_any_state`). A Candidate that keeps its lower term and stays a Candidate -/// starts its next election at a term the cluster has already passed. +/// convert to follower", whatever the RPC and whether or not the vote is granted. A Candidate that +/// keeps its lower term and stays a Candidate starts its next election at a term the cluster has +/// already passed. #[tokio::test] async fn test_candidate_higher_term_vote_request_with_stale_log_adopts_term_and_steps_down() { let (_graceful_tx, graceful_rx) = watch::channel(()); @@ -2140,9 +2146,8 @@ async fn test_candidate_tick_does_not_campaign_when_not_an_active_voter() { // No campaign while a committed membership change is still unapplied // // A node that has committed but not yet applied a membership change would count votes against -// a stale voter set. raft-rs refuses to campaign in that window -// (`test_conf_change_check_before_campaign`: "cannot campaign since there are still pending -// configuration changes to apply"). +// a stale voter set, so it must not campaign in that window: it still has pending +// configuration changes to apply. // ============================================================================ /// Builds an expired candidate whose log has committed entries up to `commit_index`, of which @@ -2280,9 +2285,9 @@ async fn test_candidate_tick_campaigns_when_everything_committed_is_applied() { // // Follower::tick already waited one full election timeout before sending BecomeCandidate. The // new Candidate used to start with a fresh random timer, so the first real election round began -// only after ANOTHER election timeout: failover took 1-2 timeouts longer than necessary. etcd and -// raft-rs campaign at the moment the first timeout fires. (Present on main, before PreVote.) -// ============================================================================ +// only after ANOTHER election timeout: failover took 1-2 timeouts longer than necessary. The first +// election round must start at the moment the first timeout fires. (Present on main, before +// PreVote.) ============================================================================ /// Test: a Candidate created from a timed-out Follower is due to campaign immediately. #[tokio::test(start_paused = true)] diff --git a/d-engine-core/src/raft_role/follower_state.rs b/d-engine-core/src/raft_role/follower_state.rs index 3cd718b4..ee0d5691 100644 --- a/d-engine-core/src/raft_role/follower_state.rs +++ b/d-engine-core/src/raft_role/follower_state.rs @@ -1,3 +1,11 @@ +use std::collections::BTreeMap; +use std::fmt::Debug; +use std::marker::PhantomData; +use std::sync::Arc; +use std::sync::atomic::AtomicBool; +use std::sync::atomic::Ordering; +use std::time::Duration; + use async_trait::async_trait; use d_engine_proto::common::LogId; use d_engine_proto::common::NodeRole::Candidate; @@ -7,13 +15,6 @@ use d_engine_proto::server::cluster::ClusterConfUpdateResponse; use d_engine_proto::server::cluster::LeaderDiscoveryResponse; use d_engine_proto::server::election::VoteResponse; use d_engine_proto::server::storage::SnapshotMetadata; -use std::collections::BTreeMap; -use std::fmt::Debug; -use std::marker::PhantomData; -use std::sync::Arc; -use std::sync::atomic::AtomicBool; -use std::sync::atomic::Ordering; -use std::time::Duration; use tokio::sync::mpsc; use tokio::time::Instant; use tonic::Status; @@ -372,14 +373,8 @@ impl RaftRoleState for FollowerState { } InboundEvent::InstallSnapshotChunk(stream, sender) => { - self.handle_install_snapshot_chunk_workflow( - stream, - sender, - ctx, - internal_event_tx, - &state_snapshot, - ) - .await?; + self.handle_install_snapshot_chunk_workflow(stream, sender, ctx, internal_event_tx) + .await?; } InboundEvent::JoinCluster(_join_request, sender) => { diff --git a/d-engine-core/src/raft_role/follower_state_test.rs b/d-engine-core/src/raft_role/follower_state_test.rs index 3ad8a306..c7cb145f 100644 --- a/d-engine-core/src/raft_role/follower_state_test.rs +++ b/d-engine-core/src/raft_role/follower_state_test.rs @@ -1,9 +1,6 @@ -use crate::client::ClientReadRequest; -use crate::client::ClientResponse; -use crate::client::ClientWriteRequest; -use crate::client::ErrorCode; -use crate::client::LeaderHint; -use crate::config::ReadConsistencyPolicy; +use std::sync::Arc; + +use d_engine_proto::client::WriteCommand; use d_engine_proto::common::LogId; use d_engine_proto::common::NodeRole; use d_engine_proto::server::cluster::ClusterConfChangeRequest; @@ -21,7 +18,10 @@ use d_engine_proto::server::replication::AppendEntriesResponse; use d_engine_proto::server::storage::SnapshotAck; use d_engine_proto::server::storage::SnapshotChunk; use d_engine_proto::server::storage::snapshot_ack::ChunkStatus; -use std::sync::Arc; +use mockall::predicate::eq; +use prost::Message; +use tokio::sync::mpsc; +use tokio::sync::watch; use tonic::Code; use tonic::Status; @@ -49,6 +49,12 @@ use crate::StateMachineCommand; use crate::StateMachineCommandSender; use crate::StateUpdate; use crate::SystemError; +use crate::client::ClientReadRequest; +use crate::client::ClientResponse; +use crate::client::ClientWriteRequest; +use crate::client::ErrorCode; +use crate::client::LeaderHint; +use crate::config::ReadConsistencyPolicy; use crate::raft_role::candidate_state::CandidateState; use crate::raft_role::follower_state::FollowerState; use crate::raft_role::leader_state::LeaderState; @@ -61,10 +67,6 @@ use crate::test_utils::mock::mock_raft_context; use crate::test_utils::mock::mock_raft_context_with_temp; use crate::test_utils::mock::mock_raft_log; use crate::test_utils::node_config; -use d_engine_proto::client::WriteCommand; -use mockall::predicate::eq; -use prost::Message; -use tokio::sync::{mpsc, watch}; // ============================================================================ // Helper Functions @@ -325,7 +327,8 @@ async fn test_handle_vote_request_rejects_when_handler_returns_none() { /// /// Scenario: /// - Follower receives VoteRequest with higher term -/// - handle_vote_request returns Ok(StateUpdate { new_voted_for: Some(...), term_update: Some(100) }) +/// - handle_vote_request returns Ok(StateUpdate { new_voted_for: Some(...), term_update: Some(100) +/// }) /// - Vote is granted /// /// Expected: @@ -383,14 +386,13 @@ async fn test_handle_vote_request_grants_and_updates_term() { /// to the caller. /// /// Scenario: -/// - Same setup as test_handle_vote_request_grants_and_updates_term above: Follower -/// grants a vote, term updates to 100. +/// - Same setup as test_handle_vote_request_grants_and_updates_term above: Follower grants a vote, +/// term updates to 100. /// - `RaftLog::save_hard_state` is documented as required "Before voting for a candidate" /// (raft_log.rs "When to Call") — this is the Raft paper's core safety requirement: -/// currentTerm/votedFor must be durable before the RPC response is sent, otherwise a -/// crash right after responding "yes" can make the node forget it already voted this -/// term and grant a second, conflicting vote after restart (double-voting, breaks -/// election safety). +/// currentTerm/votedFor must be durable before the RPC response is sent, otherwise a crash right +/// after responding "yes" can make the node forget it already voted this term and grant a second, +/// conflicting vote after restart (double-voting, breaks election safety). /// /// Expected: /// - `save_hard_state` is called exactly once with term=100 and the granted `voted_for`. @@ -478,17 +480,16 @@ async fn test_handle_vote_request_grants_vote_persists_hard_state_before_respond /// Test: Granting a vote resets the election timer. /// /// Scenario: -/// - Follower's timer is set to an already-expired deadline before the request -/// arrives (simulates "haven't heard from anyone in a while"). +/// - Follower's timer is set to an already-expired deadline before the request arrives (simulates +/// "haven't heard from anyone in a while"). /// - `VoteRequest` arrives and is legitimately granted. /// /// Expected: -/// - After processing, `state.is_timer_expired()` is false — the timer was -/// reset to a fresh future deadline as part of granting the vote, giving the -/// new candidate a fair window to complete its election before this node -/// also times out and starts a competing campaign (see decision doc for -/// ticket #422: this exact gap caused a real election-storm regression, -/// confirmed pre-existing via `git diff` before this test was added). +/// - After processing, `state.is_timer_expired()` is false — the timer was reset to a fresh future +/// deadline as part of granting the vote, giving the new candidate a fair window to complete its +/// election before this node also times out and starts a competing campaign (see decision doc for +/// ticket #422: this exact gap caused a real election-storm regression, confirmed pre-existing +/// via `git diff` before this test was added). /// /// Known bug (fixed as of 2026-07-18, ticket #422 follow-up): the vote-granting /// branch in `handle_inbound_event`'s `ReceiveVoteRequest` arm previously never @@ -548,15 +549,14 @@ async fn test_handle_vote_request_grants_vote_resets_election_timer() { /// Test: A VoteRequest rejected for an illegitimate term does NOT reset the timer. /// /// Scenario: -/// - VoteRequest carries a term that is not >= this follower's own term (stale -/// candidate), so the request is illegitimate and must be rejected outright. +/// - VoteRequest carries a term that is not >= this follower's own term (stale candidate), so the +/// request is illegitimate and must be rejected outright. /// /// Expected: -/// - The timer is untouched. If every incoming VoteRequest reset the timer -/// regardless of legitimacy, a stale/misbehaving peer could keep this node -/// from ever timing out and starting its own election — an availability bug -/// distinct from (but as real as) the missing-reset one above. This is the -/// negative case that proves the fix is scoped correctly, not blanket-applied. +/// - The timer is untouched. If every incoming VoteRequest reset the timer regardless of +/// legitimacy, a stale/misbehaving peer could keep this node from ever timing out and starting +/// its own election — an availability bug distinct from (but as real as) the missing-reset one +/// above. This is the negative case that proves the fix is scoped correctly, not blanket-applied. #[tokio::test] async fn test_handle_vote_request_rejected_does_not_reset_timer() { let (_graceful_tx, graceful_rx) = watch::channel(()); @@ -605,9 +605,9 @@ async fn test_handle_vote_request_rejected_does_not_reset_timer() { /// /// Expected: /// - `save_hard_state` is called with the leader's committed vote. -/// - The call happens BEFORE `InternalEvent::LeaderDiscovered` is sent — mirrors -/// the BecomeFollower ordering tests in candidate_state_test.rs, applied here -/// to the notification this codebase uses for `wait_ready()`. +/// - The call happens BEFORE `InternalEvent::LeaderDiscovered` is sent — mirrors the BecomeFollower +/// ordering tests in candidate_state_test.rs, applied here to the notification this codebase uses +/// for `wait_ready()`. /// /// Known bug (fixed as of 2026-07-18, ticket #422 follow-up): this handler used /// to send `LeaderDiscovered` before the single consolidated `commit_hard_state` @@ -894,10 +894,9 @@ async fn test_tick_triggers_election_on_timeout() { /// /// Expected: /// 1. Sends LeaderDiscovered event (marks new leader) -/// 2. Sends NotifyNewCommitIndex event -/// 3. Updates current_term to leader's term (2) -/// 4. Updates commit_index to new value (2) -/// 5. Returns AppendEntriesResponse with success=true +/// 2. Updates current_term to leader's term (2) +/// 3. Updates commit_index to new value (2) +/// 4. Returns AppendEntriesResponse with success=true /// 6. handle_inbound_event returns Ok(()) /// /// This validates the core Raft rule: Follower accepts entries from @@ -969,19 +968,6 @@ async fn test_handle_append_entries_success_from_new_leader() { "Should send LeaderDiscovered event" ); - // Verify: NotifyNewCommitIndex event sent - assert!( - matches!( - internal_event_rx.try_recv().unwrap(), - InternalEvent::NotifyNewCommitIndex(NewCommitData { - new_commit_index: _, - role: _, - current_term: _ - }) - ), - "Should send NotifyNewCommitIndex event" - ); - // Verify: Term and commit_index updated assert_eq!( state.current_term(), @@ -1169,18 +1155,17 @@ async fn test_handle_append_entries_rejects_stale_term() { /// Test: AppendEntries rejected for a stale term does NOT reset the election timer. /// /// Scenario: -/// - Follower (term=2) receives AppendEntries from a stale leader (term=1) — the -/// same illegitimate-sender case as `test_handle_append_entries_rejects_stale_term`. +/// - Follower (term=2) receives AppendEntries from a stale leader (term=1) — the same +/// illegitimate-sender case as `test_handle_append_entries_rejects_stale_term`. /// /// Expected: -/// - The timer is untouched. `handle_append_entries_request_workflow` -/// (role_state.rs) currently calls `self.reset_timer()` unconditionally before -/// checking whether the sender's term is stale — so today this test is RED: a -/// rejected, stale-term AppendEntries still resets the timer. If a stale leader -/// keeps sending (each one rejected), this follower's timer never expires and it -/// never starts its own election, even though the real current leader may be -/// unreachable. Same class of bug as `test_handle_vote_request_rejected_does_not_reset_timer` -/// (ticket #422), for AppendEntries instead of VoteRequest. +/// - The timer is untouched. `handle_append_entries_request_workflow` (role_state.rs) currently +/// calls `self.reset_timer()` unconditionally before checking whether the sender's term is stale +/// — so today this test is RED: a rejected, stale-term AppendEntries still resets the timer. If a +/// stale leader keeps sending (each one rejected), this follower's timer never expires and it +/// never starts its own election, even though the real current leader may be unreachable. Same +/// class of bug as `test_handle_vote_request_rejected_does_not_reset_timer` (ticket #422), for +/// AppendEntries instead of VoteRequest. #[tokio::test] async fn test_handle_append_entries_rejects_stale_term_does_not_reset_timer() { let (_graceful_tx, graceful_rx) = watch::channel(()); @@ -1659,8 +1644,8 @@ async fn test_handle_cluster_conf_update_when_leader_unknown() { /// /// Expected: /// - Returns Ok(ClientResponse) with error = NotLeader -/// - `leader_hint` is populated so the caller can actually redirect, not just -/// told "not leader" with nowhere to go +/// - `leader_hint` is populated so the caller can actually redirect, not just told "not leader" +/// with nowhere to go /// - No state changes /// /// This validates the core Raft rule: Only leader can process writes. @@ -1676,7 +1661,7 @@ async fn test_handle_client_write_request_redirects_to_leader() { state.shared_state().set_current_leader(2); let (resp_tx, mut resp_rx) = MaybeCloneOneshot::new(); - let cmd = ClientCmd::Propose( + let cmd = ClientCmd::propose( ClientWriteRequest { client_id: 1, command: Some(bytes::Bytes::from( @@ -1995,7 +1980,8 @@ async fn test_discover_leader_returns_unavailable_when_leader_unknown() { /// The term returned alongside helps the client detect staleness. #[tokio::test] async fn test_discover_leader_returns_known_leader_address() { - use d_engine_proto::common::{NodeRole::Leader, NodeStatus}; + use d_engine_proto::common::NodeRole::Leader; + use d_engine_proto::common::NodeStatus; use d_engine_proto::server::cluster::NodeMeta; let (_graceful_tx, graceful_rx) = watch::channel(()); @@ -2048,8 +2034,8 @@ async fn test_discover_leader_returns_known_leader_address() { /// Test: Follower knows a leader ID but cannot find its metadata — returns NotFound. /// /// Scenario: -/// - Follower has current_leader=3 but membership has no metadata for node 3 -/// (e.g. membership config not yet propagated after cluster reconfiguration) +/// - Follower has current_leader=3 but membership has no metadata for node 3 (e.g. membership +/// config not yet propagated after cluster reconfiguration) /// /// Expected: /// - Response carries Status::NotFound @@ -2095,9 +2081,10 @@ async fn test_discover_leader_returns_not_found_when_metadata_missing() { // ============================================================================ mod snapshot_tests { - use super::*; use std::sync::atomic::Ordering; + use super::*; + /// Test: Follower gracefully ignores stale leader-only internal events /// /// Protocol scenario: a leader steps down to follower while internal events @@ -2296,8 +2283,9 @@ mod snapshot_tests { /// Test: Complete snapshot lifecycle — create, complete, create again /// - /// Validates the full flag cycle: false → true (create) → false (created ok) → true (create again). - /// Ensures the follower can take multiple snapshots over its lifetime without getting stuck. + /// Validates the full flag cycle: false → true (create) → false (created ok) → true (create + /// again). Ensures the follower can take multiple snapshots over its lifetime without + /// getting stuck. #[tokio::test] async fn test_follower_snapshot_lifecycle() { let (_graceful_tx, graceful_rx) = watch::channel(()); @@ -2762,7 +2750,8 @@ async fn test_apply_completed_triggers_snapshot_when_condition_met() { /// Test: Follower ApplyCompleted does NOT trigger snapshot when condition is not met /// -/// Purpose: Verify that followers respect snapshot conditions and don't create unnecessary snapshots. +/// Purpose: Verify that followers respect snapshot conditions and don't create unnecessary +/// snapshots. /// /// Scenario: /// - Follower receives ApplyCompleted event @@ -3749,8 +3738,8 @@ async fn test_follower_install_snapshot_advances_commit_index_to_boundary() { /// to advance match_index and stop retrying, leaving the follower permanently behind. /// /// # Given -/// - prepare_snapshot_stream: sends Accepted ACK (transfer succeeded), -/// then returns Err (apply_snapshot_from_file failed) +/// - prepare_snapshot_stream: sends Accepted ACK (transfer succeeded), then returns Err +/// (apply_snapshot_from_file failed) /// /// # When /// - Leader pushes a snapshot (InstallSnapshotChunk event) @@ -4515,8 +4504,8 @@ fn grant_update( /// Expected: /// - Rejected without consulting the election handler (it would grant on term alone). /// - Response carries this follower's own term, not the requester's. -/// - Term unchanged and nothing persisted: adopting 2400 would make this follower reject the -/// real leader's next heartbeat and force it to step down. +/// - Term unchanged and nothing persisted: adopting 2400 would make this follower reject the real +/// leader's next heartbeat and force it to step down. #[tokio::test(start_paused = true)] async fn test_follower_rejects_vote_request_while_leader_contact_is_recent() { let (_graceful_tx, graceful_rx) = watch::channel(()); @@ -4552,8 +4541,8 @@ async fn test_follower_rejects_vote_request_while_leader_contact_is_recent() { /// handled normally. /// /// Expected: -/// - The guard is a time window, not a permanent block: if the leader really died, an -/// election must still be able to proceed. +/// - The guard is a time window, not a permanent block: if the leader really died, an election must +/// still be able to proceed. #[tokio::test(start_paused = true)] async fn test_follower_handles_vote_request_after_leader_contact_window_expires() { let (_graceful_tx, graceful_rx) = watch::channel(()); @@ -4608,14 +4597,14 @@ async fn test_follower_without_leader_contact_handles_vote_request() { /// Test: granting a vote does not start a new guard window. /// /// Scenario: -/// - Leader contact at t0, window expires, follower grants a vote to candidate 7. -/// (Granting resets the election timer, see the #422 test above.) +/// - Leader contact at t0, window expires, follower grants a vote to candidate 7. (Granting resets +/// the election timer, see the #422 test above.) /// - A moment later, candidate 8 asks for a vote at an even higher term. /// /// Expected: -/// - The second request reaches the election handler. The guard keys on leader contact only; -/// if it keyed on the election timer, the vote just granted would count as "leader alive" -/// and block a legitimate election. +/// - The second request reaches the election handler. The guard keys on leader contact only; if it +/// keyed on the election timer, the vote just granted would count as "leader alive" and block a +/// legitimate election. #[tokio::test(start_paused = true)] async fn test_follower_granting_vote_does_not_refresh_leader_contact_guard() { let (_graceful_tx, graceful_rx) = watch::channel(()); @@ -4771,8 +4760,8 @@ async fn test_follower_pre_vote_after_leader_contact_window_reports_no_active_le /// Test: answering a PreVote changes nothing on the Follower, granted or denied. /// /// Expected: -/// - Term unchanged, nothing persisted, election timer untouched. A PreVote that reset the -/// timer would let a node that cannot win keep every other follower from ever timing out. +/// - Term unchanged, nothing persisted, election timer untouched. A PreVote that reset the timer +/// would let a node that cannot win keep every other follower from ever timing out. #[tokio::test(start_paused = true)] async fn test_follower_pre_vote_changes_no_state() { for contact_leader_first in [false, true] { @@ -4811,7 +4800,6 @@ async fn test_follower_pre_vote_changes_no_state() { // A removed (or never-promoted) node that keeps timing out would otherwise start elections whose // quorum math counts its own vote as part of the cluster: with 4 real voters it needs only // "own vote + 2 grants = 3 of 5" although a majority of the 4 voters is 3 grants. -// openraft skips the election ("not a voter"); raft-rs refuses to campaign when not promotable. // // `Membership::voters()` lists the OTHER active voters (it never contains this node), so "am I a // voter" is answered by this node's own membership entry: present and `Active`. @@ -4892,3 +4880,31 @@ async fn test_follower_tick_starts_election_when_an_active_voter() { assert!(requested); } + +/// Test: a follower rejects a write with NOT_LEADER. +/// +/// Why: a write that reaches a non-leader must be redirected, never silently dropped or +/// accepted. +#[tokio::test] +async fn test_handle_client_propose_redirects_to_leader() { + let (_graceful_tx, graceful_rx) = watch::channel(()); + let (context, _temp_dir) = mock_context_with_leader_address(graceful_rx, 2, "127.0.0.1:9082"); + let mut state = + FollowerState::::new(1, context.node_config.clone(), None, None); + state.shared_state().set_current_leader(2); + + let (resp_tx, mut resp_rx) = MaybeCloneOneshot::new(); + let cmd = ClientCmd::propose( + ClientWriteRequest { + client_id: 1, + command: Some(bytes::Bytes::from( + WriteCommand::delete(bytes::Bytes::new()).encode_to_vec(), + )), + }, + resp_tx, + ); + state.push_client_cmd(cmd, &context); + + let result = resp_rx.recv().await.expect("channel should not be closed"); + assert_eq!(result.expect("Ok response").error, ErrorCode::NotLeader); +} diff --git a/d-engine-core/src/raft_role/leader_state.rs b/d-engine-core/src/raft_role/leader_state.rs index 43eac033..28601bd6 100644 --- a/d-engine-core/src/raft_role/leader_state.rs +++ b/d-engine-core/src/raft_role/leader_state.rs @@ -1,3 +1,47 @@ +use std::collections::BTreeMap; +use std::collections::HashMap; +use std::collections::VecDeque; +use std::fmt::Debug; +use std::marker::PhantomData; +use std::sync::Arc; +use std::sync::atomic::AtomicBool; +use std::sync::atomic::Ordering; +use std::time::Duration; + +use async_trait::async_trait; +use d_engine_proto::common::AddNode; +use d_engine_proto::common::BatchPromote; +use d_engine_proto::common::BatchRemove; +use d_engine_proto::common::EntryPayload; +use d_engine_proto::common::LogId; +use d_engine_proto::common::NodeRole::Follower; +use d_engine_proto::common::NodeRole::Leader; +use d_engine_proto::common::NodeStatus; +use d_engine_proto::common::entry_payload::Payload; +use d_engine_proto::common::membership_change::Change; +use d_engine_proto::server::cluster::ClusterConfUpdateResponse; +use d_engine_proto::server::cluster::JoinRequest; +use d_engine_proto::server::cluster::JoinResponse; +use d_engine_proto::server::cluster::LeaderDiscoveryResponse; +use d_engine_proto::server::cluster::NodeMeta; +use d_engine_proto::server::election::VoteResponse; +use d_engine_proto::server::election::VotedFor; +use d_engine_proto::server::replication::AppendEntriesRequest; +use d_engine_proto::server::replication::AppendEntriesResponse; +use d_engine_proto::server::replication::append_entries_response; +use d_engine_proto::server::storage::SnapshotMetadata; +use futures::StreamExt; +use parking_lot::Mutex; +use rand::distr::SampleString; +use tokio::sync::mpsc; +use tokio::time::Instant; +use tonic::Status; +use tracing::debug; +use tracing::error; +use tracing::info; +use tracing::trace; +use tracing::warn; + use super::LeaderStateSnapshot; use super::RaftRole; use super::SharedState; @@ -40,54 +84,14 @@ use crate::alias::MOF; use crate::alias::ROF; use crate::alias::SMHOF; use crate::alias::TROF; -use crate::client::{ClientReadRequest, ClientResponse, ErrorCode}; +use crate::client::ClientReadRequest; +use crate::client::ClientResponse; +use crate::client::ErrorCode; use crate::event::ClientCmd; use crate::network::Transport; use crate::role_state::PeerReplicationState; use crate::role_state::schedule_and_execute_purge; use crate::utils::cluster_printer::print_role_transition_line; -use async_trait::async_trait; -use d_engine_proto::common::AddNode; -use d_engine_proto::common::BatchPromote; -use d_engine_proto::common::BatchRemove; -use d_engine_proto::common::EntryPayload; -use d_engine_proto::common::LogId; -use d_engine_proto::common::NodeRole::Follower; -use d_engine_proto::common::NodeRole::Leader; -use d_engine_proto::common::NodeStatus; -use d_engine_proto::common::entry_payload::Payload; -use d_engine_proto::common::membership_change::Change; -use d_engine_proto::server::cluster::ClusterConfUpdateResponse; -use d_engine_proto::server::cluster::JoinRequest; -use d_engine_proto::server::cluster::JoinResponse; -use d_engine_proto::server::cluster::LeaderDiscoveryResponse; -use d_engine_proto::server::cluster::NodeMeta; -use d_engine_proto::server::election::VoteResponse; -use d_engine_proto::server::election::VotedFor; -use d_engine_proto::server::replication::AppendEntriesRequest; -use d_engine_proto::server::replication::AppendEntriesResponse; -use d_engine_proto::server::replication::append_entries_response; -use d_engine_proto::server::storage::SnapshotMetadata; -use futures::StreamExt; -use parking_lot::Mutex; -use rand::distr::SampleString; -use std::collections::BTreeMap; -use std::collections::HashMap; -use std::collections::VecDeque; -use std::fmt::Debug; -use std::marker::PhantomData; -use std::sync::Arc; -use std::sync::atomic::AtomicBool; -use std::sync::atomic::Ordering; -use std::time::Duration; -use tokio::sync::mpsc; -use tokio::time::Instant; -use tonic::Status; -use tracing::debug; -use tracing::error; -use tracing::info; -use tracing::trace; -use tracing::warn; // Type aliases for improved readability type LinearizableReadRequest = ( @@ -290,6 +294,10 @@ impl PeerMetrics { /// when `task_tx` is dropped, `task_rx.recv()` returns `None` and the worker exits cleanly. struct ReplicationWorkerHandle { task_tx: mpsc::UnboundedSender, + /// `true` while the worker holds an open replication stream to the peer. + /// Written only by the worker; the leader reads it to avoid queueing work the peer cannot + /// receive. + stream_open: Arc, /// Consecutive snapshot push failures for this peer since the last success. snapshot_failure_count: u32, /// Earliest time the next snapshot push attempt is allowed (exponential backoff window). @@ -331,7 +339,7 @@ pub struct LeaderState { /// `now_ms()` of the latest quorum ACK, or of the election for a new leader (one election /// timeout of grace to collect the first ACKs). CheckQuorum steps down when it is older than /// `election_timeout_max`, the longest silence any follower tolerates before campaigning - /// (etcd's effective tolerance is the same). Unlike `quorum_ack_deadline_ms` (a lease + /// (the same tolerance a follower effectively has). Unlike `quorum_ack_deadline_ms` (a lease /// window, shorter than the election timeout) a slow reply or brief jitter does not trip it. last_quorum_contact_ms: u64, @@ -393,9 +401,9 @@ pub struct LeaderState { /// /// Accumulates requests until either: /// 1. Batch reaches configured size limit - /// 2. Explicit flush is triggered - /// SoA layout: payloads and senders stored in separate contiguous Vecs. - /// flush() builds one RaftRequestWithSignal via mem::take — true O(1), no unzip scan. + /// 2. Explicit flush is triggered SoA layout: payloads and senders stored in separate + /// contiguous Vecs. flush() builds one RaftRequestWithSignal via mem::take — true O(1), no + /// unzip scan. pub(super) propose_buffer: Box, // -- Timing & Scheduling -- @@ -489,8 +497,8 @@ pub struct LeaderState { /// Pending client write batches indexed by end_log_index. /// Key: end_log_index of this write batch. Value: WriteMetadata (senders + deadline). - /// Drained when commit_index advances past end_log_index (quorum achieved or single-voter flush). - /// Drained with error on step-down / HigherTerm / deadline exceeded. + /// Drained when commit_index advances past end_log_index (quorum achieved or single-voter + /// flush). Drained with error on step-down / HigherTerm / deadline exceeded. pub(super) pending_client_writes: BTreeMap, /// Post-commit action queue for Raft protocol internal operations (noop, join). @@ -514,8 +522,9 @@ pub struct LeaderState { pub(super) pending_commit_actions: BTreeMap, /// Per-follower replication worker handles. Key = follower node_id. - /// Workers send AppendEntries via transport and relay results back as InternalEvent::AppendResult. - /// Dropped on role change (LeaderState drop) → workers exit via channel close. + /// Workers send AppendEntries via transport and relay results back as + /// InternalEvent::AppendResult. Dropped on role change (LeaderState drop) → workers exit + /// via channel close. replication_workers: HashMap, // -- Type System Marker -- @@ -740,7 +749,8 @@ impl RaftRoleState for LeaderState { error!("Stale learner check failed: {}", e); } - // 2. CheckQuorum: a leader no quorum has answered for an ( election_timeout_max × multiple) timeout may be cut off. + // 2. CheckQuorum: a leader no quorum has answered for an ( election_timeout_max × multiple) + // timeout may be cut off. // Step down (same term) so clients are redirected instead of queueing writes that can // never commit. A single voter is its own quorum. let quorum_silence_limit_ms = @@ -955,7 +965,10 @@ impl RaftRoleState for LeaderState { let backpressure = &ctx.node_config.raft.backpressure; match cmd { - ClientCmd::Propose(req, sender) => { + ClientCmd::Propose(req, issued_at, sender) => { + metrics::histogram!("core.raft.client_cmd.issue_to_push_ms") + .record(issued_at.elapsed().as_secs_f64() * 1_000.0); + let current_pending = self.propose_buffer.len(); // Check write backpressure limit @@ -1128,9 +1141,9 @@ impl RaftRoleState for LeaderState { } // Leader receives RequestVote(term=X, candidate=Y) - // Deviates from Raft §5.1 on purpose (dissertation §4.2.3, disruptive servers): a leader - // that a quorum confirmed within the lease window ignores a higher term, so an - // isolated node that comes back cannot depose a healthy leader. + // Deviates from Raft §5.1 on purpose (dissertation §4.2.3, disruptive servers): a + // leader that a quorum confirmed within the lease window ignores a higher + // term, so an isolated node that comes back cannot depose a healthy leader. // 1. If X > currentTerm and no quorum ACK within the lease window: // - Leader → Follower, currentTerm = X // - Replay event @@ -1449,7 +1462,8 @@ impl RaftRoleState for LeaderState { Ok(()) } - /// Handle LogFlushed(durable): re-calculate commit_index after local log entries are crash-safe. + /// Handle LogFlushed(durable): re-calculate commit_index after local log entries are + /// crash-safe. /// /// Leader also writes to the local raft log. process_batch reads durable_index() synchronously, /// which may be 0 (flush not yet complete) — so commit never advances without this handler. @@ -1477,16 +1491,8 @@ impl RaftRoleState for LeaderState { }; if let Some(new_commit) = next_commit_index { - if let Err(e) = self.update_commit_index_with_signal( - Leader as i32, - self.current_term(), - new_commit, - internal_event_tx, - ) { - error!( - ?e, - "handle_log_flushed: update_commit_index_with_signal failed" - ); + if let Err(e) = self.update_commit_index(new_commit) { + error!(?e, "handle_log_flushed: update_commit_index failed"); } else { // Drain pending writes committed via local flush (single-voter or flush path). self.drain_pending_client_writes(new_commit); @@ -1670,12 +1676,7 @@ impl RaftRoleState for LeaderState { // `majority` answers the COMMIT question only. if let Some(new_commit) = Self::new_commit_index(commit_index, majority) { - self.update_commit_index_with_signal( - Leader as i32, - self.current_term(), - new_commit, - internal_event_tx, - )?; + self.update_commit_index(new_commit)?; self.drain_pending_client_writes(new_commit); self.drain_commit_actions(new_commit, ctx, internal_event_tx).await; } @@ -2044,11 +2045,17 @@ impl LeaderState { cfg: ReplicationWorkerConfig, ) -> ReplicationWorkerHandle { let (task_tx, task_rx) = mpsc::unbounded_channel::(); - tokio::spawn(async move { - Self::run_replication_worker(peer_id, task_rx, cfg).await; + // Starts closed: the worker opens the stream and flips it. + let stream_open = Arc::new(AtomicBool::new(false)); + tokio::spawn({ + let stream_open = Arc::clone(&stream_open); + async move { + Self::run_replication_worker(peer_id, task_rx, cfg, stream_open).await; + } }); ReplicationWorkerHandle { task_tx, + stream_open, snapshot_failure_count: 0, snapshot_next_retry_at: None, } @@ -2068,6 +2075,7 @@ impl LeaderState { peer_id: u32, mut task_rx: mpsc::UnboundedReceiver, cfg: ReplicationWorkerConfig, + stream_open: Arc, ) { let ReplicationWorkerConfig { transport, @@ -2102,6 +2110,8 @@ impl LeaderState { // Open persistent bidi stream with backoff reconnection loop { + // (Re-)entering the connect phase: no open stream. + stream_open.store(false, Ordering::Relaxed); let mut backoff_ms = base_delay_ms; let stream = loop { if task_rx.is_closed() { @@ -2124,6 +2134,7 @@ impl LeaderState { "peer_id" => peer_id.to_string() ) .increment(1); + stream_open.store(true, Ordering::Relaxed); break s; } Err(e) => { @@ -2322,6 +2333,19 @@ impl LeaderState { self.replication_workers.insert(peer_id, handle); } + /// True if `peer_id` has a live worker but no open stream. Dispatching to it would only + /// queue work it cannot send until it reconnects. A dead worker (`task_tx` closed) is not + /// "down": the next send rebuilds it. With no worker yet, returns `false` so the dispatch + /// that spawns it goes through. + fn peer_stream_is_down( + &self, + peer_id: u32, + ) -> bool { + self.replication_workers + .get(&peer_id) + .is_some_and(|w| !w.stream_open.load(Ordering::Relaxed) && !w.task_tx.is_closed()) + } + // ---- Snapshot push backoff + alerting ------------------------------------------------- /// Update per-peer snapshot push state after a completed (success or failure) attempt. @@ -2339,7 +2363,9 @@ impl LeaderState { policy: &crate::InstallSnapshotBackoffPolicy, node_id: u32, ) { - // Symmetric guard: only act if this peer is still actually in Snapshot state. A stale/duplicate completion event (e.g. left over from an earlier failed attempt) arriving after the state has already moved on is a no-op — no need for a separate session id. + // Symmetric guard: only act if this peer is still actually in Snapshot state. A + // stale/duplicate completion event (e.g. left over from an earlier failed attempt) arriving + // after the state has already moved on is a no-op — no need for a separate session id. if self.peer_replication_state(peer_id) != PeerReplicationState::Snapshot { return; } @@ -2383,8 +2409,8 @@ impl LeaderState { ) .set(0.0); // #436: push resolved either way — stop blocking AppendEntries, but don't - // trust the peer yet either. Wait for a real ACK (etcd: MsgSnapStatus, - // success or reject, both -> BecomeProbe, never straight to Replicate). + // trust the peer yet either. Wait for a real ACK (success or reject both go to Probe, + // never straight to Replicate). self.set_peer_replication_state(peer_id, PeerReplicationState::Probe); } @@ -2425,14 +2451,11 @@ impl LeaderState { for (i, sender) in senders.into_iter().enumerate() { metrics::histogram!("core.raft.write.propose_to_commit_ms") .record(propose_to_commit_ms); - self.pending_write_apply.insert( - start_idx + i as u64, - PendingApply { - sender, - proposed_at, - committed_at: commit_ts, - }, - ); + self.pending_write_apply.insert(start_idx + i as u64, PendingApply { + sender, + proposed_at, + committed_at: commit_ts, + }); } } else { // Internal requests have no senders, so this loop is a no-op for them. @@ -2470,13 +2493,10 @@ impl LeaderState { // synchronous drain_commit_actions call (single-voter path) can find the entry. // No concurrent writes: Raft loop is single-threaded. let noop_index = ctx.raft_log().last_entry_id() + 1; - self.pending_commit_actions.insert( - noop_index, - PostCommitEntry { - deadline, - action: PostCommitAction::LeaderNoop { term }, - }, - ); + self.pending_commit_actions.insert(noop_index, PostCommitEntry { + deadline, + action: PostCommitAction::LeaderNoop { term }, + }); // Write noop with no sender — purely for commit confirmation. if let Err(e) = self @@ -2791,8 +2811,8 @@ impl LeaderState { /// Two distinct update strategies based on response type: /// - Success: `next_index = max(current, peer_match + 1)` — never regress a speculative /// advance; the ACK is a lower-bound confirmation, not a correction. - /// - Conflict: `next_index = conflict_hint` (floor-guarded by `match_index + 1`) — - /// the follower explicitly corrects the leader's assumption; retreat is required. + /// - Conflict: `next_index = conflict_hint` (floor-guarded by `match_index + 1`) — the follower + /// explicitly corrects the leader's assumption; retreat is required. pub(super) fn update_peer_index( &mut self, follower_id: u32, @@ -2825,7 +2845,7 @@ impl LeaderState { error!("Failed to update next index: {:?}", e); } // #436: an explicit reject means our ledger was wrong — stop trusting - // speculative advance until the next real ACK (etcd: BecomeProbe on reject). + // speculative advance until the next real ACK. self.set_peer_replication_state(follower_id, PeerReplicationState::Probe); } @@ -3221,17 +3241,14 @@ impl LeaderState { // Record post-commit action: when the config index commits, send JoinResponse. let join_index = ctx.raft_log().last_entry_id(); - self.pending_commit_actions.insert( - join_index, - PostCommitEntry { - deadline, - action: PostCommitAction::NodeJoin { - node_id, - addr: address, - sender, - }, + self.pending_commit_actions.insert(join_index, PostCommitEntry { + deadline, + action: PostCommitAction::NodeJoin { + node_id, + addr: address, + sender, }, - ); + }); debug!("JoinCluster: awaiting commit at index {}", join_index); Ok(()) @@ -3419,7 +3436,8 @@ impl LeaderState { /// /// Commit and client responses are deferred: /// - Multi-voter: driven by handle_append_result (InternalEvent::AppendResult from workers) - /// - Single-voter: driven by handle_log_flushed (InternalEvent::LogFlushed from batch_processor) + /// - Single-voter: driven by handle_log_flushed (InternalEvent::LogFlushed from + /// batch_processor) async fn execute_and_process_raft_rpc( &mut self, payloads: Vec, @@ -3435,7 +3453,8 @@ impl LeaderState { // Phase 0: Record send timestamp before any AppendEntries RPC goes out. // handle_append_result uses this to anchor deadline to send time (not ACK time), - // ensuring lease expires at send_ts + lease_duration_ms rather than ack_ts + lease_duration_ms. + // ensuring lease expires at send_ts + lease_duration_ms rather than ack_ts + + // lease_duration_ms. self.last_heartbeat_send_ts = now_ms(); // Pre-compute in-flight gating decisions for all replication targets. Single @@ -3569,12 +3588,18 @@ impl LeaderState { if self.peer_replication_state(peer_id) == PeerReplicationState::Snapshot { continue; } + // No open stream: skip before any bookkeeping (`next_index`, in-flight) so the first + // send after reconnect resumes from the same point. Raft retransmits from `next_index`. + if self.peer_stream_is_down(peer_id) { + continue; + } let is_heartbeat = request.entries.is_empty(); // Phase 1: lightweight in-flight gate (Probe state only, window=1). - // For Replicate state, window check happens in Phase 2 (retrieve_to_be_synced_logs_for_peers). - // Gated peers get no data; heartbeats bypass the gate and never occupy a slot. + // For Replicate state, window check happens in Phase 2 + // (retrieve_to_be_synced_logs_for_peers). Gated peers get no data; + // heartbeats bypass the gate and never occupy a slot. if !is_heartbeat && peer_gating_decisions.get(&peer_id).copied().unwrap_or(false) { self.peer_metrics(peer_id).dispatch_gated_total.increment(1); continue; @@ -3734,10 +3759,10 @@ impl LeaderState { node_id, "Stale learner detected: pending promotion threshold exceeded" ); - metrics::counter!( - "core.membership.stale_learner_removed", - &[("node_id", node_id.to_string())] - ) + metrics::counter!("core.membership.stale_learner_removed", &[( + "node_id", + node_id.to_string() + )]) .increment(1); if let Err(e) = self.handle_stale_learner(node_id, internal_event_tx, ctx).await { error!(node_id, ?e, "Failed to handle stale learner"); @@ -3968,9 +3993,9 @@ impl LeaderState { /// /// `send_ts` is the timestamp recorded just before the heartbeat was sent. /// Using send_ts (not now_ms()) eliminates the RTT/2 window: - /// deadline = send_ts + lease_duration_ms ≤ follower's election timer start + election_timeout - /// Single-voter callers pass now_ms() directly (no peer RTT to account for). - /// `last_applied` is the state machine's applied index at call time. + /// deadline = send_ts + lease_duration_ms ≤ follower's election timer start + + /// election_timeout Single-voter callers pass now_ms() directly (no peer RTT to account + /// for). `last_applied` is the state machine's applied index at call time. fn update_lease_timestamp( &mut self, send_ts: u64, @@ -4015,7 +4040,8 @@ impl LeaderState { senders: req.senders, wait_for_apply: req.wait_for_apply_event, deadline: Instant::now(), // placeholder; overwritten in Phase 2 - proposed_at: std::time::Instant::now(), // placeholder; overwritten in Phase 2 + proposed_at: std::time::Instant::now(), /* placeholder; overwritten in + * Phase 2 */ }), ) } else { @@ -4122,14 +4148,12 @@ impl LeaderState { ) { let (dead_tx, dead_rx) = tokio::sync::mpsc::unbounded_channel::(); drop(dead_rx); // receiver gone → sends will fail - self.replication_workers.insert( - peer_id, - ReplicationWorkerHandle { - task_tx: dead_tx, - snapshot_failure_count: 0, - snapshot_next_retry_at: None, - }, - ); + self.replication_workers.insert(peer_id, ReplicationWorkerHandle { + task_tx: dead_tx, + snapshot_failure_count: 0, + snapshot_next_retry_at: None, + stream_open: Arc::new(AtomicBool::new(true)), + }); } /// Determine effective read consistency policy based on client request and server config diff --git a/d-engine-core/src/raft_role/leader_state_test/backpressure_test.rs b/d-engine-core/src/raft_role/leader_state_test/backpressure_test.rs index 25c856f4..825efba0 100644 --- a/d-engine-core/src/raft_role/leader_state_test/backpressure_test.rs +++ b/d-engine-core/src/raft_role/leader_state_test/backpressure_test.rs @@ -1,6 +1,3 @@ -use crate::client::ClientReadRequest; -use crate::client::ClientWriteRequest; -use crate::config::ReadConsistencyPolicy; use bytes::Bytes; use d_engine_proto::client::WriteCommand; use tokio::sync::watch; @@ -9,8 +6,12 @@ use crate::BackpressureConfig; use crate::ClientCmd; use crate::MaybeCloneOneshot; use crate::RaftOneshot; +use crate::client::ClientReadRequest; +use crate::client::ClientWriteRequest; +use crate::config::ReadConsistencyPolicy; use crate::raft_role::leader_state::LeaderState; use crate::role_state::RaftRoleState; +use crate::test_utils::MetricsCapture; use crate::test_utils::mock::MockTypeConfig; use crate::test_utils::mock::mock_raft_context; @@ -39,7 +40,7 @@ async fn test_backpressure_write_limit_enforcement() { // Push first two writes - should succeed for i in 1..=2 { let (resp_tx, _resp_rx) = MaybeCloneOneshot::new(); - let cmd = ClientCmd::Propose( + let cmd = ClientCmd::propose( ClientWriteRequest { client_id: i, command: Some({ @@ -60,7 +61,7 @@ async fn test_backpressure_write_limit_enforcement() { // Push third write - should be rejected let (resp_tx, mut resp_rx) = MaybeCloneOneshot::new(); - let cmd = ClientCmd::Propose( + let cmd = ClientCmd::propose( ClientWriteRequest { client_id: 3, command: Some({ @@ -180,7 +181,7 @@ async fn test_backpressure_unlimited_when_zero() { // Push many writes - all should succeed for i in 1..=100 { let (resp_tx, _resp_rx) = MaybeCloneOneshot::new(); - let cmd = ClientCmd::Propose( + let cmd = ClientCmd::propose( ClientWriteRequest { client_id: i, command: Some({ @@ -225,7 +226,7 @@ async fn test_backpressure_write_and_read_independent() { // Fill write buffer for i in 1..=2 { let (resp_tx, _resp_rx) = MaybeCloneOneshot::new(); - let cmd = ClientCmd::Propose( + let cmd = ClientCmd::propose( ClientWriteRequest { client_id: i, command: Some({ @@ -240,7 +241,7 @@ async fn test_backpressure_write_and_read_independent() { // Writes should be rejected let (write_tx, mut write_rx) = MaybeCloneOneshot::new(); - let write_cmd = ClientCmd::Propose( + let write_cmd = ClientCmd::propose( ClientWriteRequest { client_id: 99, command: Some({ @@ -376,3 +377,42 @@ async fn test_backpressure_all_read_policies() { "Should reject with ResourceExhausted" ); } + +/// Test: `Propose` records the wait since the client issued the request. +/// +/// Why: `issue_to_push_ms` must reflect the real elapsed time since the client's +/// `issued_at`, not zero. The variant is constructed directly (not via `ClientCmd::propose`) +/// so the test can stamp a fixed past timestamp. +#[tokio::test] +async fn test_propose_records_issue_wait() { + let capture = MetricsCapture::new(); + let _guard = metrics::set_default_local_recorder(&capture); + + let (_shutdown_tx, shutdown_rx) = watch::channel(()); + let ctx = mock_raft_context("/tmp/test_propose_records_issue_wait", shutdown_rx, None); + let mut leader = LeaderState::::new(1, ctx.node_config.clone()); + + let issued_at = tokio::time::Instant::now() - std::time::Duration::from_millis(20); + let (resp_tx, _resp_rx) = MaybeCloneOneshot::new(); + let cmd = ClientCmd::Propose( + ClientWriteRequest { + client_id: 1, + command: Some({ + use prost::Message; + Bytes::from(WriteCommand::delete(Bytes::new()).encode_to_vec()) + }), + }, + issued_at, + resp_tx, + ); + leader.push_client_cmd(cmd, &ctx); + + assert_eq!(leader.propose_buffer.len(), 1, "must be buffered"); + let waits = capture.histogram("core.raft.client_cmd.issue_to_push_ms", &[]); + assert_eq!(waits.len(), 1, "exactly one wait sample per request"); + assert!( + waits[0] >= 20.0, + "wait must cover the elapsed time since issue, got {}", + waits[0] + ); +} diff --git a/d-engine-core/src/raft_role/leader_state_test/become_follower_test.rs b/d-engine-core/src/raft_role/leader_state_test/become_follower_test.rs index d6d4bfe3..5acec72b 100644 --- a/d-engine-core/src/raft_role/leader_state_test/become_follower_test.rs +++ b/d-engine-core/src/raft_role/leader_state_test/become_follower_test.rs @@ -12,18 +12,22 @@ use std::sync::Arc; -use tokio::sync::{mpsc, watch}; +use d_engine_proto::server::election::VoteRequest; +use d_engine_proto::server::replication::AppendEntriesRequest; +use d_engine_proto::server::replication::AppendEntriesResponse; +use tokio::sync::mpsc; +use tokio::sync::watch; use crate::RaftNodeConfig; -use crate::event::{InboundEvent, InternalEvent}; -use crate::maybe_clone_oneshot::{MaybeCloneOneshot, RaftOneshot}; +use crate::event::InboundEvent; +use crate::event::InternalEvent; +use crate::maybe_clone_oneshot::MaybeCloneOneshot; +use crate::maybe_clone_oneshot::RaftOneshot; use crate::now_ms; use crate::raft_role::leader_state::LeaderState; use crate::raft_role::role_state::RaftRoleState; use crate::test_utils::MockBuilder; use crate::test_utils::mock::MockTypeConfig; -use d_engine_proto::server::election::VoteRequest; -use d_engine_proto::server::replication::{AppendEntriesRequest, AppendEntriesResponse}; /// Simulates a quorum ACK of a heartbeat sent just now, with the noop already applied. /// Goes through the real `update_lease_timestamp`, so whatever it records (read lease, @@ -75,8 +79,8 @@ async fn test_become_follower_revokes_read_lease() { /// - A node returning from a partition asks for a vote with an inflated term (999). /// /// Expected: -/// - No `BecomeFollower` event: the cluster is healthy, so the returning node must not -/// force a re-election. +/// - No `BecomeFollower` event: the cluster is healthy, so the returning node must not force a +/// re-election. /// - Response is a rejection carrying the leader's own term. #[tokio::test] async fn test_receive_higher_term_vote_request_with_valid_lease_is_rejected() { @@ -169,10 +173,10 @@ async fn test_receive_higher_term_append_entries_revokes_lease() { /// Rejecting a higher-term VoteRequest has no side effects on a leader with a valid lease. /// /// Expected: -/// - Leader term unchanged: adopting 999 would make the leader reject its own followers' -/// term-1 traffic and break a healthy cluster. -/// - The read lease is untouched, so LeaseRead keeps being served; revoking it here would -/// punish readers for a vote request the leader ignored. +/// - Leader term unchanged: adopting 999 would make the leader reject its own followers' term-1 +/// traffic and break a healthy cluster. +/// - The read lease is untouched, so LeaseRead keeps being served; revoking it here would punish +/// readers for a vote request the leader ignored. #[tokio::test] async fn test_vote_request_higher_term_with_valid_lease_leaves_term_and_lease_untouched() { let (_graceful_tx, graceful_rx) = watch::channel(()); @@ -209,8 +213,8 @@ async fn test_vote_request_higher_term_with_valid_lease_leaves_term_and_lease_un /// A leader without a valid lease still steps down on a higher-term VoteRequest. /// /// Expected: -/// - The guard is bounded by the lease: with no quorum acknowledgement in the window the -/// leader may really have lost leadership, so Raft's usual "higher term wins" applies. +/// - The guard is bounded by the lease: with no quorum acknowledgement in the window the leader may +/// really have lost leadership, so Raft's usual "higher term wins" applies. /// - After the Raft loop runs `become_follower()`, the lease stays invalid. #[tokio::test] async fn test_receive_higher_term_vote_request_without_valid_lease_steps_down() { @@ -292,8 +296,9 @@ async fn test_vote_request_higher_term_without_valid_lease_adopts_term_and_repla /// Higher-term AppendEntries → lease revoked BEFORE Raft loop calls become_follower(). /// -/// Same window-period contract as test_vote_request_higher_term_lease_revoked_before_become_follower, -/// for the AppendEntries detection path. +/// Same window-period contract as +/// test_vote_request_higher_term_lease_revoked_before_become_follower, for the AppendEntries +/// detection path. #[tokio::test] async fn test_append_entries_higher_term_lease_revoked_before_become_follower() { let (_graceful_tx, graceful_rx) = watch::channel(()); @@ -392,10 +397,10 @@ async fn test_append_result_higher_term_lease_revoked_before_become_follower() { /// higher term, BEFORE stepping down to Follower. /// /// Scenario: -/// - Leader is at term=1, receives an AppendEntries response from a follower -/// carrying a higher term (999) — this is the response-side term check -/// (distinct from the request-side ones already covered in candidate/follower -/// tests), flagged during review as previously uncovered by any test. +/// - Leader is at term=1, receives an AppendEntries response from a follower carrying a higher term +/// (999) — this is the response-side term check (distinct from the request-side ones already +/// covered in candidate/follower tests), flagged during review as previously uncovered by any +/// test. /// /// Expected: /// - `save_hard_state` is called exactly once with term=999. @@ -502,13 +507,13 @@ async fn test_append_result_higher_term_revokes_lease() { /// /// Scenario: /// - Leader just elected: noop at index 101, state machine applied only up to 100. -/// - A quorum ACK arrives. The read lease stays withheld (readers fall back to the Raft loop), -/// but a quorum has just confirmed this leader. +/// - A quorum ACK arrives. The read lease stays withheld (readers fall back to the Raft loop), but +/// a quorum has just confirmed this leader. /// - A node returning from a partition asks for a vote with term 999. /// /// Expected: -/// - The vote request is rejected and the leader stays leader. "A quorum recently confirmed me" -/// is what makes a step-down pointless; it must not depend on the state machine catching up. +/// - The vote request is rejected and the leader stays leader. "A quorum recently confirmed me" is +/// what makes a step-down pointless; it must not depend on the state machine catching up. #[tokio::test] async fn test_leader_before_noop_applied_still_rejects_higher_term_vote_request() { let (_graceful_tx, graceful_rx) = watch::channel(()); @@ -554,8 +559,8 @@ async fn test_leader_before_noop_applied_still_rejects_higher_term_vote_request( /// VoteRequest makes the leader step down again. /// /// Expected: -/// - The protection is time-bounded: a leader nobody has confirmed recently may really have -/// lost leadership, so "higher term wins" applies. +/// - The protection is time-bounded: a leader nobody has confirmed recently may really have lost +/// leadership, so "higher term wins" applies. #[tokio::test] async fn test_leader_with_expired_quorum_ack_steps_down_on_higher_term_vote_request() { let (_graceful_tx, graceful_rx) = watch::channel(()); @@ -649,8 +654,8 @@ async fn leader_pre_vote( /// but still does not step down. /// /// Expected: -/// - `leader_active == false`, the handler's answer is forwarded, no event is emitted, the -/// term is unchanged. +/// - `leader_active == false`, the handler's answer is forwarded, no event is emitted, the term is +/// unchanged. #[tokio::test] async fn test_leader_pre_vote_without_quorum_ack_reports_inactive_and_does_not_step_down() { let (_graceful_tx, graceful_rx) = watch::channel(()); diff --git a/d-engine-core/src/raft_role/leader_state_test/buffer_cleanup_test.rs b/d-engine-core/src/raft_role/leader_state_test/buffer_cleanup_test.rs index 7eadda1c..a1568a81 100644 --- a/d-engine-core/src/raft_role/leader_state_test/buffer_cleanup_test.rs +++ b/d-engine-core/src/raft_role/leader_state_test/buffer_cleanup_test.rs @@ -1,3 +1,10 @@ +use std::sync::Arc; + +use bytes::Bytes; +use d_engine_proto::client::WriteCommand; +use prost::Message; +use tokio::sync::watch; + use crate::ClientCmd; use crate::InternalEvent; use crate::MaybeCloneOneshot; @@ -9,11 +16,6 @@ use crate::client::ClientWriteRequest; use crate::client::ErrorCode; use crate::raft_role::role_state::RaftRoleState; use crate::test_utils::mock::mock_membership; -use bytes::Bytes; -use d_engine_proto::client::WriteCommand; -use prost::Message; -use std::sync::Arc; -use tokio::sync::watch; // ============================================================================ // Role-Specific Behavior Tests @@ -99,7 +101,7 @@ async fn test_leader_stepdown_clears_pending_write_buffer() { )), }; - let cmd = ClientCmd::Propose(write_req, response_tx); + let cmd = ClientCmd::propose(write_req, response_tx); leader.push_client_cmd(cmd, &raft.ctx); } @@ -178,7 +180,7 @@ async fn test_leader_stepdown_clears_pending_write_buffer() { )), }; - let cmd = ClientCmd::Propose(write_req, response_tx); + let cmd = ClientCmd::propose(write_req, response_tx); follower.push_client_cmd(cmd, &raft.ctx); let result = diff --git a/d-engine-core/src/raft_role/leader_state_test/client_read_test.rs b/d-engine-core/src/raft_role/leader_state_test/client_read_test.rs index 23abb36d..c412a256 100644 --- a/d-engine-core/src/raft_role/leader_state_test/client_read_test.rs +++ b/d-engine-core/src/raft_role/leader_state_test/client_read_test.rs @@ -1,9 +1,28 @@ +use std::sync::Arc; +use std::time::Duration; + +use bytes::Bytes; +use d_engine_proto::common::NodeRole::Follower; +use d_engine_proto::common::NodeStatus; +use d_engine_proto::server::cluster::NodeMeta; +use tokio::sync::mpsc; +use tokio::sync::watch; +use tokio::time::Instant; +use tonic::Code; +use tracing_test::traced_test; + use crate::ClientCmd; +use crate::Error; use crate::MockMembership; +use crate::MockRaftLog; +use crate::MockReplicationCore; use crate::MockStateMachineHandler; +use crate::RaftNodeConfig; use crate::ReadConsistencyPolicy; use crate::ScanResult; -use crate::client::{ClientReadRequest, ClientResponsePayload, ErrorCode}; +use crate::client::ClientReadRequest; +use crate::client::ClientResponsePayload; +use crate::client::ErrorCode; use crate::convert::safe_kv_bytes; use crate::maybe_clone_oneshot::MaybeCloneOneshot; use crate::maybe_clone_oneshot::RaftOneshot; @@ -13,18 +32,6 @@ use crate::raft_role::role_state::RaftRoleState; use crate::test_utils::MockBuilder; use crate::test_utils::mock::MockTypeConfig; use crate::test_utils::node_config; -use crate::{Error, MockRaftLog, MockReplicationCore, RaftNodeConfig}; -use bytes::Bytes; -use d_engine_proto::common::NodeRole::Follower; -use d_engine_proto::common::NodeStatus; -use d_engine_proto::server::cluster::NodeMeta; -use std::sync::Arc; -use std::time::Duration; -use tokio::sync::mpsc; -use tokio::sync::watch; -use tokio::time::Instant; -use tonic::Code; -use tracing_test::traced_test; // ============================================================================ // Unit Tests for LinearizableRead Core Functions @@ -468,7 +475,6 @@ async fn test_linearizable_read_quorum_failure() { /// /// # Then /// - Leader's commit_index updates to 3 -/// - NotifyNewCommitIndex event is sent /// - Client receives successful response with data /// /// # Raft Protocol Context @@ -564,8 +570,8 @@ async fn test_linearizable_read_quorum_success() { // Then: Client receives successful response (released by ApplyCompleted) assert!(resp_rx.recv().await.unwrap().is_ok()); - // Note: In new architecture, NotifyNewCommitIndex is sent by handle_append_result, - // not by flush_cmd_buffers. This test focuses on the read path completion. + // Note: the commit index is advanced by handle_append_result, not by flush_cmd_buffers. + // This test focuses on the read path completion. } /// Test linearizable read encountering higher term during verification @@ -1377,8 +1383,7 @@ async fn test_eventual_consistency_ignores_stale_lease() { /// /// **Design Decision**: /// Developer-friendly: d-engine trusts applications to understand their -/// consistency requirements. This mirrors etcd's consistency model where -/// clients control read semantics. +/// consistency requirements: clients control read semantics. #[tokio::test] #[traced_test] async fn test_client_policy_override_allowed() { @@ -2001,8 +2006,8 @@ async fn test_linearizable_read_rejected_when_noop_not_committed() { /// The fix: `process_lease_read` no longer calls `verify_internal_quorum`. Instead it: /// 1. Pushes the request into `pending_lease_reads` /// 2. Fires an empty AppendEntries heartbeat (fire-and-forget) -/// 3. Returns immediately — `drain_pending_lease_reads` is called from `handle_append_result` -/// when quorum ACK arrives +/// 3. Returns immediately — `drain_pending_lease_reads` is called from `handle_append_result` when +/// quorum ACK arrives /// /// # Expected (Fixed) /// - `process_lease_read` returns in <1ms (no blocking) @@ -2300,8 +2305,9 @@ async fn test_scan_cmd_leader_delivers_result() { #[tokio::test] #[traced_test] async fn test_scan_cmd_leader_propagates_sm_error() { + use crate::Error; + use crate::StorageError; use crate::test_utils::mock_state_machine; - use crate::{Error, StorageError}; let mut sm = mock_state_machine(); sm.expect_scan_prefix().returning(|_prefix| { diff --git a/d-engine-core/src/raft_role/leader_state_test/client_write_test.rs b/d-engine-core/src/raft_role/leader_state_test/client_write_test.rs index ce49ee36..1b966828 100644 --- a/d-engine-core/src/raft_role/leader_state_test/client_write_test.rs +++ b/d-engine-core/src/raft_role/leader_state_test/client_write_test.rs @@ -3,6 +3,26 @@ //! This module tests the `process_raft_request` method and related client write //! operations in the leader state. +use std::collections::VecDeque; +use std::sync::Arc; +use std::sync::atomic::AtomicU64; +use std::sync::atomic::Ordering; + +use bytes::Bytes; +use d_engine_proto::client::WriteCommand; +use d_engine_proto::common::AddNode; +use d_engine_proto::common::Entry; +use d_engine_proto::common::EntryPayload; +use d_engine_proto::common::NodeRole::Follower; +use d_engine_proto::common::NodeStatus; +use d_engine_proto::common::membership_change::Change; +use d_engine_proto::server::cluster::NodeMeta; +use d_engine_proto::server::replication::AppendEntriesRequest; +use rand::distr::SampleString; +use tokio::sync::mpsc; +use tokio::sync::watch; +use tokio::time::Instant; + use crate::ApplyResult; use crate::ClientCmd; use crate::MockRaftLog; @@ -10,7 +30,7 @@ use crate::MockReplicationCore; use crate::PrepareResult; use crate::RaftRequestWithSignal; use crate::client::ClientWriteRequest; -use crate::event::{InternalEvent, NewCommitData}; +use crate::event::InternalEvent; use crate::maybe_clone_oneshot::MaybeCloneOneshot; use crate::maybe_clone_oneshot::RaftOneshot; use crate::mock_raft_context; @@ -21,23 +41,6 @@ use crate::role_state::RaftRoleState; use crate::test_utils::MockBuilder; use crate::test_utils::mock::MockTypeConfig; use crate::test_utils::node_config; -use bytes::Bytes; -use d_engine_proto::client::WriteCommand; -use d_engine_proto::common::AddNode; -use d_engine_proto::common::Entry; -use d_engine_proto::common::EntryPayload; -use d_engine_proto::common::NodeRole::Follower; -use d_engine_proto::common::NodeStatus; -use d_engine_proto::common::membership_change::Change; -use d_engine_proto::server::cluster::NodeMeta; -use d_engine_proto::server::replication::AppendEntriesRequest; -use rand::distr::SampleString; -use std::collections::VecDeque; -use std::sync::Arc; -use std::sync::atomic::AtomicU64; -use std::sync::atomic::Ordering; -use tokio::sync::{mpsc, watch}; -use tokio::time::Instant; // ============================================================================ // Test Helper Structures and Functions @@ -145,7 +148,6 @@ async fn setup_process_raft_request_test_context( /// - Request is handled immediately (not batched) /// - Commit index advances to 5 /// - Peer next_index values are updated (2 -> 6, 3 -> 6) -/// - NotifyNewCommitIndex event is sent /// - Client receives success response #[tokio::test] async fn test_process_raft_request_immediate_execution() { @@ -170,7 +172,7 @@ async fn test_process_raft_request_immediate_execution() { }; use crate::maybe_clone_oneshot::MaybeCloneOneshot; let (tx, rx) = >::new(); - let (internal_event_tx, mut internal_event_rx) = mpsc::unbounded_channel(); + let (internal_event_tx, _internal_event_rx) = mpsc::unbounded_channel(); // When: Execute the write request (fire-and-forget replication in new arch) let result = test_context @@ -190,7 +192,7 @@ async fn test_process_raft_request_immediate_execution() { // Then: Verify operation succeeded assert!(result.is_ok(), "Operation should succeed"); - // Advance commit via single-voter flush path → fires NotifyNewCommitIndex, drains write. + // Advance commit via single-voter flush path, drains write. // MemFirst: set last_entry_id=5 before flush so commit uses last_entry_id(), not durable. test_context.last_entry_id.store(5, Ordering::Relaxed); test_context.state.cluster_metadata.single_voter = true; @@ -206,18 +208,6 @@ async fn test_process_raft_request_immediate_execution() { "Commit index should be updated to 5" ); - // Then: Verify NotifyNewCommitIndex was sent - assert!( - matches!( - internal_event_rx.try_recv(), - Ok(InternalEvent::NotifyNewCommitIndex(NewCommitData { - new_commit_index: 5, - .. - })) - ), - "Expected NotifyNewCommitIndex(5)" - ); - // Simulate SM apply to resolve pending_write_apply test_context .state @@ -345,7 +335,6 @@ async fn test_client_write_waits_for_sm_apply_not_just_commit() { /// - Both requests are handled immediately (not batched) /// - Commit index advances to 5 /// - Peer next_index values are updated (2 -> 6, 3 -> 6) -/// - NotifyNewCommitIndex event is sent /// - Both clients receive success responses #[tokio::test] async fn test_process_raft_request_two_consecutive_forced_sends() { @@ -398,7 +387,7 @@ async fn test_process_raft_request_two_consecutive_forced_sends() { use crate::maybe_clone_oneshot::MaybeCloneOneshot; let (tx1, rx1) = >::new(); let (tx2, rx2) = >::new(); - let (internal_event_tx, mut internal_event_rx) = mpsc::unbounded_channel(); + let (internal_event_tx, _internal_event_rx) = mpsc::unbounded_channel(); // When: Execute first write request (immediate execution) → pending_client_writes[5] let result1 = test_context @@ -448,15 +437,6 @@ async fn test_process_raft_request_two_consecutive_forced_sends() { "Commit index should be updated to 6" ); - // Then: Verify NotifyNewCommitIndex was sent - assert!( - matches!( - internal_event_rx.try_recv(), - Ok(InternalEvent::NotifyNewCommitIndex(_)) - ), - "Expected NotifyNewCommitIndex" - ); - // Simulate SM apply for both requests let results = vec![ ApplyResult { @@ -526,7 +506,7 @@ async fn test_process_raft_request_batching_enabled() { // When: Push request into buffer (batching — not immediately sent) test_context .state - .push_client_cmd(ClientCmd::Propose(request, tx), &test_context.raft_context); + .push_client_cmd(ClientCmd::propose(request, tx), &test_context.raft_context); // Then: replication handler NOT called (verified by mockall times(0) on drop) } @@ -595,7 +575,7 @@ async fn test_drain_single_write_no_delay() { }; let (tx, mut rx) = MaybeCloneOneshot::new(); - state.push_client_cmd(ClientCmd::Propose(req, tx), &ctx); + state.push_client_cmd(ClientCmd::propose(req, tx), &ctx); // Verify: Buffer has 1 request assert_eq!( @@ -698,7 +678,7 @@ async fn test_drain_multiple_writes_natural_batch() { }), }; let (tx, rx) = MaybeCloneOneshot::new(); - state.push_client_cmd(ClientCmd::Propose(req, tx), &ctx); + state.push_client_cmd(ClientCmd::propose(req, tx), &ctx); receivers.push(rx); } @@ -787,7 +767,7 @@ async fn test_drain_max_batch_size_limit() { }), }; let (tx, _rx) = MaybeCloneOneshot::new(); - state.push_client_cmd(ClientCmd::Propose(req, tx), &ctx); + state.push_client_cmd(ClientCmd::propose(req, tx), &ctx); } // Verify: All 10 requests in buffer @@ -880,7 +860,7 @@ async fn test_write_batch_single_replication() { }), }; let (tx, rx) = MaybeCloneOneshot::new(); - state.push_client_cmd(ClientCmd::Propose(req, tx), &ctx); + state.push_client_cmd(ClientCmd::propose(req, tx), &ctx); receivers.push(rx); } @@ -927,17 +907,15 @@ async fn test_write_batch_single_replication() { // // These tests enforce the invariant that governs when a client response is sent: // -// - Client write (propose_buffer path): wait_for_apply_event = true -// → Response MUST be deferred until StateMachine apply (ApplyCompleted event). -// → This guarantees the client only sees "success" after the entry is durable -// and applied. Changing this to false would break linearizability. +// - Client write (propose_buffer path): wait_for_apply_event = true → Response MUST be deferred +// until StateMachine apply (ApplyCompleted event). → This guarantees the client only sees +// "success" after the entry is durable and applied. Changing this to false would break +// linearizability. // -// - Noop / quorum-check / config-change (verify_internal_quorum path): -// wait_for_apply_event = false -// → Response is sent immediately after quorum is achieved. -// → These operations do not produce a state-machine result that clients -// need to observe, so waiting for apply would deadlock (no ApplyCompleted -// is ever sent for internal-only entries in this path). +// - Noop / quorum-check / config-change (verify_internal_quorum path): wait_for_apply_event = +// false → Response is sent immediately after quorum is achieved. → These operations do not +// produce a state-machine result that clients need to observe, so waiting for apply would +// deadlock (no ApplyCompleted is ever sent for internal-only entries in this path). // // If a future developer accidentally swaps these values, one of the following // tests will catch the regression: @@ -1008,13 +986,13 @@ async fn test_client_write_deferred_until_sm_apply() { }), }; let (tx, mut rx) = MaybeCloneOneshot::new(); - state.push_client_cmd(ClientCmd::Propose(req, tx), &ctx); + state.push_client_cmd(ClientCmd::propose(req, tx), &ctx); // Flush: fires prepare_batch_requests, write stored in pending_client_writes state.flush_cmd_buffers(&ctx, &internal_event_tx).await.unwrap(); - // Advance commit via single-voter path: moves write to pending_write_apply (wait_for_apply=true). - // MemFirst: set last_entry_id=5 before flush. + // Advance commit via single-voter path: moves write to pending_write_apply + // (wait_for_apply=true). MemFirst: set last_entry_id=5 before flush. last_entry_id.store(5, Ordering::Relaxed); state.cluster_metadata.single_voter = true; state.handle_log_flushed(5, &ctx, &internal_event_tx).await; @@ -1108,13 +1086,15 @@ async fn test_noop_quorum_check_responds_immediately_without_sm_apply() { // Advance commit via single-voter flush path → drain_pending_client_writes. // MemFirst: set last_entry_id=5 before flush. - // Since wait_for_apply_event=false, response is sent immediately upon commit (no ApplyCompleted needed) + // Since wait_for_apply_event=false, response is sent immediately upon commit (no ApplyCompleted + // needed) last_entry_id.store(5, Ordering::Relaxed); state.cluster_metadata.single_voter = true; state.handle_log_flushed(5, &ctx, &internal_event_tx).await; // CRITICAL: response must be available WITHOUT ApplyCompleted - // If wait_for_apply_event were true, it would still be in pending_write_apply (awaiting SM apply) + // If wait_for_apply_event were true, it would still be in pending_write_apply (awaiting SM + // apply) let response = rx .try_recv() .expect("Noop MUST respond immediately after commit, without ApplyCompleted"); @@ -1124,7 +1104,8 @@ async fn test_noop_quorum_check_responds_immediately_without_sm_apply() { ); } -/// Verify that a config-change quorum-check responds immediately after quorum, without waiting for SM apply. +/// Verify that a config-change quorum-check responds immediately after quorum, without waiting for +/// SM apply. /// /// # Purpose /// Enforces `wait_for_apply_event = false` for the verify_internal_quorum (config change) path. @@ -1170,7 +1151,8 @@ async fn test_config_change_quorum_check_responds_immediately_without_sm_apply() let (internal_event_tx, _internal_event_rx) = mpsc::unbounded_channel(); - // Config change payload — used for membership change quorum verification (wait_for_apply_event = false) + // Config change payload — used for membership change quorum verification (wait_for_apply_event + // = false) let change = Change::AddNode(AddNode { node_id: 4, address: "127.0.0.1:9004".to_string(), @@ -1194,7 +1176,8 @@ async fn test_config_change_quorum_check_responds_immediately_without_sm_apply() // Advance commit via single-voter flush path → drain_pending_client_writes. // MemFirst: set last_entry_id=5 before flush. - // Since wait_for_apply_event=false, response is sent immediately upon commit (no ApplyCompleted needed) + // Since wait_for_apply_event=false, response is sent immediately upon commit (no ApplyCompleted + // needed) last_entry_id.store(5, Ordering::Relaxed); state.cluster_metadata.single_voter = true; state.handle_log_flushed(5, &ctx, &internal_event_tx).await; @@ -1287,7 +1270,7 @@ async fn test_single_voter_client_write_completes_after_log_flushed() { "must be single_voter for this test" ); - let (internal_event_tx, mut internal_event_rx) = mpsc::unbounded_channel(); + let (internal_event_tx, _internal_event_rx) = mpsc::unbounded_channel(); // Step 1: Client sends write request let (req, mut client_rx) = write_request(); @@ -1326,15 +1309,6 @@ async fn test_single_voter_client_write_completes_after_log_flushed() { "commit must advance to last_entry_id after LogFlushed" ); - // Step 6: Verify NotifyNewCommitIndex was sent - assert!( - matches!( - internal_event_rx.try_recv().unwrap(), - InternalEvent::NotifyNewCommitIndex(_) - ), - "NotifyNewCommitIndex must fire after commit advances" - ); - // Step 7: Verify pending_client_writes was drained assert_eq!( state.pending_client_writes.len(), diff --git a/d-engine-core/src/raft_role/leader_state_test/commit_index_test.rs b/d-engine-core/src/raft_role/leader_state_test/commit_index_test.rs index 2c98710c..7a4fd0af 100644 --- a/d-engine-core/src/raft_role/leader_state_test/commit_index_test.rs +++ b/d-engine-core/src/raft_role/leader_state_test/commit_index_test.rs @@ -4,24 +4,34 @@ //! never `last_entry_id` (in-memory). Covers both the single-voter fast path and //! the multi-voter quorum path. Fixed by ticket #329. -use crate::event::InternalEvent; -use crate::maybe_clone_oneshot::MaybeCloneOneshot; -use crate::maybe_clone_oneshot::RaftOneshot; -use crate::raft_role::leader_state::LeaderState; -use crate::role_state::RaftRoleState; -use crate::test_utils::mock::{MockTypeConfig, mock_raft_context}; -use crate::{MockMembership, MockRaftLog, PeerUpdate, RaftRequestWithSignal}; +use std::collections::VecDeque; +use std::sync::Arc; +use std::sync::atomic::AtomicU64; +use std::sync::atomic::Ordering; + use bytes::Bytes; -use d_engine_proto::common::{EntryPayload, LogId}; -use d_engine_proto::common::{NodeRole::Follower, NodeStatus}; +use d_engine_proto::common::EntryPayload; +use d_engine_proto::common::LogId; +use d_engine_proto::common::NodeRole::Follower; +use d_engine_proto::common::NodeStatus; use d_engine_proto::server::cluster::NodeMeta; +use d_engine_proto::server::replication::AppendEntriesResponse; +use d_engine_proto::server::replication::SuccessResult; use d_engine_proto::server::replication::append_entries_response; -use d_engine_proto::server::replication::{AppendEntriesResponse, SuccessResult}; use rand::distr::SampleString; -use std::collections::VecDeque; -use std::sync::Arc; -use std::sync::atomic::{AtomicU64, Ordering}; -use tokio::sync::{mpsc, watch}; +use tokio::sync::mpsc; +use tokio::sync::watch; + +use crate::MockMembership; +use crate::MockRaftLog; +use crate::PeerUpdate; +use crate::RaftRequestWithSignal; +use crate::maybe_clone_oneshot::MaybeCloneOneshot; +use crate::maybe_clone_oneshot::RaftOneshot; +use crate::raft_role::leader_state::LeaderState; +use crate::role_state::RaftRoleState; +use crate::test_utils::mock::MockTypeConfig; +use crate::test_utils::mock::mock_raft_context; fn success_response( term: u64, @@ -137,7 +147,7 @@ async fn test_single_voter_commit_must_not_advance_before_durable() { ); assert!( internal_event_rx.try_recv().is_err(), - "CORRECT: NotifyNewCommitIndex must not fire before durable" + "CORRECT: no internal event expected before durable" ); } @@ -167,7 +177,7 @@ async fn test_single_voter_commit_advances_after_durable() { .returning(|_, _, _, _, _, _| Ok(crate::PrepareResult::default())); let (req, _rx) = write_request(); - let (internal_event_tx, mut internal_event_rx) = mpsc::unbounded_channel(); + let (internal_event_tx, _internal_event_rx) = mpsc::unbounded_channel(); state .process_batch(VecDeque::from(vec![req]), &internal_event_tx, &ctx) .await @@ -180,13 +190,6 @@ async fn test_single_voter_commit_advances_after_durable() { 1, "single-voter commit must advance to durable_index after flush" ); - assert!( - matches!( - internal_event_rx.try_recv(), - Ok(InternalEvent::NotifyNewCommitIndex(_)) - ), - "NotifyNewCommitIndex must fire after commit advances" - ); } // ── multi-voter: quorum uses durable_index via calculate_majority_matched_index ─ @@ -268,7 +271,7 @@ async fn test_multi_voter_commit_respects_quorum_result() { ); assert!( internal_event_rx.try_recv().is_err(), - "NotifyNewCommitIndex must not fire" + "no internal event expected while commit does not advance" ); } @@ -378,7 +381,7 @@ async fn test_leader_single_voter_commit_advances_on_log_flushed() { ) .await; - let (internal_event_tx, mut internal_event_rx) = mpsc::unbounded_channel(); + let (internal_event_tx, _internal_event_rx) = mpsc::unbounded_channel(); // Precondition: commit_index still at 0 (process_batch saw durable=0) assert_eq!(state.commit_index(), 0); @@ -392,15 +395,6 @@ async fn test_leader_single_voter_commit_advances_on_log_flushed() { 1, "single-voter commit must advance to durable index after LogFlushed" ); - - // Verify: NotifyNewCommitIndex event sent to state machine - assert!( - matches!( - internal_event_rx.try_recv().unwrap(), - InternalEvent::NotifyNewCommitIndex(_) - ), - "NotifyNewCommitIndex must fire after commit advances" - ); } /// Single-voter leader: LogFlushed with durable <= commit_index is a no-op. @@ -430,7 +424,7 @@ async fn test_leader_single_voter_log_flushed_noop_when_already_committed() { assert_eq!(state.commit_index(), 1, "commit_index must not change"); assert!( internal_event_rx.try_recv().is_err(), - "NotifyNewCommitIndex must NOT fire when durable <= commit_index" + "no internal event expected when durable <= commit_index" ); } @@ -455,7 +449,8 @@ async fn test_update_match_index_only_advances() { let mut membership = crate::MockMembership::::new(); membership.expect_is_single_node_cluster().returning(|| false); membership.expect_voters().returning(|| { - use d_engine_proto::common::{NodeRole::Follower, NodeStatus}; + use d_engine_proto::common::NodeRole::Follower; + use d_engine_proto::common::NodeStatus; use d_engine_proto::server::cluster::NodeMeta; vec![NodeMeta { id: 2, @@ -541,7 +536,8 @@ async fn setup_voter_with_learner( LeaderState, crate::raft_context::RaftContext, ) { - use d_engine_proto::common::{NodeRole, NodeStatus}; + use d_engine_proto::common::NodeRole; + use d_engine_proto::common::NodeStatus; use d_engine_proto::server::cluster::NodeMeta; let mut ctx = mock_raft_context(path, graceful_rx, None); @@ -631,7 +627,7 @@ async fn test_handle_log_flushed_excludes_learner_from_quorum() { state.update_match_index(3, 4).unwrap(); state.update_commit_index(4).unwrap(); - let (internal_event_tx, mut internal_event_rx) = mpsc::unbounded_channel(); + let (internal_event_tx, _internal_event_rx) = mpsc::unbounded_channel(); // Action: leader flushes entry 5 to disk state.handle_log_flushed(5, &ctx, &internal_event_tx).await; @@ -641,13 +637,6 @@ async fn test_handle_log_flushed_excludes_learner_from_quorum() { 5, "commit must advance when learner is correctly excluded from quorum" ); - assert!( - matches!( - internal_event_rx.try_recv().unwrap(), - InternalEvent::NotifyNewCommitIndex(_) - ), - "NotifyNewCommitIndex must fire after commit advances" - ); } /// handle_append_result must not include learner match_index in quorum peer list @@ -708,7 +697,7 @@ async fn test_handle_append_result_excludes_learner_from_quorum() { state.update_match_index(3, 4).unwrap(); state.update_commit_index(4).unwrap(); - let (internal_event_tx, mut internal_event_rx) = mpsc::unbounded_channel(); + let (internal_event_tx, _internal_event_rx) = mpsc::unbounded_channel(); // Action: voter(2) ACKs entry 5 — triggers quorum re-calculation state @@ -721,13 +710,6 @@ async fn test_handle_append_result_excludes_learner_from_quorum() { 5, "commit must advance when learner is correctly excluded from quorum after voter ACK" ); - assert!( - matches!( - internal_event_rx.try_recv().unwrap(), - InternalEvent::NotifyNewCommitIndex(_) - ), - "NotifyNewCommitIndex must fire after commit advances" - ); } // ── multi-voter: LogFlushed triggers quorum re-calculation ──────────────────── @@ -742,7 +724,8 @@ async fn setup_multi_voter( LeaderState, crate::raft_context::RaftContext, ) { - use d_engine_proto::common::{NodeRole::Follower, NodeStatus}; + use d_engine_proto::common::NodeRole::Follower; + use d_engine_proto::common::NodeStatus; use d_engine_proto::server::cluster::NodeMeta; let mut ctx = mock_raft_context(path, graceful_rx, None); @@ -791,7 +774,8 @@ async fn setup_multi_voter( /// **Scenario**: /// - 3-node cluster (leader + 2 peers) /// - Leader appends entry 1, peers ACK immediately (match_index=[1,1]) -/// - Leader's durable_index=0 → quorum blocked (need 2/3, have only 2/3 peers but leader not durable) +/// - Leader's durable_index=0 → quorum blocked (need 2/3, have only 2/3 peers but leader not +/// durable) /// - LogFlushed(1) arrives → leader durable=1 → quorum satisfied ([1,1,1]) → commit advances to 1 #[tokio::test] async fn test_multi_voter_commit_advances_when_log_flushed_enables_quorum() { @@ -814,7 +798,7 @@ async fn test_multi_voter_commit_advances_when_log_flushed_enables_quorum() { state.update_match_index(2, 1).unwrap(); state.update_match_index(3, 1).unwrap(); - let (internal_event_tx, mut internal_event_rx) = mpsc::unbounded_channel(); + let (internal_event_tx, _internal_event_rx) = mpsc::unbounded_channel(); // Precondition: commit still at 0 (durable_index was 0 during process_batch) assert_eq!(state.commit_index(), 0); @@ -828,15 +812,6 @@ async fn test_multi_voter_commit_advances_when_log_flushed_enables_quorum() { 1, "multi-voter commit must advance when LogFlushed enables quorum" ); - - // Verify: NotifyNewCommitIndex event sent to state machine - assert!( - matches!( - internal_event_rx.try_recv().unwrap(), - InternalEvent::NotifyNewCommitIndex(_) - ), - "NotifyNewCommitIndex must fire after commit advances" - ); } /// Multi-voter leader: LogFlushed does not advance commit when quorum still insufficient. @@ -890,9 +865,9 @@ async fn test_multi_voter_log_flushed_no_commit_when_quorum_insufficient() { "multi-voter commit must NOT advance when quorum insufficient despite LogFlushed" ); - // Verify: no commit event should be sent + // Verify: no event should be sent assert!( internal_event_rx.try_recv().is_err(), - "NotifyNewCommitIndex must NOT fire when quorum not satisfied" + "no internal event expected when quorum not satisfied" ); } diff --git a/d-engine-core/src/raft_role/leader_state_test/deadline_test.rs b/d-engine-core/src/raft_role/leader_state_test/deadline_test.rs index 18652029..5721df02 100644 --- a/d-engine-core/src/raft_role/leader_state_test/deadline_test.rs +++ b/d-engine-core/src/raft_role/leader_state_test/deadline_test.rs @@ -4,21 +4,26 @@ //! `pending_client_writes`, `pending_reads`, `pending_lease_reads`, and //! `pending_commit_actions`, and leaves live entries intact. +use std::collections::VecDeque; +use std::time::Duration; + +use tokio::sync::mpsc; +use tokio::sync::watch; +use tokio::time::Instant; +use tonic::Code; + use crate::MaybeCloneOneshot; use crate::client::ClientReadRequest; use crate::maybe_clone_oneshot::RaftOneshot; -use crate::raft_role::leader_state::{ - LeaderState, PendingLeaseRead, PendingReadBatch, PostCommitAction, PostCommitEntry, - WriteMetadata, -}; +use crate::raft_role::leader_state::LeaderState; +use crate::raft_role::leader_state::PendingLeaseRead; +use crate::raft_role::leader_state::PendingReadBatch; +use crate::raft_role::leader_state::PostCommitAction; +use crate::raft_role::leader_state::PostCommitEntry; +use crate::raft_role::leader_state::WriteMetadata; use crate::role_state::RaftRoleState; use crate::test_utils::MockBuilder; use crate::test_utils::mock::MockTypeConfig; -use std::collections::VecDeque; -use std::time::Duration; -use tokio::sync::{mpsc, watch}; -use tokio::time::Instant; -use tonic::Code; // ============================================================================ // Helpers @@ -298,17 +303,14 @@ async fn test_tick_drains_expired_node_join_commit_action() { let (mut state, ctx, internal_event_tx, raft_tx) = setup().await; let (tx, mut rx) = MaybeCloneOneshot::new(); - state.pending_commit_actions.insert( - 42, - PostCommitEntry { - deadline: expired(), - action: PostCommitAction::NodeJoin { - node_id: 2, - addr: "127.0.0.1:8080".to_string(), - sender: tx, - }, + state.pending_commit_actions.insert(42, PostCommitEntry { + deadline: expired(), + action: PostCommitAction::NodeJoin { + node_id: 2, + addr: "127.0.0.1:8080".to_string(), + sender: tx, }, - ); + }); state.tick(&internal_event_tx, &raft_tx, &ctx).await.unwrap(); @@ -330,17 +332,14 @@ async fn test_tick_keeps_non_expired_node_join_commit_action() { let (mut state, ctx, internal_event_tx, raft_tx) = setup().await; let (tx, mut rx) = MaybeCloneOneshot::new(); - state.pending_commit_actions.insert( - 42, - PostCommitEntry { - deadline: far_future(), - action: PostCommitAction::NodeJoin { - node_id: 2, - addr: "127.0.0.1:8080".to_string(), - sender: tx, - }, + state.pending_commit_actions.insert(42, PostCommitEntry { + deadline: far_future(), + action: PostCommitAction::NodeJoin { + node_id: 2, + addr: "127.0.0.1:8080".to_string(), + sender: tx, }, - ); + }); state.tick(&internal_event_tx, &raft_tx, &ctx).await.unwrap(); @@ -363,13 +362,10 @@ async fn test_tick_expired_leader_noop_triggers_step_down() { let (internal_event_tx, mut internal_event_rx) = mpsc::unbounded_channel(); let (raft_tx, _raft_rx) = mpsc::channel(1); - state.pending_commit_actions.insert( - 1, - PostCommitEntry { - deadline: expired(), - action: PostCommitAction::LeaderNoop { term: 1 }, - }, - ); + state.pending_commit_actions.insert(1, PostCommitEntry { + deadline: expired(), + action: PostCommitAction::LeaderNoop { term: 1 }, + }); state.tick(&internal_event_tx, &raft_tx, &ctx).await.unwrap(); diff --git a/d-engine-core/src/raft_role/leader_state_test/event_handling_test.rs b/d-engine-core/src/raft_role/leader_state_test/event_handling_test.rs index 331f685b..955a6e02 100644 --- a/d-engine-core/src/raft_role/leader_state_test/event_handling_test.rs +++ b/d-engine-core/src/raft_role/leader_state_test/event_handling_test.rs @@ -3,36 +3,43 @@ //! This module tests the `handle_inbound_event` method for various inbound events //! including vote requests, append entries, client operations, and more. +use std::sync::Arc; + +use bytes::Bytes; +use d_engine_proto::client::WriteCommand; +use d_engine_proto::server::cluster::ClusterConfChangeRequest; +use d_engine_proto::server::cluster::ClusterMembership; +use d_engine_proto::server::cluster::MetadataRequest; +use d_engine_proto::server::election::VoteRequest; +use d_engine_proto::server::election::VoteResponse; +use d_engine_proto::server::replication::AppendEntriesRequest; +use d_engine_proto::server::replication::AppendEntriesResponse; +use prost::Message; +use tokio::sync::mpsc; +use tokio::sync::watch; +use tonic::Code; +use tonic::Status; +use tracing_test::traced_test; + use crate::ClientCmd; use crate::Error; use crate::MockRaftLog; use crate::MockReplicationCore; use crate::MockStateMachineHandler; -use crate::client::{ClientReadRequest, ClientWriteRequest}; +use crate::client::ClientReadRequest; +use crate::client::ClientWriteRequest; use crate::config::RaftNodeConfig; use crate::config::ReadConsistencyPolicy; use crate::event::InboundEvent; -use crate::event::{InternalEvent, NewCommitData}; +use crate::event::InternalEvent; use crate::maybe_clone_oneshot::RaftOneshot; use crate::raft_role::leader_state::LeaderState; use crate::role_state::RaftRoleState; use crate::test_utils::create_test_chunk; +use crate::test_utils::mock::MockBuilder; +use crate::test_utils::mock::MockTypeConfig; use crate::test_utils::mock::mock_raft_context; -use crate::test_utils::mock::{MockBuilder, MockTypeConfig}; use crate::utils::convert::safe_kv_bytes; -use bytes::Bytes; -use d_engine_proto::client::WriteCommand; -use d_engine_proto::server::cluster::{ - ClusterConfChangeRequest, ClusterMembership, MetadataRequest, -}; -use d_engine_proto::server::election::{VoteRequest, VoteResponse}; -use d_engine_proto::server::replication::{AppendEntriesRequest, AppendEntriesResponse}; -use prost::Message; -use std::sync::Arc; -use tokio::sync::{mpsc, watch}; -use tonic::Code; -use tonic::Status; -use tracing_test::traced_test; // ============================================================================ // Test Helper Functions @@ -627,7 +634,7 @@ async fn test_handle_client_propose_success() { // Handle inbound event use crate::maybe_clone_oneshot::MaybeCloneOneshot; let (resp_tx, _resp_rx) = >::new(); - let cmd = ClientCmd::Propose( + let cmd = ClientCmd::propose( ClientWriteRequest { client_id: 1, command: Some(Bytes::from( @@ -726,7 +733,6 @@ async fn test_handle_client_read_linearizable_failure() { /// /// # Then /// - Leader's commit_index is updated to 3 -/// - NotifyNewCommitIndex event is sent /// - Client receives successful read response #[tokio::test] #[traced_test] @@ -775,7 +781,8 @@ async fn test_handle_client_read_linearizable_success() { let mut state = LeaderState::::new(1, context.node_config.clone()); // noop committed — leader is ready to serve linearizable reads. state.noop_log_id = Some(1); - // Pre-set commit_index = 1 so read_index=max(1,1)=1 > last_applied=0 → read deferred to pending_reads. + // Pre-set commit_index = 1 so read_index=max(1,1)=1 > last_applied=0 → read deferred to + // pending_reads. state.update_commit_index(1).expect("should succeed"); // Prepare request @@ -789,7 +796,7 @@ async fn test_handle_client_read_linearizable_success() { let (resp_tx, mut resp_rx) = >::new(); let cmd = ClientCmd::Read(client_read_request, resp_tx); - let (internal_event_tx, mut internal_event_rx) = mpsc::unbounded_channel(); + let (internal_event_tx, _internal_event_rx) = mpsc::unbounded_channel(); // Push to buffer state.push_client_cmd(cmd, &context); @@ -800,7 +807,7 @@ async fn test_handle_client_read_linearizable_success() { .await .expect("should succeed"); - // Advance commit to 3 via single-voter flush path → fires NotifyNewCommitIndex + // Advance commit to 3 via single-voter flush path state.cluster_metadata.single_voter = true; state .handle_log_flushed(expect_new_commit_index, &context, &internal_event_tx) @@ -809,17 +816,6 @@ async fn test_handle_client_read_linearizable_success() { // Validation criteria 1: Leader commit should be updated to: 3 assert_eq!(state.shared_state().commit_index, expect_new_commit_index); - // Validation criteria 3: event "InternalEvent::NotifyNewCommitIndex" should be received - let event = internal_event_rx.try_recv().unwrap(); - assert!(matches!( - event, - InternalEvent::NotifyNewCommitIndex(NewCommitData { - new_commit_index: _expect_new_commit_index, - role: _, - current_term: _ - }) - )); - // Simulate SM apply: ApplyCompleted fires pending_reads for read_index <= 3 state .handle_apply_completed( @@ -1021,7 +1017,8 @@ async fn test_drain_read_buffer_clears_pending_reads_on_stepdown() { let mut state = LeaderState::::new(1, context.node_config.clone()); // noop committed — leader is ready to serve linearizable reads. state.noop_log_id = Some(1); - // Pre-set commit_index = 3 so read_index=max(3,1)=3 > last_applied=0 → read deferred to pending_reads + // Pre-set commit_index = 3 so read_index=max(3,1)=3 > last_applied=0 → read deferred to + // pending_reads state.update_commit_index(3).expect("should succeed"); // Push a linearizable read and flush: quorum succeeds, last_applied(0) < read_index(3) @@ -1326,9 +1323,10 @@ async fn test_cluster_conf_leader_id_transitions_after_noop_commits() { /// - handle_inbound_event propagates the non-fatal error (swallowed at dispatch layer) #[tokio::test] async fn test_leader_join_cluster_dispatched_via_handle_inbound_event() { - use crate::maybe_clone_oneshot::MaybeCloneOneshot; use d_engine_proto::server::cluster::JoinRequest; + use crate::maybe_clone_oneshot::MaybeCloneOneshot; + let (_graceful_tx, graceful_rx) = watch::channel(()); let mut context = mock_raft_context("/tmp/test_leader_join_cluster_dispatch", graceful_rx, None); diff --git a/d-engine-core/src/raft_role/leader_state_test/fatal_error_test.rs b/d-engine-core/src/raft_role/leader_state_test/fatal_error_test.rs index 040dc93a..3ee22bb0 100644 --- a/d-engine-core/src/raft_role/leader_state_test/fatal_error_test.rs +++ b/d-engine-core/src/raft_role/leader_state_test/fatal_error_test.rs @@ -3,6 +3,14 @@ //! These tests verify that when Leader receives FatalError events, //! it properly notifies all pending client requests before shutdown. +use std::sync::Arc; + +use bytes::Bytes; +use tokio::sync::mpsc; +use tokio::sync::watch; +use tonic::Code; +use tracing_test::traced_test; + use super::super::LeaderState; use crate::ClientCmd; use crate::InboundEvent; @@ -15,11 +23,6 @@ use crate::maybe_clone_oneshot::MaybeCloneOneshot; use crate::maybe_clone_oneshot::RaftOneshot; use crate::role_state::RaftRoleState; use crate::test_utils::MockBuilder; -use bytes::Bytes; -use std::sync::Arc; -use tokio::sync::{mpsc, watch}; -use tonic::Code; -use tracing_test::traced_test; /// Test: Leader handles FatalError and propagates error /// diff --git a/d-engine-core/src/raft_role/leader_state_test/heartbeat_scenarios_test.rs b/d-engine-core/src/raft_role/leader_state_test/heartbeat_scenarios_test.rs index b7c22777..e05e8ff7 100644 --- a/d-engine-core/src/raft_role/leader_state_test/heartbeat_scenarios_test.rs +++ b/d-engine-core/src/raft_role/leader_state_test/heartbeat_scenarios_test.rs @@ -11,8 +11,8 @@ //! - A *true* heartbeat (entries built by `prepare_batch_requests` are absent, so //! `build_append_request` produces empty entries) must always be sendable. //! - A *substitute* heartbeat (entries were withheld because the window is full, so -//! `prepare_batch_requests` did not populate `peer_entries` for this peer) is also safe -//! to send as empty entries, from the follower's point of view. +//! `prepare_batch_requests` did not populate `peer_entries` for this peer) is also safe to send +//! as empty entries, from the follower's point of view. //! //! The distinction matters for observability and correctness reasoning: we must verify //! that even when a Replicate-state peer's window is at its limit (N un-acked requests), @@ -28,21 +28,25 @@ use std::sync::Arc; -use d_engine_proto::common::{NodeRole::Follower, NodeStatus}; +use d_engine_proto::common::NodeRole::Follower; +use d_engine_proto::common::NodeStatus; use d_engine_proto::server::cluster::NodeMeta; -use d_engine_proto::server::replication::{ - AppendEntriesResponse, ConflictResult, append_entries_response, -}; -use tokio::sync::{mpsc, watch}; +use d_engine_proto::server::replication::AppendEntriesResponse; +use d_engine_proto::server::replication::ConflictResult; +use d_engine_proto::server::replication::append_entries_response; +use tokio::sync::mpsc; +use tokio::sync::watch; use tracing_test::traced_test; use crate::MockMembership; use crate::MockRaftLog; use crate::event::InternalEvent; use crate::raft_role::leader_state::LeaderState; -use crate::raft_role::role_state::{PeerReplicationState, RaftRoleState}; +use crate::raft_role::role_state::PeerReplicationState; +use crate::raft_role::role_state::RaftRoleState; use crate::test_utils::MetricsCapture; -use crate::test_utils::mock::{MockTypeConfig, mock_raft_context}; +use crate::test_utils::mock::MockTypeConfig; +use crate::test_utils::mock::mock_raft_context; fn two_peer_membership() -> MockMembership { let peers = vec![ @@ -348,11 +352,11 @@ async fn test_release_on_empty_window_is_a_noop() { /// # Scenario /// - Peer 2 in `Replicate` state, 3 un-acknowledged requests ending at indexes 3, 6 and 9 /// (in_flight = 3). -/// - A stale-term response arrives (term 0 < leader_term 1) — must release nothing (in_flight -/// stays 3). -/// - A real, term-matching success response for index 3 arrives — must release exactly the -/// request ending at 3 (in_flight becomes 2), proving the earlier stale response didn't -/// already consume it. +/// - A stale-term response arrives (term 0 < leader_term 1) — must release nothing (in_flight stays +/// 3). +/// - A real, term-matching success response for index 3 arrives — must release exactly the request +/// ending at 3 (in_flight becomes 2), proving the earlier stale response didn't already consume +/// it. #[tokio::test] #[traced_test] async fn test_replicate_stale_response_releases_nothing_real_ack_releases_covered_requests() { @@ -444,10 +448,11 @@ async fn test_replicate_stale_response_releases_nothing_real_ack_releases_covere /// `ConflictResult` differs from `SuccessResult` here: both are term-matching resolutions (so /// neither is stale-ignored), but conflict additionally transitions the peer to `Probe` /// (`update_peer_index`'s conflict branch → `set_peer_replication_state(Probe)`), and that -/// transition unconditionally resets the whole in-flight window to 0 — not a partial release. This is intentional: a conflict means the leader's whole speculative pipeline -/// for this peer was built on a wrong assumption, so every other request still in flight is -/// also suspect, not just the one this response answers. Documented here so this isn't -/// mistaken for a bug symmetrical to the success case. +/// transition unconditionally resets the whole in-flight window to 0 — not a partial release. This +/// is intentional: a conflict means the leader's whole speculative pipeline for this peer was built +/// on a wrong assumption, so every other request still in flight is also suspect, not just the one +/// this response answers. Documented here so this isn't mistaken for a bug symmetrical to the +/// success case. #[tokio::test] #[traced_test] async fn test_replicate_conflict_response_resets_all_inflight_via_probe_transition() { @@ -830,10 +835,9 @@ async fn test_last_ack_timestamp_gauge_emitted_only_on_non_stale_ack() { .await .unwrap(); assert_eq!( - capture.gauge( - "core.raft.peer.last_ack_timestamp_seconds", - &[("peer_id", "2")] - ), + capture.gauge("core.raft.peer.last_ack_timestamp_seconds", &[( + "peer_id", "2" + )]), None, "stale-term response must not set the timestamp gauge" ); @@ -858,10 +862,9 @@ async fn test_last_ack_timestamp_gauge_emitted_only_on_non_stale_ack() { .unwrap(); let ts = capture - .gauge( - "core.raft.peer.last_ack_timestamp_seconds", - &[("peer_id", "2")], - ) + .gauge("core.raft.peer.last_ack_timestamp_seconds", &[( + "peer_id", "2", + )]) .expect("timestamp gauge must be set after a real ACK"); assert!( ts > 1_700_000_000.0, @@ -1114,7 +1117,8 @@ fn success_at(index: u64) -> Option { #[tokio::test] #[traced_test] async fn test_every_response_shape_has_a_defined_window_effect() { - use PeerReplicationState::{Probe, Replicate}; + use PeerReplicationState::Probe; + use PeerReplicationState::Replicate; let conflict = Some(append_entries_response::Result::Conflict(ConflictResult { conflict_term: None, conflict_index: Some(1), diff --git a/d-engine-core/src/raft_role/leader_state_test/lease_refresh_on_log_flushed_test.rs b/d-engine-core/src/raft_role/leader_state_test/lease_refresh_on_log_flushed_test.rs index 63729268..cfc7e6f4 100644 --- a/d-engine-core/src/raft_role/leader_state_test/lease_refresh_on_log_flushed_test.rs +++ b/d-engine-core/src/raft_role/leader_state_test/lease_refresh_on_log_flushed_test.rs @@ -10,6 +10,16 @@ //! After fix: log flush in single-voter confirms leadership (leader is entire quorum), //! so lease timestamp is updated to enable fast local reads. +use std::sync::Arc; +use std::sync::atomic::AtomicU64; +use std::sync::atomic::Ordering; +use std::time::Duration; + +use tokio::sync::mpsc; +use tokio::sync::watch; +use tokio::time::sleep; +use tracing_test::traced_test; + use crate::MockMembership; use crate::MockRaftLog; use crate::MockReplicationCore; @@ -18,13 +28,6 @@ use crate::raft_role::leader_state::LeaderState; use crate::raft_role::role_state::RaftRoleState; use crate::test_utils::MockBuilder; use crate::test_utils::mock::MockTypeConfig; -use std::sync::Arc; -use std::sync::atomic::{AtomicU64, Ordering}; -use std::time::Duration; -use tokio::sync::mpsc; -use tokio::sync::watch; -use tokio::time::sleep; -use tracing_test::traced_test; // ============================================================================ // Test Utilities diff --git a/d-engine-core/src/raft_role/leader_state_test/lease_send_ts_test.rs b/d-engine-core/src/raft_role/leader_state_test/lease_send_ts_test.rs index 7b2e815e..1adbbd14 100644 --- a/d-engine-core/src/raft_role/leader_state_test/lease_send_ts_test.rs +++ b/d-engine-core/src/raft_role/leader_state_test/lease_send_ts_test.rs @@ -14,14 +14,16 @@ //! `update_lease_timestamp`, so the lease expires at `send_ts + lease_duration_ms` //! rather than `ack_ts + lease_duration_ms`. +use std::time::Duration; + +use tokio::sync::watch; + use crate::RaftNodeConfig; use crate::raft_role::leader_state::LeaderState; use crate::raft_role::read_lease::now_ms; use crate::raft_role::role_state::RaftRoleState; use crate::test_utils::MockBuilder; use crate::test_utils::mock::MockTypeConfig; -use std::time::Duration; -use tokio::sync::watch; // ── helpers ────────────────────────────────────────────────────────────────── @@ -149,14 +151,14 @@ fn test_new_impl_preserves_election_timeout_invariant_old_impl_violates_it() { /// (Raft §8). /// /// Scenario: -/// - Node is elected; the previous leader had committed entry X, but this node's state -/// machine has not applied X yet (it only learns the commit point once its noop commits). +/// - Node is elected; the previous leader had committed entry X, but this node's state machine has +/// not applied X yet (it only learns the commit point once its noop commits). /// - A quorum ACK arrives before the noop is committed. /// /// Expected: -/// - The lease stays invalid for readers (`ReadActor` only looks at this lease). -/// Otherwise a LeaseRead right after the election can return a value older than a write -/// the previous leader already acknowledged to its client. +/// - The lease stays invalid for readers (`ReadActor` only looks at this lease). Otherwise a +/// LeaseRead right after the election can return a value older than a write the previous leader +/// already acknowledged to its client. /// - Once the noop is committed, the next quorum ACK makes the lease valid. #[tokio::test] async fn test_lease_not_published_to_readers_before_noop_committed() { @@ -192,13 +194,13 @@ async fn test_lease_not_published_to_readers_before_noop_committed() { /// /// Scenario: /// - Noop is at index 101 and already committed (a quorum ACK arrived). -/// - The state machine has only applied up to 100, so an entry the previous leader committed -/// (say X=1 at index 100) may or may not be applied yet; the entries before the noop are -/// not guaranteed to be visible to a local read. +/// - The state machine has only applied up to 100, so an entry the previous leader committed (say +/// X=1 at index 100) may or may not be applied yet; the entries before the noop are not +/// guaranteed to be visible to a local read. /// /// Expected: -/// - Quorum ACK with `last_applied = 100`: lease stays invalid for readers (they fall back -/// to the Raft loop, which waits for `last_applied >= read_index`). +/// - Quorum ACK with `last_applied = 100`: lease stays invalid for readers (they fall back to the +/// Raft loop, which waits for `last_applied >= read_index`). /// - Quorum ACK with `last_applied = 101`: lease becomes valid. #[tokio::test] async fn test_lease_not_published_to_readers_before_noop_applied() { @@ -229,7 +231,7 @@ async fn test_lease_not_published_to_readers_before_noop_applied() { // // Not required for safety (reads are guarded by the lease and by quorum confirmation), but an // isolated leader would otherwise keep queueing writes that can never commit and keep telling -// clients "I am the leader". openraft's approach: while the leader has no recent quorum +// clients "I am the leader". The approach: while the leader has no recent quorum // acknowledgement it rejects new writes (clients look for another leader); there is no role // change, no term change and no election, so brief network jitter costs a few rejected writes // instead of a leadership change. @@ -318,11 +320,12 @@ async fn offer_write( ctx: &crate::RaftContext, ) -> Option { use crate::client::ClientWriteRequest; - use crate::maybe_clone_oneshot::{MaybeCloneOneshot, RaftOneshot}; + use crate::maybe_clone_oneshot::MaybeCloneOneshot; + use crate::maybe_clone_oneshot::RaftOneshot; let (resp_tx, mut resp_rx) = >::new(); state.push_client_cmd( - crate::ClientCmd::Propose( + crate::ClientCmd::propose( ClientWriteRequest { client_id: 1, command: Some(bytes::Bytes::from_static(b"write")), @@ -416,9 +419,9 @@ async fn test_single_voter_leader_always_accepts_writes() { /// `election_timeout_min` has passed, and the first ACK reopens admission. /// /// Expected: -/// - Accepted right after the election, rejected after the window with no ACK, accepted again -/// after the first quorum ACK. The leader's own noop is an internal write and never goes -/// through admission. +/// - Accepted right after the election, rejected after the window with no ACK, accepted again after +/// the first quorum ACK. The leader's own noop is an internal write and never goes through +/// admission. #[tokio::test] async fn test_new_leader_accepts_writes_for_one_window_then_needs_a_quorum_ack() { let (mut state, ctx) = write_admission_leader().await; @@ -488,8 +491,8 @@ async fn test_leader_steps_down_when_no_quorum_ack_beyond_the_limit() { /// Test: silence past the read lease but inside the limit does not step the leader down. /// /// Expected: -/// - The limit is `multiple x election_timeout_max`, not the (much shorter) lease: jitter must -/// not cost a leadership change. +/// - The limit is `multiple x election_timeout_max`, not the (much shorter) lease: jitter must not +/// cost a leadership change. #[tokio::test] async fn test_leader_stays_when_silence_is_inside_the_limit() { let (mut state, ctx) = write_admission_leader().await; diff --git a/d-engine-core/src/raft_role/leader_state_test/membership_change_test.rs b/d-engine-core/src/raft_role/leader_state_test/membership_change_test.rs index f371633f..dffaa329 100644 --- a/d-engine-core/src/raft_role/leader_state_test/membership_change_test.rs +++ b/d-engine-core/src/raft_role/leader_state_test/membership_change_test.rs @@ -11,7 +11,16 @@ use std::sync::Arc; use std::time::Duration; use bytes::Bytes; -use tokio::sync::{mpsc, watch}; +use d_engine_proto::common::LogId; +use d_engine_proto::common::NodeRole::Follower; +use d_engine_proto::common::NodeRole::Learner; +use d_engine_proto::common::NodeStatus; +use d_engine_proto::server::cluster::ClusterMembership; +use d_engine_proto::server::cluster::JoinRequest; +use d_engine_proto::server::cluster::NodeMeta; +use d_engine_proto::server::storage::SnapshotMetadata; +use tokio::sync::mpsc; +use tokio::sync::watch; use tonic::Code; use tracing_test::traced_test; @@ -28,13 +37,6 @@ use crate::raft_role::role_state::RaftRoleState; use crate::state_machine_handler::StateMachineHandler; use crate::test_utils::mock::MockTypeConfig; use crate::test_utils::mock::mock_raft_context; -use d_engine_proto::common::{ - LogId, - NodeRole::{Follower, Learner}, - NodeStatus, -}; -use d_engine_proto::server::cluster::{ClusterMembership, JoinRequest, NodeMeta}; -use d_engine_proto::server::storage::SnapshotMetadata; // ============================================================================ // Test Helper Functions @@ -162,7 +164,7 @@ async fn test_join_cluster_precondition_checks() { /// Test composition to preserve original intent: /// - test_join_cluster_precondition_checks: validates preconditions /// - test_join_cluster_creates_correct_config_change: validates AddNode config (this test) -/// - Integration: When commit applies, membership.add_learner is called by CommitHandler +/// - Integration: When commit applies, membership.add_learner is called by `dispatch_committed` #[tokio::test] #[traced_test] async fn test_join_cluster_creates_correct_config_change() { @@ -649,16 +651,18 @@ mod stale_learner_tests { use std::sync::Arc; use std::time::Duration; - use tokio::sync::{mpsc, watch}; + use d_engine_proto::common::NodeStatus; + use tokio::sync::mpsc; + use tokio::sync::watch; use tokio::time::Instant; use crate::MockMembership; use crate::MockRaftLog; - use crate::raft_role::leader_state::{LeaderState, PendingPromotion}; + use crate::raft_role::leader_state::LeaderState; + use crate::raft_role::leader_state::PendingPromotion; use crate::test_utils::mock::MockTypeConfig; use crate::test_utils::mock::mock_raft_context; use crate::test_utils::node_config; - use d_engine_proto::common::NodeStatus; /// Setup helper for creating test leader state with pending promotions fn create_test_leader_state( @@ -699,12 +703,10 @@ mod stale_learner_tests { /// - (Actual replication stop happens after config applies via MembershipApplied event) #[tokio::test] async fn test_stale_learner_creates_removal_config() { - let (mut leader, mut membership) = create_test_leader_state( - "test_stale_learner_removal_config", - vec![ + let (mut leader, mut membership) = + create_test_leader_state("test_stale_learner_removal_config", vec![ (101, Duration::from_secs(31)), // stale: over 30s threshold - ], - ); + ]); membership.expect_get_node_status().returning(|_| Some(NodeStatus::Active)); // Mock: Capture payloads to validate config change @@ -781,18 +783,20 @@ mod check_learner_progress_tests { use std::sync::Arc; use std::time::Duration; - use tokio::sync::{mpsc, watch}; + use d_engine_proto::common::NodeStatus; + use tokio::sync::mpsc; + use tokio::sync::watch; use tokio::time::Instant; use crate::MockMembership; use crate::RaftContext; use crate::event::InternalEvent; - use crate::raft_role::leader_state::{LeaderState, PendingPromotion}; + use crate::raft_role::leader_state::LeaderState; + use crate::raft_role::leader_state::PendingPromotion; use crate::raft_role::role_state::RaftRoleState; use crate::test_utils::mock::MockTypeConfig; use crate::test_utils::mock::mock_raft_context; use crate::test_utils::node_config; - use d_engine_proto::common::NodeStatus; /// Helper: Create mock context with configurable threshold fn create_mock_context_with_threshold( @@ -1124,9 +1128,14 @@ mod pending_promotion_tests { use std::sync::Arc; use std::time::Duration; + use d_engine_proto::common::EntryPayload; + use d_engine_proto::common::NodeRole::Follower; + use d_engine_proto::common::NodeStatus; + use d_engine_proto::server::cluster::NodeMeta; use futures::future; use parking_lot::Mutex; - use tokio::sync::{mpsc, watch}; + use tokio::sync::mpsc; + use tokio::sync::watch; use tokio::time::Instant; use crate::MockMembership; @@ -1134,16 +1143,13 @@ mod pending_promotion_tests { use crate::MockReplicationCore; use crate::RaftContext; use crate::event::InternalEvent; - use crate::raft_role::leader_state::{ - LeaderState, PendingPromotion, calculate_safe_batch_size, - }; + use crate::raft_role::leader_state::LeaderState; + use crate::raft_role::leader_state::PendingPromotion; + use crate::raft_role::leader_state::calculate_safe_batch_size; use crate::raft_role::role_state::RaftRoleState; use crate::test_utils::mock::MockTypeConfig; use crate::test_utils::mock::mock_raft_context; use crate::test_utils::node_config; - use d_engine_proto::common::EntryPayload; - use d_engine_proto::common::{NodeRole::Follower, NodeStatus}; - use d_engine_proto::server::cluster::NodeMeta; // Test fixture: simple setup without pre-running verify_internal_quorum. // Quorum completion requires a running Raft loop; unit tests validate up to submission. @@ -1572,16 +1578,18 @@ mod pending_promotion_tests { mod zombie_purge_tests { use std::sync::Arc; + use d_engine_proto::common::EntryPayload; + use d_engine_proto::common::NodeStatus; use parking_lot::Mutex; - use tokio::sync::{mpsc, watch}; + use tokio::sync::mpsc; + use tokio::sync::watch; use crate::MockMembership; use crate::MockRaftLog; use crate::raft_role::leader_state::LeaderState; - use crate::test_utils::mock::{MockBuilder, MockTypeConfig}; + use crate::test_utils::mock::MockBuilder; + use crate::test_utils::mock::MockTypeConfig; use crate::test_utils::node_config; - use d_engine_proto::common::EntryPayload; - use d_engine_proto::common::NodeStatus; // ========================================================================= // handle_zombie_node — warn-only, no auto-removal @@ -1596,8 +1604,8 @@ mod zombie_purge_tests { /// - handle_zombie_node(5) is called /// /// # Then - /// - Returns Ok, no membership change is proposed - /// (auto-removal is intentionally not performed; operators decide) + /// - Returns Ok, no membership change is proposed (auto-removal is intentionally not performed; + /// operators decide) #[tokio::test] async fn test_handle_zombie_node_warns_without_removal() { let (_graceful_tx, graceful_rx) = watch::channel(()); @@ -1675,7 +1683,8 @@ mod zombie_purge_tests { /// - handle_zombie_node(5) is called (e.g. stale signal arrived after BatchRemove committed) /// /// # Then - /// - No config change is proposed (node is gone; re-proposing BatchRemove would flood the Raft log) + /// - No config change is proposed (node is gone; re-proposing BatchRemove would flood the Raft + /// log) #[tokio::test] async fn test_handle_zombie_node_absent_node_is_ignored() { let (_graceful_tx, graceful_rx) = watch::channel(()); diff --git a/d-engine-core/src/raft_role/leader_state_test/pending_commit_actions_test.rs b/d-engine-core/src/raft_role/leader_state_test/pending_commit_actions_test.rs index 7414e7af..32c8264f 100644 --- a/d-engine-core/src/raft_role/leader_state_test/pending_commit_actions_test.rs +++ b/d-engine-core/src/raft_role/leader_state_test/pending_commit_actions_test.rs @@ -5,19 +5,24 @@ //! - tick() step 5: deadline scan for expired `pending_commit_actions` //! //! Three key behaviors: -//! 1. `drain_commit_actions` fires `InternalEvent::NoopCommitted` when a `LeaderNoop` entry commits. -//! 2. `drain_commit_actions` delivers `NodeJoin` completion directly to the join response sender //! (no `InternalEvent` is emitted for NodeJoin). +//! 1. `drain_commit_actions` fires `InternalEvent::NoopCommitted` when a `LeaderNoop` entry +//! commits. +//! 2. `drain_commit_actions` delivers `NodeJoin` completion directly to the join response sender +//! //! (no `InternalEvent` is emitted for NodeJoin). //! 3. tick() sends `InternalEvent::BecomeFollower` for expired noop entries (leadership lost). use std::time::Duration; -use tokio::sync::{mpsc, watch}; +use tokio::sync::mpsc; +use tokio::sync::watch; use tokio::time::Instant; use crate::MaybeCloneOneshot; use crate::event::InternalEvent; use crate::maybe_clone_oneshot::RaftOneshot; -use crate::raft_role::leader_state::{LeaderState, PostCommitAction, PostCommitEntry}; +use crate::raft_role::leader_state::LeaderState; +use crate::raft_role::leader_state::PostCommitAction; +use crate::raft_role::leader_state::PostCommitEntry; use crate::raft_role::role_state::RaftRoleState; use crate::test_utils::MockBuilder; use crate::test_utils::mock::MockTypeConfig; diff --git a/d-engine-core/src/raft_role/leader_state_test/pending_lease_reads_test.rs b/d-engine-core/src/raft_role/leader_state_test/pending_lease_reads_test.rs index 75ae9a9c..e9e48d15 100644 --- a/d-engine-core/src/raft_role/leader_state_test/pending_lease_reads_test.rs +++ b/d-engine-core/src/raft_role/leader_state_test/pending_lease_reads_test.rs @@ -5,6 +5,22 @@ //! cleanup on step-down, batching of multiple requests, and //! Unavailable delivery when leadership is lost via higher-term step-down. +use std::sync::Arc; +use std::time::Duration; + +use d_engine_proto::common::LogId; +use d_engine_proto::common::NodeRole::Follower; +use d_engine_proto::common::NodeStatus; +use d_engine_proto::server::cluster::NodeMeta; +use d_engine_proto::server::replication::AppendEntriesResponse; +use d_engine_proto::server::replication::SuccessResult; +use d_engine_proto::server::replication::append_entries_response; +use tokio::sync::mpsc; +use tokio::sync::watch; +use tokio::time::Instant; +use tonic::Code; +use tracing_test::traced_test; + use crate::ClientCmd; use crate::MockMembership; use crate::MockRaftLog; @@ -14,24 +30,13 @@ use crate::RaftNodeConfig; use crate::ReadConsistencyPolicy; use crate::client::ClientReadRequest; use crate::convert::safe_kv_bytes; -use crate::maybe_clone_oneshot::{MaybeCloneOneshot, RaftOneshot}; -use crate::raft_role::leader_state::{LeaderState, PendingLeaseRead}; +use crate::maybe_clone_oneshot::MaybeCloneOneshot; +use crate::maybe_clone_oneshot::RaftOneshot; +use crate::raft_role::leader_state::LeaderState; +use crate::raft_role::leader_state::PendingLeaseRead; use crate::raft_role::role_state::RaftRoleState; use crate::test_utils::MockBuilder; use crate::test_utils::mock::MockTypeConfig; -use d_engine_proto::common::LogId; -use d_engine_proto::common::NodeRole::Follower; -use d_engine_proto::common::NodeStatus; -use d_engine_proto::server::cluster::NodeMeta; -use d_engine_proto::server::replication::AppendEntriesResponse; -use d_engine_proto::server::replication::SuccessResult; -use d_engine_proto::server::replication::append_entries_response; -use std::sync::Arc; -use std::time::Duration; -use tokio::sync::{mpsc, watch}; -use tokio::time::Instant; -use tonic::Code; -use tracing_test::traced_test; // ============================================================================ // Helpers @@ -467,8 +472,8 @@ async fn test_multiple_lease_reads_batched_on_single_quorum_ack() { /// # Flow /// 1. Expired lease → request queued in `pending_lease_reads` /// 2. Peer 2 responds with term=10 (higher than leader term=1) -/// 3. `handle_append_result` detects higher term → `BecomeFollower` event sent -/// → returns `Err(HigherTerm)` +/// 3. `handle_append_result` detects higher term → `BecomeFollower` event sent → returns +/// `Err(HigherTerm)` /// 4. Raft loop processes `BecomeFollower` → calls `drain_read_buffer()` /// 5. All pending lease reads receive `Unavailable` /// @@ -479,9 +484,10 @@ async fn test_multiple_lease_reads_batched_on_single_quorum_ack() { #[tokio::test] #[traced_test] async fn test_pending_lease_reads_drained_on_higher_term_stepdown() { - use crate::event::InternalEvent; use d_engine_proto::server::replication::AppendEntriesResponse; + use crate::event::InternalEvent; + let (mut state, mut context, internal_event_tx, mut internal_event_rx) = setup_multi_voter_expired_lease( "/tmp/test_pending_lease_reads_drained_on_higher_term_stepdown", diff --git a/d-engine-core/src/raft_role/leader_state_test/pending_reads_test.rs b/d-engine-core/src/raft_role/leader_state_test/pending_reads_test.rs index d0075b78..9192122d 100644 --- a/d-engine-core/src/raft_role/leader_state_test/pending_reads_test.rs +++ b/d-engine-core/src/raft_role/leader_state_test/pending_reads_test.rs @@ -26,7 +26,6 @@ use std::collections::VecDeque; use std::sync::Arc; use std::time::Duration; -use crate::client::ClientReadRequest; use d_engine_proto::common::LogId; use d_engine_proto::common::NodeRole::Follower; use d_engine_proto::common::NodeStatus; @@ -34,7 +33,8 @@ use d_engine_proto::server::cluster::NodeMeta; use d_engine_proto::server::replication::AppendEntriesResponse; use d_engine_proto::server::replication::SuccessResult; use d_engine_proto::server::replication::append_entries_response; -use tokio::sync::{mpsc, watch}; +use tokio::sync::mpsc; +use tokio::sync::watch; use tokio::time::Instant; use tonic::Code; use tracing_test::traced_test; @@ -47,10 +47,13 @@ use crate::MockStateMachine; use crate::PeerUpdate; use crate::RaftNodeConfig; use crate::ReadConsistencyPolicy; +use crate::client::ClientReadRequest; use crate::convert::safe_kv_bytes; use crate::event::InboundEvent; -use crate::maybe_clone_oneshot::{MaybeCloneOneshot, RaftOneshot}; -use crate::raft_role::leader_state::{LeaderState, PendingReadBatch}; +use crate::maybe_clone_oneshot::MaybeCloneOneshot; +use crate::maybe_clone_oneshot::RaftOneshot; +use crate::raft_role::leader_state::LeaderState; +use crate::raft_role::leader_state::PendingReadBatch; use crate::raft_role::role_state::RaftRoleState; use crate::test_utils::MockBuilder; use crate::test_utils::mock::MockTypeConfig; @@ -365,13 +368,10 @@ async fn test_pending_reads_slow_path_drained_by_apply_completed() { }, resp_tx, )); - state.pending_reads.insert( - 5, - PendingReadBatch { - deadline: Instant::now() + Duration::from_secs(5), - requests, - }, - ); + state.pending_reads.insert(5, PendingReadBatch { + deadline: Instant::now() + Duration::from_secs(5), + requests, + }); assert_eq!(state.pending_reads.len(), 1, "precondition: read is queued"); let (internal_event_tx, _internal_event_rx) = mpsc::unbounded_channel(); @@ -506,13 +506,10 @@ async fn test_pending_reads_cleared_on_stepdown() { }, resp_tx, )); - state.pending_reads.insert( - read_index, - PendingReadBatch { - deadline: Instant::now() + Duration::from_secs(30), - requests, - }, - ); + state.pending_reads.insert(read_index, PendingReadBatch { + deadline: Instant::now() + Duration::from_secs(30), + requests, + }); senders.push(resp_rx); } assert_eq!(state.pending_reads.len(), 2, "precondition: 2 reads queued"); @@ -701,8 +698,8 @@ async fn test_linearizable_read_queued_when_sm_behind_despite_valid_lease() { /// /// ## Setup /// - 3-voter cluster, lease never refreshed (default: expired / None) -/// - `last_applied=0 >= read_index=0` — the fast-path condition for `last_applied` -/// is satisfied, but `is_lease_valid()` is false +/// - `last_applied=0 >= read_index=0` — the fast-path condition for `last_applied` is satisfied, +/// but `is_lease_valid()` is false /// /// ## Expected behavior /// The lease fast path is NOT taken. The read is queued in `pending_reads` and diff --git a/d-engine-core/src/raft_role/leader_state_test/probe_backpressure_test.rs b/d-engine-core/src/raft_role/leader_state_test/probe_backpressure_test.rs index 5b61051b..e30085d6 100644 --- a/d-engine-core/src/raft_role/leader_state_test/probe_backpressure_test.rs +++ b/d-engine-core/src/raft_role/leader_state_test/probe_backpressure_test.rs @@ -1,7 +1,6 @@ //! Test for the missing per-peer in-flight gate during `PeerReplicationState::Probe`. //! -//! Background (see `446-expert-q-probe-backpressure-fix-8020.md` in the product-design repo): -//! every reference Raft implementation limits a `Probe`-state +//! Background: a Raft leader must limit a `Probe`-state //! peer to at most one outstanding (unacknowledged) `AppendEntries` request. d-engine's //! `PeerReplicationState::Probe`/`Replicate` only controls whether `next_index` is optimistically //! advanced before sending — it never checks whether the peer already has a request in flight. @@ -17,26 +16,35 @@ use std::collections::VecDeque; use std::sync::Arc; +use std::sync::atomic::AtomicBool; use bytes::Bytes; -use d_engine_proto::common::{Entry, EntryPayload, NodeRole::Follower, NodeStatus}; +use d_engine_proto::common::Entry; +use d_engine_proto::common::EntryPayload; +use d_engine_proto::common::NodeRole::Follower; +use d_engine_proto::common::NodeStatus; use d_engine_proto::server::cluster::NodeMeta; -use d_engine_proto::server::replication::{ - AppendEntriesRequest, AppendEntriesResponse, ConflictResult, append_entries_response, -}; -use tokio::sync::{mpsc, watch}; +use d_engine_proto::server::replication::AppendEntriesRequest; +use d_engine_proto::server::replication::AppendEntriesResponse; +use d_engine_proto::server::replication::ConflictResult; +use d_engine_proto::server::replication::append_entries_response; +use tokio::sync::mpsc; +use tokio::sync::watch; use tracing_test::traced_test; use crate::MockMembership; use crate::MockRaftLog; use crate::RaftRequestWithSignal; use crate::event::InternalEvent; -use crate::maybe_clone_oneshot::{MaybeCloneOneshot, RaftOneshot}; +use crate::maybe_clone_oneshot::MaybeCloneOneshot; +use crate::maybe_clone_oneshot::RaftOneshot; use crate::network::PeerUpdate; use crate::raft_role::leader_state::LeaderState; -use crate::raft_role::role_state::{PeerReplicationState, RaftRoleState}; +use crate::raft_role::role_state::PeerReplicationState; +use crate::raft_role::role_state::RaftRoleState; use crate::test_utils::MetricsCapture; -use crate::test_utils::mock::{MockTypeConfig, mock_raft_context}; +use crate::test_utils::mock::MockTypeConfig; +use crate::test_utils::mock::mock_raft_context; /// Two-voter membership (peers 2 & 3) so the cluster is multi-voter — a single-voter leader /// short-circuits Phase 5 entirely (no peer work to gate), which would make this test vacuous. @@ -98,17 +106,16 @@ fn one_entry_batch() -> VecDeque { /// - Batch 1 is dispatched to peer 2: its first `Probe`, correctly limited to one outstanding /// request. /// - The follower answers that probe with a CONFLICT (reject), not a success. A reject is not -/// evidence the peer is caught up, so `update_peer_index`'s conflict branch retreats -/// `next_index` to the conflict hint and leaves peer 2 in `Probe`. +/// evidence the peer is caught up, so `update_peer_index`'s conflict branch retreats `next_index` +/// to the conflict hint and leaves peer 2 in `Probe`. /// - Batch 2 is processed afterwards. The response to batch 1 has *arrived*, so peer 2 no longer -/// has an outstanding request and the gate must release: the corrected probe must be -/// dispatched. +/// has an outstanding request and the gate must release: the corrected probe must be dispatched. /// /// # Expected -/// `Probe` means "at most one unacknowledged `AppendEntries` at a time", not "at most one ever". A response must -/// re-arm the gate, never latch it shut: a latched `Probe` peer receives nothing further — not -/// even heartbeats, since they share this dispatch path — while the frozen follower times out -/// into candidacy and the leader cannot reach quorum. +/// `Probe` means "at most one unacknowledged `AppendEntries` at a time", not "at most one ever". A +/// response must re-arm the gate, never latch it shut: a latched `Probe` peer receives nothing +/// further — not even heartbeats, since they share this dispatch path — while the frozen follower +/// times out into candidacy and the leader cannot reach quorum. /// /// # Current behavior (why this test is RED) /// The gate is set on dispatch and only cleared by `handle_peer_stream_error` (a bidi stream @@ -158,14 +165,12 @@ async fn test_probe_peer_dispatches_next_probe_after_reject() { state.init_cluster_metadata(&ctx.membership).await.unwrap(); let (task_tx, mut task_rx) = mpsc::unbounded_channel(); - state.replication_workers.insert( - 2, - super::ReplicationWorkerHandle { - task_tx, - snapshot_failure_count: 0, - snapshot_next_retry_at: None, - }, - ); + state.replication_workers.insert(2, super::ReplicationWorkerHandle { + task_tx, + snapshot_failure_count: 0, + snapshot_next_retry_at: None, + stream_open: Arc::new(AtomicBool::new(true)), + }); let (internal_event_tx, _internal_event_rx) = mpsc::unbounded_channel::(); @@ -268,14 +273,12 @@ async fn test_probe_peer_with_pending_ack_receives_no_second_dispatch() { // because this test module nests under it — same access pattern `inject_dead_worker_for_test` // already relies on for the sibling `worker_lifecycle_test.rs` file. let (task_tx, mut task_rx) = mpsc::unbounded_channel(); - state.replication_workers.insert( - 2, - super::ReplicationWorkerHandle { - task_tx, - snapshot_failure_count: 0, - snapshot_next_retry_at: None, - }, - ); + state.replication_workers.insert(2, super::ReplicationWorkerHandle { + task_tx, + snapshot_failure_count: 0, + snapshot_next_retry_at: None, + stream_open: Arc::new(AtomicBool::new(true)), + }); let (internal_event_tx, _internal_event_rx) = mpsc::unbounded_channel::(); @@ -292,8 +295,7 @@ async fn test_probe_peer_with_pending_ack_receives_no_second_dispatch() { assert!( task_rx.try_recv().is_err(), "peer 2 already has an unacknowledged request in flight — a Probe-state peer must not \ - receive a second AppendEntries until the first is acked. See \ - 446-expert-q-probe-backpressure-fix-8020.md for the etcd/raft and openraft references." + receive a second AppendEntries until the first is acked." ); } @@ -321,14 +323,12 @@ async fn setup_gate_harness( state.init_cluster_metadata(&ctx.membership).await.unwrap(); let (task_tx, task_rx) = mpsc::unbounded_channel(); - state.replication_workers.insert( - 2, - super::ReplicationWorkerHandle { - task_tx, - snapshot_failure_count: 0, - snapshot_next_retry_at: None, - }, - ); + state.replication_workers.insert(2, super::ReplicationWorkerHandle { + task_tx, + snapshot_failure_count: 0, + snapshot_next_retry_at: None, + stream_open: Arc::new(AtomicBool::new(true)), + }); let (internal_event_tx, _internal_event_rx) = mpsc::unbounded_channel::(); (ctx, state, task_rx, internal_event_tx) @@ -457,8 +457,8 @@ async fn test_heartbeat_bypasses_gate_while_probe_in_flight() { } /// A stale-term response belongs to an older request, not the outstanding probe, so it must NOT -/// reopen the gate. Liveness is instead recovered by the heartbeat backstop (see the bypass test), -/// mirroring etcd's `MaybeUpdate(n <= Match)` early-return-without-resume. +/// reopen the gate. Liveness is instead recovered by the heartbeat backstop (see the bypass test): +/// the response is ignored without resuming the probe. #[tokio::test] #[traced_test] async fn test_stale_term_response_keeps_gate_latched() { @@ -508,12 +508,12 @@ async fn test_stale_term_response_keeps_gate_latched() { /// /// # Scenario /// - Batch 1 dispatched to peer 2 — gate latches (Probe, window=1). -/// - A stale-term response arrives (term 0 < leader_term 1) — ignored, gate stays latched -/// (same setup as the sibling test above). +/// - A stale-term response arrives (term 0 < leader_term 1) — ignored, gate stays latched (same +/// setup as the sibling test above). /// - The *real* response for that outstanding probe arrives (term 1, a reject) — this is the /// response Phase 5 actually sent the probe to elicit, so it must release the slot. -/// - Batch 2 must now dispatch — proves the slot was released by the real response, not -/// permanently stuck after the stale one was ignored. +/// - Batch 2 must now dispatch — proves the slot was released by the real response, not permanently +/// stuck after the stale one was ignored. #[tokio::test] #[traced_test] async fn test_real_response_after_stale_still_releases_the_gate() { @@ -721,14 +721,12 @@ async fn setup_two_worker_harness( ) { let (ctx, mut state, task_rx2, internal_event_tx) = setup_gate_harness(path).await; let (task_tx3, task_rx3) = mpsc::unbounded_channel(); - state.replication_workers.insert( - 3, - super::ReplicationWorkerHandle { - task_tx: task_tx3, - snapshot_failure_count: 0, - snapshot_next_retry_at: None, - }, - ); + state.replication_workers.insert(3, super::ReplicationWorkerHandle { + task_tx: task_tx3, + snapshot_failure_count: 0, + snapshot_next_retry_at: None, + stream_open: Arc::new(AtomicBool::new(true)), + }); (ctx, state, task_rx2, task_rx3, internal_event_tx) } diff --git a/d-engine-core/src/raft_role/leader_state_test/replicate_unbounded_dispatch_test.rs b/d-engine-core/src/raft_role/leader_state_test/replicate_unbounded_dispatch_test.rs index 1555b6f5..099d1ba7 100644 --- a/d-engine-core/src/raft_role/leader_state_test/replicate_unbounded_dispatch_test.rs +++ b/d-engine-core/src/raft_role/leader_state_test/replicate_unbounded_dispatch_test.rs @@ -3,8 +3,7 @@ //! `next_index = effective_next_index + entries.len()` to whatever `prepare_batch_requests` hands //! it, on every batch, unconditionally. //! -//! Background (full root-cause chain: `replication-worker-backpressure-deadlock-expert-q.md` and -//! `446-replication-stall-case-study-2026-09-28.md` in the product-design repo): a real deadlock +//! Background: a real deadlock //! reproduced on embedded-bench (100K writes, 3-node localhost) traced to `ReplicationHandler` //! (`retrieve_to_be_synced_logs_for_peers` / `build_append_request`) repeatedly re-offering the //! same un-acked range — 8955 rebuilds for 28 real acks in one capture — which starved the peer's @@ -13,7 +12,7 @@ //! //! **Important scoping correction**: the fix belongs in `ReplicationHandler`, upstream of this //! layer — it must stop *offering* a duplicate un-acked range at all (checked before touching the -//! log, mirroring `raft-rs`'s `Progress::is_paused()` ahead of `maybe_send_append`'s entry fetch). +//! log: a per-peer pause check ahead of the entry fetch). //! Once that lands, `build_append_request` naturally falls back to an empty (heartbeat) request //! for a peer with nothing new to offer (`entries_per_peer.remove(..).unwrap_or_default()`), and //! Phase 5's existing formula correctly no-ops on that (`entries.len() == 0` advances nothing) — @@ -26,22 +25,30 @@ use std::collections::VecDeque; use std::sync::Arc; +use std::sync::atomic::AtomicBool; use bytes::Bytes; -use d_engine_proto::common::{Entry, EntryPayload, NodeRole::Follower, NodeStatus}; +use d_engine_proto::common::Entry; +use d_engine_proto::common::EntryPayload; +use d_engine_proto::common::NodeRole::Follower; +use d_engine_proto::common::NodeStatus; use d_engine_proto::server::cluster::NodeMeta; use d_engine_proto::server::replication::AppendEntriesRequest; -use tokio::sync::{mpsc, watch}; +use tokio::sync::mpsc; +use tokio::sync::watch; use tracing_test::traced_test; use crate::MockMembership; use crate::MockRaftLog; use crate::RaftRequestWithSignal; use crate::event::InternalEvent; -use crate::maybe_clone_oneshot::{MaybeCloneOneshot, RaftOneshot}; +use crate::maybe_clone_oneshot::MaybeCloneOneshot; +use crate::maybe_clone_oneshot::RaftOneshot; use crate::raft_role::leader_state::LeaderState; -use crate::raft_role::role_state::{PeerReplicationState, RaftRoleState}; -use crate::test_utils::mock::{MockTypeConfig, mock_raft_context}; +use crate::raft_role::role_state::PeerReplicationState; +use crate::raft_role::role_state::RaftRoleState; +use crate::test_utils::mock::MockTypeConfig; +use crate::test_utils::mock::mock_raft_context; /// Two-voter membership (peers 2 & 3) — same shape as `probe_backpressure_test.rs`'s, duplicated /// here rather than shared: these helpers are private to their own test module. @@ -120,14 +127,12 @@ async fn test_replicate_peer_dispatch_and_next_index_are_unbounded_without_any_a // Peer 2's worker channel held directly by the test — dispatch is observable synchronously, // no real transport/worker task involved (same pattern as probe_backpressure_test.rs). let (task_tx, mut task_rx) = mpsc::unbounded_channel(); - state.replication_workers.insert( - 2, - super::ReplicationWorkerHandle { - task_tx, - snapshot_failure_count: 0, - snapshot_next_retry_at: None, - }, - ); + state.replication_workers.insert(2, super::ReplicationWorkerHandle { + task_tx, + snapshot_failure_count: 0, + stream_open: Arc::new(AtomicBool::new(true)), + snapshot_next_retry_at: None, + }); // The steady state a peer settles into after its first real ACK — where this bug lives. state.set_peer_replication_state(2, PeerReplicationState::Replicate); diff --git a/d-engine-core/src/raft_role/leader_state_test/replication_test.rs b/d-engine-core/src/raft_role/leader_state_test/replication_test.rs index 39b825e9..c74f4b0c 100644 --- a/d-engine-core/src/raft_role/leader_state_test/replication_test.rs +++ b/d-engine-core/src/raft_role/leader_state_test/replication_test.rs @@ -4,10 +4,34 @@ //! `execute_request_immediately` methods which handle log replication, //! commit index calculation, and client response delivery. +use std::collections::VecDeque; +use std::sync::Arc; +use std::sync::atomic::AtomicU64; +use std::sync::atomic::Ordering; + +use bytes::Bytes; +use d_engine_proto::common::EntryPayload; +use d_engine_proto::common::LogId; +use d_engine_proto::common::NodeRole::Follower; +use d_engine_proto::common::NodeStatus; +use d_engine_proto::server::cluster::ClusterMembership; +use d_engine_proto::server::cluster::NodeMeta; +use d_engine_proto::server::replication::AppendEntriesResponse; +use d_engine_proto::server::replication::ConflictResult; +use d_engine_proto::server::replication::SuccessResult; +use d_engine_proto::server::replication::append_entries_response; +use rand::distr::SampleString; +use tokio::sync::mpsc; +use tokio::sync::watch; +use tonic::Status; +use tracing_test::traced_test; + use crate::Error; +use crate::MaybeCloneOneshot; use crate::MockMembership; use crate::MockRaftLog; use crate::MockReplicationCore; +use crate::PeerUpdate; use crate::RaftContext; use crate::RaftRequestWithSignal; use crate::client::ClientResponse; @@ -19,21 +43,6 @@ use crate::role_state::RaftRoleState; use crate::test_utils::mock::MockBuilder; use crate::test_utils::mock::MockTypeConfig; use crate::test_utils::mock::mock_raft_context; -use crate::{MaybeCloneOneshot, PeerUpdate}; -use bytes::Bytes; -use d_engine_proto::common::{EntryPayload, LogId, NodeRole::Follower, NodeStatus}; -use d_engine_proto::server::cluster::{ClusterMembership, NodeMeta}; -use d_engine_proto::server::replication::ConflictResult; -use d_engine_proto::server::replication::append_entries_response; -use d_engine_proto::server::replication::{AppendEntriesResponse, SuccessResult}; -use rand::distr::SampleString; -use std::collections::VecDeque; -use std::sync::Arc; -use std::sync::atomic::{AtomicU64, Ordering}; -use tokio::sync::mpsc; -use tokio::sync::watch; -use tonic::Status; -use tracing_test::traced_test; // ============================================================================ // Test Helper Structures and Functions @@ -159,7 +168,6 @@ fn success_response( /// /// # Then /// - Commit index advances to 6 -/// - NotifyNewCommitIndex event is sent /// - Both clients receive success responses #[tokio::test] #[traced_test] @@ -193,7 +201,7 @@ async fn test_process_batch_quorum_achieved() { let batch = VecDeque::from(vec![mock_request(tx1), mock_request(tx2)]); let receivers = vec![rx1, rx2]; - let (internal_event_tx, mut internal_event_rx) = mpsc::unbounded_channel(); + let (internal_event_tx, _internal_event_rx) = mpsc::unbounded_channel(); let result = context .state .process_batch(batch, &internal_event_tx, &context.raft_context) @@ -211,10 +219,6 @@ async fn test_process_batch_quorum_achieved() { .await; assert_eq!(context.state.shared_state().commit_index, 6); - assert!(matches!( - internal_event_rx.try_recv(), - Ok(InternalEvent::NotifyNewCommitIndex(_)) - )); // Verify client responses for mut rx in receivers { @@ -700,7 +704,7 @@ async fn test_single_node_cluster_commit_index() { let (tx1, rx1) = >::new(); let batch = VecDeque::from(vec![mock_request(tx1)]); - let (internal_event_tx, mut internal_event_rx) = mpsc::unbounded_channel(); + let (internal_event_tx, _internal_event_rx) = mpsc::unbounded_channel(); let result = context .state @@ -721,10 +725,6 @@ async fn test_single_node_cluster_commit_index() { 7, "Single-node: commit_index should advance to last_entry_id after LogFlushed" ); - assert!(matches!( - internal_event_rx.try_recv(), - Ok(InternalEvent::NotifyNewCommitIndex(_)) - )); let mut rx = rx1; let response = rx.recv().await.unwrap().unwrap(); @@ -787,7 +787,7 @@ async fn test_multi_node_cluster_empty_peer_updates_commit_index() { ); assert!( internal_event_rx.try_recv().is_err(), - "No NotifyNewCommitIndex without commit advancement" + "No internal event expected without commit advancement" ); // Write is pending — awaiting peer ACKs via handle_append_result @@ -855,7 +855,7 @@ async fn test_multi_node_cluster_with_peer_updates_commit_index() { let (tx1, rx1) = >::new(); let (tx2, rx2) = >::new(); let batch = VecDeque::from(vec![mock_request(tx1), mock_request(tx2)]); - let (internal_event_tx, mut internal_event_rx) = mpsc::unbounded_channel(); + let (internal_event_tx, _internal_event_rx) = mpsc::unbounded_channel(); let result = context .state @@ -876,10 +876,6 @@ async fn test_multi_node_cluster_with_peer_updates_commit_index() { .unwrap(); assert_eq!(context.state.shared_state().commit_index, 6); - assert!(matches!( - internal_event_rx.try_recv(), - Ok(InternalEvent::NotifyNewCommitIndex(_)) - )); let mut rx = rx1; let response = rx.recv().await.unwrap().unwrap(); @@ -913,7 +909,6 @@ async fn test_multi_node_cluster_with_peer_updates_commit_index() { /// /// # Then /// - Commit index advances to 5 -/// - NotifyNewCommitIndex event is sent /// - Response channel receives write_success #[tokio::test] #[traced_test] @@ -960,7 +955,7 @@ async fn test_verify_internal_quorum_success() { wait_for_apply_event: false, }; - let (internal_event_tx, mut internal_event_rx) = mpsc::unbounded_channel(); + let (internal_event_tx, _internal_event_rx) = mpsc::unbounded_channel(); state .execute_request_immediately(req, &raft_context, &internal_event_tx) .await @@ -971,10 +966,6 @@ async fn test_verify_internal_quorum_success() { state.handle_log_flushed(5, &raft_context, &internal_event_tx).await; assert_eq!(state.shared_state().commit_index, 5); - assert!(matches!( - internal_event_rx.try_recv(), - Ok(InternalEvent::NotifyNewCommitIndex(_)) - )); // Response received: write_success let response = resp_rx.recv().await.unwrap().unwrap(); @@ -1477,7 +1468,7 @@ async fn test_execute_and_process_raft_rpc_multi_node_empty_peer_updates() { ); assert!( internal_event_rx.try_recv().is_err(), - "No NotifyNewCommitIndex without commit advancement" + "No internal event expected without commit advancement" ); // Write request is pending — awaiting peer ACKs @@ -1562,7 +1553,6 @@ async fn test_merge_batch_to_write_metadata_empty_payload_with_senders() { /// /// # Then /// - Commit advances from 0 to 3 (quorum: leader + peer = 2/2) -/// - NotifyNewCommitIndex event sent #[tokio::test] #[traced_test] async fn test_handle_append_result_two_node_quorum_achieved() { @@ -1617,7 +1607,7 @@ async fn test_handle_append_result_two_node_quorum_achieved() { state.init_cluster_metadata(&raft_context.membership).await.unwrap(); // initial commit_index = 0 - let (internal_event_tx, mut internal_event_rx) = mpsc::unbounded_channel(); + let (internal_event_tx, _internal_event_rx) = mpsc::unbounded_channel(); state .handle_append_result( peer2_id, @@ -1633,10 +1623,6 @@ async fn test_handle_append_result_two_node_quorum_achieved() { 3, "two-node: commit advances to 3 (quorum: leader + peer2 = 2/2)" ); - assert!(matches!( - internal_event_rx.try_recv(), - Ok(InternalEvent::NotifyNewCommitIndex(_)) - )); } /// B1: Three-node cluster achieves quorum when all peers respond with success. @@ -1685,7 +1671,7 @@ async fn test_handle_append_result_three_node_quorum_all_peers() { ); context.raft_context.storage.raft_log = Arc::new(raft_log); - let (internal_event_tx, mut internal_event_rx) = mpsc::unbounded_channel(); + let (internal_event_tx, _internal_event_rx) = mpsc::unbounded_channel(); // Peer 2 responds → quorum (2/3), commit advances context @@ -1704,10 +1690,6 @@ async fn test_handle_append_result_three_node_quorum_all_peers() { 10, "commit advances after peer 2" ); - assert!(matches!( - internal_event_rx.try_recv(), - Ok(InternalEvent::NotifyNewCommitIndex(_)) - )); // Peer 3 responds → already committed, calculate_majority returns None context @@ -1768,7 +1750,7 @@ async fn test_handle_append_result_three_node_quorum_partial_timeout() { raft_log.expect_calculate_majority_matched_index().returning(|_, _, _| Some(10)); context.raft_context.storage.raft_log = Arc::new(raft_log); - let (internal_event_tx, mut internal_event_rx) = mpsc::unbounded_channel(); + let (internal_event_tx, _internal_event_rx) = mpsc::unbounded_channel(); // Only peer 2 responds; peer 3 never does context @@ -1787,10 +1769,6 @@ async fn test_handle_append_result_three_node_quorum_partial_timeout() { 10, "quorum achieved with partial timeout (2/3 nodes = majority)" ); - assert!(matches!( - internal_event_rx.try_recv(), - Ok(InternalEvent::NotifyNewCommitIndex(_)) - )); } /// B1: Five-node cluster achieves quorum when majority (3 of 4 peers) respond. @@ -1855,7 +1833,7 @@ async fn test_handle_append_result_five_node_quorum_majority() { ); context.raft_context.storage.raft_log = Arc::new(raft_log); - let (internal_event_tx, mut internal_event_rx) = mpsc::unbounded_channel(); + let (internal_event_tx, _internal_event_rx) = mpsc::unbounded_channel(); // 3 peers respond → majority of 5 (need ≥3) achieved context @@ -1874,10 +1852,6 @@ async fn test_handle_append_result_five_node_quorum_majority() { 10, "commit advances (4/5 ≥ majority)" ); - assert!(matches!( - internal_event_rx.try_recv(), - Ok(InternalEvent::NotifyNewCommitIndex(_)) - )); // Remaining peers just confirm — no second commit advance for peer_id in [3u32, 4u32] { @@ -1940,7 +1914,7 @@ async fn test_handle_append_result_three_node_no_quorum_all_timeouts() { ); assert!( internal_event_rx.try_recv().is_err(), - "no NotifyNewCommitIndex without commit advancement" + "no internal event expected without commit advancement" ); } @@ -2004,7 +1978,7 @@ async fn test_handle_append_result_three_node_no_quorum_single_peer_insufficient ); assert!( internal_event_rx.try_recv().is_err(), - "no NotifyNewCommitIndex without commit advancement" + "no internal event expected without commit advancement" ); } @@ -2085,7 +2059,7 @@ async fn test_handle_append_result_five_node_no_quorum_minority() { ); assert!( internal_event_rx.try_recv().is_err(), - "no NotifyNewCommitIndex without quorum" + "no internal event expected without quorum" ); } @@ -2510,14 +2484,11 @@ mod test_leader_update_next_index { let mut state = LeaderState::::new(1, context_inner.node_config.clone()); state.update_next_index(2, 100).unwrap(); state.update_match_index(2, 7).unwrap(); - state.update_peer_index( - 2, - &PeerUpdate { - match_index: Some(11), - next_index: 12, - success: true, - }, - ); + state.update_peer_index(2, &PeerUpdate { + match_index: Some(11), + next_index: 12, + success: true, + }); assert_eq!(state.next_index.get(&2), Some(&100)); assert_eq!(state.match_index.get(&2), Some(&11)); } @@ -2533,21 +2504,19 @@ mod test_leader_update_next_index { let mut state = LeaderState::::new(1, context_inner.node_config.clone()); state.update_next_index(2, 100).unwrap(); state.update_match_index(2, 7).unwrap(); - state.update_peer_index( - 2, - &PeerUpdate { - match_index: None, - next_index: 12, - success: false, - }, - ); + state.update_peer_index(2, &PeerUpdate { + match_index: None, + next_index: 12, + success: false, + }); assert_eq!(state.next_index.get(&2), Some(&12)); assert_eq!(state.match_index.get(&2), Some(&7)); } // On a conflict response, if the computed retry point (PeerUpdate::next_index) - // is below the confirmed match_index + 1, the floor guard must clamp next_index to match_index + 1. - // Retreating below already-confirmed entries would violate Raft's Log Matching Property. + // is below the confirmed match_index + 1, the floor guard must clamp next_index to match_index + // + 1. Retreating below already-confirmed entries would violate Raft's Log Matching + // Property. #[tokio::test] async fn test_update_peer_index_conflict_floor_guard_prevents_retreat_below_match_index() { let (_graceful_tx, graceful_rx) = watch::channel(()); @@ -2556,14 +2525,11 @@ mod test_leader_update_next_index { let mut state = LeaderState::::new(1, context_inner.node_config.clone()); state.update_next_index(2, 100).unwrap(); state.update_match_index(2, 50).unwrap(); - state.update_peer_index( - 2, - &PeerUpdate { - match_index: None, - next_index: 12, - success: false, - }, - ); + state.update_peer_index(2, &PeerUpdate { + match_index: None, + next_index: 12, + success: false, + }); assert_eq!(state.next_index.get(&2), Some(&(50 + 1))); assert_eq!(state.match_index.get(&2), Some(&50)); } @@ -2576,14 +2542,11 @@ mod test_leader_update_next_index { let context_inner = MockBuilder::new(graceful_rx).build_context(); let mut state = LeaderState::::new(1, context_inner.node_config.clone()); - state.update_peer_index( - 2, - &PeerUpdate { - match_index: Some(11), - next_index: 12, - success: true, - }, - ); + state.update_peer_index(2, &PeerUpdate { + match_index: Some(11), + next_index: 12, + success: true, + }); assert_eq!( state.peer_replication_state.get(&2), Some(&PeerReplicationState::Replicate), @@ -2599,14 +2562,11 @@ mod test_leader_update_next_index { let context_inner = MockBuilder::new(graceful_rx).build_context(); let mut state = LeaderState::::new(1, context_inner.node_config.clone()); - state.update_peer_index( - 2, - &PeerUpdate { - match_index: None, - next_index: 12, - success: false, - }, - ); + state.update_peer_index(2, &PeerUpdate { + match_index: None, + next_index: 12, + success: false, + }); assert_eq!( state.peer_replication_state.get(&2), Some(&PeerReplicationState::Probe), @@ -2626,14 +2586,11 @@ mod test_leader_update_next_index { let mut state = LeaderState::::new(1, context_inner.node_config.clone()); state.update_next_index(2, 1).unwrap(); state.update_match_index(2, 1).unwrap(); - state.update_peer_index( - 2, - &PeerUpdate { - match_index: Some(11), - next_index: 12, - success: true, - }, - ); + state.update_peer_index(2, &PeerUpdate { + match_index: Some(11), + next_index: 12, + success: true, + }); assert_eq!(state.next_index.get(&2), Some(&12)); assert_eq!(state.match_index.get(&2), Some(&11)); } @@ -2790,9 +2747,9 @@ async fn test_two_reachable_peers_of_four_confirm_quorum() { /// The partition lasts: one peer keeps answering every round, three never do. /// /// # Then -/// - After more than `election_timeout_min` the leader no longer admits writes and the vote -/// guard no longer claims a recent quorum ACK. The loop below never calls anything but the -/// normal reply path, so a refresh from stale history would keep both open forever. +/// - After more than `election_timeout_min` the leader no longer admits writes and the vote guard +/// no longer claims a recent quorum ACK. The loop below never calls anything but the normal reply +/// path, so a refresh from stale history would keep both open forever. #[tokio::test] #[traced_test] async fn test_leader_loses_write_admission_when_only_one_peer_keeps_answering() { @@ -2815,11 +2772,13 @@ async fn test_leader_loses_write_admission_when_only_one_peer_keeps_answering() #[tokio::test] #[traced_test] async fn test_queued_reads_wait_for_a_real_quorum() { + use tokio::time::Duration; + use tokio::time::Instant; + use crate::ReadConsistencyPolicy; use crate::client::ClientReadRequest; use crate::convert::safe_kv_bytes; use crate::raft_role::leader_state::PendingLeaseRead; - use tokio::time::{Duration, Instant}; let mut context = five_voter_partition_context("/tmp/test_queued_reads_wait").await; @@ -2834,13 +2793,13 @@ async fn test_queued_reads_wait_for_a_real_quorum() { read(ReadConsistencyPolicy::LinearizableRead), linearizable_tx, )); - context.state.pending_reads.insert( - 0, - crate::raft_role::leader_state::PendingReadBatch { + context + .state + .pending_reads + .insert(0, crate::raft_role::leader_state::PendingReadBatch { deadline: Instant::now() + Duration::from_secs(5), requests, - }, - ); + }); let (lease_tx, mut lease_rx) = MaybeCloneOneshot::new(); context.state.pending_lease_reads.push_back(PendingLeaseRead { request: read(ReadConsistencyPolicy::LeaseRead), @@ -2983,19 +2942,20 @@ async fn test_record_voter_ack_moves_forward_again_when_a_second_peer_recovers() /// - 3 voters (leader, B, C): one peer is a quorum, B answered the round sent at 1400 /// /// # When -/// - the cluster grows to 5 voters (D and a new E join, the old E=5 is gone), then C answers -/// the round sent at 1450 +/// - the cluster grows to 5 voters (D and a new E join, the old E=5 is gone), then C answers the +/// round sent at 1450 /// /// # Then /// - the removed peer's time is gone -/// - the quorum time is recomputed for 5 voters (2 peers needed, B:1400 and C:1450 -> 1400) -/// and reported once. Without the reset the stale 1400 of the 3-voter conclusion would hide it. +/// - the quorum time is recomputed for 5 voters (2 peers needed, B:1400 and C:1450 -> 1400) and +/// reported once. Without the reset the stale 1400 of the 3-voter conclusion would hide it. #[tokio::test] #[traced_test] async fn test_membership_change_drops_left_peers_and_recomputes_the_quorum_time() { - use crate::MockMembership; use d_engine_proto::server::cluster::NodeMeta; + use crate::MockMembership; + let mut context = five_voter_partition_context("/tmp/test_membership_change_resets").await; let state = &mut context.state; state.cluster_metadata.total_voters = 3; diff --git a/d-engine-core/src/raft_role/leader_state_test/single_voter_commit_test.rs b/d-engine-core/src/raft_role/leader_state_test/single_voter_commit_test.rs index 2f6e7ec1..b823eceb 100644 --- a/d-engine-core/src/raft_role/leader_state_test/single_voter_commit_test.rs +++ b/d-engine-core/src/raft_role/leader_state_test/single_voter_commit_test.rs @@ -12,6 +12,13 @@ //! single-voter clusters — there is no majority to absorb the risk the old design //! accepted. +use std::sync::Arc; +use std::sync::atomic::AtomicU64; +use std::sync::atomic::Ordering; + +use tokio::sync::mpsc; +use tokio::sync::watch; + use crate::MockMembership; use crate::MockRaftLog; use crate::MockReplicationCore; @@ -19,10 +26,6 @@ use crate::raft_role::leader_state::LeaderState; use crate::raft_role::role_state::RaftRoleState; use crate::test_utils::MockBuilder; use crate::test_utils::mock::MockTypeConfig; -use std::sync::Arc; -use std::sync::atomic::{AtomicU64, Ordering}; -use tokio::sync::mpsc; -use tokio::sync::watch; async fn setup_single_voter( last_entry_id_val: u64 diff --git a/d-engine-core/src/raft_role/leader_state_test/snapshot_test.rs b/d-engine-core/src/raft_role/leader_state_test/snapshot_test.rs index 18d5887f..359d1d20 100644 --- a/d-engine-core/src/raft_role/leader_state_test/snapshot_test.rs +++ b/d-engine-core/src/raft_role/leader_state_test/snapshot_test.rs @@ -7,14 +7,19 @@ use std::path::PathBuf; use std::sync::Arc; use std::sync::atomic::Ordering; +use d_engine_proto::common::LogId; +use d_engine_proto::server::storage::SnapshotMetadata; +use tokio::sync::mpsc; +use tokio::sync::watch; + +use crate::MockPurgeExecutor; +use crate::SnapshotError; use crate::config::RaftNodeConfig as CoreRaftNodeConfig; use crate::raft_role::leader_state::LeaderState; use crate::role_state::RaftRoleState; -use crate::test_utils::mock::{MockBuilder, MockTypeConfig, mock_raft_log}; -use crate::{MockPurgeExecutor, SnapshotError}; -use d_engine_proto::common::LogId; -use d_engine_proto::server::storage::SnapshotMetadata; -use tokio::sync::{mpsc, watch}; +use crate::test_utils::mock::MockBuilder; +use crate::test_utils::mock::MockTypeConfig; +use crate::test_utils::mock::mock_raft_log; // ============================================================================ // CreateSnapshot Event Tests diff --git a/d-engine-core/src/raft_role/leader_state_test/snapshot_worker_test.rs b/d-engine-core/src/raft_role/leader_state_test/snapshot_worker_test.rs index 593eebd3..1d2734dd 100644 --- a/d-engine-core/src/raft_role/leader_state_test/snapshot_worker_test.rs +++ b/d-engine-core/src/raft_role/leader_state_test/snapshot_worker_test.rs @@ -35,12 +35,15 @@ use std::sync::Arc; use std::time::Duration; -use d_engine_proto::common::{EntryPayload, NodeRole::Follower, NodeStatus}; +use d_engine_proto::common::EntryPayload; +use d_engine_proto::common::NodeRole::Follower; +use d_engine_proto::common::NodeStatus; use d_engine_proto::server::cluster::NodeMeta; use d_engine_proto::server::replication::AppendEntriesRequest; use d_engine_proto::server::storage::SnapshotMetadata; use futures::StreamExt; -use tokio::sync::{mpsc, watch}; +use tokio::sync::mpsc; +use tokio::sync::watch; use tracing_test::traced_test; use crate::MockMembership; @@ -50,11 +53,13 @@ use crate::MockTransport; use crate::RaftRequestWithSignal; use crate::SnapshotApplyResult; use crate::event::InternalEvent; -use crate::maybe_clone_oneshot::{MaybeCloneOneshot, RaftOneshot}; +use crate::maybe_clone_oneshot::MaybeCloneOneshot; +use crate::maybe_clone_oneshot::RaftOneshot; use crate::raft_role::leader_state::LeaderState; use crate::raft_role::role_state::PeerReplicationState; use crate::raft_role::role_state::RaftRoleState; -use crate::test_utils::mock::{MockTypeConfig, mock_raft_context}; +use crate::test_utils::mock::MockTypeConfig; +use crate::test_utils::mock::mock_raft_context; // ── helpers ────────────────────────────────────────────────────────────────── @@ -98,7 +103,7 @@ fn state_machine_with_snapshot() -> MockStateMachine { use d_engine_proto::common::LogId; let mut sm = MockStateMachine::new(); sm.expect_start().returning(|| Ok(())); - sm.expect_stop().returning(|| Ok(())); + sm.expect_close().returning(|| Ok(())); sm.expect_is_running().returning(|| true); sm.expect_get().returning(|_| Ok(None)); sm.expect_apply_chunk().returning(|_| Ok(vec![])); @@ -185,10 +190,9 @@ fn open_stream_capturing_appends() -> ( /// /// # Scenario /// - `prepare_batch_requests` returns snapshot_targets = [2], append_requests = [] -/// - The snapshot metadata's own boundary is index 10 (`stub_snapshot_metadata`), while -/// the leader's log tip (`raft_log.last_entry_id`) is 15 — deliberately different, so -/// the assertion below proves the event carries the snapshot's own boundary and not -/// the leader's current tip. +/// - The snapshot metadata's own boundary is index 10 (`stub_snapshot_metadata`), while the +/// leader's log tip (`raft_log.last_entry_id`) is 15 — deliberately different, so the assertion +/// below proves the event carries the snapshot's own boundary and not the leader's current tip. /// - Expected: `InternalEvent::SnapshotPushCompleted { peer_id: 2, success: true, /// last_included_index: Some(10) }` arrives #[tokio::test] @@ -231,15 +235,12 @@ async fn test_worker_handles_snapshot_task_and_emits_completed_event() { .try_recv() .expect("SnapshotPushCompleted should arrive after snapshot transfer"); assert!( - matches!( - event, - InternalEvent::SnapshotPushCompleted { - peer_id: 2, - success: true, - term: _, - last_included_index: Some(10) - } - ), + matches!(event, InternalEvent::SnapshotPushCompleted { + peer_id: 2, + success: true, + term: _, + last_included_index: Some(10) + }), "expected SnapshotPushCompleted for peer 2 carrying the snapshot's own boundary \ (index 10, not the leader's log tip of 15), got {event:?}" ); @@ -250,8 +251,8 @@ async fn test_worker_handles_snapshot_task_and_emits_completed_event() { /// separate worker-side flag is consulted. /// /// # Scenario -/// - Peer 2's replication state is preset to `Snapshot` (simulating an attempt -/// already in flight), with no real transfer running. +/// - Peer 2's replication state is preset to `Snapshot` (simulating an attempt already in flight), +/// with no real transfer running. /// - `process_batch` runs with peer 2 in `snapshot_targets`. /// - Expected: `transport.send_snapshot` is never called. #[tokio::test] @@ -303,8 +304,8 @@ async fn test_phase6_skips_redispatch_when_snapshot_already_in_flight() { /// - Peer 2 starts as `Replicate` (it was healthy before suddenly falling behind). /// - `process_batch` routes peer 2 to `snapshot_targets`. /// - Expected: `peer_replication_state(2)` is `Snapshot` immediately after dispatch — -/// synchronously, with no polling/sleep needed, because the state write and the -/// task dispatch happen in the same function call. +/// synchronously, with no polling/sleep needed, because the state write and the task dispatch +/// happen in the same function call. #[tokio::test] async fn test_phase6_snapshot_dispatch_sets_snapshot_state() { let (mut state, mut ctx) = @@ -354,11 +355,10 @@ async fn test_phase6_snapshot_dispatch_sets_snapshot_state() { /// /// # Scenario /// - Peer 2 is preset to `Snapshot` (a transfer is in flight). -/// - `process_batch` runs with an Append request for peer 2 in `append_requests` -/// (simulating a heartbeat that computed fresh data to send, unaware the peer -/// is mid-transfer). -/// - Expected: nothing reaches the worker's bidi stream, and `next_index` for -/// peer 2 is unchanged from its preset value. +/// - `process_batch` runs with an Append request for peer 2 in `append_requests` (simulating a +/// heartbeat that computed fresh data to send, unaware the peer is mid-transfer). +/// - Expected: nothing reaches the worker's bidi stream, and `next_index` for peer 2 is unchanged +/// from its preset value. #[tokio::test] async fn test_phase5_skips_append_and_leaves_next_index_untouched_while_peer_in_snapshot_state() { let (mut state, mut ctx) = new_leader_and_ctx( @@ -411,8 +411,8 @@ async fn test_phase5_skips_append_and_leaves_next_index_untouched_while_peer_in_ /// independently of whatever Phase 5 does or doesn't do. /// /// # Scenario -/// - Peer 2 is set to `Snapshot` (irrelevant to the worker, relevant only to -/// prove this test isn't accidentally exercising Phase 5's gate). +/// - Peer 2 is set to `Snapshot` (irrelevant to the worker, relevant only to prove this test isn't +/// accidentally exercising Phase 5's gate). /// - An `Append` task is handed directly to `send_to_worker_or_spawn`. /// - Expected: the request reaches the worker's bidi stream unmodified. #[tokio::test] @@ -448,8 +448,8 @@ async fn test_worker_forwards_any_append_task_it_is_given_without_inspecting_sta /// # Scenario /// - Peer 2 is dispatched to Snapshot (via a real `process_batch` round). /// - The completion handler is invoked directly with `success = true`. -/// - Expected: state is `Probe`, and a subsequent round with an Append request -/// for peer 2 is no longer blocked. +/// - Expected: state is `Probe`, and a subsequent round with an Append request for peer 2 is no +/// longer blocked. #[tokio::test] async fn test_snapshot_completion_success_returns_to_probe_and_allows_append() { let (mut state, mut ctx) = new_leader_and_ctx( @@ -490,6 +490,11 @@ async fn test_snapshot_completion_success_returns_to_probe_and_allows_append() { "a successful snapshot completion must return the peer to Probe" ); + // Let the freshly spawned worker open its stream: an Append dispatched while the worker + // has no open stream is (by design) not queued, and this test is about the Snapshot -> + // Probe transition re-enabling Append, not about dispatch to a disconnected peer. + tokio::time::sleep(Duration::from_millis(50)).await; + // Round 2: an Append for peer 2 must now go through, on the same worker. ctx.handlers .replication_handler @@ -516,9 +521,9 @@ async fn test_snapshot_completion_success_returns_to_probe_and_allows_append() { /// # Scenario /// - Peer 2 is in Snapshot state (an attempt in flight). /// - The completion handler is invoked directly with `success = false`. -/// - Expected: state is `Probe`, and the worker handle's retry-backoff timestamp -/// is set to a point in the future (so an immediate re-dispatch attempt would -/// be skipped — covered by the backoff-window test below). +/// - Expected: state is `Probe`, and the worker handle's retry-backoff timestamp is set to a point +/// in the future (so an immediate re-dispatch attempt would be skipped — covered by the +/// backoff-window test below). #[tokio::test] async fn test_snapshot_completion_failure_returns_to_probe_with_backoff() { let (mut state, mut ctx) = @@ -574,12 +579,12 @@ async fn test_snapshot_completion_failure_returns_to_probe_with_backoff() { /// corrupting bookkeeping for whatever the peer is doing now. /// /// # Scenario -/// - Peer 2 is left in `Probe` (no snapshot currently in flight — e.g. the peer -/// caught up normally through ordinary replication after an earlier attempt). -/// - A `SnapshotPushCompleted` for peer 2 is delivered anyway (simulating a -/// straggler event from a past attempt). -/// - Expected: state stays exactly `Probe` (not perturbed), proving the handler -/// checked "am I actually in Snapshot right now" before acting. +/// - Peer 2 is left in `Probe` (no snapshot currently in flight — e.g. the peer caught up normally +/// through ordinary replication after an earlier attempt). +/// - A `SnapshotPushCompleted` for peer 2 is delivered anyway (simulating a straggler event from a +/// past attempt). +/// - Expected: state stays exactly `Probe` (not perturbed), proving the handler checked "am I +/// actually in Snapshot right now" before acting. #[tokio::test] async fn test_stale_snapshot_completion_is_ignored_once_state_has_moved_on() { let (mut state, ctx) = new_leader_and_ctx( @@ -611,12 +616,10 @@ async fn test_stale_snapshot_completion_is_ignored_once_state_has_moved_on() { /// /// # Scenario /// - Peer 2 fails a snapshot attempt (state returns to `Probe`, backoff starts). -/// - The very next round classifies peer 2 as still needing a snapshot -/// (`snapshot_targets = [2]`) — simulating that nothing about its position -/// has changed since the failure. -/// - Expected: no second `send_snapshot` call happens (backoff not yet expired), -/// and the peer's state stays `Probe` (it does not incorrectly re-enter -/// `Snapshot` without a real dispatch). +/// - The very next round classifies peer 2 as still needing a snapshot (`snapshot_targets = [2]`) — +/// simulating that nothing about its position has changed since the failure. +/// - Expected: no second `send_snapshot` call happens (backoff not yet expired), and the peer's +/// state stays `Probe` (it does not incorrectly re-enter `Snapshot` without a real dispatch). #[tokio::test] async fn test_backoff_window_blocks_redispatch_even_when_classification_still_requests_snapshot() { let (mut state, mut ctx) = new_leader_and_ctx( @@ -695,3 +698,106 @@ async fn test_backoff_window_blocks_redispatch_even_when_classification_still_re "must not re-enter Snapshot state without an actual dispatch happening" ); } + +/// Snapshot pushes are state transitions with their own retry/backoff and `Snapshot` peer +/// state; the "do not feed a disconnected peer" rule applies to Append dispatch only. +/// +/// # Scenario +/// - Batch 1 spawns peer 2's worker with an Append request; the worker cannot connect yet. +/// - Batch 2 asks for a snapshot push to peer 2 (Phase 6) while the worker is disconnected. +/// - The peer then becomes reachable. +/// +/// # Guarantees checked +/// - The snapshot task is still delivered: `send_snapshot` runs exactly once and +/// `SnapshotPushCompleted { peer_id: 2, success: true }` arrives. A fix that skipped disconnected +/// peers in Phase 6 too would leave the peer stuck in `Snapshot` state forever. +/// +/// GREEN before the fix and must stay green after it. +#[tokio::test] +#[traced_test] +async fn test_snapshot_push_still_dispatched_to_disconnected_peer() { + use std::sync::atomic::AtomicBool; + use std::sync::atomic::AtomicUsize; + use std::sync::atomic::Ordering; + + let (mut state, mut ctx) = + new_leader_and_ctx("/tmp/test_snapshot_push_still_dispatched_to_disconnected_peer"); + state.init_cluster_metadata(&ctx.membership).await.unwrap(); + + let calls = Arc::new(AtomicUsize::new(0)); + ctx.handlers.replication_handler.expect_prepare_batch_requests().returning( + move |_, _, _, _, _, _| { + if calls.fetch_add(1, Ordering::SeqCst) == 0 { + Ok(crate::replication::PrepareResult { + append_requests: vec![( + 2, + AppendEntriesRequest { + entries: vec![d_engine_proto::common::Entry { + index: 1, + term: 1, + payload: None, + }], + ..Default::default() + }, + 1, + )], + snapshot_targets: vec![], + }) + } else { + Ok(crate::replication::PrepareResult { + append_requests: vec![], + snapshot_targets: vec![2], + }) + } + }, + ); + + let allow_connect = Arc::new(AtomicBool::new(false)); + let open_attempts = Arc::new(AtomicUsize::new(0)); + let (allow, attempts) = (allow_connect.clone(), open_attempts.clone()); + let mut transport = MockTransport::::new(); + transport.expect_send_snapshot().times(1).returning(|_, _, _, _, _, _| Ok(())); + transport.expect_open_replication_stream().returning(move |_, _, _, _| { + attempts.fetch_add(1, Ordering::SeqCst); + if !allow.load(Ordering::SeqCst) { + return Err(crate::NetworkError::ConnectError("peer unreachable".into()).into()); + } + let (tx, mut rx) = mpsc::channel::(128); + tokio::spawn(async move { while rx.recv().await.is_some() {} }); + Ok(crate::ReplicationStream { + sender: tx, + receiver: futures::stream::pending().boxed(), + }) + }); + ctx.transport = Arc::new(transport); + + let (internal_event_tx, mut internal_event_rx) = mpsc::unbounded_channel(); + state.process_batch(one_entry_batch(), &internal_event_tx, &ctx).await.unwrap(); + for _ in 0..1000 { + if open_attempts.load(Ordering::SeqCst) >= 1 { + break; + } + tokio::time::sleep(Duration::from_millis(5)).await; + } + assert!( + open_attempts.load(Ordering::SeqCst) >= 1, + "precondition: worker is in its reconnect loop" + ); + + state.process_batch(one_entry_batch(), &internal_event_tx, &ctx).await.unwrap(); + allow_connect.store(true, Ordering::SeqCst); + + for _ in 0..1000 { + while let Ok(event) = internal_event_rx.try_recv() { + if matches!(event, InternalEvent::SnapshotPushCompleted { + peer_id: 2, + success: true, + .. + }) { + return; + } + } + tokio::time::sleep(Duration::from_millis(5)).await; + } + panic!("snapshot push to a disconnected-then-reconnected peer never completed"); +} diff --git a/d-engine-core/src/raft_role/leader_state_test/stale_learner_deadline_test.rs b/d-engine-core/src/raft_role/leader_state_test/stale_learner_deadline_test.rs index 14a9d309..076da21d 100644 --- a/d-engine-core/src/raft_role/leader_state_test/stale_learner_deadline_test.rs +++ b/d-engine-core/src/raft_role/leader_state_test/stale_learner_deadline_test.rs @@ -17,17 +17,20 @@ use std::sync::Arc; use std::time::Duration; +use d_engine_proto::common::EntryPayload; use parking_lot::Mutex; -use tokio::sync::{mpsc, watch}; +use tokio::sync::mpsc; +use tokio::sync::watch; use tokio::time::Instant; use crate::MockMembership; use crate::MockRaftLog; -use crate::raft_role::leader_state::{LeaderState, PendingPromotion}; +use crate::raft_role::leader_state::LeaderState; +use crate::raft_role::leader_state::PendingPromotion; use crate::role_state::RaftRoleState; -use crate::test_utils::mock::{MockBuilder, MockTypeConfig}; +use crate::test_utils::mock::MockBuilder; +use crate::test_utils::mock::MockTypeConfig; use crate::test_utils::node_config; -use d_engine_proto::common::EntryPayload; // ============================================================================ // Helpers @@ -136,7 +139,8 @@ async fn test_tick_skips_stale_check_before_deadline() { // Call tick() — deadline is far in the future, stale check must be skipped let result = leader.tick(&internal_event_tx, &raft_tx, &ctx).await; assert!(result.is_ok()); - // If MockMembership::get_node_status were called, mockall would panic — test passes by not panicking + // If MockMembership::get_node_status were called, mockall would panic — test passes by not + // panicking } // ============================================================================ @@ -145,7 +149,8 @@ async fn test_tick_skips_stale_check_before_deadline() { #[tokio::test] async fn test_tick_removes_stale_learner_when_deadline_fires() { - use d_engine_proto::common::{NodeRole::Follower, NodeStatus}; + use d_engine_proto::common::NodeRole::Follower; + use d_engine_proto::common::NodeStatus; use d_engine_proto::server::cluster::NodeMeta; let threshold = Duration::from_millis(1); // Expires almost immediately diff --git a/d-engine-core/src/raft_role/leader_state_test/state_management_test.rs b/d-engine-core/src/raft_role/leader_state_test/state_management_test.rs index 028527aa..4406b18c 100644 --- a/d-engine-core/src/raft_role/leader_state_test/state_management_test.rs +++ b/d-engine-core/src/raft_role/leader_state_test/state_management_test.rs @@ -4,6 +4,18 @@ //! - Leader discovery event handling //! - State size tracking +use std::mem::size_of; +use std::sync::Arc; + +use d_engine_proto::common::LogId; +use d_engine_proto::common::NodeRole::Leader; +use d_engine_proto::common::NodeStatus; +use d_engine_proto::server::cluster::LeaderDiscoveryRequest; +use d_engine_proto::server::cluster::NodeMeta; +use tokio::sync::mpsc; +use tokio::sync::watch; +use tracing_test::traced_test; + use crate::candidate_state::CandidateState; use crate::event::InboundEvent; use crate::maybe_clone_oneshot::RaftOneshot; @@ -12,13 +24,6 @@ use crate::raft_role::leader_state::LeaderState; use crate::role_state::RaftRoleState; use crate::test_utils::mock::MockTypeConfig; use crate::test_utils::mock::mock_raft_context; -use d_engine_proto::common::LogId; -use d_engine_proto::common::{NodeRole::Leader, NodeStatus}; -use d_engine_proto::server::cluster::{LeaderDiscoveryRequest, NodeMeta}; -use std::mem::size_of; -use std::sync::Arc; -use tokio::sync::{mpsc, watch}; -use tracing_test::traced_test; // ============================================================================ // Test Helper Functions diff --git a/d-engine-core/src/raft_role/leader_state_test/worker_lifecycle_test.rs b/d-engine-core/src/raft_role/leader_state_test/worker_lifecycle_test.rs index d2717208..3edb7360 100644 --- a/d-engine-core/src/raft_role/leader_state_test/worker_lifecycle_test.rs +++ b/d-engine-core/src/raft_role/leader_state_test/worker_lifecycle_test.rs @@ -10,14 +10,19 @@ //! actually executes `send_to_worker_or_spawn`. use std::sync::Arc; -use std::sync::atomic::{AtomicUsize, Ordering}; +use std::sync::atomic::AtomicUsize; +use std::sync::atomic::Ordering; use bytes::Bytes; -use d_engine_proto::common::{EntryPayload, NodeRole::Follower, NodeStatus}; +use d_engine_proto::common::EntryPayload; +use d_engine_proto::common::NodeRole::Follower; +use d_engine_proto::common::NodeStatus; use d_engine_proto::server::cluster::NodeMeta; -use d_engine_proto::server::replication::{AppendEntriesRequest, AppendEntriesResponse}; +use d_engine_proto::server::replication::AppendEntriesRequest; +use d_engine_proto::server::replication::AppendEntriesResponse; use futures::StreamExt; -use tokio::sync::{mpsc, watch}; +use tokio::sync::mpsc; +use tokio::sync::watch; use tokio_stream::wrappers::UnboundedReceiverStream; use tracing_test::traced_test; @@ -26,10 +31,12 @@ use crate::MockRaftLog; use crate::MockTransport; use crate::RaftRequestWithSignal; use crate::event::InternalEvent; -use crate::maybe_clone_oneshot::{MaybeCloneOneshot, RaftOneshot}; +use crate::maybe_clone_oneshot::MaybeCloneOneshot; +use crate::maybe_clone_oneshot::RaftOneshot; use crate::raft_role::leader_state::LeaderState; use crate::raft_role::role_state::RaftRoleState; -use crate::test_utils::mock::{MockTypeConfig, mock_raft_context}; +use crate::test_utils::mock::MockTypeConfig; +use crate::test_utils::mock::mock_raft_context; // ── helpers ────────────────────────────────────────────────────────────────── @@ -276,7 +283,8 @@ async fn test_worker_reused_on_subsequent_requests() { /// /// # Guarantees checked /// - `worker_count_for_test() == 1` after the rebuild (old dead handle replaced). -/// - `InternalEvent::AppendResult` arrives on `internal_event_rx`, confirming the request was delivered. +/// - `InternalEvent::AppendResult` arrives on `internal_event_rx`, confirming the request was +/// delivered. #[tokio::test] #[traced_test] async fn test_worker_rebuilt_after_death() { @@ -365,14 +373,14 @@ async fn test_worker_rebuilt_after_death() { /// - Transport always fails `open_replication_stream` (peer unreachable). /// - Worker is spawned via `process_batch`; it enters the inner reconnect loop. /// - The `LeaderState` is dropped, which drops `ReplicationWorkerHandle` and closes `task_rx`. -/// - On the next iteration the worker checks `task_rx.is_closed()` and must exit cleanly -/// without making further connection attempts. +/// - On the next iteration the worker checks `task_rx.is_closed()` and must exit cleanly without +/// making further connection attempts. /// /// # Guarantees checked -/// - After the handle is dropped, `open_replication_stream` call count stops increasing. -/// This proves the `if task_rx.is_closed() { return; }` guard fires correctly, -/// preventing an infinite reconnect loop that would flood the inbound event loop with -/// zombie signals and block heartbeats (regression: zombie detection + dead peer). +/// - After the handle is dropped, `open_replication_stream` call count stops increasing. This +/// proves the `if task_rx.is_closed() { return; }` guard fires correctly, preventing an infinite +/// reconnect loop that would flood the inbound event loop with zombie signals and block +/// heartbeats (regression: zombie detection + dead peer). #[tokio::test] #[traced_test] async fn test_replication_worker_exits_when_handle_dropped() { @@ -455,12 +463,12 @@ async fn test_replication_worker_exits_when_handle_dropped() { /// /// # Scenario /// - `open_replication_stream` succeeds, but its receiver immediately yields `Err(status)`. -/// - The worker's recv task detects the error; the main loop emits `PeerStreamError` -/// and reconnects (calls `open_replication_stream` again). +/// - The worker's recv task detects the error; the main loop emits `PeerStreamError` and reconnects +/// (calls `open_replication_stream` again). /// /// # Why -/// - This is the "peer went away mid-stream" recovery path: without it, a broken -/// stream would silently stop replication for that peer. +/// - This is the "peer went away mid-stream" recovery path: without it, a broken stream would +/// silently stop replication for that peer. #[tokio::test] #[traced_test] async fn test_worker_reconnects_on_stream_recv_error() { @@ -775,14 +783,11 @@ async fn test_worker_forwards_append_result_on_recv() { .try_recv() .expect("AppendResult should arrive from the stream ACK"); assert!( - matches!( - event, - InternalEvent::AppendResult { - follower_id: 2, - result: Ok(_), - .. - } - ), + matches!(event, InternalEvent::AppendResult { + follower_id: 2, + result: Ok(_), + .. + }), "expected AppendResult for peer 2, got {event:?}" ); } @@ -791,8 +796,8 @@ async fn test_worker_forwards_append_result_on_recv() { /// permanently stops for the rest of this worker's lifetime (#446). /// /// # Scenario -/// - 16 sends land on the sampling boundary exactly once (the 16th), starting -/// an RTT sample that never resolves — the mock stream stays pending, no ACK. +/// - 16 sends land on the sampling boundary exactly once (the 16th), starting an RTT sample that +/// never resolves — the mock stream stays pending, no ACK. /// - The stream then errors, forcing `recv_handle.abort()` + reconnect. /// /// # Guarantee checked @@ -876,9 +881,21 @@ async fn test_rtt_sample_reset_on_reconnect_after_interrupted_sample() { // time (leader_state.rs:3384) — since this test never sends a real ACK, // that flag would never clear naturally, so it's reset by hand here to // let all 16 attempts actually reach the worker. - for _ in 0..16 { + for i in 0..16 { state.reset_in_flight(2); state.process_batch(one_entry_batch(), &internal_event_tx, &ctx).await.unwrap(); + if i == 0 { + // The first dispatch spawns the worker. A dispatch made while the worker has no + // open stream is (by design) not queued, so wait for the stream before sending the + // rest — otherwise the 16th send, which starts the RTT sample, would never happen. + for _ in 0..1000 { + if open_count.load(Ordering::SeqCst) >= 1 { + break; + } + tokio::time::sleep(std::time::Duration::from_millis(5)).await; + } + tokio::time::sleep(std::time::Duration::from_millis(50)).await; + } } // Give the worker time to process all 16 sends before breaking the stream. @@ -965,3 +982,352 @@ async fn test_worker_opens_stream_with_configured_send_queue_capacity() { "the worker must pass replication_send_queue_capacity to open_replication_stream" ); } + +// ── disconnected-peer dispatch (R3) ─────────────────────────────────────────── +// +// Background: while a peer's replication worker has no open stream (it sits in its reconnect +// loop) it does not dequeue tasks, but the leader keeps enqueuing one per write batch into the +// worker's UNBOUNDED task queue. Measured: 37k-66k tasks piled up in ~40 s and were drained in +// one burst after reconnect, each having waited ~20 s. +// +// Contract these tests pin (the fix: a worker-owned `connected` flag read by Phase 5 BEFORE any +// bookkeeping): while the worker has no open stream the leader must dispatch NOTHING to it — +// data and heartbeats alike — and must leave no accounting behind (`next_index`, in-flight), so +// the first send after reconnect starts from the same place. Dropping is safe for Raft: the +// leader only advances on ACKs and retransmits from `next_index`. +// +// The tests observe behaviour only (what the stream receives, what the leader accounted), so +// they compile against the current code and are RED for the right reason until the fix lands. + +use crate::raft_role::role_state::PeerReplicationState; + +/// Scripted peer: `open_replication_stream` fails until `allow_connect` is set, then succeeds. +/// Counts how many requests each established stream receives. +#[derive(Clone, Default)] +struct PeerProbe { + allow_connect: Arc, + open_attempts: Arc, + established: Arc, + /// Requests delivered to the first established stream. + received_first: Arc, + /// Requests delivered to any later (re-established) stream. + received_later: Arc, +} + +/// `break_first_stream`: the first open succeeds immediately but its receiver errors at once +/// (a stream that was up and then broke); every other open needs `allow_connect`. +fn scripted_peer(break_first_stream: bool) -> (MockTransport, PeerProbe) { + let probe = PeerProbe::default(); + let p = probe.clone(); + let mut transport = MockTransport::::new(); + transport.expect_open_replication_stream().returning(move |_, _, _, _| { + let attempt = p.open_attempts.fetch_add(1, Ordering::SeqCst) + 1; + let immediate = break_first_stream && attempt == 1; + if !immediate && !p.allow_connect.load(Ordering::SeqCst) { + return Err(crate::NetworkError::ConnectError("peer unreachable".into()).into()); + } + let n = p.established.fetch_add(1, Ordering::SeqCst) + 1; + let counter = if n == 1 { + p.received_first.clone() + } else { + p.received_later.clone() + }; + let (req_tx, mut req_rx) = mpsc::channel::(128); + let receiver = if immediate { + tokio::spawn(async move { + while req_rx.recv().await.is_some() { + counter.fetch_add(1, Ordering::SeqCst); + } + }); + futures::stream::iter(vec![Err::( + tonic::Status::unavailable("peer gone"), + )]) + .boxed() + } else { + // Keep the response side open (never answers) so the stream stays healthy. + let (resp_tx, resp_rx) = + mpsc::channel::>(1); + tokio::spawn(async move { + while req_rx.recv().await.is_some() { + counter.fetch_add(1, Ordering::SeqCst); + } + drop(resp_tx); + }); + tokio_stream::wrappers::ReceiverStream::new(resp_rx).boxed() + }; + Ok(crate::ReplicationStream { + sender: req_tx, + receiver, + }) + }); + (transport, probe) +} + +/// Leader wired to the scripted peer. `is_heartbeat(k)` decides whether the k-th +/// `prepare_batch_requests` call (1-based) yields an empty request (heartbeat) or a data request. +/// Peer 2 is put in `Replicate` state so the in-flight window (256) does not gate data requests — +/// the situation in which the leader really does enqueue one task per batch. +async fn leader_with_scripted_peer( + tmp: &str, + transport: MockTransport, + is_heartbeat: fn(usize) -> bool, +) -> ( + LeaderState, + crate::RaftContext, + mpsc::UnboundedSender, + mpsc::UnboundedReceiver, +) { + let (_graceful_tx, graceful_rx) = watch::channel(()); + let mut ctx = mock_raft_context(tmp, graceful_rx, None); + ctx.membership = Arc::new(two_peer_membership()); + + let calls = Arc::new(AtomicUsize::new(0)); + ctx.handlers.replication_handler.expect_prepare_batch_requests().returning( + move |_, _, _, _, _, _| { + let k = calls.fetch_add(1, Ordering::SeqCst) + 1; + let req = if is_heartbeat(k) { + stub_request() + } else { + stub_request_with_entry(k as u64) + }; + Ok(crate::PrepareResult { + append_requests: vec![(2, req, k as u64)], + snapshot_targets: vec![], + }) + }, + ); + ctx.transport = Arc::new(transport); + + let mut raft_log = MockRaftLog::new(); + raft_log.expect_last_entry_id().returning(|| 0); + raft_log.expect_flush().returning(|| Ok(())); + raft_log.expect_save_hard_state().returning(|_| Ok(())); + ctx.storage.raft_log = Arc::new(raft_log); + + let mut state = LeaderState::::new(1, ctx.node_config.clone()); + state.init_cluster_metadata(&ctx.membership).await.unwrap(); + state.set_peer_replication_state(2, PeerReplicationState::Replicate); + + let (tx, rx) = mpsc::unbounded_channel(); + (state, ctx, tx, rx) +} + +/// Polls `cond` every 5 ms for up to 5 s; panics with `what` on timeout. +async fn wait_until( + what: &str, + cond: impl Fn() -> bool, +) { + for _ in 0..1000 { + if cond() { + return; + } + tokio::time::sleep(std::time::Duration::from_millis(5)).await; + } + panic!("timed out waiting for: {what}"); +} + +/// Peer unreachable from the start: nothing dispatched while it is down may be replayed after it +/// comes back, and the leader must not have accounted for it. +/// +/// # Scenario +/// - Batch 1 spawns the worker (no handle exists yet, so nothing can be skipped). +/// - The worker is in its reconnect loop (every open fails). Batches 2-6 are dispatched: data +/// requests and heartbeats mixed. +/// - The peer then becomes reachable. +/// +/// # Guarantees checked +/// - The re-established stream receives exactly 1 request (the spawn task), not 6: the 5 sent while +/// the peer was down were never enqueued. (Before the fix all 6 are replayed.) +/// - In-flight accounting holds only the 1 data request that really was dispatched; the skipped +/// ones left no trace (before the fix: one slot per data request, 4 in total). +/// - Dispatch resumes once the stream is back: an unreachable peer is not skipped forever. +#[tokio::test] +#[traced_test] +async fn test_unreachable_peer_does_not_accumulate_replication_tasks() { + let (transport, probe) = scripted_peer(false); + // k = 3 and 5 are heartbeats, the rest carry an entry. + let (mut state, ctx, tx, _rx) = leader_with_scripted_peer( + "/tmp/test_unreachable_peer_does_not_accumulate_replication_tasks", + transport, + |k| k == 3 || k == 5, + ) + .await; + + state.process_batch(one_entry_batch(), &tx, &ctx).await.unwrap(); + wait_until("worker enters its reconnect loop", || { + probe.open_attempts.load(Ordering::SeqCst) >= 1 + }) + .await; + // After the spawning dispatch (k=1, one entry): next_index = effective(1) + 1 entry. + let next_index_after_spawn = state.next_index(2); + + for _ in 0..5 { + state.process_batch(one_entry_batch(), &tx, &ctx).await.unwrap(); + } + assert_eq!( + state.next_index(2), + next_index_after_spawn, + "skipped dispatches must not advance next_index (the first send after reconnect must \ + start from the same place)" + ); + + probe.allow_connect.store(true, Ordering::SeqCst); + wait_until("stream re-established", || { + probe.established.load(Ordering::SeqCst) >= 1 + }) + .await; + tokio::time::sleep(std::time::Duration::from_millis(300)).await; // let the worker drain its queue + + assert_eq!( + probe.received_first.load(Ordering::SeqCst), + 1, + "requests dispatched while the stream was down must not be queued and replayed on reconnect" + ); + assert_eq!( + state.in_flight_count(2), + 1, + "skipped dispatches must leave no in-flight accounting behind" + ); + + // Control: once the stream is back the peer is served again. + let before = probe.received_first.load(Ordering::SeqCst); + for _ in 0..100 { + state.process_batch(one_entry_batch(), &tx, &ctx).await.unwrap(); + if probe.received_first.load(Ordering::SeqCst) > before { + return; + } + tokio::time::sleep(std::time::Duration::from_millis(20)).await; + } + panic!("dispatch to the peer never resumed after the stream was re-established"); +} + +/// Heartbeats alone must not pile up either (they are empty, bypass the in-flight window, and +/// were the one request kind still enqueued for a gated peer). +/// +/// # Guarantees checked +/// - 5 heartbeats sent while the stream is down are not replayed after reconnect: only the +/// worker-spawning request is delivered. (Before the fix: 6.) +#[tokio::test] +#[traced_test] +async fn test_unreachable_peer_heartbeats_are_not_queued() { + let (transport, probe) = scripted_peer(false); + let (mut state, ctx, tx, _rx) = leader_with_scripted_peer( + "/tmp/test_unreachable_peer_heartbeats_are_not_queued", + transport, + |_| true, + ) + .await; + + state.process_batch(one_entry_batch(), &tx, &ctx).await.unwrap(); + wait_until("worker enters its reconnect loop", || { + probe.open_attempts.load(Ordering::SeqCst) >= 1 + }) + .await; + for _ in 0..5 { + state.process_batch(one_entry_batch(), &tx, &ctx).await.unwrap(); + } + + probe.allow_connect.store(true, Ordering::SeqCst); + wait_until("stream re-established", || { + probe.established.load(Ordering::SeqCst) >= 1 + }) + .await; + tokio::time::sleep(std::time::Duration::from_millis(300)).await; + + assert_eq!( + probe.received_first.load(Ordering::SeqCst), + 1, + "heartbeats dispatched while the stream was down must not be queued and replayed" + ); +} + +/// A stream that was up and then broke: while the worker reconnects it counts as disconnected +/// exactly like a peer that never connected. +/// +/// # Scenario +/// - First stream opens, then its receiver errors at once -> the worker leaves the main loop and +/// reconnects; the reconnect attempts fail. +/// - 4 batches are dispatched during that window; then the peer becomes reachable. +/// +/// # Guarantees checked +/// - The second stream receives nothing from the 4 batches sent while reconnecting. (Before the +/// fix: 4.) +/// - Dispatch resumes on the second stream afterwards. +#[tokio::test] +#[traced_test] +async fn test_broken_stream_peer_is_not_fed_while_reconnecting() { + let (transport, probe) = scripted_peer(true); + let (mut state, ctx, tx, _rx) = leader_with_scripted_peer( + "/tmp/test_broken_stream_peer_is_not_fed_while_reconnecting", + transport, + |k| k % 2 == 0, + ) + .await; + + state.process_batch(one_entry_batch(), &tx, &ctx).await.unwrap(); + // attempt 1 = the stream that breaks; attempt >= 2 = the worker is reconnecting. + wait_until("worker is reconnecting after the broken stream", || { + probe.open_attempts.load(Ordering::SeqCst) >= 2 + }) + .await; + + for _ in 0..4 { + state.process_batch(one_entry_batch(), &tx, &ctx).await.unwrap(); + } + + probe.allow_connect.store(true, Ordering::SeqCst); + wait_until("second stream established", || { + probe.established.load(Ordering::SeqCst) >= 2 + }) + .await; + tokio::time::sleep(std::time::Duration::from_millis(300)).await; + + assert_eq!( + probe.received_later.load(Ordering::SeqCst), + 0, + "requests dispatched while reconnecting after a broken stream must not reach the new stream" + ); + + for _ in 0..100 { + state.process_batch(one_entry_batch(), &tx, &ctx).await.unwrap(); + if probe.received_later.load(Ordering::SeqCst) > 0 { + return; + } + tokio::time::sleep(std::time::Duration::from_millis(20)).await; + } + panic!("dispatch never resumed on the re-established stream"); +} + +/// Control: a healthy, connected peer receives every dispatch. Guards against a fix that skips +/// too much (e.g. a flag that never turns true, or turns true too late). +/// +/// This test is GREEN before the fix and must stay green after it. +#[tokio::test] +#[traced_test] +async fn test_connected_peer_receives_every_dispatch() { + let (transport, probe) = scripted_peer(false); + probe.allow_connect.store(true, Ordering::SeqCst); + let (mut state, ctx, tx, _rx) = leader_with_scripted_peer( + "/tmp/test_connected_peer_receives_every_dispatch", + transport, + |k| k % 3 == 0, + ) + .await; + + state.process_batch(one_entry_batch(), &tx, &ctx).await.unwrap(); + wait_until("stream established", || { + probe.established.load(Ordering::SeqCst) >= 1 + }) + .await; + tokio::time::sleep(std::time::Duration::from_millis(50)).await; + + for _ in 0..5 { + state.process_batch(one_entry_batch(), &tx, &ctx).await.unwrap(); + tokio::time::sleep(std::time::Duration::from_millis(10)).await; + } + + wait_until("all 6 dispatches delivered to the connected peer", || { + probe.received_first.load(Ordering::SeqCst) == 6 + }) + .await; +} diff --git a/d-engine-core/src/raft_role/learner_state.rs b/d-engine-core/src/raft_role/learner_state.rs index 68a72e7e..aa36563a 100644 --- a/d-engine-core/src/raft_role/learner_state.rs +++ b/d-engine-core/src/raft_role/learner_state.rs @@ -1,3 +1,32 @@ +use std::collections::BTreeMap; +use std::fmt::Debug; +use std::marker::PhantomData; +use std::sync::Arc; +use std::sync::atomic::AtomicBool; +use std::sync::atomic::Ordering; +use std::time::Duration; + +use async_trait::async_trait; +use d_engine_proto::common::LogId; +use d_engine_proto::common::NodeRole; +use d_engine_proto::common::NodeRole::Follower; +use d_engine_proto::common::NodeRole::Learner; +use d_engine_proto::server::cluster::ClusterConfUpdateResponse; +use d_engine_proto::server::cluster::JoinRequest; +use d_engine_proto::server::cluster::LeaderDiscoveryRequest; +use d_engine_proto::server::cluster::LeaderDiscoveryResponse; +use d_engine_proto::server::election::VoteResponse; +use d_engine_proto::server::election::VotedFor; +use d_engine_proto::server::storage::SnapshotMetadata; +use tokio::sync::mpsc::{self}; +use tokio::time::Instant; +use tonic::Status; +use tracing::debug; +use tracing::error; +use tracing::info; +use tracing::trace; +use tracing::warn; + use super::RaftRole; use super::SharedState; use super::StateSnapshot; @@ -24,32 +53,6 @@ use crate::cluster_printer::print_learner_promoted_to_voter; use crate::cluster_printer::print_role_transition_line; use crate::role_state::PendingAcks; use crate::role_state::schedule_and_execute_purge; -use async_trait::async_trait; -use d_engine_proto::common::LogId; -use d_engine_proto::common::NodeRole; -use d_engine_proto::common::NodeRole::Follower; -use d_engine_proto::common::NodeRole::Learner; -use d_engine_proto::server::cluster::ClusterConfUpdateResponse; -use d_engine_proto::server::cluster::JoinRequest; -use d_engine_proto::server::cluster::LeaderDiscoveryRequest; -use d_engine_proto::server::cluster::LeaderDiscoveryResponse; -use d_engine_proto::server::election::VoteResponse; -use d_engine_proto::server::election::VotedFor; -use d_engine_proto::server::storage::SnapshotMetadata; -use std::collections::BTreeMap; -use std::fmt::Debug; -use std::marker::PhantomData; -use std::sync::Arc; -use std::sync::atomic::{AtomicBool, Ordering}; -use std::time::Duration; -use tokio::sync::mpsc::{self}; -use tokio::time::Instant; -use tonic::Status; -use tracing::debug; -use tracing::error; -use tracing::info; -use tracing::trace; -use tracing::warn; /// Learner node's state in Raft cluster. /// @@ -291,14 +294,8 @@ impl RaftRoleState for LearnerState { } InboundEvent::InstallSnapshotChunk(stream, sender) => { - self.handle_install_snapshot_chunk_workflow( - stream, - sender, - ctx, - internal_event_tx, - &state_snapshot, - ) - .await?; + self.handle_install_snapshot_chunk_workflow(stream, sender, ctx, internal_event_tx) + .await?; } InboundEvent::JoinCluster(_join_request, sender) => { diff --git a/d-engine-core/src/raft_role/learner_state_test.rs b/d-engine-core/src/raft_role/learner_state_test.rs index 1de5ef0e..06ed1e72 100644 --- a/d-engine-core/src/raft_role/learner_state_test.rs +++ b/d-engine-core/src/raft_role/learner_state_test.rs @@ -1,3 +1,25 @@ +use std::sync::Arc; + +use bytes::Bytes; +use d_engine_proto::client::WriteCommand; +use d_engine_proto::common::LogId; +use d_engine_proto::common::NodeRole; +use d_engine_proto::common::NodeStatus; +use d_engine_proto::server::cluster::ClusterConfChangeRequest; +use d_engine_proto::server::cluster::ClusterConfUpdateResponse; +use d_engine_proto::server::cluster::MetadataRequest; +use d_engine_proto::server::cluster::NodeMeta; +use d_engine_proto::server::cluster::cluster_conf_update_response; +use d_engine_proto::server::election::VoteRequest; +use d_engine_proto::server::storage::SnapshotAck; +use d_engine_proto::server::storage::SnapshotChunk; +use d_engine_proto::server::storage::snapshot_ack::ChunkStatus; +use mockall::predicate::eq; +use prost::Message; +use tokio::sync::mpsc; +use tokio::sync::watch; +use tonic::Code; + use crate::ClientCmd; use crate::Error; use crate::InboundEvent; @@ -28,25 +50,6 @@ use crate::test_utils::mock::mock_raft_context; use crate::test_utils::mock::mock_raft_context_with_temp; use crate::test_utils::mock::mock_raft_log; use crate::test_utils::node_config; -use bytes::Bytes; -use d_engine_proto::client::WriteCommand; -use d_engine_proto::common::LogId; -use d_engine_proto::common::NodeRole; -use d_engine_proto::common::NodeStatus; -use d_engine_proto::server::cluster::ClusterConfChangeRequest; -use d_engine_proto::server::cluster::ClusterConfUpdateResponse; -use d_engine_proto::server::cluster::MetadataRequest; -use d_engine_proto::server::cluster::NodeMeta; -use d_engine_proto::server::cluster::cluster_conf_update_response; -use d_engine_proto::server::election::VoteRequest; -use d_engine_proto::server::storage::SnapshotAck; -use d_engine_proto::server::storage::SnapshotChunk; -use d_engine_proto::server::storage::snapshot_ack::ChunkStatus; -use mockall::predicate::eq; -use prost::Message; -use std::sync::Arc; -use tokio::sync::{mpsc, watch}; -use tonic::Code; /// Test: LearnerState drain_read_buffer returns NotLeader error /// @@ -290,7 +293,6 @@ async fn test_learner_handles_cluster_conf_update_success() { /// /// Expected: /// - Sends LeaderDiscovered event -/// - Sends NotifyNewCommitIndex event /// - Updates current_term to leader's term /// - Updates commit_index /// - Returns AppendEntriesResponse with success=true @@ -354,12 +356,6 @@ async fn test_learner_handles_append_entries_success() { crate::InternalEvent::LeaderDiscovered(5, _) )); - // Verify: NotifyNewCommitIndex event - assert!(matches!( - internal_event_rx.try_recv().unwrap(), - crate::InternalEvent::NotifyNewCommitIndex(_) - )); - assert_eq!(state.current_term(), leader_term); assert_eq!(state.commit_index(), expected_commit); @@ -519,7 +515,7 @@ async fn test_learner_rejects_client_write_request() { let mut state = LearnerState::::new(1, context.node_config.clone()); let (resp_tx, mut resp_rx) = MaybeCloneOneshot::new(); - let cmd = ClientCmd::Propose( + let cmd = ClientCmd::propose( ClientWriteRequest { client_id: 1, command: Some(Bytes::from( @@ -1243,9 +1239,10 @@ async fn test_learner_node_not_found_on_membership_applied() { // ============================================================================ mod snapshot_tests { - use super::*; use std::sync::atomic::Ordering; + use super::*; + /// Test: Learner gracefully ignores stale leader-only internal events /// /// Protocol scenario: a leader steps down or a stale event arrives. @@ -1637,7 +1634,8 @@ async fn test_apply_completed_triggers_snapshot_when_condition_met() { /// Test: Learner ApplyCompleted does NOT trigger snapshot when condition is not met /// -/// Purpose: Verify that learners respect snapshot conditions and don't create unnecessary snapshots. +/// Purpose: Verify that learners respect snapshot conditions and don't create unnecessary +/// snapshots. /// /// Scenario: /// - Learner receives ApplyCompleted event @@ -2050,8 +2048,8 @@ async fn test_learner_install_snapshot_advances_commit_index_to_boundary() { /// /// # Then /// - Response MUST be success: false -/// - The Worker (`state_machine_commands`) must NOT be invoked — no fake sink is wired up, -/// so a call there would panic on the closed channel / never resolve. +/// - The Worker (`state_machine_commands`) must NOT be invoked — no fake sink is wired up, so a +/// call there would panic on the closed channel / never resolve. #[tokio::test] async fn test_learner_install_snapshot_does_not_report_success_on_mid_chunk_failure() { let (_graceful_tx, graceful_rx) = watch::channel(()); @@ -2122,8 +2120,8 @@ async fn test_learner_install_snapshot_does_not_report_success_on_mid_chunk_fail /// match_index and stop retrying, leaving the learner permanently behind. /// /// # Given -/// - prepare_snapshot_stream: sends Accepted ACKs (transfer succeeded), -/// then returns Err (apply_snapshot_from_file failed) +/// - prepare_snapshot_stream: sends Accepted ACKs (transfer succeeded), then returns Err +/// (apply_snapshot_from_file failed) /// /// # When /// - Leader pushes a snapshot (InstallSnapshotChunk event) diff --git a/d-engine-core/src/raft_role/mod.rs b/d-engine-core/src/raft_role/mod.rs index 495671d4..79af22fd 100644 --- a/d-engine-core/src/raft_role/mod.rs +++ b/d-engine-core/src/raft_role/mod.rs @@ -33,7 +33,9 @@ use follower_state::FollowerState; pub use leader_state::ClusterMetadata; use leader_state::LeaderState; use learner_state::LearnerState; -pub use read_lease::{ReadLease, init_clock, now_ms}; +pub use read_lease::ReadLease; +pub use read_lease::init_clock; +pub use read_lease::now_ms; use role_state::RaftRoleState; use serde::Deserialize; use serde::Deserializer; @@ -101,6 +103,12 @@ pub struct SharedState { /// increases monotonically) pub commit_index: u64, + /// Highest log index already handed to the state machine worker (initialized to the state + /// machine's `last_applied`, increases monotonically). Lives next to `commit_index` and + /// survives role transitions. It is NOT `applied`: that only advances once the worker has + /// finished, so using it would re-deliver entries still in flight. + pub delivered_index: u64, + /// In-memory leader ID for hot-path reads (0 = no leader) /// Performance optimization: avoid RwLock on AppendEntries path current_leader_id: AtomicU32, @@ -117,6 +125,7 @@ impl Clone for SharedState { node_id: self.node_id, hard_state: self.hard_state, commit_index: self.commit_index, + delivered_index: self.delivered_index, current_leader_id: AtomicU32::new(self.current_leader_id.load(Ordering::Acquire)), read_lease: Arc::clone(&self.read_lease), } @@ -132,6 +141,7 @@ impl std::fmt::Debug for SharedState { .field("node_id", &self.node_id) .field("hard_state", &self.hard_state) .field("commit_index", &self.commit_index) + .field("delivered_index", &self.delivered_index) .field("current_leader_id", &self.current_leader()) .finish() } @@ -175,6 +185,7 @@ impl SharedState { node_id, hard_state, commit_index: last_applied_index_option.unwrap_or(0), + delivered_index: last_applied_index_option.unwrap_or(0), current_leader_id: AtomicU32::new(0), read_lease: Arc::new(ReadLease::new()), } @@ -525,6 +536,18 @@ impl RaftRole { .await } + /// Hand newly committed entries to the state machine worker (same for every role). + pub(crate) async fn dispatch_committed( + &mut self, + ctx: &RaftContext, + internal_event_tx: &mpsc::UnboundedSender, + ) -> crate::Result<()> + where + T: TypeConfig, + { + self.state_mut().dispatch_committed(ctx, internal_event_tx).await + } + /// Dispatch LogFlushed(durable) to the current role state. /// Follower/Learner: sends deferred ACK. Leader: recalculates commit. pub(crate) async fn handle_log_flushed( diff --git a/d-engine-core/src/raft_role/pending_ack_test.rs b/d-engine-core/src/raft_role/pending_ack_test.rs index c0f4ce48..d3b765cd 100644 --- a/d-engine-core/src/raft_role/pending_ack_test.rs +++ b/d-engine-core/src/raft_role/pending_ack_test.rs @@ -1,8 +1,8 @@ //! Tests for #446's withheld-AppendEntries-ACK primitives: //! //! - `RaftRoleState::resolve_pending_acks` — the release / reject decision; -//! - `RaftRole::{take,restore}_pending_acks` — carrying the queue across a role -//! transition instead of dropping it. +//! - `RaftRole::{take,restore}_pending_acks` — carrying the queue across a role transition +//! instead of dropping it. //! //! These drive bare role structs (no `RaftContext`, no mocks, no fs), so they //! stay cheap to run and cheap to change. @@ -66,14 +66,11 @@ fn withhold( role.state_mut() .pending_append_acks_mut() .expect("role keeps a pending-ack queue") - .insert( - (term_when_withheld, index), - PendingAck { - claimed_term, - withheld_at: std::time::Instant::now(), - senders: vec![tx], - }, - ); + .insert((term_when_withheld, index), PendingAck { + claimed_term, + withheld_at: std::time::Instant::now(), + senders: vec![tx], + }); rx } @@ -139,14 +136,11 @@ fn test_resolve_answers_every_sender_on_an_index() { let mut role = follower(5); let (tx1, mut rx1) = MaybeCloneOneshot::new(); let (tx2, mut rx2) = MaybeCloneOneshot::new(); - role.state_mut().pending_append_acks_mut().unwrap().insert( - (5, 8), - PendingAck { - claimed_term: 5, - withheld_at: std::time::Instant::now(), - senders: vec![tx1, tx2], - }, - ); + role.state_mut().pending_append_acks_mut().unwrap().insert((5, 8), PendingAck { + claimed_term: 5, + withheld_at: std::time::Instant::now(), + senders: vec![tx1, tx2], + }); role.state_mut().resolve_pending_acks(8); diff --git a/d-engine-core/src/raft_role/raft_role_test.rs b/d-engine-core/src/raft_role/raft_role_test.rs index 4afa805b..43c5d823 100644 --- a/d-engine-core/src/raft_role/raft_role_test.rs +++ b/d-engine-core/src/raft_role/raft_role_test.rs @@ -1,7 +1,9 @@ -use std::{sync::Arc, thread}; +use std::sync::Arc; +use std::thread; use d_engine_proto::server::election::VotedFor; -use tokio::sync::{mpsc, watch}; +use tokio::sync::mpsc; +use tokio::sync::watch; use super::super::*; use super::candidate_state::CandidateState; diff --git a/d-engine-core/src/raft_role/read_lease.rs b/d-engine-core/src/raft_role/read_lease.rs index 9e855549..fa60d3cc 100644 --- a/d-engine-core/src/raft_role/read_lease.rs +++ b/d-engine-core/src/raft_role/read_lease.rs @@ -1,5 +1,6 @@ use std::sync::OnceLock; -use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::atomic::AtomicU64; +use std::sync::atomic::Ordering; use std::time::Instant; // Process-local monotonic epoch. Initialized on first call to now_ms(). diff --git a/d-engine-core/src/raft_role/read_lease_test.rs b/d-engine-core/src/raft_role/read_lease_test.rs index bb4e58d3..8c9c8722 100644 --- a/d-engine-core/src/raft_role/read_lease_test.rs +++ b/d-engine-core/src/raft_role/read_lease_test.rs @@ -2,7 +2,8 @@ use std::sync::Arc; use std::sync::Barrier; use std::thread; -use super::{ReadLease, now_ms}; +use super::ReadLease; +use super::now_ms; // ── now_ms ──────────────────────────────────────────────────────────────────── diff --git a/d-engine-core/src/raft_role/role_state.rs b/d-engine-core/src/raft_role/role_state.rs index 1986e19f..da6ecda7 100644 --- a/d-engine-core/src/raft_role/role_state.rs +++ b/d-engine-core/src/raft_role/role_state.rs @@ -1,8 +1,36 @@ +use std::collections::BTreeMap; +use std::sync::atomic::AtomicBool; +use std::sync::atomic::Ordering; + +use async_trait::async_trait; +use d_engine_proto::common::Entry; +use d_engine_proto::common::LogId; +use d_engine_proto::common::MembershipChange; +use d_engine_proto::common::entry_payload::Payload; +use d_engine_proto::server::election::VotedFor; +use d_engine_proto::server::replication::AppendEntriesRequest; +use d_engine_proto::server::replication::AppendEntriesResponse; +use d_engine_proto::server::replication::SuccessResult; +use d_engine_proto::server::replication::append_entries_response; +use d_engine_proto::server::storage::SnapshotAck; +use d_engine_proto::server::storage::SnapshotChunk; +use d_engine_proto::server::storage::SnapshotMetadata; +use d_engine_proto::server::storage::SnapshotResponse; +use tokio::sync::mpsc; +use tokio::time::Instant; +use tonic::Status; +use tracing::debug; +use tracing::error; +use tracing::info; +use tracing::trace; +use tracing::warn; + use super::RaftRole; use super::SharedState; use super::StateSnapshot; use crate::AppendResponseWithUpdates; use crate::ConsensusError; +use crate::Error; use crate::InboundEvent; use crate::InternalEvent; use crate::MaybeCloneOneshotSender; @@ -27,25 +55,6 @@ use crate::client::LeaderHint; use crate::event::ClientCmd; use crate::scoped_timer::ScopedTimer; use crate::utils::cluster::error; -use async_trait::async_trait; -use d_engine_proto::common::LogId; -use d_engine_proto::server::election::VotedFor; -use d_engine_proto::server::replication::AppendEntriesRequest; -use d_engine_proto::server::replication::AppendEntriesResponse; -use d_engine_proto::server::replication::SuccessResult; -use d_engine_proto::server::replication::append_entries_response; -use d_engine_proto::server::storage::SnapshotAck; -use d_engine_proto::server::storage::SnapshotChunk; -use d_engine_proto::server::storage::SnapshotMetadata; -use d_engine_proto::server::storage::SnapshotResponse; -use std::collections::BTreeMap; -use std::sync::atomic::{AtomicBool, Ordering}; -use tokio::sync::mpsc; -use tokio::time::Instant; -use tonic::Status; -use tracing::error; -use tracing::warn; -use tracing::{debug, info}; /// Per-peer replication trust state (#436): gates optimistic `next_index` advance. #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -281,34 +290,56 @@ pub(crate) trait RaftRoleState: Send + Sync + 'static { Ok(()) } - fn update_commit_index_with_signal( + /// Hand every committed entry not yet delivered to the state machine worker, in log order, + /// each exactly once, without waiting for it to apply them. Called once per loop iteration: + /// how `commit_index` advanced differs per role, delivering what it covers does not. + /// + /// Entries are handed over in batches: commands accumulate, and a config or noop entry ends + /// the batch it belongs to. A config entry's membership change is applied right after its + /// batch is sent, before the next batch. + /// Apply a committed membership change. Per Raft protocol the leader can remove itself; after + /// applying the removal it must step down immediately. + /// + /// # Errors + /// - `Error::Fatal` if a committed membership change cannot be applied: the membership view + /// would no longer match the log and nothing can repair that. Later entries are withheld and + /// replayed from the state machine's `last_applied` on restart. + /// - Any other error (log read failed, worker unavailable) means committed entries cannot be + /// applied; the caller must stop the node. + async fn dispatch_committed( &mut self, - role: i32, - current_term: u64, - new_commit_index: u64, + ctx: &RaftContext, internal_event_tx: &mpsc::UnboundedSender, ) -> Result<()> { - if let Err(e) = self.update_commit_index(new_commit_index) { - error!("Follower::update_commit_index: {:?}", e); - return Err(e); + // `applied` is ahead of `delivered_index` after a snapshot install: those entries are + // already covered. + let floor = ctx + .state_machine_handler() + .last_applied() + .max(self.shared_state().delivered_index); + let commit_index = self.commit_index(); + + if commit_index <= floor { + return Ok(()); } + + let _timer = ScopedTimer::new("dispatch_committed"); + + let entries = ctx.raft_log().get_entries_range(floor + 1..=commit_index)?; debug!( - "[Node-{}] update_commit_index_with_signal, new_commit_index: {:?}", + "[Node-{}] delivering committed entries {}..={commit_index}", self.node_id(), - new_commit_index + floor + 1 ); - internal_event_tx - .send(InternalEvent::NotifyNewCommitIndex(NewCommitData { - new_commit_index, - role, - current_term, - })) - .map_err(|e| { - error!("Failed to send NotifyNewCommitIndex: {e:?}"); - NetworkError::SingalSendFailed(format!("{:?}", e)) - })?; - + for batch in split_into_batches(entries_with_payload(entries)) { + let membership_change = config_change_at_end(&batch); + send_to_sm_worker(ctx, batch)?; + if let Some((index, change)) = membership_change { + apply_membership(self.node_id(), ctx, internal_event_tx, index, change).await?; + } + } + self.shared_state_mut().delivered_index = commit_index; Ok(()) } @@ -354,8 +385,8 @@ pub(crate) trait RaftRoleState: Send + Sync + 'static { } /// True if a membership change is committed but not yet applied. A node that campaigned in - /// that window would count votes against a stale voter set (raft-rs refuses to campaign - /// here). Only reads the log when something is actually unapplied, so the common case is + /// that window would count votes against a stale voter set, so it must not campaign here. + /// Only reads the log when something is actually unapplied, so the common case is /// two integer comparisons. /// /// A log read failure is reported as an error: the caller cannot prove that nothing is @@ -433,7 +464,7 @@ pub(crate) trait RaftRoleState: Send + Sync + 'static { use crate::config::ReadConsistencyPolicy as ClientReadConsistencyPolicy; match cmd { - ClientCmd::Propose(_, sender) => { + ClientCmd::Propose(_, _, sender) => { // Writes always require leader - reject immediately let _ = sender.send(Ok(self.create_not_leader_response(ctx))); } @@ -529,10 +560,10 @@ pub(crate) trait RaftRoleState: Send + Sync + 'static { /// Release withheld AppendEntries ACKs now that the log is durable through /// `durable`. For each withheld ACK: /// - /// - withheld under a term this node has since left → fail it with a conflict. - /// A higher-term leader may have overwritten the log at that index; within a - /// single term a follower's entries are never replaced, so the term check - /// alone is a sufficient content guard and no log lookup is needed. + /// - withheld under a term this node has since left → fail it with a conflict. A higher-term + /// leader may have overwritten the log at that index; within a single term a follower's + /// entries are never replaced, so the term check alone is a sufficient content guard and no + /// log lookup is needed. /// - claimed index now `<= durable` → rebuild and send the success. /// - otherwise → keep waiting. /// @@ -727,17 +758,9 @@ pub(crate) trait RaftRoleState: Send + Sync + 'static { commit_index_update, }) => { if let Some(commit) = commit_index_update - && let Err(e) = self.update_commit_index_with_signal( - state_snapshot.role, - state_snapshot.current_term, - commit, - &internal_event_tx, - ) + && let Err(e) = self.update_commit_index(commit) { - error!( - "update_commit_index_with_signal,commit={}, error: {:?}", - commit, e - ); + error!("update_commit_index,commit={}, error: {:?}", commit, e); return Err(e); } debug!("AppendEntriesResponse: {:?}", response); @@ -838,7 +861,6 @@ pub(crate) trait RaftRoleState: Send + Sync + 'static { sender: MaybeCloneOneshotSender>, ctx: &RaftContext, internal_event_tx: mpsc::UnboundedSender, - state_snapshot: &StateSnapshot, ) -> Result<()> { let my_term = self.current_term(); @@ -951,12 +973,7 @@ pub(crate) trait RaftRoleState: Send + Sync + 'static { let boundary = match result { SnapshotApplyResult::Applied { last_included } => { if last_included.index > self.commit_index() - && let Err(e) = self.update_commit_index_with_signal( - state_snapshot.role, - my_term, - last_included.index, - &internal_event_tx, - ) + && let Err(e) = self.update_commit_index(last_included.index) { error!(?e, "Failed to advance commit_index after snapshot install"); } @@ -1055,7 +1072,8 @@ pub(crate) trait RaftRoleState: Send + Sync + 'static { Ok(()) } - /// Returns the snapshot deduplication flag for roles that support independent snapshot creation. + /// Returns the snapshot deduplication flag for roles that support independent snapshot + /// creation. /// /// Returns `None` for roles without snapshot capability (Candidate). The default /// `handle_create_snapshot` uses this to short-circuit with `RoleViolation` instead of @@ -1163,6 +1181,121 @@ pub(crate) trait RaftRoleState: Send + Sync + 'static { } } +/// True if `change` removes `my_id` itself. +pub(super) fn is_self_removal_config( + my_id: u32, + change: &MembershipChange, +) -> bool { + matches!( + &change.change, + Some(d_engine_proto::common::membership_change::Change::RemoveNode(remove)) + if remove.node_id == my_id + ) +} + +fn payload_of(entry: &Entry) -> Option<&Payload> { + entry.payload.as_ref()?.payload.as_ref() +} + +/// Drops entries that carry no payload kind (corrupt entry or hostile peer), with a warning. +fn entries_with_payload(entries: Vec) -> Vec { + entries + .into_iter() + .filter(|entry| { + let has_payload = payload_of(entry).is_some(); + if !has_payload { + warn!("entry {} has no payload kind; skipped", entry.index); + } + has_payload + }) + .collect() +} + +/// Splits committed entries, in log order, into the messages handed to the state machine +/// worker. A config or noop entry ends its batch, so it is sent together with the commands +/// before it and nothing after it. +pub(super) fn split_into_batches(entries: Vec) -> Vec> { + let mut batches = vec![]; + let mut batch = vec![]; + for entry in entries { + let ends_batch = matches!( + payload_of(&entry), + Some(Payload::Config(_) | Payload::Noop(_)) + ); + batch.push(entry); + if ends_batch { + batches.push(std::mem::take(&mut batch)); + } + } + if !batch.is_empty() { + batches.push(batch); + } + batches +} + +/// The membership change a batch ends with, if its last entry is a config entry. +fn config_change_at_end(batch: &[Entry]) -> Option<(u64, MembershipChange)> { + let last = batch.last()?; + match payload_of(last)? { + Payload::Config(change) => Some((last.index, change.clone())), + _ => None, + } +} + +/// Hand a batch of entries to the state machine worker without waiting for it to apply. +fn send_to_sm_worker( + ctx: &RaftContext, + entries: Vec, +) -> Result<()> { + trace!("Sending batch to SM Worker: {} entries", entries.len()); + ctx.state_machine_commands().apply_entries(entries).inspect_err(|e| { + error!("SM Worker channel closed: {:?}", e); + }) +} + +/// # Errors +/// `Error::Fatal` if the change cannot be applied: the membership view would no longer match +/// the log, and nothing can repair that. +async fn apply_membership( + my_id: u32, + ctx: &RaftContext, + internal_event_tx: &mpsc::UnboundedSender, + index: u64, + change: MembershipChange, +) -> Result<()> { + // Timer name kept from the commit handler so existing dashboards keep working. + let _timer = ScopedTimer::new("apply_config_change"); + + debug!("Received config change at index {index}: {change:?}"); + + // Check before applying: afterwards the node may no longer be a member. + let is_self_removal = is_self_removal_config(my_id, &change); + + if let Err(e) = ctx.membership_ref().apply_config_change(change).await { + error!("[{my_id}] Failed to apply config change at index {index}: {e:?}"); + return Err(Error::Fatal(format!( + "committed membership change at index {index} could not be applied: {e:?}" + ))); + } + ctx.membership_ref().notify_config_applied(index); + // Must come AFTER the membership is applied: the leader refreshes its cluster metadata. + if let Err(e) = internal_event_tx.send(InternalEvent::MembershipApplied) { + warn!("Failed to send MembershipApplied event: {e:?}"); + } + + if is_self_removal { + warn!( + "[{my_id}] Node removed from cluster membership, triggering step down (index {index})" + ); + // Not fatal: the removal is already committed. + if let Err(e) = internal_event_tx.send(InternalEvent::StepDownSelfRemoved) { + error!("[{my_id}] Failed to send StepDownSelfRemoved event: {e:?}"); + } + } + + Ok(()) +} + /// Attempts to execute whatever purge target is currently pending, if any. Shared by both /// purge paths (local snapshot creation and installed-snapshot cleanup) — on success sends /// `LogPurgeCompleted` (cleared by the role's `handle_log_purge_completed`); on failure the @@ -1244,10 +1377,10 @@ pub(super) async fn schedule_and_execute_purge( } /// Cleanup after an InstallSnapshotChunk install (or a confirmed no-op at/beyond -/// `target`). Shares `pending_purge_upto` with `schedule_and_execute_purge`'s watermark — failure isn't cleared, -/// retried by whichever purge trigger runs next. No `retained_log_entries` subtraction: -/// entries below `target` are redundant with a snapshot we already have, not held back for -/// a lagging peer. +/// `target`). Shares `pending_purge_upto` with `schedule_and_execute_purge`'s watermark — failure +/// isn't cleared, retried by whichever purge trigger runs next. No `retained_log_entries` +/// subtraction: entries below `target` are redundant with a snapshot we already have, not held back +/// for a lagging peer. pub(super) async fn schedule_installed_snapshot_purge( target: LogId, ctx: &RaftContext, diff --git a/d-engine-core/src/raft_role/role_state_test.rs b/d-engine-core/src/raft_role/role_state_test.rs index da342bed..a8cbc6cb 100644 --- a/d-engine-core/src/raft_role/role_state_test.rs +++ b/d-engine-core/src/raft_role/role_state_test.rs @@ -1,8 +1,19 @@ //! Tests for role_state free functions //! //! Covers: -//! - schedule_and_execute_purge: two-phase log purge orchestration (Phase 1 = schedule, Phase 2 = execute) +//! - schedule_and_execute_purge: two-phase log purge orchestration (Phase 1 = schedule, Phase 2 = +//! execute) +use std::time::Duration; + +use d_engine_proto::common::LogId; +use d_engine_proto::server::election::VotedFor; +use d_engine_proto::server::storage::SnapshotMetadata; +use tokio::sync::mpsc; +use tokio::sync::watch; + +use super::role_state::RaftRoleState; +use super::role_state::schedule_and_execute_purge; use crate::CapturedLocalSnapshot; use crate::ConsensusError; use crate::Error; @@ -16,16 +27,10 @@ use crate::StateMachineCommand; use crate::StateMachineCommandSender; use crate::raft_role::candidate_state::CandidateState; use crate::raft_role::follower_state::FollowerState; -use crate::test_utils::mock::{MockBuilder, MockTypeConfig, mock_raft_log}; +use crate::test_utils::mock::MockBuilder; +use crate::test_utils::mock::MockTypeConfig; +use crate::test_utils::mock::mock_raft_log; use crate::test_utils::node_config; -use d_engine_proto::common::LogId; -use d_engine_proto::server::election::VotedFor; -use d_engine_proto::server::storage::SnapshotMetadata; -use std::time::Duration; -use tokio::sync::{mpsc, watch}; - -use super::role_state::RaftRoleState; -use super::role_state::schedule_and_execute_purge; // ============================================================================ // schedule_and_execute_purge Tests @@ -741,9 +746,6 @@ async fn test_handle_create_snapshot_failure_emits_error_event() { // ============================================================================ // A vote belongs to the term it was cast in (Raft §5.1: a new term starts with votedFor = null) // -// raft-rs `reset(term)` clears the vote whenever the term changes; openraft replaces the whole -// (term, node) vote. If an old vote survives a term change, it blocks a legitimate vote request -// in the new term even though this node has not voted in that term. // ============================================================================ fn recording_raft_log() -> ( @@ -981,3 +983,150 @@ async fn test_adopted_term_survives_a_restart_even_when_the_vote_was_denied() { FollowerState::::new(1, context.node_config.clone(), Some(persisted), None); assert_eq!(restarted.current_term(), 4); } + +/// Lightweight unit test verifying `is_self_removal_config()` correctly identifies when a node is +/// removing itself from the cluster. +/// +/// Related: Issue #200 +#[test] +fn test_is_self_removal_config() { + use d_engine_proto::common::AddNode; + use d_engine_proto::common::MembershipChange; + use d_engine_proto::common::RemoveNode; + use d_engine_proto::common::membership_change::Change; + + use super::role_state::is_self_removal_config; + + let self_removal = MembershipChange { + change: Some(Change::RemoveNode(RemoveNode { node_id: 1 })), + }; + assert!( + is_self_removal_config(1, &self_removal), + "Should detect self-removal when my_id == removed node_id" + ); + + let other_removal = MembershipChange { + change: Some(Change::RemoveNode(RemoveNode { node_id: 2 })), + }; + assert!( + !is_self_removal_config(1, &other_removal), + "Should NOT detect self-removal when removing different node" + ); + + let add_node = MembershipChange { + change: Some(Change::AddNode(AddNode { + node_id: 1, + address: "127.0.0.1:8080".to_string(), + status: 1, + })), + }; + assert!( + !is_self_removal_config(1, &add_node), + "Should NOT detect self-removal for AddNode changes" + ); + + let no_change = MembershipChange { change: None }; + assert!( + !is_self_removal_config(1, &no_change), + "Should NOT detect self-removal when change is None" + ); +} + +// ── split_into_batches: how committed entries are grouped into worker messages ────────────── + +mod split_into_batches_tests { + use bytes::Bytes; + use d_engine_proto::common::AddNode; + use d_engine_proto::common::Entry; + use d_engine_proto::common::EntryPayload; + use d_engine_proto::common::NodeStatus; + use d_engine_proto::common::membership_change::Change; + + use super::super::role_state::split_into_batches; + + fn command(index: u64) -> Entry { + Entry { + index, + term: 1, + payload: Some(EntryPayload::command(Bytes::from(format!("cmd{index}")))), + } + } + + fn config(index: u64) -> Entry { + Entry { + index, + term: 1, + payload: Some(EntryPayload::config(Change::AddNode(AddNode { + node_id: 9, + address: "addr".into(), + status: NodeStatus::Promotable as i32, + }))), + } + } + + fn noop(index: u64) -> Entry { + Entry { + index, + term: 1, + payload: Some(EntryPayload::noop()), + } + } + + fn indices(batches: &[Vec]) -> Vec> { + batches.iter().map(|b| b.iter().map(|e| e.index).collect()).collect() + } + + #[test] + fn test_split_into_batches_with_only_commands_returns_one_batch() { + let batches = split_into_batches(vec![command(1), command(2), command(3)]); + assert_eq!(indices(&batches), vec![vec![1, 2, 3]]); + } + + /// The config entry is sent with the commands before it, and the commands after it start + /// a new batch: membership is applied between the two messages. + #[test] + fn test_split_into_batches_ends_a_batch_at_a_config_entry() { + let batches = split_into_batches(vec![command(1), command(2), config(3), command(4)]); + assert_eq!(indices(&batches), vec![vec![1, 2, 3], vec![4]]); + } + + #[test] + fn test_split_into_batches_gives_consecutive_configs_a_batch_each() { + let batches = split_into_batches(vec![config(1), config(2)]); + assert_eq!(indices(&batches), vec![vec![1], vec![2]]); + } + + #[test] + fn test_split_into_batches_with_config_last_has_no_trailing_batch() { + let batches = split_into_batches(vec![command(1), config(2)]); + assert_eq!(indices(&batches), vec![vec![1, 2]]); + } + + #[test] + fn test_split_into_batches_ends_a_batch_at_a_noop_entry() { + let batches = split_into_batches(vec![command(1), noop(2), command(3)]); + assert_eq!(indices(&batches), vec![vec![1, 2], vec![3]]); + } + + #[test] + fn test_split_into_batches_of_nothing_is_empty() { + assert!(split_into_batches(vec![]).is_empty()); + } + + /// Whatever the grouping, the entries come out in exactly the order they went in. + #[test] + fn test_split_into_batches_keeps_log_order() { + let input = vec![ + command(1), + noop(2), + config(3), + config(4), + command(5), + command(6), + noop(7), + ]; + let flattened: Vec = + split_into_batches(input.clone()).iter().flatten().map(|e| e.index).collect(); + assert_eq!(flattened, input.iter().map(|e| e.index).collect::>()); + } +} diff --git a/d-engine-core/src/raft_test/commit_path_tests.rs b/d-engine-core/src/raft_test/commit_path_tests.rs new file mode 100644 index 00000000..2714e3f7 --- /dev/null +++ b/d-engine-core/src/raft_test/commit_path_tests.rs @@ -0,0 +1,1593 @@ +//! Commit-path delivery tests, retargeted from `commit_handler/default_commit_handler_test.rs` +//! after merging the commit handler into the Raft main loop (#455). +//! +//! Committed entries are handed to the state machine worker by `Raft::process_committed()`, the +//! last stage of each loop iteration (`RaftRoleState::dispatch_committed`), driven by the role's +//! `commit_index` and the `delivered_index` cursor next to it. These tests keep every assertion +//! from the commit-handler tests; only the subject (the Raft loop instead of the commit handler +//! task) and the feed (advancing `commit_index` instead of sending a commit notification) changed. +//! +//! Two rigs: +//! - `Rig` advances `commit_index` and calls `Raft::process_committed()` directly, so a test can +//! inspect the internal event channel (`MembershipApplied`, `StepDownSelfRemoved`) and the +//! state-machine apply channel without the loop consuming those events first. +//! - `ShutdownRig` spawns `Raft::run()` to exercise the real shutdown hand-over. + +use std::ops::RangeInclusive; +use std::path::PathBuf; +use std::sync::Arc; +use std::sync::atomic::AtomicU64; +use std::sync::atomic::AtomicUsize; +use std::sync::atomic::Ordering; +use std::time::Duration; + +use bytes::Bytes; +use d_engine_proto::common::Entry; +use d_engine_proto::common::EntryPayload; +use d_engine_proto::common::membership_change::Change; +use tokio::sync::mpsc; +use tokio::sync::watch; +use tokio::task::JoinHandle; +use tokio::time; + +use crate::Error; +use crate::InternalEvent; +use crate::MockMembership; +use crate::MockRaftLog; +use crate::MockStateMachineHandler; +use crate::MockTypeConfig; +use crate::Raft; +use crate::Result; +use crate::StateMachineCommand; +use crate::StateMachineHandler; +use crate::test_utils::MockBuilder; +use crate::test_utils::snapshot_config; + +const TEST_TERM: u64 = 1; + +enum CommandType { + Command(Bytes), + Configuration(Change), + Noop, +} + +fn build_entries( + commands: Vec, + term: u64, +) -> Vec { + let mut r = vec![]; + let mut index = 1; + for c in commands { + let entry = match c { + CommandType::Command(data) => Entry { + index, + term, + payload: Some(EntryPayload::command(data)), + }, + CommandType::Configuration(change) => Entry { + index, + term, + payload: Some(EntryPayload::config(change)), + }, + CommandType::Noop => Entry { + index, + term, + payload: Some(EntryPayload::noop()), + }, + }; + r.push(entry); + index += 1; + } + r +} + +// ------------------------------------------------------------------------------------------------ +// exactly-once delivery rig +// ------------------------------------------------------------------------------------------------ + +/// Same arithmetic as `DefaultStateMachineHandler::{pending_range, update_pending}`. +struct Progress { + /// Highest index the state machine has FINISHED applying. + applied: AtomicU64, + /// Highest commit index announced so far (only ever raised). + pending: AtomicU64, +} + +impl Progress { + fn new(initial_applied: u64) -> Self { + Self { + applied: AtomicU64::new(initial_applied), + pending: AtomicU64::new(0), + } + } + fn range(&self) -> Option> { + let applied = self.applied.load(Ordering::Acquire); + let pending = self.pending.load(Ordering::Acquire); + (pending > applied).then(|| (applied + 1)..=pending) + } + fn raise_pending( + &self, + new_commit: u64, + ) { + self.pending.fetch_max(new_commit, Ordering::AcqRel); + } +} + +/// `LOG_LEN` plain commands, optionally with `config_at` replaced by an AddNode change. +fn log_entries(config_at: Option) -> Vec { + let commands = (1..=200u64) + .map(|i| { + if Some(i) == config_at { + CommandType::Configuration(Change::AddNode(d_engine_proto::common::AddNode { + node_id: 9, + address: "addr".into(), + status: d_engine_proto::common::NodeStatus::Promotable as i32, + })) + } else { + CommandType::Command(Bytes::from(format!("cmd{i}"))) + } + }) + .collect(); + build_entries(commands, TEST_TERM) +} + +/// Mocks for the commit path: a state-machine handler whose `pending_range` mirrors `Progress`, a +/// raft log that serves `get_entries_range`, and a membership whose first config apply optionally +/// fails (to exercise the fatal path). +struct Parts { + smh: MockStateMachineHandler, + log: MockRaftLog, + membership: MockMembership, + progress: Arc, + config_applied: Arc, + /// How many times `RaftLog::close` was called. + log_closed: Arc, +} + +fn build_parts( + entries: Vec, + initial_applied: u64, + first_config_apply_fails: bool, +) -> Parts { + let progress = Arc::new(Progress::new(initial_applied)); + + let mut smh = MockStateMachineHandler::new(); + let p = progress.clone(); + smh.expect_pending_range().returning(move || p.range()); + let p = progress.clone(); + smh.expect_update_pending().returning(move |n| p.raise_pending(n)); + let p = progress.clone(); + smh.expect_last_applied().returning(move || p.applied.load(Ordering::Acquire)); + + let mut log = MockRaftLog::new(); + log.expect_load_hard_state().returning(|| Ok(None)); + log.expect_save_hard_state().returning(|_| Ok(())); + log.expect_get_entries_range().returning(move |range| { + Ok(entries.iter().filter(|e| range.contains(&e.index)).cloned().collect()) + }); + // `close` is only reached when `Raft::run()` ends; the direct-drive rig never calls it. + let log_closed = Arc::new(AtomicUsize::new(0)); + let closes = log_closed.clone(); + log.expect_close().returning(move || { + closes.fetch_add(1, Ordering::SeqCst); + }); + + let config_applied = Arc::new(AtomicUsize::new(0)); + let calls = config_applied.clone(); + let mut membership = MockMembership::new(); + membership.expect_notify_config_applied().returning(|_| {}); + membership.expect_apply_config_change().returning(move |_| { + let n = calls.fetch_add(1, Ordering::SeqCst) + 1; + if first_config_apply_fails && n == 1 { + Err(Error::Fatal( + "config change could not be applied".to_string(), + )) + } else { + Ok(()) + } + }); + + Parts { + smh, + log, + membership, + progress, + config_applied, + log_closed, + } +} + +/// Drives the commit path synchronously through `Raft::handle_internal_event`, leaving the +/// internal event channel readable so a test can assert on `MembershipApplied`/`FatalError`. +struct Rig { + raft: Raft, + progress: Arc, + /// One element per `ApplyEntries` batch the (fake) state machine worker received. + delivered: Arc>>>, + config_applied: Arc, +} + +impl Rig { + fn start( + entries: Vec, + initial_applied: u64, + first_config_apply_fails: bool, + ) -> Self { + Self::build(entries, initial_applied, first_config_apply_fails, false) + } + + /// Same as `start`, but the state-machine worker's receiving end is dropped so that + /// `send_to_sm_worker` fails — simulates the worker task having already exited. + fn start_with_closed_worker( + entries: Vec, + initial_applied: u64, + first_config_apply_fails: bool, + ) -> Self { + Self::build(entries, initial_applied, first_config_apply_fails, true) + } + + fn build( + entries: Vec, + initial_applied: u64, + first_config_apply_fails: bool, + closed_worker: bool, + ) -> Self { + let parts = build_parts(entries, initial_applied, first_config_apply_fails); + + let (sm_tx, sm_rx) = mpsc::unbounded_channel::(); + let delivered = Arc::new(std::sync::Mutex::new(Vec::new())); + if closed_worker { + drop(sm_rx); + } else { + let sink = delivered.clone(); + tokio::spawn(async move { + let mut sm_rx = sm_rx; + while let Some(cmd) = sm_rx.recv().await { + if let StateMachineCommand::ApplyEntries { entries, .. } = cmd { + sink.lock().unwrap().push(entries.iter().map(|e| e.index).collect()); + } + } + }); + } + + let (_shutdown_tx, shutdown_rx) = watch::channel(()); + let raft = MockBuilder::new(shutdown_rx) + .with_state_machine_handler(parts.smh) + .with_raft_log(parts.log) + .with_membership(parts.membership) + .with_state_machine_commands(crate::StateMachineCommandSender::new(sm_tx)) + .turn_on_election(false) + .build_raft(); + + Self { + raft, + progress: parts.progress, + delivered, + config_applied: parts.config_applied, + } + } + + /// Advance the role's `commit_index`, exactly as the role layer does, then run the stage of + /// the loop iteration that hands newly committed entries to the state machine worker. + async fn notify( + &mut self, + commit_index: u64, + ) -> Result<()> { + self.raft.role.state_mut().update_commit_index(commit_index)?; + self.raft.process_committed().await + } + + /// Wait until nothing more is being delivered. + async fn settle(&self) { + let mut stable = 0; + let mut last = self.delivered_flat().len(); + for _ in 0..200 { + tokio::time::sleep(Duration::from_millis(10)).await; + let now = self.delivered_flat().len(); + stable = if now == last { stable + 1 } else { 0 }; + last = now; + if stable >= 3 { + return; + } + } + } + + /// The state machine finished applying everything up to `index`. + fn complete_apply( + &self, + index: u64, + ) { + self.progress.applied.fetch_max(index, Ordering::AcqRel); + } + + /// A snapshot install moved the state machine to `index` without going through entries. + fn jump_applied_to( + &self, + index: u64, + ) { + self.progress.applied.store(index, Ordering::Release); + } + + fn delivered_flat(&self) -> Vec { + self.delivered.lock().unwrap().iter().flatten().copied().collect() + } + + fn assert_delivered( + &self, + expected: Vec, + why: &str, + ) { + let batches = self.delivered.lock().unwrap().clone(); + assert_eq!( + self.delivered_flat(), + expected, + "{why}\n batches handed to the state machine, in order: {batches:?}" + ); + } + + fn drain_events(&mut self) -> Vec { + let mut out = Vec::new(); + while let Ok(ev) = self.raft.internal_event_rx.try_recv() { + out.push(ev); + } + out + } + + /// Drop the internal-event receiver so sends to `internal_event_tx` fail — exercises the + /// warn-only paths for `MembershipApplied` / `StepDownSelfRemoved`. + fn drop_event_receiver(&mut self) { + let (_dummy_tx, dummy_rx) = mpsc::unbounded_channel::(); + let old_rx = std::mem::replace(&mut self.raft.internal_event_rx, dummy_rx); + drop(old_rx); + } +} + +fn range(r: RangeInclusive) -> Vec { + r.collect() +} + +/// Spawns `Raft::run()` to exercise the real shutdown hand-over; observes only the delivered +/// apply batches (the loop consumes its own internal events). The commit index is advanced before +/// the loop starts, so shutdown can be raced against commits the node already knows about. +struct ShutdownRig { + raft: Option>, + internal_event_tx: mpsc::UnboundedSender, + delivered: Arc>>>, + handle: Option>>, + shutdown_tx: watch::Sender<()>, + log_closed: Arc, + /// Sender side of the inbound event channel, as gRPC stream handlers hold it. + event_tx: mpsc::Sender, + /// Set by `spawn_loop_keeping_raft`: the loop's result together with the `Raft` itself. + kept: Option, Raft)>>, +} + +impl ShutdownRig { + fn start( + entries: Vec, + initial_applied: u64, + first_config_apply_fails: bool, + ) -> Self { + let parts = build_parts(entries, initial_applied, first_config_apply_fails); + + let (sm_tx, sm_rx) = mpsc::unbounded_channel::(); + let delivered = Arc::new(std::sync::Mutex::new(Vec::new())); + let sink = delivered.clone(); + tokio::spawn(async move { + let mut sm_rx = sm_rx; + while let Some(cmd) = sm_rx.recv().await { + if let StateMachineCommand::ApplyEntries { entries, .. } = cmd { + sink.lock().unwrap().push(entries.iter().map(|e| e.index).collect()); + } + } + }); + + let (shutdown_tx, shutdown_rx) = watch::channel(()); + let raft = MockBuilder::new(shutdown_rx) + .with_state_machine_handler(parts.smh) + .with_raft_log(parts.log) + .with_membership(parts.membership) + .with_state_machine_commands(crate::StateMachineCommandSender::new(sm_tx)) + .turn_on_election(false) + .build_raft(); + let internal_event_tx = raft.internal_event_tx.clone(); + let event_tx = raft.event_sender(); + + Self { + raft: Some(raft), + internal_event_tx, + delivered, + handle: None, + shutdown_tx, + log_closed: parts.log_closed, + event_tx, + kept: None, + } + } + + /// The node learns that `commit_index` is committed (before the loop starts). + fn announce_commit( + &mut self, + commit_index: u64, + ) { + self.raft + .as_mut() + .expect("the loop has not started") + .role + .state_mut() + .update_commit_index(commit_index) + .unwrap(); + } + + /// One delivery stage of the loop, before the loop starts. + async fn deliver_now(&mut self) { + self.raft + .as_mut() + .expect("the loop has not started") + .process_committed() + .await + .unwrap(); + } + + /// Like `spawn_loop`, but hands the `Raft` back when `run()` ends. In a real node the `Raft` + /// outlives `run()` (it sits in the `Node`), so its state after `run()` is what matters. + fn spawn_loop_keeping_raft(&mut self) { + let mut raft = self.raft.take().expect("the loop has already started"); + self.kept = Some(tokio::spawn(async move { + let result = raft.run().await; + (result, raft) + })); + } + + async fn end_of_loop_keeping_raft(&mut self) -> (Result<()>, Raft) { + time::timeout( + Duration::from_secs(2), + self.kept.as_mut().expect("loop not started"), + ) + .await + .expect("run() must end") + .expect("the commit path must not panic") + } + + fn spawn_loop(&mut self) { + let mut raft = self.raft.take().expect("the loop has already started"); + self.handle = Some(tokio::spawn(async move { raft.run().await })); + } + + /// Wait for `run()` to exit (accepting either Ok or Err) and for delivery to settle. + async fn join_and_settle(&mut self) { + let handle = self.handle.as_mut().expect("the loop has not started"); + let _ = time::timeout(Duration::from_secs(2), handle) + .await + .expect("the commit path must exit after shutdown") + .expect("the commit path must not panic"); + self.settle().await; + } + + async fn settle(&self) { + let mut stable = 0; + let mut last = self.delivered_flat().len(); + for _ in 0..200 { + tokio::time::sleep(Duration::from_millis(10)).await; + let now = self.delivered_flat().len(); + stable = if now == last { stable + 1 } else { 0 }; + last = now; + if stable >= 3 { + return; + } + } + } + + fn delivered_flat(&self) -> Vec { + self.delivered.lock().unwrap().iter().flatten().copied().collect() + } + + fn assert_delivered( + &self, + expected: Vec, + why: &str, + ) { + let batches = self.delivered.lock().unwrap().clone(); + assert_eq!( + self.delivered_flat(), + expected, + "{why}\n batches handed to the state machine, in order: {batches:?}" + ); + } +} + +// ================================================================================================ +// Exactly-once delivery contract (#455) +// ================================================================================================ +// +// CONTRACT: every committed log entry is handed to the state machine worker EXACTLY ONCE, in log +// order, with no gaps — no matter how the commit notifications arrive (back to back, repeated, +// stale) or how far behind the state machine is. + +/// A second commit notification arrives while the state machine has not applied the first batch +/// yet (the normal case under load). Entries 1..=5 must not be delivered again. +#[tokio::test] +async fn test_commit_notifications_while_apply_is_pending_deliver_each_entry_once() { + let mut rig = Rig::start(log_entries(None), 0, false); + rig.notify(5).await.unwrap(); + rig.settle().await; + rig.notify(10).await.unwrap(); // state machine still has not finished anything + rig.settle().await; + rig.notify(15).await.unwrap(); + rig.settle().await; + rig.assert_delivered( + range(1..=15), + "each committed entry must be handed to the state machine exactly once", + ); +} + +/// The commit index advances several times before the delivery stage runs (e.g. several +/// AppendResults in one loop iteration). Only the latest value matters: one delivery covers the +/// whole range, as a single batch, with nothing queued in between. +#[tokio::test] +async fn test_several_commit_advances_before_one_delivery_are_delivered_as_one_batch() { + let mut rig = Rig::start(log_entries(None), 0, false); + for commit_index in [3, 5, 8, 12] { + rig.raft.role.state_mut().update_commit_index(commit_index).unwrap(); + } + rig.raft.process_committed().await.unwrap(); + rig.settle().await; + + let batches = rig.delivered.lock().unwrap().clone(); + assert_eq!( + batches, + vec![range(1..=12)], + "one delivery must cover everything up to the latest commit index, as a single batch" + ); +} + +/// The same notification repeated (leader re-announces the same commit index, e.g. on every ack) +/// must not make the commit path re-read and re-send anything. +#[tokio::test] +async fn test_repeated_identical_notification_does_not_redeliver() { + let mut rig = Rig::start(log_entries(None), 0, false); + for _ in 0..3 { + rig.notify(8).await.unwrap(); + rig.settle().await; + } + rig.assert_delivered( + range(1..=8), + "a repeated commit index must deliver nothing new", + ); +} + +/// A stale (lower) notification arriving after a higher one must be ignored. +#[tokio::test] +async fn test_stale_lower_notification_does_not_redeliver() { + let mut rig = Rig::start(log_entries(None), 0, false); + rig.notify(10).await.unwrap(); + rig.settle().await; + rig.notify(4).await.unwrap(); + rig.settle().await; + rig.assert_delivered(range(1..=10), "a lower commit index must deliver nothing"); +} + +/// 100 single-step notifications in a row while nothing is ever reported as applied: the state +/// machine must still see 1..=100 once each, contiguous and ordered. +#[tokio::test] +async fn test_burst_of_single_step_notifications_delivers_each_entry_once() { + let mut rig = Rig::start(log_entries(None), 0, false); + for i in 1..=100 { + rig.notify(i).await.unwrap(); + } + rig.settle().await; + rig.assert_delivered( + range(1..=100), + "a burst of notifications must neither duplicate nor skip entries", + ); +} + +/// Guard: when the state machine reports progress between notifications, only the NEW entries are +/// delivered. +#[tokio::test] +async fn test_apply_progress_between_notifications_delivers_only_new_entries() { + let mut rig = Rig::start(log_entries(None), 0, false); + rig.notify(5).await.unwrap(); + rig.settle().await; + rig.complete_apply(5); + rig.notify(9).await.unwrap(); + rig.settle().await; + rig.assert_delivered( + range(1..=9), + "entries 1..=5 were applied; only 6..=9 are new", + ); +} + +/// Guard: announcing a commit index the state machine already reached delivers nothing. +#[tokio::test] +async fn test_notification_for_already_applied_index_delivers_nothing() { + let mut rig = Rig::start(log_entries(None), 0, false); + rig.notify(5).await.unwrap(); + rig.settle().await; + rig.complete_apply(5); + rig.notify(5).await.unwrap(); + rig.notify(3).await.unwrap(); + rig.settle().await; + rig.assert_delivered(range(1..=5), "nothing above index 5 was committed"); +} + +/// Guard: after a restart the state machine is already at index 20. Entries up to 20 are never +/// delivered, whatever commit index is announced first. +#[tokio::test] +async fn test_entries_at_or_below_initial_applied_index_are_never_delivered() { + let mut rig = Rig::start(log_entries(None), 20, false); + rig.notify(10).await.unwrap(); + rig.settle().await; + rig.assert_delivered( + vec![], + "commit index 10 is below what is already applied (20)", + ); + rig.notify(25).await.unwrap(); + rig.settle().await; + rig.assert_delivered(range(21..=25), "only 21..=25 are new"); +} + +/// Guard: a snapshot install moves the state machine past entries it never saw. Delivery resumes +/// after the snapshot point and never goes back for the covered entries. +#[tokio::test] +async fn test_snapshot_install_jump_skips_covered_entries() { + let mut rig = Rig::start(log_entries(None), 0, false); + rig.notify(10).await.unwrap(); + rig.settle().await; + rig.complete_apply(10); + rig.notify(30).await.unwrap(); + rig.settle().await; // 11..=30 delivered, not yet reported applied + rig.jump_applied_to(50); // snapshot at index 50 installed + rig.notify(60).await.unwrap(); + rig.settle().await; + let mut expected = range(1..=30); + expected.extend(range(51..=60)); + rig.assert_delivered(expected, "after the snapshot at 50 only 51..=60 are new"); +} + +/// A committed membership change is applied to the membership exactly once, even when the commit +/// path is notified again before the state machine reports the config entry applied. +#[tokio::test] +async fn test_config_change_is_applied_exactly_once_when_commits_are_announced_again() { + let mut rig = Rig::start(log_entries(Some(3)), 0, false); + rig.notify(3).await.unwrap(); // contains the AddNode at index 3 + rig.settle().await; + rig.notify(5).await.unwrap(); // state machine has not reported anything applied + rig.settle().await; + assert_eq!( + rig.config_applied.load(Ordering::SeqCst), + 1, + "the AddNode at index 3 must change the membership once" + ); + let applied_events = rig + .drain_events() + .into_iter() + .filter(|e| matches!(e, InternalEvent::MembershipApplied)) + .count(); + assert_eq!( + applied_events, 1, + "one committed change => one MembershipApplied event" + ); + rig.assert_delivered(range(1..=5), "entries are still delivered exactly once"); +} + +/// A committed membership change that cannot be applied is FATAL for the node (owner decision, +/// #455): the cluster has already agreed on it, so a node that skips it holds a membership view +/// that no longer matches its own log, and nothing can repair that. The commit path returns a +/// fatal error (which stops the loop), and no `MembershipApplied` is announced for a change that +/// failed (#209). +#[tokio::test] +async fn test_failed_config_change_raises_fatal_error_and_announces_nothing() { + let mut rig = Rig::start(log_entries(Some(3)), 0, true); + let result = rig.notify(3).await; // the AddNode at index 3 fails to apply + rig.settle().await; + rig.complete_apply(3); + + assert!( + matches!(result, Err(ref e) if e.is_fatal()), + "a committed membership change that cannot be applied must be fatal" + ); + let events = rig.drain_events(); + assert!( + !events.iter().any(|e| matches!(e, InternalEvent::MembershipApplied)), + "MembershipApplied must not be announced for a change that failed" + ); +} + +/// An entry whose payload kind is missing (corrupt / hostile peer) must never panic the commit +/// path (no panics in embedded core paths). Later entries are still delivered. +#[tokio::test] +async fn test_entry_with_missing_payload_kind_does_not_panic_the_commit_path() { + let mut entries = log_entries(None); + entries[1].payload = Some(EntryPayload { payload: None }); // index 2: oneof not set + let mut rig = Rig::start(entries, 0, false); + let result = rig.notify(3).await; + rig.settle().await; + + assert!( + result.is_ok(), + "the commit path must not return an error (panic) on an entry with an empty payload kind" + ); + let fatal = rig.drain_events().iter().any(|e| matches!(e, InternalEvent::FatalError { .. })); + assert!( + fatal || rig.delivered_flat().contains(&3), + "after the malformed entry, later entries were not delivered and no FatalError was raised \ + (delivered: {:?})", + rig.delivered_flat() + ); +} + +/// An entry with no payload at all (not just an unset payload kind) is skipped the same way: +/// no panic, no error, and the entries around it are still delivered. +#[tokio::test] +async fn test_entry_without_any_payload_is_skipped_and_later_entries_are_delivered() { + let mut entries = log_entries(None); + entries[1].payload = None; // index 2: no payload at all + let mut rig = Rig::start(entries, 0, false); + let result = rig.notify(3).await; + rig.settle().await; + + assert!( + result.is_ok(), + "an entry with no payload must not fail the commit path" + ); + rig.assert_delivered( + vec![1, 3], + "the malformed entry is skipped, its neighbours delivered", + ); +} + +// ------------------------------------------------------------------------------------------------ +// graceful shutdown +// ------------------------------------------------------------------------------------------------ + +/// CONTRACT (owner decision, #455): on graceful shutdown, every commit the node already knows +/// about is handed to the state machine worker before the commit path exits. 30 rounds make a +/// race between shutdown and the loop fail reliably rather than by luck. +#[tokio::test] +async fn test_graceful_shutdown_hands_all_announced_commits_to_the_state_machine() { + for round in 0..30 { + let mut rig = ShutdownRig::start(log_entries(None), 0, false); + // The commit and the shutdown are both known before the loop gets to run at all. + rig.announce_commit(10); + rig.spawn_loop(); + rig.shutdown_tx.send(()).unwrap(); + + rig.join_and_settle().await; + rig.assert_delivered( + range(1..=10), + &format!("round {round}: shutdown dropped an announced commit"), + ); + } +} + +/// Runs the real `Raft::run()` and the real `StateMachineWorker`, with commit 10 announced before +/// the loop starts, and returns the indices the state machine applied. +/// +/// It shuts down the way `Node::run` does: signal the Raft loop, wait until it has ended, and only +/// then put `Shutdown` into the worker's queue. The runtime is single-threaded, so spawn order is +/// poll order and both starting orders are reproduced exactly. +/// +/// Why this exists: the tests above stand in a fake worker, so they cannot see whether the loop's +/// final hand-over really reaches a worker and is applied before it stops. +async fn applied_after_shutdown_with_real_worker(worker_first: bool) -> Vec { + let parts = build_parts(log_entries(None), 0, false); + + let applied = Arc::new(std::sync::Mutex::new(Vec::new())); + let sink = applied.clone(); + let mut writer = crate::MockStateMachineWriterOps::new(); + writer.expect_apply_chunk().returning(move |entries| { + sink.lock().unwrap().extend(entries.iter().map(|e| e.index)); + Ok(entries.iter().map(|e| crate::ApplyResult::success(e.index)).collect()) + }); + // The worker closes the state machine as its last act. + writer.expect_close().returning(|| Ok(())); + + let (sm_tx, sm_rx) = mpsc::unbounded_channel::(); + // The worker reports apply results here; the receiver must stay open or the worker fails. + let (worker_events_tx, _worker_events_rx) = mpsc::unbounded_channel::(); + let (shutdown_tx, shutdown_rx) = watch::channel(()); + let worker = crate::StateMachineWorker::::new( + 1, + Arc::new(writer), + sm_tx.clone(), + sm_rx, + worker_events_tx, + ); + let owner_tx = sm_tx.clone(); + let mut raft = MockBuilder::new(shutdown_rx) + .with_state_machine_handler(parts.smh) + .with_raft_log(parts.log) + .with_membership(parts.membership) + .with_state_machine_commands(crate::StateMachineCommandSender::new(sm_tx)) + .turn_on_election(false) + .build_raft(); + raft.role.state_mut().update_commit_index(10).unwrap(); + + let (worker_handle, raft_handle); + if worker_first { + worker_handle = tokio::spawn(worker.run()); + raft_handle = tokio::spawn(async move { raft.run().await }); + } else { + raft_handle = tokio::spawn(async move { raft.run().await }); + worker_handle = tokio::spawn(worker.run()); + } + shutdown_tx.send(()).unwrap(); + + let _ = time::timeout(Duration::from_secs(2), raft_handle) + .await + .expect("the raft loop must exit after shutdown") + .expect("the raft loop must not panic"); + // The loop has ended, so everything it will ever hand over is already queued. + owner_tx.send(StateMachineCommand::Shutdown).unwrap(); + let _ = time::timeout(Duration::from_secs(2), worker_handle) + .await + .expect("the state machine worker must exit after shutdown") + .expect("the state machine worker must not panic"); + applied.lock().unwrap().clone() +} + +/// CONTRACT (owner decision, #455): graceful shutdown must not finish with a commit the node +/// already knew about still unapplied in the state machine. Here the worker is spawned first and +/// is already waiting on its queue when the loop makes its final hand-over. +#[tokio::test] +async fn test_graceful_shutdown_applies_announced_commits_when_worker_is_spawned_first() { + assert_eq!( + applied_after_shutdown_with_real_worker(true).await, + range(1..=10), + "announced commits were not applied after a graceful shutdown" + ); +} + +/// Same contract, other starting order: the loop is spawned first, so its final hand-over is +/// queued before the worker has run at all. +#[tokio::test] +async fn test_graceful_shutdown_applies_announced_commits_when_loop_is_spawned_first() { + assert_eq!( + applied_after_shutdown_with_real_worker(false).await, + range(1..=10), + "announced commits were not applied after a graceful shutdown" + ); +} + +/// The commit advanced in several steps and only the first steps were delivered before shutdown: +/// shutdown delivers the rest, and every entry is delivered exactly once. +#[tokio::test] +async fn test_graceful_shutdown_with_partly_delivered_commits_delivers_everything_once() { + for round in 0..30 { + let mut rig = ShutdownRig::start(log_entries(None), 0, false); + rig.announce_commit(2); + rig.deliver_now().await; + rig.announce_commit(6); + rig.deliver_now().await; + rig.announce_commit(10); // known, not yet delivered + rig.spawn_loop(); + rig.shutdown_tx.send(()).unwrap(); + + rig.join_and_settle().await; + rig.assert_delivered( + range(1..=10), + &format!("round {round}: commits were lost or duplicated at shutdown"), + ); + } +} + +/// Guard: shutdown with nothing pending exits promptly and delivers nothing. +#[tokio::test] +async fn test_graceful_shutdown_with_no_pending_commit_exits_cleanly() { + let mut rig = ShutdownRig::start(log_entries(None), 0, false); + rig.spawn_loop(); + rig.shutdown_tx.send(()).unwrap(); + rig.join_and_settle().await; + rig.assert_delivered(vec![], "nothing was committed"); +} + +/// A failed membership change is fatal and stops the loop; a shutdown that follows must not +/// deliver the same entries a second time. +#[tokio::test] +async fn test_shutdown_after_failed_config_change_does_not_redeliver_entries() { + let mut rig = ShutdownRig::start(log_entries(Some(3)), 0, true); + rig.announce_commit(5); // AddNode at index 3 fails to apply + rig.spawn_loop(); + // Wake the loop so its delivery stage runs and hits the failing config change. + rig.internal_event_tx.send(InternalEvent::LeaderDiscovered(1, 1)).unwrap(); + rig.settle().await; + // The loop has already stopped after the fatal error: nobody listens then. + let _ = rig.shutdown_tx.send(()); + rig.join_and_settle().await; + + let flat = rig.delivered_flat(); + let mut unique = flat.clone(); + unique.sort_unstable(); + unique.dedup(); + assert_eq!( + flat.len(), + unique.len(), + "entries delivered twice around fatal shutdown: {:?}", + rig.delivered.lock().unwrap().clone() + ); +} + +/// Business scenario: the operator stops a node. `Raft` is the only writer of its log, so before +/// `run()` returns it must have waited for in-flight log writes (`RaftLog::close`), exactly once. +#[tokio::test] +async fn test_raft_log_is_closed_once_when_run_ends_by_shutdown() { + let mut rig = ShutdownRig::start(log_entries(None), 0, false); + rig.spawn_loop(); + rig.shutdown_tx.send(()).unwrap(); + rig.join_and_settle().await; + + assert_eq!( + rig.log_closed.load(Ordering::SeqCst), + 1, + "the log must be closed exactly once when run() ends by shutdown" + ); +} + +/// Business scenario: a committed membership change cannot be applied, which is fatal and ends +/// `run()` with an error; nobody sent a shutdown. The log's in-flight writes must still be waited +/// for before `run()` returns: closing is not something only a clean shutdown gets to do. +#[tokio::test] +async fn test_raft_log_is_closed_when_run_ends_by_fatal_error() { + let mut rig = ShutdownRig::start(log_entries(Some(3)), 0, true); + rig.announce_commit(5); // AddNode at index 3 fails to apply + rig.spawn_loop(); + // Wake the loop so its delivery stage runs and hits the failing config change. + rig.internal_event_tx.send(InternalEvent::LeaderDiscovered(1, 1)).unwrap(); + + let result = time::timeout(Duration::from_secs(2), rig.handle.as_mut().unwrap()) + .await + .expect("the fatal error must end run()") + .expect("the commit path must not panic"); + assert!( + result.is_err(), + "the failed config change must end run() with an error" + ); + + assert_eq!( + rig.log_closed.load(Ordering::SeqCst), + 1, + "the log must be closed even though run() ended by error, not by shutdown" + ); +} + +/// Business scenario: the operator stops a node while gRPC stream handlers are feeding it events. +/// Once `run()` has ended nobody consumes events, so the channel must be closed: a handler that +/// is blocked sending, or sends later, then gets an error instead of waiting forever (which would +/// keep `serve_with_shutdown`, and with it the whole node, from ever finishing). +#[tokio::test] +async fn test_event_channel_is_closed_when_run_ends_by_shutdown() { + let mut rig = ShutdownRig::start(log_entries(None), 0, false); + rig.spawn_loop_keeping_raft(); + rig.shutdown_tx.send(()).unwrap(); + let (result, _raft) = rig.end_of_loop_keeping_raft().await; + + assert!(result.is_ok()); + assert!( + rig.event_tx.is_closed(), + "the event channel must be closed once run() ended" + ); +} + +/// Business scenario: same, but `run()` ends because a committed membership change could not be +/// applied (fatal). The stream handlers need the same release; nobody sent a shutdown. +#[tokio::test] +async fn test_event_channel_is_closed_when_run_ends_by_fatal_error() { + let mut rig = ShutdownRig::start(log_entries(Some(3)), 0, true); + rig.announce_commit(5); // AddNode at index 3 fails to apply + rig.spawn_loop_keeping_raft(); + rig.internal_event_tx.send(InternalEvent::LeaderDiscovered(1, 1)).unwrap(); + let (result, _raft) = rig.end_of_loop_keeping_raft().await; + + assert!( + result.is_err(), + "the failed config change must end run() with an error" + ); + assert!( + rig.event_tx.is_closed(), + "the event channel must be closed even though run() ended by error, not by shutdown" + ); +} + +// ------------------------------------------------------------------------------------------------ +// the rig itself +// ------------------------------------------------------------------------------------------------ + +/// The rig's `Progress` must behave exactly like the real handler's, otherwise the tests above +/// would prove nothing about production. Same inputs => same `pending_range()`. +#[test] +fn test_emulated_progress_matches_real_state_machine_handler() { + for initial_applied in [0u64, 5, 40] { + let dir = PathBuf::from(format!("/tmp/test_emulated_progress_{initial_applied}")); + let real = crate::DefaultStateMachineHandler::::new( + 1, + initial_applied, + Arc::new(crate::MockStateMachine::new()), + dir.clone(), + snapshot_config(dir), + crate::MockSnapshotPolicy::new(), + ); + let emulated = Progress::new(initial_applied); + assert_eq!(real.pending_range(), emulated.range(), "before any commit"); + for commit in [3, 7, 7, 2, 12, 12, 50, 49] { + real.update_pending(commit); + emulated.raise_pending(commit); + assert_eq!( + real.pending_range(), + emulated.range(), + "after update_pending({commit}) with last_applied={initial_applied}" + ); + } + } +} + +// ------------------------------------------------------------------------------------------------ +// process_batch behaviour (config ordering, membership events, self-removal, worker unavailable) +// ------------------------------------------------------------------------------------------------ + +/// The happy path with all entry types is delivered without error. +#[tokio::test] +async fn test_full_processing_flow() { + let entries = build_entries( + vec![ + CommandType::Command(Bytes::from(b"cmd1".to_vec())), + CommandType::Configuration(Change::AddNode(d_engine_proto::common::AddNode { + node_id: 1, + address: "addr".into(), + status: d_engine_proto::common::NodeStatus::Promotable as i32, + })), + CommandType::Noop, + CommandType::Command(Bytes::from(b"cmd2".to_vec())), + ], + 1, + ); + + let mut rig = Rig::start(entries, 0, false); + rig.notify(4).await.unwrap(); + rig.settle().await; + + rig.assert_delivered(range(1..=4), "all four committed entries must be delivered"); + assert!( + rig.drain_events().iter().any(|e| matches!(e, InternalEvent::MembershipApplied)), + "a successful membership change must announce MembershipApplied" + ); +} + +/// A config change removing a non-self node applies cleanly. +#[tokio::test] +async fn test_config_remove_node() { + let entries = build_entries( + vec![CommandType::Configuration(Change::RemoveNode( + d_engine_proto::common::RemoveNode { node_id: 2 }, // not self + ))], + 1, + ); + + let mut rig = Rig::start(entries, 0, false); + rig.notify(1).await.unwrap(); + rig.settle().await; + + rig.assert_delivered( + range(1..=1), + "the config entry must reach the state machine", + ); + assert!( + rig.drain_events().iter().any(|e| matches!(e, InternalEvent::MembershipApplied)), + "a successful removal must announce MembershipApplied" + ); +} + +/// Batch processing under load: 1000 plain commands delivered without error. +#[tokio::test] +async fn test_high_throughput_processing() { + let mut commands = Vec::new(); + for i in 1..=1000 { + commands.push(CommandType::Command(Bytes::from(format!("cmd{i}")))); + } + let entries = build_entries(commands, 1); + + let mut rig = Rig::start(entries, 0, false); + rig.notify(1000).await.unwrap(); + rig.settle().await; + + rig.assert_delivered( + range(1..=1000), + "every command must be delivered exactly once", + ); +} + +/// An empty pending range delivers nothing and is not an error. +#[tokio::test] +async fn processes_empty_batch_successfully() { + let mut rig = Rig::start(vec![], 0, false); + let result = rig.notify(1).await; + assert!(result.is_ok()); + rig.assert_delivered(vec![], "an empty batch must deliver nothing"); +} + +/// A closed state-machine worker channel is reported the same way every other Worker-unavailable +/// error is (non-fatal), not as `Error::Fatal` (reserved for conditions requiring node shutdown). +#[tokio::test] +async fn test_process_batch_reports_worker_unavailable_not_fatal_when_channel_closed() { + let entries = build_entries(vec![CommandType::Command(Bytes::from(b"cmd1".to_vec()))], 1); + let mut rig = Rig::start_with_closed_worker(entries, 0, false); + + let err = rig + .notify(1) + .await + .expect_err("process_batch must fail when the SM Worker channel is closed"); + assert!( + !err.is_fatal(), + "a closed SM Worker channel must be reported as non-fatal, not Error::Fatal" + ); +} + +/// Consecutive commands are handed to the state machine as a single batch. +#[tokio::test] +async fn batches_consecutive_commands() { + let entries = build_entries( + vec![ + CommandType::Command(Bytes::from(b"cmd1".to_vec())), + CommandType::Command(Bytes::from(b"cmd2".to_vec())), + CommandType::Command(Bytes::from(b"cmd3".to_vec())), + ], + 1, + ); + + let mut rig = Rig::start(entries, 0, false); + rig.notify(3).await.unwrap(); + rig.settle().await; + rig.assert_delivered(range(1..=3), "three consecutive commands must be delivered"); +} + +/// A config change flushes the preceding command batch before it is applied. +#[tokio::test] +async fn flushes_commands_at_config_change() { + let entries = build_entries( + vec![ + CommandType::Command(Bytes::from(b"cmd1".to_vec())), + CommandType::Configuration(Change::AddNode(d_engine_proto::common::AddNode { + node_id: 1, + address: "addr".into(), + status: d_engine_proto::common::NodeStatus::Promotable as i32, + })), + CommandType::Command(Bytes::from(b"cmd2".to_vec())), + ], + 1, + ); + + let mut rig = Rig::start(entries, 0, false); + rig.notify(3).await.unwrap(); + rig.settle().await; + rig.assert_delivered( + range(1..=3), + "commands around a config change must all be delivered", + ); +} + +/// A noop flushes the preceding command batch. +#[tokio::test] +async fn flushes_commands_at_noop() { + let entries = build_entries( + vec![ + CommandType::Command(Bytes::from(b"cmd1".to_vec())), + CommandType::Noop, + CommandType::Command(Bytes::from(b"cmd2".to_vec())), + ], + 1, + ); + + let mut rig = Rig::start(entries, 0, false); + rig.notify(3).await.unwrap(); + rig.settle().await; + rig.assert_delivered(range(1..=3), "commands around a noop must all be delivered"); +} + +/// A config change that fails to apply makes the commit path return a fatal error. +#[tokio::test] +async fn handles_config_failure_properly() { + let entries = build_entries( + vec![ + CommandType::Configuration(Change::AddNode(d_engine_proto::common::AddNode { + node_id: 1, + address: "addr".into(), + status: d_engine_proto::common::NodeStatus::Promotable as i32, + })), + CommandType::Command(Bytes::from(b"cmd1".to_vec())), + ], + 1, + ); + + let mut rig = Rig::start(entries, 0, true); + let result = rig.notify(2).await; + assert!( + result.is_err(), + "a failed config change must fail the commit path" + ); +} + +/// Mixed command/config/noop entries are processed in log order. +#[tokio::test] +async fn processes_mixed_entries_correctly() { + let entries = build_entries( + vec![ + CommandType::Command(Bytes::from(b"cmd1".to_vec())), + CommandType::Configuration(Change::AddNode(d_engine_proto::common::AddNode { + node_id: 1, + address: "addr".into(), + status: d_engine_proto::common::NodeStatus::Promotable as i32, + })), + CommandType::Noop, + CommandType::Command(Bytes::from(b"cmd2".to_vec())), + CommandType::Command(Bytes::from(b"cmd3".to_vec())), + ], + 1, + ); + + let mut rig = Rig::start(entries, 0, false); + rig.notify(5).await.unwrap(); + rig.settle().await; + rig.assert_delivered(range(1..=5), "mixed entries must all be delivered in order"); +} + +/// A large batch of commands is delivered in one apply call. +#[tokio::test] +async fn handles_large_command_batches() { + let mut commands = Vec::new(); + for _ in 1..=1000 { + commands.push(CommandType::Command(Bytes::from(b"cmd".to_vec()))); + } + let entries = build_entries(commands, 1); + + let mut rig = Rig::start(entries, 0, false); + rig.notify(1000).await.unwrap(); + rig.settle().await; + rig.assert_delivered(range(1..=1000), "1000 commands must all be delivered"); +} + +/// MembershipApplied event MUST be sent after a successful config change (#209). +#[tokio::test] +async fn membership_applied_event_sent_on_success() { + let entries = build_entries( + vec![CommandType::Configuration(Change::AddNode( + d_engine_proto::common::AddNode { + node_id: 2, + address: "127.0.0.1:8080".into(), + status: d_engine_proto::common::NodeStatus::Promotable as i32, + }, + ))], + 1, + ); + + let mut rig = Rig::start(entries, 0, false); + rig.notify(1).await.unwrap(); + assert!( + rig.drain_events().iter().any(|e| matches!(e, InternalEvent::MembershipApplied)), + "a successful config change must send MembershipApplied" + ); +} + +/// Event order: MembershipApplied is sent AFTER apply_config_change (#209). +#[tokio::test] +async fn membership_applied_event_sent_after_apply() { + let order = Arc::new(std::sync::Mutex::new(Vec::new())); + + let entries = build_entries( + vec![CommandType::Configuration(Change::AddNode( + d_engine_proto::common::AddNode { + node_id: 2, + address: "127.0.0.1:8080".into(), + status: d_engine_proto::common::NodeStatus::Promotable as i32, + }, + ))], + 1, + ); + + // Build the rig manually so `apply_config_change` records into `order` before succeeding. + let order_clone = order.clone(); + let progress = Arc::new(Progress::new(0)); + let mut smh = MockStateMachineHandler::new(); + let p = progress.clone(); + smh.expect_pending_range().returning(move || p.range()); + let p = progress.clone(); + smh.expect_update_pending().returning(move |n| p.raise_pending(n)); + smh.expect_last_applied().returning(|| 0); + + let mut log = MockRaftLog::new(); + log.expect_load_hard_state().returning(|| Ok(None)); + log.expect_save_hard_state().returning(|_| Ok(())); + let e = entries.clone(); + log.expect_get_entries_range().returning(move |range| { + Ok(e.iter().filter(|x| range.contains(&x.index)).cloned().collect()) + }); + log.expect_close().returning(|| ()); + + let mut membership = MockMembership::new(); + membership.expect_notify_config_applied().returning(|_| {}); + membership.expect_apply_config_change().returning(move |_| { + order_clone.lock().unwrap().push("apply_config_change"); + Ok(()) + }); + + let (sm_tx, sm_rx) = mpsc::unbounded_channel::(); + tokio::spawn(async move { + let mut sm_rx = sm_rx; + while sm_rx.recv().await.is_some() {} + }); + + let (_shutdown_tx, shutdown_rx) = watch::channel(()); + let mut raft = MockBuilder::new(shutdown_rx) + .with_state_machine_handler(smh) + .with_raft_log(log) + .with_membership(membership) + .with_state_machine_commands(crate::StateMachineCommandSender::new(sm_tx)) + .turn_on_election(false) + .build_raft(); + raft.role.state_mut().update_commit_index(1).unwrap(); + raft.process_committed().await.unwrap(); + + let events: Vec = { + let mut out = Vec::new(); + while let Ok(ev) = raft.internal_event_rx.try_recv() { + out.push(ev); + } + out + }; + assert!( + events.iter().any(|e| matches!(e, InternalEvent::MembershipApplied)), + "a successful config change must send MembershipApplied" + ); + order.lock().unwrap().push("MembershipApplied_event"); + + assert_eq!( + *order.lock().unwrap(), + vec!["apply_config_change", "MembershipApplied_event"], + "Event must be sent AFTER apply_config_change" + ); +} + +/// NO MembershipApplied event is sent when a config change fails (#209). A FatalError is allowed +/// and expected (#455). +#[tokio::test] +async fn no_membership_applied_event_on_config_failure() { + let entries = build_entries( + vec![CommandType::Configuration(Change::AddNode( + d_engine_proto::common::AddNode { + node_id: 2, + address: "127.0.0.1:8080".into(), + status: d_engine_proto::common::NodeStatus::Promotable as i32, + }, + ))], + 1, + ); + + let mut rig = Rig::start(entries, 0, true); + let result = rig.notify(1).await; + assert!(result.is_err(), "config change should fail"); + + let events = rig.drain_events(); + assert!( + !events.iter().any(|e| matches!(e, InternalEvent::MembershipApplied)), + "MembershipApplied sent although the membership change failed" + ); +} + +/// AddNode, RemoveNode, and Promote all trigger MembershipApplied (#209). +#[tokio::test] +async fn all_config_change_types_send_membership_applied() { + use d_engine_proto::common::NodeStatus; + use d_engine_proto::common::PromoteLearner; + + // AddNode + let entries = build_entries( + vec![CommandType::Configuration(Change::AddNode( + d_engine_proto::common::AddNode { + node_id: 2, + address: "127.0.0.1:8080".into(), + status: d_engine_proto::common::NodeStatus::Promotable as i32, + }, + ))], + 1, + ); + let mut rig = Rig::start(entries, 0, false); + rig.notify(1).await.unwrap(); + assert!( + rig.drain_events().iter().any(|e| matches!(e, InternalEvent::MembershipApplied)), + "AddNode should send MembershipApplied" + ); + + // RemoveNode + let entries = build_entries( + vec![CommandType::Configuration(Change::RemoveNode( + d_engine_proto::common::RemoveNode { node_id: 2 }, + ))], + 1, + ); + let mut rig = Rig::start(entries, 0, false); + rig.notify(1).await.unwrap(); + assert!( + rig.drain_events().iter().any(|e| matches!(e, InternalEvent::MembershipApplied)), + "RemoveNode should send MembershipApplied" + ); + + // PromoteLearner + let entries = build_entries( + vec![CommandType::Configuration(Change::Promote( + PromoteLearner { + node_id: 2, + status: NodeStatus::Active as i32, + }, + ))], + 1, + ); + let mut rig = Rig::start(entries, 0, false); + rig.notify(1).await.unwrap(); + assert!( + rig.drain_events().iter().any(|e| matches!(e, InternalEvent::MembershipApplied)), + "PromoteLearner should send MembershipApplied" + ); +} + +/// A self-removal (RemoveNode(my_id)) sends both MembershipApplied and StepDownSelfRemoved, in +/// that order (#209, #200). +#[tokio::test] +async fn self_removal_sends_both_events() { + let entries = build_entries( + vec![CommandType::Configuration(Change::RemoveNode( + d_engine_proto::common::RemoveNode { node_id: 1 }, // self (my_id = 1) + ))], + 1, + ); + + let mut rig = Rig::start(entries, 0, false); + rig.notify(1).await.unwrap(); + + let events = rig.drain_events(); + let first = events.iter().position(|e| matches!(e, InternalEvent::MembershipApplied)); + let second = events.iter().position(|e| matches!(e, InternalEvent::StepDownSelfRemoved)); + assert!( + first.is_some(), + "self-removal must announce MembershipApplied" + ); + assert!( + second.is_some(), + "self-removal must announce StepDownSelfRemoved" + ); + assert!( + first.unwrap() < second.unwrap(), + "MembershipApplied must precede StepDownSelfRemoved" + ); +} + +/// Dropping the event receiver makes the MembershipApplied send fail. The warn path must run +/// without propagating an error. +#[tokio::test] +async fn membership_applied_send_fails_gracefully_when_receiver_dropped() { + let entries = build_entries( + vec![CommandType::Configuration(Change::AddNode( + d_engine_proto::common::AddNode { + node_id: 2, + address: "127.0.0.1:8080".into(), + status: d_engine_proto::common::NodeStatus::Promotable as i32, + }, + ))], + 1, + ); + + let mut rig = Rig::start(entries, 0, false); + rig.drop_event_receiver(); + let result = rig.notify(1).await; + assert!( + result.is_ok(), + "send failure on MembershipApplied must not propagate as Err" + ); +} + +/// Dropping the event receiver makes the StepDownSelfRemoved send fail. The error path inside the +/// self-removal branch must run without panicking or returning an error. +#[tokio::test] +async fn step_down_send_fails_gracefully_when_receiver_dropped() { + let entries = build_entries( + vec![CommandType::Configuration(Change::RemoveNode( + d_engine_proto::common::RemoveNode { node_id: 1 }, // self (my_id = 1) + ))], + 1, + ); + + let mut rig = Rig::start(entries, 0, false); + rig.drop_event_receiver(); + let result = rig.notify(1).await; + assert!( + result.is_ok(), + "send failure on StepDownSelfRemoved must not propagate as Err" + ); +} + +// ------------------------------------------------------------------------------------------------ +// delivery timing +// ------------------------------------------------------------------------------------------------ + +/// CONTRACT: a commit that advances in ANY stage of a loop iteration is handed to the state +/// machine worker before that iteration ends — not one wake-up later. +/// +/// A follower learns of a commit only from an inbound `AppendEntries` (the LAST stage of an +/// iteration, after the internal-event and client-command stages). The first RPC from a new leader +/// makes the follower emit `LeaderDiscovered`, which wakes the loop a second time and would hide a +/// late delivery; so the test first lets that settle, then sends a second RPC from the same +/// leader that advances the commit. Election is off and nothing else is sent, so nothing can wake +/// the loop again: if delivery ran in an earlier stage, the commit would sit undelivered and the +/// test times out. +#[tokio::test] +async fn test_commit_advanced_by_inbound_append_entries_is_delivered_in_the_same_iteration() { + use d_engine_proto::server::replication::AppendEntriesRequest; + use d_engine_proto::server::replication::AppendEntriesResponse; + + use crate::AppendResponseWithUpdates; + use crate::InboundEvent; + use crate::MockReplicationCore; + use crate::maybe_clone_oneshot::MaybeCloneOneshot; + use crate::maybe_clone_oneshot::RaftOneshot; + + let new_commit = 5; + let mut parts = build_parts(log_entries(None), 0, false); + parts.log.expect_durable_index().returning(|| 0); + parts.log.expect_last_entry_id().returning(|| 0); + parts.log.expect_last_log_id().returning(|| None); + // The follower workflow looks its own node up in the membership. + parts.membership.expect_retrieve_node_meta().returning(|_| None); + + // Only the second RPC advances the commit. + let calls = Arc::new(AtomicUsize::new(0)); + let mut replication = MockReplicationCore::new(); + replication.expect_handle_append_entries().returning(move |_, _, _| { + let second_call = calls.fetch_add(1, Ordering::SeqCst) == 1; + Ok(AppendResponseWithUpdates { + response: AppendEntriesResponse::success(1, 1, None), + commit_index_update: second_call.then_some(new_commit), + }) + }); + + let (sm_tx, mut sm_rx) = mpsc::unbounded_channel::(); + let (_shutdown_tx, shutdown_rx) = watch::channel(()); + let mut raft = MockBuilder::new(shutdown_rx) + .with_state_machine_handler(parts.smh) + .with_raft_log(parts.log) + .with_membership(parts.membership) + .with_replication_handler(replication) + .with_state_machine_commands(crate::StateMachineCommandSender::new(sm_tx)) + .turn_on_election(false) + .build_raft(); + let event_tx = raft.event_sender(); + let _loop = tokio::spawn(async move { raft.run().await }); + + let append_entries = || { + let (resp_tx, _resp_rx) = MaybeCloneOneshot::new(); + InboundEvent::AppendEntries( + AppendEntriesRequest { + term: 1, + leader_id: 5, + prev_log_index: 0, + prev_log_term: 0, + entries: vec![], + leader_commit_index: 0, + }, + vec![resp_tx], + ) + }; + + event_tx.send(append_entries()).await.unwrap(); + time::sleep(Duration::from_millis(100)).await; // leader discovery settles; the loop goes idle + event_tx.send(append_entries()).await.unwrap(); + + // Far below the election timeout (min 500 ms): only the next tick could deliver it late. + let delivered = time::timeout(Duration::from_millis(300), async { + loop { + if let Some(StateMachineCommand::ApplyEntries { entries, .. }) = sm_rx.recv().await { + return entries.iter().map(|e| e.index).collect::>(); + } + } + }) + .await + .expect("the commit advanced by AppendEntries was not delivered in the same loop iteration"); + assert_eq!(delivered, range(1..=new_commit)); +} diff --git a/d-engine-core/src/raft_test/drain_based_batch_architecture_tests.rs b/d-engine-core/src/raft_test/drain_based_batch_architecture_tests.rs index 116b5153..33bd0d74 100644 --- a/d-engine-core/src/raft_test/drain_based_batch_architecture_tests.rs +++ b/d-engine-core/src/raft_test/drain_based_batch_architecture_tests.rs @@ -213,7 +213,7 @@ async fn test_p0_leader_single_command_drain_immediately() { // Access leader state to push command if let RaftRole::Leader(ref mut leader) = raft.role { - let cmd = crate::ClientCmd::Propose(write_req, response_tx); + let cmd = crate::ClientCmd::propose(write_req, response_tx); // **Step 2**: Push command to buffer leader.push_client_cmd(cmd, &raft.ctx); @@ -316,7 +316,7 @@ async fn test_p0_leader_high_load_max_batch_cap() { client_id: 1, command: Some(write_cmd), }; - let cmd = crate::ClientCmd::Propose(write_req, response_tx); + let cmd = crate::ClientCmd::propose(write_req, response_tx); leader.push_client_cmd(cmd, &raft.ctx); } @@ -479,7 +479,7 @@ async fn test_p0_follower_write_request_rejection() { let start = tokio::time::Instant::now(); if let RaftRole::Follower(ref mut follower) = raft.role { - let cmd = crate::ClientCmd::Propose(write_req, response_tx); + let cmd = crate::ClientCmd::propose(write_req, response_tx); // **Step 2**: Push write command (should reject immediately) follower.push_client_cmd(cmd, &raft.ctx); @@ -576,7 +576,7 @@ async fn test_p0_candidate_commands_handling() { }; if let RaftRole::Candidate(ref mut candidate) = raft.role { - let cmd = crate::ClientCmd::Propose(write_req, response_tx); + let cmd = crate::ClientCmd::propose(write_req, response_tx); candidate.push_client_cmd(cmd, &raft.ctx); let result = response_rx.recv().await; @@ -758,7 +758,7 @@ async fn test_p1_medium_load_natural_batching() { client_id: 1, command: Some(write_cmd), }; - let cmd = crate::ClientCmd::Propose(write_req, response_tx); + let cmd = crate::ClientCmd::propose(write_req, response_tx); leader.push_client_cmd(cmd, &raft.ctx); } @@ -865,7 +865,7 @@ async fn test_p1_mixed_workload_read_write_coexistence() { client_id: 1, command: Some(write_cmd), }; - let cmd = crate::ClientCmd::Propose(write_req, response_tx); + let cmd = crate::ClientCmd::propose(write_req, response_tx); leader.push_client_cmd(cmd, &raft.ctx); } @@ -973,7 +973,7 @@ async fn test_p1_burst_load_recovery_after_spike() { client_id: 1, command: Some(write_cmd), }; - let cmd = crate::ClientCmd::Propose(write_req, response_tx); + let cmd = crate::ClientCmd::propose(write_req, response_tx); leader.push_client_cmd(cmd, &raft.ctx); } @@ -1006,7 +1006,7 @@ async fn test_p1_burst_load_recovery_after_spike() { client_id: 1, command: Some(write_cmd), }; - let cmd = crate::ClientCmd::Propose(write_req, response_tx); + let cmd = crate::ClientCmd::propose(write_req, response_tx); leader.push_client_cmd(cmd, &raft.ctx); // Flush after each trickle command diff --git a/d-engine-core/src/raft_test/leader_change_tests.rs b/d-engine-core/src/raft_test/leader_change_tests.rs index b2ab17ac..01b02b60 100644 --- a/d-engine-core/src/raft_test/leader_change_tests.rs +++ b/d-engine-core/src/raft_test/leader_change_tests.rs @@ -13,7 +13,6 @@ //! - Multiple listeners receiving the same notification //! - Watch channel behavior on closure //! - Deduplication of identical notifications -//! use tokio::sync::mpsc; diff --git a/d-engine-core/src/raft_test/leader_discovered_tests.rs b/d-engine-core/src/raft_test/leader_discovered_tests.rs index db6a9d40..433c7ffb 100644 --- a/d-engine-core/src/raft_test/leader_discovered_tests.rs +++ b/d-engine-core/src/raft_test/leader_discovered_tests.rs @@ -12,7 +12,6 @@ //! - Multiple listeners receive the event //! - Event deduplication and watch channel behavior //! - InternalEvent::LeaderDiscovered creation and serialization -//! use tokio::sync::watch; @@ -148,3 +147,34 @@ fn test_internal_event_leader_discovered_creation() { _ => panic!("Should be LeaderDiscovered variant"), } } + +/// Test: the central loop records how long an `ApplyCompleted` event waited before it was +/// handled (`core.raft.central_loop.apply_completed_queue_wait_ms`). +/// +/// Why: the apply result is only turned into a client response when the single-threaded +/// loop gets to it; if the loop is busy that wait is part of commit->apply latency and is +/// otherwise invisible. +#[tokio::test] +async fn test_apply_completed_records_central_loop_queue_wait() { + let capture = crate::test_utils::MetricsCapture::new(); + let _guard = metrics::set_default_local_recorder(&capture); + + let (_graceful_tx, graceful_rx) = watch::channel(()); + let mut raft: Raft = MockBuilder::new(graceful_rx).build_raft(); + + raft.handle_internal_event(InternalEvent::ApplyCompleted { + last_index: 0, + results: vec![], + sent_at: tokio::time::Instant::now() - std::time::Duration::from_millis(25), + }) + .await + .expect("ApplyCompleted must be handled"); + + let waits = capture.histogram("core.raft.central_loop.apply_completed_queue_wait_ms", &[]); + assert_eq!(waits.len(), 1); + assert!( + waits[0] >= 25.0, + "wait must cover time since send, got {}", + waits[0] + ); +} diff --git a/d-engine-core/src/raft_test/merge_append_entries_tests.rs b/d-engine-core/src/raft_test/merge_append_entries_tests.rs index bbea3e38..89911652 100644 --- a/d-engine-core/src/raft_test/merge_append_entries_tests.rs +++ b/d-engine-core/src/raft_test/merge_append_entries_tests.rs @@ -6,9 +6,9 @@ //! //! ## Design decisions under test //! - Only the leading run of contiguous same-term AppendEntries is merged. -//! - Heartbeats (empty entries, same term and prev_log_index as the next slot) are absorbed -//! into the merge group: their sender is collected and their `leader_commit_index` is folded -//! in via `max()`, advancing the commit index without a separate dispatch. +//! - Heartbeats (empty entries, same term and prev_log_index as the next slot) are absorbed into +//! the merge group: their sender is collected and their `leader_commit_index` is folded in via +//! `max()`, advancing the commit index without a separate dispatch. //! - Non-AppendEntries events stop the merge and remain in FIFO order. //! - All senders from merged events are collected into a Vec for broadcast. //! @@ -17,15 +17,23 @@ //! directly into `raft.buffered_inbound_event`, calls `raft.merge_append_entries()`, //! then inspects the resulting buffer. -use crate::maybe_clone_oneshot::MaybeCloneOneshot; -use crate::test_utils::MockBuilder; -use crate::{InboundEvent, MaybeCloneOneshotReceiver, RaftNodeConfig, RaftOneshot, mock_entries}; -use d_engine_proto::common::LogId; -use d_engine_proto::server::election::{VoteRequest, VoteResponse}; -use d_engine_proto::server::replication::{AppendEntriesRequest, AppendEntriesResponse}; use std::collections::VecDeque; + +use d_engine_proto::common::LogId; +use d_engine_proto::server::election::VoteRequest; +use d_engine_proto::server::election::VoteResponse; +use d_engine_proto::server::replication::AppendEntriesRequest; +use d_engine_proto::server::replication::AppendEntriesResponse; use tokio::sync::watch; +use crate::InboundEvent; +use crate::MaybeCloneOneshotReceiver; +use crate::RaftNodeConfig; +use crate::RaftOneshot; +use crate::maybe_clone_oneshot::MaybeCloneOneshot; +use crate::mock_entries; +use crate::test_utils::MockBuilder; + // ----------------------------------------------------------------------- // Helpers // ----------------------------------------------------------------------- diff --git a/d-engine-core/src/raft_test/mod.rs b/d-engine-core/src/raft_test/mod.rs index 2a1d2a79..8e262540 100644 --- a/d-engine-core/src/raft_test/mod.rs +++ b/d-engine-core/src/raft_test/mod.rs @@ -85,10 +85,9 @@ //! - **B4.1**: Valid transition from Follower to Learner //! - **B4.2**: leader_change notification sent with None //! -//! #### B5. NotifyNewCommitIndex Event -//! - **B5.1**: Notification broadcast to all registered listeners -//! - **B5.2**: Correct new_commit_index passed to listeners -//! - **B5.3**: Multiple commit listeners receive notification +//! #### B5. (removed in #455) +//! Commit notifications no longer exist: committed entries are delivered by `process_committed()` +//! at the end of each loop iteration, driven by `commit_index`. See `commit_path_tests.rs`. //! //! #### B6. LeaderDiscovered Event //! - **B6.1**: Notification sent to leader_change listeners @@ -174,9 +173,8 @@ //! - **E1.3**: Multiple listeners receive all notifications //! - **E1.4**: Watch channel deduplication (identical notifications) //! -//! #### E2. New Commit Notifications -//! - **E2.1**: Notification sent with correct commit index -//! - **E2.2**: Multiple listeners receive notification +//! #### E2. (removed in #455) +//! Commit notifications no longer exist; see B5. //! //! #### E3. Role Transition Notifications (Test-Only) //! - **E3.1**: Role transition listener triggered on every role change @@ -263,7 +261,7 @@ //! - **G2.3**: No peer replication needed //! //! #### G3. Concurrent Event Handling -//! - **G3.1**: Multiple listeners receiving notifications simultaneously +//! - **G3.1**: (removed in #455) commit listeners no longer exist //! - **G3.2**: InternalEvents arriving while processing previous event //! //! #### G4. Listener Registration Edge Cases diff --git a/d-engine-core/src/raft_test/process_inbound_events_tests.rs b/d-engine-core/src/raft_test/process_inbound_events_tests.rs index e14d04a7..1201983f 100644 --- a/d-engine-core/src/raft_test/process_inbound_events_tests.rs +++ b/d-engine-core/src/raft_test/process_inbound_events_tests.rs @@ -9,20 +9,26 @@ //! The drain loop empties the buffer, so buffer length is not meaningful here. //! Dispatch counts are asserted via mock `times(N)`: //! - `handle_append_entries.times(1)` proves two consecutive AEs were merged. -//! - `handle_append_entries.times(2)` would prove they were dispatched separately. -//! Mock expectations are verified on drop at the end of each test. +//! - `handle_append_entries.times(2)` would prove they were dispatched separately. Mock +//! expectations are verified on drop at the end of each test. + +use std::collections::VecDeque; + +use d_engine_proto::server::election::VoteRequest; +use d_engine_proto::server::replication::AppendEntriesRequest; +use d_engine_proto::server::replication::AppendEntriesResponse; +use tokio::sync::watch; use crate::AppendResponseWithUpdates; +use crate::InboundEvent; use crate::InternalEvent; +use crate::MockElectionCore; +use crate::MockReplicationCore; use crate::StateUpdate; use crate::maybe_clone_oneshot::MaybeCloneOneshot; use crate::maybe_clone_oneshot::RaftOneshot; +use crate::mock_entries; use crate::test_utils::MockBuilder; -use crate::{InboundEvent, MockElectionCore, MockReplicationCore, mock_entries}; -use d_engine_proto::server::election::VoteRequest; -use d_engine_proto::server::replication::{AppendEntriesRequest, AppendEntriesResponse}; -use std::collections::VecDeque; -use tokio::sync::watch; // ----------------------------------------------------------------------- // Helpers @@ -299,31 +305,31 @@ async fn test_process_internal_events_propagates_fatal_error() { } // ----------------------------------------------------------------------- -// handle_shutdown (#438) +// shutdown (#438) // ----------------------------------------------------------------------- /// Events still queued when shutdown fires are dropped, not answered — their -/// resp_tx senders are never called. Drives the real `handle_shutdown` path, -/// not a re-implementation of it. +/// resp_tx senders are never called. Drives the real `run()` path (shutdown signalled +/// before it starts, so its first step is the shutdown branch), not a re-implementation. #[tokio::test] -async fn test_handle_shutdown_drops_pending_events_without_responding() { - let (_graceful_tx, graceful_rx) = watch::channel(()); +async fn test_shutdown_drops_pending_events_without_responding() { + let (graceful_tx, graceful_rx) = watch::channel(()); let mut raft = MockBuilder::new(graceful_rx).build_raft(); let (tx1, rx1) = MaybeCloneOneshot::new(); let (tx2, rx2) = MaybeCloneOneshot::new(); let mut queue = VecDeque::new(); - queue.push_back(InboundEvent::AppendEntries( - make_request(5, 9, 5, 1), - vec![tx1], - )); + queue.push_back(InboundEvent::AppendEntries(make_request(5, 9, 5, 1), vec![ + tx1, + ])); queue.push_back(InboundEvent::AppendEntries( make_request(5, 14, 5, 1), vec![tx2], )); raft.buffered_inbound_event = queue; - raft.handle_shutdown().await.unwrap(); + graceful_tx.send(()).unwrap(); + raft.run().await.unwrap(); drop(raft); // matches production: Raft is dropped shortly after run() returns assert!( diff --git a/d-engine-core/src/raft_test/raft_comprehensive_tests.rs b/d-engine-core/src/raft_test/raft_comprehensive_tests.rs index e10251fd..a665ad69 100644 --- a/d-engine-core/src/raft_test/raft_comprehensive_tests.rs +++ b/d-engine-core/src/raft_test/raft_comprehensive_tests.rs @@ -20,16 +20,17 @@ //! - tokio::time::pause() for deterministic timing //! - Mock membership and log storage //! - Listener notification verification -//! - -use d_engine_proto::common::NodeRole::{Candidate, Follower, Leader, Learner}; use std::sync::Arc; + +use d_engine_proto::common::NodeRole::Candidate; +use d_engine_proto::common::NodeRole::Follower; +use d_engine_proto::common::NodeRole::Leader; +use d_engine_proto::common::NodeRole::Learner; use tokio::sync::mpsc; use tokio::sync::watch; use crate::InternalEvent; -use crate::NewCommitData; use crate::test_utils::MockBuilder; use crate::test_utils::MockTypeConfig; @@ -56,7 +57,8 @@ fn prepare_succeed_majority_confirmation() -> ( crate::MockReplicationCore, ) { use std::sync::Arc; - use std::sync::atomic::{AtomicU64, Ordering}; + use std::sync::atomic::AtomicU64; + use std::sync::atomic::Ordering; // Dynamic counter: starts at 11 (pre-noop state). // prepare_batch_requests increments it by the number of payloads written, @@ -954,7 +956,8 @@ async fn test_leader_ready_notification_suppressed_when_noop_fails() { raft_log.expect_save_hard_state().returning(|_| Ok(())); // prepare_batch_requests returning Err simulates noop failure: - // verify_internal_quorum returns Err → BecomeFollower queued → notify_leader_change(Some) never sent. + // verify_internal_quorum returns Err → BecomeFollower queued → notify_leader_change(Some) never + // sent. let mut replication_handler = crate::MockReplicationCore::new(); replication_handler .expect_prepare_batch_requests() @@ -1007,41 +1010,6 @@ async fn test_become_learner_no_leader() { // Leader notification sent with None (no leader for learners) } -// B5. NotifyNewCommitIndex Event Tests - -/// Test: NotifyNewCommitIndex broadcasts to all listeners -/// -/// Verifies that commit index notification reaches all registered listeners. -/// Multiple listeners should all receive the notification. -/// See B5.1-B5.3 in test scenarios. -#[tokio::test] -async fn test_notify_new_commit_index_broadcasts() { - let (_graceful_tx, graceful_rx) = watch::channel(()); - let mut raft = MockBuilder::new(graceful_rx).build_raft(); - - // Register commit listeners - let (tx1, mut rx1) = mpsc::unbounded_channel(); - let (tx2, mut rx2) = mpsc::unbounded_channel(); - raft.register_new_commit_listener(tx1); - raft.register_new_commit_listener(tx2); - - // Send NotifyNewCommitIndex event - let new_commit_index = 10; - raft.handle_internal_event(InternalEvent::NotifyNewCommitIndex(NewCommitData { - new_commit_index, - role: Follower.into(), - current_term: 1, - })) - .await - .expect("Should handle NotifyNewCommitIndex"); - - // Both listeners should receive the notification - let msg1 = rx1.recv().await.expect("Listener 1 should receive"); - let msg2 = rx2.recv().await.expect("Listener 2 should receive"); - assert_eq!(msg1.new_commit_index, new_commit_index); - assert_eq!(msg2.new_commit_index, new_commit_index); -} - // B6. LeaderDiscovered Event Tests (already has good coverage, but documenting here) /// Test: LeaderDiscovered does NOT change role state @@ -1440,8 +1408,8 @@ async fn test_leader_init_three_peers() { /// /// # Scenario /// - Leader's log tip (`raft_log.last_entry_id()`) is at 12. -/// - The just-completed snapshot's own boundary is index 5 (simulates the snapshot -/// having been captured well before the leader's tip advanced to 12). +/// - The just-completed snapshot's own boundary is index 5 (simulates the snapshot having been +/// captured well before the leader's tip advanced to 12). /// - Expected: `next_index(peer) == 6` (5 + 1), NOT `13` (12 + 1). #[tokio::test] async fn test_snapshot_push_completed_uses_snapshot_boundary_not_leader_tip() { @@ -1560,8 +1528,8 @@ async fn test_snapshot_push_completed_missing_boundary_does_not_fall_back_to_lea /// /// # Scenario /// - Leader elected at term `T`. -/// - A `SnapshotPushCompleted` arrives carrying `term = T - 1` (a straggler from -/// the previous leadership period). +/// - A `SnapshotPushCompleted` arrives carrying `term = T - 1` (a straggler from the previous +/// leadership period). /// - Expected: `next_index` is left exactly as it was — not seeded. #[tokio::test] async fn test_stale_term_snapshot_push_completed_does_not_seed_next_index() { @@ -1666,11 +1634,10 @@ async fn test_snapshot_push_completed_for_peer_not_in_snapshot_does_not_seed_nex /// attempt. /// /// # Scenario -/// - Peer 42 is in `Snapshot` state, with `next_index` at some value the leader -/// recorded when the transfer started. +/// - Peer 42 is in `Snapshot` state, with `next_index` at some value the leader recorded when the +/// transfer started. /// - A `PeerStreamError` for peer 42 arrives (the bidi stream broke). -/// - Expected: `peer_replication_state(42)` is still `Snapshot`, and `next_index` -/// is unchanged. +/// - Expected: `peer_replication_state(42)` is still `Snapshot`, and `next_index` is unchanged. #[tokio::test] async fn test_peer_stream_error_does_not_touch_peer_in_snapshot_state() { let (_graceful_tx, graceful_rx) = watch::channel(()); @@ -2130,36 +2097,6 @@ async fn test_leader_change_notification_correctness() { // Listeners receive notification with correct leader_id and term } -// E2. New Commit Notifications - -/// Test: New commit notifications with correct index -/// -/// Verifies commit notifications reach listeners with correct commit_index. -/// See E2.1-E2.2 in test scenarios. -#[tokio::test] -async fn test_new_commit_notification_correctness() { - let (_graceful_tx, graceful_rx) = watch::channel(()); - let mut raft = MockBuilder::new(graceful_rx).build_raft(); - - // Register commit listener - let (tx, mut rx) = mpsc::unbounded_channel(); - raft.register_new_commit_listener(tx); - - // Send commit notification - let new_commit_index = 25; - raft.handle_internal_event(InternalEvent::NotifyNewCommitIndex(NewCommitData { - new_commit_index, - role: Follower.into(), - current_term: 5, - })) - .await - .unwrap(); - - // Verify notification contains correct index - let msg = rx.recv().await.expect("Should receive notification"); - assert_eq!(msg.new_commit_index, new_commit_index); -} - // E3. Role Transition Notifications (Test-Only) /// Test: Role transition listeners receive all transitions @@ -2463,38 +2400,6 @@ async fn test_single_node_cluster_election() { assert!(is_leader(raft.role.as_i32())); } -/// Test: Multiple listeners receiving notifications -/// -/// Verifies listener system handles concurrent receivers. -/// See G3.1 in test scenarios. -#[tokio::test] -async fn test_multiple_listeners_concurrent() { - let (_graceful_tx, graceful_rx) = watch::channel(()); - let mut raft = MockBuilder::new(graceful_rx).build_raft(); - - // Register multiple commit listeners - let (tx1, mut rx1) = mpsc::unbounded_channel(); - let (tx2, mut rx2) = mpsc::unbounded_channel(); - let (tx3, mut rx3) = mpsc::unbounded_channel(); - raft.register_new_commit_listener(tx1); - raft.register_new_commit_listener(tx2); - raft.register_new_commit_listener(tx3); - - // Send notification - raft.handle_internal_event(InternalEvent::NotifyNewCommitIndex(NewCommitData { - new_commit_index: 5, - role: Follower.into(), - current_term: 1, - })) - .await - .unwrap(); - - // All listeners should receive - assert!(rx1.recv().await.is_some()); - assert!(rx2.recv().await.is_some()); - assert!(rx3.recv().await.is_some()); -} - /// Test: Register listeners after role transitions /// /// Verifies late listener registration still receives future events. diff --git a/d-engine-core/src/replication/mod.rs b/d-engine-core/src/replication/mod.rs index c3c77e25..f2ec3c2b 100644 --- a/d-engine-core/src/replication/mod.rs +++ b/d-engine-core/src/replication/mod.rs @@ -15,7 +15,6 @@ pub mod replication_handler_test; use std::collections::HashMap; use std::sync::Arc; -use crate::client::ClientResponse; use async_trait::async_trait; use d_engine_proto::common::Entry; use d_engine_proto::common::EntryPayload; @@ -33,6 +32,7 @@ use crate::MaybeCloneOneshotSender; use crate::Result; use crate::TypeConfig; use crate::alias::ROF; +use crate::client::ClientResponse; /// Request with response channel that can handle all Raft payload types #[derive(Debug)] @@ -116,9 +116,9 @@ where /// /// Performs two operations that MUST remain in the Raft loop (serial, single-threaded): /// 1. Write new log entries to the local raft log (`generate_new_entries`) - /// 2. Build per-peer request payloads (`retrieve_to_be_synced_logs_for_peers` + `build_append_request`) - /// Peers whose `next_index < first_entry_id` (below purge boundary) are routed to - /// `PrepareResult.snapshot_targets` instead of `append_requests`. + /// 2. Build per-peer request payloads (`retrieve_to_be_synced_logs_for_peers` + + /// `build_append_request`) Peers whose `next_index < first_entry_id` (below purge + /// boundary) are routed to `PrepareResult.snapshot_targets` instead of `append_requests`. /// /// The caller (leader Raft loop) then fires each request to a per-follower /// `ReplicationWorker` task and returns immediately — no blocking `.await`. @@ -131,8 +131,8 @@ where /// but not at dispatch (or vice versa) would block that peer's heartbeat. /// /// # Returns - /// - `Ok(PrepareResult)` — append requests per in-range peer + snapshot targets. - /// Empty `append_requests` when there are no replication targets or all need snapshots. + /// - `Ok(PrepareResult)` — append requests per in-range peer + snapshot targets. Empty + /// `append_requests` when there are no replication targets or all need snapshots. /// - `Err` for log write failures or ID allocation errors async fn prepare_batch_requests( &self, @@ -182,9 +182,8 @@ where /// /// For each peer behind the leader's latest index: /// - `next_index < first_index` (purged prefix) → `NeedSnapshot` - /// - Fetched range is short or has a gap despite being in-range → `CorruptGap` - /// (storage-layer inconsistency, not a purge — caller must not silently - /// fall back to snapshot) + /// - Fetched range is short or has a gap despite being in-range → `CorruptGap` (storage-layer + /// inconsistency, not a purge — caller must not silently fall back to snapshot) /// - Otherwise → `Ready(entries)`, contiguous from `next_index` onward /// /// `first_index` is the leader's retained-log boundary (`raft_log.first_entry_id()`), diff --git a/d-engine-core/src/replication/replication_handler.rs b/d-engine-core/src/replication/replication_handler.rs index 2bf7ea09..ff7a01ad 100644 --- a/d-engine-core/src/replication/replication_handler.rs +++ b/d-engine-core/src/replication/replication_handler.rs @@ -1,3 +1,10 @@ +use std::cmp; +use std::collections::HashMap; +use std::collections::HashSet; +use std::fmt::Debug; +use std::marker::PhantomData; +use std::sync::Arc; + use async_trait::async_trait; use bytes::BytesMut; use d_engine_proto::client::WriteCommand; @@ -10,12 +17,6 @@ use d_engine_proto::server::replication::AppendEntriesResponse; use d_engine_proto::server::replication::ConflictResult; use d_engine_proto::server::replication::SuccessResult; use prost::Message; -use std::cmp; -use std::collections::HashMap; -use std::collections::HashSet; -use std::fmt::Debug; -use std::marker::PhantomData; -use std::sync::Arc; use tracing::debug; use tracing::error; use tracing::trace; @@ -127,7 +128,8 @@ where count, } => { // #436/#439: isolate to this one peer, don't fall back to snapshot — - // that would hide a storage-layer bug. Structured event reporting upgrades this from a bare log later. + // that would hide a storage-layer bug. Structured event reporting upgrades this + // from a bare log later. error!( peer_id, start, @@ -317,16 +319,13 @@ where if !entries_are_contiguous_in_range(&legacy_entries, peer_next_id, until_index) { - peer_entries.insert( - id, - PeerEntriesResult::CorruptGap { - start: peer_next_id, - end: until_index, - first_seen: legacy_entries.first().map(|e| e.index), - last_seen: legacy_entries.last().map(|e| e.index), - count: legacy_entries.len(), - }, - ); + peer_entries.insert(id, PeerEntriesResult::CorruptGap { + start: peer_next_id, + end: until_index, + first_seen: legacy_entries.first().map(|e| e.index), + last_seen: legacy_entries.last().map(|e| e.index), + count: legacy_entries.len(), + }); continue; } @@ -615,20 +614,17 @@ where entries.len() ); - Ok(( - peer_id, - BuildAppendOutcome::Append { - request: AppendEntriesRequest { - term: data.current_term, - leader_id: self.my_id, - prev_log_index: prev_index, - prev_log_term, - entries, - leader_commit_index: data.commit_index, - }, - effective_next_index: next_index, + Ok((peer_id, BuildAppendOutcome::Append { + request: AppendEntriesRequest { + term: data.current_term, + leader_id: self.my_id, + prev_log_index: prev_index, + prev_log_term, + entries, + leader_commit_index: data.commit_index, }, - )) + effective_next_index: next_index, + })) } } diff --git a/d-engine-core/src/replication/replication_handler_test/basic_scenarios_test.rs b/d-engine-core/src/replication/replication_handler_test/basic_scenarios_test.rs index b4e96c1f..06a63a97 100644 --- a/d-engine-core/src/replication/replication_handler_test/basic_scenarios_test.rs +++ b/d-engine-core/src/replication/replication_handler_test/basic_scenarios_test.rs @@ -10,6 +10,12 @@ use std::collections::HashMap; use std::sync::Arc; +use d_engine_proto::common::NodeRole; +use d_engine_proto::common::NodeRole::Leader; +use d_engine_proto::common::NodeStatus; +use d_engine_proto::server::cluster::NodeMeta; +use tokio::sync::watch; + use crate::ClusterMetadata; use crate::LeaderStateSnapshot; use crate::MockRaftLog; @@ -18,11 +24,6 @@ use crate::ReplicationCore; use crate::ReplicationHandler; use crate::StateSnapshot; use crate::test_utils::mock_raft_context; -use d_engine_proto::common::NodeRole; -use d_engine_proto::common::NodeRole::Leader; -use d_engine_proto::common::NodeStatus; -use d_engine_proto::server::cluster::NodeMeta; -use tokio::sync::watch; /// Single-node cluster builds no replication requests (no peers to replicate to). /// diff --git a/d-engine-core/src/replication/replication_handler_test/heartbeat_match_index_test.rs b/d-engine-core/src/replication/replication_handler_test/heartbeat_match_index_test.rs index f90c815c..72b709e6 100644 --- a/d-engine-core/src/replication/replication_handler_test/heartbeat_match_index_test.rs +++ b/d-engine-core/src/replication/replication_handler_test/heartbeat_match_index_test.rs @@ -12,7 +12,8 @@ use std::sync::Arc; use d_engine_proto::common::Entry; -use d_engine_proto::server::replication::{AppendEntriesRequest, append_entries_response}; +use d_engine_proto::server::replication::AppendEntriesRequest; +use d_engine_proto::server::replication::append_entries_response; use crate::MockRaftLog; use crate::MockTypeConfig; @@ -20,7 +21,8 @@ use crate::RaftLog; use crate::ReplicationCore; use crate::ReplicationHandler; use crate::StateSnapshot; -use crate::test_utils::{RaftLogCoreTestContext, mock_entries}; +use crate::test_utils::RaftLogCoreTestContext; +use crate::test_utils::mock_entries; const OLD_TERM: u64 = 1; const NEW_TERM: u64 = 2; diff --git a/d-engine-core/src/replication/replication_handler_test/helper_functions_test.rs b/d-engine-core/src/replication/replication_handler_test/helper_functions_test.rs index 98262f60..24ad59b4 100644 --- a/d-engine-core/src/replication/replication_handler_test/helper_functions_test.rs +++ b/d-engine-core/src/replication/replication_handler_test/helper_functions_test.rs @@ -4,17 +4,9 @@ //! - generate_new_entries: Convert commands to log entries //! - build_append_request: Build AppendEntriesRequest for peers -use crate::ConsensusError; -use crate::Error; -use crate::MockRaftLog; -use crate::MockTypeConfig; -use crate::ReplicationCore; -use crate::ReplicationData; -use crate::ReplicationError; -use crate::ReplicationHandler; -use crate::client_command_to_entry_payloads; -use crate::test_utils::mock_insert_log_entries; -use crate::test_utils::setup_mock_replication_test_context; +use std::collections::HashMap; +use std::sync::Arc; + use bytes::Bytes; use d_engine_proto::client::WriteCommand; use d_engine_proto::client::write_command::Insert; @@ -27,8 +19,18 @@ use d_engine_proto::server::replication::ConflictResult; use d_engine_proto::server::replication::SuccessResult; use d_engine_proto::server::replication::append_entries_response; use prost::Message; -use std::collections::HashMap; -use std::sync::Arc; + +use crate::ConsensusError; +use crate::Error; +use crate::MockRaftLog; +use crate::MockTypeConfig; +use crate::ReplicationCore; +use crate::ReplicationData; +use crate::ReplicationError; +use crate::ReplicationHandler; +use crate::client_command_to_entry_payloads; +use crate::test_utils::mock_insert_log_entries; +use crate::test_utils::setup_mock_replication_test_context; /// Generate new entries with empty commands should return empty vector. /// @@ -833,11 +835,10 @@ async fn test_build_append_request_rejects_inconsistent_prev_log_term() { // `entry_term(0)` is always `None` (index 0 is the Raft "before the log" // sentinel, never a real entry), so this always falls into the `None if ...` // guard chain below. Two cases must be told apart: -// - first_index == 1 (log never purged): prev_index == 0 is the ordinary -// "peer starting from scratch" case -> normal Append with prev_log_term=0. -// - first_index > 1 (leader purged its prefix): prev_index == 0 means the -// peer needs everything from the very start, which no longer exists on -// this leader -> NeedSnapshot. +// - first_index == 1 (log never purged): prev_index == 0 is the ordinary "peer starting from +// scratch" case -> normal Append with prev_log_term=0. +// - first_index > 1 (leader purged its prefix): prev_index == 0 means the peer needs everything +// from the very start, which no longer exists on this leader -> NeedSnapshot. // A naive "just swap the match arms" fix breaks the first case (every fresh // peer / cluster bootstrap would wrongly get routed to snapshot instead of // a normal AppendEntries) — these three tests pin down all three branches diff --git a/d-engine-core/src/replication/replication_handler_test/log_retrieval_test.rs b/d-engine-core/src/replication/replication_handler_test/log_retrieval_test.rs index 38060ebe..d974a7a4 100644 --- a/d-engine-core/src/replication/replication_handler_test/log_retrieval_test.rs +++ b/d-engine-core/src/replication/replication_handler_test/log_retrieval_test.rs @@ -7,6 +7,12 @@ //! - Case 4: Multiple scenarios with different peer positions //! - Case 5: Edge cases in log retrieval +use std::collections::HashMap; +use std::sync::Arc; + +use d_engine_proto::common::Entry; +use d_engine_proto::common::EntryPayload; + use crate::MockTypeConfig; use crate::ReplicationCore; use crate::ReplicationHandler; @@ -14,10 +20,6 @@ use crate::mock_insert_log_entries; use crate::mock_log_entries_exist; use crate::test_utils::generate_insert_commands; use crate::test_utils::setup_mock_replication_test_context; -use d_engine_proto::common::Entry; -use d_engine_proto::common::EntryPayload; -use std::collections::HashMap; -use std::sync::Arc; /// Peer at end of old log should receive only new entries. /// @@ -328,7 +330,8 @@ async fn test_leader_id_excluded_from_replication_targets() { // Arrange: Peer map includes BOTH leader and peer3 // peer3's next_index is strictly past the leader's last old entry, so the - // legacy-fetch branch is skipped entirely (see test_retrieve_only_new_entries_when_peer_caught_up). + // legacy-fetch branch is skipped entirely (see + // test_retrieve_only_new_entries_when_peer_caught_up). let peer_next_indices = HashMap::from([ (my_id, 1_u64), // Leader itself (peer3_id, leader_last_index_before_inserting_new_entries + 1), // Peer3 diff --git a/d-engine-core/src/replication/replication_handler_test/quorum_calculation_test.rs b/d-engine-core/src/replication/replication_handler_test/quorum_calculation_test.rs index 93e22afd..7bad6e42 100644 --- a/d-engine-core/src/replication/replication_handler_test/quorum_calculation_test.rs +++ b/d-engine-core/src/replication/replication_handler_test/quorum_calculation_test.rs @@ -8,6 +8,12 @@ use std::collections::HashMap; use std::sync::Arc; +use d_engine_proto::common::NodeRole::Follower; +use d_engine_proto::common::NodeRole::Leader; +use d_engine_proto::common::NodeStatus; +use d_engine_proto::server::cluster::NodeMeta; +use tokio::sync::watch; + use crate::ClusterMetadata; use crate::LeaderStateSnapshot; use crate::MockRaftLog; @@ -16,11 +22,6 @@ use crate::ReplicationCore; use crate::ReplicationHandler; use crate::StateSnapshot; use crate::test_utils::mock_raft_context; -use d_engine_proto::common::NodeRole::Follower; -use d_engine_proto::common::NodeRole::Leader; -use d_engine_proto::common::NodeStatus; -use d_engine_proto::server::cluster::NodeMeta; -use tokio::sync::watch; /// Two-node cluster builds one replication request targeting the peer. /// diff --git a/d-engine-core/src/replication/replication_handler_test/snapshot_trigger_test.rs b/d-engine-core/src/replication/replication_handler_test/snapshot_trigger_test.rs index 38c327d8..ee063ec1 100644 --- a/d-engine-core/src/replication/replication_handler_test/snapshot_trigger_test.rs +++ b/d-engine-core/src/replication/replication_handler_test/snapshot_trigger_test.rs @@ -17,6 +17,14 @@ use std::collections::HashMap; use std::sync::Arc; +use d_engine_proto::common::Entry; +use d_engine_proto::common::EntryPayload; +use d_engine_proto::common::NodeRole; +use d_engine_proto::common::NodeRole::Leader; +use d_engine_proto::common::NodeStatus; +use d_engine_proto::server::cluster::NodeMeta; +use tokio::sync::watch; + use crate::ClusterMetadata; use crate::LeaderStateSnapshot; use crate::MockRaftLog; @@ -25,13 +33,6 @@ use crate::ReplicationCore; use crate::ReplicationHandler; use crate::StateSnapshot; use crate::test_utils::mock_raft_context; -use d_engine_proto::common::Entry; -use d_engine_proto::common::EntryPayload; -use d_engine_proto::common::NodeRole; -use d_engine_proto::common::NodeRole::Leader; -use d_engine_proto::common::NodeStatus; -use d_engine_proto::server::cluster::NodeMeta; -use tokio::sync::watch; /// Both tests below mock a log window covering indices 11..=15 (`first_entry_id`=11, /// `last_entry_id`=15). `retrieve_to_be_synced_logs_for_peers` now enforces that any @@ -63,8 +64,8 @@ fn voter(id: u32) -> NodeMeta { /// # Scenario /// - Leader log: entries 1-10 purged (first_entry_id = 11, last_entry_id = 15) /// - Peer 2 next_index = 4 → prev_log_index = 3, which is purged (non-boundary) -/// - Expected: PrepareResult.snapshot_targets = [2] -/// PrepareResult.append_requests does NOT contain peer 2 +/// - Expected: PrepareResult.snapshot_targets = [2] PrepareResult.append_requests does NOT contain +/// peer 2 /// /// # Before fix (will FAIL) /// `prepare_batch_requests` returns an AppendEntriesRequest for peer 2 with @@ -296,8 +297,7 @@ async fn test_prepare_batch_requests_splits_snapshot_and_append_peers() { /// # Scenario /// - Leader: first_entry_id = 1 (nothing ever purged), last_entry_id = 5 /// - Peer 2: next_index = 1 (never replicated to before) -> prev_log_index = 0 -/// - Expected: peer 2 in append_requests (prev_log_term = 0), NOT in -/// snapshot_targets +/// - Expected: peer 2 in append_requests (prev_log_term = 0), NOT in snapshot_targets /// /// # Regression this guards (PR #442 review) /// The purge-boundary fix in `build_append_request` must distinguish @@ -375,10 +375,10 @@ async fn test_prepare_batch_requests_routes_fresh_peer_to_append_when_log_never_ /// /// # Scenario /// - Leader: first_entry_id = 11 (indices 1-10 purged), last_entry_id = 15 -/// - Peer 2: next_index = 1 -> prev_log_index = 0, and everything from index -/// 1 no longer exists on this leader -/// - Expected: peer 2 appears in snapshot_targets exactly once, and does -/// NOT appear in append_requests +/// - Peer 2: next_index = 1 -> prev_log_index = 0, and everything from index 1 no longer exists on +/// this leader +/// - Expected: peer 2 appears in snapshot_targets exactly once, and does NOT appear in +/// append_requests #[tokio::test] async fn test_prepare_batch_requests_routes_never_replicated_peer_to_snapshot_exactly_once() { let (_graceful_tx, graceful_rx) = watch::channel(()); diff --git a/d-engine-core/src/state_machine_handler/applied_state.rs b/d-engine-core/src/state_machine_handler/applied_state.rs index 22fa1a3a..8d604427 100644 --- a/d-engine-core/src/state_machine_handler/applied_state.rs +++ b/d-engine-core/src/state_machine_handler/applied_state.rs @@ -1,5 +1,6 @@ use std::sync::Arc; -use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::atomic::AtomicU64; +use std::sync::atomic::Ordering; use tokio::sync::watch; diff --git a/d-engine-core/src/state_machine_handler/command.rs b/d-engine-core/src/state_machine_handler/command.rs index 182662f0..39408e64 100644 --- a/d-engine-core/src/state_machine_handler/command.rs +++ b/d-engine-core/src/state_machine_handler/command.rs @@ -2,18 +2,20 @@ //! all state-machine mutations. //! one serial writer. -use crate::Result; -use crate::SnapshotApplyResult; -use crate::SnapshotError; -use d_engine_proto::common::Entry; -use d_engine_proto::server::storage::SnapshotMetadata; use std::path::Path; use std::path::PathBuf; + +use d_engine_proto::common::Entry; +use d_engine_proto::server::storage::SnapshotMetadata; use tempfile::TempDir; use tokio::sync::mpsc; use tokio::sync::oneshot; use tracing::warn; +use crate::Result; +use crate::SnapshotApplyResult; +use crate::SnapshotError; + /// A snapshot stream already fully received and decompressed to a local temp /// directory. Owns the directory's lifetime: dropping it (e.g. once the /// Worker finishes consuming it) cleans up the files automatically. @@ -118,8 +120,14 @@ pub struct CapturedLocalSnapshot { pub enum StateMachineCommand { /// Fire-and-forget by design: completion is reported via /// `InternalEvent::ApplyCompleted`, not through this variant, to keep - /// `CommitHandler` decoupled from apply I/O (see worker.rs doc comment). - ApplyEntries { entries: Vec }, + /// the Raft loop decoupled from apply I/O (see worker.rs doc comment). + /// + /// `sent_at` is when the Raft loop enqueued the batch; the Worker records the + /// queue wait (`core.raft.sm_worker.queue_wait_ms`) when it dequeues it. + ApplyEntries { + entries: Vec, + sent_at: tokio::time::Instant, + }, /// Carries a response channel because the caller (Raft main loop) must /// block until the install genuinely completes before replying to the @@ -148,6 +156,11 @@ pub enum StateMachineCommand { response: oneshot::Sender>, guard: tokio::sync::OwnedMutexGuard<()>, }, + + /// Last command the Raft node's owner puts in the queue. The Worker applies everything + /// queued before it, then stops. Queue order is what makes "everything" include the + /// last `ApplyEntries`. + Shutdown, } /// The only way role code (follower/learner) may reach `StateMachineWorker` — hides the @@ -170,7 +183,10 @@ impl StateMachineCommandSender { entries: Vec, ) -> Result<()> { self.tx - .send(StateMachineCommand::ApplyEntries { entries }) + .send(StateMachineCommand::ApplyEntries { + entries, + sent_at: tokio::time::Instant::now(), + }) .map_err(|_| SnapshotError::WorkerUnavailable.into()) } diff --git a/d-engine-core/src/state_machine_handler/default_state_machine_handler.rs b/d-engine-core/src/state_machine_handler/default_state_machine_handler.rs index e767fc3c..6f53e05b 100644 --- a/d-engine-core/src/state_machine_handler/default_state_machine_handler.rs +++ b/d-engine-core/src/state_machine_handler/default_state_machine_handler.rs @@ -1,10 +1,12 @@ -#[cfg(feature = "watch")] -use crate::BatchOp; +use std::ops::RangeInclusive; +use std::path::Path; +use std::path::PathBuf; +use std::sync::Arc; +use std::sync::atomic::AtomicBool; +use std::sync::atomic::AtomicU64; +use std::sync::atomic::Ordering; +use std::time::Duration; -use crate::CapturedLocalSnapshot; -use crate::PreparedSnapshot; -use crate::client::KvEntry; -use crate::state_machine_handler::applied_state::AppliedState; use async_compression::tokio::bufread::GzipDecoder; use async_compression::tokio::write::GzipEncoder; use async_stream::try_stream; @@ -18,14 +20,6 @@ use d_engine_proto::server::storage::snapshot_ack::ChunkStatus; use futures::stream::BoxStream; use memmap2::Mmap; use memmap2::MmapOptions; -use std::ops::RangeInclusive; -use std::path::Path; -use std::path::PathBuf; -use std::sync::Arc; -use std::sync::atomic::AtomicBool; -use std::sync::atomic::AtomicU64; -use std::sync::atomic::Ordering; -use std::time::Duration; use tokio::fs; use tokio::fs::File; use tokio::fs::remove_dir_all; @@ -50,8 +44,12 @@ use crate::ApplyEntry; #[cfg(feature = "watch")] use crate::ApplyResult; #[cfg(feature = "watch")] +use crate::BatchOp; +use crate::CapturedLocalSnapshot; +#[cfg(feature = "watch")] use crate::Command; use crate::NewCommitData; +use crate::PreparedSnapshot; use crate::Result; use crate::SnapshotConfig; use crate::SnapshotError; @@ -61,9 +59,11 @@ use crate::StorageError; use crate::TypeConfig; use crate::alias::SMOF; use crate::alias::SNP; +use crate::client::KvEntry; use crate::file_io::validate_checksum; use crate::file_io::validate_compressed_format; use crate::scoped_timer::ScopedTimer; +use crate::state_machine_handler::applied_state::AppliedState; /// Unified snapshot metadata with precomputed values #[derive(Debug, Clone)] diff --git a/d-engine-core/src/state_machine_handler/default_state_machine_handler_test.rs b/d-engine-core/src/state_machine_handler/default_state_machine_handler_test.rs index 67915722..34cc61fc 100644 --- a/d-engine-core/src/state_machine_handler/default_state_machine_handler_test.rs +++ b/d-engine-core/src/state_machine_handler/default_state_machine_handler_test.rs @@ -1,21 +1,18 @@ -use super::DefaultStateMachineHandler; -use super::DefaultStateMachineWriter; -use super::StateMachineHandler; -use super::StateMachineWriterOps; -#[cfg(feature = "watch")] -use super::broadcast_watch_events; -use super::new_reader_writer_pair; -use crate::Error; -use crate::MockSnapshotPolicy; -use crate::MockStateMachine; -use crate::MockTypeConfig; -use crate::SnapshotError; -use crate::test_utils::create_test_compressed_snapshot; -use crate::test_utils::snapshot_config; +use std::collections::HashSet; +use std::fs; +use std::path::Path; +use std::path::PathBuf; +use std::sync::Arc; +use std::sync::atomic::AtomicUsize; +use std::time::Duration; + use bytes::Bytes; use d_engine_proto::client::WriteCommand; +use d_engine_proto::client::write_command::Batch; +use d_engine_proto::client::write_command::BatchOp as ProtoBatchOp; +use d_engine_proto::client::write_command::Insert; +use d_engine_proto::client::write_command::Operation; use d_engine_proto::client::write_command::batch_op::Op; -use d_engine_proto::client::write_command::{Batch, BatchOp as ProtoBatchOp, Insert, Operation}; use d_engine_proto::common::Entry; use d_engine_proto::common::EntryPayload; use d_engine_proto::common::LogId; @@ -27,13 +24,6 @@ use d_engine_proto::server::storage::snapshot_ack::ChunkStatus; use futures::StreamExt; use mockall::Sequence; use prost::Message; -use std::collections::HashSet; -use std::fs; -use std::path::Path; -use std::path::PathBuf; -use std::sync::Arc; -use std::sync::atomic::AtomicUsize; -use std::time::Duration; use tempfile::TempDir; use tempfile::tempdir; use tokio::fs::File; @@ -43,6 +33,21 @@ use tokio::sync::mpsc; use tracing::debug; use tracing_test::traced_test; +use super::DefaultStateMachineHandler; +use super::DefaultStateMachineWriter; +use super::StateMachineHandler; +use super::StateMachineWriterOps; +#[cfg(feature = "watch")] +use super::broadcast_watch_events; +use super::new_reader_writer_pair; +use crate::Error; +use crate::MockSnapshotPolicy; +use crate::MockStateMachine; +use crate::MockTypeConfig; +use crate::SnapshotError; +use crate::test_utils::create_test_compressed_snapshot; +use crate::test_utils::snapshot_config; + fn batch_entry( index: u64, key: &[u8], @@ -430,7 +435,8 @@ mod apply_chunk_wakes_wait_applied_test { use std::sync::Arc; use std::time::Duration; - use d_engine_proto::common::{Entry, EntryPayload}; + use d_engine_proto::common::Entry; + use d_engine_proto::common::EntryPayload; use super::StateMachineHandler; use super::StateMachineWriterOps; @@ -492,16 +498,20 @@ mod apply_chunk_wakes_wait_applied_test { mod decode_boundary_tests { use bytes::Bytes; use d_engine_proto::client::WriteCommand; - use d_engine_proto::client::write_command::{Insert, Operation}; + use d_engine_proto::client::write_command::Insert; + use d_engine_proto::client::write_command::Operation; use d_engine_proto::common::EntryPayload; use d_engine_proto::common::entry_payload::Payload; use prost::Message; use super::*; + use crate::ApplyResult; + use crate::BatchOp; + use crate::Command; + use crate::MockSnapshotPolicy; + use crate::MockStateMachine; + use crate::MockTypeConfig; use crate::test_utils::snapshot_config; - use crate::{ - ApplyResult, BatchOp, Command, MockSnapshotPolicy, MockStateMachine, MockTypeConfig, - }; fn noop_entry(index: u64) -> Entry { Entry { @@ -954,8 +964,8 @@ mod create_snapshot_tests { /// # Scenario /// - `generate_snapshot_data` exports state and returns a checksum (no publish). /// - `build_local_snapshot` produces the archive, then publishes metadata. - /// - Expected: `persist_last_snapshot_metadata` is called exactly once, and only - /// after `generate_snapshot_data` (i.e. after the archive is built). + /// - Expected: `persist_last_snapshot_metadata` is called exactly once, and only after + /// `generate_snapshot_data` (i.e. after the archive is built). #[tokio::test] async fn test_create_snapshot_publishes_metadata_only_after_archive_built() { let temp_dir = tempfile::tempdir().unwrap(); @@ -1405,7 +1415,8 @@ mod create_snapshot_tests { ); } - /// create_snapshot when last_applied.index is zero must not panic or produce an invalid snapshot. + /// create_snapshot when last_applied.index is zero must not panic or produce an invalid + /// snapshot. /// /// # Purpose /// Boundary: snapshot triggered before any entry has been applied. @@ -1457,7 +1468,8 @@ mod create_snapshot_tests { assert_eq!(metadata.last_included, Some(LogId { term: 0, index: 0 })); } - /// A crash mid-checkpoint must not leave the temp dir behind after create_snapshot() returns Err. + /// A crash mid-checkpoint must not leave the temp dir behind after create_snapshot() returns + /// Err. #[tokio::test] async fn test_create_snapshot_removes_temp_dir_on_generate_failure() { let temp_dir = tempfile::tempdir().unwrap(); @@ -1490,7 +1502,8 @@ mod create_snapshot_tests { ); } - /// A stale temp dir left by a previous crash must not block the next snapshot at the same index. + /// A stale temp dir left by a previous crash must not block the next snapshot at the same + /// index. #[tokio::test] async fn test_create_snapshot_succeeds_despite_stale_temp_dir() { let temp_dir = tempfile::tempdir().unwrap(); @@ -2017,16 +2030,16 @@ async fn test_snapshot_compression() { /// write" window the bug leaves open. /// /// # Scenario -/// - `temp_dir` (captured snapshot) contains one normal file and one file -/// with all read permissions stripped +/// - `temp_dir` (captured snapshot) contains one normal file and one file with all read permissions +/// stripped /// - `build_local_snapshot` is called on this capture /// /// # Then /// - `build_local_snapshot` returns `Err` -/// - the partial `.tar.gz` at the standard `final_path` must NOT be left on -/// disk (this is the assertion that fails before the fix) -/// - `temp_dir` must still be cleaned up too (pre-existing behavior, must -/// not regress alongside the new cleanup) +/// - the partial `.tar.gz` at the standard `final_path` must NOT be left on disk (this is the +/// assertion that fails before the fix) +/// - `temp_dir` must still be cleaned up too (pre-existing behavior, must not regress alongside the +/// new cleanup) #[tokio::test] #[cfg(unix)] async fn test_build_local_snapshot_removes_partial_archive_on_compression_failure() { @@ -2100,7 +2113,10 @@ mod broadcast_watch_events_tests { use d_engine_proto::client::WatchEventType; use super::*; - use crate::{ApplyEntry, ApplyResult, BatchOp, Command}; + use crate::ApplyEntry; + use crate::ApplyResult; + use crate::BatchOp; + use crate::Command; /// Build an `ApplyEntry` directly from a `Command` — no proto encoding needed. /// `broadcast_watch_events` receives already-decoded entries; tests mirror that reality. @@ -2133,14 +2149,11 @@ mod broadcast_watch_events_tests { async fn test_broadcast_insert_event() { let (tx, mut rx) = tokio::sync::broadcast::channel(10); - let entry = apply_entry( - 1, - Command::Insert { - key: Bytes::from_static(b"key1"), - value: Bytes::from_static(b"value1"), - ttl_secs: None, - }, - ); + let entry = apply_entry(1, Command::Insert { + key: Bytes::from_static(b"key1"), + value: Bytes::from_static(b"value1"), + ttl_secs: None, + }); broadcast_watch_events(&[entry], &[succeeded(1)], &tx, None); @@ -2154,12 +2167,9 @@ mod broadcast_watch_events_tests { async fn test_broadcast_delete_event() { let (tx, mut rx) = tokio::sync::broadcast::channel(10); - let entry = apply_entry( - 1, - Command::Delete { - key: Bytes::from_static(b"key1"), - }, - ); + let entry = apply_entry(1, Command::Delete { + key: Bytes::from_static(b"key1"), + }); broadcast_watch_events(&[entry], &[succeeded(1)], &tx, None); @@ -2174,14 +2184,11 @@ mod broadcast_watch_events_tests { async fn test_broadcast_cas_success() { let (tx, mut rx) = tokio::sync::broadcast::channel(10); - let entry = apply_entry( - 1, - Command::CompareAndSwap { - key: Bytes::from_static(b"lock"), - expected: Some(Bytes::from_static(b"owner1")), - value: Bytes::from_static(b"owner2"), - }, - ); + let entry = apply_entry(1, Command::CompareAndSwap { + key: Bytes::from_static(b"lock"), + expected: Some(Bytes::from_static(b"owner1")), + value: Bytes::from_static(b"owner2"), + }); broadcast_watch_events(&[entry], &[succeeded(1)], &tx, None); @@ -2196,14 +2203,11 @@ mod broadcast_watch_events_tests { async fn test_broadcast_cas_failure_no_event() { let (tx, mut rx) = tokio::sync::broadcast::channel(10); - let entry = apply_entry( - 1, - Command::CompareAndSwap { - key: Bytes::from_static(b"lock"), - expected: Some(Bytes::from_static(b"wrong_owner")), - value: Bytes::from_static(b"owner2"), - }, - ); + let entry = apply_entry(1, Command::CompareAndSwap { + key: Bytes::from_static(b"lock"), + expected: Some(Bytes::from_static(b"wrong_owner")), + value: Bytes::from_static(b"owner2"), + }); broadcast_watch_events(&[entry], &[failed(1)], &tx, None); @@ -2219,30 +2223,21 @@ mod broadcast_watch_events_tests { let (tx, mut rx) = tokio::sync::broadcast::channel(10); let entries = vec![ - apply_entry( - 1, - Command::CompareAndSwap { - key: Bytes::from_static(b"k1"), - expected: Some(Bytes::from_static(b"v0")), - value: Bytes::from_static(b"v1"), - }, - ), - apply_entry( - 2, - Command::CompareAndSwap { - key: Bytes::from_static(b"k2"), - expected: Some(Bytes::from_static(b"wrong")), - value: Bytes::from_static(b"v2"), - }, - ), - apply_entry( - 3, - Command::CompareAndSwap { - key: Bytes::from_static(b"k3"), - expected: None, - value: Bytes::from_static(b"v3"), - }, - ), + apply_entry(1, Command::CompareAndSwap { + key: Bytes::from_static(b"k1"), + expected: Some(Bytes::from_static(b"v0")), + value: Bytes::from_static(b"v1"), + }), + apply_entry(2, Command::CompareAndSwap { + key: Bytes::from_static(b"k2"), + expected: Some(Bytes::from_static(b"wrong")), + value: Bytes::from_static(b"v2"), + }), + apply_entry(3, Command::CompareAndSwap { + key: Bytes::from_static(b"k3"), + expected: None, + value: Bytes::from_static(b"v3"), + }), ]; let results = vec![succeeded(1), failed(2), succeeded(3)]; @@ -2269,28 +2264,19 @@ mod broadcast_watch_events_tests { let (tx, mut rx) = tokio::sync::broadcast::channel(10); let entries = vec![ - apply_entry( - 1, - Command::Insert { - key: Bytes::from_static(b"k1"), - value: Bytes::from_static(b"v1"), - ttl_secs: None, - }, - ), - apply_entry( - 2, - Command::CompareAndSwap { - key: Bytes::from_static(b"k2"), - expected: Some(Bytes::from_static(b"wrong")), - value: Bytes::from_static(b"v2"), - }, - ), - apply_entry( - 3, - Command::Delete { - key: Bytes::from_static(b"k3"), - }, - ), + apply_entry(1, Command::Insert { + key: Bytes::from_static(b"k1"), + value: Bytes::from_static(b"v1"), + ttl_secs: None, + }), + apply_entry(2, Command::CompareAndSwap { + key: Bytes::from_static(b"k2"), + expected: Some(Bytes::from_static(b"wrong")), + value: Bytes::from_static(b"v2"), + }), + apply_entry(3, Command::Delete { + key: Bytes::from_static(b"k3"), + }), ]; let results = vec![succeeded(1), failed(2), succeeded(3)]; @@ -2313,22 +2299,16 @@ mod broadcast_watch_events_tests { let (tx, mut rx) = tokio::sync::broadcast::channel(10); let entries = vec![ - apply_entry( - 10, - Command::CompareAndSwap { - key: Bytes::from_static(b"key-a"), - expected: Some(Bytes::from_static(b"old")), - value: Bytes::from_static(b"new-a"), - }, - ), - apply_entry( - 11, - Command::CompareAndSwap { - key: Bytes::from_static(b"key-b"), - expected: Some(Bytes::from_static(b"old")), - value: Bytes::from_static(b"new-b"), - }, - ), + apply_entry(10, Command::CompareAndSwap { + key: Bytes::from_static(b"key-a"), + expected: Some(Bytes::from_static(b"old")), + value: Bytes::from_static(b"new-a"), + }), + apply_entry(11, Command::CompareAndSwap { + key: Bytes::from_static(b"key-b"), + expected: Some(Bytes::from_static(b"old")), + value: Bytes::from_static(b"new-b"), + }), ]; let results = vec![failed(10), succeeded(11)]; @@ -2373,21 +2353,18 @@ mod broadcast_watch_events_tests { async fn test_broadcast_batch_insert_ops_each_produce_put_event() { let (tx, mut rx) = tokio::sync::broadcast::channel(10); - let entry = apply_entry( - 1, - Command::Batch { - ops: vec![ - BatchOp::Insert { - key: Bytes::from_static(b"k1"), - value: Bytes::from_static(b"v1"), - }, - BatchOp::Insert { - key: Bytes::from_static(b"k2"), - value: Bytes::from_static(b"v2"), - }, - ], - }, - ); + let entry = apply_entry(1, Command::Batch { + ops: vec![ + BatchOp::Insert { + key: Bytes::from_static(b"k1"), + value: Bytes::from_static(b"v1"), + }, + BatchOp::Insert { + key: Bytes::from_static(b"k2"), + value: Bytes::from_static(b"v2"), + }, + ], + }); broadcast_watch_events(&[entry], &[succeeded(1)], &tx, None); @@ -2423,19 +2400,16 @@ mod broadcast_watch_events_tests { async fn test_broadcast_batch_delete_ops_each_produce_delete_event() { let (tx, mut rx) = tokio::sync::broadcast::channel(10); - let entry = apply_entry( - 1, - Command::Batch { - ops: vec![ - BatchOp::Delete { - key: Bytes::from_static(b"k1"), - }, - BatchOp::Delete { - key: Bytes::from_static(b"k2"), - }, - ], - }, - ); + let entry = apply_entry(1, Command::Batch { + ops: vec![ + BatchOp::Delete { + key: Bytes::from_static(b"k1"), + }, + BatchOp::Delete { + key: Bytes::from_static(b"k2"), + }, + ], + }); broadcast_watch_events(&[entry], &[succeeded(1)], &tx, None); @@ -2464,31 +2438,27 @@ mod broadcast_watch_events_tests { /// - apply succeeds for the batch entry /// /// # Expected - /// - broadcast channel receives exactly 3 events in order: - /// Put(k1=v1), Delete(k2), Put(k3=v3) + /// - broadcast channel receives exactly 3 events in order: Put(k1=v1), Delete(k2), Put(k3=v3) /// - no reordering, no missing events #[tokio::test] async fn test_broadcast_batch_mixed_ops_events_preserve_order() { let (tx, mut rx) = tokio::sync::broadcast::channel(10); - let entry = apply_entry( - 1, - Command::Batch { - ops: vec![ - BatchOp::Insert { - key: Bytes::from_static(b"k1"), - value: Bytes::from_static(b"v1"), - }, - BatchOp::Delete { - key: Bytes::from_static(b"k2"), - }, - BatchOp::Insert { - key: Bytes::from_static(b"k3"), - value: Bytes::from_static(b"v3"), - }, - ], - }, - ); + let entry = apply_entry(1, Command::Batch { + ops: vec![ + BatchOp::Insert { + key: Bytes::from_static(b"k1"), + value: Bytes::from_static(b"v1"), + }, + BatchOp::Delete { + key: Bytes::from_static(b"k2"), + }, + BatchOp::Insert { + key: Bytes::from_static(b"k3"), + value: Bytes::from_static(b"v3"), + }, + ], + }); broadcast_watch_events(&[entry], &[succeeded(1)], &tx, None); @@ -2518,19 +2488,24 @@ mod broadcast_watch_events_tests { #[cfg(feature = "watch")] mod prev_kv_apply_tests { use std::sync::Arc; - use std::sync::atomic::{AtomicUsize, Ordering}; + use std::sync::atomic::AtomicUsize; + use std::sync::atomic::Ordering; use bytes::Bytes; - use d_engine_proto::client::{ - WriteCommand, - write_command::{Insert, Operation}, - }; - use d_engine_proto::common::{Entry, EntryPayload, entry_payload::Payload}; + use d_engine_proto::client::WriteCommand; + use d_engine_proto::client::write_command::Insert; + use d_engine_proto::client::write_command::Operation; + use d_engine_proto::common::Entry; + use d_engine_proto::common::EntryPayload; + use d_engine_proto::common::entry_payload::Payload; use prost::Message; use super::*; + use crate::ApplyResult; + use crate::MockSnapshotPolicy; + use crate::MockStateMachine; + use crate::MockTypeConfig; use crate::test_utils::snapshot_config; - use crate::{ApplyResult, MockSnapshotPolicy, MockStateMachine, MockTypeConfig}; fn insert_entry( index: u64, @@ -2687,8 +2662,10 @@ mod prev_kv_apply_tests { mod wait_applied_tests { use super::*; + use crate::MockSnapshotPolicy; + use crate::MockStateMachine; + use crate::MockTypeConfig; use crate::test_utils::snapshot_config; - use crate::{MockSnapshotPolicy, MockStateMachine, MockTypeConfig}; /// wait_applied returns immediately when last_applied is already >= target index. /// @@ -2746,7 +2723,8 @@ mod wait_applied_tests { /// - does not unblock at index 4 or earlier #[tokio::test] async fn test_wait_applied_blocks_until_target_index_reached() { - use std::sync::atomic::{AtomicBool, Ordering}; + use std::sync::atomic::AtomicBool; + use std::sync::atomic::Ordering; let handler = Arc::new(DefaultStateMachineHandler::::new( 1, @@ -2800,7 +2778,8 @@ mod wait_applied_tests { /// - all 3 return Ok(()) #[tokio::test] async fn test_wait_applied_multiple_waiters_all_wake_correctly() { - use std::sync::atomic::{AtomicBool, Ordering}; + use std::sync::atomic::AtomicBool; + use std::sync::atomic::Ordering; let handler = Arc::new(DefaultStateMachineHandler::::new( 1, diff --git a/d-engine-core/src/state_machine_handler/mod.rs b/d-engine-core/src/state_machine_handler/mod.rs index df63031b..94bfc5c8 100644 --- a/d-engine-core/src/state_machine_handler/mod.rs +++ b/d-engine-core/src/state_machine_handler/mod.rs @@ -37,6 +37,7 @@ mod default_state_machine_handler; mod snapshot_assembler; mod snapshot_policy; mod worker; +mod worker_handle; mod writer; pub use command::*; @@ -44,6 +45,7 @@ pub use default_state_machine_handler::*; pub(crate) use snapshot_assembler::*; pub use snapshot_policy::*; pub use worker::*; +pub use worker_handle::*; pub use writer::*; #[cfg(test)] @@ -55,9 +57,10 @@ mod snapshot_assembler_test; #[cfg(test)] mod wait_applied_test; #[cfg(test)] +mod worker_handle_test; +#[cfg(test)] mod worker_test; -use crate::client::KvEntry; use async_trait::async_trait; use d_engine_proto::server::storage::SnapshotChunk; use d_engine_proto::server::storage::SnapshotMetadata; @@ -69,6 +72,7 @@ use super::NewCommitData; use crate::ApplyResult; use crate::Result; use crate::TypeConfig; +use crate::client::KvEntry; #[cfg_attr(any(test, feature = "__test_support"), automock)] #[async_trait] @@ -165,7 +169,7 @@ where /// Destructive state-machine mutation — deliberately a **separate trait** from /// `StateMachineHandler`, not two methods on it. `StateMachineHandler` stays importable -/// and callable from anywhere (role layer, CommitHandler, etc.); this trait is +/// and callable from anywhere (role layer, Raft loop, etc.); this trait is /// module-restricted so only code inside `state_machine_handler` (i.e. `StateMachineWorker`) /// can even name it, let alone call it. This is real compile-time enforcement, not convention. #[cfg_attr(any(test, feature = "__test_support"), automock)] @@ -196,4 +200,8 @@ where /// `StateMachineCommand::CaptureLocalSnapshot`) so this read is ordered against /// concurrent `apply_chunk`/`apply_snapshot_from_file` calls. async fn capture_local_snapshot(&self) -> Result; + + /// Closes the state machine (see `StateMachine::close`). The worker calls it as its last + /// act, so nothing is applied after the storage is released. Synchronous and idempotent. + fn close(&self) -> Result<()>; } diff --git a/d-engine-core/src/state_machine_handler/snapshot_policy/log_size.rs b/d-engine-core/src/state_machine_handler/snapshot_policy/log_size.rs index da59406e..0fbeb9f5 100644 --- a/d-engine-core/src/state_machine_handler/snapshot_policy/log_size.rs +++ b/d-engine-core/src/state_machine_handler/snapshot_policy/log_size.rs @@ -4,6 +4,7 @@ use std::sync::atomic::AtomicBool; use std::sync::atomic::AtomicU64; use std::sync::atomic::Ordering; + use tracing::error; use tracing::trace; use tracing::warn; diff --git a/d-engine-core/src/state_machine_handler/wait_applied_test.rs b/d-engine-core/src/state_machine_handler/wait_applied_test.rs index 3bd1861b..8f91a9ff 100644 --- a/d-engine-core/src/state_machine_handler/wait_applied_test.rs +++ b/d-engine-core/src/state_machine_handler/wait_applied_test.rs @@ -1,5 +1,6 @@ use std::sync::Arc; -use std::time::{Duration, Instant}; +use std::time::Duration; +use std::time::Instant; use super::DefaultStateMachineHandler; use super::StateMachineHandler; diff --git a/d-engine-core/src/state_machine_handler/worker.rs b/d-engine-core/src/state_machine_handler/worker.rs index d4aefcd4..b63e689d 100644 --- a/d-engine-core/src/state_machine_handler/worker.rs +++ b/d-engine-core/src/state_machine_handler/worker.rs @@ -1,17 +1,23 @@ use std::sync::Arc; -use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::atomic::AtomicU64; +use std::sync::atomic::Ordering; use std::time::Duration; -use tokio::sync::{mpsc, watch}; + +use tokio::sync::mpsc; +use tracing::debug; +use tracing::error; +use tracing::info; use tracing::warn; -use tracing::{debug, error, info}; +use crate::InternalEvent; +use crate::PreparedSnapshot; use crate::Result; +use crate::SnapshotApplyResult; +use crate::SnapshotError; use crate::StateMachineCommand; use crate::StateMachineWriterOps; use crate::TypeConfig; use crate::alias::SMWOF; -use crate::{InternalEvent, SnapshotApplyResult}; -use crate::{PreparedSnapshot, SnapshotError}; /// Shared, read-only-after-construction context `handle_command` needs — split out /// from `sm_apply_rx` (which needs `&mut` exclusive access in the loop) purely to @@ -28,8 +34,8 @@ struct HandlerContext { /// State Machine Worker /// -/// Decouples state machine apply operations from CommitHandler. -/// Runs in independent task, allowing apply_chunk I/O to not block commit processing. +/// Decouples state machine apply operations from the Raft loop. +/// Runs on its own thread, so apply_chunk I/O never blocks the Raft loop. /// /// Sole authority for ordering and outcome of state-machine mutations and snapshot /// lifecycle decisions (ApplyEntries, InstallSnapshot, CaptureLocalSnapshot — see @@ -41,16 +47,15 @@ struct HandlerContext { /// /// Holds its own clone of `sm_apply_tx` for that self-report, so the channel never /// closes on its own — dropping every external `StateMachineCommandSender` does NOT -/// stop this Worker. Shutdown must go through `shutdown_signal`. +/// stop this Worker. Shutdown goes through `StateMachineCommand::Shutdown`. /// /// InstallSnapshot always waits for the capture/install lock inline — while it /// waits, subsequent queued commands (including ApplyEntries) wait behind it too. /// Only CaptureLocalSnapshot is guaranteed not to block the queue. /// -/// On shutdown: -/// - Drains whatever's currently queued in sm_apply_rx (non-blocking) before exiting -/// - Completes all pending work before shutdown -/// - This ensures no data loss and all results are propagated to Leader +/// On `Shutdown`: +/// - everything queued before it has already been applied (the queue is FIFO); +/// - commands queued after it are ignored; a still-running capture is waited for. pub struct StateMachineWorker { /// Executes the actual mutations. state_machine_writer: Arc>, @@ -68,8 +73,6 @@ pub struct StateMachineWorker { capture_cleanup_lock: Arc>, /// Bumped on each successful install; lets a capture detect it was superseded. install_epoch: Arc, - /// Signals graceful shutdown. - shutdown_signal: watch::Receiver<()>, /// For log lines only. node_id: u32, } @@ -81,7 +84,6 @@ impl StateMachineWorker { sm_apply_tx: mpsc::UnboundedSender, sm_apply_rx: mpsc::UnboundedReceiver, internal_event_tx: mpsc::UnboundedSender, - shutdown_signal: watch::Receiver<()>, ) -> Self { Self { state_machine_writer, @@ -91,7 +93,6 @@ impl StateMachineWorker { capture_install_lock: Arc::new(tokio::sync::Mutex::new(())), capture_cleanup_lock: Arc::new(tokio::sync::Mutex::new(())), install_epoch: Arc::new(AtomicU64::new(0)), - shutdown_signal, node_id, } } @@ -102,11 +103,9 @@ impl StateMachineWorker { pub async fn run(self) -> Result<()> { debug!("[Node-{}] SM Worker started", self.node_id); - let mut sm_apply_rx = self.sm_apply_rx; - let mut shutdown_signal = self.shutdown_signal.clone(); - let mut shutdown = false; let node_id = self.node_id; + let mut sm_apply_rx = self.sm_apply_rx; let ctx = HandlerContext { node_id, state_machine_writer: self.state_machine_writer, @@ -117,41 +116,36 @@ impl StateMachineWorker { install_epoch: self.install_epoch, }; - // Tracks the in-flight capture task so shutdown can wait for it (#436). - let mut in_flight_capture: Option> = None; + let result = Self::drain_until_shutdown(&ctx, &mut sm_apply_rx).await; - loop { - tokio::select! { - // Shutdown signal received: drain remaining entries before exit - _ = shutdown_signal.changed() => { - info!("[Node-{}] SM Worker shutdown signal received", node_id); - shutdown = true; - } + // Last act, on every exit path: nothing is applied after this. + let closed = ctx.state_machine_writer.close(); + if let Err(e) = &closed { + error!( + "[Node-{}] SM Worker failed to close the state machine: {:?}", + node_id, e + ); + } + info!("[Node-{}] SM Worker shutdown complete", node_id); + // A drain failure is the root cause; a close failure is reported only when the drain + // itself was fine. + result.and(closed) + } - // Normal apply processing - command = sm_apply_rx.recv() => { - match command { - Some(command) => { - Self::handle_command(&ctx, &mut in_flight_capture, command).await?; - } - None => { - debug!("[Node-{}] SM Worker: apply channel closed", node_id); - return Ok(()); - } - } - } - } + /// Applies queued commands in order until `Shutdown`, then waits for a running capture. + async fn drain_until_shutdown( + ctx: &HandlerContext, + sm_apply_rx: &mut mpsc::UnboundedReceiver, + ) -> Result<()> { + // Tracks the in-flight capture task so shutdown can wait for it (#436). + let mut in_flight_capture: Option> = None; - // Exit loop after shutdown signal to start graceful drain - if shutdown { + while let Some(command) = sm_apply_rx.recv().await { + if matches!(command, StateMachineCommand::Shutdown) { + info!("[Node-{}] SM Worker shutdown command received", ctx.node_id); break; } - } - - // Graceful drain: process remaining entries after shutdown signal - debug!("[Node-{}] SM Worker draining pending applies", node_id); - while let Ok(command) = sm_apply_rx.try_recv() { - Self::handle_command(&ctx, &mut in_flight_capture, command).await?; + Self::handle_command(ctx, &mut in_flight_capture, command).await?; } // Wait for a still-running capture rather than leaving it orphaned — see @@ -162,11 +156,10 @@ impl StateMachineWorker { { error!( "[Node-{}] in-flight capture task panicked during shutdown: {:?}", - node_id, e + ctx.node_id, e ); } - info!("[Node-{}] SM Worker shutdown complete", node_id); Ok(()) } @@ -178,7 +171,13 @@ impl StateMachineWorker { command: StateMachineCommand, ) -> Result<()> { match command { - StateMachineCommand::ApplyEntries { entries } => { + StateMachineCommand::Shutdown => { + // Acted on by `run()`; nothing to do here. + Ok(()) + } + StateMachineCommand::ApplyEntries { entries, sent_at } => { + metrics::histogram!("core.raft.sm_worker.queue_wait_ms") + .record(sent_at.elapsed().as_secs_f64() * 1_000.0); Self::apply_and_notify( &ctx.node_id, &ctx.state_machine_writer, @@ -320,6 +319,7 @@ impl StateMachineWorker { if let Err(e) = internal_event_tx.send(InternalEvent::ApplyCompleted { last_index: last.index, results, + sent_at: tokio::time::Instant::now(), }) { error!( "[Node-{}] Failed to send ApplyCompleted event: {:?}", @@ -351,6 +351,11 @@ impl StateMachineWorker { } } } + + /// Closes the state machine of a worker that was never started. + pub(super) fn close(self) -> Result<()> { + self.state_machine_writer.close() + } } /// Runs `call` to completion, warning (not cancelling) if it takes longer than diff --git a/d-engine-core/src/state_machine_handler/worker_handle.rs b/d-engine-core/src/state_machine_handler/worker_handle.rs new file mode 100644 index 00000000..d6bf996d --- /dev/null +++ b/d-engine-core/src/state_machine_handler/worker_handle.rs @@ -0,0 +1,193 @@ +//! Owns the life of the state machine worker: built unstarted, started once, stopped through +//! its own queue so everything queued before the stop request is applied first. +//! +//! The handle's sender is private and only ever sends [`StateMachineCommand::Shutdown`]: whoever +//! owns the handle can start and stop the worker but cannot feed it. + +use std::fmt; +use std::io; +use std::sync::Mutex; +use std::sync::MutexGuard; +use std::sync::PoisonError; +use std::time::Duration; + +use tokio::sync::mpsc; +use tokio::sync::watch; +use tracing::error; +use tracing::info; + +use super::StateMachineCommand; +use super::StateMachineWorker; +use crate::TypeConfig; + +/// Why [`SmWorkerHandle::stop_and_join`] could not confirm a clean stop. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum StopError { + /// The worker did not finish within the given time. Its thread keeps running and a later + /// call may wait for it again. + TimedOut(Duration), + /// The worker thread panicked. + Panicked, + /// The worker ended with an error (a failed apply, or a failed `close`). + Failed(String), +} + +impl fmt::Display for StopError { + fn fmt( + &self, + f: &mut fmt::Formatter<'_>, + ) -> fmt::Result { + match self { + Self::TimedOut(t) => write!(f, "state machine worker did not stop within {t:?}"), + Self::Panicked => write!(f, "state machine worker thread panicked"), + Self::Failed(e) => write!(f, "state machine worker failed: {e}"), + } + } +} + +impl std::error::Error for StopError {} + +enum State { + Pending(StateMachineWorker), + Running { + join: std::thread::JoinHandle>, + done: watch::Receiver<()>, + }, + Stopped, +} + +impl State { + fn name(&self) -> &'static str { + match self { + Self::Pending(_) => "Pending", + Self::Running { .. } => "Running", + Self::Stopped => "Stopped", + } + } +} + +pub struct SmWorkerHandle { + /// Only ever used to send `Shutdown`. + tx: mpsc::UnboundedSender, + state: Mutex>, +} + +impl SmWorkerHandle { + /// Wraps an unstarted worker. `tx` must be a sender of the queue the worker reads from. + pub fn new( + worker: StateMachineWorker, + tx: mpsc::UnboundedSender, + ) -> Self { + Self { + tx, + state: Mutex::new(State::Pending(worker)), + } + } + + /// Starts the worker on a dedicated thread (`sm-apply-`) that drives it on the + /// current Tokio runtime. Only the first call has an effect. + /// + /// # Errors + /// No Tokio runtime is available, or the OS refused to create the thread. + pub fn start(&self) -> io::Result<()> { + let mut state = self.lock(); + let State::Pending(_) = &*state else { + return Ok(()); + }; + let runtime = tokio::runtime::Handle::try_current() + .map_err(|e| io::Error::other(format!("no Tokio runtime to run the worker: {e}")))?; + let State::Pending(worker) = std::mem::replace(&mut *state, State::Stopped) else { + unreachable!("checked Pending above while holding the lock"); + }; + + let node_id = worker.node_id(); + // Dropped when the thread ends, however it ends (return or panic): the closed channel + // is the "done" signal, `stop_and_join` waits for it. + let (done_tx, done_rx) = watch::channel(()); + let join = + std::thread::Builder::new().name(format!("sm-apply-{node_id}")).spawn(move || { + let _done = done_tx; + runtime.block_on(async move { + let result = worker.run().await; + match &result { + Ok(()) => info!("[Node-{node_id}] state machine worker exit"), + Err(e) => { + error!("[Node-{node_id}] state machine worker exit with error: {e:?}") + } + } + result + }) + })?; + *state = State::Running { + join, + done: done_rx, + }; + Ok(()) + } + + /// Asks the worker to stop after everything already queued, and waits up to `timeout` for + /// its thread to end. A worker that was never started is simply discarded. Safe to call more + /// than once; after a clean stop it returns `Ok` immediately. + pub async fn stop_and_join( + &self, + timeout: Duration, + ) -> Result<(), StopError> { + // Fails only if the worker is already gone, which is the goal anyway. + let _ = self.tx.send(StateMachineCommand::Shutdown); + + let mut done = { + let mut state = self.lock(); + match &*state { + State::Running { done, .. } => done.clone(), + State::Pending(_) | State::Stopped => { + if let State::Pending(worker) = std::mem::replace(&mut *state, State::Stopped) { + // never started: close what it would have closed + return worker.close().map_err(|e| StopError::Failed(format!("{e:?}"))); + } + return Ok(()); + } + } + }; + + if tokio::time::timeout(timeout, done.changed()).await.is_err() { + return Err(StopError::TimedOut(timeout)); + } + + match std::mem::replace(&mut *self.lock(), State::Stopped) { + State::Running { join, .. } => join + .join() + .map_err(|_| StopError::Panicked)? + .map_err(|e| StopError::Failed(format!("{e:?}"))), + State::Pending(_) | State::Stopped => Ok(()), + } + } + + fn lock(&self) -> MutexGuard<'_, State> { + // The state only moves between plain variants, so a poisoned lock is still consistent. + self.state.lock().unwrap_or_else(PoisonError::into_inner) + } +} + +impl Drop for SmWorkerHandle { + /// Last resort for an owner that is dropped without calling `stop_and_join` (task aborted, + /// panic): ask the worker to stop. Does not wait. + fn drop(&mut self) { + let _ = self.tx.send(StateMachineCommand::Shutdown); + if let State::Pending(worker) = std::mem::replace( + self.state.get_mut().unwrap_or_else(PoisonError::into_inner), + State::Stopped, + ) && let Err(e) = worker.close() + { + error!("state machine close failed for a worker that never started: {e:?}"); + } + } +} + +impl fmt::Debug for SmWorkerHandle { + fn fmt( + &self, + f: &mut fmt::Formatter<'_>, + ) -> fmt::Result { + f.debug_struct("SmWorkerHandle").field("state", &self.lock().name()).finish() + } +} diff --git a/d-engine-core/src/state_machine_handler/worker_handle_test.rs b/d-engine-core/src/state_machine_handler/worker_handle_test.rs new file mode 100644 index 00000000..15b76cc4 --- /dev/null +++ b/d-engine-core/src/state_machine_handler/worker_handle_test.rs @@ -0,0 +1,286 @@ +//! Tests for `SmWorkerHandle`: starting the worker thread, stopping it through its queue, and the +//! cases where the stop does not go to plan (never started, stuck, panicked, owner dropped). +//! +//! The worker is real; only the state machine writer is mocked. The multi-thread runtime is +//! deliberate: the worker thread drives its future on the runtime of the test. + +use std::sync::Arc; +use std::sync::Mutex; +use std::time::Duration; +use std::time::Instant; + +use d_engine_proto::common::Entry; +use tokio::sync::mpsc; + +use super::SmWorkerHandle; +use super::StateMachineCommand; +use super::StateMachineWorker; +use super::StopError; +use crate::ApplyResult; +use crate::InternalEvent; +use crate::MockStateMachineWriterOps; +use crate::MockTypeConfig; + +type Applied = Arc>>; + +/// A handle around a real worker whose writer records the indices it applies. The test keeps its +/// own sender (`tx`) to feed the queue the way the Raft loop would. +struct Rig { + handle: SmWorkerHandle, + tx: mpsc::UnboundedSender, + applied: Applied, + // Kept alive so the worker's sends of apply results never fail. + _events: mpsc::UnboundedReceiver, +} + +fn rig_with( + configure: impl FnOnce(&mut MockStateMachineWriterOps, Applied) +) -> Rig { + let applied: Applied = Arc::default(); + let mut writer = MockStateMachineWriterOps::new(); + configure(&mut writer, applied.clone()); + // Added last, so a test's own `expect_close` keeps precedence. + writer.expect_close().returning(|| Ok(())); + + let (tx, rx) = mpsc::unbounded_channel(); + let (events_tx, events_rx) = mpsc::unbounded_channel(); + let worker = + StateMachineWorker::::new(1, Arc::new(writer), tx.clone(), rx, events_tx); + Rig { + handle: SmWorkerHandle::new(worker, tx.clone()), + tx, + applied, + _events: events_rx, + } +} + +/// A writer that records every applied index and succeeds. +fn recording( + writer: &mut MockStateMachineWriterOps, + applied: Applied, +) { + writer.expect_apply_chunk().returning(move |entries| { + applied.lock().unwrap().extend(entries.iter().map(|e| e.index)); + Ok(entries.iter().map(|e| ApplyResult::success(e.index)).collect()) + }); +} + +fn apply(index: u64) -> StateMachineCommand { + StateMachineCommand::ApplyEntries { + sent_at: tokio::time::Instant::now(), + entries: vec![Entry { + index, + term: 1, + payload: Some(d_engine_proto::common::EntryPayload::command( + bytes::Bytes::from("data"), + )), + }], + } +} + +async fn eventually(mut condition: impl FnMut() -> bool) -> bool { + let deadline = Instant::now() + Duration::from_secs(2); + while Instant::now() < deadline { + if condition() { + return true; + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + condition() +} + +fn state_of(handle: &SmWorkerHandle) -> String { + format!("{handle:?}") +} + +/// Business scenario: a node starts. Purpose: `start` brings the worker to life on its own +/// thread, so entries the Raft loop queues are applied without anyone driving the worker. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_started_worker_applies_entries_queued_by_the_raft_loop() { + let rig = rig_with(recording); + rig.handle.start().expect("start"); + + rig.tx.send(apply(1)).unwrap(); + rig.tx.send(apply(2)).unwrap(); + + assert!( + eventually(|| *rig.applied.lock().unwrap() == vec![1, 2]).await, + "queued entries were not applied: {:?}", + rig.applied.lock().unwrap() + ); + assert!(state_of(&rig.handle).contains("Running")); + rig.handle.stop_and_join(Duration::from_secs(2)).await.expect("stop"); +} + +/// Business scenario: the node shuts down right after the Raft loop queued its last committed +/// batches. Purpose: stopping applies everything queued before the stop request, in order, and +/// only then ends the thread. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_stop_and_join_applies_everything_queued_before_it() { + let rig = rig_with(recording); + rig.handle.start().expect("start"); + for index in 1..=3 { + rig.tx.send(apply(index)).unwrap(); + } + + rig.handle.stop_and_join(Duration::from_secs(2)).await.expect("clean stop"); + + assert_eq!(*rig.applied.lock().unwrap(), vec![1, 2, 3]); + assert!(state_of(&rig.handle).contains("Stopped")); +} + +/// Business scenario: the node's shutdown path runs unconditionally, so it may run twice and may +/// follow a start that was already done. Purpose: a second `start` and a second `stop_and_join` +/// are harmless. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_start_and_stop_and_join_are_idempotent() { + let rig = rig_with(recording); + rig.handle.start().expect("start"); + rig.handle.start().expect("second start is a no-op"); + assert!(state_of(&rig.handle).contains("Running")); + + rig.handle.stop_and_join(Duration::from_secs(2)).await.expect("first stop"); + rig.handle.stop_and_join(Duration::from_secs(2)).await.expect("second stop"); + assert!(state_of(&rig.handle).contains("Stopped")); +} + +/// Business scenario: the state machine is wedged in a slow storage call during shutdown. +/// Purpose: `stop_and_join` gives up at the timeout instead of hanging the node, says so, keeps +/// the thread's handle, and a later call succeeds once the worker is free again. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_stop_and_join_times_out_on_a_stuck_worker_and_can_be_retried() { + let (release_tx, release_rx) = std::sync::mpsc::channel::<()>(); + let release_rx = Mutex::new(release_rx); + let (started_tx, started_rx) = std::sync::mpsc::channel::<()>(); + let started_tx = Mutex::new(started_tx); + let rig = rig_with(|writer, _| { + writer.expect_apply_chunk().returning(move |entries| { + let _ = started_tx.lock().unwrap().send(()); + let _ = release_rx.lock().unwrap().recv(); // blocks the worker thread + Ok(entries.iter().map(|e| ApplyResult::success(e.index)).collect()) + }); + }); + rig.handle.start().expect("start"); + rig.tx.send(apply(1)).unwrap(); + started_rx.recv_timeout(Duration::from_secs(2)).expect("apply started"); + + let timeout = Duration::from_millis(100); + assert_eq!( + rig.handle.stop_and_join(timeout).await, + Err(StopError::TimedOut(timeout)) + ); + assert!( + state_of(&rig.handle).contains("Running"), + "thread handle must be kept" + ); + + release_tx.send(()).unwrap(); + rig.handle + .stop_and_join(Duration::from_secs(2)) + .await + .expect("a free worker stops on the retry"); +} + +/// Business scenario: start-up fails before the node ever starts its worker. Purpose: stopping +/// a handle that was never started succeeds, never runs the worker, and a later `start` cannot +/// revive it. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_stop_before_start_discards_the_worker() { + // No expectation on apply_chunk: any call to it would panic. + let rig = rig_with(|_, _| {}); + assert!(state_of(&rig.handle).contains("Pending")); + + rig.handle.stop_and_join(Duration::from_secs(1)).await.expect("stop"); + assert!(state_of(&rig.handle).contains("Stopped")); + + rig.handle.start().expect("start after stop is a no-op"); + assert!(state_of(&rig.handle).contains("Stopped")); +} + +/// Business scenario: the task running the node is aborted, or panics, so the unconditional +/// shutdown path never runs and the handle is simply dropped. Purpose: dropping the handle still +/// stops the worker, after it applied what was queued, without waiting for it. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_dropping_the_handle_stops_the_worker_after_what_was_queued() { + let rig = rig_with(recording); + rig.handle.start().expect("start"); + rig.tx.send(apply(1)).unwrap(); + rig.tx.send(apply(2)).unwrap(); + + let Rig { + handle, + tx, + applied, + .. + } = rig; + drop(handle); + + // The receiver is dropped when the worker thread ends. + assert!( + eventually(|| tx.is_closed()).await, + "the worker did not stop after its handle was dropped" + ); + assert_eq!(*applied.lock().unwrap(), vec![1, 2]); +} + +/// Business scenario: a bug in a custom state machine panics inside `apply_chunk`. Purpose: the +/// node's shutdown path learns about it (`Panicked`) instead of believing the stop was clean. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_stop_and_join_reports_a_panicked_worker() { + let rig = rig_with(|writer, _| { + writer.expect_apply_chunk().returning(|_| panic!("state machine bug")); + }); + rig.handle.start().expect("start"); + rig.tx.send(apply(1)).unwrap(); + + assert_eq!( + rig.handle.stop_and_join(Duration::from_secs(2)).await, + Err(StopError::Panicked) + ); +} + +/// Business scenario: the node shuts down while the disk is full, so the state machine cannot +/// persist its final state in `close`. Purpose: the node's shutdown path learns about it +/// (`Failed`, carrying the reason) instead of believing the stop was clean. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_stop_and_join_reports_a_failed_close() { + let rig = rig_with(|writer, _| { + writer + .expect_close() + .times(1) + .returning(|| Err(crate::SnapshotError::OperationFailed("disk full".into()).into())); + }); + rig.handle.start().expect("start"); + + let result = rig.handle.stop_and_join(Duration::from_secs(2)).await; + + match result { + Err(StopError::Failed(reason)) => { + assert!( + reason.contains("disk full"), + "reason lost the cause: {reason}" + ) + } + other => panic!("a failed close must be reported as Failed, got {other:?}"), + } +} + +/// Same as above for a worker that was never started: its state machine is still closed on stop, +/// and a failure there must not be swallowed just because no thread ever ran. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_stop_before_start_reports_a_failed_close() { + let rig = rig_with(|writer, _| { + writer + .expect_close() + .times(1) + .returning(|| Err(crate::SnapshotError::OperationFailed("disk full".into()).into())); + }); + + let result = rig.handle.stop_and_join(Duration::from_secs(1)).await; + + assert!( + matches!(&result, Err(StopError::Failed(reason)) if reason.contains("disk full")), + "got {result:?}" + ); + assert!(state_of(&rig.handle).contains("Stopped")); +} diff --git a/d-engine-core/src/state_machine_handler/worker_test.rs b/d-engine-core/src/state_machine_handler/worker_test.rs index 018e1806..039178f1 100644 --- a/d-engine-core/src/state_machine_handler/worker_test.rs +++ b/d-engine-core/src/state_machine_handler/worker_test.rs @@ -8,17 +8,25 @@ use std::sync::Arc; -use d_engine_proto::common::{Entry, LogId}; +use d_engine_proto::common::Entry; +use d_engine_proto::common::LogId; use d_engine_proto::server::storage::SnapshotMetadata; +use tokio::sync::mpsc; use tokio::sync::oneshot; -use tokio::sync::{mpsc, watch}; use super::StateMachineWorker; -use crate::{ - ApplyResult, CapturedLocalSnapshot, ConsensusError, Error, InternalEvent, - MockStateMachineWriterOps, MockTypeConfig, OwnedSnapshotDir, PreparedSnapshot, - SnapshotApplyResult, SnapshotError, StateMachineCommand, -}; +use crate::ApplyResult; +use crate::CapturedLocalSnapshot; +use crate::ConsensusError; +use crate::Error; +use crate::InternalEvent; +use crate::MockStateMachineWriterOps; +use crate::MockTypeConfig; +use crate::OwnedSnapshotDir; +use crate::PreparedSnapshot; +use crate::SnapshotApplyResult; +use crate::SnapshotError; +use crate::StateMachineCommand; /// A real, throwaway directory adopted by a fresh `OwnedSnapshotDir` — stands in for /// the export `capture_local_snapshot` would have produced. `OwnedSnapshotDir::from_existing` @@ -114,6 +122,7 @@ fn create_test_entry( #[tokio::test] async fn test_apply_success_sends_apply_completed() { let mut mock_smw = MockStateMachineWriterOps::new(); + mock_smw.expect_close().returning(|| Ok(())); mock_smw.expect_apply_chunk().times(1).returning(|entries| { let results: Vec = entries.iter().map(|e| crate::ApplyResult::success(e.index)).collect(); @@ -122,7 +131,6 @@ async fn test_apply_success_sends_apply_completed() { let (sm_apply_tx, sm_apply_rx) = mpsc::unbounded_channel(); let (internal_event_tx, mut internal_event_rx) = mpsc::unbounded_channel::(); - let (_shutdown_tx, shutdown_rx) = watch::channel(()); let worker = StateMachineWorker::::new( 1, @@ -130,7 +138,6 @@ async fn test_apply_success_sends_apply_completed() { sm_apply_tx.clone(), sm_apply_rx, internal_event_tx, - shutdown_rx, ); tokio::spawn(async move { @@ -139,6 +146,7 @@ async fn test_apply_success_sends_apply_completed() { sm_apply_tx .send(StateMachineCommand::ApplyEntries { + sent_at: tokio::time::Instant::now(), entries: vec![create_test_entry(1, 1), create_test_entry(2, 1)], }) .unwrap(); @@ -152,6 +160,7 @@ async fn test_apply_success_sends_apply_completed() { Ok(Some(InternalEvent::ApplyCompleted { last_index, results, + .. })) => { assert_eq!(last_index, 2, "last_index should match last entry"); assert_eq!(results.len(), 2, "results should contain 2 entries"); @@ -182,6 +191,7 @@ async fn test_apply_success_sends_apply_completed() { #[tokio::test] async fn test_apply_failure_sends_fatal_error() { let mut mock_smw = MockStateMachineWriterOps::new(); + mock_smw.expect_close().returning(|| Ok(())); mock_smw.expect_apply_chunk().times(1).returning(|_| { Err(Error::Fatal( "Disk failure - cannot write to storage".to_string(), @@ -190,7 +200,6 @@ async fn test_apply_failure_sends_fatal_error() { let (sm_apply_tx, sm_apply_rx) = mpsc::unbounded_channel(); let (internal_event_tx, mut internal_event_rx) = mpsc::unbounded_channel::(); - let (_shutdown_tx, shutdown_rx) = watch::channel(()); let worker = StateMachineWorker::::new( 1, @@ -198,13 +207,13 @@ async fn test_apply_failure_sends_fatal_error() { sm_apply_tx.clone(), sm_apply_rx, internal_event_tx, - shutdown_rx, ); let worker_handle = tokio::spawn(async move { worker.run().await }); sm_apply_tx .send(StateMachineCommand::ApplyEntries { + sent_at: tokio::time::Instant::now(), entries: vec![create_test_entry(1, 1)], }) .unwrap(); @@ -237,8 +246,8 @@ async fn test_apply_failure_sends_fatal_error() { /// Test: Shutdown - Drains Remaining Entries /// /// # Test Objective -/// Verify that when shutdown signal is received, SM Worker drains all -/// pending entries before exit (no data loss). +/// Verify that when the `Shutdown` command is received, SM Worker has applied all +/// entries queued before it, and propagated their results, before it exits (no data loss). /// /// # Success Criteria /// - All 3 entries applied (verified via ApplyCompleted count on internal_event_rx) @@ -246,6 +255,7 @@ async fn test_apply_failure_sends_fatal_error() { #[tokio::test] async fn test_shutdown_drains_remaining_entries() { let mut mock_smw = MockStateMachineWriterOps::new(); + mock_smw.expect_close().returning(|| Ok(())); mock_smw.expect_apply_chunk().times(3).returning(|entries| { let results: Vec = entries.iter().map(|e| crate::ApplyResult::success(e.index)).collect(); @@ -254,7 +264,6 @@ async fn test_shutdown_drains_remaining_entries() { let (sm_apply_tx, sm_apply_rx) = mpsc::unbounded_channel(); let (internal_event_tx, mut internal_event_rx) = mpsc::unbounded_channel::(); - let (shutdown_tx, shutdown_rx) = watch::channel(()); let worker = StateMachineWorker::::new( 1, @@ -262,7 +271,6 @@ async fn test_shutdown_drains_remaining_entries() { sm_apply_tx.clone(), sm_apply_rx, internal_event_tx, - shutdown_rx, ); let worker_handle = tokio::spawn(async move { worker.run().await }); @@ -270,13 +278,14 @@ async fn test_shutdown_drains_remaining_entries() { for i in 1..=3 { sm_apply_tx .send(StateMachineCommand::ApplyEntries { + sent_at: tokio::time::Instant::now(), entries: vec![create_test_entry(i, 1)], }) .unwrap(); } tokio::time::sleep(std::time::Duration::from_millis(10)).await; - shutdown_tx.send(()).unwrap(); + sm_apply_tx.send(StateMachineCommand::Shutdown).unwrap(); let mut apply_count = 0; while apply_count < 3 { @@ -302,18 +311,79 @@ async fn test_shutdown_drains_remaining_entries() { assert!(result.is_ok(), "Worker should exit cleanly after shutdown"); } +/// Test: Shutdown command - applies everything queued before it, then stops +/// +/// # Business scenario +/// The node is shutting down. The Raft loop has already handed the last committed batches to +/// the state machine worker, and the node's owner then puts `Shutdown` into the same queue. +/// +/// # Purpose +/// Queue order alone must decide when the worker stops: everything queued BEFORE `Shutdown` is +/// applied, in order; the worker then exits; anything queued AFTER it is never applied. +/// +/// # Success Criteria +/// - Entries 1..=3 applied, in order +/// - The entry queued after `Shutdown` is not applied +/// - `run()` returns `Ok(())` +#[tokio::test] +async fn test_shutdown_command_applies_everything_queued_before_it_then_stops() { + let applied = Arc::new(std::sync::Mutex::new(Vec::::new())); + let sink = applied.clone(); + let mut mock_smw = MockStateMachineWriterOps::new(); + mock_smw.expect_close().returning(|| Ok(())); + mock_smw.expect_apply_chunk().returning(move |entries| { + sink.lock().unwrap().extend(entries.iter().map(|e| e.index)); + Ok(entries.iter().map(|e| ApplyResult::success(e.index)).collect()) + }); + + let (sm_apply_tx, sm_apply_rx) = mpsc::unbounded_channel(); + let (internal_event_tx, _internal_event_rx) = mpsc::unbounded_channel::(); + + let worker = StateMachineWorker::::new( + 1, + Arc::new(mock_smw), + sm_apply_tx.clone(), + sm_apply_rx, + internal_event_tx, + ); + let apply = |index| StateMachineCommand::ApplyEntries { + sent_at: tokio::time::Instant::now(), + entries: vec![create_test_entry(index, 1)], + }; + // Queued before the worker first runs, so the order is fixed: 1, 2, 3, Shutdown, 4. + for index in 1..=3 { + sm_apply_tx.send(apply(index)).unwrap(); + } + sm_apply_tx.send(StateMachineCommand::Shutdown).unwrap(); + sm_apply_tx.send(apply(4)).unwrap(); + + let result = tokio::time::timeout( + std::time::Duration::from_secs(2), + tokio::spawn(async move { worker.run().await }), + ) + .await + .expect("worker must stop when it reaches Shutdown, with no signal sent") + .expect("worker task must not panic"); + + assert!(result.is_ok(), "worker should exit cleanly on Shutdown"); + assert_eq!( + *applied.lock().unwrap(), + vec![1, 2, 3], + "applied entries must be exactly those queued before Shutdown, in order" + ); +} + /// #436: Worker now holds its own clone of `sm_apply_tx` (to route /// `LocalSnapshotReady` back to itself), so dropping every *external* clone no /// longer closes the channel — Worker's own clone keeps the sender count above /// zero for as long as it's running. This replaces the old /// `test_channel_closed_worker_exits`, which asserted the opposite (now-false) -/// behavior. Shutdown must go through `shutdown_signal` — see -/// `test_worker_exits_when_shutdown_sender_dropped_without_send` below for the -/// mechanism embedders actually rely on when they drop their handle without -/// calling `stop()`. +/// behavior. Shutdown goes through `StateMachineCommand::Shutdown`; a dropped +/// `SmWorkerHandle` sends it too (see `worker_handle_test`). #[tokio::test] async fn test_dropping_external_senders_alone_does_not_stop_worker() { let mut mock_smw = MockStateMachineWriterOps::new(); + mock_smw.expect_close().returning(|| Ok(())); mock_smw .expect_apply_chunk() .times(1) @@ -321,7 +391,6 @@ async fn test_dropping_external_senders_alone_does_not_stop_worker() { let (sm_apply_tx, sm_apply_rx) = mpsc::unbounded_channel(); let (internal_event_tx, internal_event_rx) = mpsc::unbounded_channel::(); - let (_shutdown_tx, shutdown_rx) = watch::channel(()); let worker = StateMachineWorker::::new( 1, @@ -329,7 +398,6 @@ async fn test_dropping_external_senders_alone_does_not_stop_worker() { sm_apply_tx.clone(), sm_apply_rx, internal_event_tx, - shutdown_rx, ); let worker_handle = tokio::spawn(async move { worker.run().await }); @@ -355,52 +423,6 @@ async fn test_dropping_external_senders_alone_does_not_stop_worker() { worker_handle.abort(); } -/// #436 / embedded-mode safety net: if the caller drops the `shutdown_signal` -/// Sender without ever calling `.send(())` — this is exactly what happens when -/// an embedder drops `EmbeddedEngine` without awaiting `stop()` first, since -/// `Inner::shutdown_tx` then drops as part of the struct's own teardown — Worker -/// must still exit. `tokio::sync::watch::Receiver::changed()` resolves (with an -/// error) once its Sender is dropped, and Worker's shutdown arm (`_ = -/// shutdown_signal.changed()`) doesn't discriminate Ok from Err, so this must be -/// sufficient on its own — independent of whether `sm_apply_tx` has any activity, -/// since Worker now permanently holds its own clone (see test above). -#[tokio::test] -async fn test_worker_exits_when_shutdown_sender_dropped_without_send() { - let mock_smw = MockStateMachineWriterOps::new(); - - let (sm_apply_tx, sm_apply_rx) = mpsc::unbounded_channel(); - let (internal_event_tx, _internal_event_rx) = mpsc::unbounded_channel::(); - let (shutdown_tx, shutdown_rx) = watch::channel(()); - - let worker = StateMachineWorker::::new( - 1, - Arc::new(mock_smw), - sm_apply_tx.clone(), - sm_apply_rx, - internal_event_tx, - shutdown_rx, - ); - let worker_handle = tokio::spawn(async move { worker.run().await }); - - // Keep sm_apply_tx alive throughout, unlike the old channel-closed test — - // this proves shutdown works even when the command channel is untouched. - let _keep_alive = sm_apply_tx; - - // Drop the sender without ever calling .send(()) — no explicit shutdown signal. - drop(shutdown_tx); - - match tokio::time::timeout(std::time::Duration::from_millis(200), worker_handle).await { - Ok(Ok(Ok(()))) => {} - Ok(Ok(Err(e))) => panic!("Worker returned error: {e:?}"), - Ok(Err(e)) => panic!("Worker task panicked: {e:?}"), - Err(_) => panic!( - "Worker did not exit when shutdown_signal's Sender was dropped \ - without .send() — embedders who drop EmbeddedEngine without \ - calling stop() would leak the sm-apply OS thread forever" - ), - } -} - /// Test: Multiple Batches - Sequential Processing /// /// Verify SM Worker processes batches in order, last_index increases monotonically. @@ -410,6 +432,7 @@ async fn test_worker_exits_when_shutdown_sender_dropped_without_send() { #[tokio::test] async fn test_multiple_batches_sequential_processing() { let mut mock_smw = MockStateMachineWriterOps::new(); + mock_smw.expect_close().returning(|| Ok(())); mock_smw.expect_apply_chunk().times(3).returning(|entries| { let results: Vec = entries.iter().map(|e| crate::ApplyResult::success(e.index)).collect(); @@ -418,7 +441,6 @@ async fn test_multiple_batches_sequential_processing() { let (sm_apply_tx, sm_apply_rx) = mpsc::unbounded_channel(); let (internal_event_tx, mut internal_event_rx) = mpsc::unbounded_channel::(); - let (_shutdown_tx, shutdown_rx) = watch::channel(()); let worker = StateMachineWorker::::new( 1, @@ -426,7 +448,6 @@ async fn test_multiple_batches_sequential_processing() { sm_apply_tx.clone(), sm_apply_rx, internal_event_tx, - shutdown_rx, ); tokio::spawn(async move { @@ -435,16 +456,19 @@ async fn test_multiple_batches_sequential_processing() { sm_apply_tx .send(StateMachineCommand::ApplyEntries { + sent_at: tokio::time::Instant::now(), entries: vec![create_test_entry(1, 1), create_test_entry(2, 1)], }) .unwrap(); sm_apply_tx .send(StateMachineCommand::ApplyEntries { + sent_at: tokio::time::Instant::now(), entries: vec![create_test_entry(3, 1), create_test_entry(4, 1)], }) .unwrap(); sm_apply_tx .send(StateMachineCommand::ApplyEntries { + sent_at: tokio::time::Instant::now(), entries: vec![create_test_entry(5, 1), create_test_entry(6, 1)], }) .unwrap(); @@ -487,6 +511,7 @@ async fn test_multiple_batches_sequential_processing() { #[tokio::test] async fn test_install_snapshot_applied_forwards_result_via_response() { let mut mock_smw = MockStateMachineWriterOps::new(); + mock_smw.expect_close().returning(|| Ok(())); mock_smw.expect_install_prepared_snapshot().times(1).returning(|_, _| { Ok(SnapshotApplyResult::Applied { last_included: LogId { index: 10, term: 2 }, @@ -495,7 +520,6 @@ async fn test_install_snapshot_applied_forwards_result_via_response() { let (sm_apply_tx, sm_apply_rx) = mpsc::unbounded_channel(); let (internal_event_tx, _internal_event_rx) = mpsc::unbounded_channel::(); - let (_shutdown_tx, shutdown_rx) = watch::channel(()); let worker = StateMachineWorker::::new( 1, @@ -503,7 +527,6 @@ async fn test_install_snapshot_applied_forwards_result_via_response() { sm_apply_tx.clone(), sm_apply_rx, internal_event_tx, - shutdown_rx, ); tokio::spawn(async move { let _ = worker.run().await; @@ -536,6 +559,7 @@ async fn test_install_snapshot_applied_forwards_result_via_response() { #[tokio::test] async fn test_install_snapshot_ignored_stale_forwards_as_ok_not_error() { let mut mock_smw = MockStateMachineWriterOps::new(); + mock_smw.expect_close().returning(|| Ok(())); mock_smw.expect_install_prepared_snapshot().times(1).returning(|_, _| { Ok(SnapshotApplyResult::IgnoredStale { current: LogId { index: 20, term: 3 }, @@ -544,7 +568,6 @@ async fn test_install_snapshot_ignored_stale_forwards_as_ok_not_error() { let (sm_apply_tx, sm_apply_rx) = mpsc::unbounded_channel(); let (internal_event_tx, _internal_event_rx) = mpsc::unbounded_channel::(); - let (_shutdown_tx, shutdown_rx) = watch::channel(()); let worker = StateMachineWorker::::new( 1, @@ -552,7 +575,6 @@ async fn test_install_snapshot_ignored_stale_forwards_as_ok_not_error() { sm_apply_tx.clone(), sm_apply_rx, internal_event_tx, - shutdown_rx, ); tokio::spawn(async move { let _ = worker.run().await; @@ -582,11 +604,12 @@ async fn test_install_snapshot_ignored_stale_forwards_as_ok_not_error() { /// /// # Then /// - response_rx receives Err -/// - Worker keeps running afterward (verified by successfully processing a -/// subsequent ApplyEntries command on the same worker) +/// - Worker keeps running afterward (verified by successfully processing a subsequent ApplyEntries +/// command on the same worker) #[tokio::test] async fn test_install_snapshot_error_forwards_as_err_worker_survives() { let mut mock_smw = MockStateMachineWriterOps::new(); + mock_smw.expect_close().returning(|| Ok(())); mock_smw .expect_install_prepared_snapshot() .times(1) @@ -599,7 +622,6 @@ async fn test_install_snapshot_error_forwards_as_err_worker_survives() { let (sm_apply_tx, sm_apply_rx) = mpsc::unbounded_channel(); let (internal_event_tx, mut internal_event_rx) = mpsc::unbounded_channel::(); - let (_shutdown_tx, shutdown_rx) = watch::channel(()); let worker = StateMachineWorker::::new( 1, @@ -607,7 +629,6 @@ async fn test_install_snapshot_error_forwards_as_err_worker_survives() { sm_apply_tx.clone(), sm_apply_rx, internal_event_tx, - shutdown_rx, ); tokio::spawn(async move { let _ = worker.run().await; @@ -625,6 +646,7 @@ async fn test_install_snapshot_error_forwards_as_err_worker_survives() { // Worker must still be alive: a subsequent ApplyEntries command is processed normally. sm_apply_tx .send(StateMachineCommand::ApplyEntries { + sent_at: tokio::time::Instant::now(), entries: vec![create_test_entry(1, 1)], }) .unwrap(); @@ -647,13 +669,14 @@ async fn test_install_snapshot_error_forwards_as_err_worker_survives() { /// - Send order: ApplyEntries(A) -> InstallSnapshot -> ApplyEntries(B) /// /// # Then -/// - Both apply_chunk calls and the install call are observed in that exact -/// order (recorded via a shared, mutex-guarded log) +/// - Both apply_chunk calls and the install call are observed in that exact order (recorded via a +/// shared, mutex-guarded log) #[tokio::test] async fn test_apply_and_install_share_single_queue_fifo_order() { let order: Arc>> = Arc::new(std::sync::Mutex::new(vec![])); let mut mock_smw = MockStateMachineWriterOps::new(); + mock_smw.expect_close().returning(|| Ok(())); { let order = order.clone(); mock_smw.expect_apply_chunk().times(2).returning(move |entries| { @@ -675,7 +698,6 @@ async fn test_apply_and_install_share_single_queue_fifo_order() { let (sm_apply_tx, sm_apply_rx) = mpsc::unbounded_channel(); let (internal_event_tx, mut internal_event_rx) = mpsc::unbounded_channel::(); - let (_shutdown_tx, shutdown_rx) = watch::channel(()); let worker = StateMachineWorker::::new( 1, @@ -683,7 +705,6 @@ async fn test_apply_and_install_share_single_queue_fifo_order() { sm_apply_tx.clone(), sm_apply_rx, internal_event_tx, - shutdown_rx, ); tokio::spawn(async move { let _ = worker.run().await; @@ -691,6 +712,7 @@ async fn test_apply_and_install_share_single_queue_fifo_order() { sm_apply_tx .send(StateMachineCommand::ApplyEntries { + sent_at: tokio::time::Instant::now(), entries: vec![create_test_entry(1, 1)], }) .unwrap(); @@ -698,6 +720,7 @@ async fn test_apply_and_install_share_single_queue_fifo_order() { sm_apply_tx.send(install_command).unwrap(); sm_apply_tx .send(StateMachineCommand::ApplyEntries { + sent_at: tokio::time::Instant::now(), entries: vec![create_test_entry(2, 1)], }) .unwrap(); @@ -813,6 +836,7 @@ async fn test_capture_local_snapshot_does_not_block_subsequent_apply_entries() { let release_rx = std::sync::Mutex::new(Some(release_rx)); let mut mock_smw = MockStateMachineWriterOps::new(); + mock_smw.expect_close().returning(|| Ok(())); mock_smw.expect_capture_local_snapshot().times(1).returning(move || { let rx = release_rx.lock().unwrap().take().expect("called once"); let _ = rx.recv(); @@ -828,7 +852,6 @@ async fn test_capture_local_snapshot_does_not_block_subsequent_apply_entries() { let (sm_apply_tx, sm_apply_rx) = mpsc::unbounded_channel(); let (internal_event_tx, mut internal_event_rx) = mpsc::unbounded_channel::(); - let (_shutdown_tx, shutdown_rx) = watch::channel(()); let worker = StateMachineWorker::::new( 1, @@ -836,7 +859,6 @@ async fn test_capture_local_snapshot_does_not_block_subsequent_apply_entries() { sm_apply_tx.clone(), sm_apply_rx, internal_event_tx, - shutdown_rx, ); tokio::spawn(async move { let _ = worker.run().await; @@ -856,6 +878,7 @@ async fn test_capture_local_snapshot_does_not_block_subsequent_apply_entries() { // Now send ApplyEntries — must be processed without waiting for capture to finish. sm_apply_tx .send(StateMachineCommand::ApplyEntries { + sent_at: tokio::time::Instant::now(), entries: vec![create_test_entry(1, 1)], }) .unwrap(); @@ -904,6 +927,7 @@ async fn test_capture_skipped_when_lock_already_held() { let (gate_reached_tx, gate_reached_rx) = std::sync::mpsc::channel::<()>(); let mut mock_smw = MockStateMachineWriterOps::new(); + mock_smw.expect_close().returning(|| Ok(())); mock_smw.expect_capture_local_snapshot().times(1).returning(move || { let _ = gate_reached_tx.send(()); let rx = release_rx.lock().unwrap().take().expect("called once"); @@ -916,7 +940,6 @@ async fn test_capture_skipped_when_lock_already_held() { let (sm_apply_tx, sm_apply_rx) = mpsc::unbounded_channel(); let (internal_event_tx, _internal_event_rx) = mpsc::unbounded_channel::(); - let (_shutdown_tx, shutdown_rx) = watch::channel(()); let worker = StateMachineWorker::::new( 1, @@ -924,7 +947,6 @@ async fn test_capture_skipped_when_lock_already_held() { sm_apply_tx.clone(), sm_apply_rx, internal_event_tx, - shutdown_rx, ); tokio::spawn(async move { let _ = worker.run().await; @@ -968,6 +990,7 @@ async fn test_install_waits_for_capture_to_release_lock() { let release_rx = std::sync::Mutex::new(Some(release_rx)); let mut mock_smw = MockStateMachineWriterOps::new(); + mock_smw.expect_close().returning(|| Ok(())); mock_smw.expect_capture_local_snapshot().times(1).returning(move || { let rx = release_rx.lock().unwrap().take().expect("called once"); let _ = rx.recv(); @@ -984,7 +1007,6 @@ async fn test_install_waits_for_capture_to_release_lock() { let (sm_apply_tx, sm_apply_rx) = mpsc::unbounded_channel(); let (internal_event_tx, _internal_event_rx) = mpsc::unbounded_channel::(); - let (_shutdown_tx, shutdown_rx) = watch::channel(()); let worker = StateMachineWorker::::new( 1, @@ -992,7 +1014,6 @@ async fn test_install_waits_for_capture_to_release_lock() { sm_apply_tx.clone(), sm_apply_rx, internal_event_tx, - shutdown_rx, ); tokio::spawn(async move { let _ = worker.run().await; @@ -1067,6 +1088,7 @@ async fn test_capture_superseded_by_install_even_when_capture_is_more_advanced() let superseded_dir_guard = std::sync::Mutex::new(Some(superseded_dir_guard)); let mut mock_smw = MockStateMachineWriterOps::new(); + mock_smw.expect_close().returning(|| Ok(())); // Capture's own boundary (index=100) is ahead of what the leader is pushing // (index=90) — proves rejection is about "install happened", not "who's newer". mock_smw.expect_capture_local_snapshot().times(1).returning(move || { @@ -1091,7 +1113,6 @@ async fn test_capture_superseded_by_install_even_when_capture_is_more_advanced() let (sm_apply_tx, sm_apply_rx) = mpsc::unbounded_channel(); let (internal_event_tx, _internal_event_rx) = mpsc::unbounded_channel::(); - let (_shutdown_tx, shutdown_rx) = watch::channel(()); let worker = StateMachineWorker::::new( 1, @@ -1099,7 +1120,6 @@ async fn test_capture_superseded_by_install_even_when_capture_is_more_advanced() sm_apply_tx.clone(), sm_apply_rx, internal_event_tx, - shutdown_rx, ); tokio::spawn(async move { let _ = worker.run().await; @@ -1182,6 +1202,7 @@ async fn test_second_capture_at_same_path_waits_for_superseded_cleanup_to_finish )); let mut mock_smw = MockStateMachineWriterOps::new(); + mock_smw.expect_close().returning(|| Ok(())); mock_smw.expect_capture_local_snapshot().times(1).returning(move || { let rx = release_rx.lock().unwrap().take().expect("called once"); let _ = rx.recv(); @@ -1204,7 +1225,6 @@ async fn test_second_capture_at_same_path_waits_for_superseded_cleanup_to_finish let (sm_apply_tx, sm_apply_rx) = mpsc::unbounded_channel(); let (internal_event_tx, _internal_event_rx) = mpsc::unbounded_channel::(); - let (_shutdown_tx, shutdown_rx) = watch::channel(()); let worker = StateMachineWorker::::new( 1, @@ -1212,7 +1232,6 @@ async fn test_second_capture_at_same_path_waits_for_superseded_cleanup_to_finish sm_apply_tx.clone(), sm_apply_rx, internal_event_tx, - shutdown_rx, ); tokio::spawn(async move { let _ = worker.run().await; @@ -1305,10 +1324,10 @@ async fn test_local_snapshot_ready_reports_operation_failed_when_superseded_clea let guard = OwnedSnapshotDir::from_existing(dir_path.clone()).unwrap(); - let mock_smw = MockStateMachineWriterOps::new(); + let mut mock_smw = MockStateMachineWriterOps::new(); + mock_smw.expect_close().returning(|| Ok(())); let (sm_apply_tx, sm_apply_rx) = mpsc::unbounded_channel(); let (internal_event_tx, _internal_event_rx) = mpsc::unbounded_channel::(); - let (_shutdown_tx, shutdown_rx) = watch::channel(()); let worker = StateMachineWorker::::new( 1, @@ -1316,7 +1335,6 @@ async fn test_local_snapshot_ready_reports_operation_failed_when_superseded_clea sm_apply_tx.clone(), sm_apply_rx, internal_event_tx, - shutdown_rx, ); tokio::spawn(async move { let _ = worker.run().await; @@ -1379,10 +1397,10 @@ async fn test_local_snapshot_ready_reports_operation_failed_when_superseded_clea /// no race to win, `send()` is guaranteed to find a dead receiver. #[tokio::test] async fn test_local_snapshot_ready_cleans_up_capture_when_response_receiver_dropped() { - let mock_smw = MockStateMachineWriterOps::new(); + let mut mock_smw = MockStateMachineWriterOps::new(); + mock_smw.expect_close().returning(|| Ok(())); let (sm_apply_tx, sm_apply_rx) = mpsc::unbounded_channel(); let (internal_event_tx, _internal_event_rx) = mpsc::unbounded_channel::(); - let (_shutdown_tx, shutdown_rx) = watch::channel(()); let worker = StateMachineWorker::::new( 1, @@ -1390,7 +1408,6 @@ async fn test_local_snapshot_ready_cleans_up_capture_when_response_receiver_drop sm_apply_tx.clone(), sm_apply_rx, internal_event_tx, - shutdown_rx, ); tokio::spawn(async move { let _ = worker.run().await; @@ -1438,6 +1455,7 @@ async fn test_shutdown_waits_for_in_flight_capture_before_returning() { let (gate_reached_tx, gate_reached_rx) = std::sync::mpsc::channel::<()>(); let mut mock_smw = MockStateMachineWriterOps::new(); + mock_smw.expect_close().returning(|| Ok(())); mock_smw.expect_capture_local_snapshot().times(1).returning(move || { let _ = gate_reached_tx.send(()); let rx = release_rx.lock().unwrap().take().expect("called once"); @@ -1450,7 +1468,6 @@ async fn test_shutdown_waits_for_in_flight_capture_before_returning() { let (sm_apply_tx, sm_apply_rx) = mpsc::unbounded_channel(); let (internal_event_tx, _internal_event_rx) = mpsc::unbounded_channel::(); - let (shutdown_tx, shutdown_rx) = watch::channel(()); let worker = StateMachineWorker::::new( 1, @@ -1458,7 +1475,6 @@ async fn test_shutdown_waits_for_in_flight_capture_before_returning() { sm_apply_tx.clone(), sm_apply_rx, internal_event_tx, - shutdown_rx, ); let worker_handle = tokio::spawn(async move { worker.run().await }); @@ -1473,7 +1489,7 @@ async fn test_shutdown_waits_for_in_flight_capture_before_returning() { // "in flight" — before triggering shutdown. gate_reached_rx.recv().unwrap(); - shutdown_tx.send(()).unwrap(); + sm_apply_tx.send(StateMachineCommand::Shutdown).unwrap(); // Give shutdown handling a real chance to run, then prove it hasn't returned — // it must be blocked waiting on the still-gated capture, not racing past it. @@ -1517,10 +1533,10 @@ async fn test_superseded_capture_directory_gone_by_the_time_response_resolves() } let guard = OwnedSnapshotDir::from_existing(dir_path.clone()).unwrap(); - let mock_smw = MockStateMachineWriterOps::new(); + let mut mock_smw = MockStateMachineWriterOps::new(); + mock_smw.expect_close().returning(|| Ok(())); let (sm_apply_tx, sm_apply_rx) = mpsc::unbounded_channel(); let (internal_event_tx, _internal_event_rx) = mpsc::unbounded_channel::(); - let (_shutdown_tx, shutdown_rx) = watch::channel(()); let worker = StateMachineWorker::::new( 1, @@ -1528,7 +1544,6 @@ async fn test_superseded_capture_directory_gone_by_the_time_response_resolves() sm_apply_tx.clone(), sm_apply_rx, internal_event_tx, - shutdown_rx, ); tokio::spawn(async move { let _ = worker.run().await; @@ -1569,3 +1584,183 @@ async fn test_superseded_capture_directory_gone_by_the_time_response_resolves() this path before cleanup has actually run" ); } + +/// Test: the SM Worker records how long a batch sat in its queue before being applied. +/// +/// Why: commit->apply latency has a hidden part -- time between the Raft loop +/// enqueueing a batch and the Worker picking it up (it applies one batch at a time). +/// `sent_at` must carry the real enqueue time, otherwise that wait stays invisible. +#[tokio::test] +async fn test_apply_entries_records_sm_worker_queue_wait() { + let capture = crate::test_utils::MetricsCapture::new(); + let _guard = metrics::set_default_local_recorder(&capture); + + let mut mock_smw = MockStateMachineWriterOps::new(); + mock_smw.expect_close().returning(|| Ok(())); + mock_smw.expect_apply_chunk().times(1).returning(|entries| { + Ok(entries.iter().map(|e| crate::ApplyResult::success(e.index)).collect()) + }); + + let (sm_apply_tx, sm_apply_rx) = mpsc::unbounded_channel(); + let (internal_event_tx, mut internal_event_rx) = mpsc::unbounded_channel::(); + let worker = StateMachineWorker::::new( + 1, + Arc::new(mock_smw), + sm_apply_tx.clone(), + sm_apply_rx, + internal_event_tx, + ); + tokio::spawn(async move { + let _ = worker.run().await; + }); + + sm_apply_tx + .send(StateMachineCommand::ApplyEntries { + sent_at: tokio::time::Instant::now() - std::time::Duration::from_millis(30), + entries: vec![create_test_entry(1, 1)], + }) + .unwrap(); + tokio::time::timeout( + std::time::Duration::from_millis(500), + internal_event_rx.recv(), + ) + .await + .expect("ApplyCompleted not received") + .expect("channel closed"); + + let waits = capture.histogram("core.raft.sm_worker.queue_wait_ms", &[]); + assert_eq!(waits.len(), 1, "one queue-wait sample per batch"); + assert!( + waits[0] >= 30.0, + "wait must cover time since enqueue, got {}", + waits[0] + ); +} + +/// Test: `ApplyCompleted` is stamped when the Worker sends it, so the central loop can +/// measure how long the event waited behind other events before being handled. +#[tokio::test] +async fn test_apply_completed_carries_send_timestamp() { + let mut mock_smw = MockStateMachineWriterOps::new(); + mock_smw.expect_close().returning(|| Ok(())); + mock_smw.expect_apply_chunk().times(1).returning(|entries| { + Ok(entries.iter().map(|e| crate::ApplyResult::success(e.index)).collect()) + }); + + let (sm_apply_tx, sm_apply_rx) = mpsc::unbounded_channel(); + let (internal_event_tx, mut internal_event_rx) = mpsc::unbounded_channel::(); + let worker = StateMachineWorker::::new( + 1, + Arc::new(mock_smw), + sm_apply_tx.clone(), + sm_apply_rx, + internal_event_tx, + ); + tokio::spawn(async move { + let _ = worker.run().await; + }); + + let before = tokio::time::Instant::now(); + sm_apply_tx + .send(StateMachineCommand::ApplyEntries { + sent_at: tokio::time::Instant::now(), + entries: vec![create_test_entry(1, 1)], + }) + .unwrap(); + match tokio::time::timeout( + std::time::Duration::from_millis(500), + internal_event_rx.recv(), + ) + .await + { + Ok(Some(InternalEvent::ApplyCompleted { sent_at, .. })) => { + assert!( + sent_at >= before, + "sent_at must be taken at send time, not earlier" + ); + assert!(sent_at <= tokio::time::Instant::now()); + } + other => panic!("expected ApplyCompleted, got {other:?}"), + } +} + +/// Business scenario: the node is stopped cleanly but the state machine cannot persist its +/// final state when it is closed (disk full). +/// +/// Purpose: `run` returns that error instead of reporting a clean exit, and the state machine +/// was still closed exactly once, so its resources are released. +#[tokio::test] +async fn test_run_returns_the_close_error_after_a_clean_drain() { + let mut mock_smw = MockStateMachineWriterOps::new(); + mock_smw + .expect_close() + .times(1) + .returning(|| Err(SnapshotError::OperationFailed("disk full".into()).into())); + + let (sm_apply_tx, sm_apply_rx) = mpsc::unbounded_channel(); + let (internal_event_tx, _internal_event_rx) = mpsc::unbounded_channel::(); + let worker = StateMachineWorker::::new( + 1, + Arc::new(mock_smw), + sm_apply_tx.clone(), + sm_apply_rx, + internal_event_tx, + ); + sm_apply_tx.send(StateMachineCommand::Shutdown).unwrap(); + + let result = worker.run().await; + + let error = result.expect_err("a failed close must make run() fail"); + assert!( + format!("{error:?}").contains("disk full"), + "the close error was lost: {error:?}" + ); +} + +/// Business scenario: an entry fails to apply, and then closing the state machine fails too. +/// +/// Purpose: the apply failure stays the reported error (it is the root cause), and the state +/// machine is still closed once. +#[tokio::test] +async fn test_run_keeps_the_apply_error_when_close_also_fails() { + let mut mock_smw = MockStateMachineWriterOps::new(); + mock_smw + .expect_apply_chunk() + .times(1) + .returning(|_| Err(SnapshotError::OperationFailed("apply broke".into()).into())); + mock_smw + .expect_close() + .times(1) + .returning(|| Err(SnapshotError::OperationFailed("close broke".into()).into())); + + let (sm_apply_tx, sm_apply_rx) = mpsc::unbounded_channel(); + let (internal_event_tx, _internal_event_rx) = mpsc::unbounded_channel::(); + let worker = StateMachineWorker::::new( + 1, + Arc::new(mock_smw), + sm_apply_tx.clone(), + sm_apply_rx, + internal_event_tx, + ); + sm_apply_tx + .send(StateMachineCommand::ApplyEntries { + sent_at: tokio::time::Instant::now(), + entries: vec![create_test_entry(1, 1)], + }) + .unwrap(); + + let result = worker.run().await; + + let error = format!( + "{:?}", + result.expect_err("a failed apply must make run() fail") + ); + assert!( + error.contains("apply broke"), + "the root cause was lost: {error}" + ); + assert!( + !error.contains("close broke"), + "close error must not replace the apply error: {error}" + ); +} diff --git a/d-engine-core/src/state_machine_handler/writer.rs b/d-engine-core/src/state_machine_handler/writer.rs index 43a95820..ab6217db 100644 --- a/d-engine-core/src/state_machine_handler/writer.rs +++ b/d-engine-core/src/state_machine_handler/writer.rs @@ -1,8 +1,9 @@ +use std::path::PathBuf; +use std::sync::Arc; + use async_trait::async_trait; use d_engine_proto::common::Entry; use d_engine_proto::server::storage::SnapshotMetadata; -use std::path::PathBuf; -use std::sync::Arc; use tracing::trace; use super::StateMachineWriterOps; @@ -81,10 +82,10 @@ impl StateMachineWriterOps for DefaultStateMachineWriter { let start = std::time::Instant::now(); let chunk_size = chunk.len(); - metrics::counter!( - "core.state_machine.apply_chunk.count", - &[("node_id", self.node_id.to_string())] - ) + metrics::counter!("core.state_machine.apply_chunk.count", &[( + "node_id", + self.node_id.to_string() + )]) .increment(1); let last_index = chunk.last().map(|entry| entry.index); @@ -111,10 +112,10 @@ impl StateMachineWriterOps for DefaultStateMachineWriter { let apply_t0 = std::time::Instant::now(); let apply_result = sm.apply_chunk(&apply_entries).await; let apply_elapsed = apply_t0.elapsed(); - metrics::counter!( - "core.state_machine.apply.busy_nanos_total", - &[("node_id", self.node_id.to_string())] - ) + metrics::counter!("core.state_machine.apply.busy_nanos_total", &[( + "node_id", + self.node_id.to_string() + )]) .increment(apply_elapsed.as_nanos() as u64); #[cfg(feature = "watch")] @@ -130,42 +131,39 @@ impl StateMachineWriterOps for DefaultStateMachineWriter { } let duration_ms = start.elapsed().as_secs_f64() * 1000.0; - metrics::histogram!( - "core.state_machine.apply_chunk.duration_ms", - &[("node_id", self.node_id.to_string())] - ) + metrics::histogram!("core.state_machine.apply_chunk.duration_ms", &[( + "node_id", + self.node_id.to_string() + )]) .record(duration_ms); - metrics::histogram!( - "core.state_machine.apply_chunk.batch_size", - &[("node_id", self.node_id.to_string())] - ) + metrics::histogram!("core.state_machine.apply_chunk.batch_size", &[( + "node_id", + self.node_id.to_string() + )]) .record(chunk_size as f64); match &apply_result { Ok(_) => { if let Some(idx) = last_index { self.applied.advance(idx); - metrics::gauge!( - "core.raft.apply_index", - &[("node_id", self.node_id.to_string())] - ) + metrics::gauge!("core.raft.apply_index", &[( + "node_id", + self.node_id.to_string() + )]) .set(idx as f64); } - metrics::counter!( - "core.state_machine.apply_chunk.success", - &[("node_id", self.node_id.to_string())] - ) + metrics::counter!("core.state_machine.apply_chunk.success", &[( + "node_id", + self.node_id.to_string() + )]) .increment(1); } Err(e) => { let error_type = classify_error(e); - metrics::counter!( - "core.state_machine.apply_chunk.error", - &[ - ("node_id", self.node_id.to_string()), - ("error_type", error_type) - ] - ) + metrics::counter!("core.state_machine.apply_chunk.error", &[ + ("node_id", self.node_id.to_string()), + ("error_type", error_type) + ]) .increment(1); } } @@ -224,4 +222,8 @@ impl StateMachineWriterOps for DefaultStateMachineWriter { temp_dir: OwnedSnapshotDir::from_existing(temp_path)?, }) } + + fn close(&self) -> Result<()> { + self.state_machine.close() + } } diff --git a/d-engine-core/src/storage/fsync_worker.rs b/d-engine-core/src/storage/fsync_worker.rs index 07b87dec..e4bf1da0 100644 --- a/d-engine-core/src/storage/fsync_worker.rs +++ b/d-engine-core/src/storage/fsync_worker.rs @@ -1,16 +1,20 @@ -use crate::Error; -use crate::InternalEvent; -use crate::LogStore; -use crate::Result; +use std::sync::Arc; +use std::sync::atomic::AtomicBool; +use std::sync::atomic::AtomicU64; +use std::sync::atomic::Ordering; + use d_engine_proto::common::LogId; use parking_lot::Mutex; -use std::sync::Arc; -use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; use tokio::sync::mpsc; use tokio::sync::oneshot; use tokio::time::Instant; use tracing::error; +use crate::Error; +use crate::InternalEvent; +use crate::LogStore; +use crate::Result; + /// Schedules physical fsync calls — batches concurrent requests into one /// flush() at a time. The final content check is `RaftLogCore:: /// try_advance_durable_index`; this only orders the pending mark term-first. diff --git a/d-engine-core/src/storage/fsync_worker_test.rs b/d-engine-core/src/storage/fsync_worker_test.rs index 442538e4..c61a0561 100644 --- a/d-engine-core/src/storage/fsync_worker_test.rs +++ b/d-engine-core/src/storage/fsync_worker_test.rs @@ -10,16 +10,20 @@ //! Scope: protocol/logic correctness of `FsyncWorker`'s own state machine. //! Not performance — see `benches/` for throughput regression guards. +use std::sync::Arc; +use std::sync::atomic::AtomicBool; +use std::sync::atomic::Ordering; + +use d_engine_proto::common::LogId; +use tokio::sync::mpsc; +use tokio::sync::oneshot; + use super::*; use crate::InternalEvent; use crate::MockLogStore; use crate::MockStorageEngine; use crate::Result; use crate::StorageEngine; -use d_engine_proto::common::LogId; -use std::sync::Arc; -use std::sync::atomic::{AtomicBool, Ordering}; -use tokio::sync::{mpsc, oneshot}; /// Build a bare `FsyncWorker` over a mock log store — no tokio runtime, no /// `RaftLogCore`. `internal_event_tx` is `None`, so the worker never sends @@ -38,8 +42,7 @@ fn minimal_worker(log_store: Arc) -> Arc /// `FsyncWorker::new()` starts with no work pending and no fence armed. /// /// Expected: -/// - `inflight == false`, `pending_max == 0`, `pending_replies` empty, -/// `generation == 0`. +/// - `inflight == false`, `pending_max == 0`, `pending_replies` empty, `generation == 0`. #[test] fn test_new_initializes_empty_state() { let worker = minimal_worker(Arc::new(MockLogStore::new())); diff --git a/d-engine-core/src/storage/lease.rs b/d-engine-core/src/storage/lease.rs index f6868a14..98d15c2e 100644 --- a/d-engine-core/src/storage/lease.rs +++ b/d-engine-core/src/storage/lease.rs @@ -20,7 +20,7 @@ //! //! Implementations should follow these guidelines: //! - **Read path (hot)**: Lock-free, supports high concurrency -//! - **Write path (cold)**: Single-threaded (CommitHandler), Mutex acceptable +//! - **Write path (cold)**: Single-threaded (state machine worker), Mutex acceptable //! - **Read-write**: Concurrent safe, reads don't block on cleanup use std::time::SystemTime; diff --git a/d-engine-core/src/storage/mod.rs b/d-engine-core/src/storage/mod.rs index dd3bf7e2..54976f8d 100644 --- a/d-engine-core/src/storage/mod.rs +++ b/d-engine-core/src/storage/mod.rs @@ -5,11 +5,10 @@ mod raft_log_core; mod snapshot_path_manager; mod state_machine; mod storage_engine; -pub use raft_log_core::*; - pub use lease::*; #[doc(hidden)] pub use raft_log::*; +pub use raft_log_core::*; pub(crate) use snapshot_path_manager::*; pub use state_machine::*; pub use storage_engine::*; diff --git a/d-engine-core/src/storage/raft_log_core.rs b/d-engine-core/src/storage/raft_log_core.rs index 97c419bf..d606dc29 100644 --- a/d-engine-core/src/storage/raft_log_core.rs +++ b/d-engine-core/src/storage/raft_log_core.rs @@ -3,25 +3,39 @@ //! no dedicated OS thread, no command channel. Only the physical fsync is //! handed off, to `FsyncWorker`'s own execution context (see fsync_worker.rs). -use crate::{ - Error, HardState, InternalEvent, LogStore, MetaStore, NetworkError, RaftLog, Result, - StorageEngine, TypeConfig, alias::SOF, fsync_worker::FsyncWorker, scoped_timer::ScopedTimer, -}; +use std::collections::HashMap; +use std::ops::RangeInclusive; +use std::sync::Arc; +use std::sync::atomic::AtomicBool; +use std::sync::atomic::AtomicU64; +use std::sync::atomic::AtomicUsize; +use std::sync::atomic::Ordering; +use std::time::Duration; + use crossbeam_skiplist::SkipMap; -use d_engine_proto::common::{Entry, LogId}; +use d_engine_proto::common::Entry; +use d_engine_proto::common::LogId; use parking_lot::RwLock; -use std::{ - collections::HashMap, - ops::RangeInclusive, - sync::{ - Arc, - atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering}, - }, - time::Duration, -}; -use tokio::sync::{mpsc, oneshot}; +use tokio::sync::mpsc; +use tokio::sync::oneshot; use tonic::async_trait; -use tracing::{debug, error, warn}; +use tracing::debug; +use tracing::error; +use tracing::warn; + +use crate::Error; +use crate::HardState; +use crate::InternalEvent; +use crate::LogStore; +use crate::MetaStore; +use crate::NetworkError; +use crate::RaftLog; +use crate::Result; +use crate::StorageEngine; +use crate::TypeConfig; +use crate::alias::SOF; +use crate::fsync_worker::FsyncWorker; +use crate::scoped_timer::ScopedTimer; pub struct RaftLogCore where @@ -60,8 +74,9 @@ where // --- In-memory index --- // O(1) answer to "is this index currently held in memory" — lets callers // (e.g. entry_term()) reject an out-of-range index without touching `entries`. - min_index: AtomicU64, // Smallest log index (0 if empty) - memory_max_index: AtomicU64, // Largest log index held in memory (0 if empty) — may be ahead of what's persisted/durable + min_index: AtomicU64, // Smallest log index (0 if empty) + memory_max_index: AtomicU64, /* Largest log index held in memory (0 if empty) — may be ahead + * of what's persisted/durable */ // The term of the last entry ever purged (compacted away after a snapshot). // Raft's AppendEntries consistency check needs the term at prev_log_index @@ -312,9 +327,10 @@ where // Overlap is safe to skip entirely when all three conditions hold: // (a) The entire overlap range falls within follower's current term segment // (first.index >= last_term_start) — no older-term entries below overlap start. - // (b) Incoming overlap entries all carry follower's current term (first.term == last_term). - // (c) No term change within incoming overlap (last.term == last_term); guaranteed by - // Raft non-decreasing term property when combined with (b). + // (b) Incoming overlap entries all carry follower's current term (first.term == + // last_term). (c) No term change within incoming overlap (last.term == + // last_term); guaranteed by Raft non-decreasing term property when combined + // with (b). // // Covers the steady-state pipeline path (99%+ of calls). Election recovery (new leader, // term conflict in overlap) is caught by condition (b) failing → slow path. @@ -671,7 +687,8 @@ where self.update_term_indexes(entries); self.term_segments.on_append(entries); - let max_index = entries.iter().map(|e| e.index).max().unwrap_or(0); + // `entries` are in log order: the last one has the highest index. + let max_index = entries.last().map_or(0, |e| e.index); let current_next = self.next_id.load(Ordering::Acquire); if max_index >= current_next { self.next_id.store(max_index + 1, Ordering::Release); @@ -713,20 +730,22 @@ where &self, entries: &[Entry], ) { - for entry in entries { - let term = entry.term; + // `entries` are in log order, so equal terms are contiguous: one lookup pair per term + // run instead of per entry (a batch is almost always a single term). + for run in entries.chunk_by(|a, b| a.term == b.term) { + let term = run[0].term; // Update first index self.term_first_index .get_or_insert(term, AtomicU64::new(u64::MAX)) .value() - .fetch_min(entry.index, Ordering::AcqRel); + .fetch_min(run[0].index, Ordering::AcqRel); // Update last index self.term_last_index .get_or_insert(term, AtomicU64::new(0)) .value() - .fetch_max(entry.index, Ordering::AcqRel); + .fetch_max(run[run.len() - 1].index, Ordering::AcqRel); } } diff --git a/d-engine-core/src/storage/raft_log_core_test/basic_operations_test.rs b/d-engine-core/src/storage/raft_log_core_test/basic_operations_test.rs index c445607f..ced99bf8 100644 --- a/d-engine-core/src/storage/raft_log_core_test/basic_operations_test.rs +++ b/d-engine-core/src/storage/raft_log_core_test/basic_operations_test.rs @@ -7,12 +7,14 @@ //! - Log purging //! - Metadata updates -use d_engine_proto::common::{Entry, LogId}; +use d_engine_proto::common::Entry; +use d_engine_proto::common::LogId; use crate::RaftLog; -use crate::test_utils::{ - RaftLogCoreTestContext, mock_empty_entries, simulate_delete_command, simulate_insert_command, -}; +use crate::test_utils::RaftLogCoreTestContext; +use crate::test_utils::mock_empty_entries; +use crate::test_utils::simulate_delete_command; +use crate::test_utils::simulate_insert_command; /// Test get_entries_range returns correct subset /// @@ -544,8 +546,8 @@ async fn test_last_log_id_after_appends() { /// /// # Scenario /// - Insert entry and explicitly drop context -/// - Expected: No panic (RaftLogCore has no custom Drop — this just guards -/// against a panic during normal field teardown) +/// - Expected: No panic (RaftLogCore has no custom Drop — this just guards against a panic during +/// normal field teardown) #[tokio::test] async fn test_drop_with_pending_writes_does_not_panic() { let ctx = RaftLogCoreTestContext::new("test_drop_with_pending_writes_does_not_panic"); diff --git a/d-engine-core/src/storage/raft_log_core_test/concurrent_fsync_test.rs b/d-engine-core/src/storage/raft_log_core_test/concurrent_fsync_test.rs index a30a16f5..5024ae0b 100644 --- a/d-engine-core/src/storage/raft_log_core_test/concurrent_fsync_test.rs +++ b/d-engine-core/src/storage/raft_log_core_test/concurrent_fsync_test.rs @@ -5,15 +5,20 @@ //! - **Logic**: `FsyncWorker` (submit / run_until_caught_up) / try_advance_durable_index contract //! - **Concurrency**: Reset races, out-of-order completion, crash recovery -use crate::test_utils::drain_and_apply_fsync_completions; -use crate::test_utils::wait_for_durable_index; -use crate::{MockStorageEngine, MockTypeConfig, RaftLog, RaftLogCore}; -use d_engine_proto::common::Entry; -use d_engine_proto::common::LogId; use std::sync::Arc; use std::time::Duration; + +use d_engine_proto::common::Entry; +use d_engine_proto::common::LogId; use tokio::sync::mpsc; +use crate::MockStorageEngine; +use crate::MockTypeConfig; +use crate::RaftLog; +use crate::RaftLogCore; +use crate::test_utils::drain_and_apply_fsync_completions; +use crate::test_utils::wait_for_durable_index; + // ── Protocol correctness ────────────────────────────────────────────────────── /// `durable_index` must not advance before the physical fsync actually completes. @@ -519,13 +524,11 @@ async fn test_shutdown_with_pending_flush_caller_still_receives_err_reply() { /// caller waiting on that stale round. /// /// Scenario: -/// 1. Write an entry, gate the physical `flush()` call so the round stays -/// in flight. -/// 2. While gated, call `reset()` — bumps `FsyncWorker`'s fence generation, -/// then clears `durable_index`/`memory_max_index`/entries to 0. -/// 3. Release the gate — the stale round's `flush()` returns, but its -/// generation no longer matches; it must discard its result instead of -/// resolving durable_index or the queued reply. +/// 1. Write an entry, gate the physical `flush()` call so the round stays in flight. +/// 2. While gated, call `reset()` — bumps `FsyncWorker`'s fence generation, then clears +/// `durable_index`/`memory_max_index`/entries to 0. +/// 3. Release the gate — the stale round's `flush()` returns, but its generation no longer +/// matches; it must discard its result instead of resolving durable_index or the queued reply. #[tokio::test] async fn test_reset_during_inflight_fsync_does_not_resurrect_stale_durable_index() { let (storage, flush_gate) = MockStorageEngine::not_durable_gated_flush( diff --git a/d-engine-core/src/storage/raft_log_core_test/concurrent_operations_test.rs b/d-engine-core/src/storage/raft_log_core_test/concurrent_operations_test.rs index 616bae67..1afea457 100644 --- a/d-engine-core/src/storage/raft_log_core_test/concurrent_operations_test.rs +++ b/d-engine-core/src/storage/raft_log_core_test/concurrent_operations_test.rs @@ -1,11 +1,13 @@ use std::time::Duration; +use d_engine_proto::common::Entry; +use d_engine_proto::common::LogId; use futures::future::join_all; use tokio; use crate::storage::raft_log::RaftLog; -use crate::test_utils::{RaftLogCoreTestContext, simulate_insert_command}; -use d_engine_proto::common::{Entry, LogId}; +use crate::test_utils::RaftLogCoreTestContext; +use crate::test_utils::simulate_insert_command; #[tokio::test] async fn test_remove_range_with_concurrent_reads() { @@ -102,8 +104,8 @@ async fn test_concurrent_append_and_purge() { /// This does NOT test the leader's own replication path (`retrieve_to_be_synced_logs_for_peers` /// only ever runs on the single-threaded raft core loop, serialized with its own /// purge calls — see 436 design docs). It tests the lower-level primitive any -/// concurrent reader (e.g. `DefaultCommitHandler`, which runs as its own spawned -/// task and also calls `get_entries_range` on the same shared log) depends on. +/// concurrent reader (any task that calls `get_entries_range` on the shared log) +/// depends on. /// /// Non-deterministic by nature: a torn read is a real, reproducible bug (fails /// this test if hit), but a clean pass does not prove the race is absent — only diff --git a/d-engine-core/src/storage/raft_log_core_test/content_validated_watermark_test.rs b/d-engine-core/src/storage/raft_log_core_test/content_validated_watermark_test.rs index 0f5d63d6..4eab10fb 100644 --- a/d-engine-core/src/storage/raft_log_core_test/content_validated_watermark_test.rs +++ b/d-engine-core/src/storage/raft_log_core_test/content_validated_watermark_test.rs @@ -15,9 +15,11 @@ use std::time::Duration; use d_engine_proto::common::Entry; use d_engine_proto::common::LogId; +use crate::MockStorageEngine; +use crate::MockTypeConfig; +use crate::RaftLogCore; use crate::storage::raft_log::RaftLog; use crate::test_utils::RaftLogCoreTestContext; -use crate::{MockStorageEngine, MockTypeConfig, RaftLogCore}; fn entry( index: u64, diff --git a/d-engine-core/src/storage/raft_log_core_test/drain_fsync_test.rs b/d-engine-core/src/storage/raft_log_core_test/drain_fsync_test.rs index 3233a445..c83cef2a 100644 --- a/d-engine-core/src/storage/raft_log_core_test/drain_fsync_test.rs +++ b/d-engine-core/src/storage/raft_log_core_test/drain_fsync_test.rs @@ -2,33 +2,35 @@ //! //! These tests verify three core properties: //! -//! 1. **Write/flush separation**: `durable_index` advances only after fsync, not -//! after the write-only phase. This ensures crash-safety semantics are preserved. +//! 1. **Write/flush separation**: `durable_index` advances only after fsync, not after the +//! write-only phase. This ensures crash-safety semantics are preserved. //! -//! 2. **Batch efficiency**: rapid concurrent writes are covered by far fewer fsyncs -//! than individual writes. The fsync execution time itself acts as the batch window. +//! 2. **Batch efficiency**: rapid concurrent writes are covered by far fewer fsyncs than individual +//! writes. The fsync execution time itself acts as the batch window. //! -//! 3. **Explicit flush barrier**: `flush()` waits until all entries written before -//! the call are durable, regardless of how many internal fsyncs occurred. +//! 3. **Explicit flush barrier**: `flush()` waits until all entries written before the call are +//! durable, regardless of how many internal fsyncs occurred. -use crate::Error; -use crate::HardState; -use crate::MockLogStore; -use crate::MockMetaStore; -use crate::MockStorageEngine; -use crate::MockTypeConfig; -use crate::RaftLogCore; -use d_engine_proto::common::Entry; -use d_engine_proto::common::LogId; use std::sync::Arc; use std::sync::Mutex; use std::sync::atomic::AtomicU64; use std::sync::atomic::Ordering; + +use d_engine_proto::common::Entry; +use d_engine_proto::common::LogId; use tokio::sync::mpsc; +use tokio::time::Duration; +use tokio::time::sleep; use tokio::time::timeout; -use tokio::time::{Duration, sleep}; +use crate::Error; +use crate::HardState; +use crate::MockLogStore; +use crate::MockMetaStore; +use crate::MockStorageEngine; +use crate::MockTypeConfig; use crate::RaftLog; +use crate::RaftLogCore; use crate::test_utils::RaftLogCoreTestContext; /// Each `append_entries` persists the new tail and auto-submits it to `FsyncWorker` @@ -384,22 +386,18 @@ async fn test_replace_range_failure_poisons() { // A new leader (term 2) overwrites from index 2 onward — a real conflict // that must truncate + replace, routing through `replace_range_and_submit`. let result = raft_log - .filter_out_conflicts_and_append( - 1, - 1, - vec![ - Entry { - index: 2, - term: 2, - payload: None, - }, - Entry { - index: 3, - term: 2, - payload: None, - }, - ], - ) + .filter_out_conflicts_and_append(1, 1, vec![ + Entry { + index: 2, + term: 2, + payload: None, + }, + Entry { + index: 3, + term: 2, + payload: None, + }, + ]) .await; assert!( result.is_err(), @@ -613,22 +611,18 @@ async fn test_poisoned_skips_replace_range() { raft_log.poisoned.store(true, Ordering::SeqCst); let result = raft_log - .filter_out_conflicts_and_append( - 1, - 1, - vec![ - Entry { - index: 2, - term: 2, - payload: None, - }, - Entry { - index: 3, - term: 2, - payload: None, - }, - ], - ) + .filter_out_conflicts_and_append(1, 1, vec![ + Entry { + index: 2, + term: 2, + payload: None, + }, + Entry { + index: 3, + term: 2, + payload: None, + }, + ]) .await; match result { diff --git a/d-engine-core/src/storage/raft_log_core_test/durable_index_test.rs b/d-engine-core/src/storage/raft_log_core_test/durable_index_test.rs index 5f78f1a3..2172fd23 100644 --- a/d-engine-core/src/storage/raft_log_core_test/durable_index_test.rs +++ b/d-engine-core/src/storage/raft_log_core_test/durable_index_test.rs @@ -1,13 +1,18 @@ use std::sync::Arc; use std::time::Duration; +use d_engine_proto::common::Entry; +use d_engine_proto::common::LogId; use futures::future::join_all; use tokio::sync::mpsc; +use crate::InternalEvent; +use crate::MockTypeConfig; +use crate::RaftLogCore; use crate::storage::raft_log::RaftLog; -use crate::test_utils::{MockStorageEngine, RaftLogCoreTestContext, simulate_insert_command}; -use crate::{InternalEvent, MockTypeConfig, RaftLogCore}; -use d_engine_proto::common::{Entry, LogId}; +use crate::test_utils::MockStorageEngine; +use crate::test_utils::RaftLogCoreTestContext; +use crate::test_utils::simulate_insert_command; #[tokio::test] async fn test_durable_index_monotonic_under_concurrency() { diff --git a/d-engine-core/src/storage/raft_log_core_test/durable_index_truncation_clamp_test.rs b/d-engine-core/src/storage/raft_log_core_test/durable_index_truncation_clamp_test.rs index f4e7b69d..53762050 100644 --- a/d-engine-core/src/storage/raft_log_core_test/durable_index_truncation_clamp_test.rs +++ b/d-engine-core/src/storage/raft_log_core_test/durable_index_truncation_clamp_test.rs @@ -12,9 +12,13 @@ use std::time::Duration; use d_engine_proto::common::Entry; use d_engine_proto::common::LogId; +use crate::MockLogStore; +use crate::MockMetaStore; +use crate::MockStorageEngine; +use crate::MockTypeConfig; +use crate::RaftLogCore; use crate::storage::raft_log::RaftLog; use crate::test_utils::RaftLogCoreTestContext; -use crate::{MockLogStore, MockMetaStore, MockStorageEngine, MockTypeConfig, RaftLogCore}; fn entry( index: u64, @@ -81,19 +85,16 @@ async fn test_durable_index_lands_on_new_tail_after_truncate_and_resync() { /// `durable_index` into the range the truncation discarded. /// /// Timeline (deterministic via the persist gate): -/// 1. Old leader (term 1) replicates 1..=10 — on its own task, since the -/// inline persist path now blocks the calling task on the gate (unlike -/// the old dedicated-IO-thread design, where `append_entries` returned -/// immediately and the gated persist ran elsewhere). -/// 2. New leader (term 2): index 2 conflicts, on a second task, concurrent -/// with the still-gated persist from step 1. In-memory truncation -/// (`remove_range`) is synchronous, so it's visible immediately; the -/// `replace_range_and_submit` await behind it does not depend on step 1's -/// gate at all — the two tasks touch independent code paths, only -/// `entries`'s `RwLock` briefly serializes them. -/// 3. Release the gate: the stale persist (step 1) completes and reports a -/// stale fsync-completion for index 10; a flush() drives one legitimate -/// fsync-completion for index 2. +/// 1. Old leader (term 1) replicates 1..=10 — on its own task, since the inline persist path now +/// blocks the calling task on the gate (unlike the old dedicated-IO-thread design, where +/// `append_entries` returned immediately and the gated persist ran elsewhere). +/// 2. New leader (term 2): index 2 conflicts, on a second task, concurrent with the still-gated +/// persist from step 1. In-memory truncation (`remove_range`) is synchronous, so it's visible +/// immediately; the `replace_range_and_submit` await behind it does not depend on step 1's gate +/// at all — the two tasks touch independent code paths, only `entries`'s `RwLock` briefly +/// serializes them. +/// 3. Release the gate: the stale persist (step 1) completes and reports a stale fsync-completion +/// for index 10; a flush() drives one legitimate fsync-completion for index 2. /// /// Expected: draining the fsync completions advances `durable_index` to 2 and /// rejects the stale report for index 10. diff --git a/d-engine-core/src/storage/raft_log_core_test/edge_cases_test.rs b/d-engine-core/src/storage/raft_log_core_test/edge_cases_test.rs index 1243436e..5049d63f 100644 --- a/d-engine-core/src/storage/raft_log_core_test/edge_cases_test.rs +++ b/d-engine-core/src/storage/raft_log_core_test/edge_cases_test.rs @@ -1,8 +1,10 @@ use bytes::Bytes; +use d_engine_proto::common::Entry; +use d_engine_proto::common::EntryPayload; +use d_engine_proto::common::LogId; use crate::storage::raft_log::RaftLog; use crate::test_utils::RaftLogCoreTestContext; -use d_engine_proto::common::{Entry, EntryPayload, LogId}; #[tokio::test] async fn test_empty_log_operations() { diff --git a/d-engine-core/src/storage/raft_log_core_test/flush_strategy_test.rs b/d-engine-core/src/storage/raft_log_core_test/flush_strategy_test.rs index 754da743..183c7d43 100644 --- a/d-engine-core/src/storage/raft_log_core_test/flush_strategy_test.rs +++ b/d-engine-core/src/storage/raft_log_core_test/flush_strategy_test.rs @@ -9,8 +9,10 @@ //! Integration tests with real FileStorageEngine are in d-engine-server/tests/integration. use bytes::Bytes; -use d_engine_proto::common::{Entry, EntryPayload}; -use tokio::time::{Duration, sleep}; +use d_engine_proto::common::Entry; +use d_engine_proto::common::EntryPayload; +use tokio::time::Duration; +use tokio::time::sleep; use crate::RaftLog; use crate::test_utils::RaftLogCoreTestContext; diff --git a/d-engine-core/src/storage/raft_log_core_test/id_allocation_test.rs b/d-engine-core/src/storage/raft_log_core_test/id_allocation_test.rs index 4ec756ff..02e57a25 100644 --- a/d-engine-core/src/storage/raft_log_core_test/id_allocation_test.rs +++ b/d-engine-core/src/storage/raft_log_core_test/id_allocation_test.rs @@ -10,7 +10,10 @@ use std::ops::RangeInclusive; use std::sync::Arc; use std::sync::atomic::Ordering; -use crate::{MockStorageEngine, MockTypeConfig, RaftLog, RaftLogCore}; +use crate::MockStorageEngine; +use crate::MockTypeConfig; +use crate::RaftLog; +use crate::RaftLogCore; fn setup_memory() -> Arc> { let storage = Arc::new(MockStorageEngine::new()); diff --git a/d-engine-core/src/storage/raft_log_core_test/performance_test.rs b/d-engine-core/src/storage/raft_log_core_test/performance_test.rs index c11ea47d..6488f5db 100644 --- a/d-engine-core/src/storage/raft_log_core_test/performance_test.rs +++ b/d-engine-core/src/storage/raft_log_core_test/performance_test.rs @@ -7,11 +7,17 @@ use std::sync::Arc; use std::time::Duration; use bytes::Bytes; +use d_engine_proto::common::Entry; +use d_engine_proto::common::EntryPayload; use tokio::sync::Barrier; use tokio::time::Instant; -use crate::{MockLogStore, MockMetaStore, MockStorageEngine, MockTypeConfig, RaftLog, RaftLogCore}; -use d_engine_proto::common::{Entry, EntryPayload}; +use crate::MockLogStore; +use crate::MockMetaStore; +use crate::MockStorageEngine; +use crate::MockTypeConfig; +use crate::RaftLog; +use crate::RaftLogCore; // Test helper: Creates storage with controllable delay fn create_delayed_storage(delay_ms: u64) -> Arc { @@ -115,15 +121,11 @@ async fn test_filter_conflicts_performance_during_flush() { // Measure performance during active flush let start = Instant::now(); - log.filter_out_conflicts_and_append( - 500, - 1, - vec![Entry { - index: 501, - term: 1, - payload: Some(EntryPayload::command(Bytes::from(vec![1; 256]))), - }], - ) + log.filter_out_conflicts_and_append(500, 1, vec![Entry { + index: 501, + term: 1, + payload: Some(EntryPayload::command(Bytes::from(vec![1; 256]))), + }]) .await .unwrap(); diff --git a/d-engine-core/src/storage/raft_log_core_test/pipeline_overlap_test.rs b/d-engine-core/src/storage/raft_log_core_test/pipeline_overlap_test.rs index 53808657..fd9d2b61 100644 --- a/d-engine-core/src/storage/raft_log_core_test/pipeline_overlap_test.rs +++ b/d-engine-core/src/storage/raft_log_core_test/pipeline_overlap_test.rs @@ -6,12 +6,18 @@ //! - Partial overlap with real term conflict (truncate only from conflict point) use std::sync::Arc; -use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::atomic::AtomicU64; +use std::sync::atomic::Ordering; use d_engine_proto::common::Entry; +use crate::MockLogStore; +use crate::MockMetaStore; +use crate::MockStorageEngine; +use crate::MockTypeConfig; +use crate::RaftLog; +use crate::RaftLogCore; use crate::test_utils::RaftLogCoreTestContext; -use crate::{MockLogStore, MockMetaStore, MockStorageEngine, MockTypeConfig, RaftLog, RaftLogCore}; fn ctx(name: &str) -> RaftLogCoreTestContext { RaftLogCoreTestContext::new(name) @@ -300,7 +306,8 @@ async fn test_filter_conflicts_bench_pattern_no_truncation() { /// After recovery: [1,2,3(t1), 4(t2), 5(t2)] — index=4 must be term=2 #[tokio::test] async fn test_replace_range_conflict_persists_correctly_after_crash() { - use tokio::time::{Duration, sleep}; + use tokio::time::Duration; + use tokio::time::sleep; let ctx = ctx("replace_range_crash_recovery"); @@ -361,11 +368,12 @@ async fn test_replace_range_conflict_persists_correctly_after_crash() { /// /// # Scenario /// Follower: [1,2,3,4,5] all term=1 (flushed) -/// Leader sends: prev_index=2, prev_term=1, entries=[3(t2), 4(t2)] — conflict at index=3, no entry 5 -/// After: log must be [1,2,3(t2),4(t2)] — entry 5 must not exist +/// Leader sends: prev_index=2, prev_term=1, entries=[3(t2), 4(t2)] — conflict at index=3, no entry +/// 5 After: log must be [1,2,3(t2),4(t2)] — entry 5 must not exist #[tokio::test] async fn test_replace_range_removes_entries_beyond_new_tail() { - use tokio::time::{Duration, sleep}; + use tokio::time::Duration; + use tokio::time::sleep; let ctx = ctx("replace_range_removes_trailing"); diff --git a/d-engine-core/src/storage/raft_log_core_test/prev_log_index_zero_idempotency_test.rs b/d-engine-core/src/storage/raft_log_core_test/prev_log_index_zero_idempotency_test.rs index eb98ce8c..d2833306 100644 --- a/d-engine-core/src/storage/raft_log_core_test/prev_log_index_zero_idempotency_test.rs +++ b/d-engine-core/src/storage/raft_log_core_test/prev_log_index_zero_idempotency_test.rs @@ -16,10 +16,12 @@ //! progress. These tests are RED until `prev_log_index == 0` is folded into the same //! overlap/conflict comparison used for `prev_log_index > 0`. +use std::time::Duration; + +use d_engine_proto::common::Entry; + use crate::storage::raft_log::RaftLog; use crate::test_utils::RaftLogCoreTestContext; -use d_engine_proto::common::Entry; -use std::time::Duration; fn ctx(name: &str) -> RaftLogCoreTestContext { RaftLogCoreTestContext::new(name) @@ -61,17 +63,13 @@ async fn test_filter_conflicts_zero_prev_on_genuinely_empty_log_appends_all() { // Act: the very first AppendEntries this follower ever receives. let result = ctx .raft_log - .filter_out_conflicts_and_append( - 0, - 0, - vec![ - entry(1, 1), - entry(2, 1), - entry(3, 1), - entry(4, 1), - entry(5, 1), - ], - ) + .filter_out_conflicts_and_append(0, 0, vec![ + entry(1, 1), + entry(2, 1), + entry(3, 1), + entry(4, 1), + entry(5, 1), + ]) .await .unwrap(); @@ -112,17 +110,13 @@ async fn test_filter_conflicts_zero_prev_duplicate_resend_preserves_durable_inde // hasn't processed a response yet), not a genuinely new peer. let result = ctx .raft_log - .filter_out_conflicts_and_append( - 0, - 0, - vec![ - entry(1, 1), - entry(2, 1), - entry(3, 1), - entry(4, 1), - entry(5, 1), - ], - ) + .filter_out_conflicts_and_append(0, 0, vec![ + entry(1, 1), + entry(2, 1), + entry(3, 1), + entry(4, 1), + entry(5, 1), + ]) .await .unwrap(); diff --git a/d-engine-core/src/storage/raft_log_core_test/quorum_durability_test.rs b/d-engine-core/src/storage/raft_log_core_test/quorum_durability_test.rs index 337f6f21..ebf5ff47 100644 --- a/d-engine-core/src/storage/raft_log_core_test/quorum_durability_test.rs +++ b/d-engine-core/src/storage/raft_log_core_test/quorum_durability_test.rs @@ -31,15 +31,17 @@ //! runtime), and it was intermittently losing under load — flaky, not broken logic. Do not //! reintroduce that pattern here. +use std::sync::Arc; +use std::time::Duration; + +use d_engine_proto::common::Entry; +use d_engine_proto::common::LogId; + use crate::MockStorageEngine; use crate::MockTypeConfig; use crate::RaftLogCore; use crate::storage::raft_log::RaftLog; use crate::test_utils::RaftLogCoreTestContext; -use d_engine_proto::common::Entry; -use d_engine_proto::common::LogId; -use std::sync::Arc; -use std::time::Duration; /// Entries `1..=n`, all at `term`, no payload — the shape these tests need. fn entries( diff --git a/d-engine-core/src/storage/raft_log_core_test/raft_properties_test.rs b/d-engine-core/src/storage/raft_log_core_test/raft_properties_test.rs index 13515bde..76bc42a8 100644 --- a/d-engine-core/src/storage/raft_log_core_test/raft_properties_test.rs +++ b/d-engine-core/src/storage/raft_log_core_test/raft_properties_test.rs @@ -1,7 +1,9 @@ -use crate::storage::raft_log::RaftLog; -use crate::test_utils::{RaftLogCoreTestContext, simulate_insert_command}; use d_engine_proto::common::Entry; +use crate::storage::raft_log::RaftLog; +use crate::test_utils::RaftLogCoreTestContext; +use crate::test_utils::simulate_insert_command; + #[tokio::test] async fn test_log_matching_property() { let ctx = RaftLogCoreTestContext::new("test_log_matching"); @@ -19,15 +21,11 @@ async fn test_log_matching_property() { // Conflict resolution: new leader with [1,1] [2,1] [3,2] [4,3] let result = ctx .raft_log - .filter_out_conflicts_and_append( - 3, - 2, - vec![Entry { - index: 4, - term: 3, - payload: None, - }], - ) + .filter_out_conflicts_and_append(3, 2, vec![Entry { + index: 4, + term: 3, + payload: None, + }]) .await .unwrap(); @@ -211,7 +209,8 @@ async fn test_majority_matched_index_is_none_when_it_does_not_pass_the_commit_in None, "median == commit_index: nothing new to commit" ); - // A majority below the commit index (possible after a membership change) is not an advance either. + // A majority below the commit index (possible after a membership change) is not an advance + // either. assert_eq!( ctx.raft_log.calculate_majority_matched_index(1, 50, vec![50, 40, 40, 40]), None diff --git a/d-engine-core/src/storage/raft_log_core_test/remove_range_test.rs b/d-engine-core/src/storage/raft_log_core_test/remove_range_test.rs index 11ce3da4..45d61788 100644 --- a/d-engine-core/src/storage/raft_log_core_test/remove_range_test.rs +++ b/d-engine-core/src/storage/raft_log_core_test/remove_range_test.rs @@ -1,7 +1,9 @@ -use d_engine_proto::common::{Entry, LogId}; +use d_engine_proto::common::Entry; +use d_engine_proto::common::LogId; use crate::storage::raft_log::RaftLog; -use crate::test_utils::{RaftLogCoreTestContext, simulate_insert_command}; +use crate::test_utils::RaftLogCoreTestContext; +use crate::test_utils::simulate_insert_command; fn entry( index: u64, @@ -247,8 +249,8 @@ async fn test_purge_prefix_removes_entries_and_records_boundary_together() { /// # Expected /// - term1's index is fully gone: `first/last_index_for_term(1) == None`. /// - term2's surviving range starts at 46: `first_index_for_term(2) == Some(46)`. -/// - The purge boundary itself resolves to term 2 (the cutoff's own term), and -/// the surviving term3 range is untouched. +/// - The purge boundary itself resolves to term 2 (the cutoff's own term), and the surviving term3 +/// range is untouched. #[tokio::test] async fn test_purge_prefix_multi_term_cutoff_updates_term_indexes_and_boundary() { let ctx = RaftLogCoreTestContext::new("test_purge_prefix_multi_term_cutoff"); diff --git a/d-engine-core/src/storage/raft_log_core_test/shutdown_test.rs b/d-engine-core/src/storage/raft_log_core_test/shutdown_test.rs index ff337a2a..adafa1f2 100644 --- a/d-engine-core/src/storage/raft_log_core_test/shutdown_test.rs +++ b/d-engine-core/src/storage/raft_log_core_test/shutdown_test.rs @@ -2,11 +2,17 @@ use std::sync::Arc; use std::time::Duration; use bytes::Bytes; - +use d_engine_proto::common::Entry; +use d_engine_proto::common::EntryPayload; + +use crate::MockLogStore; +use crate::MockMetaStore; +use crate::MockStorageEngine; +use crate::MockTypeConfig; +use crate::RaftLogCore; use crate::storage::raft_log::RaftLog; -use crate::test_utils::{RaftLogCoreTestContext, drain_and_apply_fsync_completions}; -use crate::{MockLogStore, MockMetaStore, MockStorageEngine, MockTypeConfig, RaftLogCore}; -use d_engine_proto::common::{Entry, EntryPayload}; +use crate::test_utils::RaftLogCoreTestContext; +use crate::test_utils::drain_and_apply_fsync_completions; fn entry( index: u64, @@ -155,8 +161,8 @@ async fn test_close_flushes_multiple_batches() { /// Verifies that a fatal `replace_range` failure: /// 1. Propagates the error synchronously to the caller. -/// 2. Poisons the log — writes after the failure are rejected outright, not -/// silently accepted into memory with nothing left to ever persist them. +/// 2. Poisons the log — writes after the failure are rejected outright, not silently accepted into +/// memory with nothing left to ever persist them. #[tokio::test] async fn test_replace_range_failure_poisons_log_and_rejects_future_writes() { let mut log_store = MockLogStore::new(); diff --git a/d-engine-core/src/storage/raft_log_core_test/term_segments_test.rs b/d-engine-core/src/storage/raft_log_core_test/term_segments_test.rs index 53d77b65..6e1d440a 100644 --- a/d-engine-core/src/storage/raft_log_core_test/term_segments_test.rs +++ b/d-engine-core/src/storage/raft_log_core_test/term_segments_test.rs @@ -214,27 +214,23 @@ async fn test_entry_term_after_truncation_reappend() { .unwrap(); } ctx.raft_log - .filter_out_conflicts_and_append( - 6, - 2, - vec![ - Entry { - index: 7, - term: 3, - payload: None, - }, - Entry { - index: 8, - term: 3, - payload: None, - }, - Entry { - index: 9, - term: 3, - payload: None, - }, - ], - ) + .filter_out_conflicts_and_append(6, 2, vec![ + Entry { + index: 7, + term: 3, + payload: None, + }, + Entry { + index: 8, + term: 3, + payload: None, + }, + Entry { + index: 9, + term: 3, + payload: None, + }, + ]) .await .unwrap(); diff --git a/d-engine-core/src/storage/raft_log_core_test/truncation_fsync_fence_test.rs b/d-engine-core/src/storage/raft_log_core_test/truncation_fsync_fence_test.rs index 0b44fba8..d82ba9f1 100644 --- a/d-engine-core/src/storage/raft_log_core_test/truncation_fsync_fence_test.rs +++ b/d-engine-core/src/storage/raft_log_core_test/truncation_fsync_fence_test.rs @@ -21,8 +21,10 @@ use std::time::Duration; use d_engine_proto::common::Entry; +use crate::MockStorageEngine; +use crate::MockTypeConfig; +use crate::RaftLogCore; use crate::storage::raft_log::RaftLog; -use crate::{MockStorageEngine, MockTypeConfig, RaftLogCore}; fn entry( index: u64, diff --git a/d-engine-core/src/storage/state_machine.rs b/d-engine-core/src/storage/state_machine.rs index ecb94309..66830fe3 100644 --- a/d-engine-core/src/storage/state_machine.rs +++ b/d-engine-core/src/storage/state_machine.rs @@ -2,17 +2,17 @@ //! //! See the [server customization guide](https://github.com/deventlab/d-engine/blob/master/d-engine/src/docs/server_guide/customize-state-machine.md) for details. -use crate::Error; -use crate::StorageError; use async_trait::async_trait; use bytes::Bytes; use d_engine_proto::common::LogId; use d_engine_proto::server::storage::SnapshotMetadata; - -use crate::ApplyEntry; #[cfg(any(test, feature = "__test_support"))] use mockall::automock; +use crate::ApplyEntry; +use crate::Error; +use crate::StorageError; + /// All `(key, value)` pairs returned by a prefix scan, plus the revision anchor. /// /// `revision` equals `last_applied_index` at scan time — clients use it as the @@ -94,20 +94,23 @@ pub trait StateMachine: Send + Sync + 'static { /// Called once during node startup. Performance is not critical. async fn start(&self) -> Result<(), Error>; - /// Stops the state machine service gracefully. - /// This is typically a sync operation for state management. - fn stop(&self) -> Result<(), Error>; - - /// Permanently close underlying storage resources (e.g. DB handle, file descriptors). + /// Permanently closes the state machine: stops serving, persists what must survive a + /// restart (e.g. lease state, metadata), flushes, and releases exclusive resources + /// (e.g. a DB handle or its lock file). /// - /// Called by `EmbeddedEngine::stop()` before the Raft loop exits to release - /// OS-level resources (e.g. RocksDB LOCK file) without waiting for all - /// `Arc` clones to drop. + /// Called once by the state machine worker as its last act, after everything queued for + /// it has been applied. The developer may keep their own `Arc` to the state machine, so + /// this must not rely on `Drop`. Must be synchronous and idempotent; afterwards every + /// operation returns `NotServing`. /// - /// Default is a no-op — custom state machine implementations do not need to - /// override this unless they hold exclusive OS resources that must be released - /// before the process exits or a new engine instance is started. - fn close_storage(&self) {} + /// When it returns, reopening the same storage restores every applied entry and a + /// `last_applied` that is not older than the one before the call, and every exclusive + /// resource is released. If persisting fails, still release the resources and return the + /// first error. A second call returns `Ok(())`. + /// Default is a no-op, for implementations with nothing to persist or release. + fn close(&self) -> Result<(), Error> { + Ok(()) + } /// Checks if the state machine is currently running. /// Sync operation as it just checks an atomic boolean. diff --git a/d-engine-core/src/storage/state_machine_test.rs b/d-engine-core/src/storage/state_machine_test.rs index fa0689ca..07702d1f 100644 --- a/d-engine-core/src/storage/state_machine_test.rs +++ b/d-engine-core/src/storage/state_machine_test.rs @@ -1,18 +1,20 @@ -use crate::ApplyEntry; -use crate::Command; -use crate::Error; -use crate::SnapshotApplyResult; -use crate::storage::StateMachine; -use async_trait::async_trait; -use bytes::Bytes; -use d_engine_proto::common::LogId; -use d_engine_proto::server::storage::SnapshotMetadata; use std::path::PathBuf; use std::sync::Arc; use std::time::Duration; use std::time::Instant; + +use async_trait::async_trait; +use bytes::Bytes; +use d_engine_proto::common::LogId; +use d_engine_proto::server::storage::SnapshotMetadata; use tempfile::TempDir; +use crate::ApplyEntry; +use crate::Command; +use crate::Error; +use crate::SnapshotApplyResult; +use crate::storage::StateMachine; + /// Test suite for StateMachine implementations /// /// This suite provides comprehensive tests that can be used to validate @@ -34,7 +36,7 @@ pub trait StateMachineBuilder: Send + Sync { impl StateMachineTestSuite { /// Run all state machine tests pub async fn run_all_tests(builder: B) -> Result<(), Error> { - Self::test_start_stop(builder.build().await?).await?; + Self::test_start_close(builder.build().await?).await?; Self::test_basic_kv_operations(builder.build().await?).await?; Self::test_apply_chunk_functionality(builder.build().await?).await?; Self::test_cas_operations(builder.build().await?).await?; @@ -51,6 +53,8 @@ impl StateMachineTestSuite { Self::test_data_survives_reopen(&builder).await?; Self::test_batch_survives_reopen(&builder).await?; Self::test_ungraceful_shutdown_recovery(&builder).await?; + Self::test_close_persists_without_drop(&builder).await?; + Self::test_operations_fail_after_close(builder.build().await?).await?; Self::test_reset_operation(builder.build().await?).await?; Self::test_get_multi_returns_values_in_key_order(builder.build().await?).await?; @@ -76,18 +80,18 @@ impl StateMachineTestSuite { Ok(()) } - /// Test start/stop functionality - pub async fn test_start_stop(state_machine: Arc) -> Result<(), Error> { + /// Test start/close lifecycle: a state machine is running after `start`, and not running + /// once `close`d. `close` is permanent and idempotent (#455), so there is no restart. + pub async fn test_start_close(state_machine: Arc) -> Result<(), Error> { // Test default state assert!(state_machine.is_running()); - // Test explicit start/stop state_machine.start().await?; assert!(state_machine.is_running()); - state_machine.stop()?; + state_machine.close()?; + assert!(!state_machine.is_running()); + state_machine.close()?; // idempotent assert!(!state_machine.is_running()); - state_machine.start().await?; - assert!(state_machine.is_running()); Ok(()) } @@ -152,7 +156,7 @@ impl StateMachineTestSuite { sm.apply_chunk(&[entry3]).await?; assert_eq!(sm.get(b"lock")?, Some(b"owner2".to_vec().into())); // Still owner2 - sm.stop()?; + sm.close()?; Ok(()) } @@ -163,22 +167,19 @@ impl StateMachineTestSuite { sm.start().await?; - let entry = create_batch_entry( - 1, - vec![ - BatchOp::Insert { - key: b"k1".to_vec().into(), - value: b"v1".to_vec().into(), - }, - BatchOp::Insert { - key: b"k2".to_vec().into(), - value: b"v2".to_vec().into(), - }, - BatchOp::Delete { - key: b"k1".to_vec().into(), - }, - ], - ); + let entry = create_batch_entry(1, vec![ + BatchOp::Insert { + key: b"k1".to_vec().into(), + value: b"v1".to_vec().into(), + }, + BatchOp::Insert { + key: b"k2".to_vec().into(), + value: b"v2".to_vec().into(), + }, + BatchOp::Delete { + key: b"k1".to_vec().into(), + }, + ]); sm.apply_chunk(&[entry]).await?; // k1 was deleted after insert, k2 survived @@ -194,7 +195,7 @@ impl StateMachineTestSuite { ); assert_eq!(sm.last_applied().index, 1); - sm.stop()?; + sm.close()?; Ok(()) } @@ -630,8 +631,8 @@ impl StateMachineTestSuite { "Data should exist before stop" ); - // Call stop() for graceful shutdown - stop() must persist data - sm.stop()?; + // Call close() for graceful shutdown - close() must persist data + sm.close()?; // Explicitly drop Arc to release database locks (critical for RocksDB) drop(sm); @@ -648,12 +649,12 @@ impl StateMachineTestSuite { assert_eq!( value1, Some(Bytes::from(b"drop_test_value1".to_vec())), - "BUG: Data lost after stop! StateMachine.stop() must persist data." + "BUG: Data lost after close! StateMachine.close() must persist data." ); assert_eq!( value2, Some(Bytes::from(b"drop_test_value2".to_vec())), - "BUG: Data lost after stop! StateMachine.stop() must persist data." + "BUG: Data lost after close! StateMachine.close() must persist data." ); Ok(()) @@ -867,8 +868,8 @@ impl StateMachineTestSuite { last_applied_before = sm.last_applied(); assert_eq!(last_applied_before.index, 3); - // Graceful stop - sm.stop()?; + // Graceful close + sm.close()?; } // Step 2: Reopen and verify @@ -914,22 +915,19 @@ impl StateMachineTestSuite { let sm = builder.build().await?; sm.start().await?; - let entry = create_batch_entry( - 1, - vec![ - BatchOp::Insert { - key: b"bk1".to_vec().into(), - value: b"bv1".to_vec().into(), - }, - BatchOp::Insert { - key: b"bk2".to_vec().into(), - value: b"bv2".to_vec().into(), - }, - BatchOp::Delete { - key: b"bk1".to_vec().into(), - }, - ], - ); + let entry = create_batch_entry(1, vec![ + BatchOp::Insert { + key: b"bk1".to_vec().into(), + value: b"bv1".to_vec().into(), + }, + BatchOp::Insert { + key: b"bk2".to_vec().into(), + value: b"bv2".to_vec().into(), + }, + BatchOp::Delete { + key: b"bk1".to_vec().into(), + }, + ]); sm.apply_chunk(&[entry]).await?; assert_eq!(sm.get(b"bk1")?, None); @@ -955,7 +953,7 @@ impl StateMachineTestSuite { ); assert_eq!(sm2.last_applied().index, 1, "last_applied must be restored"); - sm2.stop()?; + sm2.close()?; Ok(()) } @@ -1048,6 +1046,102 @@ impl StateMachineTestSuite { Ok(()) } + /// `close()` alone must persist everything a restart needs, even while the developer still + /// holds the state machine. + /// + /// Business scenario: the node shuts down cleanly, but the application keeps its own `Arc` + /// to the state machine, so `Drop` never runs. A restart must still see every applied key + /// and the applied position; otherwise Raft replays entries that were already applied. + /// Every other reopen test in this suite drops the state machine first, which lets a `Drop` + /// impl hide a `close()` that forgot to persist. + pub async fn test_close_persists_without_drop( + builder: &B + ) -> Result<(), Error> { + let sm = builder.build().await?; + sm.start().await?; + sm.apply_chunk(&[ + create_insert_entry( + 1, + Bytes::from_static(b"close_key1"), + Bytes::from_static(b"v1"), + ), + create_insert_entry( + 2, + Bytes::from_static(b"close_key2"), + Bytes::from_static(b"v2"), + ), + create_insert_entry( + 3, + Bytes::from_static(b"close_key3"), + Bytes::from_static(b"v3"), + ), + ]) + .await?; + assert_eq!(sm.last_applied().index, 3); + + sm.close()?; + // `sm` stays alive until the end of the test: Drop must not be what saves the state. + + let sm2 = builder.build().await?; + sm2.start().await?; + + assert_eq!( + sm2.last_applied().index, + 3, + "close() must persist the applied position without relying on Drop" + ); + assert_eq!( + sm2.get(b"close_key1")?, + Some(Bytes::from_static(b"v1")), + "close() must persist applied data without relying on Drop" + ); + assert_eq!(sm2.get(b"close_key3")?, Some(Bytes::from_static(b"v3"))); + + drop(sm); + Ok(()) + } + + /// After `close()` every operation must fail with `NotServing`, including writes. + /// + /// Business scenario: the application retains the state machine after the node has shut + /// down. A late read must not answer from state that is no longer maintained, and a late + /// `apply_chunk` must not mutate a state machine that already persisted its final state. + pub async fn test_operations_fail_after_close(sm: Arc) -> Result<(), Error> { + sm.start().await?; + sm.apply_chunk(&[create_insert_entry( + 1, + Bytes::from_static(b"serving_key"), + Bytes::from_static(b"v"), + )]) + .await?; + + sm.close()?; + + let not_serving = |err: Error| format!("{err:?}").contains("NotServing"); + assert!( + not_serving(sm.get(b"serving_key").expect_err("get must fail after close")), + "get after close must return NotServing" + ); + assert!( + not_serving(sm.scan_prefix(b"serving").expect_err("scan_prefix must fail after close")), + "scan_prefix after close must return NotServing" + ); + let late_write = sm + .apply_chunk(&[create_insert_entry( + 2, + Bytes::from_static(b"late_key"), + Bytes::from_static(b"v"), + )]) + .await + .expect_err("apply_chunk must fail after close"); + assert!( + not_serving(late_write), + "apply_chunk after close must return NotServing" + ); + + Ok(()) + } + /// /// This test verifies that the reset operation: /// 1. Clears all data from memory @@ -1651,15 +1745,19 @@ fn create_batch_entry( // scan_prefix, ensuring the default arm is reachable and exercised. #[cfg(test)] mod default_scan_prefix_tests { - use std::sync::atomic::{AtomicBool, Ordering}; + use std::sync::atomic::AtomicBool; + use std::sync::atomic::Ordering; use async_trait::async_trait; use bytes::Bytes; use d_engine_proto::common::LogId; use d_engine_proto::server::storage::SnapshotMetadata; - use crate::storage::state_machine::{ApplyResult, StateMachine}; - use crate::{ApplyEntry, Error, SnapshotApplyResult}; + use crate::ApplyEntry; + use crate::Error; + use crate::SnapshotApplyResult; + use crate::storage::state_machine::ApplyResult; + use crate::storage::state_machine::StateMachine; struct MinimalSm { running: AtomicBool, @@ -1672,7 +1770,7 @@ mod default_scan_prefix_tests { Ok(()) } - fn stop(&self) -> Result<(), Error> { + fn close(&self) -> Result<(), Error> { self.running.store(false, Ordering::SeqCst); Ok(()) } diff --git a/d-engine-core/src/test_utils/entry_builder_test.rs b/d-engine-core/src/test_utils/entry_builder_test.rs index 2506c30c..7fd35f49 100644 --- a/d-engine-core/src/test_utils/entry_builder_test.rs +++ b/d-engine-core/src/test_utils/entry_builder_test.rs @@ -184,10 +184,9 @@ fn test_entry_builder_multiple_commands() { let (builder, e3) = builder.command(b"cmd3"); let (_, e4) = builder.command(b"cmd4"); - assert_eq!( - vec![e1.index, e2.index, e3.index, e4.index], - vec![1, 2, 3, 4] - ); + assert_eq!(vec![e1.index, e2.index, e3.index, e4.index], vec![ + 1, 2, 3, 4 + ]); } #[test] diff --git a/d-engine-core/src/test_utils/metrics_capture.rs b/d-engine-core/src/test_utils/metrics_capture.rs index cf393804..52235864 100644 --- a/d-engine-core/src/test_utils/metrics_capture.rs +++ b/d-engine-core/src/test_utils/metrics_capture.rs @@ -1,9 +1,19 @@ -use metrics::{ - Counter, CounterFn, Gauge, GaugeFn, Histogram, HistogramFn, Key, KeyName, Metadata, Recorder, - SharedString, Unit, -}; use std::collections::HashMap; -use std::sync::{Arc, Mutex}; +use std::sync::Arc; +use std::sync::Mutex; + +use metrics::Counter; +use metrics::CounterFn; +use metrics::Gauge; +use metrics::GaugeFn; +use metrics::Histogram; +use metrics::HistogramFn; +use metrics::Key; +use metrics::KeyName; +use metrics::Metadata; +use metrics::Recorder; +use metrics::SharedString; +use metrics::Unit; #[derive(Default)] struct Store { diff --git a/d-engine-core/src/test_utils/mock/mock_raft_builder.rs b/d-engine-core/src/test_utils/mock/mock_raft_builder.rs index 6da00be4..715b1c0a 100644 --- a/d-engine-core/src/test_utils/mock/mock_raft_builder.rs +++ b/d-engine-core/src/test_utils/mock/mock_raft_builder.rs @@ -384,7 +384,7 @@ pub fn mock_state_machine() -> MockStateMachine { let mut mock = MockStateMachine::new(); mock.expect_start().returning(|| Ok(())); - mock.expect_stop().returning(|| Ok(())); + mock.expect_close().returning(|| Ok(())); mock.expect_is_running().returning(|| true); mock.expect_get().returning(|_| Ok(None)); @@ -416,6 +416,8 @@ pub fn mock_state_machine() -> MockStateMachine { pub fn mock_state_machine_handler() -> MockStateMachineHandler { let mut state_machine_handler = MockStateMachineHandler::new(); state_machine_handler.expect_update_pending().returning(|_| {}); + // Commit delivery (also run on shutdown) reads the applied position every iteration. + state_machine_handler.expect_last_applied().returning(|| 0); state_machine_handler.expect_read_from_state_machine().returning(|_| None); state_machine_handler.expect_should_snapshot().returning(|_| false); state_machine_handler.expect_get_latest_snapshot_metadata().returning(|| None); diff --git a/d-engine-core/src/test_utils/mock/mock_storage_engine.rs b/d-engine-core/src/test_utils/mock/mock_storage_engine.rs index 171b7815..995e735e 100644 --- a/d-engine-core/src/test_utils/mock/mock_storage_engine.rs +++ b/d-engine-core/src/test_utils/mock/mock_storage_engine.rs @@ -227,7 +227,9 @@ impl MockStorageEngine { }); } - /// Default `replace_range()`: always succeeds, atomically deletes entries >= `from_index` then inserts `new_entries`. Extracted out of `configure_mocks` for the same reason as `configure_persist_entries_success`. + /// Default `replace_range()`: always succeeds, atomically deletes entries >= `from_index` then + /// inserts `new_entries`. Extracted out of `configure_mocks` for the same reason as + /// `configure_persist_entries_success`. fn configure_replace_range_success( log_store: &mut MockLogStore, instance_id: &str, diff --git a/d-engine-core/src/test_utils/mock/mock_type_config.rs b/d-engine-core/src/test_utils/mock/mock_type_config.rs index 300b5842..11b1196e 100644 --- a/d-engine-core/src/test_utils/mock/mock_type_config.rs +++ b/d-engine-core/src/test_utils/mock/mock_type_config.rs @@ -1,4 +1,3 @@ -use crate::MockCommitHandler; use crate::MockElectionCore; use crate::MockMembership; use crate::MockPurgeExecutor; @@ -30,8 +29,6 @@ impl TypeConfig for MockTypeConfig { type REP = MockReplicationCore; - type C = MockCommitHandler; - type SMH = MockStateMachineHandler; type SMW = MockStateMachineWriterOps; diff --git a/d-engine-core/src/test_utils/mod.rs b/d-engine-core/src/test_utils/mod.rs index 4041c103..09e06473 100644 --- a/d-engine-core/src/test_utils/mod.rs +++ b/d-engine-core/src/test_utils/mod.rs @@ -22,11 +22,10 @@ pub use log_capture::*; #[cfg(any(test, feature = "__test_support"))] mod metrics_capture; -#[cfg(any(test, feature = "__test_support"))] -pub use metrics_capture::MetricsCapture; - pub use common::*; pub use entry_builder::*; +#[cfg(any(test, feature = "__test_support"))] +pub use metrics_capture::MetricsCapture; pub use mock::*; pub use raft_log_core_test_helpers::*; pub use replication_test_helpers::*; diff --git a/d-engine-core/src/test_utils/raft_log_core_test_helpers.rs b/d-engine-core/src/test_utils/raft_log_core_test_helpers.rs index 3d8be0a4..83b387b4 100644 --- a/d-engine-core/src/test_utils/raft_log_core_test_helpers.rs +++ b/d-engine-core/src/test_utils/raft_log_core_test_helpers.rs @@ -9,9 +9,13 @@ use std::sync::Arc; use std::sync::atomic::AtomicU64; use bytes::Bytes; -use d_engine_proto::common::{Entry, EntryPayload}; +use d_engine_proto::common::Entry; +use d_engine_proto::common::EntryPayload; -use crate::{MockStorageEngine, MockTypeConfig, RaftLog, RaftLogCore}; +use crate::MockStorageEngine; +use crate::MockTypeConfig; +use crate::RaftLog; +use crate::RaftLogCore; /// Test context for RaftLogCore tests pub struct RaftLogCoreTestContext { diff --git a/d-engine-core/src/test_utils/replication_test_helpers.rs b/d-engine-core/src/test_utils/replication_test_helpers.rs index 5545b1f2..ecb5ee70 100644 --- a/d-engine-core/src/test_utils/replication_test_helpers.rs +++ b/d-engine-core/src/test_utils/replication_test_helpers.rs @@ -5,10 +5,11 @@ use std::sync::Arc; -use crate::MockRaftLog; use d_engine_proto::common::Entry; use d_engine_proto::common::EntryPayload; +use crate::MockRaftLog; + /// Simplified test context for replication handler unit tests. /// /// This context contains only the essential components needed for testing diff --git a/d-engine-core/src/type_config.rs b/d-engine-core/src/type_config.rs index d426413a..3bbc7660 100644 --- a/d-engine-core/src/type_config.rs +++ b/d-engine-core/src/type_config.rs @@ -1,6 +1,3 @@ -use std::fmt::Debug; - -use crate::CommitHandler; use crate::ElectionCore; use crate::Membership; use crate::PurgeExecutor; @@ -13,18 +10,18 @@ use crate::StateMachineWriterOps; use crate::StorageEngine; use crate::Transport; -pub trait TypeConfig: Sync + Send + Sized + Debug + 'static { +pub trait TypeConfig: Sync + Send + Sized + std::fmt::Debug + 'static { // Required type - specified by the application layer // - type SE: StorageEngine + Debug; + type SE: StorageEngine + std::fmt::Debug; - type SM: StateMachine + Debug; + type SM: StateMachine + std::fmt::Debug; // Optional type - has default implementation // - type R: RaftLog + Debug; + type R: RaftLog + std::fmt::Debug; - type M: Membership + Debug; + type M: Membership + std::fmt::Debug; type TR: Transport; @@ -32,13 +29,11 @@ pub trait TypeConfig: Sync + Send + Sized + Debug + 'static { type REP: ReplicationCore; - type C: CommitHandler; - - type SMH: StateMachineHandler + Debug; + type SMH: StateMachineHandler + std::fmt::Debug; - type SMW: StateMachineWriterOps + Debug; + type SMW: StateMachineWriterOps + std::fmt::Debug; - type SNP: SnapshotPolicy + Debug; + type SNP: SnapshotPolicy + std::fmt::Debug; type PE: PurgeExecutor + Send + Sync + 'static; } @@ -60,8 +55,6 @@ pub mod alias { pub type REPOF = ::REP; - pub type COF = ::C; - pub type SMHOF = ::SMH; pub type SMWOF = ::SMW; diff --git a/d-engine-core/src/utils/cluster_printer.rs b/d-engine-core/src/utils/cluster_printer.rs index 6a167718..466a8fa9 100644 --- a/d-engine-core/src/utils/cluster_printer.rs +++ b/d-engine-core/src/utils/cluster_printer.rs @@ -262,7 +262,8 @@ pub fn print_role_transition( println!("{}", "=".repeat(80)); } -/// Single-line role transition log (lighter than `print_role_transition`'s banner); no node ID, callers already get one from the Makefile's `sed` prefix. +/// Single-line role transition log (lighter than `print_role_transition`'s banner); no node ID, +/// callers already get one from the Makefile's `sed` prefix. pub fn print_role_transition_line( from_role: i32, to_role: i32, diff --git a/d-engine-core/src/utils/convert_test.rs b/d-engine-core/src/utils/convert_test.rs index faf128ff..2e437a68 100644 --- a/d-engine-core/src/utils/convert_test.rs +++ b/d-engine-core/src/utils/convert_test.rs @@ -98,10 +98,9 @@ fn test_classify_error() { fn test_u64_to_bytes() { // Test basic conversion let result = u64_to_bytes(0x1234_5678_9ABC_DEF0); - assert_eq!( - result.as_ref(), - &[0x12, 0x34, 0x56, 0x78, 0x9A, 0xBC, 0xDE, 0xF0] - ); + assert_eq!(result.as_ref(), &[ + 0x12, 0x34, 0x56, 0x78, 0x9A, 0xBC, 0xDE, 0xF0 + ]); // Test zero let result = u64_to_bytes(0); @@ -109,10 +108,9 @@ fn test_u64_to_bytes() { // Test max value let result = u64_to_bytes(u64::MAX); - assert_eq!( - result.as_ref(), - &[0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF] - ); + assert_eq!(result.as_ref(), &[ + 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF + ]); } #[test] diff --git a/d-engine-core/src/utils/file_io_test.rs b/d-engine-core/src/utils/file_io_test.rs index 479649b9..b7e8a84a 100644 --- a/d-engine-core/src/utils/file_io_test.rs +++ b/d-engine-core/src/utils/file_io_test.rs @@ -1,23 +1,25 @@ -use crate::Error; -use crate::FileError; -use crate::StorageError; -use crate::SystemError; -use crate::file_io; -use crate::file_io::convert_vec_checksum; -use crate::file_io::create_parent_dir_if_not_exist; -use crate::file_io::delete_file; +use std::io::Write; +use std::os::fd::AsRawFd; +use std::os::unix::fs::PermissionsExt; + use nix::libc::LOCK_EX; use nix::libc::flock; use sha2::Digest; use sha2::Sha256; -use std::io::Write; -use std::os::fd::AsRawFd; -use std::os::unix::fs::PermissionsExt; use tempfile::NamedTempFile; use tempfile::tempdir; use tokio::fs; use tracing_test::traced_test; +use crate::Error; +use crate::FileError; +use crate::StorageError; +use crate::SystemError; +use crate::file_io; +use crate::file_io::convert_vec_checksum; +use crate::file_io::create_parent_dir_if_not_exist; +use crate::file_io::delete_file; + /// Passed: "/tmp/files/data.txt" /// Expected: "/tmp/files" created #[tokio::test] diff --git a/d-engine-core/src/utils/stream.rs b/d-engine-core/src/utils/stream.rs index aad9db19..3e0da583 100644 --- a/d-engine-core/src/utils/stream.rs +++ b/d-engine-core/src/utils/stream.rs @@ -1,5 +1,6 @@ -use bytes::BufMut; use std::marker::PhantomData; + +use bytes::BufMut; use tonic::Code; use tonic::Status; diff --git a/d-engine-core/src/watch/manager.rs b/d-engine-core/src/watch/manager.rs index d0b92643..c2b06f8d 100644 --- a/d-engine-core/src/watch/manager.rs +++ b/d-engine-core/src/watch/manager.rs @@ -21,15 +21,18 @@ //! - **No hidden resource allocation**: All tokio::spawn calls are explicit in Builder //! - **Minimal abstraction**: Only essential data structures, no unnecessary wrappers //! - **Composable**: Registry and Dispatcher are independent, composed in Builder -//! - **O(depth) prefix dispatch**: Decompose event key into slash-terminated path segments, -//! O(1) DashMap lookup per segment — dispatch cost is independent of prefix watcher count +//! - **O(depth) prefix dispatch**: Decompose event key into slash-terminated path segments, O(1) +//! DashMap lookup per segment — dispatch cost is independent of prefix watcher count //! - **Proto boundary**: WatchResponse (proto) lives only in the broadcast channel and handler; //! WatchEvent (opaque) is what callers see — no proto import required. use std::collections::hash_map::DefaultHasher; -use std::hash::{Hash, Hasher}; +use std::hash::Hash; +use std::hash::Hasher; use std::sync::Arc; -use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering}; +use std::sync::atomic::AtomicU64; +use std::sync::atomic::AtomicUsize; +use std::sync::atomic::Ordering; use std::time::Duration; use bytes::Bytes; @@ -57,10 +60,10 @@ pub enum WatchEventType { /// All fields use standard Rust types — callers never need to import proto types. /// /// `prev_value` semantics: -/// - `None` — watcher was registered with `prev_kv = false`, or the event is -/// `Progress` / `Canceled` (not a data mutation). -/// - `Some(Bytes::new())` — watcher has `prev_kv = true` and the key did not exist -/// before this write (fresh insert). +/// - `None` — watcher was registered with `prev_kv = false`, or the event is `Progress` / +/// `Canceled` (not a data mutation). +/// - `Some(Bytes::new())` — watcher has `prev_kv = true` and the key did not exist before this +/// write (fresh insert). /// - `Some(v)` — watcher has `prev_kv = true` and `v` is the previous value. #[derive(Debug, Clone)] pub struct WatchEvent { diff --git a/d-engine-core/src/watch/manager_test.rs b/d-engine-core/src/watch/manager_test.rs index 58ddd365..c10983e5 100644 --- a/d-engine-core/src/watch/manager_test.rs +++ b/d-engine-core/src/watch/manager_test.rs @@ -15,11 +15,10 @@ use d_engine_proto::client::WatchResponse; use tokio::sync::broadcast; use tokio::sync::mpsc; use tokio::time::Duration; - -use crate::watch::manager::prefix_segments; use tokio::time::timeout; use super::*; +use crate::watch::manager::prefix_segments; /// Helper to create test watch system components. /// `heartbeat_interval_ms` = 0 disables heartbeat (avoids Progress noise in most tests). @@ -735,14 +734,11 @@ fn setup_watch_system_with_max( fn test_prefix_segments_decomposes_path_correctly() { let key = Bytes::from("/config/db/host"); let segments = prefix_segments(&key); - assert_eq!( - segments, - vec![ - Bytes::from("/"), - Bytes::from("/config/"), - Bytes::from("/config/db/"), - ] - ); + assert_eq!(segments, vec![ + Bytes::from("/"), + Bytes::from("/config/"), + Bytes::from("/config/db/"), + ]); } #[test] diff --git a/d-engine-server/benches/lease_performance.rs b/d-engine-server/benches/lease_performance.rs index bc07b70a..1384dcac 100644 --- a/d-engine-server/benches/lease_performance.rs +++ b/d-engine-server/benches/lease_performance.rs @@ -10,7 +10,8 @@ //! 3. Batch operations with TTL use std::sync::Arc; -use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::atomic::AtomicU64; +use std::sync::atomic::Ordering; use bytes::Bytes; use criterion::BenchmarkId; @@ -18,7 +19,9 @@ use criterion::Criterion; use criterion::black_box; use criterion::criterion_group; use criterion::criterion_main; -use d_engine_core::{ApplyEntry, Command, StateMachine}; +use d_engine_core::ApplyEntry; +use d_engine_core::Command; +use d_engine_core::StateMachine; use d_engine_server::storage::FileStateMachine; use d_engine_server::storage::TtlLease; use tempfile::TempDir; diff --git a/d-engine-server/benches/state_machine.rs b/d-engine-server/benches/state_machine.rs index 13d50eea..f7dc9b4a 100644 --- a/d-engine-server/benches/state_machine.rs +++ b/d-engine-server/benches/state_machine.rs @@ -10,20 +10,24 @@ //! - End-to-end Watch notification latency: < 100µs //! - Batch operations: Linear scaling +use std::sync::Arc; +use std::sync::atomic::AtomicU64; +use std::sync::atomic::Ordering; +use std::time::Duration; + use bytes::Bytes; use criterion::BenchmarkId; use criterion::Criterion; use criterion::black_box; use criterion::criterion_group; use criterion::criterion_main; +use d_engine_core::ApplyEntry; +use d_engine_core::Command; +use d_engine_core::StateMachine; use d_engine_core::watch::WatchDispatcher; use d_engine_core::watch::WatchRegistry; use d_engine_core::watch::WatcherHandle; -use d_engine_core::{ApplyEntry, Command, StateMachine}; use d_engine_server::storage::FileStateMachine; -use std::sync::Arc; -use std::sync::atomic::{AtomicU64, Ordering}; -use std::time::Duration; use tempfile::TempDir; use tokio::sync::broadcast; use tokio::sync::mpsc; diff --git a/d-engine-server/benches/ttl.rs b/d-engine-server/benches/ttl.rs index 87a7ccdf..97d914b0 100644 --- a/d-engine-server/benches/ttl.rs +++ b/d-engine-server/benches/ttl.rs @@ -8,7 +8,8 @@ //! - TTL registration: < 100ns per entry //! - Expired check: < 50ns per key -use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::atomic::AtomicU64; +use std::sync::atomic::Ordering; use std::time::Duration; use bytes::Bytes; @@ -17,7 +18,10 @@ use criterion::Criterion; use criterion::black_box; use criterion::criterion_group; use criterion::criterion_main; -use d_engine_core::{ApplyEntry, Command, Lease, StateMachine}; +use d_engine_core::ApplyEntry; +use d_engine_core::Command; +use d_engine_core::Lease; +use d_engine_core::StateMachine; use d_engine_server::storage::FileStateMachine; use tempfile::TempDir; diff --git a/d-engine-server/benches/watch_overhead.rs b/d-engine-server/benches/watch_overhead.rs index fa55b0a7..7604bd74 100644 --- a/d-engine-server/benches/watch_overhead.rs +++ b/d-engine-server/benches/watch_overhead.rs @@ -20,7 +20,9 @@ use criterion::Criterion; use criterion::black_box; use criterion::criterion_group; use criterion::criterion_main; -use d_engine_core::{ApplyEntry, Command, StateMachine}; +use d_engine_core::ApplyEntry; +use d_engine_core::Command; +use d_engine_core::StateMachine; use d_engine_server::RocksDBUnifiedEngine; use d_engine_server::api::DefaultEmbeddedEngine; use tempfile::TempDir; diff --git a/d-engine-server/src/api/embedded.rs b/d-engine-server/src/api/embedded.rs index d8fd8c19..225d6f05 100644 --- a/d-engine-server/src/api/embedded.rs +++ b/d-engine-server/src/api/embedded.rs @@ -102,13 +102,26 @@ //! //! **Why doesn't EmbeddedEngine auto-forward writes?** //! -//! 1. **Preserves zero-overhead guarantee** - Auto-forwarding requires gRPC client (adds network/serialization) +//! 1. **Preserves zero-overhead guarantee** - Auto-forwarding requires gRPC client (adds +//! network/serialization) //! 2. **Maintains simplicity** - No hidden network calls in "embedded" mode -//! 3. **Flexible deployment** - Applications choose load balancer health checks, HTTP redirect, or other strategies +//! 3. **Flexible deployment** - Applications choose load balancer health checks, HTTP redirect, or +//! other strategies //! 4. **Performance transparency** - Developers know exactly when network calls occur //! //! For auto-forwarding with gRPC overhead, use standalone mode with `GrpcClient`. +use std::fmt::Debug; +use std::sync::Arc; +use std::time::Duration; + +use d_engine_core::TypeConfig; +use tokio::sync::Mutex; +use tokio::sync::watch; +use tokio::task::JoinHandle; +use tracing::error; +use tracing::info; + use super::embedded_client::EmbeddedClient; use super::embedded_read_handle::EmbeddedReadHandle; use crate::Result; @@ -122,20 +135,8 @@ use crate::StateMachine; use crate::StorageEngine; use crate::node::NodeBuilder; use crate::node::RaftTypeConfig; -use d_engine_core::TypeConfig; -use std::fmt::Debug; -use std::sync::Arc; -use std::time::Duration; -use tokio::sync::Mutex; -use tokio::sync::watch; -use tokio::task::JoinHandle; -use tracing::error; -use tracing::info; struct Inner { - /// SM reference for shutdown: close_storage() releases OS resources (e.g. RocksDB LOCK) - /// before the Raft loop exits, without waiting for all Arc clones to drop. - sm: Arc, node_handle: Mutex>>>, shutdown_tx: watch::Sender<()>, client: Arc>, @@ -398,7 +399,6 @@ where // for the direct-SM fast path. LOCK release is decoupled from Arc counting // (via close_db()), so holding multiple Arc clones is safe. let sm_for_client = Arc::clone(&state_machine); - let sm_for_engine = Arc::clone(&state_machine); let mut builder = NodeBuilder::init(data_dir, node_config.validate()?, shutdown_rx) .storage_engine(storage_engine) @@ -454,7 +454,6 @@ where Ok(Self { inner: Arc::new(Inner { - sm: sm_for_engine, node_handle: Mutex::new(Some(node_handle)), shutdown_tx, client, @@ -571,15 +570,18 @@ impl EmbeddedEngine { /// (node join, removal, or learner promotion) is committed by a majority. /// /// All nodes — leader, follower, and learner — fire the notification because - /// every node walks the same `CommitHandler::apply_config_change()` path. + /// every node applies a committed config change through the same delivery path + /// (`dispatch_committed`). /// /// The first `borrow()` returns the current membership state immediately /// without waiting for a change. /// /// ## Distinguishing this from other notifiers /// - /// - [`Self::leader_change_notifier`]: fires on leader election changes, **not** membership changes - /// - [`Self::wait_ready`]: resolves when a leader is elected and the cluster is ready, **not** membership changes + /// - [`Self::leader_change_notifier`]: fires on leader election changes, **not** membership + /// changes + /// - [`Self::wait_ready`]: resolves when a leader is elected and the cluster is ready, **not** + /// membership changes /// /// ## Usage /// ```ignore @@ -698,11 +700,6 @@ impl EmbeddedEngine { info!("Stopping embedded d-engine"); - // Close SM storage resources first (releases RocksDB LOCK immediately). - // Must happen before the Raft loop exits so the LOCK is freed before any - // subsequent engine restart on the same data directory. - self.inner.sm.close_storage(); - // Send shutdown signal to Raft node let _ = self.inner.shutdown_tx.send(()); diff --git a/d-engine-server/src/api/embedded_client.rs b/d-engine-server/src/api/embedded_client.rs index 45a02a3e..1ea927f0 100644 --- a/d-engine-server/src/api/embedded_client.rs +++ b/d-engine-server/src/api/embedded_client.rs @@ -15,13 +15,10 @@ //! client.put(b"key", b"value").await?; //! ``` -use super::embedded_read_handle::EmbeddedReadHandle; -pub(crate) use super::standalone_read_handle::channel_closed_error; -pub(crate) use super::standalone_read_handle::map_error_response; -pub(crate) use super::standalone_read_handle::server_error; -pub(crate) use super::standalone_read_handle::timeout_error; -use crate::api::types::WriteOperation; -use crate::proto_convert; +#[cfg(feature = "watch")] +use std::sync::Arc; +use std::time::Duration; + use bytes::Bytes; use d_engine_core::BatchOp; use d_engine_core::InboundEvent; @@ -38,11 +35,16 @@ use d_engine_core::config::ReadConsistencyPolicy; use d_engine_core::types::ScanResults; #[cfg(feature = "watch")] use d_engine_core::watch::WatchRegistry; -#[cfg(feature = "watch")] -use std::sync::Arc; -use std::time::Duration; use tokio::sync::mpsc; +use super::embedded_read_handle::EmbeddedReadHandle; +pub(crate) use super::standalone_read_handle::channel_closed_error; +pub(crate) use super::standalone_read_handle::map_error_response; +pub(crate) use super::standalone_read_handle::server_error; +pub(crate) use super::standalone_read_handle::timeout_error; +use crate::api::types::WriteOperation; +use crate::proto_convert; + /// Zero-overhead KV client for embedded mode. Obtained via `EmbeddedEngine::client()`. /// /// `T` is the [`TypeConfig`] that carries the concrete `StateMachine` type, enabling @@ -122,7 +124,7 @@ impl EmbeddedClient { self.read_handle .cmd_tx - .send(d_engine_core::ClientCmd::Propose(request, resp_tx)) + .send(d_engine_core::ClientCmd::propose(request, resp_tx)) .await .map_err(|_| channel_closed_error())?; @@ -291,7 +293,7 @@ impl EmbeddedClient { self.read_handle .cmd_tx - .send(d_engine_core::ClientCmd::Propose(request, resp_tx)) + .send(d_engine_core::ClientCmd::propose(request, resp_tx)) .await .map_err(|_| channel_closed_error())?; @@ -367,10 +369,9 @@ impl EmbeddedClient { /// # Arguments /// /// * `key` - The exact key to watch. - /// * `prev_kv` - When `true`, every `Put` and `Delete` event carries the value the - /// key held **before** the mutation in `event.prev_value`. When - /// `false` (the default via [`watch`](Self::watch)), `prev_value` is - /// always empty. + /// * `prev_kv` - When `true`, every `Put` and `Delete` event carries the value the key held + /// **before** the mutation in `event.prev_value`. When `false` (the default via + /// [`watch`](Self::watch)), `prev_value` is always empty. /// /// # Performance note /// @@ -603,7 +604,7 @@ impl ClientApi for EmbeddedClient { self.read_handle .cmd_tx - .send(d_engine_core::ClientCmd::Propose(request, resp_tx)) + .send(d_engine_core::ClientCmd::propose(request, resp_tx)) .await .map_err(|_| channel_closed_error())?; @@ -658,7 +659,7 @@ impl ClientApi for EmbeddedClient { self.read_handle .cmd_tx - .send(d_engine_core::ClientCmd::Propose(request, resp_tx)) + .send(d_engine_core::ClientCmd::propose(request, resp_tx)) .await .map_err(|_| channel_closed_error())?; @@ -723,7 +724,7 @@ impl ClientApi for EmbeddedClient { self.read_handle .cmd_tx - .send(d_engine_core::ClientCmd::Propose(request, resp_tx)) + .send(d_engine_core::ClientCmd::propose(request, resp_tx)) .await .map_err(|_| channel_closed_error())?; diff --git a/d-engine-server/src/api/embedded_read_handle.rs b/d-engine-server/src/api/embedded_read_handle.rs index b7c9cfe3..110fe190 100644 --- a/d-engine-server/src/api/embedded_read_handle.rs +++ b/d-engine-server/src/api/embedded_read_handle.rs @@ -21,6 +21,7 @@ //! as any other shutdown scenario. use std::marker::PhantomData; +use std::sync::Arc; use std::time::Duration; use bytes::Bytes; @@ -29,15 +30,18 @@ use d_engine_core::RaftOneshot; use d_engine_core::ReadLease; use d_engine_core::StateMachine; use d_engine_core::TypeConfig; -use d_engine_core::client::{ClientApiResult, ClientReadRequest, ErrorCode}; +use d_engine_core::client::ClientApiResult; +use d_engine_core::client::ClientReadRequest; +use d_engine_core::client::ErrorCode; use d_engine_core::config::ReadConsistencyPolicy; use d_engine_core::now_ms; -use std::sync::Arc; use tokio::sync::mpsc; -use super::standalone_read_handle::{ - channel_closed_error, extract_read_payload, map_error_response, server_error, timeout_error, -}; +use super::standalone_read_handle::channel_closed_error; +use super::standalone_read_handle::extract_read_payload; +use super::standalone_read_handle::map_error_response; +use super::standalone_read_handle::server_error; +use super::standalone_read_handle::timeout_error; // ── EmbeddedReadHandle ──────────────────────────────────────────────────────── diff --git a/d-engine-server/src/api/embedded_read_handle_test.rs b/d-engine-server/src/api/embedded_read_handle_test.rs index c8ec8461..44cdf3cb 100644 --- a/d-engine-server/src/api/embedded_read_handle_test.rs +++ b/d-engine-server/src/api/embedded_read_handle_test.rs @@ -9,8 +9,11 @@ mod embedded_read_handle_tests { use std::time::Duration; use bytes::Bytes; + use d_engine_core::MockStateMachine; + use d_engine_core::MockTypeConfig; + use d_engine_core::ReadLease; use d_engine_core::config::ReadConsistencyPolicy; - use d_engine_core::{MockStateMachine, MockTypeConfig, ReadLease, now_ms}; + use d_engine_core::now_ms; use tokio::sync::mpsc; use super::super::EmbeddedReadHandle; diff --git a/d-engine-server/src/api/embedded_test/embedded_test.rs b/d-engine-server/src/api/embedded_test/embedded_test.rs index dccf5af7..8874b62a 100644 --- a/d-engine-server/src/api/embedded_test/embedded_test.rs +++ b/d-engine-server/src/api/embedded_test/embedded_test.rs @@ -519,9 +519,6 @@ listen_addr = "127.0.0.1:0" mod start_node_tests { use std::net::SocketAddr; - use super::*; - use serial_test::serial; - use bytes::Bytes; use d_engine_core::ClientApi; use d_engine_core::StateMachine; @@ -529,6 +526,9 @@ listen_addr = "127.0.0.1:0" use d_engine_proto::common::NodeRole; use d_engine_proto::common::NodeStatus; use d_engine_proto::server::cluster::NodeMeta; + use serial_test::serial; + + use super::*; /// Build a minimal valid `RaftNodeConfig` backed by a temp directory. /// Uses a fixed high port; tests are serialized to avoid conflicts. diff --git a/d-engine-server/src/api/embedded_test/mod.rs b/d-engine-server/src/api/embedded_test/mod.rs index c9c4cfc0..ea70aecc 100644 --- a/d-engine-server/src/api/embedded_test/mod.rs +++ b/d-engine-server/src/api/embedded_test/mod.rs @@ -3,3 +3,6 @@ mod embedded_test; #[cfg(test)] mod embedded_env_test; + +#[cfg(test)] +mod shutdown_order_test; diff --git a/d-engine-server/src/api/embedded_test/shutdown_order_test.rs b/d-engine-server/src/api/embedded_test/shutdown_order_test.rs new file mode 100644 index 00000000..154d981d --- /dev/null +++ b/d-engine-server/src/api/embedded_test/shutdown_order_test.rs @@ -0,0 +1,189 @@ +//! Shutdown ordering of `EmbeddedEngine::stop()`. +//! +//! `close_storage()` flushes the state machine and releases its storage (for RocksDB: the DB +//! handle and the LOCK file). It may therefore only run once the state machine worker has applied +//! everything the node had already committed; otherwise those commits are applied onto closed +//! storage, or never applied. + +use std::sync::Arc; +use std::sync::atomic::AtomicBool; +use std::sync::atomic::AtomicUsize; +use std::sync::atomic::Ordering; +use std::time::Duration; + +use async_trait::async_trait; +use bytes::Bytes; +use d_engine_core::ApplyEntry; +use d_engine_core::ApplyResult; +use d_engine_core::Error; +use d_engine_core::ScanResult; +use d_engine_core::SnapshotApplyResult; +use d_engine_core::StateMachine; +use d_engine_proto::common::LogId; +use d_engine_proto::server::storage::SnapshotMetadata; + +use crate::api::EmbeddedEngine; +use crate::storage::FileStateMachine; +use crate::storage::FileStorageEngine; + +/// A file state machine whose `apply_chunk` is slow (so commits are still being applied when the +/// engine is stopped) and which records whether storage was closed while applying. +#[derive(Debug)] +struct SlowApplyProbe { + inner: FileStateMachine, + apply_delay: Duration, + apply_started: AtomicBool, + storage_closed: AtomicBool, + /// Entries whose apply started or finished after `close_storage()` had been called. + applied_after_close: AtomicUsize, +} + +#[async_trait] +impl StateMachine for SlowApplyProbe { + async fn apply_chunk( + &self, + chunk: &[ApplyEntry], + ) -> Result, Error> { + self.apply_started.store(true, Ordering::SeqCst); + let closed_at_start = self.storage_closed.load(Ordering::SeqCst); + tokio::time::sleep(self.apply_delay).await; + let result = self.inner.apply_chunk(chunk).await; + if closed_at_start || self.storage_closed.load(Ordering::SeqCst) { + self.applied_after_close.fetch_add(chunk.len(), Ordering::SeqCst); + } + result + } + + fn close(&self) -> Result<(), Error> { + self.storage_closed.store(true, Ordering::SeqCst); + self.inner.close() + } + + // Everything below delegates unchanged. + async fn start(&self) -> Result<(), Error> { + self.inner.start().await + } + fn is_running(&self) -> bool { + self.inner.is_running() + } + fn get( + &self, + key_buffer: &[u8], + ) -> Result, Error> { + self.inner.get(key_buffer) + } + fn len(&self) -> usize { + self.inner.len() + } + fn update_last_applied( + &self, + last_applied: LogId, + ) { + self.inner.update_last_applied(last_applied) + } + fn last_applied(&self) -> LogId { + self.inner.last_applied() + } + fn persist_last_applied( + &self, + last_applied: LogId, + ) -> Result<(), Error> { + self.inner.persist_last_applied(last_applied) + } + fn update_last_snapshot_metadata( + &self, + snapshot_metadata: &SnapshotMetadata, + ) -> Result<(), Error> { + self.inner.update_last_snapshot_metadata(snapshot_metadata) + } + fn snapshot_metadata(&self) -> Option { + self.inner.snapshot_metadata() + } + fn persist_last_snapshot_metadata( + &self, + snapshot_metadata: &SnapshotMetadata, + ) -> Result<(), Error> { + self.inner.persist_last_snapshot_metadata(snapshot_metadata) + } + async fn apply_snapshot_from_file( + &self, + metadata: &SnapshotMetadata, + snapshot_path: std::path::PathBuf, + ) -> Result { + self.inner.apply_snapshot_from_file(metadata, snapshot_path).await + } + async fn generate_snapshot_data( + &self, + new_snapshot_dir: std::path::PathBuf, + last_included: LogId, + ) -> Result { + self.inner.generate_snapshot_data(new_snapshot_dir, last_included).await + } + fn save_hard_state(&self) -> Result<(), Error> { + self.inner.save_hard_state() + } + fn flush(&self) -> Result<(), Error> { + self.inner.flush() + } + async fn flush_async(&self) -> Result<(), Error> { + self.inner.flush_async().await + } + async fn reset(&self) -> Result<(), Error> { + self.inner.reset().await + } + fn scan_prefix( + &self, + prefix: &[u8], + ) -> Result { + self.inner.scan_prefix(prefix) + } +} + +/// Business scenario: a single-node embedded service is busy writing when its operator stops it +/// (rolling restart). Commits the node already made are still being applied by the slow state +/// machine at that moment. +/// +/// Purpose: `stop()` must let the worker finish applying those commits BEFORE it closes the +/// state machine's storage. Closing first means applies run on (or are cut off by) closed +/// storage, so committed writes do not reliably reach the state machine's disk. +#[tokio::test] +async fn test_stop_closes_state_machine_storage_only_after_in_flight_applies_finish() { + unsafe { + std::env::remove_var("CONFIG_PATH"); + } + let temp_dir = tempfile::tempdir().expect("temp dir"); + let storage = + Arc::new(FileStorageEngine::new(temp_dir.path().join("storage")).expect("storage")); + let probe = Arc::new(SlowApplyProbe { + inner: FileStateMachine::new(temp_dir.path().join("sm")).await.expect("sm"), + apply_delay: Duration::from_millis(300), + apply_started: AtomicBool::new(false), + storage_closed: AtomicBool::new(false), + applied_after_close: AtomicUsize::new(0), + }); + + let engine = EmbeddedEngine::start_custom(temp_dir.path(), storage, probe.clone(), None) + .await + .expect("start engine"); + engine.wait_ready(Duration::from_secs(5)).await.expect("leader"); + + // Writes in flight: not awaited, the operator does not wait for them. + let client = engine.client(); + for i in 0..20u32 { + let client = client.clone(); + tokio::spawn(async move { + let _ = client.put(format!("key-{i}"), "value").await; + }); + } + // Wait until the state machine is busy applying, then stop the engine. + while !probe.apply_started.load(Ordering::SeqCst) { + tokio::time::sleep(Duration::from_millis(5)).await; + } + engine.stop().await.expect("stop"); + + assert_eq!( + probe.applied_after_close.load(Ordering::SeqCst), + 0, + "stop() closed the state machine's storage while committed entries were still being applied" + ); +} diff --git a/d-engine-server/src/api/standalone_read_handle.rs b/d-engine-server/src/api/standalone_read_handle.rs index 0591e34f..80b66ec5 100644 --- a/d-engine-server/src/api/standalone_read_handle.rs +++ b/d-engine-server/src/api/standalone_read_handle.rs @@ -11,14 +11,19 @@ use std::time::Duration; use bytes::Bytes; use d_engine_core::MaybeCloneOneshot; use d_engine_core::RaftOneshot; -use d_engine_core::client::{ - ClientApiError, ClientApiResult, ClientReadRequest, ClientResponsePayload, ErrorCode, - LeaderHint, ReadResults, -}; +use d_engine_core::client::ClientApiError; +use d_engine_core::client::ClientApiResult; +use d_engine_core::client::ClientReadRequest; +use d_engine_core::client::ClientResponsePayload; +use d_engine_core::client::ErrorCode; +use d_engine_core::client::LeaderHint; +use d_engine_core::client::ReadResults; use d_engine_core::config::ReadConsistencyPolicy; -use tokio::sync::{mpsc, oneshot}; +use tokio::sync::mpsc; +use tokio::sync::oneshot; -use crate::read_actor::{ReadActorError, ReadCmd}; +use crate::read_actor::ReadActorError; +use crate::read_actor::ReadCmd; // ── Error helpers ───────────────────────────────────────────────────────────── diff --git a/d-engine-server/src/api/standalone_read_handle_test.rs b/d-engine-server/src/api/standalone_read_handle_test.rs index 0dc8d2b6..a0335e2b 100644 --- a/d-engine-server/src/api/standalone_read_handle_test.rs +++ b/d-engine-server/src/api/standalone_read_handle_test.rs @@ -14,15 +14,21 @@ mod read_handle_tests { use std::sync::Arc; use std::time::Duration; - use crate::read_actor::run_read_actor; use bytes::Bytes; - use d_engine_core::client::{ClientResponse, KvEntry}; + use d_engine_core::ClientApiError; + use d_engine_core::ClientCmd; + use d_engine_core::Error; + use d_engine_core::MockStateMachine; + use d_engine_core::ReadLease; + use d_engine_core::client::ClientResponse; + use d_engine_core::client::KvEntry; use d_engine_core::config::ReadConsistencyPolicy; - use d_engine_core::{ClientApiError, ClientCmd, Error, MockStateMachine, ReadLease, now_ms}; + use d_engine_core::now_ms; use tokio::sync::mpsc; use tokio::task::JoinHandle; use super::super::StandaloneReadHandle; + use crate::read_actor::run_read_actor; // ── Helpers ─────────────────────────────────────────────────────────────── @@ -612,10 +618,17 @@ mod read_handle_tests { #[cfg(test)] mod error_helper_tests { use bytes::Bytes; - use d_engine_core::client::{ClientApiError, ClientResponsePayload, ErrorCode}; - use d_engine_core::client::{KvEntry, LeaderHint, ReadResults, ScanResults}; - - use super::super::{extract_read_payload, map_error_response, not_leader_error}; + use d_engine_core::client::ClientApiError; + use d_engine_core::client::ClientResponsePayload; + use d_engine_core::client::ErrorCode; + use d_engine_core::client::KvEntry; + use d_engine_core::client::LeaderHint; + use d_engine_core::client::ReadResults; + use d_engine_core::client::ScanResults; + + use super::super::extract_read_payload; + use super::super::map_error_response; + use super::super::not_leader_error; // ─── not_leader_error ──────────────────────────────────────────────────── diff --git a/d-engine-server/src/api/standalone_test.rs b/d-engine-server/src/api/standalone_test.rs index 2cfdff60..d7517954 100644 --- a/d-engine-server/src/api/standalone_test.rs +++ b/d-engine-server/src/api/standalone_test.rs @@ -247,13 +247,12 @@ mod start_node_tests { use std::sync::Arc; use std::time::Duration; - use serial_test::serial; - use tokio::sync::watch; - use d_engine_core::config::ClusterConfig; use d_engine_proto::common::NodeRole; use d_engine_proto::common::NodeStatus; use d_engine_proto::server::cluster::NodeMeta; + use serial_test::serial; + use tokio::sync::watch; use crate::RocksDBStateMachine; use crate::RocksDBStorageEngine; diff --git a/d-engine-server/src/api/types.rs b/d-engine-server/src/api/types.rs index abcf20a7..5f867624 100644 --- a/d-engine-server/src/api/types.rs +++ b/d-engine-server/src/api/types.rs @@ -16,12 +16,14 @@ use bytes::Bytes; use d_engine_core::BatchOp; -use d_engine_proto::client::{ - WriteCommand, - write_command::{ - Batch, BatchOp as ProtoBatchOp, CompareAndSwap, Delete, Insert, Operation, batch_op, - }, -}; +use d_engine_proto::client::WriteCommand; +use d_engine_proto::client::write_command::Batch; +use d_engine_proto::client::write_command::BatchOp as ProtoBatchOp; +use d_engine_proto::client::write_command::CompareAndSwap; +use d_engine_proto::client::write_command::Delete; +use d_engine_proto::client::write_command::Insert; +use d_engine_proto::client::write_command::Operation; +use d_engine_proto::client::write_command::batch_op; /// Decoded write operation — the unit submitted by a client. /// diff --git a/d-engine-server/src/lib.rs b/d-engine-server/src/lib.rs index f5322f6e..4ade9797 100644 --- a/d-engine-server/src/lib.rs +++ b/d-engine-server/src/lib.rs @@ -99,7 +99,8 @@ mod proto_convert; mod proto_convert_test; /// ReadActor — fast-path read task for Eventual and LeaseRead. -/// Kept in server (not core) because it is a performance optimization, not a Raft protocol component. +/// Kept in server (not core) because it is a performance optimization, not a Raft protocol +/// component. pub(crate) mod read_actor; #[cfg(test)] mod read_actor_test; @@ -128,16 +129,34 @@ pub mod storage; pub use api::DefaultEmbeddedClient; #[cfg(feature = "rocksdb")] pub use api::DefaultEmbeddedEngine; +// -------------------- Client API -------------------- +pub use api::EmbeddedClient; pub use api::EmbeddedEngine; pub use api::StandaloneEngine; -pub use membership::MembershipSnapshot; +pub use d_engine_core::ApplyEntry; +// Re-export core types needed by applications +pub use d_engine_core::ApplyResult; +// Batch operation variants (Insert, Delete) used in [`Command::Batch`]. +pub use d_engine_core::BatchOp; +pub use d_engine_core::Command; // -------------------- Error Types -------------------- /// Unified error type for all d-engine operations pub use d_engine_core::Error; +// Internal types required by storage implementations — not part of user API +/// Persisted Raft hard state (current term + voted-for). Required by +/// [`MetaStore::save_hard_state`]/[`MetaStore::load_hard_state`] — implement these if +/// you're building a custom storage engine. +pub use d_engine_core::HardState; /// Log storage trait pub use d_engine_core::LogStore; /// Metadata storage trait pub use d_engine_core::MetaStore; +// Purely internal — only reachable via `?`/`From` conversions inside `Error`. +// No public trait method requires implementors to construct this directly. +#[doc(hidden)] +pub use d_engine_core::ProstError; +// Raft Node Config +pub use d_engine_core::RaftNodeConfig; /// Unified result type (equivalent to Result) pub use d_engine_core::Result; /// Prefix scan result: matching entries + revision anchor for watch deduplication @@ -147,25 +166,6 @@ pub use d_engine_core::Result; pub use d_engine_core::ScanResult; /// Client-facing prefix scan result, returned by `EmbeddedClient::scan_prefix()`. pub use d_engine_core::ScanResults; -/// Storage-specific error type -pub use d_engine_core::StorageError; -// Raft Node Config -pub use d_engine_core::RaftNodeConfig; -// Batch operation variants (Insert, Delete) used in [`Command::Batch`]. -pub use d_engine_core::BatchOp; - -// Internal types required by storage implementations — not part of user API - -/// Persisted Raft hard state (current term + voted-for). Required by -/// [`MetaStore::save_hard_state`]/[`MetaStore::load_hard_state`] — implement these if -/// you're building a custom storage engine. -pub use d_engine_core::HardState; - -// Purely internal — only reachable via `?`/`From` conversions inside `Error`. -// No public trait method requires implementors to construct this directly. -#[doc(hidden)] -pub use d_engine_core::ProstError; - /// Outcome of installing a snapshot via [`StateMachine::apply_snapshot_from_file`]. /// /// Distinguishes "the install actually ran" (`Applied`) from two no-op cases that are not @@ -174,14 +174,30 @@ pub use d_engine_core::ProstError; /// (checksum mismatch, corrupted data, conflicting boundary) should be returned as `Err`, not /// encoded as a variant here. pub use d_engine_core::SnapshotApplyResult; - /// Snapshot-specific error type. Construct these in /// [`StateMachine::apply_snapshot_from_file`] (e.g. `SnapshotError::ChecksumMismatch`) to /// report a genuine install failure — the no-op cases are already covered by /// [`SnapshotApplyResult`], not this type. pub use d_engine_core::SnapshotError; -// -------------------- Client API -------------------- -pub use api::EmbeddedClient; +/// Storage trait for implementing custom storage backends +/// +/// Implement this trait to create your own storage engine. +pub use d_engine_core::StateMachine; +/// Storage trait for implementing custom storage backends +/// +/// Implement this trait to create your own storage engine. +pub use d_engine_core::StorageEngine; +/// Storage-specific error type +pub use d_engine_core::StorageError; +/// Unified client operations trait (put/get/delete/CAS/watch). +/// Re-exported here so embedded-mode users don't need a separate d-engine-client dependency. +pub use d_engine_core::client::ClientApi; +/// Unified client operations trait (put/get/delete/CAS/watch). +/// Re-exported here so embedded-mode users don't need a separate d-engine-client dependency. +pub use d_engine_core::client::ClientApiError; +/// Unified client operations trait (put/get/delete/CAS/watch). +/// Re-exported here so embedded-mode users don't need a separate d-engine-client dependency. +pub use d_engine_core::client::ClientApiResult; /// Error code discriminator returned by [`ClientApiError::code()`]. /// Use this to pattern-match error categories without inspecting message strings: /// `e.code() == ErrorCode::NotLeader` @@ -189,23 +205,6 @@ pub use d_engine_core::client::ErrorCode; /// Optional leader location hint on `ErrorCode::NotLeader`, so a client can /// retry against the right node without a blind broadcast. pub use d_engine_core::client::LeaderHint; -/// Unified client operations trait (put/get/delete/CAS/watch). -/// Re-exported here so embedded-mode users don't need a separate d-engine-client dependency. -pub use d_engine_core::client::{ClientApi, ClientApiError, ClientApiResult}; -/// Storage trait for implementing custom storage backends -/// -/// Implement this trait to create your own storage engine. -pub use d_engine_core::{StateMachine, StorageEngine}; -pub use node::Node; -// Re-export storage implementations -pub use storage::{FileStateMachine, FileStorageEngine}; -// Re-export core types needed by applications -pub use d_engine_core::ApplyResult; -pub use d_engine_core::{ApplyEntry, Command}; -// Conditional RocksDB exports -#[cfg(feature = "rocksdb")] -pub use storage::{RocksDBStateMachine, RocksDBStorageEngine, RocksDBUnifiedEngine}; - // -------------------- Watch API Types -------------------- /// Watch event types and handles — available when the `watch` feature is enabled. /// @@ -215,6 +214,13 @@ pub use storage::{RocksDBStateMachine, RocksDBStorageEngine, RocksDBUnifiedEngin /// ``` #[cfg(feature = "watch")] pub use d_engine_core::{WatchError, WatchEvent, WatchEventType, WatcherHandle}; +pub use membership::MembershipSnapshot; +pub use node::Node; +// Re-export storage implementations +pub use storage::{FileStateMachine, FileStorageEngine}; +// Conditional RocksDB exports +#[cfg(feature = "rocksdb")] +pub use storage::{RocksDBStateMachine, RocksDBStorageEngine, RocksDBUnifiedEngine}; // -------------------- Data Types -------------------- @@ -247,17 +253,23 @@ mod utils; // ==================== Test Utilities ==================== -/// Standardized test suite for custom [`StateMachine`] implementations. +/// Deterministically freezes an outbound snapshot push transfer to a given +/// peer right before its first chunk is sent, for tests that need to observe +/// a "transfer started but not completed" window without racing wall-clock +/// timing. /// -/// Enable the `__test_support` feature in your `[dev-dependencies]` to access this module: +/// Enable the `__test_support` feature in your `[dev-dependencies]` to access this: /// ```toml /// [dev-dependencies] /// d-engine = { version = "...", features = ["server", "__test_support"] } /// ``` #[cfg(feature = "__test_support")] -pub use d_engine_core::state_machine_test; - -/// Standardized test suite for custom [`StorageEngine`] implementations. +pub use d_engine_core::SnapshotTransferGate; +#[cfg(feature = "__test_support")] +pub use d_engine_core::SnapshotTransferGateGuard; +#[cfg(feature = "__test_support")] +pub use d_engine_core::install_snapshot_transfer_gate; +/// Standardized test suite for custom [`StateMachine`] implementations. /// /// Enable the `__test_support` feature in your `[dev-dependencies]` to access this module: /// ```toml @@ -265,24 +277,16 @@ pub use d_engine_core::state_machine_test; /// d-engine = { version = "...", features = ["server", "__test_support"] } /// ``` #[cfg(feature = "__test_support")] -pub use d_engine_core::storage_engine_test; - -/// Deterministically freezes an outbound snapshot push transfer to a given -/// peer right before its first chunk is sent, for tests that need to observe -/// a "transfer started but not completed" window without racing wall-clock -/// timing. +pub use d_engine_core::state_machine_test; +/// Standardized test suite for custom [`StorageEngine`] implementations. /// -/// Enable the `__test_support` feature in your `[dev-dependencies]` to access this: +/// Enable the `__test_support` feature in your `[dev-dependencies]` to access this module: /// ```toml /// [dev-dependencies] /// d-engine = { version = "...", features = ["server", "__test_support"] } /// ``` #[cfg(feature = "__test_support")] -pub use d_engine_core::SnapshotTransferGate; -#[cfg(feature = "__test_support")] -pub use d_engine_core::SnapshotTransferGateGuard; -#[cfg(feature = "__test_support")] -pub use d_engine_core::install_snapshot_transfer_gate; +pub use d_engine_core::storage_engine_test; /// Test utilities for d-engine-server /// diff --git a/d-engine-server/src/membership/raft_membership.rs b/d-engine-server/src/membership/raft_membership.rs index c38c7aca..b9e392a7 100644 --- a/d-engine-server/src/membership/raft_membership.rs +++ b/d-engine-server/src/membership/raft_membership.rs @@ -638,6 +638,15 @@ where info!("Applying membership change: {:?}", membership_change); match membership_change.change { Some(Change::AddNode(add)) => { + // A committed AddNode for a node the local view already has is already reflected: + // it happens when this node's `initial_cluster` was seeded after the node joined, + // and it then replays the log entry that added it. Applying it again would fail, + // and a failed committed membership change is fatal for the node. The existing + // role is kept, so a voter is never demoted back to a learner. + if self.retrieve_node_meta(add.node_id).is_some() { + return Ok(()); + } + let status = NodeStatus::try_from(add.status).unwrap_or(NodeStatus::Promotable); self.add_learner(add.node_id, add.address, status).await } diff --git a/d-engine-server/src/membership/raft_membership_test.rs b/d-engine-server/src/membership/raft_membership_test.rs index addc1ea9..ddd78aa6 100644 --- a/d-engine-server/src/membership/raft_membership_test.rs +++ b/d-engine-server/src/membership/raft_membership_test.rs @@ -1172,6 +1172,57 @@ mod add_learner_test { } } +/// Business scenario: a new node (3) joins a running cluster. Its `initial_cluster` already lists +/// node 2 as a voter, because node 2 was promoted before node 3 started. Node 3 then catches up by +/// replaying the leader's committed log, which still contains the `AddNode(2, learner)` entry from +/// when node 2 first joined. +/// +/// Purpose: applying a committed `AddNode` for a node the local view already has is a no-op. It +/// must not fail (a failed committed membership change is fatal and kills the node's Raft loop) +/// and must not demote the voter back to a learner. +#[tokio::test] +async fn test_apply_committed_add_node_for_existing_voter_is_idempotent() { + let (membership, _) = + RaftMembership::>::new( + 3, + vec![ + NodeMeta { + id: 2, + address: "127.0.0.1:1002".to_string(), + role: Follower as i32, + status: NodeStatus::Active.into(), + }, + NodeMeta { + id: 3, + address: "127.0.0.1:1003".to_string(), + role: Learner as i32, + status: NodeStatus::Promotable.into(), + }, + ], + RaftNodeConfig::default(), + ); + + let replayed = MembershipChange { + change: Some(Change::AddNode(AddNode { + node_id: 2, + address: "127.0.0.1:1002".to_string(), + status: NodeStatus::Promotable as i32, + })), + }; + let result = membership.apply_config_change(replayed).await; + + assert!( + result.is_ok(), + "replaying AddNode for a node already in the local view must not fail: {result:?}" + ); + let node2 = membership.retrieve_node_meta(2).expect("node 2 stays in the membership"); + assert_eq!( + node2.role, Follower as i32, + "the voter must not be demoted to a learner" + ); + assert_eq!(node2.status, NodeStatus::Active as i32); +} + #[tokio::test] #[traced_test] async fn test_ensure_safe_join() { @@ -1232,7 +1283,8 @@ async fn test_health_monitoring_integration() { // Test 3: get_peer_channel success does NOT reset failure counts. // tonic Endpoint::connect() is lazy — channel creation succeeds even for a stopped peer. - // Health state must be driven by RPC results (on_peer_stream_failed), not by channel acquisition. + // Health state must be driven by RPC results (on_peer_stream_failed), not by channel + // acquisition. let (_tx, rx) = oneshot::channel::<()>(); let service = MockRpcService::default(); let (port, _addr) = MockNode::mock_listener(service, rx, true).await.unwrap(); diff --git a/d-engine-server/src/network/connection_cache.rs b/d-engine-server/src/network/connection_cache.rs index f5ed40d0..272e6c86 100644 --- a/d-engine-server/src/network/connection_cache.rs +++ b/d-engine-server/src/network/connection_cache.rs @@ -62,14 +62,11 @@ impl ConnectionCache { let channel = self.create_channel(current_address.clone(), conn_type).await?; trace!(?key, "Cache updated: address: {}", current_address.clone()); - self.cache.insert( - key, - CachedChannel { - channel: channel.clone(), - address: current_address, - last_used: Instant::now(), - }, - ); + self.cache.insert(key, CachedChannel { + channel: channel.clone(), + address: current_address, + last_used: Instant::now(), + }); Ok(channel) } diff --git a/d-engine-server/src/network/grpc/grpc_raft_service.rs b/d-engine-server/src/network/grpc/grpc_raft_service.rs index 57b3cc31..1100bb8b 100644 --- a/d-engine-server/src/network/grpc/grpc_raft_service.rs +++ b/d-engine-server/src/network/grpc/grpc_raft_service.rs @@ -2,8 +2,10 @@ //! and client requests. Implements core Raft protocol logic for leader election, //! log replication, and cluster configuration management. -use crate::Node; -use crate::proto_convert; +use std::future::Future; +use std::pin::Pin; +use std::time::Duration; + use d_engine_core::InboundEvent; use d_engine_core::MaybeCloneOneshot; use d_engine_core::RaftOneshot; @@ -40,9 +42,6 @@ use d_engine_proto::server::storage::SnapshotResponse; use d_engine_proto::server::storage::snapshot_service_server::SnapshotService; use futures::Stream; use futures::StreamExt; -use std::future::Future; -use std::pin::Pin; -use std::time::Duration; use tokio::select; use tokio::sync::mpsc; use tokio::time::timeout; @@ -56,6 +55,9 @@ use tracing::error; use tracing::info; use tracing::warn; +use crate::Node; +use crate::proto_convert; + #[tonic::async_trait] impl RaftElectionService for Node where @@ -144,10 +146,9 @@ where let (resp_tx, resp_rx) = MaybeCloneOneshot::new(); self.event_tx - .send(InboundEvent::AppendEntries( - request.into_inner(), - vec![resp_tx], - )) + .send(InboundEvent::AppendEntries(request.into_inner(), vec![ + resp_tx, + ])) .await .map_err(|_| Status::internal("Event channel closed"))?; @@ -482,7 +483,7 @@ where let (resp_tx, resp_rx) = MaybeCloneOneshot::new(); let t0 = std::time::Instant::now(); cmd_tx - .send(d_engine_core::ClientCmd::Propose(core_req, resp_tx)) + .send(d_engine_core::ClientCmd::propose(core_req, resp_tx)) .await .map_err(|_| Status::internal("Command channel closed"))?; diff --git a/d-engine-server/src/network/grpc/grpc_raft_service_fast_path_test.rs b/d-engine-server/src/network/grpc/grpc_raft_service_fast_path_test.rs index ed7d50c0..73b5581f 100644 --- a/d-engine-server/src/network/grpc/grpc_raft_service_fast_path_test.rs +++ b/d-engine-server/src/network/grpc/grpc_raft_service_fast_path_test.rs @@ -14,9 +14,11 @@ mod grpc_fast_path_tests { use std::sync::Arc; use std::sync::atomic::Ordering; - use crate::read_actor::run_read_actor; use bytes::Bytes; - use d_engine_core::{MockStateMachine, ReadLease, now_ms}; + use d_engine_core::MockStateMachine; + use d_engine_core::MockTypeConfig; + use d_engine_core::ReadLease; + use d_engine_core::now_ms; use d_engine_proto::client::ClientReadRequest; use d_engine_proto::client::ReadConsistencyPolicy as ProtoPolicy; use d_engine_proto::client::client_response::SuccessResult; @@ -27,8 +29,8 @@ mod grpc_fast_path_tests { use crate::Node; use crate::api::StandaloneReadHandle; + use crate::read_actor::run_read_actor; use crate::test_utils::mock_node; - use d_engine_core::MockTypeConfig; // ── Helpers ─────────────────────────────────────────────────────────────── diff --git a/d-engine-server/src/network/grpc/grpc_raft_service_test.rs b/d-engine-server/src/network/grpc/grpc_raft_service_test.rs index e79511aa..ca1d4f1c 100644 --- a/d-engine-server/src/network/grpc/grpc_raft_service_test.rs +++ b/d-engine-server/src/network/grpc/grpc_raft_service_test.rs @@ -1,6 +1,5 @@ use std::time::Duration; -use crate::ApplyResult; use d_engine_core::AppendResponseWithUpdates; use d_engine_core::InternalEvent; use d_engine_core::MaybeCloneOneshot; @@ -19,6 +18,8 @@ use d_engine_proto::client::ReadConsistencyPolicy; use d_engine_proto::client::WatchMembershipRequest; use d_engine_proto::client::WriteCommand; use d_engine_proto::client::raft_client_service_server::RaftClientService; +use d_engine_proto::common::Entry; +use d_engine_proto::common::EntryPayload; use d_engine_proto::common::LogId; use d_engine_proto::server::cluster::ClusterConfChangeRequest; use d_engine_proto::server::cluster::ClusterConfUpdateResponse; @@ -37,6 +38,7 @@ use tonic::Code; use tonic::Request; use tracing_test::traced_test; +use crate::ApplyResult; use crate::test_utils::MockBuilder; use crate::test_utils::mock_node; @@ -223,7 +225,8 @@ async fn test_handle_rpc_services_successfully() { // Dynamic log index: starts at 0, incremented by prepare_batch_requests per payload written. // Used so the test can read the correct log index when sending LogFlushed to trigger commit. use std::sync::Arc; - use std::sync::atomic::{AtomicU64, Ordering}; + use std::sync::atomic::AtomicU64; + use std::sync::atomic::Ordering; let log_index = Arc::new(AtomicU64::new(0)); let li_last = log_index.clone(); let li_flush = log_index.clone(); @@ -238,6 +241,17 @@ async fn test_handle_rpc_services_successfully() { raft_log.expect_load_hard_state().returning(|| Ok(None)); raft_log.expect_save_hard_state().returning(|_| Ok(())); raft_log.expect_last_log_id().returning(|| None); + // The raft loop reads committed entries back from the log to hand them to the state machine + // worker, so the mock log must serve them (any payload will do for this test). + raft_log.expect_get_entries_range().returning(|range| { + Ok(range + .map(|index| Entry { + index, + term: 1, + payload: Some(EntryPayload::noop()), + }) + .collect()) + }); let mut replication_handler = MockReplicationCore::::new(); replication_handler @@ -311,7 +325,8 @@ async fn test_handle_rpc_services_successfully() { .is_ok() ); - // Wait for raft loop to complete BecomeLeader (tick → broadcast_vote_requests → verify noop). + // Wait for raft loop to complete BecomeLeader (tick → broadcast_vote_requests → verify + // noop). tokio::time::sleep(Duration::from_millis(10)).await; // handle_client_write and handle_client_read must be sent while node is leader. @@ -340,6 +355,7 @@ async fn test_handle_rpc_services_successfully() { } // ApplyCompleted is now sent via internal_event_tx (P2) to avoid priority inversion. let _ = test_internal_event_tx.send(InternalEvent::ApplyCompleted { + sent_at: tokio::time::Instant::now(), last_index: 2, results: vec![ApplyResult { index: 2, @@ -531,12 +547,13 @@ async fn test_handle_client_scan_timeout() { #[tokio::test] #[traced_test] async fn test_handle_client_scan_success_returns_entries_and_revision() { + use std::sync::atomic::Ordering; + use bytes::Bytes; use d_engine_core::ClientCmd; use d_engine_core::client::ClientResponse; use d_engine_proto::client::ScanRequest; use d_engine_proto::client::client_response::SuccessResult; - use std::sync::atomic::Ordering; use tokio::sync::mpsc; let (_graceful_tx, graceful_rx) = watch::channel(()); @@ -581,12 +598,13 @@ async fn test_handle_client_scan_success_returns_entries_and_revision() { #[tokio::test] #[traced_test] async fn test_handle_client_scan_not_leader_carries_leader_hint_in_metadata() { + use std::sync::atomic::Ordering; + use d_engine_core::ClientCmd; use d_engine_core::client::ClientResponse; use d_engine_core::client::LeaderHint; use d_engine_proto::client::ScanRequest; use d_engine_proto::error::ErrorCode as ProtoErrorCode; - use std::sync::atomic::Ordering; use tokio::sync::mpsc; let (_graceful_tx, graceful_rx) = watch::channel(()); @@ -847,7 +865,8 @@ async fn test_stream_append_entries_closes_response_stream_after_inbound_ends() raft_log.expect_durable_index().returning(|| 5); let mut replication_handler = MockReplicationCore::::new(); - // The first election now starts at once, so a single-node cluster becomes leader and proposes its noop. + // The first election now starts at once, so a single-node cluster becomes leader and proposes + // its noop. replication_handler .expect_prepare_batch_requests() .returning(|_, _, _, _, _, _| Ok(d_engine_core::PrepareResult::default())); @@ -996,7 +1015,8 @@ async fn test_pre_vote_unanswered_request_times_out() { let timeout_ms = node.node_config.raft.election.election_timeout_min; let call = tokio::spawn(async move { node.pre_vote(Request::new(pre_vote_request())).await }); - // Keep the event (and its reply sender) alive so the failure is a timeout, not a closed channel. + // Keep the event (and its reply sender) alive so the failure is a timeout, not a closed + // channel. let _unanswered = event_rx.recv().await.expect("event reaches the loop"); tokio::time::advance(Duration::from_millis(timeout_ms + 1)).await; diff --git a/d-engine-server/src/network/grpc/grpc_transport.rs b/d-engine-server/src/network/grpc/grpc_transport.rs index 2d7fa341..2f0cb4cc 100644 --- a/d-engine-server/src/network/grpc/grpc_transport.rs +++ b/d-engine-server/src/network/grpc/grpc_transport.rs @@ -1,6 +1,10 @@ //! Centerialized all RPC client operations will make unit test eaiser. //! We also want to refactor all the APIs based its similar parttern. +use std::collections::HashSet; +use std::marker::PhantomData; +use std::sync::Arc; + use async_trait::async_trait; use d_engine_core::BackgroundSnapshotTransfer; use d_engine_core::BackoffPolicy; @@ -32,9 +36,6 @@ use d_engine_proto::server::replication::raft_replication_service_client::RaftRe use futures::FutureExt; use futures::StreamExt; use futures::stream::FuturesUnordered; -use std::collections::HashSet; -use std::marker::PhantomData; -use std::sync::Arc; use tokio::sync::mpsc; use tokio::task; use tokio_stream::wrappers::ReceiverStream; diff --git a/d-engine-server/src/network/grpc/grpc_transport_test.rs b/d-engine-server/src/network/grpc/grpc_transport_test.rs index 6e177b38..e5ec579f 100644 --- a/d-engine-server/src/network/grpc/grpc_transport_test.rs +++ b/d-engine-server/src/network/grpc/grpc_transport_test.rs @@ -331,8 +331,8 @@ async fn test_send_vote_request_returns_granted_response() { // # Case: vote denied but RPC itself succeeds // // ## Criteria: -// 1. Must be Ok(VoteResponse), NOT Err — a denial is a valid protocol -// response, not a transport failure +// 1. Must be Ok(VoteResponse), NOT Err — a denial is a valid protocol response, not a transport +// failure #[tokio::test] #[traced_test] async fn test_send_vote_request_returns_denied_response_as_ok() { @@ -371,8 +371,8 @@ async fn test_send_vote_request_returns_denied_response_as_ok() { // # Case: peer response carries a higher term than the candidate's // // ## Criteria: -// 1. send_vote_request passes the response through unchanged — it must not -// interpret the higher term itself; that's broadcast_vote_requests' job +// 1. send_vote_request passes the response through unchanged — it must not interpret the higher +// term itself; that's broadcast_vote_requests' job #[tokio::test] #[traced_test] async fn test_send_vote_request_passes_through_higher_term_response() { @@ -413,8 +413,8 @@ async fn test_send_vote_request_passes_through_higher_term_response() { // # Case: peer response carries a more up-to-date log than the candidate's // // ## Criteria: -// 1. send_vote_request passes the response through unchanged — log-conflict -// detection happens in broadcast_vote_requests, not here +// 1. send_vote_request passes the response through unchanged — log-conflict detection happens in +// broadcast_vote_requests, not here #[tokio::test] #[traced_test] async fn test_send_vote_request_passes_through_more_recent_log_response() { @@ -455,8 +455,7 @@ async fn test_send_vote_request_passes_through_more_recent_log_response() { // # Case: peer channel not found in membership // // ## Criteria: -// 1. Err(NetworkError::PeerConnectionNotFound(peer_id)) without attempting -// any RPC +// 1. Err(NetworkError::PeerConnectionNotFound(peer_id)) without attempting any RPC #[tokio::test] #[traced_test] async fn test_send_vote_request_missing_channel_returns_peer_connection_not_found() { @@ -564,28 +563,22 @@ async fn send_pre_vote_to_mock_peer( #[tokio::test] #[traced_test] async fn test_send_pre_vote_request_returns_granted_and_denied_responses_as_ok() { - let granted = send_pre_vote_to_mock_peer( - "/tmp/test_send_pre_vote_granted", - VoteResponse { - term: 1, - vote_granted: true, - last_log_index: 1, - last_log_term: 1, - }, - ) + let granted = send_pre_vote_to_mock_peer("/tmp/test_send_pre_vote_granted", VoteResponse { + term: 1, + vote_granted: true, + last_log_index: 1, + last_log_term: 1, + }) .await .expect("granted must be Ok"); assert!(granted.vote_granted); - let denied = send_pre_vote_to_mock_peer( - "/tmp/test_send_pre_vote_denied", - VoteResponse { - term: 1, - vote_granted: false, - last_log_index: 1, - last_log_term: 1, - }, - ) + let denied = send_pre_vote_to_mock_peer("/tmp/test_send_pre_vote_denied", VoteResponse { + term: 1, + vote_granted: false, + last_log_index: 1, + last_log_term: 1, + }) .await .expect("a denial must be Ok, not Err"); assert!(!denied.vote_granted); @@ -596,15 +589,12 @@ async fn test_send_pre_vote_request_returns_granted_and_denied_responses_as_ok() #[tokio::test] #[traced_test] async fn test_send_pre_vote_request_passes_through_term_and_log_of_the_response() { - let res = send_pre_vote_to_mock_peer( - "/tmp/test_send_pre_vote_passes_through", - VoteResponse { - term: 99, - vote_granted: false, - last_log_index: 42, - last_log_term: 5, - }, - ) + let res = send_pre_vote_to_mock_peer("/tmp/test_send_pre_vote_passes_through", VoteResponse { + term: 99, + vote_granted: false, + last_log_index: 42, + last_log_term: 5, + }) .await .expect("should succeed"); @@ -615,8 +605,8 @@ async fn test_send_pre_vote_request_passes_through_term_and_log_of_the_response( // # Case: PreVote goes out through the PreVote RPC, never through RequestVote. // // ## Criteria: -// 1. The peer's RequestVote handler answers with an error; the PreVote handler answers -// "granted". Getting "granted" proves the PreVote RPC was used. +// 1. The peer's RequestVote handler answers with an error; the PreVote handler answers "granted". +// Getting "granted" proves the PreVote RPC was used. #[tokio::test] #[traced_test] async fn test_send_pre_vote_request_uses_the_pre_vote_rpc_not_request_vote() { diff --git a/d-engine-server/src/network/health_checker_test.rs b/d-engine-server/src/network/health_checker_test.rs index 56869631..4f029021 100644 --- a/d-engine-server/src/network/health_checker_test.rs +++ b/d-engine-server/src/network/health_checker_test.rs @@ -4,11 +4,10 @@ use tokio::net::TcpStream; use tokio::sync::oneshot; use tracing_test::traced_test; +use super::*; use crate::test_utils::MockNode; use crate::test_utils::MockRpcService; -use super::*; - /// Case 1: server is not ready #[tokio::test] #[traced_test] diff --git a/d-engine-server/src/node/builder.rs b/d-engine-server/src/node/builder.rs index 9edadd05..3308f6ff 100644 --- a/d-engine-server/src/node/builder.rs +++ b/d-engine-server/src/node/builder.rs @@ -28,15 +28,14 @@ //! - **Configuration Loading**: Supports loading cluster configuration from file or in-memory //! config. -use crate::read_actor::run_read_actor; -use d_engine_core::CommitHandler; -use d_engine_core::CommitHandlerDependencies; -use d_engine_core::DefaultCommitHandler; +use std::fmt::Debug; +use std::sync::Arc; +use std::sync::atomic::AtomicBool; + use d_engine_core::DefaultPurgeExecutor; use d_engine_core::ElectionHandler; use d_engine_core::InternalEvent; use d_engine_core::LogSizePolicy; -use d_engine_core::NewCommitData; use d_engine_core::Raft; use d_engine_core::RaftCoreHandlers; use d_engine_core::RaftLog; @@ -47,6 +46,7 @@ use d_engine_core::RaftStorageHandles; use d_engine_core::ReplicationHandler; use d_engine_core::Result; use d_engine_core::SignalParams; +use d_engine_core::SmWorkerHandle; use d_engine_core::StateMachine; use d_engine_core::StateMachineCommand; use d_engine_core::StateMachineWorker; @@ -63,9 +63,6 @@ use d_engine_core::learner_state::LearnerState; use d_engine_core::watch::WatchDispatcher; #[cfg(feature = "watch")] use d_engine_core::watch::WatchRegistry; -use std::fmt::Debug; -use std::sync::Arc; -use std::sync::atomic::AtomicBool; use tokio::sync::Mutex; use tokio::sync::mpsc; use tokio::sync::watch; @@ -79,6 +76,7 @@ use crate::Node; use crate::membership::RaftMembership; use crate::network::grpc; use crate::network::grpc::grpc_transport::GrpcTransport; +use crate::read_actor::run_read_actor; type StateMachineHandler = ( Arc>>, @@ -304,9 +302,6 @@ where // snapshots_dir is not configurable — always data_dir/snapshots (see #10). let snapshots_dir = data_dir.snapshots_dir()?; - // Init CommitHandler - let (new_commit_event_tx, new_commit_event_rx) = mpsc::unbounded_channel::(); - // Handle state machine initialization let state_machine = self.state_machine.take().ok_or_else(|| { SystemError::NodeStartFailed( @@ -416,7 +411,8 @@ where unregister_tx, )); - // Shared last_applied for Progress event revision; written by handler, read by dispatcher. + // Shared last_applied for Progress event revision; written by handler, read by + // dispatcher. let last_applied_ref = Arc::new(std::sync::atomic::AtomicU64::new(last_applied_index)); // Create dispatcher @@ -441,7 +437,8 @@ where #[cfg(feature = "watch")] let watch_event_tx = watch_system.as_ref().map(|(tx, _, _, _)| tx.clone()); - // Share the registry's live prev_kv counter Arc so the handler sees real-time updates. + // Share the registry's live prev_kv counter Arc so the handler sees real-time + // updates. #[cfg(feature = "watch")] let prev_kv_count = watch_system .as_ref() @@ -492,10 +489,11 @@ where let purge_executor = DefaultPurgeExecutor::new(raft_log.clone()); - // internal_event_tx / internal_event_rx created earlier (before raft_log) to pass log_flush_tx to start(). + // internal_event_tx / internal_event_rx created earlier (before raft_log) to pass + // log_flush_tx to start(). let (event_tx, event_rx) = mpsc::channel(10240); let (cmd_tx, cmd_rx) = mpsc::channel(node_config.raft.cmd_channel_capacity); - let internal_event_tx_clone = internal_event_tx.clone(); + let internal_event_tx_for_sm = internal_event_tx.clone(); // used in SM worker (ApplyCompleted → P2) // Bridge zombie signals from health monitor → internal event loop. @@ -516,7 +514,8 @@ where }); // Bridge peer stream failures from transport → health monitor. - // Exits naturally when transport (and its peer_failure_tx) drops (channel closed → recv None). + // Exits naturally when transport (and its peer_failure_tx) drops (channel closed → recv + // None). let membership_for_peer_failure = Arc::clone(&membership); tokio::spawn(async move { let mut peer_failure_rx = peer_failure_rx; @@ -526,7 +525,8 @@ where }); // Bridge peer stream successes from transport → health monitor. - // Resets failure counter so transient errors during failover don't accumulate into false zombies. + // Resets failure counter so transient errors during failover don't accumulate into false + // zombies. let membership_for_peer_success = Arc::clone(&membership); tokio::spawn(async move { let mut peer_success_rx = peer_success_rx; @@ -598,46 +598,21 @@ where node_config_arc.clone(), ); - // Register commit event listener - raft_core.register_new_commit_listener(new_commit_event_tx); - // Create leader notification channel and register with Raft core let leader_notifier = LeaderNotifier::new(); raft_core.register_leader_change_listener(leader_notifier.sender()); - // Spawn SM Worker task - let sm_worker = StateMachineWorker::new( - node_id, - state_machine_writer.clone(), + // Assemble the SM Worker; `Node::run` starts it. + let sm_worker = SmWorkerHandle::new( + StateMachineWorker::new( + node_id, + state_machine_writer.clone(), + sm_apply_tx.clone(), + sm_apply_rx, + internal_event_tx_for_sm, + ), sm_apply_tx.clone(), - sm_apply_rx, - internal_event_tx_for_sm, - self.shutdown_signal.clone(), - ); - let sm_worker_handle = Self::spawn_state_machine_worker(sm_worker); - - // Start CommitHandler in a single thread - let deps = CommitHandlerDependencies { - state_machine_handler, - raft_log: raft_core.ctx.storage.raft_log.clone(), - membership: membership.clone(), - internal_event_tx: internal_event_tx_clone, - sm_apply_tx: state_machine_commands, - shutdown_signal, - max_batch_size: raft_core.ctx.node_config.raft.batching.max_batch_size, - }; - - let commit_handler = DefaultCommitHandler::>::new( - node_id, - my_role_i32, - my_current_term, - deps, - new_commit_event_rx, ); - // Spawn commit listener via Builder method - // This ensures all tokio::spawn calls are visible in one place (Builder impl) - // Following the "one page visible" Builder pattern principle - let commit_handler_handle = Self::spawn_state_machine_commit_listener(commit_handler); let event_tx = raft_core.event_sender(); let read_lease = raft_core.read_lease(); @@ -670,10 +645,9 @@ where watch_registry: watch_system.as_ref().map(|(_, reg, _, _)| Arc::clone(reg)), #[cfg(feature = "watch")] _watch_dispatcher_handle: watch_system.map(|(_, _, handle, _)| handle), - sm_worker_handle: std::sync::Mutex::new(Some(sm_worker_handle)), + sm_worker, read_actor_handle: std::sync::Mutex::new(Some(read_actor_handle)), rpc_server_handle: std::sync::Mutex::new(None), - _commit_handler_handle: Some(commit_handler_handle), _lease_cleanup_handle: lease_cleanup_handle, shutdown_signal: self.shutdown_signal.clone(), read_lease, @@ -685,56 +659,6 @@ where Ok(self) } - /// Spawn state machine commit listener as background task. - /// - /// This method is called during node build() to start the commit handler thread. - /// All spawn_* methods are centralized in NodeBuilder so developers can see - /// all resource consumption (threads/tasks) in one place. - /// - /// # Returns - /// * `JoinHandle` - Task handle for lifecycle management - fn spawn_state_machine_commit_listener( - mut commit_handler: DefaultCommitHandler> - ) -> tokio::task::JoinHandle<()> { - tokio::spawn(async move { - match commit_handler.run().await { - Ok(_) => { - info!("commit_handler exit program"); - } - Err(e) => { - error!("commit_handler exit program with unexpected error: {:?}", e); - println!("commit_handler exit program"); - } - } - }) - } - - /// Spawn state machine worker on a dedicated OS thread. - /// - /// Runs on `sm-apply-{node_id}` OS thread (not a tokio worker) so that - /// synchronous RocksDB writes inside `apply_chunk` never block the async runtime - /// or compete with tokio's blocking thread pool. - /// - /// # Returns - /// * `JoinHandle` - Thread handle for lifecycle management - fn spawn_state_machine_worker( - sm_worker: StateMachineWorker> - ) -> std::thread::JoinHandle<()> { - let node_id = sm_worker.node_id(); - let handle = tokio::runtime::Handle::current(); - std::thread::Builder::new() - .name(format!("sm-apply-{}", node_id)) - .spawn(move || { - handle.block_on(async move { - match sm_worker.run().await { - Ok(_) => info!("state_machine_worker exit"), - Err(e) => error!("state_machine_worker exit with error: {:?}", e), - } - }); - }) - .expect("failed to spawn sm-apply thread") - } - /// Spawn lease background cleanup task (if enabled). /// /// This task runs independently from Raft apply pipeline, avoiding any blocking. @@ -821,7 +745,7 @@ where /// This is the primary method to initialize and start a node. It performs: /// 1. State machine initialization (including lease injection if applicable) /// 2. Raft core construction - /// 3. Background task spawning (commit handler, replication, election) + /// 3. Background task spawning (state machine worker, read actor, lease cleanup) /// 4. gRPC server startup for cluster communication /// /// # Returns diff --git a/d-engine-server/src/node/mod.rs b/d-engine-server/src/node/mod.rs index a01df19f..9cc0bae0 100644 --- a/d-engine-server/src/node/mod.rs +++ b/d-engine-server/src/node/mod.rs @@ -21,6 +21,7 @@ pub(crate) use leader_notifier::*; #[doc(hidden)] mod type_config; +use tracing::error; use tracing::info; #[doc(hidden)] pub use type_config::*; @@ -46,6 +47,7 @@ use d_engine_core::Raft; use d_engine_core::RaftNodeConfig; use d_engine_core::ReadLease; use d_engine_core::Result; +use d_engine_core::SmWorkerHandle; use d_engine_core::TypeConfig; use d_engine_core::alias::MOF; #[cfg(feature = "watch")] @@ -102,10 +104,9 @@ where #[cfg(feature = "watch")] pub(crate) _watch_dispatcher_handle: Option>, - /// State machine worker thread handle (dedicated OS thread, not a tokio task). - /// Wrapped in Mutex so run(&self) can take it for joining after Raft loop exits, - /// ensuring `Arc` is released before run() returns. - pub(crate) sm_worker_handle: std::sync::Mutex>>, + /// State machine worker (dedicated OS thread). Started by `run()` and stopped there, so + /// its `Arc` is released before `run()` returns. + pub(crate) sm_worker: SmWorkerHandle, /// ReadActor task handle. Aborted in run() after sm_worker exits so that the /// `Arc` (and any exclusive resources the storage backend holds) is @@ -116,9 +117,6 @@ where /// long-lived streams (the Raft replication bidi stream never ends on its own) pub(crate) rpc_server_handle: std::sync::Mutex>>, - /// Commit handler task handle (background log application) - pub(crate) _commit_handler_handle: Option>, - /// Lease cleanup task handle (background TTL cleanup) pub(crate) _lease_cleanup_handle: Option>, @@ -166,7 +164,9 @@ where /// Both paths converge to the inbound event processing loop. /// /// # Errors - /// Returns `Err` if any bootstrap step or Raft execution fails. + /// Returns `Err` if the state machine worker cannot be started (no Tokio runtime, or the OS + /// refuses a thread), a bootstrap step fails, or Raft execution fails. The components `run()` + /// owns are stopped before it returns in every one of these cases. /// /// # Example /// ```ignore @@ -176,6 +176,14 @@ where /// }); /// ``` pub async fn run(&self) -> Result<()> { + let result = self.serve().await; + self.stop_components().await; + result + } + + /// Starts the state machine worker, bootstraps, then runs the Raft loop until it ends. + async fn serve(&self) -> Result<()> { + self.sm_worker.start()?; let mut shutdown_signal = self.shutdown_signal.clone(); shutdown_signal.borrow_and_update(); @@ -188,8 +196,12 @@ where // Start Raft main loop. // Note: IO thread is closed inside Raft::run() on shutdown before returning. - self.start_raft_loop().await?; + self.start_raft_loop().await + } + /// Stops what `run()` owns, in reverse startup order. Runs on every exit path of `serve()`, + /// so it must not fail: problems are logged and shutdown continues. + async fn stop_components(&self) { // Bounded wait for graceful shutdown, then force-abort — the replication // bidi stream never ends on its own (#438). let rpc_handle = self.rpc_server_handle.lock().unwrap().take(); @@ -203,28 +215,25 @@ where } } - // Shutdown in reverse startup order: join sm-worker thread first so its - // Arc clone is dropped before we return, letting the storage backend - // release any exclusive resources it holds. - let handle = self.sm_worker_handle.lock().unwrap().take(); - if let Some(handle) = handle { - tokio::task::spawn_blocking(move || { - let _ = handle.join(); - }) - .await - .ok(); + // Shutdown in reverse startup order: stop the sm worker first so its Arc clone is + // dropped before we return, letting the storage backend release any exclusive + // resources it holds. + let timeout = + Duration::from_millis(self.node_config.raft.state_machine.shutdown_apply_timeout_ms); + if let Err(e) = self.sm_worker.stop_and_join(timeout).await { + error!( + "state machine worker did not stop cleanly: {e}; shutting down anyway, \ + committed entries may be unapplied until restart" + ); } - // Abort ReadActor after sm_worker exits so Arc ref-count reaches zero, - // letting the storage backend release any exclusive resources (e.g. an - // on-disk lock file) before run() returns. + // Abort the ReadActor last. The worker has already closed the state machine, and the + // ReadActor only serves reads, so there is nothing left to wait for. let ra_handle = self.read_actor_handle.lock().unwrap().take(); if let Some(ra_handle) = ra_handle { ra_handle.abort(); let _ = ra_handle.await; } - - Ok(()) } /// Learner bootstrap: skip cluster ready check, join after warmup. diff --git a/d-engine-server/src/node/node_test.rs b/d-engine-server/src/node/node_test.rs index 427b0494..f873f250 100644 --- a/d-engine-server/src/node/node_test.rs +++ b/d-engine-server/src/node/node_test.rs @@ -80,7 +80,8 @@ fn prepare_succeed_majority_confirmation() -> (MockRaftLog, MockReplicationCore< // Initialize the mock object let mut replication_handler = MockReplicationCore::::new(); let mut raft_log = MockRaftLog::new(); - // The first election now starts at once, so a single-node cluster becomes leader and proposes its noop. + // The first election now starts at once, so a single-node cluster becomes leader and proposes + // its noop. replication_handler .expect_prepare_batch_requests() .returning(|_, _, _, _, _, _| Ok(d_engine_core::PrepareResult::default())); @@ -922,3 +923,283 @@ mod bootstrap_strategy_tests { ); } } + +/// Shutdown hands committed entries to the state machine worker and then waits for it to finish +/// applying them. That wait must be bounded: a wedged state machine may not hang `run()`. +mod shutdown_apply_wait_tests { + use std::sync::atomic::AtomicBool; + use std::sync::atomic::Ordering; + use std::time::Instant; + + use d_engine_core::ApplyResult; + use d_engine_core::MockStateMachineWriterOps; + use d_engine_core::StateMachineCommand; + use d_engine_proto::common::Entry; + use d_engine_proto::common::EntryPayload; + + use super::*; + use crate::test_utils::sm_worker_with_writer; + + /// A node whose state machine worker is real and whose writer is `writer`, with one entry + /// already queued for it to apply when `run()` starts it. + fn node_applying( + writer: MockStateMachineWriterOps, + timeout_ms: u64, + shutdown_rx: watch::Receiver<()>, + ) -> crate::Node { + let mut config = RaftNodeConfig::new().expect("default config").validate().expect("valid"); + config.raft.state_machine.shutdown_apply_timeout_ms = timeout_ms; + let (sm_worker, queue) = sm_worker_with_writer(writer); + queue + .send(StateMachineCommand::ApplyEntries { + sent_at: tokio::time::Instant::now(), + entries: vec![Entry { + index: 1, + term: 1, + payload: Some(EntryPayload::command(bytes::Bytes::from("data"))), + }], + }) + .unwrap(); + MockBuilder::new(shutdown_rx) + .with_node_config(config) + .with_sm_worker(sm_worker, queue) + .build_node() + } + + /// A state machine worker that never finishes (stuck apply) must not hang shutdown: + /// `run()` gives up after the configured timeout and says so at ERROR level, because + /// entries the node reported as committed may then be unapplied until the next start. + #[tokio::test] + #[traced_test] + async fn test_run_shutdown_with_stuck_state_machine_returns_after_timeout_and_logs_error() { + let (shutdown_tx, shutdown_rx) = watch::channel(()); + let (release_tx, release_rx) = std::sync::mpsc::channel::<()>(); + let release_rx = std::sync::Mutex::new(release_rx); + let mut writer = MockStateMachineWriterOps::new(); + writer.expect_apply_chunk().returning(move |entries| { + let _ = release_rx.lock().unwrap().recv(); // blocks until the test releases it + Ok(entries.iter().map(|e| ApplyResult::success(e.index)).collect()) + }); + let node = node_applying(writer, 300, shutdown_rx); + + shutdown_tx.send(()).unwrap(); + let started = Instant::now(); + let result = tokio::time::timeout(Duration::from_secs(5), node.run()) + .await + .expect("run() hung on a stuck state machine worker; the wait must be bounded"); + let elapsed = started.elapsed(); + let _ = release_tx.send(()); + + assert!( + result.is_ok(), + "timeout while draining must not turn shutdown into a panic" + ); + assert!( + elapsed >= Duration::from_millis(250), + "returned after {elapsed:?}: gave up before the configured timeout" + ); + logs_assert(|lines: &[&str]| { + lines + .iter() + .any(|l| l.contains("ERROR") && l.to_lowercase().contains("state machine")) + .then_some(()) + .ok_or_else(|| "no ERROR log about the state machine apply timeout".to_string()) + }); + } + + /// A healthy worker that needs a moment to finish applying must be waited for + /// (that is the whole guarantee), and no error is logged. + #[tokio::test] + #[traced_test] + async fn test_run_shutdown_waits_for_state_machine_that_finishes_within_timeout() { + let (shutdown_tx, shutdown_rx) = watch::channel(()); + let applied = Arc::new(AtomicBool::new(false)); + let applied_in_worker = applied.clone(); + let mut writer = MockStateMachineWriterOps::new(); + writer.expect_apply_chunk().returning(move |entries| { + std::thread::sleep(Duration::from_millis(300)); + applied_in_worker.store(true, Ordering::SeqCst); + Ok(entries.iter().map(|e| ApplyResult::success(e.index)).collect()) + }); + let node = node_applying(writer, 5_000, shutdown_rx); + + shutdown_tx.send(()).unwrap(); + node.run().await.expect("shutdown ok"); + + assert!( + applied.load(Ordering::SeqCst), + "run() returned before the worker finished applying" + ); + logs_assert(|lines: &[&str]| { + match lines + .iter() + .find(|l| l.contains("ERROR") && l.to_lowercase().contains("state machine")) + { + Some(l) => Err(format!("unexpected error log: {l}")), + None => Ok(()), + } + }); + } +} + +/// The state machine worker is a dedicated OS thread started by `run()`. Whatever ends +/// `run()`, that thread must not outlive it: it holds the state machine +/// (and with it the storage lock), so a leaked thread blocks restarting on the same data dir. +mod sm_worker_lifecycle_tests { + use std::time::Instant; + + use super::*; + use crate::FileStateMachine; + use crate::FileStorageEngine; + use crate::node::NodeBuilder; + + /// `true` once the node's worker has been stopped and joined. + fn sm_worker_stopped(node: &crate::Node) -> bool { + format!("{:?}", node.sm_worker).contains("Stopped") + } + + /// Business scenario: an embedder starts a 3-node cluster member whose peers are down. Start-up + /// gives up at `startup_quorum_timeout` and `run()` returns an error, but the embedder still + /// holds the engine (so the shutdown signal is NOT sent or dropped) and will retry. + /// + /// Purpose: after `run()` fails during bootstrap, the sm-apply thread must have stopped. + /// Before the fix nobody told it to stop, so it stayed blocked forever, holding the state + /// machine. + #[tokio::test] + async fn test_run_failing_during_bootstrap_stops_sm_apply_thread() { + let temp_dir = tempfile::tempdir().unwrap(); + let (_shutdown_tx, shutdown_rx) = watch::channel(()); + + let mut config = RaftNodeConfig::new().unwrap().validate().unwrap(); + config.cluster.node_id = 1; + config.cluster.initial_cluster = (1..=3u32) + .map(|id| NodeMeta { + id, + // Nothing listens on these ports: peers 2 and 3 are unreachable. + address: format!("127.0.0.1:{}", 1 + id), + role: Follower as i32, + status: NodeStatus::Active as i32, + }) + .collect(); + config.raft.membership.startup_quorum_timeout = Duration::from_millis(200); + + let storage = + Arc::new(FileStorageEngine::new(temp_dir.path().join("storage")).expect("storage")); + let sm = Arc::new(FileStateMachine::new(temp_dir.path().join("sm")).await.expect("sm")); + let node = NodeBuilder::::from_node_config( + temp_dir.path(), + config, + shutdown_rx, + ) + .storage_engine(storage) + .state_machine(sm) + .build() + .await + .expect("build") + .node + .expect("node built"); + + let result = tokio::time::timeout(Duration::from_secs(10), node.run()) + .await + .expect("run() must give up at startup_quorum_timeout"); + assert!( + result.is_err(), + "bootstrap should fail with unreachable peers" + ); + + let deadline = Instant::now() + Duration::from_secs(2); + let stopped = loop { + let stopped = sm_worker_stopped(&node); + if stopped || Instant::now() > deadline { + break stopped; + } + tokio::time::sleep(Duration::from_millis(20)).await; + }; + assert!( + stopped, + "the sm-apply thread is still running after run() failed during bootstrap" + ); + } + + /// Business scenario: an embedder drops its `EmbeddedEngine` without calling `stop()`. The + /// shutdown sender is dropped with it, which the Raft loop treats as a shutdown request. + /// + /// Purpose: guard for the safety net that must survive any change to how the worker is + /// stopped: a running node whose shutdown sender is dropped must still end `run()` and leave + /// no sm-apply thread behind. + #[tokio::test] + async fn test_run_stops_sm_apply_thread_when_shutdown_sender_is_dropped() { + let temp_dir = tempfile::tempdir().unwrap(); + let (shutdown_tx, shutdown_rx) = watch::channel(()); + let config = RaftNodeConfig::new().unwrap().validate().unwrap(); + + let storage = + Arc::new(FileStorageEngine::new(temp_dir.path().join("storage")).expect("storage")); + let sm = Arc::new(FileStateMachine::new(temp_dir.path().join("sm")).await.expect("sm")); + let node = NodeBuilder::::from_node_config( + temp_dir.path(), + config, + shutdown_rx, + ) + .storage_engine(storage) + .state_machine(sm) + .build() + .await + .expect("build") + .node + .expect("node built"); + + let running = tokio::spawn({ + let node = node.clone(); + async move { node.run().await } + }); + tokio::time::sleep(Duration::from_millis(300)).await; + drop(shutdown_tx); + + let result = tokio::time::timeout(Duration::from_secs(10), running) + .await + .expect("run() must end once the shutdown sender is dropped") + .expect("run() task must not panic"); + assert!( + result.is_ok(), + "shutdown by dropping the sender is a clean stop" + ); + assert!( + sm_worker_stopped(&node), + "the sm-apply thread is still running after run() ended" + ); + } + + /// Business scenario: the worker thread cannot be started (here: `run()` is driven by an + /// executor that is not a Tokio runtime, so the worker has no runtime to run on). `run()` + /// fails at its first step, before anything else was started. + /// + /// Purpose: the unconditional tail still runs and hands the state machine back closed even + /// though the worker never ran, because the developer may keep their own `Arc` to it. + #[test] + fn test_run_failing_to_start_the_worker_still_closes_the_state_machine() { + let closed = Arc::new(std::sync::atomic::AtomicUsize::new(0)); + let counter = closed.clone(); + let mut writer = d_engine_core::MockStateMachineWriterOps::new(); + writer.expect_close().returning(move || { + counter.fetch_add(1, std::sync::atomic::Ordering::SeqCst); + Ok(()) + }); + let (_shutdown_tx, shutdown_rx) = watch::channel(()); + let (sm_worker, queue) = crate::test_utils::sm_worker_with_writer(writer); + let node = MockBuilder::new(shutdown_rx).with_sm_worker(sm_worker, queue).build_node(); + + // No Tokio runtime on this thread: `SmWorkerHandle::start` has nowhere to run the worker. + let result = futures::executor::block_on(node.run()); + + assert!( + result.is_err(), + "starting the worker without a runtime must fail run()" + ); + assert_eq!( + closed.load(std::sync::atomic::Ordering::SeqCst), + 1, + "the never-started worker's state machine must be closed by run()'s tail" + ); + } +} diff --git a/d-engine-server/src/node/type_config/raft_type_config.rs b/d-engine-server/src/node/type_config/raft_type_config.rs index 60c6f785..9c620684 100644 --- a/d-engine-server/src/node/type_config/raft_type_config.rs +++ b/d-engine-server/src/node/type_config/raft_type_config.rs @@ -1,6 +1,3 @@ -use std::fmt::Debug; - -use d_engine_core::DefaultCommitHandler; use d_engine_core::DefaultPurgeExecutor; use d_engine_core::DefaultStateMachineHandler; use d_engine_core::DefaultStateMachineWriter; @@ -18,16 +15,16 @@ use crate::network::grpc::grpc_transport::GrpcTransport; #[derive(Debug)] pub struct RaftTypeConfig where - SE: StorageEngine + Debug, - SM: StateMachine + Debug, + SE: StorageEngine + std::fmt::Debug, + SM: StateMachine + std::fmt::Debug, { _marker: std::marker::PhantomData<(SE, SM)>, } impl TypeConfig for RaftTypeConfig where - SE: StorageEngine + Debug, - SM: StateMachine + Debug, + SE: StorageEngine + std::fmt::Debug, + SM: StateMachine + std::fmt::Debug, { type SE = SE; @@ -43,8 +40,6 @@ where type E = ElectionHandler; - type C = DefaultCommitHandler; - type SMH = DefaultStateMachineHandler; type SMW = DefaultStateMachineWriter; diff --git a/d-engine-server/src/proto_convert.rs b/d-engine-server/src/proto_convert.rs index 3ee62ba1..bda2be62 100644 --- a/d-engine-server/src/proto_convert.rs +++ b/d-engine-server/src/proto_convert.rs @@ -12,23 +12,32 @@ //! //! - All functions are `#[inline]` //! - `Bytes` fields are moved (refcount unchanged, zero allocation) -//! - `ErrorCode` conversion uses exhaustive match — LLVM optimises to identity; -//! a missing variant is a compile error (correctness guarantee) -use crate::api::types::WriteOperation; +//! - `ErrorCode` conversion uses exhaustive match — LLVM optimises to identity; a missing variant +//! is a compile error (correctness guarantee) use bytes::Bytes; -use d_engine_core::client::{ - ClientReadRequest, ClientResponse, ClientResponsePayload, ClientWriteRequest, ErrorCode, - KvEntry, LeaderHint, ReadResults, WriteResult, -}; +use d_engine_core::client::ClientReadRequest; +use d_engine_core::client::ClientResponse; +use d_engine_core::client::ClientResponsePayload; +use d_engine_core::client::ClientWriteRequest; +use d_engine_core::client::ErrorCode; +use d_engine_core::client::KvEntry; +use d_engine_core::client::LeaderHint; +use d_engine_core::client::ReadResults; +use d_engine_core::client::WriteResult; use d_engine_core::config::ReadConsistencyPolicy; use d_engine_core::types::ScanResults; -use d_engine_proto::client::{ - self as proto_client, ClientResult, ReadResults as ProtoReadResults, - ScanResults as ProtoScanResults, WriteCommand, WriteResult as ProtoWriteResult, -}; -use d_engine_proto::error::{ErrorCode as ProtoErrorCode, ErrorMetadata}; +use d_engine_proto::client::ClientResult; +use d_engine_proto::client::ReadResults as ProtoReadResults; +use d_engine_proto::client::ScanResults as ProtoScanResults; +use d_engine_proto::client::WriteCommand; +use d_engine_proto::client::WriteResult as ProtoWriteResult; +use d_engine_proto::client::{self as proto_client}; +use d_engine_proto::error::ErrorCode as ProtoErrorCode; +use d_engine_proto::error::ErrorMetadata; use prost::Message; +use crate::api::types::WriteOperation; + #[inline] pub(crate) fn write_op_to_bytes(op: WriteOperation) -> Bytes { Bytes::from(WriteCommand::from(op).encode_to_vec()) diff --git a/d-engine-server/src/proto_convert_test.rs b/d-engine-server/src/proto_convert_test.rs index 175660ac..132c6e25 100644 --- a/d-engine-server/src/proto_convert_test.rs +++ b/d-engine-server/src/proto_convert_test.rs @@ -1,20 +1,28 @@ use bytes::Bytes; -use d_engine_core::client::{ - ClientReadRequest, ClientResponse, ClientResponsePayload, ClientWriteRequest, ErrorCode, - KvEntry, LeaderHint, ReadResults, WriteResult, -}; +use d_engine_core::client::ClientReadRequest; +use d_engine_core::client::ClientResponse; +use d_engine_core::client::ClientResponsePayload; +use d_engine_core::client::ClientWriteRequest; +use d_engine_core::client::ErrorCode; +use d_engine_core::client::KvEntry; +use d_engine_core::client::LeaderHint; +use d_engine_core::client::ReadResults; +use d_engine_core::client::WriteResult; use d_engine_core::config::ReadConsistencyPolicy; use d_engine_core::types::ScanResults; use d_engine_proto::client as proto_client; use d_engine_proto::client::WriteCommand; -use d_engine_proto::client::write_command::{Insert, Operation}; +use d_engine_proto::client::write_command::Insert; +use d_engine_proto::client::write_command::Operation; use d_engine_proto::error::ErrorCode as ProtoErrorCode; use prost::Message; -use crate::proto_convert::{ - core_error_to_proto, proto_error_to_core, to_core_read_req, to_core_response, - to_core_write_req, to_proto_response, -}; +use crate::proto_convert::core_error_to_proto; +use crate::proto_convert::proto_error_to_core; +use crate::proto_convert::to_core_read_req; +use crate::proto_convert::to_core_response; +use crate::proto_convert::to_core_write_req; +use crate::proto_convert::to_proto_response; // ─── to_core_write_req ──────────────────────────────────────────────────────── #[test] diff --git a/d-engine-server/src/read_actor.rs b/d-engine-server/src/read_actor.rs index 0cd633d7..7bc520eb 100644 --- a/d-engine-server/src/read_actor.rs +++ b/d-engine-server/src/read_actor.rs @@ -18,13 +18,15 @@ //! `read_rx` is drained and closed, at which point `Arc` is dropped and the //! RocksDB LOCK file is released — before `stop()` signals the Raft loop. +use std::sync::Arc; + use bytes::Bytes; use d_engine_core::ReadLease; use d_engine_core::StateMachine; use d_engine_core::config::ReadConsistencyPolicy; use d_engine_core::now_ms; -use std::sync::Arc; -use tokio::sync::{mpsc, oneshot}; +use tokio::sync::mpsc; +use tokio::sync::oneshot; // ── ReadCmd ──────────────────────────────────────────────────────────────────── diff --git a/d-engine-server/src/read_actor_test.rs b/d-engine-server/src/read_actor_test.rs index f3704589..8bc3202d 100644 --- a/d-engine-server/src/read_actor_test.rs +++ b/d-engine-server/src/read_actor_test.rs @@ -1,11 +1,17 @@ use std::sync::Arc; use bytes::Bytes; +use d_engine_core::Error; +use d_engine_core::MockStateMachine; +use d_engine_core::ReadLease; use d_engine_core::config::ReadConsistencyPolicy; -use d_engine_core::{Error, MockStateMachine, ReadLease, now_ms}; -use tokio::sync::{mpsc, oneshot}; +use d_engine_core::now_ms; +use tokio::sync::mpsc; +use tokio::sync::oneshot; -use crate::read_actor::{ReadActorError, ReadCmd, run_read_actor}; +use crate::read_actor::ReadActorError; +use crate::read_actor::ReadCmd; +use crate::read_actor::run_read_actor; fn make_sm_running() -> MockStateMachine { let mut sm = MockStateMachine::new(); diff --git a/d-engine-server/src/storage/adaptors/file/file_state_machine.rs b/d-engine-server/src/storage/adaptors/file/file_state_machine.rs index 17c09291..4b6bc761 100644 --- a/d-engine-server/src/storage/adaptors/file/file_state_machine.rs +++ b/d-engine-server/src/storage/adaptors/file/file_state_machine.rs @@ -81,30 +81,33 @@ //! - **Background Cleanup**: Periodic async worker scans and deletes expired keys //! - **Zero Overhead**: When TTL feature is disabled, no lease components are initialized -use crate::storage::TtlLease; +use std::collections::HashMap; +use std::collections::HashSet; +use std::io::Write; +use std::path::PathBuf; +use std::sync::Arc; +use std::sync::Mutex; +use std::sync::atomic::AtomicBool; +use std::sync::atomic::AtomicU64; +use std::sync::atomic::Ordering; +use std::time::SystemTime; + use async_trait::async_trait; use bytes::Bytes; +use d_engine_core::ApplyEntry; +use d_engine_core::ApplyResult; use d_engine_core::Command; use d_engine_core::Error; use d_engine_core::Lease; use d_engine_core::ScanResult; +use d_engine_core::SnapshotApplyResult; +use d_engine_core::SnapshotError; use d_engine_core::StateMachine; use d_engine_core::StorageError; use d_engine_core::file_io::compute_checksum_from_folder_path; -use d_engine_core::{ApplyEntry, SnapshotApplyResult}; -use d_engine_core::{ApplyResult, SnapshotError}; use d_engine_proto::common::LogId; use d_engine_proto::server::storage::SnapshotMetadata; use parking_lot::RwLock; -use std::collections::{HashMap, HashSet}; -use std::io::Write; -use std::path::PathBuf; -use std::sync::Arc; -use std::sync::Mutex; -use std::sync::atomic::AtomicBool; -use std::sync::atomic::AtomicU64; -use std::sync::atomic::Ordering; -use std::time::SystemTime; use tokio::fs; use tokio::fs::File; use tokio::fs::OpenOptions; @@ -116,6 +119,8 @@ use tracing::error; use tracing::info; use tracing::warn; +use crate::storage::TtlLease; + type FileStateMachineDataType = RwLock>; /// WAL operation codes for fixed-size encoding @@ -980,6 +985,31 @@ impl FileStateMachine { Ok(()) } + /// Reads are served only while running. After `close()` the developer may still hold a clone + /// of the state machine, and it must not answer from state that is no longer maintained. + fn ensure_serving(&self) -> Result<(), Error> { + if self.running.load(Ordering::Acquire) { + Ok(()) + } else { + Err(StorageError::NotServing("state machine stopped".into()).into()) + } + } + + /// Writes the lease table to `ttl_state.bin`. Once a checkpoint has cleared the WAL this + /// file is the only record of key expiry, so it is written to a temp file and renamed: a + /// crash mid-write cannot leave a truncated file. + fn persist_lease(&self) -> Result<(), Error> { + let Some(ref lease) = self.lease else { + return Ok(()); + }; + let tmp_path = self.data_dir.join("ttl_state.bin.tmp"); + let file = std::fs::File::create(&tmp_path)?; + std::io::Write::write_all(&mut &file, &lease.to_snapshot())?; + file.sync_all()?; + std::fs::rename(&tmp_path, self.data_dir.join("ttl_state.bin"))?; + Ok(()) + } + /// Batch WAL writes with proper durability guarantees. /// /// `cas_outcomes[i]` is consulted only when `entries[i].command` is @@ -1031,9 +1061,9 @@ impl Drop for FileStateMachine { let timer = Instant::now(); // Save state into local database including flush operation - match self.save_hard_state() { - Ok(_) => debug!("StateMachine saved in {:?}", timer.elapsed()), - Err(e) => error!("Failed to save StateMachine: {}", e), + match StateMachine::close(self) { + Ok(()) => debug!("StateMachine closed in {:?}", timer.elapsed()), + Err(e) => error!("Failed to close StateMachine: {}", e), } } } @@ -1053,29 +1083,16 @@ impl StateMachine for FileStateMachine { Ok(()) } - fn close_storage(&self) { - self.running.store(false, Ordering::SeqCst); - // FileStateMachine has no exclusive OS lock file. - // Marking not-running is sufficient; Drop handles final flush. - } - - fn stop(&self) -> Result<(), Error> { - // Ensure all data is flushed to disk before stopping - self.running.store(false, Ordering::SeqCst); - - // Graceful shutdown: persist TTL state to disk - // This ensures lease data survives across restarts - if let Some(ref lease) = self.lease { - let ttl_snapshot = lease.to_snapshot(); - let ttl_path = self.data_dir.join("ttl_state.bin"); - // Use blocking write since stop() is sync - std::fs::write(&ttl_path, ttl_snapshot) - .map_err(d_engine_core::StorageError::IoError)?; - debug!("Persisted TTL state on shutdown"); + fn close(&self) -> Result<(), Error> { + if !self.running.swap(false, Ordering::SeqCst) { + return Ok(()); // already closed } - - info!("File state machine stopped"); - Ok(()) + // The developer may keep an `Arc` to this state machine, so `Drop` may never run: + // persist everything a restart needs here. Both steps run; the first error is returned. + // No exclusive OS lock file to release. + let saved = self.save_hard_state(); + let leased = self.persist_lease(); + saved.and(leased) } fn is_running(&self) -> bool { @@ -1086,6 +1103,7 @@ impl StateMachine for FileStateMachine { &self, key_buffer: &[u8], ) -> Result, Error> { + self.ensure_serving()?; // Lazy cleanup: only check expiration in Lazy strategy // Background strategy handles cleanup in dedicated async task // Background cleanup strategy: expired keys are cleaned by background worker @@ -1094,11 +1112,12 @@ impl StateMachine for FileStateMachine { Ok(data.get(key_buffer).map(|(value, _)| value.clone())) } - /// Thread-safe: called serially by single-task CommitHandler + /// Thread-safe: called serially by the state machine worker async fn apply_chunk( &self, chunk: &[ApplyEntry], ) -> Result, Error> { + self.ensure_serving()?; let chunk_len = chunk.len(); let mut results = Vec::with_capacity(chunk_len); let mut highest_log_id: Option = None; @@ -1661,6 +1680,7 @@ impl StateMachine for FileStateMachine { &self, prefix: &[u8], ) -> Result { + self.ensure_serving()?; let data = self.data.read(); let entries: Vec<(Bytes, Bytes)> = data .iter() @@ -1675,6 +1695,7 @@ impl StateMachine for FileStateMachine { &self, keys: &[Bytes], ) -> d_engine_core::Result>> { + self.ensure_serving()?; // Hold read lock for the full batch so all keys come from the same consistent view. // apply_chunk holds the write lock; parking_lot RwLock ensures they don't interleave. let data = self.data.read(); diff --git a/d-engine-server/src/storage/adaptors/file/file_state_machine_test.rs b/d-engine-server/src/storage/adaptors/file/file_state_machine_test.rs index f68fd9b0..a0e5ef5e 100644 --- a/d-engine-server/src/storage/adaptors/file/file_state_machine_test.rs +++ b/d-engine-server/src/storage/adaptors/file/file_state_machine_test.rs @@ -1,41 +1,89 @@ +use std::sync::Arc; + +use async_trait::async_trait; use bytes::Bytes; +use d_engine_core::ApplyEntry; +use d_engine_core::Command; +use d_engine_core::Error; +use d_engine_core::SnapshotApplyResult; +use d_engine_core::StateMachine; use d_engine_core::file_io::compute_checksum_from_folder_path; -use d_engine_core::{ApplyEntry, Command, SnapshotApplyResult, StateMachine}; +use d_engine_core::storage::state_machine_test::StateMachineBuilder; +use d_engine_core::storage::state_machine_test::StateMachineTestSuite; use d_engine_proto::common::LogId; use d_engine_proto::server::storage::SnapshotMetadata; -use std::sync::Arc; +use tempfile::TempDir; use tokio::io::AsyncWriteExt; use crate::storage::adaptors::file::FileStateMachine; -use async_trait::async_trait; -use d_engine_core::Error; -use d_engine_core::storage::state_machine_test::{StateMachineBuilder, StateMachineTestSuite}; -use tempfile::TempDir; -// ── close_storage() tests ───────────────────────────────────────────────────── +// ── close() tests ───────────────────────────────────────────────────────────── -/// close_storage() must mark the SM as not running. -/// FileStateMachine has no exclusive OS lock, so close_storage() is a lifecycle -/// signal — subsequent reads must fail with NotServing. +/// close() must mark the SM as not running. +/// FileStateMachine has no exclusive OS lock, so close() is a lifecycle +/// signal and persists lease state. #[tokio::test] -async fn test_file_sm_close_storage_marks_not_running() { +async fn test_file_sm_close_marks_not_running() { let dir = TempDir::new().unwrap(); let sm = FileStateMachine::new(dir.path().to_path_buf()).await.unwrap(); sm.start().await.unwrap(); assert!(sm.is_running()); - sm.close_storage(); + sm.close().unwrap(); + + assert!(!sm.is_running(), "close must mark SM not running"); +} + +/// Business scenario: the node shuts down while the developer still holds a clone of the state +/// machine (for example an embedded client reading directly from it). +/// +/// Purpose: once closed, every read fails with NotServing instead of answering from state that is +/// no longer maintained. Same contract as the RocksDB state machine. +#[tokio::test] +async fn test_file_sm_reads_fail_after_close() { + let dir = TempDir::new().unwrap(); + let sm = FileStateMachine::new(dir.path().to_path_buf()).await.unwrap(); + sm.start().await.unwrap(); + sm.apply_chunk(&[ApplyEntry { + index: 1, + term: 1, + command: Command::Insert { + key: Bytes::from_static(b"k"), + value: Bytes::from_static(b"v"), + ttl_secs: None, + }, + }]) + .await + .unwrap(); + assert_eq!( + sm.get(b"k").unwrap(), + Some(Bytes::from_static(b"v")), + "readable while running" + ); - assert!(!sm.is_running(), "close_storage must mark SM not running"); + sm.close().unwrap(); + + let not_serving = |err: d_engine_core::Error| format!("{err:?}").contains("NotServing"); + assert!(not_serving( + sm.get(b"k").expect_err("get must fail after close") + )); + assert!(not_serving( + sm.get_multi(&[Bytes::from_static(b"k")]) + .expect_err("get_multi must fail after close") + )); + assert!(not_serving( + sm.scan_prefix(b"k").expect_err("scan_prefix must fail after close") + )); } -/// stop() after close_storage() must not panic (idempotent shutdown). +/// close() is idempotent (the worker and the `Drop` safety net may both close). #[tokio::test] -async fn test_file_sm_stop_after_close_storage_is_safe() { +async fn test_file_sm_close_twice_is_safe() { let dir = TempDir::new().unwrap(); let sm = FileStateMachine::new(dir.path().to_path_buf()).await.unwrap(); - sm.close_storage(); - assert!(sm.stop().is_ok(), "stop after close_storage must not error"); + sm.close().unwrap(); + sm.close().unwrap(); + assert!(!sm.is_running()); } /// Builder for FileStateMachine test instances diff --git a/d-engine-server/src/storage/adaptors/file/file_storage_engine_test.rs b/d-engine-server/src/storage/adaptors/file/file_storage_engine_test.rs index 657ef83c..fd23afc7 100644 --- a/d-engine-server/src/storage/adaptors/file/file_storage_engine_test.rs +++ b/d-engine-server/src/storage/adaptors/file/file_storage_engine_test.rs @@ -472,9 +472,8 @@ mod tests { use async_trait::async_trait; use d_engine_core::Error; - use d_engine_core::storage::storage_engine_test::{ - StorageEngineBuilder, StorageEngineTestSuite, - }; + use d_engine_core::storage::storage_engine_test::StorageEngineBuilder; + use d_engine_core::storage::storage_engine_test::StorageEngineTestSuite; /// Builder for FileStorageEngine test instances struct FileStorageEngineBuilder { @@ -597,21 +596,18 @@ mod tests { // Act: conflict at index=3, new suffix [3(t2), 4(t2)] — entry 5 must vanish log_store - .replace_range( - 3, - vec![ - Entry { - index: 3, - term: 2, - payload: None, - }, - Entry { - index: 4, - term: 2, - payload: None, - }, - ], - ) + .replace_range(3, vec![ + Entry { + index: 3, + term: 2, + payload: None, + }, + Entry { + index: 4, + term: 2, + payload: None, + }, + ]) .await .unwrap(); diff --git a/d-engine-server/src/storage/adaptors/rocksdb/rocksdb_state_machine.rs b/d-engine-server/src/storage/adaptors/rocksdb/rocksdb_state_machine.rs index 90e1387d..d7f80dd0 100644 --- a/d-engine-server/src/storage/adaptors/rocksdb/rocksdb_state_machine.rs +++ b/d-engine-server/src/storage/adaptors/rocksdb/rocksdb_state_machine.rs @@ -1,4 +1,10 @@ -use crate::storage::TtlLease; +use std::path::Path; +use std::sync::Arc; +use std::sync::atomic::AtomicBool; +use std::sync::atomic::AtomicU64; +use std::sync::atomic::Ordering; +use std::time::SystemTime; + use arc_swap::ArcSwapOption; use async_trait::async_trait; use bytes::Bytes; @@ -31,12 +37,6 @@ use rocksdb::WriteBatchWithIndex; use rocksdb::WriteOptions; use serde::Deserialize; use serde::Serialize; -use std::path::Path; -use std::sync::Arc; -use std::sync::atomic::AtomicBool; -use std::sync::atomic::AtomicU64; -use std::sync::atomic::Ordering; -use std::time::SystemTime; use tracing::debug; use tracing::error; use tracing::info; @@ -45,6 +45,7 @@ use tracing::warn; use super::STATE_MACHINE_CF; use super::STATE_MACHINE_META_CF; +use crate::storage::TtlLease; const LAST_APPLIED_INDEX_KEY: &[u8] = b"last_applied_index"; const LAST_APPLIED_TERM_KEY: &[u8] = b"last_applied_term"; @@ -587,41 +588,6 @@ impl RocksDBStateMachine { Ok(ScanResult { entries, revision }) } - /// Permanently close the RocksDB handle and release the LOCK file. - /// - /// Unlike `stop()`, this is not reversible with `start()`. - /// Called by `EmbeddedEngine::stop()` before the Raft loop exits so the LOCK - /// is released without waiting for `Arc` reference counts to reach zero. - /// - /// Sequence: flush → save hard state → cancel background work → store None. - /// After this, `load_db()` returns `NotServing`; `stop()` afterwards is safe. - pub(crate) fn close_db(&self) { - self.is_serving.store(false, Ordering::SeqCst); - - if let Err(e) = self.persist_ttl_metadata() { - error!("close_db: failed to persist TTL metadata: {e:?}"); - } - - if let Err(e) = self.flush() { - error!("close_db: failed to flush: {e:?}"); - } - - if let Err(e) = self.save_hard_state() { - error!("close_db: failed to save hard state: {e:?}"); - } - - { - let guard = self.db.load(); - if let Some(db) = guard.as_deref() { - db.cancel_all_background_work(true); - } - } - - // Release Arc: refcount decrements here, RocksDB LOCK file released. - self.db.store(None); - info!("RocksDB state machine: DB closed, LOCK released"); - } - /// Test-only: quiesce RocksDB's background compaction/flush WITHOUT flushing /// MemTable, saving hard state, or persisting TTL metadata — simulates what a /// real `kill -9` leaves behind on disk (no "dying gasp" persistence a real @@ -661,26 +627,34 @@ impl StateMachine for RocksDBStateMachine { Ok(()) } - fn close_storage(&self) { - self.close_db(); - } + /// Sequence: stop serving → persist TTL → flush → save hard state → cancel background + /// work → drop the DB handle (releasing the LOCK). A second call finds no DB and does nothing. + fn close(&self) -> Result<(), Error> { + if self.db.load().is_none() { + return Ok(()); + } - fn stop(&self) -> Result<(), Error> { self.is_serving.store(false, Ordering::SeqCst); - // Skip TTL persist if close_db() was already called (DB is None). - // Node::stop() calls sm.stop() after EmbeddedEngine has already called close_db(), - // so this path must be safe to call on a closed DB. - // Skip TTL persist if close_db() has already set DB to None. - if self.db.load().is_some() - && let Err(e) = self.persist_ttl_metadata() - { - error!("Failed to persist TTL metadata on shutdown: {:?}", e); - return Err(e); + // Every step runs even if an earlier one failed, so the LOCK is always released; + // the first error is returned. + let mut first_err: Option = None; + let mut record = |step: &str, result: Result<(), Error>| { + if let Err(e) = result { + error!("close: {step} failed: {e:?}"); + first_err.get_or_insert(e); + } + }; + record("persist TTL metadata", self.persist_ttl_metadata()); + record("flush", self.flush()); + record("save hard state", self.save_hard_state()); + if let Some(db) = self.db.load().as_deref() { + db.cancel_all_background_work(true); } - - info!("RocksDB state machine stopped"); - Ok(()) + // Release Arc: refcount decrements here, RocksDB LOCK file released. + self.db.store(None); + info!("RocksDB state machine: DB closed, LOCK released"); + first_err.map_or(Ok(()), Err) } fn is_running(&self) -> bool { @@ -714,7 +688,7 @@ impl StateMachine for RocksDBStateMachine { }) } - /// Thread-safe: called serially by single-task CommitHandler + /// Thread-safe: called serially by the state machine worker #[instrument(skip(self, chunk))] async fn apply_chunk( &self, @@ -1212,32 +1186,9 @@ impl StateMachine for RocksDBStateMachine { } impl Drop for RocksDBStateMachine { fn drop(&mut self) { - // If close_db() was already called, DB is None — nothing to flush or cancel. - if self.db.load().is_none() { - return; - } - - // last_applied durability no longer depends on ordering here — it is - // already atomic with the SM data as of apply_chunk(). flush() persists - // both the SM data and snapshot_metadata (via save_hard_state) to SST. - if let Err(e) = self.flush() { - error!("Failed to flush on drop: {}", e); - } else { - debug!("RocksDBStateMachine flushed successfully on drop"); - } - - // save_hard_state() only persists snapshot_metadata now — last_applied is - // already durable as of apply_chunk(), so there is no ordering hazard - // between this call and the flush() above. - if let Err(e) = self.save_hard_state() { - error!("Failed to save hard state on drop: {}", e); - } - - // Ensure flush operations are truly finished (no Arc clone needed — just &DB) - let guard = self.db.load(); - if let Some(db) = guard.as_deref() { - db.cancel_all_background_work(true); - debug!("RocksDB background work cancelled on drop"); + // Safety net for an owner that never called `close()`; a no-op if it already did. + if let Err(e) = StateMachine::close(self) { + error!("RocksDB state machine: close on drop failed: {e:?}"); } } } diff --git a/d-engine-server/src/storage/adaptors/rocksdb/rocksdb_state_machine_test.rs b/d-engine-server/src/storage/adaptors/rocksdb/rocksdb_state_machine_test.rs index f80be4f8..89533f38 100644 --- a/d-engine-server/src/storage/adaptors/rocksdb/rocksdb_state_machine_test.rs +++ b/d-engine-server/src/storage/adaptors/rocksdb/rocksdb_state_machine_test.rs @@ -1,17 +1,23 @@ -use super::RocksDBStorageEngine; -use super::RocksDBUnifiedEngine; -use crate::{Error, StateMachine}; +use std::sync::Arc; +use std::sync::Mutex; + use async_trait::async_trait; use bytes::Bytes; +use d_engine_core::ApplyEntry; +use d_engine_core::Command; +use d_engine_core::SnapshotApplyResult; use d_engine_core::file_io::compute_checksum_from_folder_path; -use d_engine_core::state_machine_test::{StateMachineBuilder, StateMachineTestSuite}; -use d_engine_core::{ApplyEntry, Command, SnapshotApplyResult}; +use d_engine_core::state_machine_test::StateMachineBuilder; +use d_engine_core::state_machine_test::StateMachineTestSuite; use d_engine_proto::common::LogId; use d_engine_proto::server::storage::SnapshotMetadata; -use std::sync::Arc; -use std::sync::Mutex; use tempfile::TempDir; +use super::RocksDBStorageEngine; +use super::RocksDBUnifiedEngine; +use crate::Error; +use crate::StateMachine; + struct RocksDBStateMachineBuilder { temp_dir: TempDir, // Partner storage kept alive alongside SM; must be dropped before the next build(). @@ -74,78 +80,6 @@ async fn test_rocksdb_state_machine_performance() { .expect("RocksDBStateMachine should pass performance tests"); } -/// Test that read operations are rejected during snapshot restoration -/// -/// This test verifies the fix for concurrent read safety during apply_snapshot_from_file(). -/// Without the is_serving check in get(), reads could access the temporary empty database -/// during PHASE 2.5, returning incorrect empty results instead of proper errors. -/// -/// Test flow: -/// 1. Create state machine with data -/// 2. Stop serving (simulating snapshot restoration PHASE 1) -/// 3. Attempt read operation -/// 4. Verify NotServing error is returned (not empty result) -/// 5. Resume serving -/// 6. Verify read succeeds -#[tokio::test] -async fn test_get_rejected_when_not_serving() { - let temp_dir = TempDir::new().expect("Failed to create temp dir"); - let db_path = temp_dir.path().join("test_not_serving"); - let (_storage, state_machine) = - RocksDBUnifiedEngine::open(&db_path).expect("Failed to open unified DB"); - - // Start state machine - state_machine.start().await.expect("Failed to start"); - - // Insert test data - let test_key = b"test_key"; - let test_value = b"test_value"; - - let entry = ApplyEntry { - index: 1, - term: 1, - command: Command::Insert { - key: Bytes::from(test_key.to_vec()), - value: Bytes::from(test_value.to_vec()), - ttl_secs: None, - }, - }; - - state_machine.apply_chunk(&[entry]).await.expect("Failed to apply entry"); - - // Verify data exists before stopping - let value = state_machine.get(test_key).expect("Failed to get"); - assert_eq!(value, Some(Bytes::from(test_value.to_vec()))); - - // Simulate snapshot restoration: stop serving - state_machine.stop().expect("Failed to stop"); - - // Attempt read while not serving - should return NotServing error - let result = state_machine.get(test_key); - assert!(result.is_err(), "Read should fail when not serving"); - - // Check the error chain for NotServing - let err = result.unwrap_err(); - let err_debug = format!("{err:?}"); // Use Debug format to see full error chain - assert!( - err_debug.contains("NotServing") - || err_debug.contains("not serving") - || err_debug.contains("restoring from snapshot"), - "Error should indicate not serving state, got: {err_debug}" - ); - - // Resume serving - state_machine.start().await.expect("Failed to restart"); - - // Read should succeed now - let value = state_machine.get(test_key).expect("Failed to get after resuming"); - assert_eq!( - value, - Some(Bytes::from(test_value.to_vec())), - "Data should be accessible after resuming service" - ); -} - // ── scan_prefix tests (#378) ────────────────────────────────────────────────── /// scan_prefix returns exactly the entries whose key starts with the given prefix, @@ -578,52 +512,55 @@ async fn test_get_multi_two_keys_still_uses_snapshot_isolation() { // ── close_db() tests ────────────────────────────────────────────────────────── // -// close_db() is a permanent, one-way shutdown that releases the RocksDB LOCK file -// immediately, without waiting for Arc to be the sole owner. It is separate -// from stop() because stop()/start() must remain a reversible cycle (used during -// snapshot restoration). +// close() is a permanent, one-way shutdown that releases the RocksDB LOCK file +// immediately, without waiting for Arc to be the sole owner (the developer may keep +// their own clone). -/// close_db() marks SM not-running and makes reads return an error. +/// close() marks SM not-running and makes reads return a NotServing error, never an empty result. #[tokio::test] -async fn test_close_db_prevents_reads() { +async fn test_close_prevents_reads() { let dir = TempDir::new().unwrap(); let (_storage, sm) = RocksDBUnifiedEngine::open(dir.path()).unwrap(); sm.start().await.unwrap(); sm.apply_chunk(&[insert_at(b"k", b"v", 1)]).await.unwrap(); assert_eq!(sm.get(b"k").unwrap(), Some(Bytes::from_static(b"v"))); - sm.close_db(); + sm.close().unwrap(); - assert!(!sm.is_running(), "is_running must be false after close_db"); - assert!(sm.get(b"k").is_err(), "get must fail after close_db"); + assert!(!sm.is_running(), "is_running must be false after close"); + let err = sm.get(b"k").expect_err("get must fail after close"); + assert!( + format!("{err:?}").contains("NotServing"), + "a closed state machine must say NotServing, got: {err:?}" + ); assert!( sm.get_multi(&[Bytes::from_static(b"k")]).is_err(), - "get_multi must fail after close_db" + "get_multi must fail after close" ); } -/// stop() after close_db() must succeed without panicking. -/// Node::stop() calls sm.stop() after EmbeddedEngine has already called close_db(); -/// that sequence must be safe. +/// close() is idempotent: the worker closes the state machine and the `Drop` safety net may +/// close it again, so a second call must be a harmless no-op. #[test] -fn test_stop_after_close_db_is_safe() { +fn test_close_twice_is_safe() { let dir = TempDir::new().unwrap(); let (_storage, sm) = RocksDBUnifiedEngine::open(dir.path()).unwrap(); - sm.close_db(); - assert!(sm.stop().is_ok(), "stop after close_db must not error"); + sm.close().unwrap(); + sm.close().unwrap(); + assert!(!sm.is_running()); } -/// Drop after close_db() must not panic or double-flush. -/// EmbeddedEngine::stop() calls close_db() before the Raft loop exits; the SM -/// is then dropped — Drop must detect the closed state and skip the flush. +/// Drop after close() must not panic or double-flush. +/// The worker closes the state machine as its last act; the SM is dropped later (possibly by a +/// developer who kept a clone) -- Drop must detect the closed state and skip the flush. #[tokio::test] -async fn test_drop_after_close_db_does_not_panic() { +async fn test_drop_after_close_does_not_panic() { let dir = TempDir::new().unwrap(); { let (_storage, sm) = RocksDBUnifiedEngine::open(dir.path()).unwrap(); sm.start().await.unwrap(); sm.apply_chunk(&[insert_at(b"k", b"v", 1)]).await.unwrap(); - sm.close_db(); + sm.close().unwrap(); // sm drops here — Drop must be a no-op } // reaching here without panic means the test passes @@ -823,7 +760,8 @@ async fn test_apply_chunk_delete_removes_key() { /// read-only DB via swap_db_for_test to reliably trigger the error. #[tokio::test] async fn test_apply_chunk_returns_error_on_read_only_db() { - use rocksdb::{DB, Options}; + use rocksdb::DB; + use rocksdb::Options; let dir = tempfile::TempDir::new().unwrap(); let db_path = dir.path().join("rocksdb"); diff --git a/d-engine-server/src/storage/adaptors/rocksdb/rocksdb_storage_engine.rs b/d-engine-server/src/storage/adaptors/rocksdb/rocksdb_storage_engine.rs index e16b16ef..e247bd7a 100644 --- a/d-engine-server/src/storage/adaptors/rocksdb/rocksdb_storage_engine.rs +++ b/d-engine-server/src/storage/adaptors/rocksdb/rocksdb_storage_engine.rs @@ -1,8 +1,10 @@ use std::ops::RangeInclusive; +use std::path::Path; use std::sync::Arc; use std::sync::atomic::AtomicU64; use std::sync::atomic::Ordering; +use async_trait::async_trait; use d_engine_core::Error; use d_engine_core::HardState; use d_engine_core::LogStore; @@ -12,16 +14,13 @@ use d_engine_core::StorageError; use d_engine_proto::common::Entry; use d_engine_proto::common::LogId; use prost::Message; -use rocksdb::WriteOptions; -use std::path::Path; - -use async_trait::async_trait; use rocksdb::Cache; use rocksdb::ColumnFamilyDescriptor; use rocksdb::DB; use rocksdb::Direction; use rocksdb::IteratorMode; use rocksdb::WriteBatch; +use rocksdb::WriteOptions; use tracing::instrument; use super::LOG_CF; diff --git a/d-engine-server/src/storage/adaptors/rocksdb/rocksdb_unified_engine_test.rs b/d-engine-server/src/storage/adaptors/rocksdb/rocksdb_unified_engine_test.rs index 2f7332e0..527d31ca 100644 --- a/d-engine-server/src/storage/adaptors/rocksdb/rocksdb_unified_engine_test.rs +++ b/d-engine-server/src/storage/adaptors/rocksdb/rocksdb_unified_engine_test.rs @@ -7,8 +7,10 @@ use d_engine_core::Lease; use d_engine_core::LogStore; use d_engine_core::StorageEngine; use d_engine_core::config::LeaseConfig; -use d_engine_core::state_machine_test::{StateMachineBuilder, StateMachineTestSuite}; -use d_engine_core::storage_engine_test::{StorageEngineBuilder, StorageEngineTestSuite}; +use d_engine_core::state_machine_test::StateMachineBuilder; +use d_engine_core::state_machine_test::StateMachineTestSuite; +use d_engine_core::storage_engine_test::StorageEngineBuilder; +use d_engine_core::storage_engine_test::StorageEngineTestSuite; use tempfile::TempDir; use super::RocksDBStateMachine; @@ -20,8 +22,8 @@ use crate::storage::TtlLease; // Both test suites call build() twice within a single persistence test (write data → drop → // build() again → verify data survived). Both builders therefore: // 1. Use a fixed path (`self.temp_dir`) so data persists across build() calls. -// 2. Drop the "partner" handle inside build() before reopening, to release the shared -// Arc and allow RocksDB to close before the next open. +// 2. Drop the "partner" handle inside build() before reopening, to release the shared Arc and +// allow RocksDB to close before the next open. // ── Storage engine suite via unified open ──────────────────────────────────── @@ -203,8 +205,8 @@ async fn test_persist_and_reload_lease_round_trip() { sm.set_lease(Arc::clone(&lease)); // Register a key with a long TTL so it survives the reload. lease.register(bytes::Bytes::from("persistent_key"), 3600); - // stop() calls persist_ttl_metadata() → writes TTL_STATE_KEY to RocksDB. - sm.stop().expect("stop"); + // close() calls persist_ttl_metadata() → writes TTL_STATE_KEY to RocksDB. + sm.close().unwrap(); // Ensure DB is flushed before drop. } // Wait for RocksDB to release the file lock. diff --git a/d-engine-server/src/storage/lease_integration_test.rs b/d-engine-server/src/storage/lease_integration_test.rs index 0ff1d60e..02e6030a 100644 --- a/d-engine-server/src/storage/lease_integration_test.rs +++ b/d-engine-server/src/storage/lease_integration_test.rs @@ -104,8 +104,10 @@ mod file_state_machine_tests { use std::time::Duration; use bytes::Bytes; + use d_engine_core::ApplyEntry; + use d_engine_core::Command; + use d_engine_core::StateMachine; use d_engine_core::file_io::compute_checksum_from_folder_path; - use d_engine_core::{ApplyEntry, Command, StateMachine}; use tempfile::TempDir; use tokio::time::sleep; @@ -190,10 +192,10 @@ mod file_state_machine_tests { // Create snapshot let snapshot_dir = temp_dir.path().join("snapshot"); - sm.generate_snapshot_data( - snapshot_dir.clone(), - d_engine_proto::common::LogId { index: 1, term: 1 }, - ) + sm.generate_snapshot_data(snapshot_dir.clone(), d_engine_proto::common::LogId { + index: 1, + term: 1, + }) .await .unwrap(); @@ -253,8 +255,8 @@ mod file_state_machine_tests { ); assert_eq!(sm.get(b"no_ttl_key").unwrap(), Some(Bytes::from("value3"))); - // Gracefully stop state machine to persist TTL data - sm.stop().unwrap(); + // Gracefully close state machine to persist TTL data + sm.close().unwrap(); // State machine drops here } @@ -412,16 +414,16 @@ mod file_state_machine_tests { "Permanent key should be restored from WAL" ); - sm.stop().unwrap(); + sm.close().unwrap(); } } /// WAL replay stops at the first truncated entry and discards the tail. /// - /// A WAL tail truncation is the expected outcome of a crash mid-write. Replay stops at the first incomplete entry rather - /// than continuing with garbage bytes or silently loading a partial entry with wrong TTL. - /// Complete entries before the truncation point are applied; the truncated entry and - /// anything after it are dropped. + /// A WAL tail truncation is the expected outcome of a crash mid-write. Replay stops at the + /// first incomplete entry rather than continuing with garbage bytes or silently loading a + /// partial entry with wrong TTL. Complete entries before the truncation point are applied; + /// the truncated entry and anything after it are dropped. #[tokio::test] async fn test_wal_replay_handles_incomplete_entries() { let temp_dir = TempDir::new().unwrap(); @@ -765,7 +767,9 @@ mod rocksdb_state_machine_tests { use std::time::Duration; use bytes::Bytes; - use d_engine_core::{ApplyEntry, Command, StateMachine}; + use d_engine_core::ApplyEntry; + use d_engine_core::Command; + use d_engine_core::StateMachine; use tempfile::TempDir; use tokio::time::sleep; @@ -868,10 +872,10 @@ mod rocksdb_state_machine_tests { // Create snapshot let snapshot_dir = temp_dir.path().join("snapshot"); - sm.generate_snapshot_data( - snapshot_dir.clone(), - d_engine_proto::common::LogId { index: 1, term: 1 }, - ) + sm.generate_snapshot_data(snapshot_dir.clone(), d_engine_proto::common::LogId { + index: 1, + term: 1, + }) .await .unwrap(); @@ -1043,8 +1047,8 @@ mod rocksdb_state_machine_tests { ); assert_eq!(sm.get(b"no_ttl_key").unwrap(), Some(Bytes::from("value3"))); - // Gracefully stop state machine to persist TTL data - sm.stop().unwrap(); + // Gracefully close state machine to persist TTL data + sm.close().unwrap(); // State machine drops here } diff --git a/d-engine-server/src/test_utils/integration/mod.rs b/d-engine-server/src/test_utils/integration/mod.rs index f7c2da0c..0a872a04 100644 --- a/d-engine-server/src/test_utils/integration/mod.rs +++ b/d-engine-server/src/test_utils/integration/mod.rs @@ -43,6 +43,8 @@ use std::path::PathBuf; use std::sync::Arc; use bytes::Bytes; +use d_engine_core::ApplyEntry; +use d_engine_core::Command; use d_engine_core::DefaultStateMachineHandler; use d_engine_core::ElectionHandler; use d_engine_core::LogSizePolicy; @@ -60,7 +62,6 @@ use d_engine_core::alias::SMOF; use d_engine_core::alias::TROF; use d_engine_core::generate_delete_commands; use d_engine_core::generate_insert_commands; -use d_engine_core::{ApplyEntry, Command}; use d_engine_proto::common::Entry; use d_engine_proto::common::EntryPayload; use d_engine_proto::common::NodeRole::Follower; diff --git a/d-engine-server/src/test_utils/mock/mock_node_builder.rs b/d-engine-server/src/test_utils/mock/mock_node_builder.rs index 124e204b..2100d39e 100644 --- a/d-engine-server/src/test_utils/mock/mock_node_builder.rs +++ b/d-engine-server/src/test_utils/mock/mock_node_builder.rs @@ -1,11 +1,9 @@ //! Mock node builder for fluent configuration of test Raft instances -use super::MockTypeConfig; -use crate::Node; -use crate::membership::MembershipSnapshot; -use crate::network::grpc; -use crate::node::DataDirLock; -use crate::node::LeaderNotifier; +use std::path::Path; +use std::sync::Arc; +use std::sync::atomic::AtomicBool; + use bytes::Bytes; use d_engine_core::ElectionConfig; use d_engine_core::InboundEvent; @@ -17,6 +15,7 @@ use d_engine_core::MockRaftLog; use d_engine_core::MockReplicationCore; use d_engine_core::MockStateMachine; use d_engine_core::MockStateMachineHandler; +use d_engine_core::MockStateMachineWriterOps; use d_engine_core::MockTransport; use d_engine_core::Raft; use d_engine_core::RaftConfig; @@ -27,21 +26,28 @@ use d_engine_core::RaftNodeConfig; use d_engine_core::RaftRole; use d_engine_core::RaftStorageHandles; use d_engine_core::SignalParams; +use d_engine_core::SmWorkerHandle; use d_engine_core::SnapshotApplyResult; use d_engine_core::StateMachine; +use d_engine_core::StateMachineCommand; +use d_engine_core::StateMachineWorker; use d_engine_core::follower_state::FollowerState; use d_engine_core::mock_membership as mock_membership_fn; use d_engine_proto::common::LogId; use d_engine_proto::server::cluster::ClusterMembership; -use std::path::Path; -use std::sync::Arc; -use std::sync::atomic::AtomicBool; use tokio::sync::Mutex; use tokio::sync::mpsc; use tokio::sync::watch; use tracing::error; use tracing::trace; +use super::MockTypeConfig; +use crate::Node; +use crate::membership::MembershipSnapshot; +use crate::network::grpc; +use crate::node::DataDirLock; +use crate::node::LeaderNotifier; + /// Builder for constructing mock Raft components with customizable defaults /// /// This builder provides a fluent API for creating test fixtures with @@ -110,6 +116,11 @@ pub struct MockBuilder { pub replication_handler: Option>, /// Optional state machine handler pub state_machine_handler: Option>>, + /// Optional state machine worker the built `Node` runs (defaults to an idle one) + pub sm_worker: Option>, + /// Queue of `sm_worker`: the Raft loop delivers committed entries here, so lifecycle tests + /// see them reach the worker's writer instead of a channel nobody reads + sm_queue: Option>, /// Optional node configuration pub node_config: Option, /// Optional election toggle (defaults to true) @@ -149,6 +160,8 @@ impl MockBuilder { election_handler: None, replication_handler: None, state_machine_handler: None, + sm_worker: None, + sm_queue: None, node_config: None, turn_on_election: None, shutdown_signal, @@ -241,7 +254,9 @@ impl MockBuilder { self.replication_handler.unwrap_or_else(mock_replication_handler), self.state_machine_handler .unwrap_or_else(|| Arc::new(mock_state_machine_handler())), - default_state_machine_commands(), + self.sm_queue + .map(d_engine_core::StateMachineCommandSender::new) + .unwrap_or_else(default_state_machine_commands), self.membership.unwrap_or_else(|| Arc::new(mock_membership_fn())), self.purge_executor.unwrap_or_else(mock_purge_exewcutor), self.node_config.unwrap_or_else(|| { @@ -315,8 +330,9 @@ impl MockBuilder { /// Build a Node instance (without RPC server) /// /// Wraps the Raft instance in a Node container for integration testing. - pub fn build_node(self) -> Node { + pub fn build_node(mut self) -> Node { let shutdown_signal = self.shutdown_signal.clone(); + let sm_worker = self.take_or_idle_sm_worker(); let raft = self.build_raft(); let event_tx = raft.event_sender(); let cmd_tx = raft.cmd_sender(); @@ -346,10 +362,9 @@ impl MockBuilder { watch_registry: None, #[cfg(feature = "watch")] _watch_dispatcher_handle: None, - sm_worker_handle: std::sync::Mutex::new(None), + sm_worker, rpc_server_handle: std::sync::Mutex::new(None), read_actor_handle: std::sync::Mutex::new(None), - _commit_handler_handle: None, _lease_cleanup_handle: None, shutdown_signal, read_lease, @@ -362,9 +377,10 @@ impl MockBuilder { /// /// This variant starts the RPC server in a background task, /// useful for end-to-end testing with network communication. - pub fn build_node_with_rpc_server(self) -> Arc> { + pub fn build_node_with_rpc_server(mut self) -> Arc> { let shutdown = self.shutdown_signal.clone(); let node_config_option = self.node_config.clone(); + let sm_worker = self.take_or_idle_sm_worker(); let raft = self.build_raft(); let event_tx = raft.event_sender(); @@ -405,10 +421,9 @@ impl MockBuilder { watch_registry: None, #[cfg(feature = "watch")] _watch_dispatcher_handle: None, - sm_worker_handle: std::sync::Mutex::new(None), + sm_worker, rpc_server_handle: std::sync::Mutex::new(None), read_actor_handle: std::sync::Mutex::new(None), - _commit_handler_handle: None, _lease_cleanup_handle: None, shutdown_signal: shutdown.clone(), read_lease, @@ -512,6 +527,30 @@ impl MockBuilder { self } + /// Set the state machine worker the built `Node` starts and stops in `run()`, together with + /// its queue (the second value of `sm_worker_with_writer`), which the Raft loop delivers + /// committed entries to. + pub fn with_sm_worker( + mut self, + sm_worker: SmWorkerHandle, + queue: mpsc::UnboundedSender, + ) -> Self { + self.sm_worker = Some(sm_worker); + self.sm_queue = Some(queue); + self + } + + /// The worker the node runs: the one supplied, or an idle one whose queue is wired into the + /// Raft loop the same way. + fn take_or_idle_sm_worker(&mut self) -> SmWorkerHandle { + if let Some(sm_worker) = self.sm_worker.take() { + return sm_worker; + } + let (sm_worker, queue) = idle_sm_worker(); + self.sm_queue = Some(queue); + sm_worker + } + /// Set custom node configuration pub fn with_node_config( mut self, @@ -589,7 +628,7 @@ pub(crate) fn mock_state_machine() -> MockStateMachine { let mut mock = MockStateMachine::new(); mock.expect_start().returning(|| Ok(())); - mock.expect_stop().returning(|| Ok(())); + mock.expect_close().returning(|| Ok(())); mock.expect_is_running().returning(|| true); mock.expect_get().returning(|_| Ok(None)); @@ -622,21 +661,51 @@ pub(crate) fn mock_state_machine() -> MockStateMachine { pub(crate) fn mock_state_machine_handler() -> MockStateMachineHandler { let mut state_machine_handler = MockStateMachineHandler::new(); state_machine_handler.expect_update_pending().returning(|_| {}); + // Commit delivery (also run on shutdown) reads the applied position every iteration. + state_machine_handler.expect_last_applied().returning(|| 0); state_machine_handler.expect_read_from_state_machine().returning(|_| None); state_machine_handler.expect_should_snapshot().returning(|_| false); state_machine_handler } -/// Default `StateMachineCommandSender` for tests that don't care about it. No production -/// call site sends through this yet (#436), so the receiver is simply -/// dropped rather than drained by a spawned task — spawning unconditionally here broke -/// every non-`#[tokio::test]` caller of `MockBuilder::build_context()`/`build_raft()` -/// with "there is no reactor running" (`tokio::spawn` requires an active runtime). +/// Default `StateMachineCommandSender` for tests that don't care about it. The raft loop hands +/// committed entries to it, so the receiver must stay open or delivery fails with +/// `WorkerUnavailable` and the loop exits. Nothing drains it: a spawned drain task would break +/// every non-`#[tokio::test]` caller of `MockBuilder::build_context()`/`build_raft()` with +/// "there is no reactor running", so the receiver is leaked instead (test-only, tiny). pub(crate) fn default_state_machine_commands() -> d_engine_core::StateMachineCommandSender { - let (tx, _rx) = mpsc::unbounded_channel(); + let (tx, rx) = mpsc::unbounded_channel(); + std::mem::forget(rx); d_engine_core::StateMachineCommandSender::new(tx) } +/// A real state machine worker around `writer`, not started yet. The second value is a sender of +/// its queue, for tests to feed it the way the Raft loop would. +pub(crate) fn sm_worker_with_writer( + mut writer: MockStateMachineWriterOps +) -> ( + SmWorkerHandle, + mpsc::UnboundedSender, +) { + // The worker (or the handle, if never started) closes the state machine. Added last, so a + // test's own `expect_close` keeps precedence. + writer.expect_close().returning(|| Ok(())); + let (tx, rx) = mpsc::unbounded_channel(); + let (events_tx, events_rx) = mpsc::unbounded_channel(); + // Apply results must have somewhere to go; leaked like `default_state_machine_commands`. + std::mem::forget(events_rx); + let worker = StateMachineWorker::new(1, Arc::new(writer), tx.clone(), rx, events_tx); + (SmWorkerHandle::new(worker, tx.clone()), tx) +} + +/// The worker mock nodes get unless a test supplies one: its writer expects no apply at all. +pub(crate) fn idle_sm_worker() -> ( + SmWorkerHandle, + mpsc::UnboundedSender, +) { + sm_worker_with_writer(MockStateMachineWriterOps::new()) +} + /// Create a mock purge executor pub(crate) fn mock_purge_exewcutor() -> MockPurgeExecutor { let mut purge_exewcutor = MockPurgeExecutor::new(); diff --git a/d-engine-server/src/test_utils/mock/mock_rpc_service.rs b/d-engine-server/src/test_utils/mock/mock_rpc_service.rs index 1087e151..154967d1 100644 --- a/d-engine-server/src/test_utils/mock/mock_rpc_service.rs +++ b/d-engine-server/src/test_utils/mock/mock_rpc_service.rs @@ -13,7 +13,6 @@ use d_engine_proto::server::election::VoteResponse; use d_engine_proto::server::election::raft_election_service_server::RaftElectionServiceServer; use d_engine_proto::server::replication::AppendEntriesResponse; use d_engine_proto::server::replication::raft_replication_service_server::RaftReplicationServiceServer; - use d_engine_proto::server::storage::SnapshotResponse; use d_engine_proto::server::storage::snapshot_service_server::SnapshotServiceServer; use tokio::net::TcpListener; @@ -25,11 +24,10 @@ use tonic_health::server::health_reporter; use tracing::debug; use tracing::info; +use super::mock_rpc::MockRpcService; use crate::Result; use crate::StorageError; -use super::mock_rpc::MockRpcService; - pub struct MockNode; impl MockNode { diff --git a/d-engine-server/src/test_utils/mock/mod.rs b/d-engine-server/src/test_utils/mock/mod.rs index 56d74ef8..92325f47 100644 --- a/d-engine-server/src/test_utils/mock/mod.rs +++ b/d-engine-server/src/test_utils/mock/mod.rs @@ -10,13 +10,14 @@ mod mock_rpc_service; mod snapshot; mod stream; +use d_engine_core::MockTypeConfig; +use d_engine_core::Raft; +use d_engine_core::node_config; +use d_engine_proto::server::cluster::NodeMeta; pub use mock_node_builder::*; pub use mock_rpc::*; pub use mock_rpc_service::*; pub use snapshot::*; - -use d_engine_core::{MockTypeConfig, Raft, node_config}; -use d_engine_proto::server::cluster::NodeMeta; use tokio::sync::watch; use crate::Node; diff --git a/d-engine-server/tests/cas_operations/bank_transfer_invariant_embedded.rs b/d-engine-server/tests/cas_operations/bank_transfer_invariant_embedded.rs index af577dbe..02e5a3a1 100644 --- a/d-engine-server/tests/cas_operations/bank_transfer_invariant_embedded.rs +++ b/d-engine-server/tests/cas_operations/bank_transfer_invariant_embedded.rs @@ -1,12 +1,13 @@ #![cfg(feature = "rocksdb")] +use std::sync::Arc; +use std::time::Duration; + use d_engine_core::ClientApi; use d_engine_core::ClientApiError; use d_engine_server::RocksDBUnifiedEngine; use d_engine_server::api::DefaultEmbeddedEngine; use serial_test::serial; -use std::sync::Arc; -use std::time::Duration; use tracing::info; use tracing_test::traced_test; diff --git a/d-engine-server/tests/cas_operations/distributed_lock_embedded.rs b/d-engine-server/tests/cas_operations/distributed_lock_embedded.rs index 701a1ff0..ddcc04c6 100644 --- a/d-engine-server/tests/cas_operations/distributed_lock_embedded.rs +++ b/d-engine-server/tests/cas_operations/distributed_lock_embedded.rs @@ -1,10 +1,9 @@ #![cfg(feature = "rocksdb")] -use d_engine_server::RocksDBUnifiedEngine; - use std::sync::Arc; use std::time::Duration; use d_engine_core::ClientApi; +use d_engine_server::RocksDBUnifiedEngine; use d_engine_server::api::DefaultEmbeddedEngine; use tracing::info; use tracing_test::traced_test; diff --git a/d-engine-server/tests/cas_operations/leader_failover_cas_embedded.rs b/d-engine-server/tests/cas_operations/leader_failover_cas_embedded.rs index 6d31cad7..efab88d9 100644 --- a/d-engine-server/tests/cas_operations/leader_failover_cas_embedded.rs +++ b/d-engine-server/tests/cas_operations/leader_failover_cas_embedded.rs @@ -1,11 +1,11 @@ #![cfg(feature = "rocksdb")] -use d_engine_server::RocksDBUnifiedEngine; +use std::sync::Arc; +use std::time::Duration; use d_engine_core::ClientApi; +use d_engine_server::RocksDBUnifiedEngine; use d_engine_server::api::DefaultEmbeddedEngine; use serial_test::serial; -use std::sync::Arc; -use std::time::Duration; use tracing::info; use tracing_test::traced_test; @@ -15,9 +15,6 @@ use crate::common::wait_for_new_leader; /// Test CAS operation behavior during leader failover (Embedded mode) /// -/// Based on etcd's TestTxnWriteFail pattern: -/// https://github.com/etcd-io/etcd/blob/main/tests/integration/clientv3/txn_test.go -/// /// Scenario: /// 1. Start 3-node cluster, elect leader /// 2. Client A sends CAS request to acquire lock diff --git a/d-engine-server/tests/cas_operations/leader_failover_cas_standalone.rs b/d-engine-server/tests/cas_operations/leader_failover_cas_standalone.rs index d832a514..c76133ed 100644 --- a/d-engine-server/tests/cas_operations/leader_failover_cas_standalone.rs +++ b/d-engine-server/tests/cas_operations/leader_failover_cas_standalone.rs @@ -1,8 +1,9 @@ +use std::time::Duration; + use d_engine_client::Client; use d_engine_core::ClientApi; use d_engine_core::ClientApiError; use d_engine_core::client::ErrorCode; -use std::time::Duration; use tempfile::TempDir; use tracing::info; use tracing_test::traced_test; @@ -19,10 +20,6 @@ use crate::common::wait_for_stable_leader; /// Test CAS operation behavior during leader failover (Standalone/gRPC mode) /// -/// Based on etcd's TestTxnWriteFail and TestFailover patterns: -/// - https://github.com/etcd-io/etcd/blob/main/tests/integration/clientv3/txn_test.go -/// - https://github.com/etcd-io/etcd/blob/main/tests/integration/v3_failover_test.go -/// /// Scenario: /// 1. Start 3-node cluster via gRPC, elect leader /// 2. Client A sends CAS request to acquire lock diff --git a/d-engine-server/tests/cas_operations/snapshot_recovery_embedded.rs b/d-engine-server/tests/cas_operations/snapshot_recovery_embedded.rs index 280299a6..d6207f98 100644 --- a/d-engine-server/tests/cas_operations/snapshot_recovery_embedded.rs +++ b/d-engine-server/tests/cas_operations/snapshot_recovery_embedded.rs @@ -1,10 +1,10 @@ #![cfg(feature = "rocksdb")] -use d_engine_server::RocksDBUnifiedEngine; +use std::sync::Arc; +use std::time::Duration; use d_engine_core::ClientApi; +use d_engine_server::RocksDBUnifiedEngine; use d_engine_server::api::DefaultEmbeddedEngine; -use std::sync::Arc; -use std::time::Duration; use tracing::info; use tracing_test::traced_test; diff --git a/d-engine-server/tests/cas_operations/snapshot_recovery_standalone.rs b/d-engine-server/tests/cas_operations/snapshot_recovery_standalone.rs index ca55e920..0cf3e523 100644 --- a/d-engine-server/tests/cas_operations/snapshot_recovery_standalone.rs +++ b/d-engine-server/tests/cas_operations/snapshot_recovery_standalone.rs @@ -1,7 +1,9 @@ +use std::time::Duration; + use d_engine_client::Client; use d_engine_core::ClientApi; use d_engine_core::ClientApiError; -use std::time::Duration; +use tempfile::TempDir; use tracing::info; use tracing_test::traced_test; @@ -15,7 +17,6 @@ use crate::common::get_available_ports; use crate::common::node_config; use crate::common::start_node; use crate::common::wait_for_stable_leader; -use tempfile::TempDir; /// Test CAS state survives snapshot and recovery (standalone mode) /// diff --git a/d-engine-server/tests/client_manager/mod.rs b/d-engine-server/tests/client_manager/mod.rs index 7bd370fc..d9021eb7 100644 --- a/d-engine-server/tests/client_manager/mod.rs +++ b/d-engine-server/tests/client_manager/mod.rs @@ -2,8 +2,6 @@ use std::time::Duration; -use crate::common::ClientCommands; -use crate::common::{self}; use d_engine_client::Client; use d_engine_client::ClientBuilder; use d_engine_core::ClientApi; @@ -18,6 +16,9 @@ use tracing::debug; use tracing::error; use tracing::info; +use crate::common::ClientCommands; +use crate::common::{self}; + const MAX_RETRIES: u32 = 10; const RETRY_DELAY_MS: u64 = 100; diff --git a/d-engine-server/tests/cluster_lifecycle/config_loading_embedded.rs b/d-engine-server/tests/cluster_lifecycle/config_loading_embedded.rs index 321184f1..26d505ae 100644 --- a/d-engine-server/tests/cluster_lifecycle/config_loading_embedded.rs +++ b/d-engine-server/tests/cluster_lifecycle/config_loading_embedded.rs @@ -2,10 +2,11 @@ //! //! Verifies the bug fix for start_with() failing when CONFIG_PATH env is not set. -use d_engine_server::DefaultEmbeddedEngine; use std::io::Write; use std::time::Duration; +use d_engine_server::DefaultEmbeddedEngine; + #[tokio::test] async fn test_start_with_no_config_path_env() { // Clear CONFIG_PATH to reproduce the bug scenario diff --git a/d-engine-server/tests/cluster_lifecycle/mod.rs b/d-engine-server/tests/cluster_lifecycle/mod.rs index 2ba6080b..63eb0952 100644 --- a/d-engine-server/tests/cluster_lifecycle/mod.rs +++ b/d-engine-server/tests/cluster_lifecycle/mod.rs @@ -5,24 +5,24 @@ //! //! ## Business Scenarios Covered //! -//! 1. **Single-Node Bootstrap (Embedded)**: Verify a standalone node can self-elect as leader -//! and accept KV operations immediately after startup -//! 2. **Multi-Node Cluster Formation (Embedded)**: Form a 3-node cluster with automatic -//! leader election among nodes +//! 1. **Single-Node Bootstrap (Embedded)**: Verify a standalone node can self-elect as leader and +//! accept KV operations immediately after startup +//! 2. **Multi-Node Cluster Formation (Embedded)**: Form a 3-node cluster with automatic leader +//! election among nodes //! 3. **Multi-Node Cluster Formation (Standalone)**: Same via gRPC for remote nodes //! 4. **Cluster Scaling (1→3 nodes)**: Seamlessly transition from single-node to 3-node cluster //! without data loss or downtime -//! 5. **Configuration Loading**: Parse and apply cluster configuration from TOML files, -//! supporting both single-node and multi-node modes +//! 5. **Configuration Loading**: Parse and apply cluster configuration from TOML files, supporting +//! both single-node and multi-node modes //! //! ## Test Files //! //! - `single_node_bootstrap_embedded.rs` - Single-node lifecycle and self-election //! - `cluster_bootstrap_embedded.rs` - Multi-node cluster bootstrap (embedded) - ❌ MISSING -//! - `cluster_bootstrap_standalone.rs` - Multi-node cluster bootstrap (standalone/gRPC) - ❌ MISSING +//! - `cluster_bootstrap_standalone.rs` - Multi-node cluster bootstrap (standalone/gRPC) - ❌ +//! MISSING //! - `scale_single_to_three_node_embedded.rs` - Scaling without data loss //! - `config_loading_embedded.rs` - Configuration file parsing and validation -//! // mod cluster_bootstrap_embedded; // mod cluster_bootstrap_standalone; diff --git a/d-engine-server/tests/cluster_lifecycle/scale_single_to_three_node_embedded.rs b/d-engine-server/tests/cluster_lifecycle/scale_single_to_three_node_embedded.rs index 19f3c952..3f54a630 100644 --- a/d-engine-server/tests/cluster_lifecycle/scale_single_to_three_node_embedded.rs +++ b/d-engine-server/tests/cluster_lifecycle/scale_single_to_three_node_embedded.rs @@ -1,8 +1,8 @@ -use d_engine_server::RocksDBUnifiedEngine; use std::sync::Arc; use std::time::Duration; use d_engine_server::DefaultEmbeddedEngine; +use d_engine_server::RocksDBUnifiedEngine; use tracing::info; use tracing_test::traced_test; @@ -14,7 +14,8 @@ use crate::common::retry_until; /// /// Scenario: /// 1. Start single node, write data -/// 2. Start another two nodes and join node 1 as cluster. Node 1 should remain as Leader, no data lost +/// 2. Start another two nodes and join node 1 as cluster. Node 1 should remain as Leader, no data +/// lost /// 3. Verify cluster healthy and data preserved #[tokio::test] #[traced_test] diff --git a/d-engine-server/tests/cluster_lifecycle/single_node_bootstrap_embedded.rs b/d-engine-server/tests/cluster_lifecycle/single_node_bootstrap_embedded.rs index 9b8449a1..84a2647b 100644 --- a/d-engine-server/tests/cluster_lifecycle/single_node_bootstrap_embedded.rs +++ b/d-engine-server/tests/cluster_lifecycle/single_node_bootstrap_embedded.rs @@ -1,5 +1,6 @@ -use d_engine_server::DefaultEmbeddedEngine; use std::time::Duration; + +use d_engine_server::DefaultEmbeddedEngine; use tracing_test::traced_test; /// Test single-node DefaultEmbeddedEngine basic lifecycle diff --git a/d-engine-server/tests/cluster_membership/join_cluster_concurrent_joins_standalone.rs b/d-engine-server/tests/cluster_membership/join_cluster_concurrent_joins_standalone.rs index bac42d00..1baa39e2 100644 --- a/d-engine-server/tests/cluster_membership/join_cluster_concurrent_joins_standalone.rs +++ b/d-engine-server/tests/cluster_membership/join_cluster_concurrent_joins_standalone.rs @@ -56,7 +56,7 @@ use crate::common::test_put_get; /// 11) and verifies that all nodes, including the newly joined ones, successfully replicate and /// persist the data to disk. This ensures: /// - Raft log replication across all 5 nodes -/// - Commit handler processing on all nodes +/// - Commit delivery on all nodes /// - State machine persistence (FileStateMachine.persist_data_async()) /// /// 5. **Cluster Consistency**: Final assertions verify: @@ -319,14 +319,14 @@ async fn test_join_cluster_scenario2() -> Result<(), ClientApiError> { let mut client_manager = ClientManager::new(&create_bootstrap_urls(&all_ports)).await?; - // Insert a new entry to trigger commit handler and ensure all nodes are in sync + // Insert a new entry to trigger commit delivery and ensure all nodes are in sync test_put_get(&mut client_manager, 11, 200).await?; // Wait for all nodes to complete replication and persistence // This ensures entry 11 has been: // 1. Replicated from leader to all followers via AppendEntries RPC // 2. Committed by the leader once majority acknowledgment received - // 3. Applied to state machine on all nodes via commit handler + // 3. Applied to state machine on all nodes via commit delivery // 4. Persisted to disk via FileStateMachine.persist_data_async() sleep(Duration::from_secs(2)).await; diff --git a/d-engine-server/tests/cluster_membership/mod.rs b/d-engine-server/tests/cluster_membership/mod.rs index 4e9d19b3..20e11c51 100644 --- a/d-engine-server/tests/cluster_membership/mod.rs +++ b/d-engine-server/tests/cluster_membership/mod.rs @@ -4,8 +4,10 @@ //! //! ## Test Coverage //! -//! - `join_cluster_single_join_standalone.rs` - Single node joins existing cluster (standalone mode) -//! - `join_cluster_concurrent_joins_standalone.rs` - Multiple nodes join simultaneously (standalone mode) +//! - `join_cluster_single_join_standalone.rs` - Single node joins existing cluster (standalone +//! mode) +//! - `join_cluster_concurrent_joins_standalone.rs` - Multiple nodes join simultaneously (standalone +//! mode) mod join_cluster_concurrent_joins_standalone; mod join_cluster_single_join_standalone; diff --git a/d-engine-server/tests/cluster_state_and_metadata/cluster_state_consensus_embedded.rs b/d-engine-server/tests/cluster_state_and_metadata/cluster_state_consensus_embedded.rs index c9d6bd2b..d80af582 100644 --- a/d-engine-server/tests/cluster_state_and_metadata/cluster_state_consensus_embedded.rs +++ b/d-engine-server/tests/cluster_state_and_metadata/cluster_state_consensus_embedded.rs @@ -5,14 +5,15 @@ //! - Leader failover //! - Cluster view consistency -use crate::common::get_available_ports; -use d_engine_server::DefaultEmbeddedEngine; use std::sync::Arc; use std::time::Duration; + +use d_engine_server::DefaultEmbeddedEngine; +use d_engine_server::RocksDBUnifiedEngine; use tracing::info; use tracing_test::traced_test; -use d_engine_server::RocksDBUnifiedEngine; +use crate::common::get_available_ports; /// Test: 3-node cluster should elect exactly one leader /// diff --git a/d-engine-server/tests/cluster_state_and_metadata/mod.rs b/d-engine-server/tests/cluster_state_and_metadata/mod.rs index cf315e60..fec6c323 100644 --- a/d-engine-server/tests/cluster_state_and_metadata/mod.rs +++ b/d-engine-server/tests/cluster_state_and_metadata/mod.rs @@ -4,7 +4,8 @@ //! //! ## Test Coverage //! -//! - `cluster_state_consensus_embedded.rs` - Leader uniqueness verification and all-nodes consensus on leader identity +//! - `cluster_state_consensus_embedded.rs` - Leader uniqueness verification and all-nodes consensus +//! on leader identity //! - `metadata_api_standalone.rs` - Metadata API queries via gRPC mod cluster_state_consensus_embedded; diff --git a/d-engine-server/tests/common/mod.rs b/d-engine-server/tests/common/mod.rs index f82e8cd9..1a0398f4 100644 --- a/d-engine-server/tests/common/mod.rs +++ b/d-engine-server/tests/common/mod.rs @@ -1,5 +1,10 @@ #![allow(dead_code)] +use std::path::Path; +use std::path::PathBuf; +use std::sync::Arc; +use std::time::Duration; + use bytes::Bytes; use bytes::BytesMut; use config::Config; @@ -30,10 +35,6 @@ use d_engine_server::StandaloneEngine; use d_engine_server::StorageEngine; use d_engine_server::node::RaftTypeConfig; use prost::Message; -use std::path::Path; -use std::path::PathBuf; -use std::sync::Arc; -use std::time::Duration; use tokio::fs::remove_dir_all; use tokio::fs::{self}; use tokio::net::TcpStream; diff --git a/d-engine-server/tests/consistent_reads/lease_read_embedded.rs b/d-engine-server/tests/consistent_reads/lease_read_embedded.rs index b1e6a61d..1c0c534b 100644 --- a/d-engine-server/tests/consistent_reads/lease_read_embedded.rs +++ b/d-engine-server/tests/consistent_reads/lease_read_embedded.rs @@ -4,7 +4,6 @@ //! - Lease Read with valid lease (local read, no quorum) //! - Lease Read consistency across sequential writes //! - Lease Read vs Linearizable Read comparison -//! use std::time::Duration; diff --git a/d-engine-server/tests/consistent_reads/linearizable_read_batching_embedded.rs b/d-engine-server/tests/consistent_reads/linearizable_read_batching_embedded.rs index ffa438aa..2e39e92a 100644 --- a/d-engine-server/tests/consistent_reads/linearizable_read_batching_embedded.rs +++ b/d-engine-server/tests/consistent_reads/linearizable_read_batching_embedded.rs @@ -11,7 +11,6 @@ use bytes::Bytes; use d_engine_server::DefaultEmbeddedEngine; use tempfile::TempDir; use tokio::time::Instant; - use tracing_test::traced_test; use crate::common::get_available_ports; diff --git a/d-engine-server/tests/consistent_reads/linearizable_read_consistency_embedded.rs b/d-engine-server/tests/consistent_reads/linearizable_read_consistency_embedded.rs index 442b5e94..f970356f 100644 --- a/d-engine-server/tests/consistent_reads/linearizable_read_consistency_embedded.rs +++ b/d-engine-server/tests/consistent_reads/linearizable_read_consistency_embedded.rs @@ -5,15 +5,18 @@ //! //! Uses embedded mode with DefaultEmbeddedEngine for production-ready testing. -use crate::common::{create_node_config, get_available_ports}; -use d_engine_server::DefaultEmbeddedEngine; -use d_engine_server::RocksDBUnifiedEngine; use std::sync::Arc; use std::time::Duration; + +use d_engine_server::DefaultEmbeddedEngine; +use d_engine_server::RocksDBUnifiedEngine; use tempfile::TempDir; use tokio::time::Instant; use tracing_test::traced_test; +use crate::common::create_node_config; +use crate::common::get_available_ports; + /// Helper to create a test EmbeddedEngine async fn create_test_engine(test_name: &str) -> (DefaultEmbeddedEngine, TempDir) { let temp_dir = tempfile::tempdir().expect("Failed to create temp dir"); diff --git a/d-engine-server/tests/drain_batching/select_fairness_embedded.rs b/d-engine-server/tests/drain_batching/select_fairness_embedded.rs index 056efb97..79b5254c 100644 --- a/d-engine-server/tests/drain_batching/select_fairness_embedded.rs +++ b/d-engine-server/tests/drain_batching/select_fairness_embedded.rs @@ -8,7 +8,8 @@ //! Test plan: 001_todo_raft_test_plan_2215.md P3-2 use std::sync::Arc; -use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::atomic::AtomicU64; +use std::sync::atomic::Ordering; use std::time::Duration; use d_engine_server::DefaultEmbeddedEngine; diff --git a/d-engine-server/tests/embedded_client/mod.rs b/d-engine-server/tests/embedded_client/mod.rs index b964837f..8044c44f 100644 --- a/d-engine-server/tests/embedded_client/mod.rs +++ b/d-engine-server/tests/embedded_client/mod.rs @@ -6,8 +6,8 @@ //! //! ## Test Files //! -//! - `embedded_client_operations.rs` - EmbeddedClient API testing: -//! put, put_with_ttl, get (eventual/linearizable/lease), get_multi alignment, -//! delete, concurrent ops, watch (put/delete events, key isolation) +//! - `embedded_client_operations.rs` - EmbeddedClient API testing: put, put_with_ttl, get +//! (eventual/linearizable/lease), get_multi alignment, delete, concurrent ops, watch (put/delete +//! events, key isolation) mod embedded_client_operations; diff --git a/d-engine-server/tests/failover_and_recovery/leader_failover_embedded.rs b/d-engine-server/tests/failover_and_recovery/leader_failover_embedded.rs index 7ee7c24d..8a32b1cc 100644 --- a/d-engine-server/tests/failover_and_recovery/leader_failover_embedded.rs +++ b/d-engine-server/tests/failover_and_recovery/leader_failover_embedded.rs @@ -522,8 +522,8 @@ async fn test_minority_failure_blocks_writes() -> Result<(), Box= 6) +/// 4. The leader's term must be small: without PreVote node 3 would have raised its term once per +/// round and the whole cluster would have adopted it (term >= 6) #[tokio::test] #[traced_test] async fn test_isolated_node_does_not_inflate_term() -> Result<(), Box> { diff --git a/d-engine-server/tests/failover_and_recovery/mod.rs b/d-engine-server/tests/failover_and_recovery/mod.rs index 93eb617a..8e56b6e6 100644 --- a/d-engine-server/tests/failover_and_recovery/mod.rs +++ b/d-engine-server/tests/failover_and_recovery/mod.rs @@ -5,17 +5,18 @@ //! //! ## Business Scenarios Covered //! -//! 1. **Leader Failover (Embedded)**: Kill the leader node in a 3-node cluster and verify -//! automatic re-election of a new leader, data consistency, and cluster operational status +//! 1. **Leader Failover (Embedded)**: Kill the leader node in a 3-node cluster and verify automatic +//! re-election of a new leader, data consistency, and cluster operational status //! 2. **Leader Failover (Standalone)**: Same scenario via gRPC for remote nodes -//! 3. **Node Rejoin (Embedded)**: Kill a follower, verify cluster still works with 2/3 nodes, -//! then restart the follower and confirm it syncs all missing data +//! 3. **Node Rejoin (Embedded)**: Kill a follower, verify cluster still works with 2/3 nodes, then +//! restart the follower and confirm it syncs all missing data //! 4. **Minority Failure Blocks Writes (Both Modes)**: Kill 2 out of 3 nodes to lose majority, //! verify that remaining node cannot serve writes (Raft safety property) //! //! ## Raft Protocol Robustness //! -//! - **Quorum-Based Consensus**: Leader requires majority (2/3) of nodes to confirm before committing +//! - **Quorum-Based Consensus**: Leader requires majority (2/3) of nodes to confirm before +//! committing //! - **Election Timeout**: Followers detect leader failure and trigger new election //! - **New Leader Confirmation**: Verify new leader's committed entries match previous leader //! - **No Data Loss on Failover**: All committed entries survive leader failure @@ -52,7 +53,8 @@ //! ## Test Files (Current Structure) //! //! **Phase 1 (Current)**: -//! - `leader_failover_embedded.rs` - COMBINED: test_embedded_leader_failover() + test_embedded_node_rejoin() + test_minority_failure_blocks_writes() +//! - `leader_failover_embedded.rs` - COMBINED: test_embedded_leader_failover() + +//! test_embedded_node_rejoin() + test_minority_failure_blocks_writes() //! - `leader_failover_standalone.rs` - COMBINED: test_3_node_failover() + test_minority_failure() //! //! **Phase 2 (Planned Refactor)**: diff --git a/d-engine-server/tests/leader_election/leader_election_log_term_index_standalone.rs b/d-engine-server/tests/leader_election/leader_election_log_term_index_standalone.rs index bfc32d5b..c7a712c8 100644 --- a/d-engine-server/tests/leader_election/leader_election_log_term_index_standalone.rs +++ b/d-engine-server/tests/leader_election/leader_election_log_term_index_standalone.rs @@ -9,8 +9,8 @@ //! 2. Node A appends 10 log entries with Term=2. //! 3. Node B appends 8 log entries with Term=3 (higher term). //! 4. Node C is a new node with no logs. -//! 5. Trigger a leader election (all three nodes use the same default randomized timeout — -//! the rejection is no longer sourced from this race; see below). +//! 5. Trigger a leader election (all three nodes use the same default randomized timeout — the +//! rejection is no longer sourced from this race; see below). //! //! Expected Result: //! @@ -26,9 +26,9 @@ //! rejection organically: whichever of A/C's randomized timers fired first would ask node B for //! a vote and (having a worse log) get rejected. Two problems with that: //! -//! 1. It's not guaranteed. If node B's own timer happens to fire first, it becomes candidate -//! itself and never receives a vote request to reject — the assertion would then have nothing -//! to observe, purely due to timer luck (observed in CI: ~1/9 runs). +//! 1. It's not guaranteed. If node B's own timer happens to fire first, it becomes candidate itself +//! and never receives a vote request to reject — the assertion would then have nothing to +//! observe, purely due to timer luck (observed in CI: ~1/9 runs). //! 2. An earlier attempt to fix (1) by giving node B a much longer timeout than A/C — forcing A/C //! to go first — backfired: A and C's logs disagree with each other too (A has no entries in //! common with C's empty log, different terms), and they ended up in a prolonged split-vote @@ -46,6 +46,13 @@ //! The natural election (who the cluster actually elects) is untouched and still runs exactly as //! before. +use std::time::Duration; + +use d_engine_core::ClientApiError; +use d_engine_proto::server::election::VoteRequest; +use d_engine_proto::server::election::raft_election_service_client::RaftElectionServiceClient; +use tracing::debug; + use crate::client_manager::ClientManager; use crate::common::TestContext; use crate::common::WAIT_FOR_NODE_READY_IN_SEC; @@ -59,11 +66,6 @@ use crate::common::node_config; use crate::common::prepare_storage_engine; use crate::common::reset; use crate::common::start_node; -use d_engine_core::ClientApiError; -use d_engine_proto::server::election::VoteRequest; -use d_engine_proto::server::election::raft_election_service_client::RaftElectionServiceClient; -use std::time::Duration; -use tracing::debug; // Constants for test configuration const ELECTION_CASE1_DIR: &str = "election/case1"; @@ -138,10 +140,11 @@ async fn test_leader_election_based_on_log_term_and_index() -> Result<(), Client .map_err(|e| tonic::Status::unavailable(e.to_string()))?; client .request_vote(tonic::Request::new(VoteRequest { - term: 10, // comfortably above any term the natural election reaches here - candidate_id: 99, // synthetic candidate — not a real cluster member + term: 10, /* comfortably above any term the natural election + * reaches here */ + candidate_id: 99, // synthetic candidate — not a real cluster member last_log_index: 10, // node A's real log profile: more entries... - last_log_term: 2, // ...but a lower term than node 2's (3) + last_log_term: 2, // ...but a lower term than node 2's (3) })) .await } diff --git a/d-engine-server/tests/leader_election/mod.rs b/d-engine-server/tests/leader_election/mod.rs index 8dd0c226..49be0d4a 100644 --- a/d-engine-server/tests/leader_election/mod.rs +++ b/d-engine-server/tests/leader_election/mod.rs @@ -4,6 +4,7 @@ //! //! ## Test Coverage //! -//! - `leader_election_log_term_index_standalone.rs` - Node with highest log term/index becomes leader (gRPC mode) +//! - `leader_election_log_term_index_standalone.rs` - Node with highest log term/index becomes +//! leader (gRPC mode) mod leader_election_log_term_index_standalone; diff --git a/d-engine-server/tests/leader_tick_heartbeat/mod.rs b/d-engine-server/tests/leader_tick_heartbeat/mod.rs index 83a78f95..5d026888 100644 --- a/d-engine-server/tests/leader_tick_heartbeat/mod.rs +++ b/d-engine-server/tests/leader_tick_heartbeat/mod.rs @@ -9,7 +9,8 @@ //! ## Architecture Context //! //! In drain-based architecture: -//! - **Primary flush path**: Drain via recv() blocking on command channel, then try_recv() draining pending +//! - **Primary flush path**: Drain via recv() blocking on command channel, then try_recv() draining +//! pending //! - **Secondary flush path**: Heartbeat timer (safety net for buffered commands) //! - **Heartbeat interval**: Configurable (default 100ms), must be < election timeout (300ms) //! diff --git a/d-engine-server/tests/readonly_and_learner_mode/learner_readonly_sync_standalone.rs b/d-engine-server/tests/readonly_and_learner_mode/learner_readonly_sync_standalone.rs index 9c6c14db..3cc4a12f 100644 --- a/d-engine-server/tests/readonly_and_learner_mode/learner_readonly_sync_standalone.rs +++ b/d-engine-server/tests/readonly_and_learner_mode/learner_readonly_sync_standalone.rs @@ -40,7 +40,8 @@ use crate::common::test_put_get; /// **Standalone (RPC)**: Nodes communicate via gRPC, client accesses via ClientManager /// /// ## Test Purpose -/// Verify that a Learner node with READ_ONLY status (permanent Analytics node) works in Standalone/RPC mode: +/// Verify that a Learner node with READ_ONLY status (permanent Analytics node) works in +/// Standalone/RPC mode: /// 1. **Data Sync**: Node 4 syncs all data from the primary cluster (read-only capability) /// 2. **Non-Voting**: Node 4 does NOT participate in voting/quorum /// 3. **Permanent Learner**: Node 4 remains as Learner forever (never auto-promotes to Voter) @@ -49,7 +50,8 @@ use crate::common::test_put_get; /// ## Test Scenario /// - **Phase 1**: Bootstrap 3-node primary cluster (nodes 1,2,3 = Voter+ACTIVE via RPC) /// - **Phase 2**: Write test data through RPC client -/// - **Phase 3**: Start node 4 as standalone Learner with READ_ONLY status, joins via JoinCluster RPC +/// - **Phase 3**: Start node 4 as standalone Learner with READ_ONLY status, joins via JoinCluster +/// RPC /// - **Phase 4**: Verify node 4 can read all replicated data via RPC /// - **Phase 5**: Verify new writes propagate to node 4, quorum unchanged /// - **Phase 6**: Verify node 4 status remains READ_ONLY (cannot be promoted to Voter) diff --git a/d-engine-server/tests/readonly_and_learner_mode/mod.rs b/d-engine-server/tests/readonly_and_learner_mode/mod.rs index 4ce6d372..7b7575cf 100644 --- a/d-engine-server/tests/readonly_and_learner_mode/mod.rs +++ b/d-engine-server/tests/readonly_and_learner_mode/mod.rs @@ -4,7 +4,8 @@ //! //! ## Test Coverage //! -//! - `learner_readonly_sync_embedded.rs` - Learner data synchronization and eventual consistency reads (embedded mode) +//! - `learner_readonly_sync_embedded.rs` - Learner data synchronization and eventual consistency +//! reads (embedded mode) //! - `learner_readonly_sync_standalone.rs` - Learner operations via gRPC (standalone mode) mod learner_readonly_sync_embedded; diff --git a/d-engine-server/tests/replication_and_sync/mod.rs b/d-engine-server/tests/replication_and_sync/mod.rs index 6f3af4e3..a9fb41e1 100644 --- a/d-engine-server/tests/replication_and_sync/mod.rs +++ b/d-engine-server/tests/replication_and_sync/mod.rs @@ -4,6 +4,7 @@ //! //! ## Test Coverage //! -//! - `append_entries_out_of_sync_standalone.rs` - Leader handles followers with divergent logs via backtracking (gRPC mode) +//! - `append_entries_out_of_sync_standalone.rs` - Leader handles followers with divergent logs via +//! backtracking (gRPC mode) mod append_entries_out_of_sync_standalone; diff --git a/d-engine-server/tests/snapshot_and_recovery/mod.rs b/d-engine-server/tests/snapshot_and_recovery/mod.rs index 88433b70..3c260c8c 100644 --- a/d-engine-server/tests/snapshot_and_recovery/mod.rs +++ b/d-engine-server/tests/snapshot_and_recovery/mod.rs @@ -6,11 +6,16 @@ //! //! - `snapshot_generation_standalone.rs` - Snapshot creation with log compaction (gRPC mode) //! - `snapshot_concurrent_writes_embedded.rs` - Writes during snapshot generation (embedded mode) -//! - `snapshot_concurrent_replication_embedded.rs` - Learner receives snapshot during replication (embedded mode) -//! - `snapshot_follower_generation_embedded.rs` - Follower independent snapshot trigger + replication safety (fix #270) -//! - `snapshot_interrupted_transfer_embedded.rs` - Learner recovers from stale temp file left by prior interrupted transfer -//! - `snapshot_leader_change_during_transfer_embedded.rs` - Learner catches up after snapshot source leader dies mid-transfer -//! - `snapshot_transfer_does_not_block_apply_embedded.rs` - Large throttled snapshot push does not stall Leader apply +//! - `snapshot_concurrent_replication_embedded.rs` - Learner receives snapshot during replication +//! (embedded mode) +//! - `snapshot_follower_generation_embedded.rs` - Follower independent snapshot trigger + +//! replication safety (fix #270) +//! - `snapshot_interrupted_transfer_embedded.rs` - Learner recovers from stale temp file left by +//! prior interrupted transfer +//! - `snapshot_leader_change_during_transfer_embedded.rs` - Learner catches up after snapshot +//! source leader dies mid-transfer +//! - `snapshot_transfer_does_not_block_apply_embedded.rs` - Large throttled snapshot push does not +//! stall Leader apply mod snapshot_concurrent_replication_embedded; mod snapshot_concurrent_writes_embedded; diff --git a/d-engine-server/tests/snapshot_and_recovery/snapshot_concurrent_replication_embedded.rs b/d-engine-server/tests/snapshot_and_recovery/snapshot_concurrent_replication_embedded.rs index 17e61303..68d10f28 100644 --- a/d-engine-server/tests/snapshot_and_recovery/snapshot_concurrent_replication_embedded.rs +++ b/d-engine-server/tests/snapshot_and_recovery/snapshot_concurrent_replication_embedded.rs @@ -11,13 +11,12 @@ //! 5. Verify no data loss during PHASE 2.5 temporary DB window #![cfg(feature = "rocksdb")] -use d_engine_server::RocksDBUnifiedEngine; +use std::sync::Arc; +use std::time::Duration; use d_engine_server::DefaultEmbeddedEngine; - +use d_engine_server::RocksDBUnifiedEngine; use serial_test::serial; -use std::sync::Arc; -use std::time::Duration; use tracing::info; use tracing_test::traced_test; diff --git a/d-engine-server/tests/snapshot_and_recovery/snapshot_concurrent_writes_embedded.rs b/d-engine-server/tests/snapshot_and_recovery/snapshot_concurrent_writes_embedded.rs index c21c745b..f417fbc2 100644 --- a/d-engine-server/tests/snapshot_and_recovery/snapshot_concurrent_writes_embedded.rs +++ b/d-engine-server/tests/snapshot_and_recovery/snapshot_concurrent_writes_embedded.rs @@ -16,12 +16,11 @@ //! - Ensures no data loss during concurrent AppendEntries #![cfg(feature = "rocksdb")] -use d_engine_server::RocksDBUnifiedEngine; - use std::sync::Arc; use std::time::Duration; use d_engine_server::DefaultEmbeddedEngine; +use d_engine_server::RocksDBUnifiedEngine; use tracing::info; use tracing_test::traced_test; diff --git a/d-engine-server/tests/snapshot_and_recovery/snapshot_correctness_after_install_embedded.rs b/d-engine-server/tests/snapshot_and_recovery/snapshot_correctness_after_install_embedded.rs index b1483302..122e81d7 100644 --- a/d-engine-server/tests/snapshot_and_recovery/snapshot_correctness_after_install_embedded.rs +++ b/d-engine-server/tests/snapshot_and_recovery/snapshot_correctness_after_install_embedded.rs @@ -38,12 +38,12 @@ use crate::common::wait_for_snapshot; /// ## Test Flow /// /// 1. Start a 3-node cluster (`snapshot_threshold=100`, `retained_log_entries=30`). -/// 2. Write 120 entries → snapshot fires at 100, purge up to index 70. -/// Leader retains entries 71..120 for lagging followers. +/// 2. Write 120 entries → snapshot fires at 100, purge up to index 70. Leader retains entries +/// 71..120 for lagging followers. /// 3. Wait for snapshot to appear on the leader (confirms cluster is synced). /// 4. Stop one non-leader follower (node C). -/// 5. Write 20 more entries (121..140) to the leader while C is offline. -/// C now lags by 20 entries; 20 < retained=30, so entries 121..140 remain in leader's log. +/// 5. Write 20 more entries (121..140) to the leader while C is offline. C now lags by 20 entries; +/// 20 < retained=30, so entries 121..140 remain in leader's log. /// 6. Restart C with its persisted DB (state through index 120). /// 7. Wait for C to catch up to all 140 entries. /// @@ -270,24 +270,22 @@ retained_log_entries = {RETAINED_LOGS} /// /// ## Purpose (ticket #436) /// -/// This is the companion scenario to `test_follower_catchup_within_retained_buffer_and_data_consistency` -/// above — same judgment point (`prev_log_term` lookup deciding AppendEntries vs snapshot), -/// opposite branch. A manual, larger-scale repro (`deventlab-product-design` snapshot-test.sh -/// scenario_4) found that a follower whose lag exceeds the retained-log window can still end +/// This is the companion scenario to +/// `test_follower_catchup_within_retained_buffer_and_data_consistency` above — same judgment point +/// (`prev_log_term` lookup deciding AppendEntries vs snapshot), opposite branch. A manual, +/// larger-scale repro found that a follower whose lag exceeds the retained-log window can still end /// up "catching up" through ordinary AppendEntries instead of `InstallSnapshot` — the data /// happened to come out correct, but the mechanism was wrong, and it stayed wrong through a -/// leader-killed-and-reconnected cycle without any test catching it. See -/// `d-engine/tickets/milestones/v0.2.5/436-plan-prev-log-term-fallback.md`. +/// leader-killed-and-reconnected cycle without any test catching it. /// /// ## Test Flow /// /// 1. Start a 3-node cluster (`snapshot_threshold=20`, `retained_log_entries=5`). /// 2. Write 5 entries so all 3 nodes are in sync at index 5. /// 3. Stop one non-leader follower (node C) — it is now frozen at index 5. -/// 4. Write 40 more entries (index 6..45) to the leader while C is offline. This crosses -/// the snapshot threshold (20), so the leader's retained window ends up starting well -/// above index 5 — C's frozen position is now behind the purge boundary, not just behind -/// the leader. +/// 4. Write 40 more entries (index 6..45) to the leader while C is offline. This crosses the +/// snapshot threshold (20), so the leader's retained window ends up starting well above index 5 +/// — C's frozen position is now behind the purge boundary, not just behind the leader. /// 5. Restart C with its persisted DB and wait for it to catch up to all 45 entries. /// /// ## Expected Results (this is the assertion that must currently FAIL before #436 is fixed) diff --git a/d-engine-server/tests/snapshot_and_recovery/snapshot_follower_generation_embedded.rs b/d-engine-server/tests/snapshot_and_recovery/snapshot_follower_generation_embedded.rs index 9aec19de..39556b72 100644 --- a/d-engine-server/tests/snapshot_and_recovery/snapshot_follower_generation_embedded.rs +++ b/d-engine-server/tests/snapshot_and_recovery/snapshot_follower_generation_embedded.rs @@ -6,19 +6,21 @@ //! 1. **Trigger correctness**: Follower triggers snapshot at the configured //! `max_log_entries_before_snapshot` threshold, not before or never. //! -//! 2. **Replication safety**: While a Follower is generating its own snapshot -//! (background async task), it continues to receive and apply AppendEntries -//! from the Leader without data loss or stall. +//! 2. **Replication safety**: While a Follower is generating its own snapshot (background async +//! task), it continues to receive and apply AppendEntries from the Leader without data loss or +//! stall. #![cfg(feature = "rocksdb")] -use d_engine_server::{DefaultEmbeddedEngine, RocksDBUnifiedEngine}; - use std::sync::Arc; use std::time::Duration; + +use d_engine_server::DefaultEmbeddedEngine; +use d_engine_server::RocksDBUnifiedEngine; use tracing::info; use tracing_test::traced_test; -use crate::common::{get_available_ports, wait_for_snapshot}; +use crate::common::get_available_ports; +use crate::common::wait_for_snapshot; /// Test: Follower generates snapshot at the correct threshold and continues replication /// @@ -48,8 +50,8 @@ use crate::common::{get_available_ports, wait_for_snapshot}; /// - Entry #100 triggers automatic snapshot on Leader AND Followers (Raft §7). /// 3. Poll until the Follower snapshot file appears (max 15s). /// - Fails fast if the trigger is broken (never fires). -/// 4. While the Follower snapshot background task may still be running, -/// write 50 additional concurrent entries to the Leader. +/// 4. While the Follower snapshot background task may still be running, write 50 additional +/// concurrent entries to the Leader. /// 5. Wait for all Followers to replicate all 200 entries. /// 6. Verify: Follower has all 200 entries with correct values (no data loss). /// 7. Verify: cluster term did not increase (no unintended re-election during snapshot). diff --git a/d-engine-server/tests/snapshot_and_recovery/snapshot_transfer_does_not_block_apply_embedded.rs b/d-engine-server/tests/snapshot_and_recovery/snapshot_transfer_does_not_block_apply_embedded.rs index f5117e27..37a2d295 100644 --- a/d-engine-server/tests/snapshot_and_recovery/snapshot_transfer_does_not_block_apply_embedded.rs +++ b/d-engine-server/tests/snapshot_and_recovery/snapshot_transfer_does_not_block_apply_embedded.rs @@ -73,12 +73,12 @@ fn deterministic_value( /// /// 1. Start a 3-node cluster; write enough large entries to trigger snapshot + purge. /// 2. Record the Leader's applied index. -/// 3. Register a transfer gate for the Learner's node id, then start the Learner (behind -/// the purge boundary → needs a snapshot). +/// 3. Register a transfer gate for the Learner's node id, then start the Learner (behind the purge +/// boundary → needs a snapshot). /// 4. Wait for the gate to report the transfer has started (deterministic handshake). /// 5. Write `CONCURRENT_ENTRIES` small entries to the Leader. -/// 6. Assert the Leader's applied index advances past the concurrent writes **while the -/// transfer gate confirms the snapshot transfer has still not completed**. +/// 6. Assert the Leader's applied index advances past the concurrent writes **while the transfer +/// gate confirms the snapshot transfer has still not completed**. /// 7. Release the gate, assert the Learner eventually catches up, then stop all engines. #[tokio::test] #[serial] diff --git a/d-engine-server/tests/storage_raft_log/crash_recovery_test.rs b/d-engine-server/tests/storage_raft_log/crash_recovery_test.rs index 1b955773..f9fbdfe6 100644 --- a/d-engine-server/tests/storage_raft_log/crash_recovery_test.rs +++ b/d-engine-server/tests/storage_raft_log/crash_recovery_test.rs @@ -8,9 +8,13 @@ use std::sync::Arc; use std::time::Duration; use bytes::Bytes; -use d_engine_core::{RaftLog, RaftLogCore}; -use d_engine_proto::common::{Entry, EntryPayload}; -use d_engine_server::{FileStateMachine, FileStorageEngine, node::RaftTypeConfig}; +use d_engine_core::RaftLog; +use d_engine_core::RaftLogCore; +use d_engine_proto::common::Entry; +use d_engine_proto::common::EntryPayload; +use d_engine_server::FileStateMachine; +use d_engine_server::FileStorageEngine; +use d_engine_server::node::RaftTypeConfig; use tokio::time::sleep; use super::TestContext; @@ -157,8 +161,8 @@ async fn test_partial_flush_with_graceful_shutdown() { /// True crash (kill -9 / power loss) means data MAY survive if the fsync had time /// to complete before the crash. This test verifies: /// - Entries flushed before the crash (first batch, waited 150ms) always survive. -/// - Entries written immediately before crash (second batch, no wait) may or may not -/// survive depending on IO thread scheduling at crash time. +/// - Entries written immediately before crash (second batch, no wait) may or may not survive +/// depending on IO thread scheduling at crash time. /// - `mem::forget` simulates a hard crash: no graceful shutdown, no final flush. #[tokio::test] async fn test_partial_flush_after_crash() { diff --git a/d-engine-server/tests/storage_raft_log/mod.rs b/d-engine-server/tests/storage_raft_log/mod.rs index ad40a14b..ed62aa1a 100644 --- a/d-engine-server/tests/storage_raft_log/mod.rs +++ b/d-engine-server/tests/storage_raft_log/mod.rs @@ -14,9 +14,14 @@ use std::path::PathBuf; use std::sync::Arc; use bytes::Bytes; -use d_engine_core::{RaftLog, RaftLogCore, alias::ROF}; -use d_engine_proto::common::{Entry, EntryPayload}; -use d_engine_server::{FileStateMachine, FileStorageEngine, node::RaftTypeConfig}; +use d_engine_core::RaftLog; +use d_engine_core::RaftLogCore; +use d_engine_core::alias::ROF; +use d_engine_proto::common::Entry; +use d_engine_proto::common::EntryPayload; +use d_engine_server::FileStateMachine; +use d_engine_server::FileStorageEngine; +use d_engine_server::node::RaftTypeConfig; use tempfile::tempdir; mod crash_recovery_test; diff --git a/d-engine-server/tests/storage_raft_log/performance_test.rs b/d-engine-server/tests/storage_raft_log/performance_test.rs index 89997074..d770beff 100644 --- a/d-engine-server/tests/storage_raft_log/performance_test.rs +++ b/d-engine-server/tests/storage_raft_log/performance_test.rs @@ -4,17 +4,23 @@ //! Most tests are marked with #[ignore] and can be run explicitly or //! skipped in CI environments. -use super::TestContext; -use bytes::Bytes; -use d_engine_core::{RaftLog, RaftLogCore}; -use d_engine_proto::common::{Entry, EntryPayload}; -use d_engine_server::{FileStateMachine, FileStorageEngine, node::RaftTypeConfig}; use std::collections::HashMap; use std::sync::Arc; use std::time::Duration; + +use bytes::Bytes; +use d_engine_core::RaftLog; +use d_engine_core::RaftLogCore; +use d_engine_proto::common::Entry; +use d_engine_proto::common::EntryPayload; +use d_engine_server::FileStateMachine; +use d_engine_server::FileStorageEngine; +use d_engine_server::node::RaftTypeConfig; use tempfile::tempdir; use tokio::time::Instant; +use super::TestContext; + mod filter_out_conflicts_and_append_performance_tests { use super::*; @@ -75,7 +81,6 @@ mod filter_out_conflicts_and_append_performance_tests { // Verify correctness: index=501 term=1 already exists (populated above), so this // prev_log_index=0 resend is a pure duplicate — must be a no-op, not a reset. - // See 446-expert-q-probe-backpressure-fix-8020.md. assert_eq!( log.last_entry_id(), 1000, diff --git a/d-engine-server/tests/storage_raft_log/quorum_crash_recovery_test.rs b/d-engine-server/tests/storage_raft_log/quorum_crash_recovery_test.rs index a307885f..e2b6f51d 100644 --- a/d-engine-server/tests/storage_raft_log/quorum_crash_recovery_test.rs +++ b/d-engine-server/tests/storage_raft_log/quorum_crash_recovery_test.rs @@ -38,9 +38,10 @@ //! already found to be a poor fit for this class of bug (see mempalace notes on the //! Jepsen/lazyfs work for #444). -use super::TestContext; use d_engine_core::RaftLog; +use super::TestContext; + #[tokio::test] async fn test_quorum_acknowledged_index_survives_real_crash_and_reopen() { let mut ctx = TestContext::new("test_quorum_ack_survives_crash"); diff --git a/d-engine-server/tests/storage_raft_log/stress_test.rs b/d-engine-server/tests/storage_raft_log/stress_test.rs index d3b079fc..4f61fe62 100644 --- a/d-engine-server/tests/storage_raft_log/stress_test.rs +++ b/d-engine-server/tests/storage_raft_log/stress_test.rs @@ -6,8 +6,11 @@ use std::time::Duration; use bytes::Bytes; -use d_engine_core::{LogStore, RaftLog, StorageEngine}; -use d_engine_proto::common::{Entry, EntryPayload}; +use d_engine_core::LogStore; +use d_engine_core::RaftLog; +use d_engine_core::StorageEngine; +use d_engine_proto::common::Entry; +use d_engine_proto::common::EntryPayload; use futures::future::join_all; use tokio::time::Instant; use tracing_test::traced_test; diff --git a/d-engine-server/tests/watch_and_subscriptions/watch_events_embedded.rs b/d-engine-server/tests/watch_and_subscriptions/watch_events_embedded.rs index 9e8422cd..8bb6089a 100644 --- a/d-engine-server/tests/watch_and_subscriptions/watch_events_embedded.rs +++ b/d-engine-server/tests/watch_and_subscriptions/watch_events_embedded.rs @@ -1,10 +1,9 @@ #![cfg(feature = "rocksdb")] -use d_engine_server::RocksDBUnifiedEngine; - use std::sync::Arc; use std::time::Duration; use d_engine_core::watch::WatchEventType; +use d_engine_server::RocksDBUnifiedEngine; use d_engine_server::api::DefaultEmbeddedEngine; use tempfile::TempDir; use tokio::time::sleep; diff --git a/d-engine-server/tests/watch_and_subscriptions/watch_events_grpc_standalone.rs b/d-engine-server/tests/watch_and_subscriptions/watch_events_grpc_standalone.rs index 19d81059..1e884b18 100644 --- a/d-engine-server/tests/watch_and_subscriptions/watch_events_grpc_standalone.rs +++ b/d-engine-server/tests/watch_and_subscriptions/watch_events_grpc_standalone.rs @@ -6,10 +6,6 @@ use std::sync::Arc; use std::time::Duration; -use crate::common::TestContext; -use crate::common::create_node_config; -use crate::common::node_config; -use crate::common::start_node; use d_engine_client::ClientBuilder; use d_engine_core::ClientApi; use d_engine_core::client::ErrorCode; @@ -20,6 +16,11 @@ use futures::StreamExt; use tempfile::TempDir; use tokio::time::sleep; +use crate::common::TestContext; +use crate::common::create_node_config; +use crate::common::node_config; +use crate::common::start_node; + /// Helper function to create a 3-node standalone cluster with watch enabled async fn setup_standalone_cluster() -> Result<(d_engine_client::Client, TestContext, TempDir), Box> { diff --git a/d-engine-server/tests/watch_and_subscriptions/watch_membership_embedded.rs b/d-engine-server/tests/watch_and_subscriptions/watch_membership_embedded.rs index 051bd020..bbd454a0 100644 --- a/d-engine-server/tests/watch_and_subscriptions/watch_membership_embedded.rs +++ b/d-engine-server/tests/watch_and_subscriptions/watch_membership_embedded.rs @@ -16,30 +16,30 @@ //! //! ## Design notes //! -//! - `watch::Receiver` semantics: lossy (latest value only), no diff carried. -//! Callers compute diff from their own previous snapshot. -//! - `committed_index` is the idempotency key; callers use it to detect replayed -//! snapshots after reconnect or restart. -//! - Notification fires on **every** node (leader / follower / learner) because -//! all walk the same `CommitHandler::apply_config_change()` path. +//! - `watch::Receiver` semantics: lossy (latest value only), no diff carried. Callers compute diff +//! from their own previous snapshot. +//! - `committed_index` is the idempotency key; callers use it to detect replayed snapshots after +//! reconnect or restart. +//! - Notification fires on **every** node (leader / follower / learner) because all apply a +//! committed config change through the same path (`dispatch_committed`). //! //! ## How conf changes reach the watch channel //! //! Only Raft-consensus-backed conf changes trigger the watch. The three paths used //! in these tests are: //! -//! 1. **`JoinCluster` RPC**: New learner node (role=4) calls `join_cluster()` on boot. -//! Leader appends `AddNode(Config)` entry → commits → `CommitHandler::apply_config_change` -//! → `notify_config_applied` → watch fires on all nodes. +//! 1. **`JoinCluster` RPC**: New learner node (role=4) calls `join_cluster()` on boot. Leader +//! appends `AddNode(Config)` entry → commits → `dispatch_committed` → `apply_membership` → +//! `notify_config_applied` → watch fires on all nodes. //! //! 2. **Zombie detection** (warn-only): When a node's connection failure count reaches //! `zombie.threshold`, the leader emits a `warn` log — no `BatchRemove` is proposed. -//! Auto-removal is intentionally absent; see zombie_detection_decision.md. -//! Test 3 uses `threshold=1` to trigger the warning quickly and verify no watch fires. +//! Auto-removal is intentionally absent; see zombie_detection_decision.md. Test 3 uses +//! `threshold=1` to trigger the warning quickly and verify no watch fires. //! -//! 3. **Auto-promotion** (`BatchPromote`): When a learner with `status=Promotable(2)` catches -//! up to the leader's commit index (within `learner_check_throttle_ms = 100 ms`), the -//! leader proposes `BatchPromote(Config)` → commits → watch fires. +//! 3. **Auto-promotion** (`BatchPromote`): When a learner with `status=Promotable(2)` catches up to +//! the leader's commit index (within `learner_check_throttle_ms = 100 ms`), the leader proposes +//! `BatchPromote(Config)` → commits → watch fires. //! //! `ClusterManagementService::update_cluster_conf` is NOT used here; it bypasses the Raft //! log and modifies follower membership state directly without calling `notify_config_applied`. @@ -49,7 +49,8 @@ use std::sync::Arc; use std::time::Duration; -use d_engine_server::{DefaultEmbeddedEngine, RocksDBUnifiedEngine}; +use d_engine_server::DefaultEmbeddedEngine; +use d_engine_server::RocksDBUnifiedEngine; use tempfile::TempDir; use tokio::time::timeout; use tracing_test::traced_test; @@ -366,12 +367,11 @@ async fn test_watch_membership_fires_on_node_join() -> Result<(), Box Result<(), Box Result<(), Box Result<(), Error> { - // Graceful shutdown: stop accepting requests. - // Called by the Raft loop on shutdown. May be called multiple times. - Ok(()) - } - - // close_storage() has a default no-op implementation. - // Override ONLY if your backend holds an exclusive OS resource (e.g. a lock - // file or an open DB handle) that must be released immediately when - // EmbeddedEngine::stop() is called — without waiting for all Arc clones - // held by EmbeddedClient to drop. + // close() has a default no-op implementation. + // Override it if your backend must persist something at shutdown (e.g. lease state) or + // holds an exclusive OS resource (a lock file, an open DB handle). d-engine calls it once, + // as the worker's last step: after every committed entry has been applied when the worker + // drains cleanly, but shutdown continues without that guarantee if + // `shutdown_apply_timeout_ms` expires. It must be synchronous and idempotent, release its + // resources even if persisting fails, and return the first error. + // You may keep your own Arc to the state machine, so do not rely on Drop alone. // - // fn close_storage(&self) { + // fn close(&self) -> Result<(), Error> { + // let flushed = self.backend.flush(); // self.backend.close(); // release lock file / DB handle + // flushed // } fn is_running(&self) -> bool { @@ -123,8 +122,7 @@ impl StateMachine for CustomStateMachine { | Method | Purpose | Sync/Async | Criticality | | ---------------------------------- | -------------------------------------------------------------------------------------------- | ---------- | ----------- | | `start()` | Initialize state machine service | Sync | High | -| `stop()` | Graceful shutdown (reversible) | Sync | High | -| `close_storage()` | Release exclusive OS resources (e.g. DB lock file). Default no-op — override only if needed. | Sync | Medium | +| `close()` | Final step at shutdown: persist what must survive a restart and release exclusive OS resources (e.g. DB lock file), even if persisting fails; returns the first error. Idempotent. Default `Ok(())`. | Sync | Medium | | `is_running()` | Check service status | Sync | Medium | | `get()` | Read value by key | Sync | High | | `apply_chunk()` | Apply committed entries | Async | Critical | @@ -184,7 +182,7 @@ async fn test_custom_state_machine() -> Result<(), Error> { let checksum = sm.generate_snapshot_data(temp_dir.path().to_path_buf(), LogId::new(1, 1)).await?; assert!(!checksum.is_zero()); - sm.stop()?; + sm.close()?; Ok(()) } diff --git a/d-engine/src/lib.rs b/d-engine/src/lib.rs index a9c68196..8f2d4b53 100644 --- a/d-engine/src/lib.rs +++ b/d-engine/src/lib.rs @@ -2,12 +2,11 @@ #![doc = include_str!("docs/overview.md")] // Re-export server components when server feature is enabled -#[cfg(feature = "server")] -pub use d_engine_server::*; - // Re-export client components when client feature is enabled #[cfg(feature = "client")] pub use d_engine_client::*; +#[cfg(feature = "server")] +pub use d_engine_server::*; /// Convenient prelude for importing common types /// @@ -15,19 +14,40 @@ pub use d_engine_client::*; /// use d_engine::prelude::*; /// ``` pub mod prelude { + #[cfg(feature = "client")] + pub use d_engine_client::Client; + #[cfg(feature = "client")] + pub use d_engine_client::ClientApi; + #[cfg(feature = "client")] + pub use d_engine_client::ClientBuilder; + #[cfg(feature = "rocksdb")] + pub use d_engine_server::DefaultEmbeddedClient; + #[cfg(feature = "rocksdb")] + pub use d_engine_server::DefaultEmbeddedEngine; #[cfg(feature = "server")] - pub use d_engine_server::{ - EmbeddedClient, EmbeddedEngine, Error, FileStateMachine, FileStorageEngine, Node, Result, - StandaloneEngine, StateMachine, StorageEngine, - }; - + pub use d_engine_server::EmbeddedClient; + #[cfg(feature = "server")] + pub use d_engine_server::EmbeddedEngine; + #[cfg(feature = "server")] + pub use d_engine_server::Error; + #[cfg(feature = "server")] + pub use d_engine_server::FileStateMachine; + #[cfg(feature = "server")] + pub use d_engine_server::FileStorageEngine; + #[cfg(feature = "server")] + pub use d_engine_server::Node; + #[cfg(feature = "server")] + pub use d_engine_server::Result; #[cfg(feature = "rocksdb")] - pub use d_engine_server::{ - DefaultEmbeddedClient, DefaultEmbeddedEngine, RocksDBStateMachine, RocksDBStorageEngine, - }; - - #[cfg(feature = "client")] - pub use d_engine_client::{Client, ClientApi, ClientBuilder}; + pub use d_engine_server::RocksDBStateMachine; + #[cfg(feature = "rocksdb")] + pub use d_engine_server::RocksDBStorageEngine; + #[cfg(feature = "server")] + pub use d_engine_server::StandaloneEngine; + #[cfg(feature = "server")] + pub use d_engine_server::StateMachine; + #[cfg(feature = "server")] + pub use d_engine_server::StorageEngine; } /// Documentation modules diff --git a/examples/sled-cluster/src/sled_state_machine.rs b/examples/sled-cluster/src/sled_state_machine.rs index 0d467e14..438b32ab 100644 --- a/examples/sled-cluster/src/sled_state_machine.rs +++ b/examples/sled-cluster/src/sled_state_machine.rs @@ -2,7 +2,7 @@ use crate::compute_checksum_from_folder_path; use crate::{safe_kv, safe_vk_ivec}; -use arc_swap::ArcSwap; +use arc_swap::ArcSwapOption; use async_trait::async_trait; use bincode::config; use bytes::Bytes; @@ -46,7 +46,7 @@ pub struct SledStateMachine { /// *here*, not in the caller's temporary decompression directory (#442 review). path: PathBuf, - db: ArcSwap, + db: ArcSwapOption, is_serving: Arc, @@ -69,7 +69,7 @@ impl std::fmt::Debug for SledStateMachine { f: &mut std::fmt::Formatter<'_>, ) -> std::fmt::Result { f.debug_struct("SledStateMachine") - .field("tree_len", &self.current_tree().len()) + .field("tree_len", &self.current_tree().map(|t| t.len()).ok()) .finish() } } @@ -79,9 +79,9 @@ impl Drop for SledStateMachine { let timer = Instant::now(); // Save state into local database including flush operation - match self.save_hard_state() { - Ok(_) => debug!("StateMachine saved in {:?}", timer.elapsed()), - Err(e) => error!("Failed to save StateMachine: {}", e), + match StateMachine::close(self) { + Ok(()) => debug!("StateMachine closed in {:?}", timer.elapsed()), + Err(e) => error!("Failed to close StateMachine: {}", e), } } } @@ -94,16 +94,16 @@ impl StateMachine for SledStateMachine { Ok(()) } - fn close_storage(&self) { + fn close(&self) -> Result<()> { + if self.db.load().is_none() { + return Ok(()); + } self.is_serving.store(false, Ordering::Release); - // Flush pending sled writes. The sled::Db is released when SledStateMachine drops. - let _ = self.db.load().flush(); - } - - fn stop(&self) -> Result<()> { - debug!("stop state machine"); - self.is_serving.store(false, Ordering::Release); - Ok(()) + // Persist, then release the db handle so sled's file lock is freed even if persisting + // failed. The lock is released once no other handle to the db is alive. + let saved = self.save_hard_state(); + self.db.store(None); + saved } fn is_running(&self) -> bool { @@ -165,7 +165,7 @@ impl StateMachine for SledStateMachine { &self, key_buffer: &[u8], ) -> Result> { - match self.current_tree().get(key_buffer) { + match self.current_tree()?.get(key_buffer) { Ok(Some(v)) => Ok(Some(Bytes::copy_from_slice(&v))), Ok(None) => Ok(None), Err(e) => { @@ -180,6 +180,7 @@ impl StateMachine for SledStateMachine { chunk: &[ApplyEntry], ) -> Result> { trace!("Applying chunk: {} entries", chunk.len()); + self.db()?; let mut highest_log_id: Option = None; let mut batch = Batch::default(); @@ -221,7 +222,7 @@ impl StateMachine for SledStateMachine { self.apply_batch(std::mem::take(&mut batch))?; // Perform native Sled CAS - let tree = self.current_tree(); + let tree = self.current_tree()?; let old = expected.as_ref().map(|v| v.as_ref()); let cas_success = @@ -288,7 +289,7 @@ impl StateMachine for SledStateMachine { last_applied: LogId, ) -> Result<()> { debug!(%self.node_id, ?last_applied, "persist_last_applied"); - let db = self.db.load(); + let db = self.db()?; let tree = db .open_tree(STATE_MACHINE_META_NAMESPACE) .map_err(|e| StorageError::DbError(e.to_string()))?; @@ -312,7 +313,7 @@ impl StateMachine for SledStateMachine { last_snapshot_metadata: &SnapshotMetadata, ) -> Result<()> { debug!(%self.node_id, ?last_snapshot_metadata, "persist_last_snapshot_metadata"); - let db = self.db.load(); + let db = self.db()?; let tree = db .open_tree(STATE_SNAPSHOT_METADATA_TREE) .map_err(|e| StorageError::DbError(e.to_string()))?; @@ -322,20 +323,12 @@ impl StateMachine for SledStateMachine { } fn flush(&self) -> Result<()> { - let db = self.db.load(); - match db.flush() { - Ok(bytes) => { - info!( - "Successfully flushed State Machine, bytes flushed: {}", - bytes - ); - println!("Successfully flushed State Machine, bytes flushed: {bytes}"); - } - Err(e) => { - error!("Failed to flush State Machine: {}", e); - eprintln!("Failed to flush State Machine: {e}"); - } - } + let db = self.db()?; + let bytes = db.flush().map_err(|e| StorageError::DbError(e.to_string()))?; + info!( + "Successfully flushed State Machine, bytes flushed: {}", + bytes + ); Ok(()) } @@ -357,7 +350,7 @@ impl StateMachine for SledStateMachine { let new_db = init_sled_state_machine_db(&new_snapshot_dir) .map_err(|e| StorageError::DbError(e.to_string()))?; - let exist_db_tree = self.current_tree(); + let exist_db_tree = self.current_tree()?; let new_state_machine_tree = new_tree(&new_db, STATE_MACHINE_TREE)?; let mut batch = sled::Batch::default(); @@ -510,7 +503,7 @@ impl StateMachine for SledStateMachine { debug!("Reopening state machine database at its permanent path"); let db = init_sled_state_machine_db(&self.path) .map_err(|e| StorageError::DbError(e.to_string()))?; - self.db.store(Arc::new(db)); + self.db.store(Some(Arc::new(db))); // Best-effort: a leftover `*.pre-install-backup` is harmless (just disk space) // if this fails or the process dies here — it's never read again once @@ -533,11 +526,11 @@ impl StateMachine for SledStateMachine { } fn len(&self) -> usize { - self.current_tree().len() + self.current_tree().map(|t| t.len()).unwrap_or(0) } async fn reset(&self) -> Result<()> { - let db = self.db.load(); + let db = self.db()?; for tree_name in &[ STATE_MACHINE_TREE, STATE_MACHINE_META_NAMESPACE, @@ -566,7 +559,7 @@ impl StateMachine for SledStateMachine { &self, prefix: &[u8], ) -> Result { - let tree = self.current_tree(); + let tree = self.current_tree()?; let mut entries = Vec::new(); for result in tree.scan_prefix(prefix) { let (k, v) = result.map_err(|e| StorageError::DbError(e.to_string()))?; @@ -615,7 +608,7 @@ impl SledStateMachine { .map_err(|e| StorageError::DbError(e.to_string()))?; let sm = SledStateMachine { - db: ArcSwap::from(db), + db: ArcSwapOption::from(Some(db)), path, is_serving: Arc::new(AtomicBool::new(true)), node_id, @@ -696,7 +689,7 @@ impl SledStateMachine { &self, batch: Batch, ) -> Result<()> { - if let Err(e) = self.current_tree().apply_batch(batch) { + if let Err(e) = self.current_tree()?.apply_batch(batch) { error!("state_machine apply_batch failed: {}", e); return Err(StorageError::DbError(e.to_string()).into()); } @@ -705,9 +698,17 @@ impl SledStateMachine { // Dynamically obtain the current Tree (lock-free cache optimization) #[inline] - fn current_tree(&self) -> sled::Tree { - // Each Db instance only needs to get the Tree once - self.db.load().open_tree(STATE_MACHINE_TREE).unwrap() + fn current_tree(&self) -> Result { + self.db()? + .open_tree(STATE_MACHINE_TREE) + .map_err(|e| StorageError::DbError(e.to_string()).into()) + } + + /// The live db, or `NotServing` once `close()` has released it. + fn db(&self) -> Result> { + self.db + .load_full() + .ok_or_else(|| StorageError::NotServing("state machine stopped".into()).into()) } pub(super) fn persist_last_snapshot_metadata_with_tree(