Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,12 @@ jobs:
toolchain: 1.89.0
components: rustfmt, clippy

- name: Install nightly rustfmt
run: rustup toolchain install nightly --profile minimal --component rustfmt

# rustfmt.toml uses nightly-only options, so formatting must be checked with nightly.
- name: Check Formatting
run: cargo fmt --all -- --check
run: cargo +nightly fmt --all -- --check

- name: Run Clippy
run: cargo clippy --all-targets --all-features -- -D warnings
Expand Down
14 changes: 13 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,10 @@ All notable changes to this project will be documented in this file.

- **Write admission and CheckQuorum** (#423): A leader without a recent quorum ACK rejects new client writes with `NotLeader` after `election_timeout_min`, before anything is appended, so clients can retry safely. After `election_timeout_max x write_admission_election_timeout_multiple` (new key in `[raft.backpressure]`, default `2`) without a quorum ACK it steps down. A new leader gets one window for its first ACK; single-voter clusters are exempt.

- **Queue-wait metrics** (#455): `core.raft.client_cmd.issue_to_push_ms`, `core.raft.sm_worker.queue_wait_ms` and `core.raft.central_loop.apply_completed_queue_wait_ms` show where a write waits between the client, the Raft loop and the state machine worker.

- **`raft.state_machine.shutdown_apply_timeout_ms`** (#455, default `5000`): how long shutdown waits for the state machine worker to apply the entries already committed. On timeout it logs an error and shuts down anyway; a restart replays what was not applied.

### Fixed

- **fix(ttl) #398**: Removed `lease.enabled` flag — TTL expiration is always active. Fixes fatal crash when calling `put_with_ttl` without setting the (now-removed) `lease.enabled = true`.
Expand Down Expand Up @@ -54,6 +58,14 @@ All notable changes to this project will be documented in this file.

### Changed

- **⚠️ `StateMachine::stop()` removed; `StateMachine::close_storage()` renamed to `close()`** (#455): `stop()` had no caller in d-engine. `close()` is now the single, permanent, idempotent end-of-life hook and also persists what must survive a restart (lease state). It returns `Result<(), Error>`: release your resources even if persisting fails, and return the first error; the node logs it at ERROR level. Custom implementations: delete `stop()`, rename `close_storage()` to `close()`, and move any lease persistence from `stop()` into `close()`. See `MIGRATION_GUIDE.md`.

- **State machine worker lifecycle** (#455): the `NodeBuilder` only assembles the worker; `Node::run()` starts it and stops it. `StateMachineWorker::new` no longer takes a shutdown signal.

- **Known limitation: re-opening a RocksDB directory while you still hold the storage engine** (#455): the state machine now releases its RocksDB handle at shutdown, but the log store keeps its own handle until the `RocksDBStorageEngine` is dropped. If you keep your own clone of the storage engine after `stop()`, opening the same directory again fails with a lock error until that clone is dropped. Passing the engine to `EmbeddedEngine` without keeping a clone is unaffected.

- **Internal commit path** (#455): the separate commit handler task is merged into the Raft loop (`dispatch_committed`). `ClientCmd::Propose`, `StateMachineCommand::ApplyEntries` and `InternalEvent::ApplyCompleted` each carry one more timestamp field (internal types).

- **MSRV raised to Rust 1.89**: The `data_dir` startup lock (prevents two node processes from
sharing the same directory) uses `std::fs::File::try_lock()`, stable since 1.89.0. CI and
`rust-toolchain.toml` were already pinned to 1.89.0 — only the `rust-version` field in
Expand Down Expand Up @@ -99,7 +111,7 @@ All notable changes to this project will be documented in this file.
- **⚠️ `[raft.persistence] strategy` removed** (#446): `PersistenceStrategy` was a single-variant enum
(`MemFirst`) left over from #268; its only meaning now lives in whether an entry has reached
`durable_index`, which is no longer a configurable choice. Existing configs setting `strategy =
"MemFirst"` or `"DiskFirst"` are silently ignored, not an error — remove the field, `flush_policy`
"MemFirst"` or `"DiskFirst"` are silently ignored, not an error — remove the field, `flush_policy`
is the only persistence knob now.

---
Expand Down
39 changes: 39 additions & 0 deletions MIGRATION_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -352,6 +352,45 @@ fn apply_chunk(&mut self, entries: &[ApplyEntry]) -> Result<(), ...>

---

## 🚨 For v0.2.4 Users: `StateMachine` Lifecycle Changes in v0.2.5 (#455)

### What Changed

| Before (v0.2.4) | After (v0.2.5) |
| ---------------------------------------------------------------- | ----------------------------------------------------------------- |
| `fn stop(&self) -> Result<(), Error>` (required) | **Removed** |
| `fn close_storage(&self)` (default no-op) | **Renamed** `fn close(&self) -> Result<(), Error>` (default `Ok`) |
| `[raft.storage]` accepted as an alias of `[raft.state_machine]` | Use `[raft.state_machine]` |

`stop()` was never called by d-engine. `close()` is now the one end-of-life hook. d-engine calls it once, as the last step of the
state machine worker. When the worker drains cleanly, every committed entry has been applied by then; if
`shutdown_apply_timeout_ms` expires first, shutdown continues and committed entries may remain unapplied until the next start.
Because you may keep your own `Arc` to the state machine, do not rely on `Drop` alone. `close()` returns an error if persisting
fails: release your resources anyway and return the first error. The node logs it at ERROR level.

### Migration Steps

1. Delete your `fn stop(...)`.
2. Rename `fn close_storage(&self)` to `fn close(&self) -> Result<(), Error>`.
3. If your `stop()` persisted something that must survive a restart (for example lease state), do it in `close()`.
4. `close()` must be synchronous and idempotent: after it returns, every operation should return a "not serving" error.
5. In config files, rename `[raft.storage]` to `[raft.state_machine]`.

```rust
// Before
fn stop(&self) -> Result<(), Error> { self.persist_ttl()?; Ok(()) }
fn close_storage(&self) { self.db.flush(); }

// After
fn close(&self) -> Result<(), Error> {
let persisted = self.persist_ttl();
self.db.flush();
persisted
}
```

---

## 🚨 For v0.2.3 Users: API Surface Changes in v0.2.4 (#326)

### What Changed
Expand Down
13 changes: 8 additions & 5 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@

# Cargo executable
CARGO ?= cargo
# rustfmt.toml uses nightly-only options (group_imports, imports_granularity, ...); stable rustfmt ignores them.
CARGO_FMT ?= cargo +nightly fmt

# On macOS with Homebrew: auto-detect compression lib paths to skip bundled C++
# compilation of RocksDB dependencies, which fails under macOS 26 + Xcode 26
Expand Down Expand Up @@ -127,6 +129,7 @@ check-env:
install-tools: check-env
@echo "$(BLUE)Installing Rust development components...$(NC)"
@rustup component add rustfmt clippy rust-src rust-analyzer 2>/dev/null || true
@rustup toolchain install nightly --profile minimal --component rustfmt 2>/dev/null || true
@if ! command -v cargo-nextest >/dev/null 2>&1; then \
echo "$(YELLOW)Installing cargo-nextest for fast parallel testing...$(NC)"; \
cargo install cargo-nextest --locked; \
Expand Down Expand Up @@ -163,14 +166,14 @@ check-workspace: check-env
## fmt-check Check code formatting without modifying files (dry-run)
fmt-check: check-workspace
@echo "$(BLUE)Checking code formatting...$(NC)"
@$(CARGO) fmt --all -- --check || \
@$(CARGO_FMT) --all -- --check || \
{ echo "$(RED)✗ Formatting issues found. Run 'make fmt-fix' to fix them.$(NC)"; exit 1; }
@echo "$(GREEN)✓ Code formatting check passed$(NC)"

## fmt fmt-fix Fix all code formatting issues automatically
fmt fmt-fix: install-tools check-workspace
@echo "$(BLUE)Applying code formatting fixes...$(NC)"
@$(CARGO) fmt --all
@$(CARGO_FMT) --all
@echo "$(GREEN)✓ Code formatting fixed in all crates$(NC)"

# ============================================================================
Expand Down Expand Up @@ -232,7 +235,7 @@ check: fmt-check clippy deny
quick-check: check-workspace
@echo "$(BLUE)Running quick validation...$(NC)"
@$(BREW_ROCKSDB_ENV) $(CARGO) check --workspace --lib --tests --all-features
@$(CARGO) fmt --all -- --check || exit 1
@$(CARGO_FMT) --all -- --check || exit 1
@echo "$(GREEN)✓ Quick validation passed$(NC)"

# ============================================================================
Expand All @@ -257,7 +260,7 @@ build-release: check check-workspace
# ============================================================================

## test Run all tests with nextest (fast, parallel)
test: install-tools check-workspace check-all-projects
test: install-tools check-workspace fmt-check check-all-projects
@echo "$(BLUE)Running all tests with nextest...$(NC)"
@CI=1 RUST_LOG=$(RUST_LOG_LEVEL) RUST_BACKTRACE=$(RUST_BACKTRACE) \
$(BREW_ROCKSDB_ENV) $(CARGO) nextest run --all-features --workspace --no-fail-fast
Expand Down Expand Up @@ -378,7 +381,7 @@ clean-deps: clean
# ============================================================================

## pre-release Full pre-release validation (comprehensive checks)
pre-release: install-tools check-workspace check docs-check test audit build-release
pre-release: install-tools check-workspace check docs-check fmt-check test audit build-release
@echo "$(BLUE)Checking version consistency across workspace...$(NC)"
@version=$$(grep '^version' d-engine/Cargo.toml | head -n1 | cut -d'"' -f2); \
for crate in d-engine-proto d-engine-core d-engine-client d-engine-server d-engine; do \
Expand Down
15 changes: 9 additions & 6 deletions d-engine-client/src/grpc_client.rs
Original file line number Diff line number Diff line change
@@ -1,13 +1,12 @@
use super::ClientInner;
use crate::ClientApiError;
use crate::ClientResponseExt;
use crate::scoped_timer::ScopedTimer;
use std::sync::Arc;

use arc_swap::ArcSwap;
use bytes::Bytes;
use d_engine_core::BatchOp;
use d_engine_core::client::ClientApi;
use d_engine_core::client::ClientApiResult;
use d_engine_core::client::ErrorCode;
use d_engine_core::client::KvEntry;
use d_engine_core::client::{ClientApi, ClientApiResult};
use d_engine_core::config::ReadConsistencyPolicy;
use d_engine_core::types::ScanResults;
use d_engine_proto::client::ClientReadRequest;
Expand All @@ -26,13 +25,17 @@ use d_engine_proto::client::write_command::batch_op;
use rand::Rng;
use rand::SeedableRng;
use rand::rngs::StdRng;
use std::sync::Arc;
use tonic::codec::CompressionEncoding;
use tonic::transport::Channel;
use tracing::debug;
use tracing::error;
use tracing::warn;

use super::ClientInner;
use crate::ClientApiError;
use crate::ClientResponseExt;
use crate::scoped_timer::ScopedTimer;

/// gRPC-based KV client for standalone mode. Obtained via [`crate::ClientBuilder`].
///
/// For embedded mode use `EmbeddedClient` instead. Both implement `ClientApi`.
Expand Down
12 changes: 8 additions & 4 deletions d-engine-client/src/grpc_client_test.rs
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@ use tracing_test::traced_test;
use crate::Client;
use crate::ClientConfig;
use crate::ClientInner;

use crate::ConnectionPool;
use crate::GrpcClient;
use crate::mock_rpc_service::MockNode;
Expand Down Expand Up @@ -1435,10 +1434,11 @@ mod cas_operations {
// =============================================================================

mod watch_membership_tests {
use super::*;
use d_engine_proto::client::MembershipSnapshot;
use tokio_stream::StreamExt;

use super::*;

async fn make_client(port: u16) -> GrpcClient {
let endpoints = vec![format!("http://localhost:{}", port)];
let config = ClientConfig::default();
Expand Down Expand Up @@ -1547,9 +1547,10 @@ mod watch_membership_tests {
// =============================================================================

mod watch_tests {
use super::*;
use tokio_stream::StreamExt;

use super::*;

async fn make_client(port: u16) -> GrpcClient {
let endpoints = vec![format!("http://localhost:{}", port)];
let config = ClientConfig::default();
Expand Down Expand Up @@ -1664,8 +1665,11 @@ mod scan_tests {
use tokio::sync::oneshot;
use tracing_test::traced_test;

use crate::ClientConfig;
use crate::ClientInner;
use crate::ConnectionPool;
use crate::GrpcClient;
use crate::mock_rpc_service::MockNode;
use crate::{ClientConfig, ClientInner, ConnectionPool, GrpcClient};

async fn make_client(port: u16) -> GrpcClient {
let endpoints = vec![format!("http://localhost:{port}")];
Expand Down
28 changes: 15 additions & 13 deletions d-engine-client/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -73,9 +73,11 @@ mod utils;

pub use builder::*;
pub use config::*;
pub use d_engine_core::client::ClientApi;
pub use d_engine_core::client::ClientApiError;
pub use d_engine_core::client::ClientApiResult;
pub use d_engine_core::client::ErrorCode;
pub use d_engine_core::client::ScanResults;
pub use d_engine_core::client::{ClientApi, ClientApiError, ClientApiResult};
pub use grpc_client::*;
pub(crate) use pool::*;

Expand Down Expand Up @@ -170,21 +172,21 @@ impl Client {
/// or `ClientConfig::cluster_ready_timeout` elapses.
///
/// **What this does:**
/// - Probes all endpoints in round-robin until one reports a ready leader
/// (`current_leader_id` is `Some` and present in the member list)
/// - Probes all endpoints in round-robin until one reports a ready leader (`current_leader_id`
/// is `Some` and present in the member list)
/// - Atomically replaces the cached leader/follower connections
/// - After `Ok(())`, `get_leader_id()` returns the new leader and all
/// write/read operations are routed to the correct node
/// - After `Ok(())`, `get_leader_id()` returns the new leader and all write/read operations are
/// routed to the correct node
///
/// **What this does NOT do:**
/// - Does not guarantee that the caller's in-flight requests succeeded;
/// requests sent before `refresh()` may have failed and need to be retried
/// - Writes are at-least-once across a failover: a write that was already proposed when
/// the leader stepped down may still be committed by the new leader even though the
/// caller received an error. Re-issuing a non-idempotent command can apply it twice
/// (the engine has no request-id dedup yet).
/// - Does not update endpoints permanently; pass `new_endpoints` to change
/// the bootstrap list for this and future refreshes
/// - Does not guarantee that the caller's in-flight requests succeeded; requests sent before
/// `refresh()` may have failed and need to be retried
/// - Writes are at-least-once across a failover: a write that was already proposed when the
/// leader stepped down may still be committed by the new leader even though the caller
/// received an error. Re-issuing a non-idempotent command can apply it twice (the engine has
/// no request-id dedup yet).
/// - Does not update endpoints permanently; pass `new_endpoints` to change the bootstrap list
/// for this and future refreshes
///
/// **Typical usage after leader failover:**
/// ```ignore
Expand Down
13 changes: 8 additions & 5 deletions d-engine-client/src/mock_rpc.rs
Original file line number Diff line number Diff line change
Expand Up @@ -66,11 +66,14 @@ impl MockRpcService {
/// Convert core `ClientResponse` to the proto wire type for test mock gRPC handlers.
/// Both `ErrorCode` enums are `#[repr(i32)]` with identical discriminants.
fn core_to_proto_response(r: CoreClientResponse) -> ClientResponse {
use d_engine_core::client::{ClientResponsePayload, WriteResult};
use d_engine_proto::client::{
ClientResult, KvEntry, ReadResults as ProtoReadResults, ScanResults as ProtoScanResults,
WriteResult as ProtoWriteResult, client_response::SuccessResult,
};
use d_engine_core::client::ClientResponsePayload;
use d_engine_core::client::WriteResult;
use d_engine_proto::client::ClientResult;
use d_engine_proto::client::KvEntry;
use d_engine_proto::client::ReadResults as ProtoReadResults;
use d_engine_proto::client::ScanResults as ProtoScanResults;
use d_engine_proto::client::WriteResult as ProtoWriteResult;
use d_engine_proto::client::client_response::SuccessResult;
use d_engine_proto::error::ErrorMetadata;

let error_code = r.error as i32;
Expand Down
3 changes: 2 additions & 1 deletion d-engine-client/src/mock_rpc_service.rs
Original file line number Diff line number Diff line change
Expand Up @@ -187,7 +187,8 @@ impl MockNode {
///
/// # Parameters
/// - `rx`: Shutdown signal receiver
/// - `cas_should_succeed`: true = CAS succeeds (comparison matched), false = CAS fails (mismatch)
/// - `cas_should_succeed`: true = CAS succeeds (comparison matched), false = CAS fails
/// (mismatch)
///
/// # Returns
/// - Channel and port for the mock server
Expand Down
8 changes: 4 additions & 4 deletions d-engine-client/src/pool.rs
Original file line number Diff line number Diff line change
Expand Up @@ -85,8 +85,8 @@ impl ConnectionPool {
config: &ClientConfig,
) -> std::result::Result<(Channel, Vec<Channel>, Vec<NodeMeta>, Option<u32>), ClientApiError>
{
// 1. Load cluster metadata + establish verified leader channel atomically.
// Returns only when a ready leader is confirmed AND its channel is connectable.
// 1. Load cluster metadata + establish verified leader channel atomically. Returns only
// when a ready leader is confirmed AND its channel is connectable.
let (membership, leader_conn) = Self::load_cluster_metadata(endpoints, config).await?;
info!("Cluster members discovered: {:?}", membership.nodes);

Expand Down Expand Up @@ -151,8 +151,8 @@ impl ConnectionPool {
///
/// Returns:
/// - `Some(Ok(membership))` — leader is known and present in member list (ready)
/// - `Some(Err(()))` — node responded but cluster not yet ready (election in progress
/// or stale leader ID); caller should try next node
/// - `Some(Err(()))` — node responded but cluster not yet ready (election in progress or
/// stale leader ID); caller should try next node
/// - `None` — node unreachable; caller should try next node
pub(super) async fn probe_endpoint(
addr: &str,
Expand Down
8 changes: 4 additions & 4 deletions d-engine-client/src/proto/client_ext_test.rs
Original file line number Diff line number Diff line change
Expand Up @@ -286,10 +286,10 @@ fn test_into_scan_results_success() {

let scan = response.into_scan_results().unwrap();
assert_eq!(scan.revision, 7);
assert_eq!(
scan.entries,
vec![(Bytes::from("/services/node1"), Bytes::from("10.0.0.1"))]
);
assert_eq!(scan.entries, vec![(
Bytes::from("/services/node1"),
Bytes::from("10.0.0.1")
)]);
}

#[test]
Expand Down
Loading
Loading