Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 0 additions & 5 deletions .github/workflows/benchmark.yml
Original file line number Diff line number Diff line change
Expand Up @@ -47,11 +47,6 @@ jobs:
echo "Running d-engine Performance Benchmarks"
echo "========================================="
echo ""
echo "Performance Targets:"
echo " • Without TTL: < 10ns overhead"
echo " • With TTL passive check: < 50ns overhead"
echo " • Piggyback cleanup: < 1ms"
echo ""

# Run benchmarks and save output
cargo bench --package d-engine-server 2>&1 | tee benchmark_output.txt
Expand Down
14 changes: 11 additions & 3 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ All notable changes to this project will be documented in this file.

---

## [v0.2.5] - 2026-XX-XX
## [v0.3.0] - 2026-XX-XX

### Added

Expand All @@ -28,8 +28,6 @@ All notable changes to this project will be documented in this file.

### Fixed

- **fix(ttl) #398**: Removed `lease.enabled` flag — TTL expiration is always active. Fixes fatal crash when calling `put_with_ttl` without setting the (now-removed) `lease.enabled = true`.

- **🛑 Snapshot data corruption — label/data mismatch** (#418): `last_included` in snapshot metadata was computed by subtracting `retained_log_entries` from `last_applied`, but the snapshot data already reflected the full `last_applied` state. Followers installing such snapshots would re-apply entries 91–100 after restoring state through index 100, causing double-application and permanent cluster state divergence. Also fixes a fabricated `LogId` where `last_included.term` was copied from `last_applied.term` instead of queried from the log. Snapshot label is now always truthful: `last_included == last_applied`.

- **`ClusterConfig::default()` now consistent with serde default** (#415): `Default::default()` previously produced an empty `initial_cluster`, but `#[serde(default)]` returned `[{id:1, ...}]`. `RaftNodeConfig::new()` uses the Rust `Default` impl, so programmatic configs always got an empty cluster. Fixed.
Expand All @@ -56,8 +54,12 @@ All notable changes to this project will be documented in this file.

- **Quorum confirmation no longer derives from replicated history** (#423): In a 5+ voter cluster, a leader cut off from the majority but still reached by one follower kept renewing its read lease and never stepped down, because the commit median over every peer's stored `match_index` stayed true. A leader is now confirmed only when a quorum of voters acknowledged a newer heartbeat round; unreachable peers stop counting. Applies to lease reads, queued linearizable reads, CheckQuorum, write admission and the vote guard.

- **File state machine lost empty values on WAL replay** (#464): `replay_wal` treated a zero value length as "no value", so an `Insert` with an empty value was dropped after a restart. It now restores the empty value.

### Changed

- **⚠️ Wire protocol is not compatible with v0.2.4**: `ClientResponse.metadata` moved from field 4 to 5 (field 4 is now `scan_data`); `HandleClientScan` returns `ClientResponse` (the `ScanResponse` message is now `ScanResults`); the `ChunkStatus` values are renumbered with `CHUNK_STATUS_UNSPECIFIED = 0`; `WriteCommand.Insert.ttl_secs` (field 3) is reserved; new RPC `PreVote`. Do not run v0.2.4 and v0.3.0 nodes or clients together. See `MIGRATION_GUIDE.md`.

- **⚠️ `StateMachine::stop()` removed; `StateMachine::close_storage()` renamed to `close()`** (#455): `stop()` had no caller in d-engine. `close()` is now the single, permanent, idempotent end-of-life hook and also persists what must survive a restart (lease state). It returns `Result<(), Error>`: release your resources even if persisting fails, and return the first error; the node logs it at ERROR level. Custom implementations: delete `stop()`, rename `close_storage()` to `close()`, and move any lease persistence from `stop()` into `close()`. See `MIGRATION_GUIDE.md`.

- **State machine worker lifecycle** (#455): the `NodeBuilder` only assembles the worker; `Node::run()` starts it and stops it. `StateMachineWorker::new` no longer takes a shutdown signal.
Expand Down Expand Up @@ -114,6 +116,12 @@ All notable changes to this project will be documented in this file.
"MemFirst"` or `"DiskFirst"` are silently ignored, not an error — remove the field, `flush_policy`
is the only persistence knob now.

### Removed

- **TTL and per-key lease** (#464): d-engine now does consensus and replication only. Removed `put_with_ttl`, the `Lease` trait, `TtlLease`, the `[raft.state_machine.lease]` config section and `StateMachine::lease_background_cleanup`. `Command::Insert` no longer has `ttl_secs`. In the proto, `WriteCommand.Insert.ttl_secs` (field 3) is removed and the number is reserved. Key expiry is now the job of the layer above d-engine. The leader read lease (`LeaseRead`) is a different feature and is unchanged.

- **File state machine storage format** (#464): The WAL record no longer has an `expire_at` slot, and `snapshot.bin` no longer ends with a TTL section. See [MIGRATION_GUIDE.md](./MIGRATION_GUIDE.md).

---

## [v0.2.4] - 2026-05-23
Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ We're in **active development** based on real user feedback. Our priorities evol

## Versioning Policy

Starting after v0.2.5:
Starting after v0.2.4:

- **Patch (0.2.x → 0.2.x+1)**: bug fixes, docs, performance, internal-only changes — no breaking changes
- **Minor (0.2.x → 0.3.0)**: breaking API/behavior changes allowed
Expand Down
20 changes: 5 additions & 15 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

82 changes: 42 additions & 40 deletions MIGRATION_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,34 @@

---

## Upgrading to v0.3.0: TTL Removed

- Remove calls to `put_with_ttl` and the `ttl_secs` field of `Command::Insert`. Implement key expiry in the layer above d-engine.
Comment thread
JoshuaChi marked this conversation as resolved.
Resolved
- Delete the `[raft.state_machine.lease]` section from your config files.
- Custom state machines: remove `lease_background_cleanup`.
- Proto: `WriteCommand.Insert.ttl_secs` (field 3) is removed. Regenerate your client code.
- The file state machine storage format changed. Start with a fresh data directory; old files are not supported.
- RocksDB state machine: keys written with `put_with_ttl` under v0.2.x are kept forever after the upgrade. v0.3.0 neither reads the old `ttl_state` entry nor deletes expired keys. Before upgrading, either delete the keys you no longer want, or record their deadlines and have your own expiry layer delete them. Or start with a fresh data directory.

---

## For v0.2.4 Users: Wire Protocol Changes in v0.3.0

v0.3.0 nodes and clients cannot talk to v0.2.4 ones. There is no rolling upgrade: stop the whole cluster, upgrade every node, then upgrade every client.

| Proto item | v0.2.4 | v0.3.0 |
| ------------------------------- | ---------------------------- | ------------------------------------------------------------------ |
| `ClientResponse.metadata` | field 4 | field 5 (field 4 is now `scan_data`) |
| `HandleClientScan` response | `ScanResponse` | `ClientResponse`; the scan payload is the `ScanResults` message |
| `ChunkStatus` | `ACCEPTED = 0` … `FAILED = 4` | `CHUNK_STATUS_UNSPECIFIED = 0`, `CHUNK_STATUS_ACCEPTED = 1` … `CHUNK_STATUS_FAILED = 5` |
| `WriteCommand.Insert.ttl_secs` | field 3 | removed, number 3 and the name are reserved |
| `StreamSnapshot` RPC | present | removed (see the learner snapshot section below) |
| `PreVote` RPC | - | new |

Clients in other languages: regenerate your stubs from the v0.3.0 `.proto` files. Rust users get this by upgrading the crate.

---

## 🚨 For v0.1.x Users: WAL Format Change

### What Changed
Expand Down Expand Up @@ -134,7 +162,7 @@ grep "WAL" /var/log/d-engine.log
| v0.2.0–v0.2.2 | Absolute expiration | Compatible | ✅ Yes (clear WAL from v0.1.x) |
| v0.2.3 | Same as v0.2.0+ | **Incompatible** | ✅ Yes (protobuf enum changes + API changes) |
| v0.2.4 | Same as v0.2.0+ | Compatible (additive) | ✅ Yes (delete `snapshot/` — format changed to CF export) |
| v0.2.5 | Same as v0.2.0+ | Compatible (additive) | ⚠️ Minor (remove `lease.enabled`; remove `entry_term()` from custom SM; use `start_node` instead of `start_custom_with_config`) |
| v0.3.0 | File SM format changed | **Incompatible** (proto field and enum renumbering; upgrade all nodes and clients together) | ⚠️ Yes (TTL removed; fresh data directory; custom SM: remove `entry_term()`, `lease_background_cleanup`, rename `close_storage()` to `close()`; use `start_node` instead of `start_custom_with_config`) |

---

Expand Down Expand Up @@ -352,11 +380,11 @@ fn apply_chunk(&mut self, entries: &[ApplyEntry]) -> Result<(), ...>

---

## 🚨 For v0.2.4 Users: `StateMachine` Lifecycle Changes in v0.2.5 (#455)
## 🚨 For v0.2.4 Users: `StateMachine` Lifecycle Changes in v0.3.0 (#455)

### What Changed

| Before (v0.2.4) | After (v0.2.5) |
| Before (v0.2.4) | After (v0.3.0) |
| ---------------------------------------------------------------- | ----------------------------------------------------------------- |
| `fn stop(&self) -> Result<(), Error>` (required) | **Removed** |
| `fn close_storage(&self)` (default no-op) | **Renamed** `fn close(&self) -> Result<(), Error>` (default `Ok`) |
Expand Down Expand Up @@ -534,7 +562,7 @@ See [Watch Feature Guide](https://docs.rs/d-engine/latest/d_engine/docs/server_g

---

## For v0.2.4 Users: `StateMachine::entry_term()` Removed in v0.2.5 (#418)
## For v0.2.4 Users: `StateMachine::entry_term()` Removed in v0.3.0 (#418)

### What Changed

Expand All @@ -560,7 +588,7 @@ Compilation will fail with "method not found in trait" if you forget to remove i

---

## For v0.2.4 Users: `StateMachine::apply_snapshot_from_file` Return Type Changed in v0.2.5 (#436)
## For v0.2.4 Users: `StateMachine::apply_snapshot_from_file` Return Type Changed in v0.3.0 (#436)

### What Changed

Expand Down Expand Up @@ -591,7 +619,7 @@ async fn apply_snapshot_from_file(
Ok(())
}

// New (v0.2.5)
// New (v0.3.0)
async fn apply_snapshot_from_file(
&self,
metadata: &SnapshotMetadata,
Expand All @@ -615,7 +643,7 @@ Compilation fails with "incompatible type for trait" until a custom implementati

---

## For v0.2.4 Users: Learner Initial-Snapshot PULL Path Removed in v0.2.5 (#436)
## For v0.2.4 Users: Learner Initial-Snapshot PULL Path Removed in v0.3.0 (#436)

### What Changed

Expand Down Expand Up @@ -649,7 +677,7 @@ The public API now exposes `start_node()` directly instead of routing through a
// Old (v0.2.4) — removed
EmbeddedEngine::start_custom_with_config(storage, sm, config).await?;

// New (v0.2.5)
// New (v0.3.0)
EmbeddedEngine::start_node(config, storage, sm).await?; // embedded
StandaloneEngine::start_node(config, storage, sm, shutdown_rx).await?; // standalone
```
Expand All @@ -669,38 +697,12 @@ always.
### Impact

Transparent — no action required. Nodes that installed snapshots with the old label may have
double-applied entries; upgrading to v0.2.5 prevents this from recurring. Existing snapshots
double-applied entries; upgrading to v0.3.0 prevents this from recurring. Existing snapshots
Comment thread
JoshuaChi marked this conversation as resolved.
Resolved
are compatible.

---

## For v0.2.4 Users: TTL Config Change in v0.2.5 (#398)

### What Changed

The `enabled` flag under `[raft.state_machine.lease]` has been removed. TTL expiration is now **always active** — no opt-in required.

In v0.2.4, omitting `enabled = true` caused a fatal crash on `put_with_ttl`. This bug is fixed in v0.2.5.

### Migration

If your config contains `enabled = true` or `enabled = false`, remove the line:

```toml
# Old (v0.2.4) — remove this line
[raft.state_machine.lease]
enabled = true # ← delete

# New (v0.2.5) — TTL always active, no flag needed
[raft.state_machine.lease]
cleanup_interval_ms = 1000
```

**Impact**: None if you do not touch the config — unrecognised fields are ignored. The only behavioral change is that TTL expiration is now unconditionally enabled.

---

## For v0.2.4 Users: New `[raft.read_actor]` Config Section in v0.2.5 (#392)
## For v0.2.4 Users: New `[raft.read_actor]` Config Section in v0.3.0 (#392)

This section is **optional** — both fields have defaults and existing configs work without changes.

Expand All @@ -718,7 +720,7 @@ No migration action required unless you want to tune read concurrency.

---

## For v0.2.4 Users: `data_dir` moved out of config in v0.2.5
## For v0.2.4 Users: `data_dir` moved out of config in v0.3.0

`data_dir` (formerly `db_root_dir`) is no longer a config file field. Pass it as an explicit argument to the engine constructor instead:

Expand All @@ -727,7 +729,7 @@ No migration action required unless you want to tune read concurrency.
let engine = EmbeddedEngine::start_with("config.toml").await?;
// config.toml had [cluster] data_dir = "./db"

// After (v0.2.5)
// After (v0.3.0)
let engine = EmbeddedEngine::start_with("./db", "config.toml").await?;
```

Expand All @@ -739,7 +741,7 @@ All engine constructors (`start`, `start_with`, `start_custom`, `start_node`, `r

---

## For v0.2.4 Users: `NodeBuilder` is no longer public in v0.2.5
## For v0.2.4 Users: `NodeBuilder` is no longer public in v0.3.0

Only affects code that plugged in a custom storage engine or state machine directly via `NodeBuilder`. If you used `EmbeddedEngine` or `StandaloneEngine`, no change needed.

Expand All @@ -752,7 +754,7 @@ let node = NodeBuilder::new(data_dir, None, shutdown_rx)
.await?;
node.run().await?;

// After (v0.2.5)
// After (v0.3.0)
EmbeddedEngine::start_custom(data_dir, storage_engine, state_machine, None).await?;
// or, for a standalone gRPC server:
StandaloneEngine::run_custom(data_dir, storage_engine, state_machine, shutdown_rx, None).await?;
Expand Down
7 changes: 3 additions & 4 deletions benches/embedded-bench/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
# ===============================
# Storage engine under test. Default: Raft log in raft-engine (experimental).
# ENGINE=rocksdb builds and runs the stock RocksDB-WAL binary, which is the
# configuration of the v0.2.4 / v0.2.5 reports. Each engine has its own target dir
# configuration of the v0.2.4 / v0.3.0 reports. Each engine has its own target dir
# and its own data dir, so switching never mixes their files.
ENGINE ?= raft-engine
$(if $(filter raft-engine rocksdb,$(ENGINE)),,$(error ENGINE must be raft-engine or rocksdb, got '$(ENGINE)'))
Expand Down Expand Up @@ -44,8 +44,7 @@ VERIFY_FLAG := $(if $(filter true,$(VERIFY)),--verify-write,)

# RAM disk (macOS only, optional) — isolates this node's storage on its own
# independent RAM disk volume, to strip physical-disk latency out of a
# benchmark. Not a general fix for unrelated flakiness — see
# tickets/milestones/v0.2.5/446-perf-batching-measurement-2026-09-13.md.
# benchmark. Not a general fix for unrelated flakiness.
# One-shot: `make all-ram-tests`. Manual/single-node: add RAMDISK=true to
# any test-* target (needs the other two nodes already running for quorum).
RAMDISK ?= false
Expand Down Expand Up @@ -312,7 +311,7 @@ all-tests: build
# Isolates each node's storage on its own independent RAM disk volume — use
# this to strip physical-disk latency out of a benchmark (e.g. to study
# fsync/scheduling behavior in isolation), not as a general fix for
# unrelated flakiness. See tickets/milestones/v0.2.5/446-perf-batching-measurement-2026-09-13.md.
# unrelated flakiness.

# Ensures the RAM disks exist, then runs the full --batch suite for this
# node on its own RAM disk volume — same scope/shape as `make all-tests
Expand Down
Binary file not shown.
Binary file not shown.
Binary file not shown.
Loading
Loading