diff --git a/.github/workflows/benchmark.yml b/.github/workflows/benchmark.yml index c9e5c45a..bc939fbc 100644 --- a/.github/workflows/benchmark.yml +++ b/.github/workflows/benchmark.yml @@ -47,11 +47,6 @@ jobs: echo "Running d-engine Performance Benchmarks" echo "=========================================" echo "" - echo "Performance Targets:" - echo " • Without TTL: < 10ns overhead" - echo " • With TTL passive check: < 50ns overhead" - echo " • Piggyback cleanup: < 1ms" - echo "" # Run benchmarks and save output cargo bench --package d-engine-server 2>&1 | tee benchmark_output.txt diff --git a/CHANGELOG.md b/CHANGELOG.md index 4847a933..8fbb698b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,7 @@ All notable changes to this project will be documented in this file. --- -## [v0.2.5] - 2026-XX-XX +## [v0.3.0] - 2026-XX-XX ### Added @@ -28,8 +28,6 @@ All notable changes to this project will be documented in this file. ### Fixed -- **fix(ttl) #398**: Removed `lease.enabled` flag — TTL expiration is always active. Fixes fatal crash when calling `put_with_ttl` without setting the (now-removed) `lease.enabled = true`. - - **🛑 Snapshot data corruption — label/data mismatch** (#418): `last_included` in snapshot metadata was computed by subtracting `retained_log_entries` from `last_applied`, but the snapshot data already reflected the full `last_applied` state. Followers installing such snapshots would re-apply entries 91–100 after restoring state through index 100, causing double-application and permanent cluster state divergence. Also fixes a fabricated `LogId` where `last_included.term` was copied from `last_applied.term` instead of queried from the log. Snapshot label is now always truthful: `last_included == last_applied`. - **`ClusterConfig::default()` now consistent with serde default** (#415): `Default::default()` previously produced an empty `initial_cluster`, but `#[serde(default)]` returned `[{id:1, ...}]`. `RaftNodeConfig::new()` uses the Rust `Default` impl, so programmatic configs always got an empty cluster. Fixed. @@ -56,8 +54,12 @@ All notable changes to this project will be documented in this file. - **Quorum confirmation no longer derives from replicated history** (#423): In a 5+ voter cluster, a leader cut off from the majority but still reached by one follower kept renewing its read lease and never stepped down, because the commit median over every peer's stored `match_index` stayed true. A leader is now confirmed only when a quorum of voters acknowledged a newer heartbeat round; unreachable peers stop counting. Applies to lease reads, queued linearizable reads, CheckQuorum, write admission and the vote guard. +- **File state machine lost empty values on WAL replay** (#464): `replay_wal` treated a zero value length as "no value", so an `Insert` with an empty value was dropped after a restart. It now restores the empty value. + ### Changed +- **⚠️ Wire protocol is not compatible with v0.2.4**: `ClientResponse.metadata` moved from field 4 to 5 (field 4 is now `scan_data`); `HandleClientScan` returns `ClientResponse` (the `ScanResponse` message is now `ScanResults`); the `ChunkStatus` values are renumbered with `CHUNK_STATUS_UNSPECIFIED = 0`; `WriteCommand.Insert.ttl_secs` (field 3) is reserved; new RPC `PreVote`. Do not run v0.2.4 and v0.3.0 nodes or clients together. See `MIGRATION_GUIDE.md`. + - **⚠️ `StateMachine::stop()` removed; `StateMachine::close_storage()` renamed to `close()`** (#455): `stop()` had no caller in d-engine. `close()` is now the single, permanent, idempotent end-of-life hook and also persists what must survive a restart (lease state). It returns `Result<(), Error>`: release your resources even if persisting fails, and return the first error; the node logs it at ERROR level. Custom implementations: delete `stop()`, rename `close_storage()` to `close()`, and move any lease persistence from `stop()` into `close()`. See `MIGRATION_GUIDE.md`. - **State machine worker lifecycle** (#455): the `NodeBuilder` only assembles the worker; `Node::run()` starts it and stops it. `StateMachineWorker::new` no longer takes a shutdown signal. @@ -114,6 +116,12 @@ All notable changes to this project will be documented in this file. "MemFirst"` or `"DiskFirst"` are silently ignored, not an error — remove the field, `flush_policy` is the only persistence knob now. +### Removed + +- **TTL and per-key lease** (#464): d-engine now does consensus and replication only. Removed `put_with_ttl`, the `Lease` trait, `TtlLease`, the `[raft.state_machine.lease]` config section and `StateMachine::lease_background_cleanup`. `Command::Insert` no longer has `ttl_secs`. In the proto, `WriteCommand.Insert.ttl_secs` (field 3) is removed and the number is reserved. Key expiry is now the job of the layer above d-engine. The leader read lease (`LeaseRead`) is a different feature and is unchanged. + +- **File state machine storage format** (#464): The WAL record no longer has an `expire_at` slot, and `snapshot.bin` no longer ends with a TTL section. See [MIGRATION_GUIDE.md](./MIGRATION_GUIDE.md). + --- ## [v0.2.4] - 2026-05-23 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 6ed80a8e..a0dd611a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -63,7 +63,7 @@ We're in **active development** based on real user feedback. Our priorities evol ## Versioning Policy -Starting after v0.2.5: +Starting after v0.2.4: - **Patch (0.2.x → 0.2.x+1)**: bug fixes, docs, performance, internal-only changes — no breaking changes - **Minor (0.2.x → 0.3.0)**: breaking API/behavior changes allowed diff --git a/Cargo.lock b/Cargo.lock index bbd536d8..cf1e468e 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -190,7 +190,7 @@ dependencies = [ "bitflags", "cexpr", "clang-sys", - "itertools 0.13.0", + "itertools 0.10.5", "proc-macro2", "quote", "regex", @@ -1107,15 +1107,6 @@ dependencies = [ "either", ] -[[package]] -name = "itertools" -version = "0.13.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "413ee7dfc52ee1a4949ceeb7dbc8a33f2d6c088194d9f922fb8318faf1f01186" -dependencies = [ - "either", -] - [[package]] name = "itertools" version = "0.14.0" @@ -1183,9 +1174,9 @@ dependencies = [ [[package]] name = "libz-sys" -version = "1.1.28" +version = "1.1.31" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fc3a226e576f50782b3305c5ccf458698f92798987f551c6a02efe8276721e22" +checksum = "ff20308ecabd635a2dc1dd5834a86c462cca20efba1d5d3ddab1f8aea3455cb2" dependencies = [ "cc", "pkg-config", @@ -2909,11 +2900,10 @@ checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa" [[package]] name = "zstd-sys" -version = "2.0.16+zstd.1.5.7" +version = "2.1.2+zstd.1.5.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "91e19ebc2adc8f83e43039e79776e3fda8ca919132d68a1fed6a5faca2683748" +checksum = "481ada9c543b2323cbb7bbdbb5249177fbf0188ed959ed2c15e4b22421c84ed3" dependencies = [ - "bindgen", "cc", "pkg-config", ] diff --git a/MIGRATION_GUIDE.md b/MIGRATION_GUIDE.md index 97480ea8..fbbac79d 100644 --- a/MIGRATION_GUIDE.md +++ b/MIGRATION_GUIDE.md @@ -6,6 +6,34 @@ --- +## Upgrading to v0.3.0: TTL Removed + +- Remove calls to `put_with_ttl` and the `ttl_secs` field of `Command::Insert`. Implement key expiry in the layer above d-engine. +- Delete the `[raft.state_machine.lease]` section from your config files. +- Custom state machines: remove `lease_background_cleanup`. +- Proto: `WriteCommand.Insert.ttl_secs` (field 3) is removed. Regenerate your client code. +- The file state machine storage format changed. Start with a fresh data directory; old files are not supported. +- RocksDB state machine: keys written with `put_with_ttl` under v0.2.x are kept forever after the upgrade. v0.3.0 neither reads the old `ttl_state` entry nor deletes expired keys. Before upgrading, either delete the keys you no longer want, or record their deadlines and have your own expiry layer delete them. Or start with a fresh data directory. + +--- + +## For v0.2.4 Users: Wire Protocol Changes in v0.3.0 + +v0.3.0 nodes and clients cannot talk to v0.2.4 ones. There is no rolling upgrade: stop the whole cluster, upgrade every node, then upgrade every client. + +| Proto item | v0.2.4 | v0.3.0 | +| ------------------------------- | ---------------------------- | ------------------------------------------------------------------ | +| `ClientResponse.metadata` | field 4 | field 5 (field 4 is now `scan_data`) | +| `HandleClientScan` response | `ScanResponse` | `ClientResponse`; the scan payload is the `ScanResults` message | +| `ChunkStatus` | `ACCEPTED = 0` … `FAILED = 4` | `CHUNK_STATUS_UNSPECIFIED = 0`, `CHUNK_STATUS_ACCEPTED = 1` … `CHUNK_STATUS_FAILED = 5` | +| `WriteCommand.Insert.ttl_secs` | field 3 | removed, number 3 and the name are reserved | +| `StreamSnapshot` RPC | present | removed (see the learner snapshot section below) | +| `PreVote` RPC | - | new | + +Clients in other languages: regenerate your stubs from the v0.3.0 `.proto` files. Rust users get this by upgrading the crate. + +--- + ## 🚨 For v0.1.x Users: WAL Format Change ### What Changed @@ -134,7 +162,7 @@ grep "WAL" /var/log/d-engine.log | v0.2.0–v0.2.2 | Absolute expiration | Compatible | ✅ Yes (clear WAL from v0.1.x) | | v0.2.3 | Same as v0.2.0+ | **Incompatible** | ✅ Yes (protobuf enum changes + API changes) | | v0.2.4 | Same as v0.2.0+ | Compatible (additive) | ✅ Yes (delete `snapshot/` — format changed to CF export) | -| v0.2.5 | Same as v0.2.0+ | Compatible (additive) | ⚠️ Minor (remove `lease.enabled`; remove `entry_term()` from custom SM; use `start_node` instead of `start_custom_with_config`) | +| v0.3.0 | File SM format changed | **Incompatible** (proto field and enum renumbering; upgrade all nodes and clients together) | ⚠️ Yes (TTL removed; fresh data directory; custom SM: remove `entry_term()`, `lease_background_cleanup`, rename `close_storage()` to `close()`; use `start_node` instead of `start_custom_with_config`) | --- @@ -352,11 +380,11 @@ fn apply_chunk(&mut self, entries: &[ApplyEntry]) -> Result<(), ...> --- -## 🚨 For v0.2.4 Users: `StateMachine` Lifecycle Changes in v0.2.5 (#455) +## 🚨 For v0.2.4 Users: `StateMachine` Lifecycle Changes in v0.3.0 (#455) ### What Changed -| Before (v0.2.4) | After (v0.2.5) | +| Before (v0.2.4) | After (v0.3.0) | | ---------------------------------------------------------------- | ----------------------------------------------------------------- | | `fn stop(&self) -> Result<(), Error>` (required) | **Removed** | | `fn close_storage(&self)` (default no-op) | **Renamed** `fn close(&self) -> Result<(), Error>` (default `Ok`) | @@ -534,7 +562,7 @@ See [Watch Feature Guide](https://docs.rs/d-engine/latest/d_engine/docs/server_g --- -## For v0.2.4 Users: `StateMachine::entry_term()` Removed in v0.2.5 (#418) +## For v0.2.4 Users: `StateMachine::entry_term()` Removed in v0.3.0 (#418) ### What Changed @@ -560,7 +588,7 @@ Compilation will fail with "method not found in trait" if you forget to remove i --- -## For v0.2.4 Users: `StateMachine::apply_snapshot_from_file` Return Type Changed in v0.2.5 (#436) +## For v0.2.4 Users: `StateMachine::apply_snapshot_from_file` Return Type Changed in v0.3.0 (#436) ### What Changed @@ -591,7 +619,7 @@ async fn apply_snapshot_from_file( Ok(()) } -// New (v0.2.5) +// New (v0.3.0) async fn apply_snapshot_from_file( &self, metadata: &SnapshotMetadata, @@ -615,7 +643,7 @@ Compilation fails with "incompatible type for trait" until a custom implementati --- -## For v0.2.4 Users: Learner Initial-Snapshot PULL Path Removed in v0.2.5 (#436) +## For v0.2.4 Users: Learner Initial-Snapshot PULL Path Removed in v0.3.0 (#436) ### What Changed @@ -649,7 +677,7 @@ The public API now exposes `start_node()` directly instead of routing through a // Old (v0.2.4) — removed EmbeddedEngine::start_custom_with_config(storage, sm, config).await?; -// New (v0.2.5) +// New (v0.3.0) EmbeddedEngine::start_node(config, storage, sm).await?; // embedded StandaloneEngine::start_node(config, storage, sm, shutdown_rx).await?; // standalone ``` @@ -669,38 +697,12 @@ always. ### Impact Transparent — no action required. Nodes that installed snapshots with the old label may have -double-applied entries; upgrading to v0.2.5 prevents this from recurring. Existing snapshots +double-applied entries; upgrading to v0.3.0 prevents this from recurring. Existing snapshots are compatible. --- -## For v0.2.4 Users: TTL Config Change in v0.2.5 (#398) - -### What Changed - -The `enabled` flag under `[raft.state_machine.lease]` has been removed. TTL expiration is now **always active** — no opt-in required. - -In v0.2.4, omitting `enabled = true` caused a fatal crash on `put_with_ttl`. This bug is fixed in v0.2.5. - -### Migration - -If your config contains `enabled = true` or `enabled = false`, remove the line: - -```toml -# Old (v0.2.4) — remove this line -[raft.state_machine.lease] -enabled = true # ← delete - -# New (v0.2.5) — TTL always active, no flag needed -[raft.state_machine.lease] -cleanup_interval_ms = 1000 -``` - -**Impact**: None if you do not touch the config — unrecognised fields are ignored. The only behavioral change is that TTL expiration is now unconditionally enabled. - ---- - -## For v0.2.4 Users: New `[raft.read_actor]` Config Section in v0.2.5 (#392) +## For v0.2.4 Users: New `[raft.read_actor]` Config Section in v0.3.0 (#392) This section is **optional** — both fields have defaults and existing configs work without changes. @@ -718,7 +720,7 @@ No migration action required unless you want to tune read concurrency. --- -## For v0.2.4 Users: `data_dir` moved out of config in v0.2.5 +## For v0.2.4 Users: `data_dir` moved out of config in v0.3.0 `data_dir` (formerly `db_root_dir`) is no longer a config file field. Pass it as an explicit argument to the engine constructor instead: @@ -727,7 +729,7 @@ No migration action required unless you want to tune read concurrency. let engine = EmbeddedEngine::start_with("config.toml").await?; // config.toml had [cluster] data_dir = "./db" -// After (v0.2.5) +// After (v0.3.0) let engine = EmbeddedEngine::start_with("./db", "config.toml").await?; ``` @@ -739,7 +741,7 @@ All engine constructors (`start`, `start_with`, `start_custom`, `start_node`, `r --- -## For v0.2.4 Users: `NodeBuilder` is no longer public in v0.2.5 +## For v0.2.4 Users: `NodeBuilder` is no longer public in v0.3.0 Only affects code that plugged in a custom storage engine or state machine directly via `NodeBuilder`. If you used `EmbeddedEngine` or `StandaloneEngine`, no change needed. @@ -752,7 +754,7 @@ let node = NodeBuilder::new(data_dir, None, shutdown_rx) .await?; node.run().await?; -// After (v0.2.5) +// After (v0.3.0) EmbeddedEngine::start_custom(data_dir, storage_engine, state_machine, None).await?; // or, for a standalone gRPC server: StandaloneEngine::run_custom(data_dir, storage_engine, state_machine, shutdown_rx, None).await?; diff --git a/benches/embedded-bench/Makefile b/benches/embedded-bench/Makefile index 11237e73..d307c9ff 100644 --- a/benches/embedded-bench/Makefile +++ b/benches/embedded-bench/Makefile @@ -9,7 +9,7 @@ # =============================== # Storage engine under test. Default: Raft log in raft-engine (experimental). # ENGINE=rocksdb builds and runs the stock RocksDB-WAL binary, which is the -# configuration of the v0.2.4 / v0.2.5 reports. Each engine has its own target dir +# configuration of the v0.2.4 / v0.3.0 reports. Each engine has its own target dir # and its own data dir, so switching never mixes their files. ENGINE ?= raft-engine $(if $(filter raft-engine rocksdb,$(ENGINE)),,$(error ENGINE must be raft-engine or rocksdb, got '$(ENGINE)')) @@ -44,8 +44,7 @@ VERIFY_FLAG := $(if $(filter true,$(VERIFY)),--verify-write,) # RAM disk (macOS only, optional) — isolates this node's storage on its own # independent RAM disk volume, to strip physical-disk latency out of a -# benchmark. Not a general fix for unrelated flakiness — see -# tickets/milestones/v0.2.5/446-perf-batching-measurement-2026-09-13.md. +# benchmark. Not a general fix for unrelated flakiness. # One-shot: `make all-ram-tests`. Manual/single-node: add RAMDISK=true to # any test-* target (needs the other two nodes already running for quorum). RAMDISK ?= false @@ -312,7 +311,7 @@ all-tests: build # Isolates each node's storage on its own independent RAM disk volume — use # this to strip physical-disk latency out of a benchmark (e.g. to study # fsync/scheduling behavior in isolation), not as a general fix for -# unrelated flakiness. See tickets/milestones/v0.2.5/446-perf-batching-measurement-2026-09-13.md. +# unrelated flakiness. # Ensures the RAM disks exist, then runs the full --batch suite for this # node on its own RAM disk volume — same scope/shape as `make all-tests diff --git a/benches/reports/v0.2.5/d-engine_comparison_v0.2.5.png b/benches/reports/v0.2.5/d-engine_comparison_v0.2.5.png deleted file mode 100644 index dccb0b6d..00000000 Binary files a/benches/reports/v0.2.5/d-engine_comparison_v0.2.5.png and /dev/null differ diff --git a/benches/reports/v0.2.5/d-engine_v0.2.5_vs_v0.2.4_embedded_mode.png b/benches/reports/v0.2.5/d-engine_v0.2.5_vs_v0.2.4_embedded_mode.png deleted file mode 100644 index 960c2387..00000000 Binary files a/benches/reports/v0.2.5/d-engine_v0.2.5_vs_v0.2.4_embedded_mode.png and /dev/null differ diff --git a/benches/reports/v0.2.5/d-engine_v0.2.5_vs_v0.2.4_standalone_mode.png b/benches/reports/v0.2.5/d-engine_v0.2.5_vs_v0.2.4_standalone_mode.png deleted file mode 100644 index 348d50ac..00000000 Binary files a/benches/reports/v0.2.5/d-engine_v0.2.5_vs_v0.2.4_standalone_mode.png and /dev/null differ diff --git a/benches/reports/v0.2.5/bench_report_v0.2.5.md b/benches/reports/v0.3.0/bench_report_v0.3.0.md similarity index 87% rename from benches/reports/v0.2.5/bench_report_v0.2.5.md rename to benches/reports/v0.3.0/bench_report_v0.3.0.md index acc1aa49..34e5df0e 100644 --- a/benches/reports/v0.2.5/bench_report_v0.2.5.md +++ b/benches/reports/v0.3.0/bench_report_v0.3.0.md @@ -1,32 +1,32 @@ -# d-engine v0.2.5 Benchmark Report +# d-engine v0.3.0 Benchmark Report **Date**: October 18, 2026 | **Platform**: AWS us-east-1, 3 x c5.2xlarge nodes + 1 x c5.4xlarge client, 50 GB gp3 root volume (3,000 IOPS, 125 MB/s) | 3-round means | 8 B keys, 256 B values -**Which numbers:** the d-engine v0.2.5 columns in Summary and Sections 1 and 2 are the best configuration measured (October 20, 2026): the Raft log in the optional raft-engine log store, an experimental setting that is not part of the v0.2.5 release. The default log store (RocksDB WAL, October 18) is in Sections 1b and 1c; full tables are in [`bench_report_v0.2.5_raft_engine.md`](bench_report_v0.2.5_raft_engine.md). +**Which numbers:** the d-engine v0.3.0 columns in Summary and Sections 1 and 2 are the best configuration measured (October 20, 2026): the Raft log in the optional raft-engine log store, an experimental setting that is not part of the v0.3.0 release. The default log store (RocksDB WAL, October 18) is in Sections 1b and 1c; full tables are in [`bench_report_v0.3.0_raft_engine.md`](bench_report_v0.3.0_raft_engine.md). -Details: [`bench_report_v0.2.5_environment.md`](bench_report_v0.2.5_environment.md) (exact setup, commands, configuration) and [`bench_report_v0.2.5_details.md`](bench_report_v0.2.5_details.md) (every number, spreads, p99). +Details: [`bench_report_v0.3.0_environment.md`](bench_report_v0.3.0_environment.md) (exact setup, commands, configuration) and [`bench_report_v0.3.0_details.md`](bench_report_v0.3.0_details.md) (every number, spreads, p99). ## Summary -1. **v0.2.5 makes every commit wait for a physical `fdatasync` (RPO=0, #446), and on AWS EBS that costs one disk sync per commit.** At the same concurrency, single-client write falls from 4,393 to 836 ops/s (avg latency 0.228 to 1.196 ms) and 100-in-flight embedded write from 115,641 to 19,860 ops/s (0.864 to 5.036 ms). The added latency is close to one `fdatasync` that overwrites preallocated files, 0.89 ms measured: 0.228 + 0.89 = 1.12 ms against 1.196 ms measured. With the default log store the sync costs an append `fdatasync` (2.71 ms with fio on the same volume type) and the single-client write is 337 ops/s (2.962 ms), Section 1b. +1. **v0.3.0 makes every commit wait for a physical `fdatasync` (RPO=0, #446), and on AWS EBS that costs one disk sync per commit.** At the same concurrency, single-client write falls from 4,393 to 836 ops/s (avg latency 0.228 to 1.196 ms) and 100-in-flight embedded write from 115,641 to 19,860 ops/s (0.864 to 5.036 ms). The added latency is close to one `fdatasync` that overwrites preallocated files, 0.89 ms measured: 0.228 + 0.89 = 1.12 ms against 1.196 ms measured. With the default log store the sync costs an append `fdatasync` (2.71 ms with fio on the same volume type) and the single-client write is 337 ops/s (2.962 ms), Section 1b. 2. **Reads are mostly unaffected or faster.** In embedded mode lease and eventual reads are 2.6x faster (ReadActor, #392); in standalone mode they gain 12.3% and 4.1%. Linearizable read is 17% lower in embedded mode and 4.0% lower in standalone mode (within noise; the v0.2.4 value is one round). -3. **Against etcd v3.7.2 on the same AWS nodes, d-engine is ahead on writes and on eventual reads, in both modes.** Standalone (200 in-flight): writes 35,080 vs 17,143 ops/s (**2.05x**), single-client write 712 vs 605 ops/s (**1.18x**), eventual read vs etcd's serializable read 212,625 vs 147,426 ops/s (**1.44x**); linearizable read is 0.90x. Embedded (in-process calls, no network hop, 100 in-flight): writes 1.97x, single-client write 1.38x, linearizable read 5.74x, eventual read 8.43x. Every acknowledged write is synced to disk first, as in etcd. These are the numbers with the optional raft-engine log store, an experimental setting that is not part of the v0.2.5 release; with the default log store the single-client write is behind etcd (0.52x standalone, 0.56x embedded) and the 200-in-flight write is 1.87x. +3. **Against etcd v3.7.2 on the same AWS nodes, d-engine is ahead on writes and on eventual reads, in both modes.** Standalone (200 in-flight): writes 35,080 vs 17,143 ops/s (**2.05x**), single-client write 712 vs 605 ops/s (**1.18x**), eventual read vs etcd's serializable read 212,625 vs 147,426 ops/s (**1.44x**); linearizable read is 0.90x. Embedded (in-process calls, no network hop, 100 in-flight): writes 1.97x, single-client write 1.38x, linearizable read 5.74x, eventual read 8.43x. Every acknowledged write is synced to disk first, as in etcd. These are the numbers with the optional raft-engine log store, an experimental setting that is not part of the v0.3.0 release; with the default log store the single-client write is behind etcd (0.52x standalone, 0.56x embedded) and the 200-in-flight write is 1.87x. 4. **With the RocksDB as storage, etcd's single-client write is faster because its WAL sync is cheaper: 1.05 ms per sync in etcd's own metrics, against about 2.7 ms for an appending write on this volume.** fio shows a preallocated overwrite costs 1.29 ms on average against 2.72 ms for an append, with about half the device writes per sync. Measured since: the RocksDB WAL sync averages 2.7 ms per commit on this volume (the append figure), against 0.9 ms for a log that overwrites preallocated files (Sections 1b and 1c). 5. **The cost depends on the disk.** d-engine provides developer the flexibility to switch to any storage engine. -![d-engine v0.2.4 vs v0.2.5, Embedded Mode, 100 in-flight](d-engine_v0.2.5_vs_v0.2.4_embedded_mode.png) +![d-engine v0.2.4 vs v0.3.0, Embedded Mode, 100 in-flight](d-engine_v0.3.0_vs_v0.2.4_embedded_mode.png) -![d-engine v0.2.4 vs v0.2.5, Standalone Mode, 200 in-flight](d-engine_v0.2.5_vs_v0.2.4_standalone_mode.png) +![d-engine v0.2.4 vs v0.3.0, Standalone Mode, 200 in-flight](d-engine_v0.3.0_vs_v0.2.4_standalone_mode.png) -![d-engine v0.2.5 (Embedded) vs etcd v3.7.2, 100 in-flight](d-engine_comparison_v0.2.5.png) +![d-engine v0.3.0 (Embedded) vs etcd v3.7.2, 100 in-flight](d-engine_comparison_v0.3.0.png) -Charts (log scale, same data as Sections 1 and 2): v0.2.4 vs v0.2.5 in embedded mode at 100 in-flight; the same in standalone mode at 200 in-flight; d-engine v0.2.5 embedded against etcd v3.7.2 at 100 in-flight. There is no chart for standalone against etcd; that pair is in the Section 2 table. +Charts (log scale, same data as Sections 1 and 2): v0.2.4 vs v0.3.0 in embedded mode at 100 in-flight; the same in standalone mode at 200 in-flight; d-engine v0.3.0 embedded against etcd v3.7.2 at 100 in-flight. There is no chart for standalone against etcd; that pair is in the Section 2 table. -## 1. v0.2.5 vs v0.2.4, same concurrency +## 1. v0.3.0 vs v0.2.4, same concurrency -Embedded mode, 100 in-flight requests (single client: 1). Cells are ops/s (average latency). v0.2.5 = the best configuration (see the note at the top). +Embedded mode, 100 in-flight requests (single client: 1). Cells are ops/s (average latency). v0.3.0 = the best configuration (see the note at the top). -| Scenario | v0.2.4 | v0.2.5 | Throughput | +| Scenario | v0.2.4 | v0.3.0 | Throughput | | ---------------------- | ------------------ | -------------------- | ---------- | | Single client write | 4,393 (0.228 ms) | 836 (1.196 ms) | -81.0% | | High concurrency write | 115,641 (0.864 ms) | 19,860 (5.036 ms) | -82.8% | @@ -37,7 +37,7 @@ Embedded mode, 100 in-flight requests (single client: 1). Cells are ops/s (avera Standalone mode, 200 in-flight (single client: 1). Starred cells use the one clean round of v0.2.4 (the cluster was not ready in rounds 1 and 2, so they logged timeouts). -| Scenario | v0.2.4 | v0.2.5 | Throughput | +| Scenario | v0.2.4 | v0.3.0 | Throughput | | ---------------------- | ------------------- | ------------------ | ---------- | | Single client write | 2,042 (0.494 ms) | 712 (1.404 ms) | -65.1% | | High concurrency write | 63,308* (3.155 ms) | 35,080 (5.696 ms) | -44.6% | @@ -48,7 +48,7 @@ Standalone mode, 200 in-flight (single client: 1). Starred cells use the one cle ### 1b. Log store effect, embedded: default RocksDB WAL vs raft-engine (experimental) -Both columns are v0.2.5, ops/s (average latency). The default column is October 18 on another instance set; raft-engine is October 20. +Both columns are v0.3.0, ops/s (average latency). The default column is October 18 on another instance set; raft-engine is October 20. | Scenario | RocksDB WAL, default | raft-engine | Throughput | | ---------------------- | -------------------- | -------------------- | ---------- | @@ -66,13 +66,13 @@ Both columns are v0.2.5, ops/s (average latency). The default column is October | Linearizable read | 82,215 (2.432 ms) | 81,610 (2.448 ms) | -0.7% | | Eventual read | 217,464 (0.916 ms) | 212,625 (0.937 ms) | -2.2% | -## 2. d-engine v0.2.5 vs etcd v3.7.2 on the same AWS nodes +## 2. d-engine v0.3.0 vs etcd v3.7.2 on the same AWS nodes etcd ran in a separate session on new instances of the same shape, three times from a freshly wiped cluster, with etcd's documented `benchmark` commands (leader address instead of the removed `--target-leader`). Both d-engine modes are compared, because d-engine is used both ways. Standalone is the like-for-like pair: etcd is also reached through a network client. d-engine = the best configuration (see the note at the top). **Standalone** (200 in-flight; single client: 1) -| Case | etcd v3.7.2 | d-engine v0.2.5 standalone | d-engine / etcd | +| Case | etcd v3.7.2 | d-engine v0.3.0 standalone | d-engine / etcd | | --------------------------------------------------- | ----------------------------------- | ------------------------------------ | --------------- | | Write, 1 in-flight | 605 ops/s, 1.65 ms, p99 5.07 ms | 712 ops/s, 1.40 ms, p99 3.15 ms | 1.18x | | Write, 200 in-flight | 17,143 ops/s, 11.7 ms, p99 19.7 ms | 35,080 ops/s, 5.70 ms, p99 12.28 ms | 2.05x | @@ -81,7 +81,7 @@ etcd ran in a separate session on new instances of the same shape, three times f **Embedded** (100 in-flight; single client: 1). The embedded bench ran at 100 in-flight, so etcd's 100 in-flight rows are the matched ones. -| Case | etcd v3.7.2 | d-engine v0.2.5 embedded | d-engine / etcd | +| Case | etcd v3.7.2 | d-engine v0.3.0 embedded | d-engine / etcd | | --------------------------------------------------- | ------------------------------------ | --------------------------------------- | --------------- | | Write, 1 in-flight | 605 ops/s, 1.65 ms, p99 5.07 ms | 836 ops/s, 1.20 ms, p99 3.00 ms | 1.38x | | Write, 100 in-flight | 10,064 ops/s, 9.93 ms, p99 17.0 ms | 19,860 ops/s, 5.04 ms, p99 12.04 ms | 1.97x | diff --git a/benches/reports/v0.2.5/bench_report_v0.2.5_details.md b/benches/reports/v0.3.0/bench_report_v0.3.0_details.md similarity index 94% rename from benches/reports/v0.2.5/bench_report_v0.2.5_details.md rename to benches/reports/v0.3.0/bench_report_v0.3.0_details.md index c89feb6d..103aeddd 100644 --- a/benches/reports/v0.2.5/bench_report_v0.2.5_details.md +++ b/benches/reports/v0.3.0/bench_report_v0.3.0_details.md @@ -1,11 +1,11 @@ -# d-engine v0.2.5 Benchmark: Full Results +# d-engine v0.3.0 Benchmark: Full Results -Companion to `bench_report_v0.2.5.md`. Environment: `bench_report_v0.2.5_environment.md`. +Companion to `bench_report_v0.3.0.md`. Environment: `bench_report_v0.3.0_environment.md`. All values are 3-round means unless marked. Spread = (max - min) / mean between rounds. ## 1. Embedded, 100 in-flight (instance set A) -| Scenario | v0.2.4 ops/s | avg / p99 ms | v0.2.5 ops/s | avg / p99 ms | Throughput change | +| Scenario | v0.2.4 ops/s | avg / p99 ms | v0.3.0 ops/s | avg / p99 ms | Throughput change | |---|---|---|---|---|---| | Single client write (1 in-flight) | 4,393 (spread 17.7%) | 0.228 / 0.305 | 337 (0.5%) | 2.962 / 3.153 | -92.3% | | High concurrency write | 115,641 (5.4%) | 0.864 / 1.064 | 18,198 (1.1%) | 5.494 / 6.203 | -84.3% | @@ -19,7 +19,7 @@ about 1, not 100: the client side is the limit there. Every other row measured 1 ## 2. Standalone, 200 in-flight (instance set A) -| Scenario | v0.2.4 ops/s | avg / p99 ms | v0.2.5 ops/s | avg / p99 ms | Throughput change | +| Scenario | v0.2.4 ops/s | avg / p99 ms | v0.3.0 ops/s | avg / p99 ms | Throughput change | |---|---|---|---|---|---| | Single client write | 2,042 (23.7%) | 0.494 / 0.650 | 312 (1.2%) | 3.204 / 3.400 | -84.7% | | High concurrency write | 63,308 (round 3 only) | 3.155 / 4.727 | 32,068 (0.8%) | 6.228 / 7.428 | -49.3% | @@ -31,7 +31,7 @@ about 1, not 100: the client side is the limit there. Every other row measured 1 "Round 3 only": in rounds 1 and 2 of v0.2.4 the cluster was not ready when the bench started and the client logged `ConnectionTimeout` / `DeadlineExceeded` (write: 5,488 and 16,076 ops/s; linearizable read and hot-key also affected). Their means are meaningless, so only the clean round is used and the spread is unknown. -v0.2.5 had no errors in any round. +v0.3.0 had no errors in any round. ## 3. etcd v3.7.2 (instance set B, three fresh-cluster runs) diff --git a/benches/reports/v0.2.5/bench_report_v0.2.5_environment.md b/benches/reports/v0.3.0/bench_report_v0.3.0_environment.md similarity index 90% rename from benches/reports/v0.2.5/bench_report_v0.2.5_environment.md rename to benches/reports/v0.3.0/bench_report_v0.3.0_environment.md index ff02486b..ca016a08 100644 --- a/benches/reports/v0.2.5/bench_report_v0.2.5_environment.md +++ b/benches/reports/v0.3.0/bench_report_v0.3.0_environment.md @@ -1,7 +1,7 @@ -# d-engine v0.2.5 Benchmark: Test Environment +# d-engine v0.3.0 Benchmark: Test Environment -Companion to `bench_report_v0.2.5.md`. Everything here was measured on **October 18, 2026**, except the raft-engine run of -**October 20, 2026** (new instance sets E and F, same shapes), described in `bench_report_v0.2.5_raft_engine.md`. +Companion to `bench_report_v0.3.0.md`. Everything here was measured on **October 18, 2026**, except the raft-engine run of +**October 20, 2026** (new instance sets E and F, same shapes), described in `bench_report_v0.3.0_raft_engine.md`. ## 1. Instance sets @@ -11,7 +11,7 @@ the instances differ. | Set | Used for | |---|---| -| A | d-engine v0.2.4 and v0.2.5 runs (v0.2.4 first, v0.2.5 about 12 minutes later) | +| A | d-engine v0.2.4 and v0.3.0 runs (v0.2.4 first, v0.3.0 about 12 minutes later) | | B | etcd v3.7.2 runs (three fresh-cluster runs) and the second fio calibration | | C | the first fio calibration and node-to-node ping | @@ -28,13 +28,13 @@ the instances differ. ## 3. d-engine runs (set A) -- v0.2.5 = `main` at `b5fce385`. v0.2.4 = tag `v0.2.4`. Both built on the instances with the release profile. +- v0.3.0 = `main` at `b5fce385`. v0.2.4 = tag `v0.2.4`. Both built on the instances with the release profile. - **Embedded**: `embedded-bench --mode local --batch --key-size 8 --value-size 256 --sequential-keys put`, one process per node, the leader's output is used. 100 in-flight (v0.2.4's `--batch` hard-codes 100; - v0.2.5 gets `--clients 100`). The bench runtime has 2 worker threads on both versions (v0.2.4's bench was - patched to match v0.2.5's). + v0.3.0 gets `--clients 100`). The bench runtime has 2 worker threads on both versions (v0.2.4's bench was + patched to match v0.3.0's). - **Standalone**: `three-nodes-standalone` on each node, `standalone-bench` on the client with three - endpoints. `--conns 200 --clients 200` (v0.2.4 limits in-flight requests by `--conns`, v0.2.5 by `--clients`; + endpoints. `--conns 200 --clients 200` (v0.2.4 limits in-flight requests by `--conns`, v0.3.0 by `--clients`; both are 200). Single-client write: `--conns 1 --clients 1`. - Payload: 8 B keys, 256 B values, sequential keys; hot-key uses 10 keys. 10,000 requests for the single-client write, 100,000 for the others. 3 rounds; cluster restarted and data wiped before each round. @@ -42,7 +42,7 @@ the instances differ. `max_batch_size = 200`, snapshot off, `unified_db = false`, backpressure 10,000 writes / 50,000 reads, election timeout 1,500 to 2,500 ms and lease 500 ms (standalone), ReadActor 2,048/1,024 (embedded) and 10,240/2,000 (standalone). Replication settings are left at their defaults. -- **Persistence**: v0.2.4 `MemFirst` (asynchronous flush, 1,000 ms idle interval). v0.2.5 advances the durable +- **Persistence**: v0.2.4 `MemFirst` (asynchronous flush, 1,000 ms idle interval). v0.3.0 advances the durable index only after `fdatasync` (RPO=0). The raft log is stored in RocksDB. ## 4. etcd runs (set B) diff --git a/benches/reports/v0.2.5/bench_report_v0.2.5_raft_engine.md b/benches/reports/v0.3.0/bench_report_v0.3.0_raft_engine.md similarity index 97% rename from benches/reports/v0.2.5/bench_report_v0.2.5_raft_engine.md rename to benches/reports/v0.3.0/bench_report_v0.3.0_raft_engine.md index 5fe476ce..dc54a1fe 100644 --- a/benches/reports/v0.2.5/bench_report_v0.2.5_raft_engine.md +++ b/benches/reports/v0.3.0/bench_report_v0.3.0_raft_engine.md @@ -1,6 +1,6 @@ -# d-engine v0.2.5 Benchmark: raft-engine log store (embedded and standalone) +# d-engine v0.3.0 Benchmark: raft-engine log store (embedded and standalone) -Companion to `bench_report_v0.2.5.md`. Measured on **October 20, 2026** on two new instance sets (E and F). +Companion to `bench_report_v0.3.0.md`. Measured on **October 20, 2026** on two new instance sets (E and F). Experimental: the log store is a spike (#460), not part of a release. Values are 3-round means; spread = (max - min) / mean. ## 1. What was measured @@ -8,7 +8,7 @@ Experimental: the log store is a spike (#460), not part of a release. Values are | Item | Value | |---|---| | Engine | The Raft log and the hard state are stored in raft-engine (a fork pinned at `e5ef995`: upstream master `097c499` with the `lz4-sys` pin relaxed); the state machine is RocksDB. Embedded: `EmbeddedEngine`; standalone: `three-nodes-standalone` built with `--features raft-engine` (`StandaloneEngine::run_custom`). Branch `spike/460-fdatasync` | -| Control | Stock v0.2.5: Raft log in a RocksDB WAL, `unified_db = false`; the October 18 report (set A). For standalone there is also a RocksDB run on set F itself (Section 3) | +| Control | Stock v0.3.0: Raft log in a RocksDB WAL, `unified_db = false`; the October 18 report (set A). For standalone there is also a RocksDB run on set F itself (Section 3) | | Instances | Same shapes as the other sets: 3 x c5.2xlarge, 50 GB gp3 root volume (3,000 IOPS, 125 MB/s), us-east-1. Set E: embedded, 07:24 to 07:27 UTC. Set F: embedded 14:20 to 14:23 UTC, standalone from 14:41 UTC. Instance IDs were not recorded | | Bench | Embedded: `embedded-bench --mode local --batch`, 100 in-flight, leader output (set F: node2, node3, node3). Standalone: `standalone-bench` on the client, 200 in-flight, single client 1. 8 B keys, 256 B values, data wiped between rounds | | Config | Snapshots disabled, as in the earlier reports. `[raft_engine]`: `target_file_size_mib = 1`, `purge_threshold_mib = 512`, `prefill_limit_mib = 512` (every start created 768 files, which is 512 x 1.5) | diff --git a/benches/reports/v0.3.0/d-engine_comparison_v0.3.0.png b/benches/reports/v0.3.0/d-engine_comparison_v0.3.0.png new file mode 100644 index 00000000..cde402c0 Binary files /dev/null and b/benches/reports/v0.3.0/d-engine_comparison_v0.3.0.png differ diff --git a/benches/reports/v0.3.0/d-engine_v0.3.0_vs_v0.2.4_embedded_mode.png b/benches/reports/v0.3.0/d-engine_v0.3.0_vs_v0.2.4_embedded_mode.png new file mode 100644 index 00000000..2691156e Binary files /dev/null and b/benches/reports/v0.3.0/d-engine_v0.3.0_vs_v0.2.4_embedded_mode.png differ diff --git a/benches/reports/v0.3.0/d-engine_v0.3.0_vs_v0.2.4_standalone_mode.png b/benches/reports/v0.3.0/d-engine_v0.3.0_vs_v0.2.4_standalone_mode.png new file mode 100644 index 00000000..901d6910 Binary files /dev/null and b/benches/reports/v0.3.0/d-engine_v0.3.0_vs_v0.2.4_standalone_mode.png differ diff --git a/d-engine-client/src/grpc_client.rs b/d-engine-client/src/grpc_client.rs index 9d351998..a301f5e2 100644 --- a/d-engine-client/src/grpc_client.rs +++ b/d-engine-client/src/grpc_client.rs @@ -322,44 +322,6 @@ impl ClientApi for GrpcClient { } } - async fn put_with_ttl( - &self, - key: impl AsRef<[u8]> + Send, - value: impl AsRef<[u8]> + Send, - ttl_secs: u64, - ) -> ClientApiResult<()> { - // Performance tracking for put_with_ttl operation - let _timer = ScopedTimer::new("client::put_with_ttl"); - - let client_inner = self.client_inner.load(); - - // Build write request with TTL-enabled insert command - let command = WriteCommand::insert_with_ttl( - Bytes::copy_from_slice(key.as_ref()), - Bytes::copy_from_slice(value.as_ref()), - ttl_secs, - ); - - let request = ClientWriteRequest { - client_id: client_inner.client_id, - command: Some(command), - }; - - // Send write request to leader node (strong consistency required) - let mut client = self.make_leader_client().await?; - match client.handle_client_write(request).await { - Ok(response) => { - debug!("[:GrpcClient:put_with_ttl] response: {:?}", response); - let client_response = response.get_ref(); - client_response.validate_error() - } - Err(status) => { - error!("[:GrpcClient:put_with_ttl] status: {:?}", status); - Err(Into::::into(ClientApiError::from(status))) - } - } - } - async fn batch( &self, ops: Vec, @@ -380,11 +342,7 @@ impl ClientApi for GrpcClient { .into_iter() .map(|op| match op { BatchOp::Insert { key, value } => ProtoBatchOp { - op: Some(batch_op::Op::Insert(Insert { - key, - value, - ttl_secs: 0, - })), + op: Some(batch_op::Op::Insert(Insert { key, value })), }, BatchOp::Delete { key } => ProtoBatchOp { op: Some(batch_op::Op::Delete(Delete { key })), diff --git a/d-engine-client/src/grpc_client_test.rs b/d-engine-client/src/grpc_client_test.rs index f3a10cea..8e44c5e2 100644 --- a/d-engine-client/src/grpc_client_test.rs +++ b/d-engine-client/src/grpc_client_test.rs @@ -58,116 +58,6 @@ async fn test_put_success() { assert!(result.is_ok()); } -#[tokio::test] -#[traced_test] -async fn test_put_with_ttl_success() { - let (_tx, rx) = oneshot::channel::<()>(); - let (_channel, port) = MockNode::simulate_client_write_mock_server( - rx, - None::< - Box std::result::Result + Send + Sync>, - >, - ClientResponse::write_success(), - ) - .await - .unwrap(); - - let endpoints = vec![format!("http://localhost:{}", port)]; - let config = ClientConfig::default(); - - let pool = ConnectionPool::create(endpoints.clone(), config.clone()) - .await - .expect("Should create connection pool"); - - let client = GrpcClient::new(Arc::new(ArcSwap::from_pointee(ClientInner { - pool, - client_id: 123, - config, - endpoints, - }))); - - let key = "ttl_key".to_string().into_bytes(); - let value = "ttl_value".to_string().into_bytes(); - let ttl_secs = 3600; // 1 hour - - let result = client.put_with_ttl(key, value, ttl_secs).await; - println!("Result: {result:?}"); - assert!(result.is_ok()); -} - -#[tokio::test] -#[traced_test] -async fn test_put_with_ttl_failure() { - let (_tx, rx) = oneshot::channel::<()>(); - let (_channel, port) = MockNode::simulate_client_write_mock_server( - rx, - None::< - Box std::result::Result + Send + Sync>, - >, - ClientResponse::client_error(ErrorCode::ConnectionTimeout), - ) - .await - .unwrap(); - - let endpoints = vec![format!("http://localhost:{}", port)]; - let config = ClientConfig::default(); - - let pool = ConnectionPool::create(endpoints.clone(), config.clone()) - .await - .expect("Should create connection pool"); - - let client = GrpcClient::new(Arc::new(ArcSwap::from_pointee(ClientInner { - pool, - client_id: 123, - config, - endpoints, - }))); - - let key = "ttl_key".to_string().into_bytes(); - let value = "ttl_value".to_string().into_bytes(); - let ttl_secs = 3600; - - let result = client.put_with_ttl(key, value, ttl_secs).await; - assert!(result.is_err()); - assert_eq!(result.unwrap_err().code(), ErrorCode::ConnectionTimeout); -} - -#[tokio::test] -#[traced_test] -async fn test_put_with_zero_ttl() { - let (_tx, rx) = oneshot::channel::<()>(); - let (_channel, port) = MockNode::simulate_client_write_mock_server( - rx, - None::< - Box std::result::Result + Send + Sync>, - >, - ClientResponse::write_success(), - ) - .await - .unwrap(); - - let endpoints = vec![format!("http://localhost:{}", port)]; - let config = ClientConfig::default(); - - let pool = ConnectionPool::create(endpoints.clone(), config.clone()) - .await - .expect("Should create connection pool"); - - let client = GrpcClient::new(Arc::new(ArcSwap::from_pointee(ClientInner { - pool, - client_id: 123, - config, - endpoints, - }))); - - let key = "no_ttl_key".to_string().into_bytes(); - let value = "no_ttl_value".to_string().into_bytes(); - let ttl_secs = 0; // Zero TTL should still succeed but key won't expire - - let result = client.put_with_ttl(key, value, ttl_secs).await; - assert!(result.is_ok()); -} - #[tokio::test] #[traced_test] async fn test_put_failure() { @@ -1821,3 +1711,178 @@ async fn test_get_multi_with_policy_returns_native_kv_entries() { assert_eq!(results[0], Some(KvEntry { key: k1, value: v1 })); assert_eq!(results[1], Some(KvEntry { key: k2, value: v2 })); } + +// ============================================================================= +// batch tests +// ============================================================================= + +mod batch_tests { + use d_engine_core::BatchOp; + use d_engine_proto::client::ClientWriteRequest; + use d_engine_proto::client::write_command::Delete; + use d_engine_proto::client::write_command::Insert; + use d_engine_proto::client::write_command::Operation; + use d_engine_proto::client::write_command::batch_op; + use d_engine_proto::common::NodeRole; + use d_engine_proto::common::NodeStatus; + use d_engine_proto::server::cluster::NodeMeta; + + use super::*; + use crate::mock_rpc::MockRpcService; + + type Captured = Arc>>; + + /// Starts a leader mock that answers every write with `response` and records + /// each write request it receives. The returned sender keeps the server alive. + async fn start( + response: Result + ) -> (GrpcClient, Captured, oneshot::Sender<()>) { + let (tx, rx) = oneshot::channel::<()>(); + let service = MockRpcService { + expected_client_propose_response: Some(response), + ..Default::default() + } + .with_metadata_response(|port| { + Ok(ClusterMembership { + version: 1, + nodes: vec![NodeMeta { + id: 1, + role: NodeRole::Leader as i32, + address: format!("127.0.0.1:{port}"), + status: NodeStatus::Active.into(), + }], + current_leader_id: Some(1), + }) + }); + let captured = service.captured_write_requests.clone(); + let (port, _addr) = MockNode::mock_listener(service, rx, true).await.unwrap(); + + let endpoints = vec![format!("http://localhost:{port}")]; + let config = ClientConfig::default(); + let pool = ConnectionPool::create(endpoints.clone(), config.clone()) + .await + .expect("Should create connection pool"); + let client = GrpcClient::new(Arc::new(ArcSwap::from_pointee(ClientInner { + pool, + client_id: 42, + config, + endpoints, + }))); + (client, captured, tx) + } + + /// One `batch()` call must reach the server as exactly one write request that + /// carries every op, in the caller's order, with keys and values unchanged. + /// An Insert with an empty value must stay an Insert with an empty value. + #[tokio::test] + #[traced_test] + async fn test_batch_sends_all_ops_in_order_as_one_request() { + let (client, captured, _tx) = start(Ok(ClientResponse::write_success())).await; + + client + .batch(vec![ + BatchOp::Insert { + key: Bytes::from("a"), + value: Bytes::from("1"), + }, + BatchOp::Delete { + key: Bytes::from("b"), + }, + BatchOp::Insert { + key: Bytes::from("c"), + value: Bytes::new(), + }, + ]) + .await + .expect("batch should succeed"); + + let requests = captured.lock().unwrap(); + assert_eq!( + requests.len(), + 1, + "a batch must be one write request, not one per op" + ); + assert_eq!(requests[0].client_id, 42); + + let Some(Operation::Batch(batch)) = + requests[0].command.as_ref().and_then(|c| c.operation.clone()) + else { + panic!("the request must carry a Batch command"); + }; + let ops: Vec = + batch.ops.into_iter().map(|o| o.op.expect("op must be set")).collect(); + assert_eq!(ops, vec![ + batch_op::Op::Insert(Insert { + key: Bytes::from("a"), + value: Bytes::from("1"), + }), + batch_op::Op::Delete(Delete { + key: Bytes::from("b"), + }), + batch_op::Op::Insert(Insert { + key: Bytes::from("c"), + value: Bytes::new(), + }), + ]); + } + + /// An empty batch is rejected by the client itself: the caller gets + /// InvalidRequest and no request is sent, even though the server would accept it. + #[tokio::test] + #[traced_test] + async fn test_batch_empty_is_rejected_without_sending() { + let (client, captured, _tx) = start(Ok(ClientResponse::write_success())).await; + + let result = client.batch(vec![]).await; + + assert_eq!(result.unwrap_err().code(), ErrorCode::InvalidRequest); + assert!( + captured.lock().unwrap().is_empty(), + "an empty batch must not reach the server" + ); + } + + /// When the server answers with an error code, the caller must see that same + /// code instead of a silent success. + #[tokio::test] + #[traced_test] + async fn test_batch_server_error_code_is_returned() { + let (client, captured, _tx) = start(Ok(ClientResponse::client_error( + ErrorCode::ConnectionTimeout, + ))) + .await; + + let result = client + .batch(vec![BatchOp::Delete { + key: Bytes::from("k"), + }]) + .await; + + assert_eq!(result.unwrap_err().code(), ErrorCode::ConnectionTimeout); + assert_eq!( + captured.lock().unwrap().len(), + 1, + "the request must have been sent before the error came back" + ); + } + + /// A gRPC transport failure must reach the caller as an error. + #[tokio::test] + #[traced_test] + async fn test_batch_grpc_status_error_is_returned() { + let (client, _captured, _tx) = + start(Err(tonic::Status::unavailable("leader is down"))).await; + + let result = client + .batch(vec![BatchOp::Insert { + key: Bytes::from("k"), + value: Bytes::from("v"), + }]) + .await; + + assert!( + result.is_err(), + "a failed RPC must not be reported as success" + ); + } +} diff --git a/d-engine-client/src/mock_rpc.rs b/d-engine-client/src/mock_rpc.rs index 7c53fefa..82445411 100644 --- a/d-engine-client/src/mock_rpc.rs +++ b/d-engine-client/src/mock_rpc.rs @@ -29,6 +29,8 @@ pub struct MockRpcService { pub expected_update_cluster_conf_response: Option>, pub expected_client_propose_response: Option>, + /// Every `ClientWriteRequest` this mock received, in arrival order. + pub captured_write_requests: Arc>>, pub expected_client_read_response: Option>, #[allow(clippy::type_complexity)] @@ -171,8 +173,9 @@ impl RaftClientService for MockRpcService { async fn handle_client_write( &self, - _request: tonic::Request, + request: tonic::Request, ) -> std::result::Result, tonic::Status> { + self.captured_write_requests.lock().unwrap().push(request.into_inner()); match &self.expected_client_propose_response { Some(Ok(response)) => Ok(tonic::Response::new(core_to_proto_response( response.clone(), diff --git a/d-engine-core/src/client/client_api.rs b/d-engine-core/src/client/client_api.rs index 9cabba44..71cff912 100644 --- a/d-engine-core/src/client/client_api.rs +++ b/d-engine-core/src/client/client_api.rs @@ -73,33 +73,6 @@ pub trait ClientApi: Send + Sync { value: impl AsRef<[u8]> + Send, ) -> ClientApiResult<()>; - /// Stores a key-value pair with time-to-live (TTL). - /// - /// The key will automatically expire after `ttl_secs` seconds. - /// - /// # Arguments - /// - /// * `key` - The key to store - /// * `value` - The value to store - /// * `ttl_secs` - Time-to-live in seconds - /// - /// # Errors - /// - /// Same as [`put()`](Self::put) - /// - /// # Example - /// - /// ```rust,ignore - /// // Session expires after 1 hour - /// client.put_with_ttl(b"session:abc", b"user_data", 3600).await?; - /// ``` - async fn put_with_ttl( - &self, - key: impl AsRef<[u8]> + Send, - value: impl AsRef<[u8]> + Send, - ttl_secs: u64, - ) -> ClientApiResult<()>; - /// Atomically commits multiple write operations as a single Raft log entry. /// /// All operations succeed or none apply — all-or-nothing. Accepts only writes @@ -129,13 +102,6 @@ pub trait ClientApi: Send + Sync { /// Uses linearizable reads by default, ensuring the returned value /// reflects all previously committed writes. /// - /// # Future direction - /// Any TTL-expiry filtering (as noted below) is an application-specific - /// semantic layered on top of a generic read, not a consensus concern. - /// See d-lmdb's `wire::Value` envelope + `decode_live_payload`, which - /// implements TTL entirely outside d-engine. Don't build long-term - /// architecture around this method owning TTL semantics. - /// /// # Arguments /// /// * `key` - The key to retrieve @@ -143,7 +109,7 @@ pub trait ClientApi: Send + Sync { /// # Returns /// /// * `Ok(Some(value))` if key exists - /// * `Ok(None)` if key does not exist or has expired + /// * `Ok(None)` if key does not exist /// * `Err(_)` if operation failed /// /// # Errors @@ -313,7 +279,7 @@ pub trait ClientApi: Send + Sync { /// # Returns /// /// * `Ok(Some(value))` if key exists - /// * `Ok(None)` if key does not exist or has expired + /// * `Ok(None)` if key does not exist /// /// # Errors /// @@ -342,7 +308,7 @@ pub trait ClientApi: Send + Sync { /// # Returns /// /// * `Ok(Some(value))` if key exists - /// * `Ok(None)` if key does not exist or has expired + /// * `Ok(None)` if key does not exist /// /// # Errors /// @@ -371,7 +337,7 @@ pub trait ClientApi: Send + Sync { /// # Returns /// /// * `Ok(Some(value))` if key exists (may be stale) - /// * `Ok(None)` if key does not exist or has expired + /// * `Ok(None)` if key does not exist /// /// # Errors /// diff --git a/d-engine-core/src/command.rs b/d-engine-core/src/command.rs index 6ccaa546..9e9cedc0 100644 --- a/d-engine-core/src/command.rs +++ b/d-engine-core/src/command.rs @@ -41,9 +41,6 @@ pub enum Command { Insert { key: Bytes, value: Bytes, - /// `None` = no expiration. Proto encodes this as `ttl_secs = 0`; we convert - /// to `Option` here so SM implementors use idiomatic Rust instead of magic zeros. - ttl_secs: Option, }, Delete { @@ -59,9 +56,8 @@ pub enum Command { /// Atomic batch of Insert/Delete operations — all succeed or all fail. /// - /// Individual ops within a batch do not support TTL. Batch is designed for - /// atomic multi-key writes (service registration, config updates, distributed - /// locks) where per-key expiration does not apply. + /// Use it for atomic multi-key writes such as service registration, + /// config updates and distributed locks. Batch { ops: Vec, }, @@ -70,8 +66,7 @@ pub enum Command { /// Single operation within a [`Command::Batch`]. #[derive(Debug, Clone, PartialEq)] pub enum BatchOp { - /// Insert a key-value pair. TTL is not supported in batch mode — - /// use [`Command::Insert`] directly if you need per-key expiration. + /// Insert a key-value pair. Insert { key: Bytes, value: Bytes, @@ -104,11 +99,6 @@ impl TryFrom for Command { Some(Operation::Insert(i)) => Ok(Command::Insert { key: i.key, value: i.value, - ttl_secs: if i.ttl_secs == 0 { - None - } else { - Some(i.ttl_secs) - }, }), Some(Operation::Delete(d)) => Ok(Command::Delete { key: d.key }), Some(Operation::CompareAndSwap(c)) => Ok(Command::CompareAndSwap { @@ -121,18 +111,10 @@ impl TryFrom for Command { .ops .into_iter() .map(|op| match op.op { - Some(batch_op::Op::Insert(i)) => { - if i.ttl_secs != 0 { - return Err(StorageError::StateMachineError( - "BatchOp does not support TTL; use Command::Insert directly for per-key expiration".into(), - ) - .into()); - } - Ok(BatchOp::Insert { - key: i.key, - value: i.value, - }) - } + Some(batch_op::Op::Insert(i)) => Ok(BatchOp::Insert { + key: i.key, + value: i.value, + }), Some(batch_op::Op::Delete(d)) => Ok(BatchOp::Delete { key: d.key }), None => Err(StorageError::StateMachineError( "BatchOp has no operation".into(), diff --git a/d-engine-core/src/command_test.rs b/d-engine-core/src/command_test.rs index 173cba60..44c081a8 100644 --- a/d-engine-core/src/command_test.rs +++ b/d-engine-core/src/command_test.rs @@ -60,7 +60,6 @@ fn insert_entry( term: u64, key: &[u8], value: &[u8], - ttl_secs: u64, ) -> Entry { Entry { index, @@ -69,7 +68,6 @@ fn insert_entry( Insert { key: Bytes::copy_from_slice(key), value: Bytes::copy_from_slice(value), - ttl_secs, }, )))), } @@ -121,7 +119,6 @@ fn batch_insert_op( op: Some(batch_op::Op::Insert(Insert { key: Bytes::copy_from_slice(key), value: Bytes::copy_from_slice(value), - ttl_secs: 0, })), } } @@ -134,21 +131,6 @@ fn batch_delete_op(key: &[u8]) -> BatchOp { } } -/// Helper: create a BatchOp::Insert with non-zero TTL for rejection testing. -fn batch_insert_op_with_ttl( - key: &[u8], - value: &[u8], - ttl_secs: u64, -) -> BatchOp { - BatchOp { - op: Some(batch_op::Op::Insert(Insert { - key: Bytes::copy_from_slice(key), - value: Bytes::copy_from_slice(value), - ttl_secs, - })), - } -} - fn batch_entry( index: u64, term: u64, @@ -195,25 +177,13 @@ fn test_decode_config_entry_becomes_noop() { } #[test] -fn test_decode_insert_without_ttl() { - let entries = vec![insert_entry(2, 1, b"k", b"v", 0)]; +fn test_decode_insert_produces_insert_command() { + let entries = vec![insert_entry(2, 1, b"k", b"v")]; let result = decode_entries(entries).unwrap(); assert_eq!(result.len(), 1); assert_eq!(result[0].command, Command::Insert { key: Bytes::from_static(b"k"), value: Bytes::from_static(b"v"), - ttl_secs: None, - }); -} - -#[test] -fn test_decode_insert_with_ttl_converts_to_some() { - let entries = vec![insert_entry(3, 1, b"key", b"val", 60)]; - let result = decode_entries(entries).unwrap(); - assert_eq!(result[0].command, Command::Insert { - key: Bytes::from_static(b"key"), - value: Bytes::from_static(b"val"), - ttl_secs: Some(60), }); } @@ -253,10 +223,7 @@ fn test_decode_cas_with_no_expected_value_means_key_must_not_exist() { #[test] fn test_decode_preserves_index_and_term() { - let entries = vec![ - insert_entry(10, 3, b"a", b"1", 0), - delete_entry(11, 3, b"b"), - ]; + let entries = vec![insert_entry(10, 3, b"a", b"1"), delete_entry(11, 3, b"b")]; let result = decode_entries(entries).unwrap(); assert_eq!(result[0].index, 10); assert_eq!(result[0].term, 3); @@ -272,7 +239,7 @@ fn test_decode_mixed_batch_config_becomes_noop_keeps_order() { let entries = vec![ noop_entry(1, 1), config_entry(2, 1), - insert_entry(3, 1, b"x", b"y", 0), + insert_entry(3, 1, b"x", b"y"), delete_entry(4, 1, b"x"), ]; let result = decode_entries(entries).unwrap(); @@ -397,15 +364,6 @@ fn test_decode_batch_op_without_operation_returns_error() { assert!(result.is_err(), "BatchOp with no operation must return Err"); } -#[test] -fn test_decode_batch_insert_with_ttl_returns_error() { - // Batch ops do not support TTL — reject with a clear error. - let ops = vec![batch_insert_op_with_ttl(b"k", b"v", 60)]; - let entries = vec![batch_entry(12, 2, ops)]; - let result = decode_entries(entries); - assert!(result.is_err(), "BatchOp with non-zero TTL must return Err"); -} - // ── decode: error cases ───────────────────────────────────────────────────── #[test] diff --git a/d-engine-core/src/config/lease.rs b/d-engine-core/src/config/lease.rs deleted file mode 100644 index 67920462..00000000 --- a/d-engine-core/src/config/lease.rs +++ /dev/null @@ -1,137 +0,0 @@ -//! Lease configuration for time-based key expiration management -//! -//! Lease provides TTL (time-to-live) functionality for keys with automatic -//! background cleanup of expired entries. -//! -//! # Design Philosophy -//! -//! - **Always On**: TTL is part of the public API — no feature flag needed -//! - **Best Practice**: Fixed background cleanup (industry standard) -//! - **Low Overhead**: When no TTL keys exist, cleanup cycle is a single atomic load (~10ns) -//! -//! # Usage -//! -//! ```toml -//! # Optional: tune cleanup behaviour (TTL is always active) -//! [raft.state_machine.lease] -//! cleanup_interval_ms = 1000 # default 1 second -//! ``` - -use config::ConfigError; -use serde::Deserialize; -use serde::Serialize; - -use crate::errors::Error; -use crate::errors::Result; - -/// Lease configuration for TTL-based key expiration. -/// -/// TTL is always active — no enable flag. Background cleanup runs automatically. -/// -/// # Performance -/// -/// Background cleanup overhead: ~0.001% CPU -/// - Wakes up every `cleanup_interval_ms` (default 1000ms) -/// - No-op when no TTL keys exist (~10ns atomic load) -/// - Deletes expired entries when keys are present -/// -/// # Examples -/// -/// ```rust -/// use d_engine_core::config::LeaseConfig; -/// -/// let config = LeaseConfig::default(); -/// assert_eq!(config.cleanup_interval_ms, 1000); -/// ``` -#[derive(Serialize, Deserialize, Clone, Debug)] -pub struct LeaseConfig { - /// How often the background worker wakes up to scan and delete expired keys (milliseconds). - /// - /// This setting controls MEMORY EFFICIENCY only, not TTL correctness. - /// Expired keys are always rejected at read time via `is_expired()` regardless of cleanup - /// timing. Cleanup only reclaims the memory occupied by stale DashMap entries. - /// - /// Range: 100–60000 (100ms to 60 seconds) - /// Default: 1000 (1 second) - /// - /// Upper bound rationale: at 60s, a workload of 10K TTL writes/s with TTL=1s accumulates - /// ~600K stale entries (~30MB) between cleanups — acceptable for most deployments. - /// Beyond 60s, memory growth becomes unbounded in high-throughput short-TTL workloads. - /// - /// Tuning guide: - /// - 1000ms (default): Good for most workloads - /// - 100-500ms: High-throughput short-TTL workloads (e.g., rate limiting, sessions) - /// - 5000-60000ms: Low-frequency TTL usage where memory pressure is not a concern - /// - /// Always active — no enable flag needed. - #[serde(default = "default_cleanup_interval_ms")] - pub cleanup_interval_ms: u64, - - /// Maximum cleanup duration per cycle (milliseconds) - /// - /// Limits how long a single cleanup cycle can run to prevent - /// excessive CPU usage in the background worker. - /// - /// Range: 1-100ms - /// Default: 1ms - /// - /// Tuning guide: - /// - 1ms (default): Safe for latency-sensitive apps - /// - 5-10ms: Aggressive cleanup when backlog exists - /// - >10ms: Not recommended (can impact foreground operations) - /// - /// Always active — no enable flag needed. - #[serde(default = "default_max_cleanup_duration_ms")] - pub max_cleanup_duration_ms: u64, -} - -fn default_cleanup_interval_ms() -> u64 { - 1000 -} - -fn default_max_cleanup_duration_ms() -> u64 { - 1 -} - -impl Default for LeaseConfig { - fn default() -> Self { - Self { - cleanup_interval_ms: default_cleanup_interval_ms(), - max_cleanup_duration_ms: default_max_cleanup_duration_ms(), - } - } -} - -impl LeaseConfig { - /// Validates configuration parameters - /// - /// Returns error if: - /// - `cleanup_interval_ms` is out of range (100-60000) - /// - `max_cleanup_duration_ms` is out of range (1-100) - pub fn validate(&self) -> Result<()> { - // Validate cleanup_interval_ms - // Range: 100ms to 60s - // - Lower bound (100ms): Prevents excessive wakeups (CPU waste) - // - Upper bound (60000ms): Caps stale-entry accumulation; beyond 60s, high-throughput - // short-TTL workloads accumulate unbounded memory (see field doc comment) - if !(100..=60_000).contains(&self.cleanup_interval_ms) { - return Err(Error::Config(ConfigError::Message(format!( - "lease cleanup_interval_ms must be between 100 and 60000, got {}", - self.cleanup_interval_ms - )))); - } - - // Validate max_cleanup_duration_ms - // Range: 1-100ms - // - Lower bound (1ms): Minimum useful duration - // - Upper bound (100ms): Prevents blocking too long - if !(1..=100).contains(&self.max_cleanup_duration_ms) { - return Err(Error::Config(ConfigError::Message(format!( - "max_cleanup_duration_ms must be between 1 and 100, got {}", - self.max_cleanup_duration_ms - )))); - } - - Ok(()) - } -} diff --git a/d-engine-core/src/config/lease_test.rs b/d-engine-core/src/config/lease_test.rs deleted file mode 100644 index aca1fd79..00000000 --- a/d-engine-core/src/config/lease_test.rs +++ /dev/null @@ -1,145 +0,0 @@ -use super::lease::LeaseConfig; - -#[test] -fn test_default_config() { - let config = LeaseConfig::default(); - assert_eq!(config.cleanup_interval_ms, 1000); - assert_eq!(config.max_cleanup_duration_ms, 1); - assert!(config.validate().is_ok()); -} - -#[test] -fn test_explicit_config() { - let config = LeaseConfig { - cleanup_interval_ms: 5000, - max_cleanup_duration_ms: 5, - }; - assert_eq!(config.cleanup_interval_ms, 5000); - assert_eq!(config.max_cleanup_duration_ms, 5); - assert!(config.validate().is_ok()); -} - -#[test] -fn test_validation_cleanup_interval_ms() { - let base = LeaseConfig::default(); - - // Too small - assert!( - LeaseConfig { - cleanup_interval_ms: 99, - ..base - } - .validate() - .is_err() - ); - // Too large - assert!( - LeaseConfig { - cleanup_interval_ms: 60_001, - ..base - } - .validate() - .is_err() - ); - // Valid range - assert!( - LeaseConfig { - cleanup_interval_ms: 100, - ..base - } - .validate() - .is_ok() - ); - assert!( - LeaseConfig { - cleanup_interval_ms: 1000, - ..base - } - .validate() - .is_ok() - ); - assert!( - LeaseConfig { - cleanup_interval_ms: 60_000, - ..base - } - .validate() - .is_ok() - ); -} - -#[test] -fn test_validation_max_cleanup_duration() { - let base = LeaseConfig::default(); - - // Too small - assert!( - LeaseConfig { - max_cleanup_duration_ms: 0, - ..base - } - .validate() - .is_err() - ); - // Too large - assert!( - LeaseConfig { - max_cleanup_duration_ms: 101, - ..base - } - .validate() - .is_err() - ); - // Valid range - assert!( - LeaseConfig { - max_cleanup_duration_ms: 1, - ..base - } - .validate() - .is_ok() - ); - assert!( - LeaseConfig { - max_cleanup_duration_ms: 50, - ..base - } - .validate() - .is_ok() - ); - assert!( - LeaseConfig { - max_cleanup_duration_ms: 100, - ..base - } - .validate() - .is_ok() - ); -} - -#[test] -fn test_typical_production_config() { - let config = LeaseConfig { - cleanup_interval_ms: 1000, - max_cleanup_duration_ms: 1, - }; - assert!(config.validate().is_ok()); -} - -#[test] -fn test_aggressive_cleanup_config() { - let config = LeaseConfig { - cleanup_interval_ms: 100, - max_cleanup_duration_ms: 10, - }; - assert!(config.validate().is_ok()); -} - -#[test] -fn test_relaxed_cleanup_config() { - let config = LeaseConfig { - cleanup_interval_ms: 60_000, - max_cleanup_duration_ms: 5, - }; - assert!(config.validate().is_ok()); -} diff --git a/d-engine-core/src/config/mod.rs b/d-engine-core/src/config/mod.rs index b9ecbbb8..04e8dd14 100644 --- a/d-engine-core/src/config/mod.rs +++ b/d-engine-core/src/config/mod.rs @@ -6,14 +6,12 @@ //! - Configuration file support //! - Component-wise validation mod cluster; -mod lease; mod network; mod raft; mod retry; mod storage; mod tls; pub use cluster::*; -pub use lease::*; pub use network::*; pub use raft::*; pub use retry::*; @@ -22,8 +20,6 @@ pub use tls::*; #[cfg(test)] mod config_test; #[cfg(test)] -mod lease_test; -#[cfg(test)] mod network_test; #[cfg(test)] mod raft_test; diff --git a/d-engine-core/src/config/raft.rs b/d-engine-core/src/config/raft.rs index 67a8e661..2ff103d8 100644 --- a/d-engine-core/src/config/raft.rs +++ b/d-engine-core/src/config/raft.rs @@ -6,7 +6,6 @@ use serde::Deserialize; use serde::Serialize; use tracing::warn; -use super::lease::LeaseConfig; use crate::Error; use crate::Result; @@ -553,12 +552,6 @@ impl MembershipConfig { /// processes applied log entries and manages data. #[derive(Serialize, Deserialize, Clone, Debug)] pub struct StateMachineConfig { - /// Lease (time-based expiration) configuration - /// - /// For backward compatibility, can also be configured via `ttl` in TOML files. - #[serde(alias = "ttl")] - pub lease: LeaseConfig, - /// Maximum time to wait, on shutdown, for the state machine worker to finish /// applying the entries already handed to it. On timeout an error is logged and /// shutdown proceeds: entries the node knew as committed may then be unapplied @@ -569,7 +562,6 @@ pub struct StateMachineConfig { impl StateMachineConfig { pub fn validate(&self) -> Result<()> { - self.lease.validate()?; if self.shutdown_apply_timeout_ms == 0 { return Err(Error::Config(ConfigError::Message( "shutdown_apply_timeout_ms must be greater than 0".into(), @@ -586,7 +578,6 @@ fn default_shutdown_apply_timeout_ms() -> u64 { impl Default for StateMachineConfig { fn default() -> Self { Self { - lease: LeaseConfig::default(), shutdown_apply_timeout_ms: default_shutdown_apply_timeout_ms(), } } diff --git a/d-engine-core/src/errors.rs b/d-engine-core/src/errors.rs index a0039480..65fbb142 100644 --- a/d-engine-core/src/errors.rs +++ b/d-engine-core/src/errors.rs @@ -376,10 +376,6 @@ pub enum SystemError { #[error("Internal server error")] ServerUnavailable, - - /// State machine does not support lease-based expiration - #[error("State machine does not support lease management")] - LeaseNotSupported, } // Serialization is classified separately (across protocol layers and system layers) diff --git a/d-engine-core/src/raft_test/drain_based_batch_architecture_tests.rs b/d-engine-core/src/raft_test/drain_based_batch_architecture_tests.rs index 33bd0d74..d6d60cf4 100644 --- a/d-engine-core/src/raft_test/drain_based_batch_architecture_tests.rs +++ b/d-engine-core/src/raft_test/drain_based_batch_architecture_tests.rs @@ -201,9 +201,7 @@ async fn test_p0_leader_single_command_drain_immediately() { operation: Some(Operation::Insert( d_engine_proto::client::write_command::Insert { key: Bytes::from("test_key"), - value: Bytes::from("test_value"), - ttl_secs: 0, - }, + value: Bytes::from("test_value"), }, )), }; let write_req = ClientWriteRequest { @@ -307,9 +305,7 @@ async fn test_p0_leader_high_load_max_batch_cap() { operation: Some(Operation::Insert( d_engine_proto::client::write_command::Insert { key: Bytes::from(format!("key_{round}_{i}")), - value: Bytes::from(format!("value_{round}_{i}")), - ttl_secs: 0, - }, + value: Bytes::from(format!("value_{round}_{i}")), }, )), }; let write_req = ClientWriteRequest { @@ -466,9 +462,7 @@ async fn test_p0_follower_write_request_rejection() { operation: Some(Operation::Insert( d_engine_proto::client::write_command::Insert { key: Bytes::from("test_key"), - value: Bytes::from("test_value"), - ttl_secs: 0, - }, + value: Bytes::from("test_value"), }, )), }; let write_req = ClientWriteRequest { @@ -565,9 +559,7 @@ async fn test_p0_candidate_commands_handling() { operation: Some(Operation::Insert( d_engine_proto::client::write_command::Insert { key: Bytes::from("write_key"), - value: Bytes::from("write_value"), - ttl_secs: 0, - }, + value: Bytes::from("write_value"), }, )), }; let write_req = ClientWriteRequest { @@ -749,9 +741,7 @@ async fn test_p1_medium_load_natural_batching() { operation: Some(Operation::Insert( d_engine_proto::client::write_command::Insert { key: Bytes::from(format!("key_{i}")), - value: Bytes::from(format!("value_{i}")), - ttl_secs: 0, - }, + value: Bytes::from(format!("value_{i}")), }, )), }; let write_req = ClientWriteRequest { @@ -856,9 +846,7 @@ async fn test_p1_mixed_workload_read_write_coexistence() { operation: Some(Operation::Insert( d_engine_proto::client::write_command::Insert { key: Bytes::from(format!("write_key_{i}")), - value: Bytes::from(format!("write_value_{i}")), - ttl_secs: 0, - }, + value: Bytes::from(format!("write_value_{i}")), }, )), }; let write_req = ClientWriteRequest { @@ -964,9 +952,7 @@ async fn test_p1_burst_load_recovery_after_spike() { operation: Some(Operation::Insert( d_engine_proto::client::write_command::Insert { key: Bytes::from(format!("burst_key_{i}")), - value: Bytes::from(format!("burst_value_{i}")), - ttl_secs: 0, - }, + value: Bytes::from(format!("burst_value_{i}")), }, )), }; let write_req = ClientWriteRequest { @@ -997,9 +983,7 @@ async fn test_p1_burst_load_recovery_after_spike() { operation: Some(Operation::Insert( d_engine_proto::client::write_command::Insert { key: Bytes::from(format!("trickle_key_{i}")), - value: Bytes::from(format!("trickle_value_{i}")), - ttl_secs: 0, - }, + value: Bytes::from(format!("trickle_value_{i}")), }, )), }; let write_req = ClientWriteRequest { diff --git a/d-engine-core/src/replication/replication_handler_test/helper_functions_test.rs b/d-engine-core/src/replication/replication_handler_test/helper_functions_test.rs index 24ad59b4..726b445d 100644 --- a/d-engine-core/src/replication/replication_handler_test/helper_functions_test.rs +++ b/d-engine-core/src/replication/replication_handler_test/helper_functions_test.rs @@ -732,14 +732,12 @@ fn test_command_conversion_with_multiple_commands() { let commands = vec![ WriteCommand { operation: Some(Operation::Insert(Insert { - ttl_secs: 0, key: Bytes::from(b"key1".to_vec()), value: Bytes::from(b"value1".to_vec()), })), }, WriteCommand { operation: Some(Operation::Insert(Insert { - ttl_secs: 0, key: Bytes::from(b"key2".to_vec()), value: Bytes::from(b"value2".to_vec()), })), @@ -754,7 +752,7 @@ fn test_command_conversion_with_multiple_commands() { let decoded = WriteCommand::decode(bytes.as_ref()).unwrap(); assert!(matches!( decoded.operation, - Some(Operation::Insert(Insert { key, value, ttl_secs: _ })) + Some(Operation::Insert(Insert { key, value })) if key == b"key1".as_ref() && value == b"value1".as_ref() )); } else { @@ -765,7 +763,7 @@ fn test_command_conversion_with_multiple_commands() { let decoded = WriteCommand::decode(bytes.as_ref()).unwrap(); assert!(matches!( decoded.operation, - Some(Operation::Insert(Insert { key, value, ttl_secs: _ })) + Some(Operation::Insert(Insert { key, value })) if key == b"key2".as_ref() && value == b"value2".as_ref() )); } else { diff --git a/d-engine-core/src/state_machine_handler/default_state_machine_handler_test.rs b/d-engine-core/src/state_machine_handler/default_state_machine_handler_test.rs index 34cc61fc..210b2e15 100644 --- a/d-engine-core/src/state_machine_handler/default_state_machine_handler_test.rs +++ b/d-engine-core/src/state_machine_handler/default_state_machine_handler_test.rs @@ -59,7 +59,6 @@ fn batch_entry( op: Some(Op::Insert(Insert { key: Bytes::copy_from_slice(key), value: Bytes::copy_from_slice(value), - ttl_secs: 0, })), }], })), @@ -320,7 +319,6 @@ mod apply_chunk_test { op: Some(Op::Insert(Insert { key: Bytes::from_static(b"k1"), value: Bytes::from_static(b"v1"), - ttl_secs: 0, })), }], })), @@ -381,7 +379,6 @@ mod apply_chunk_test { op: Some(Op::Insert(Insert { key: Bytes::from_static(b"k1"), value: Bytes::from_static(b"v1"), - ttl_secs: 0, })), }], })), @@ -530,7 +527,6 @@ mod decode_boundary_tests { operation: Some(Operation::Insert(Insert { key: Bytes::copy_from_slice(key), value: Bytes::copy_from_slice(value), - ttl_secs: 0, })), }; let mut buf = Vec::new(); @@ -2152,7 +2148,6 @@ mod broadcast_watch_events_tests { let entry = apply_entry(1, Command::Insert { key: Bytes::from_static(b"key1"), value: Bytes::from_static(b"value1"), - ttl_secs: None, }); broadcast_watch_events(&[entry], &[succeeded(1)], &tx, None); @@ -2267,7 +2262,6 @@ mod broadcast_watch_events_tests { apply_entry(1, Command::Insert { key: Bytes::from_static(b"k1"), value: Bytes::from_static(b"v1"), - ttl_secs: None, }), apply_entry(2, Command::CompareAndSwap { key: Bytes::from_static(b"k2"), @@ -2516,7 +2510,6 @@ mod prev_kv_apply_tests { operation: Some(Operation::Insert(Insert { key: Bytes::copy_from_slice(key), value: Bytes::copy_from_slice(value), - ttl_secs: 0, })), }; let mut buf = Vec::new(); diff --git a/d-engine-core/src/storage/lease.rs b/d-engine-core/src/storage/lease.rs deleted file mode 100644 index 98d15c2e..00000000 --- a/d-engine-core/src/storage/lease.rs +++ /dev/null @@ -1,204 +0,0 @@ -//! Lease (Time-based key expiration) trait for d-engine. -//! -//! Provides the interface for automatic key expiration through lease-based lifecycle management. -//! This is a framework-level abstraction that allows custom lease implementations. -//! -//! # Design Philosophy -//! -//! d-engine provides lease-based key expiration management as a framework-level -//! feature that all state machines (including custom implementations) can leverage. -//! -//! # Architecture -//! -//! - **Trait-based**: `Lease` trait defines the interface -//! - **Implementation**: d-engine-server provides `DefaultLease` with high-performance dual-index -//! design -//! - **Zero overhead**: Completely disabled when not used -//! - **Snapshot support**: Full persistence through Raft snapshots -//! -//! # Concurrency Model -//! -//! Implementations should follow these guidelines: -//! - **Read path (hot)**: Lock-free, supports high concurrency -//! - **Write path (cold)**: Single-threaded (state machine worker), Mutex acceptable -//! - **Read-write**: Concurrent safe, reads don't block on cleanup - -use std::time::SystemTime; - -use bytes::Bytes; - -use crate::Result; - -/// Lease management interface for key expiration. -/// -/// Manages key lifecycles through time-based leases. d-engine provides a default -/// implementation (`DefaultLease` in d-engine-server), but developers can implement -/// custom lease management strategies. -/// -/// # Thread Safety -/// -/// All methods must be thread-safe as they will be called concurrently from: -/// - Read path: Multiple concurrent client reads -/// - Write path: Single-threaded apply operations -/// -/// # Performance Requirements -/// -/// - `is_expired()`: Must be O(1) and lock-free (hot path, called on every read) -/// - `register()` / `unregister()`: Should be O(log N) or better -/// - `get_expired_keys()`: Should be O(K log N) where K = expired keys -/// -/// # Example Implementation -/// -/// ```ignore -/// use d_engine_core::storage::Lease; -/// use bytes::Bytes; -/// use std::time::SystemTime; -/// -/// struct MyCustomLease { -/// // Your data structures -/// } -/// -/// impl Lease for MyCustomLease { -/// fn register(&self, key: Bytes, ttl_secs: u64) { -/// // Your implementation -/// } -/// -/// fn is_expired(&self, key: &[u8]) -> bool { -/// // Your implementation -/// } -/// -/// // ... other methods -/// } -/// ``` -pub trait Lease: Send + Sync + 'static { - /// Register a lease for a key. - /// - /// If the key already has a lease, updates to the new expiration time. - /// - /// # Arguments - /// * `key` - Key to set expiration for - /// * `ttl_secs` - Time-to-live in seconds from now - /// - /// # Performance - /// Should be O(log N) or better. Called on every insert with TTL. - fn register( - &self, - key: Bytes, - ttl_secs: u64, - ); - - /// Remove lease for a key (on update/delete). - /// - /// Called when a key is updated without TTL or explicitly deleted. - /// - /// # Performance - /// Should be O(log N) or better. - fn unregister( - &self, - key: &[u8], - ); - - /// Check if a key's lease has expired. - /// - /// # Returns - /// * `true` - Key has expired and should be treated as non-existent - /// * `false` - Key has not expired or has no lease - /// - /// # Performance - /// **CRITICAL**: Must be O(1) and lock-free. This is the hot path, - /// called on every read operation. - fn is_expired( - &self, - key: &[u8], - ) -> bool; - - /// Get all keys with expired leases. - /// - /// Removes returned keys from internal lease indexes. - /// - /// # Arguments - /// * `now` - Current time to check expiration against - /// - /// # Returns - /// List of expired keys (keys are removed from lease tracking) - /// - /// # Performance - /// Should be O(K log N) where K = number of expired keys. - fn get_expired_keys( - &self, - now: SystemTime, - ) -> Vec; - - /// Called on every apply operation (piggyback cleanup). - /// - /// The lease implementation decides internally whether to perform cleanup - /// based on its configuration (e.g., piggyback strategy with frequency control). - /// - /// # Returns - /// List of expired keys that were cleaned up (may be empty) - /// - /// # Performance - /// Should be O(1) most of the time (fast path when not cleaning). - fn on_apply(&self) -> Vec; - - /// Check if any key has ever been registered with a lease. - /// - /// Used for fast-path optimization to skip lease logic entirely when - /// leases are not used. - /// - /// # Returns - /// * `true` - At least one key has been registered (even if expired) - /// * `false` - No keys have ever been registered - /// - /// # Performance - /// Must be O(1) - simple flag check. - fn has_lease_keys(&self) -> bool; - - /// Fast check: returns true if there might be expired keys. - /// - /// This is an O(1) optimization to avoid full expired key scan. - /// - /// # Returns - /// * `false` - Definitely no expired keys (fast path) - /// * `true` - Maybe has expired keys (need full check) - /// - /// # Performance - /// Should be O(1) or O(log N) at most. - fn may_have_expired_keys( - &self, - now: SystemTime, - ) -> bool; - - /// Get number of keys with active leases. - /// - /// # Performance - /// Should be O(1). - fn len(&self) -> usize; - - /// Check if no keys have active leases. - fn is_empty(&self) -> bool { - self.len() == 0 - } - - /// Serialize lease state for snapshot. - /// - /// Used by Raft snapshot mechanism to persist lease metadata. - /// - /// # Returns - /// Serialized bytes (format is implementation-defined) - fn to_snapshot(&self) -> Vec; - - /// Reload lease state from snapshot data. - /// - /// Used during snapshot restoration. Should filter out already-expired leases. - /// - /// # Arguments - /// * `data` - Serialized snapshot data - /// - /// # Errors - /// Returns error if deserialization fails. - fn reload( - &self, - data: &[u8], - ) -> Result<()>; -} diff --git a/d-engine-core/src/storage/mod.rs b/d-engine-core/src/storage/mod.rs index 54976f8d..5141f1cd 100644 --- a/d-engine-core/src/storage/mod.rs +++ b/d-engine-core/src/storage/mod.rs @@ -1,11 +1,9 @@ pub(super) mod fsync_worker; -mod lease; mod raft_log; mod raft_log_core; mod snapshot_path_manager; mod state_machine; mod storage_engine; -pub use lease::*; #[doc(hidden)] pub use raft_log::*; pub use raft_log_core::*; diff --git a/d-engine-core/src/storage/raft_log_core_test/drain_fsync_test.rs b/d-engine-core/src/storage/raft_log_core_test/drain_fsync_test.rs index c83cef2a..c7397bcb 100644 --- a/d-engine-core/src/storage/raft_log_core_test/drain_fsync_test.rs +++ b/d-engine-core/src/storage/raft_log_core_test/drain_fsync_test.rs @@ -500,10 +500,9 @@ async fn test_reset_failure_poisons() { } /// A `save_hard_state()` failure (persisting current_term/voted_for) poisons -/// the log — this is the Election Safety gap the expert flagged as highest -/// priority: a silently-failed vote write, if not caught, lets a restarted -/// node believe it never voted this term and cast a second vote, breaking -/// "at most one leader per term." +/// the log. This protects Election Safety: a silently-failed vote write, if +/// not caught, lets a restarted node believe it never voted this term and +/// cast a second vote, breaking "at most one leader per term." #[tokio::test] async fn test_save_hard_state_failure_poisons() { let storage = Arc::new(MockStorageEngine::not_durable_save_hard_state_fails( diff --git a/d-engine-core/src/storage/state_machine.rs b/d-engine-core/src/storage/state_machine.rs index 66830fe3..62178fe8 100644 --- a/d-engine-core/src/storage/state_machine.rs +++ b/d-engine-core/src/storage/state_machine.rs @@ -89,13 +89,13 @@ pub trait StateMachine: Send + Sync + 'static { /// /// This method: /// 1. Flips internal state flags - /// 2. Loads persisted data (e.g., TTL state from disk) + /// 2. Loads persisted data from disk /// /// Called once during node startup. Performance is not critical. async fn start(&self) -> Result<(), Error>; /// Permanently closes the state machine: stops serving, persists what must survive a - /// restart (e.g. lease state, metadata), flushes, and releases exclusive resources + /// restart (e.g. data, metadata), flushes, and releases exclusive resources /// (e.g. a DB handle or its lock file). /// /// Called once by the state machine worker as its last act, after everything queued for @@ -305,36 +305,4 @@ pub trait StateMachine: Send + Sync + 'static { ), ))) } - - /// Background lease cleanup hook. - /// - /// Called periodically by the background cleanup task (if enabled). - /// Returns keys that were cleaned up. - /// - /// # Default Implementation - /// No-op, suitable for state machines without lease support. - /// - /// # Called By - /// Framework calls this from background cleanup task spawned in NodeBuilder::build(). - /// Only called when cleanup_strategy = "background". - /// - /// # Returns - /// - Vec of deleted keys (for logging/metrics) - /// - /// # Example (d-engine built-in state machines) - /// ```ignore - /// async fn lease_background_cleanup(&self) -> Result, Error> { - /// if let Some(ref lease) = self.lease { - /// let expired_keys = lease.get_expired_keys(SystemTime::now()); - /// if !expired_keys.is_empty() { - /// self.delete_batch(&expired_keys).await?; - /// } - /// return Ok(expired_keys); - /// } - /// Ok(vec![]) - /// } - /// ``` - async fn lease_background_cleanup(&self) -> Result, Error> { - Ok(vec![]) - } } diff --git a/d-engine-core/src/storage/state_machine_test.rs b/d-engine-core/src/storage/state_machine_test.rs index 07702d1f..350ba27c 100644 --- a/d-engine-core/src/storage/state_machine_test.rs +++ b/d-engine-core/src/storage/state_machine_test.rs @@ -1685,11 +1685,7 @@ fn create_insert_entry( ApplyEntry { index, term: 1, - command: Command::Insert { - key, - value, - ttl_secs: None, - }, + command: Command::Insert { key, value }, } } diff --git a/d-engine-core/src/storage/storage_engine_test.rs b/d-engine-core/src/storage/storage_engine_test.rs index 42f4e3ab..153d1701 100644 --- a/d-engine-core/src/storage/storage_engine_test.rs +++ b/d-engine-core/src/storage/storage_engine_test.rs @@ -559,11 +559,7 @@ fn create_test_command_payload(index: u64) -> d_engine_proto::common::EntryPaylo let key = Bytes::from(format!("key_{index}")); let value = Bytes::from(format!("value_{index}")); - let insert = Insert { - key, - value, - ttl_secs: 0, - }; + let insert = Insert { key, value }; let operation = d_engine_proto::client::write_command::Operation::Insert(insert); let write_cmd = d_engine_proto::client::WriteCommand { operation: Some(operation), diff --git a/d-engine-proto/Makefile b/d-engine-proto/Makefile index d99d7194..15be5522 100644 --- a/d-engine-proto/Makefile +++ b/d-engine-proto/Makefile @@ -25,12 +25,14 @@ generate-go: # package foo to be in foo/ directory # Move top-level proto files (error, common) - @mkdir -p go/error go/common - @mv go/proto/error.pb.go go/error/ 2>/dev/null || true - @mv go/proto/common.pb.go go/common/ 2>/dev/null || true - - # Move client directory - @mv go/proto/client go/ 2>/dev/null || true + @mkdir -p go/error go/common go/client + @mv -f go/proto/error.pb.go go/error/ + @mv -f go/proto/common.pb.go go/common/ + + # Move client files. Move the files, not the directory: `mv go/proto/client go/` + # fails when go/client already exists, and the later `rm -rf go/proto` then + # silently throws the new files away. + @mv -f go/proto/client/*.go go/client/ # Move server subdirectories @mkdir -p go/server diff --git a/d-engine-proto/go/client/client_api.pb.go b/d-engine-proto/go/client/client_api.pb.go index 5bac5b08..d6a01bc8 100644 --- a/d-engine-proto/go/client/client_api.pb.go +++ b/d-engine-proto/go/client/client_api.pb.go @@ -101,6 +101,9 @@ const ( // Watcher forcibly canceled by the server (e.g. buffer overflow). // Client should re-sync via Read API and re-register the watch. WatchEventType_WATCH_EVENT_TYPE_CANCELED WatchEventType = 2 + // Periodic heartbeat with no data change. + // Carries current revision so clients can confirm the stream is alive. + WatchEventType_WATCH_EVENT_TYPE_PROGRESS WatchEventType = 3 ) // Enum value maps for WatchEventType. @@ -109,11 +112,13 @@ var ( 0: "WATCH_EVENT_TYPE_PUT", 1: "WATCH_EVENT_TYPE_DELETE", 2: "WATCH_EVENT_TYPE_CANCELED", + 3: "WATCH_EVENT_TYPE_PROGRESS", } WatchEventType_value = map[string]int32{ "WATCH_EVENT_TYPE_PUT": 0, "WATCH_EVENT_TYPE_DELETE": 1, "WATCH_EVENT_TYPE_CANCELED": 2, + "WATCH_EVENT_TYPE_PROGRESS": 3, } ) @@ -152,6 +157,7 @@ type WriteCommand struct { // *WriteCommand_Insert_ // *WriteCommand_Delete_ // *WriteCommand_CompareAndSwap_ + // *WriteCommand_Batch_ Operation isWriteCommand_Operation `protobuf_oneof:"operation"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache @@ -221,6 +227,15 @@ func (x *WriteCommand) GetCompareAndSwap() *WriteCommand_CompareAndSwap { return nil } +func (x *WriteCommand) GetBatch() *WriteCommand_Batch { + if x != nil { + if x, ok := x.Operation.(*WriteCommand_Batch_); ok { + return x.Batch + } + } + return nil +} + type isWriteCommand_Operation interface { isWriteCommand_Operation() } @@ -237,12 +252,18 @@ type WriteCommand_CompareAndSwap_ struct { CompareAndSwap *WriteCommand_CompareAndSwap `protobuf:"bytes,3,opt,name=compare_and_swap,json=compareAndSwap,proto3,oneof"` } +type WriteCommand_Batch_ struct { + Batch *WriteCommand_Batch `protobuf:"bytes,4,opt,name=batch,proto3,oneof"` +} + func (*WriteCommand_Insert_) isWriteCommand_Operation() {} func (*WriteCommand_Delete_) isWriteCommand_Operation() {} func (*WriteCommand_CompareAndSwap_) isWriteCommand_Operation() {} +func (*WriteCommand_Batch_) isWriteCommand_Operation() {} + type ClientWriteRequest struct { state protoimpl.MessageState `protogen:"open.v1"` ClientId uint32 `protobuf:"varint,1,opt,name=client_id,json=clientId,proto3" json:"client_id,omitempty"` @@ -411,8 +432,9 @@ type ClientResponse struct { // // *ClientResponse_WriteResult // *ClientResponse_ReadData + // *ClientResponse_ScanData SuccessResult isClientResponse_SuccessResult `protobuf_oneof:"success_result"` - Metadata *error1.ErrorMetadata `protobuf:"bytes,4,opt,name=metadata,proto3" json:"metadata,omitempty"` + Metadata *error1.ErrorMetadata `protobuf:"bytes,5,opt,name=metadata,proto3" json:"metadata,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } @@ -479,6 +501,15 @@ func (x *ClientResponse) GetReadData() *ReadResults { return nil } +func (x *ClientResponse) GetScanData() *ScanResults { + if x != nil { + if x, ok := x.SuccessResult.(*ClientResponse_ScanData); ok { + return x.ScanData + } + } + return nil +} + func (x *ClientResponse) GetMetadata() *error1.ErrorMetadata { if x != nil { return x.Metadata @@ -498,10 +529,16 @@ type ClientResponse_ReadData struct { ReadData *ReadResults `protobuf:"bytes,3,opt,name=read_data,json=readData,proto3,oneof"` } +type ClientResponse_ScanData struct { + ScanData *ScanResults `protobuf:"bytes,4,opt,name=scan_data,json=scanData,proto3,oneof"` +} + func (*ClientResponse_WriteResult) isClientResponse_SuccessResult() {} func (*ClientResponse_ReadData) isClientResponse_SuccessResult() {} +func (*ClientResponse_ScanData) isClientResponse_SuccessResult() {} + type ClientResult struct { state protoimpl.MessageState `protogen:"open.v1"` Key []byte `protobuf:"bytes,1,opt,name=key,proto3" json:"key,omitempty"` @@ -598,6 +635,171 @@ func (x *ReadResults) GetResults() []*ClientResult { return nil } +// A single key-value pair returned by a prefix scan. +type KvEntry struct { + state protoimpl.MessageState `protogen:"open.v1"` + Key []byte `protobuf:"bytes,1,opt,name=key,proto3" json:"key,omitempty"` + Value []byte `protobuf:"bytes,2,opt,name=value,proto3" json:"value,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *KvEntry) Reset() { + *x = KvEntry{} + mi := &file_proto_client_client_api_proto_msgTypes[7] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *KvEntry) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*KvEntry) ProtoMessage() {} + +func (x *KvEntry) ProtoReflect() protoreflect.Message { + mi := &file_proto_client_client_api_proto_msgTypes[7] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use KvEntry.ProtoReflect.Descriptor instead. +func (*KvEntry) Descriptor() ([]byte, []int) { + return file_proto_client_client_api_proto_rawDescGZIP(), []int{7} +} + +func (x *KvEntry) GetKey() []byte { + if x != nil { + return x.Key + } + return nil +} + +func (x *KvEntry) GetValue() []byte { + if x != nil { + return x.Value + } + return nil +} + +// Request to scan all keys under a namespace prefix. +// Scan is served from the leader's state machine at last_applied. +// consistency_policy is not supported; tracked in #381. +type ScanRequest struct { + state protoimpl.MessageState `protogen:"open.v1"` + ClientId uint32 `protobuf:"varint,1,opt,name=client_id,json=clientId,proto3" json:"client_id,omitempty"` + // Prefix to scan. Must start with '/'. e.g. b"/services/". + Prefix []byte `protobuf:"bytes,2,opt,name=prefix,proto3" json:"prefix,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *ScanRequest) Reset() { + *x = ScanRequest{} + mi := &file_proto_client_client_api_proto_msgTypes[8] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *ScanRequest) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*ScanRequest) ProtoMessage() {} + +func (x *ScanRequest) ProtoReflect() protoreflect.Message { + mi := &file_proto_client_client_api_proto_msgTypes[8] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use ScanRequest.ProtoReflect.Descriptor instead. +func (*ScanRequest) Descriptor() ([]byte, []int) { + return file_proto_client_client_api_proto_rawDescGZIP(), []int{8} +} + +func (x *ScanRequest) GetClientId() uint32 { + if x != nil { + return x.ClientId + } + return 0 +} + +func (x *ScanRequest) GetPrefix() []byte { + if x != nil { + return x.Prefix + } + return nil +} + +// Payload carried in a successful ClientResponse for a prefix scan. +type ScanResults struct { + state protoimpl.MessageState `protogen:"open.v1"` + // All (key, value) pairs whose key starts with the requested prefix. + Entries []*KvEntry `protobuf:"bytes,1,rep,name=entries,proto3" json:"entries,omitempty"` + // Raft applied index at the moment of the scan. + // Use as the watch-event filter anchor: skip events where event.revision <= revision. + Revision uint64 `protobuf:"varint,2,opt,name=revision,proto3" json:"revision,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *ScanResults) Reset() { + *x = ScanResults{} + mi := &file_proto_client_client_api_proto_msgTypes[9] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *ScanResults) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*ScanResults) ProtoMessage() {} + +func (x *ScanResults) ProtoReflect() protoreflect.Message { + mi := &file_proto_client_client_api_proto_msgTypes[9] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use ScanResults.ProtoReflect.Descriptor instead. +func (*ScanResults) Descriptor() ([]byte, []int) { + return file_proto_client_client_api_proto_rawDescGZIP(), []int{9} +} + +func (x *ScanResults) GetEntries() []*KvEntry { + if x != nil { + return x.Entries + } + return nil +} + +func (x *ScanResults) GetRevision() uint64 { + if x != nil { + return x.Revision + } + return 0 +} + // Request to start a server-side membership watch stream. type WatchMembershipRequest struct { state protoimpl.MessageState `protogen:"open.v1"` @@ -608,7 +810,7 @@ type WatchMembershipRequest struct { func (x *WatchMembershipRequest) Reset() { *x = WatchMembershipRequest{} - mi := &file_proto_client_client_api_proto_msgTypes[7] + mi := &file_proto_client_client_api_proto_msgTypes[10] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -620,7 +822,7 @@ func (x *WatchMembershipRequest) String() string { func (*WatchMembershipRequest) ProtoMessage() {} func (x *WatchMembershipRequest) ProtoReflect() protoreflect.Message { - mi := &file_proto_client_client_api_proto_msgTypes[7] + mi := &file_proto_client_client_api_proto_msgTypes[10] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -633,7 +835,7 @@ func (x *WatchMembershipRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use WatchMembershipRequest.ProtoReflect.Descriptor instead. func (*WatchMembershipRequest) Descriptor() ([]byte, []int) { - return file_proto_client_client_api_proto_rawDescGZIP(), []int{7} + return file_proto_client_client_api_proto_rawDescGZIP(), []int{10} } func (x *WatchMembershipRequest) GetClientId() uint32 { @@ -662,7 +864,7 @@ type MembershipSnapshot struct { func (x *MembershipSnapshot) Reset() { *x = MembershipSnapshot{} - mi := &file_proto_client_client_api_proto_msgTypes[8] + mi := &file_proto_client_client_api_proto_msgTypes[11] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -674,7 +876,7 @@ func (x *MembershipSnapshot) String() string { func (*MembershipSnapshot) ProtoMessage() {} func (x *MembershipSnapshot) ProtoReflect() protoreflect.Message { - mi := &file_proto_client_client_api_proto_msgTypes[8] + mi := &file_proto_client_client_api_proto_msgTypes[11] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -687,7 +889,7 @@ func (x *MembershipSnapshot) ProtoReflect() protoreflect.Message { // Deprecated: Use MembershipSnapshot.ProtoReflect.Descriptor instead. func (*MembershipSnapshot) Descriptor() ([]byte, []int) { - return file_proto_client_client_api_proto_rawDescGZIP(), []int{8} + return file_proto_client_client_api_proto_rawDescGZIP(), []int{11} } func (x *MembershipSnapshot) GetMembers() []uint32 { @@ -711,22 +913,30 @@ func (x *MembershipSnapshot) GetCommittedIndex() uint64 { return 0 } -// Request to watch for changes on a specific key -// -// In v1, only exact key matching is supported. -// Prefix watching may be added in future versions. +// Request to watch for changes on a key or key prefix. type WatchRequest struct { state protoimpl.MessageState `protogen:"open.v1"` ClientId uint32 `protobuf:"varint,1,opt,name=client_id,json=clientId,proto3" json:"client_id,omitempty"` - // Exact key to watch for changes - Key []byte `protobuf:"bytes,2,opt,name=key,proto3" json:"key,omitempty"` + // Key to watch. + // prefix=false: exact key match. + // prefix=true: prefix match — key must start with "/" and end with "/". + // + // e.g. "/config/" watches all keys under /config/. + Key []byte `protobuf:"bytes,2,opt,name=key,proto3" json:"key,omitempty"` + // When true, key is treated as a path prefix (must end with "/"). + // Default false preserves backwards-compatible exact-match behaviour. + Prefix bool `protobuf:"varint,3,opt,name=prefix,proto3" json:"prefix,omitempty"` + // When true, each WatchResponse includes the value that existed before + // the mutation (prev_value). Default false; server skips the extra read + // when no watcher requests it. + PrevKv bool `protobuf:"varint,4,opt,name=prev_kv,json=prevKv,proto3" json:"prev_kv,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } func (x *WatchRequest) Reset() { *x = WatchRequest{} - mi := &file_proto_client_client_api_proto_msgTypes[9] + mi := &file_proto_client_client_api_proto_msgTypes[12] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -738,7 +948,7 @@ func (x *WatchRequest) String() string { func (*WatchRequest) ProtoMessage() {} func (x *WatchRequest) ProtoReflect() protoreflect.Message { - mi := &file_proto_client_client_api_proto_msgTypes[9] + mi := &file_proto_client_client_api_proto_msgTypes[12] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -751,7 +961,7 @@ func (x *WatchRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use WatchRequest.ProtoReflect.Descriptor instead. func (*WatchRequest) Descriptor() ([]byte, []int) { - return file_proto_client_client_api_proto_rawDescGZIP(), []int{9} + return file_proto_client_client_api_proto_rawDescGZIP(), []int{12} } func (x *WatchRequest) GetClientId() uint32 { @@ -768,6 +978,20 @@ func (x *WatchRequest) GetKey() []byte { return nil } +func (x *WatchRequest) GetPrefix() bool { + if x != nil { + return x.Prefix + } + return false +} + +func (x *WatchRequest) GetPrevKv() bool { + if x != nil { + return x.PrevKv + } + return false +} + // Response containing a watch event notification type WatchResponse struct { state protoimpl.MessageState `protogen:"open.v1"` @@ -778,14 +1002,23 @@ type WatchResponse struct { // Type of change that occurred EventType WatchEventType `protobuf:"varint,3,opt,name=event_type,json=eventType,proto3,enum=d_engine.client.WatchEventType" json:"event_type,omitempty"` // Error information if watch failed - Error error1.ErrorCode `protobuf:"varint,4,opt,name=error,proto3,enum=d_engine.error.ErrorCode" json:"error,omitempty"` + Error error1.ErrorCode `protobuf:"varint,4,opt,name=error,proto3,enum=d_engine.error.ErrorCode" json:"error,omitempty"` + // Raft applied index at the time this event was produced. + // Monotonically increasing. Clients use this as an anchor after receiving + // CANCELED: re-read state, then re-register watching from revision+1. + Revision uint64 `protobuf:"varint,5,opt,name=revision,proto3" json:"revision,omitempty"` + // Value before this mutation. Empty when: + // - key did not exist before the write + // - event_type is PROGRESS or CANCELED + // - watcher was registered with prev_kv = false + PrevValue []byte `protobuf:"bytes,6,opt,name=prev_value,json=prevValue,proto3" json:"prev_value,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } func (x *WatchResponse) Reset() { *x = WatchResponse{} - mi := &file_proto_client_client_api_proto_msgTypes[10] + mi := &file_proto_client_client_api_proto_msgTypes[13] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -797,7 +1030,7 @@ func (x *WatchResponse) String() string { func (*WatchResponse) ProtoMessage() {} func (x *WatchResponse) ProtoReflect() protoreflect.Message { - mi := &file_proto_client_client_api_proto_msgTypes[10] + mi := &file_proto_client_client_api_proto_msgTypes[13] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -810,7 +1043,7 @@ func (x *WatchResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use WatchResponse.ProtoReflect.Descriptor instead. func (*WatchResponse) Descriptor() ([]byte, []int) { - return file_proto_client_client_api_proto_rawDescGZIP(), []int{10} + return file_proto_client_client_api_proto_rawDescGZIP(), []int{13} } func (x *WatchResponse) GetKey() []byte { @@ -841,20 +1074,31 @@ func (x *WatchResponse) GetError() error1.ErrorCode { return error1.ErrorCode(0) } +func (x *WatchResponse) GetRevision() uint64 { + if x != nil { + return x.Revision + } + return 0 +} + +func (x *WatchResponse) GetPrevValue() []byte { + if x != nil { + return x.PrevValue + } + return nil +} + type WriteCommand_Insert struct { - state protoimpl.MessageState `protogen:"open.v1"` - Key []byte `protobuf:"bytes,1,opt,name=key,proto3" json:"key,omitempty"` - Value []byte `protobuf:"bytes,2,opt,name=value,proto3" json:"value,omitempty"` - // Time-to-live in seconds. 0 means no expiration (default). - // Non-zero values specify expiration time in seconds from insertion. - TtlSecs uint64 `protobuf:"varint,3,opt,name=ttl_secs,json=ttlSecs,proto3" json:"ttl_secs,omitempty"` + state protoimpl.MessageState `protogen:"open.v1"` + Key []byte `protobuf:"bytes,1,opt,name=key,proto3" json:"key,omitempty"` + Value []byte `protobuf:"bytes,2,opt,name=value,proto3" json:"value,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } func (x *WriteCommand_Insert) Reset() { *x = WriteCommand_Insert{} - mi := &file_proto_client_client_api_proto_msgTypes[11] + mi := &file_proto_client_client_api_proto_msgTypes[14] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -866,7 +1110,7 @@ func (x *WriteCommand_Insert) String() string { func (*WriteCommand_Insert) ProtoMessage() {} func (x *WriteCommand_Insert) ProtoReflect() protoreflect.Message { - mi := &file_proto_client_client_api_proto_msgTypes[11] + mi := &file_proto_client_client_api_proto_msgTypes[14] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -896,13 +1140,6 @@ func (x *WriteCommand_Insert) GetValue() []byte { return nil } -func (x *WriteCommand_Insert) GetTtlSecs() uint64 { - if x != nil { - return x.TtlSecs - } - return 0 -} - type WriteCommand_Delete struct { state protoimpl.MessageState `protogen:"open.v1"` Key []byte `protobuf:"bytes,1,opt,name=key,proto3" json:"key,omitempty"` @@ -912,7 +1149,7 @@ type WriteCommand_Delete struct { func (x *WriteCommand_Delete) Reset() { *x = WriteCommand_Delete{} - mi := &file_proto_client_client_api_proto_msgTypes[12] + mi := &file_proto_client_client_api_proto_msgTypes[15] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -924,7 +1161,7 @@ func (x *WriteCommand_Delete) String() string { func (*WriteCommand_Delete) ProtoMessage() {} func (x *WriteCommand_Delete) ProtoReflect() protoreflect.Message { - mi := &file_proto_client_client_api_proto_msgTypes[12] + mi := &file_proto_client_client_api_proto_msgTypes[15] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -958,7 +1195,7 @@ type WriteCommand_CompareAndSwap struct { func (x *WriteCommand_CompareAndSwap) Reset() { *x = WriteCommand_CompareAndSwap{} - mi := &file_proto_client_client_api_proto_msgTypes[13] + mi := &file_proto_client_client_api_proto_msgTypes[16] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -970,7 +1207,7 @@ func (x *WriteCommand_CompareAndSwap) String() string { func (*WriteCommand_CompareAndSwap) ProtoMessage() {} func (x *WriteCommand_CompareAndSwap) ProtoReflect() protoreflect.Message { - mi := &file_proto_client_client_api_proto_msgTypes[13] + mi := &file_proto_client_client_api_proto_msgTypes[16] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1007,26 +1244,158 @@ func (x *WriteCommand_CompareAndSwap) GetNewValue() []byte { return nil } +type WriteCommand_BatchOp struct { + state protoimpl.MessageState `protogen:"open.v1"` + // Types that are valid to be assigned to Op: + // + // *WriteCommand_BatchOp_Insert + // *WriteCommand_BatchOp_Delete + Op isWriteCommand_BatchOp_Op `protobuf_oneof:"op"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *WriteCommand_BatchOp) Reset() { + *x = WriteCommand_BatchOp{} + mi := &file_proto_client_client_api_proto_msgTypes[17] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *WriteCommand_BatchOp) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*WriteCommand_BatchOp) ProtoMessage() {} + +func (x *WriteCommand_BatchOp) ProtoReflect() protoreflect.Message { + mi := &file_proto_client_client_api_proto_msgTypes[17] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use WriteCommand_BatchOp.ProtoReflect.Descriptor instead. +func (*WriteCommand_BatchOp) Descriptor() ([]byte, []int) { + return file_proto_client_client_api_proto_rawDescGZIP(), []int{0, 3} +} + +func (x *WriteCommand_BatchOp) GetOp() isWriteCommand_BatchOp_Op { + if x != nil { + return x.Op + } + return nil +} + +func (x *WriteCommand_BatchOp) GetInsert() *WriteCommand_Insert { + if x != nil { + if x, ok := x.Op.(*WriteCommand_BatchOp_Insert); ok { + return x.Insert + } + } + return nil +} + +func (x *WriteCommand_BatchOp) GetDelete() *WriteCommand_Delete { + if x != nil { + if x, ok := x.Op.(*WriteCommand_BatchOp_Delete); ok { + return x.Delete + } + } + return nil +} + +type isWriteCommand_BatchOp_Op interface { + isWriteCommand_BatchOp_Op() +} + +type WriteCommand_BatchOp_Insert struct { + Insert *WriteCommand_Insert `protobuf:"bytes,1,opt,name=insert,proto3,oneof"` +} + +type WriteCommand_BatchOp_Delete struct { + Delete *WriteCommand_Delete `protobuf:"bytes,2,opt,name=delete,proto3,oneof"` +} + +func (*WriteCommand_BatchOp_Insert) isWriteCommand_BatchOp_Op() {} + +func (*WriteCommand_BatchOp_Delete) isWriteCommand_BatchOp_Op() {} + +type WriteCommand_Batch struct { + state protoimpl.MessageState `protogen:"open.v1"` + Ops []*WriteCommand_BatchOp `protobuf:"bytes,1,rep,name=ops,proto3" json:"ops,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *WriteCommand_Batch) Reset() { + *x = WriteCommand_Batch{} + mi := &file_proto_client_client_api_proto_msgTypes[18] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *WriteCommand_Batch) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*WriteCommand_Batch) ProtoMessage() {} + +func (x *WriteCommand_Batch) ProtoReflect() protoreflect.Message { + mi := &file_proto_client_client_api_proto_msgTypes[18] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use WriteCommand_Batch.ProtoReflect.Descriptor instead. +func (*WriteCommand_Batch) Descriptor() ([]byte, []int) { + return file_proto_client_client_api_proto_rawDescGZIP(), []int{0, 4} +} + +func (x *WriteCommand_Batch) GetOps() []*WriteCommand_BatchOp { + if x != nil { + return x.Ops + } + return nil +} + var File_proto_client_client_api_proto protoreflect.FileDescriptor const file_proto_client_client_api_proto_rawDesc = "" + "\n" + - "\x1dproto/client/client_api.proto\x12\x0fd_engine.client\x1a\x11proto/error.proto\"\xde\x03\n" + + "\x1dproto/client/client_api.proto\x12\x0fd_engine.client\x1a\x11proto/error.proto\"\xe4\x05\n" + "\fWriteCommand\x12>\n" + "\x06insert\x18\x01 \x01(\v2$.d_engine.client.WriteCommand.InsertH\x00R\x06insert\x12>\n" + "\x06delete\x18\x02 \x01(\v2$.d_engine.client.WriteCommand.DeleteH\x00R\x06delete\x12X\n" + - "\x10compare_and_swap\x18\x03 \x01(\v2,.d_engine.client.WriteCommand.CompareAndSwapH\x00R\x0ecompareAndSwap\x1aK\n" + + "\x10compare_and_swap\x18\x03 \x01(\v2,.d_engine.client.WriteCommand.CompareAndSwapH\x00R\x0ecompareAndSwap\x12;\n" + + "\x05batch\x18\x04 \x01(\v2#.d_engine.client.WriteCommand.BatchH\x00R\x05batch\x1a@\n" + "\x06Insert\x12\x10\n" + "\x03key\x18\x01 \x01(\fR\x03key\x12\x14\n" + - "\x05value\x18\x02 \x01(\fR\x05value\x12\x19\n" + - "\bttl_secs\x18\x03 \x01(\x04R\attlSecs\x1a\x1a\n" + + "\x05value\x18\x02 \x01(\fR\x05valueJ\x04\b\x03\x10\x04R\bttl_secs\x1a\x1a\n" + "\x06Delete\x12\x10\n" + "\x03key\x18\x01 \x01(\fR\x03key\x1a~\n" + "\x0eCompareAndSwap\x12\x10\n" + "\x03key\x18\x01 \x01(\fR\x03key\x12*\n" + "\x0eexpected_value\x18\x02 \x01(\fH\x00R\rexpectedValue\x88\x01\x01\x12\x1b\n" + "\tnew_value\x18\x03 \x01(\fR\bnewValueB\x11\n" + - "\x0f_expected_valueB\v\n" + + "\x0f_expected_value\x1a\x8f\x01\n" + + "\aBatchOp\x12>\n" + + "\x06insert\x18\x01 \x01(\v2$.d_engine.client.WriteCommand.InsertH\x00R\x06insert\x12>\n" + + "\x06delete\x18\x02 \x01(\v2$.d_engine.client.WriteCommand.DeleteH\x00R\x06deleteB\x04\n" + + "\x02op\x1a@\n" + + "\x05Batch\x127\n" + + "\x03ops\x18\x01 \x03(\v2%.d_engine.client.WriteCommand.BatchOpR\x03opsB\v\n" + "\toperation\"j\n" + "\x12ClientWriteRequest\x12\x1b\n" + "\tclient_id\x18\x01 \x01(\rR\bclientId\x127\n" + @@ -1037,44 +1406,61 @@ const file_proto_client_client_api_proto_rawDesc = "" + "\x12consistency_policy\x18\x03 \x01(\x0e2&.d_engine.client.ReadConsistencyPolicyH\x00R\x11consistencyPolicy\x88\x01\x01B\x15\n" + "\x13_consistency_policy\"+\n" + "\vWriteResult\x12\x1c\n" + - "\tsucceeded\x18\x01 \x01(\bR\tsucceeded\"\x8e\x02\n" + + "\tsucceeded\x18\x01 \x01(\bR\tsucceeded\"\xcb\x02\n" + "\x0eClientResponse\x12/\n" + "\x05error\x18\x01 \x01(\x0e2\x19.d_engine.error.ErrorCodeR\x05error\x12A\n" + "\fwrite_result\x18\x02 \x01(\v2\x1c.d_engine.client.WriteResultH\x00R\vwriteResult\x12;\n" + - "\tread_data\x18\x03 \x01(\v2\x1c.d_engine.client.ReadResultsH\x00R\breadData\x129\n" + - "\bmetadata\x18\x04 \x01(\v2\x1d.d_engine.error.ErrorMetadataR\bmetadataB\x10\n" + + "\tread_data\x18\x03 \x01(\v2\x1c.d_engine.client.ReadResultsH\x00R\breadData\x12;\n" + + "\tscan_data\x18\x04 \x01(\v2\x1c.d_engine.client.ScanResultsH\x00R\bscanData\x129\n" + + "\bmetadata\x18\x05 \x01(\v2\x1d.d_engine.error.ErrorMetadataR\bmetadataB\x10\n" + "\x0esuccess_result\"6\n" + "\fClientResult\x12\x10\n" + "\x03key\x18\x01 \x01(\fR\x03key\x12\x14\n" + "\x05value\x18\x02 \x01(\fR\x05value\"F\n" + "\vReadResults\x127\n" + - "\aresults\x18\x01 \x03(\v2\x1d.d_engine.client.ClientResultR\aresults\"5\n" + + "\aresults\x18\x01 \x03(\v2\x1d.d_engine.client.ClientResultR\aresults\"1\n" + + "\aKvEntry\x12\x10\n" + + "\x03key\x18\x01 \x01(\fR\x03key\x12\x14\n" + + "\x05value\x18\x02 \x01(\fR\x05value\"B\n" + + "\vScanRequest\x12\x1b\n" + + "\tclient_id\x18\x01 \x01(\rR\bclientId\x12\x16\n" + + "\x06prefix\x18\x02 \x01(\fR\x06prefix\"]\n" + + "\vScanResults\x122\n" + + "\aentries\x18\x01 \x03(\v2\x18.d_engine.client.KvEntryR\aentries\x12\x1a\n" + + "\brevision\x18\x02 \x01(\x04R\brevision\"5\n" + "\x16WatchMembershipRequest\x12\x1b\n" + "\tclient_id\x18\x01 \x01(\rR\bclientId\"s\n" + "\x12MembershipSnapshot\x12\x18\n" + "\amembers\x18\x01 \x03(\rR\amembers\x12\x1a\n" + "\blearners\x18\x02 \x03(\rR\blearners\x12'\n" + - "\x0fcommitted_index\x18\x03 \x01(\x04R\x0ecommittedIndex\"=\n" + + "\x0fcommitted_index\x18\x03 \x01(\x04R\x0ecommittedIndex\"n\n" + "\fWatchRequest\x12\x1b\n" + "\tclient_id\x18\x01 \x01(\rR\bclientId\x12\x10\n" + - "\x03key\x18\x02 \x01(\fR\x03key\"\xa8\x01\n" + + "\x03key\x18\x02 \x01(\fR\x03key\x12\x16\n" + + "\x06prefix\x18\x03 \x01(\bR\x06prefix\x12\x17\n" + + "\aprev_kv\x18\x04 \x01(\bR\x06prevKv\"\xe3\x01\n" + "\rWatchResponse\x12\x10\n" + "\x03key\x18\x01 \x01(\fR\x03key\x12\x14\n" + "\x05value\x18\x02 \x01(\fR\x05value\x12>\n" + "\n" + "event_type\x18\x03 \x01(\x0e2\x1f.d_engine.client.WatchEventTypeR\teventType\x12/\n" + - "\x05error\x18\x04 \x01(\x0e2\x19.d_engine.error.ErrorCodeR\x05error*\xa0\x01\n" + + "\x05error\x18\x04 \x01(\x0e2\x19.d_engine.error.ErrorCodeR\x05error\x12\x1a\n" + + "\brevision\x18\x05 \x01(\x04R\brevision\x12\x1d\n" + + "\n" + + "prev_value\x18\x06 \x01(\fR\tprevValue*\xa0\x01\n" + "\x15ReadConsistencyPolicy\x12&\n" + "\"READ_CONSISTENCY_POLICY_LEASE_READ\x10\x00\x12-\n" + ")READ_CONSISTENCY_POLICY_LINEARIZABLE_READ\x10\x01\x120\n" + - ",READ_CONSISTENCY_POLICY_EVENTUAL_CONSISTENCY\x10\x02*f\n" + + ",READ_CONSISTENCY_POLICY_EVENTUAL_CONSISTENCY\x10\x02*\x85\x01\n" + "\x0eWatchEventType\x12\x18\n" + "\x14WATCH_EVENT_TYPE_PUT\x10\x00\x12\x1b\n" + "\x17WATCH_EVENT_TYPE_DELETE\x10\x01\x12\x1d\n" + - "\x19WATCH_EVENT_TYPE_CANCELED\x10\x022\xf4\x02\n" + + "\x19WATCH_EVENT_TYPE_CANCELED\x10\x02\x12\x1d\n" + + "\x19WATCH_EVENT_TYPE_PROGRESS\x10\x032\xc7\x03\n" + "\x11RaftClientService\x12Y\n" + "\x11HandleClientWrite\x12#.d_engine.client.ClientWriteRequest\x1a\x1f.d_engine.client.ClientResponse\x12W\n" + - "\x10HandleClientRead\x12\".d_engine.client.ClientReadRequest\x1a\x1f.d_engine.client.ClientResponse\x12H\n" + + "\x10HandleClientRead\x12\".d_engine.client.ClientReadRequest\x1a\x1f.d_engine.client.ClientResponse\x12Q\n" + + "\x10HandleClientScan\x12\x1c.d_engine.client.ScanRequest\x1a\x1f.d_engine.client.ClientResponse\x12H\n" + "\x05Watch\x12\x1d.d_engine.client.WatchRequest\x1a\x1e.d_engine.client.WatchResponse0\x01\x12a\n" + "\x0fWatchMembership\x12'.d_engine.client.WatchMembershipRequest\x1a#.d_engine.client.MembershipSnapshot0\x01B,Z*github.com/deventlab/d-engine/proto/clientb\x06proto3" @@ -1091,7 +1477,7 @@ func file_proto_client_client_api_proto_rawDescGZIP() []byte { } var file_proto_client_client_api_proto_enumTypes = make([]protoimpl.EnumInfo, 2) -var file_proto_client_client_api_proto_msgTypes = make([]protoimpl.MessageInfo, 14) +var file_proto_client_client_api_proto_msgTypes = make([]protoimpl.MessageInfo, 19) var file_proto_client_client_api_proto_goTypes = []any{ (ReadConsistencyPolicy)(0), // 0: d_engine.client.ReadConsistencyPolicy (WatchEventType)(0), // 1: d_engine.client.WatchEventType @@ -1102,42 +1488,55 @@ var file_proto_client_client_api_proto_goTypes = []any{ (*ClientResponse)(nil), // 6: d_engine.client.ClientResponse (*ClientResult)(nil), // 7: d_engine.client.ClientResult (*ReadResults)(nil), // 8: d_engine.client.ReadResults - (*WatchMembershipRequest)(nil), // 9: d_engine.client.WatchMembershipRequest - (*MembershipSnapshot)(nil), // 10: d_engine.client.MembershipSnapshot - (*WatchRequest)(nil), // 11: d_engine.client.WatchRequest - (*WatchResponse)(nil), // 12: d_engine.client.WatchResponse - (*WriteCommand_Insert)(nil), // 13: d_engine.client.WriteCommand.Insert - (*WriteCommand_Delete)(nil), // 14: d_engine.client.WriteCommand.Delete - (*WriteCommand_CompareAndSwap)(nil), // 15: d_engine.client.WriteCommand.CompareAndSwap - (error1.ErrorCode)(0), // 16: d_engine.error.ErrorCode - (*error1.ErrorMetadata)(nil), // 17: d_engine.error.ErrorMetadata + (*KvEntry)(nil), // 9: d_engine.client.KvEntry + (*ScanRequest)(nil), // 10: d_engine.client.ScanRequest + (*ScanResults)(nil), // 11: d_engine.client.ScanResults + (*WatchMembershipRequest)(nil), // 12: d_engine.client.WatchMembershipRequest + (*MembershipSnapshot)(nil), // 13: d_engine.client.MembershipSnapshot + (*WatchRequest)(nil), // 14: d_engine.client.WatchRequest + (*WatchResponse)(nil), // 15: d_engine.client.WatchResponse + (*WriteCommand_Insert)(nil), // 16: d_engine.client.WriteCommand.Insert + (*WriteCommand_Delete)(nil), // 17: d_engine.client.WriteCommand.Delete + (*WriteCommand_CompareAndSwap)(nil), // 18: d_engine.client.WriteCommand.CompareAndSwap + (*WriteCommand_BatchOp)(nil), // 19: d_engine.client.WriteCommand.BatchOp + (*WriteCommand_Batch)(nil), // 20: d_engine.client.WriteCommand.Batch + (error1.ErrorCode)(0), // 21: d_engine.error.ErrorCode + (*error1.ErrorMetadata)(nil), // 22: d_engine.error.ErrorMetadata } var file_proto_client_client_api_proto_depIdxs = []int32{ - 13, // 0: d_engine.client.WriteCommand.insert:type_name -> d_engine.client.WriteCommand.Insert - 14, // 1: d_engine.client.WriteCommand.delete:type_name -> d_engine.client.WriteCommand.Delete - 15, // 2: d_engine.client.WriteCommand.compare_and_swap:type_name -> d_engine.client.WriteCommand.CompareAndSwap - 2, // 3: d_engine.client.ClientWriteRequest.command:type_name -> d_engine.client.WriteCommand - 0, // 4: d_engine.client.ClientReadRequest.consistency_policy:type_name -> d_engine.client.ReadConsistencyPolicy - 16, // 5: d_engine.client.ClientResponse.error:type_name -> d_engine.error.ErrorCode - 5, // 6: d_engine.client.ClientResponse.write_result:type_name -> d_engine.client.WriteResult - 8, // 7: d_engine.client.ClientResponse.read_data:type_name -> d_engine.client.ReadResults - 17, // 8: d_engine.client.ClientResponse.metadata:type_name -> d_engine.error.ErrorMetadata - 7, // 9: d_engine.client.ReadResults.results:type_name -> d_engine.client.ClientResult - 1, // 10: d_engine.client.WatchResponse.event_type:type_name -> d_engine.client.WatchEventType - 16, // 11: d_engine.client.WatchResponse.error:type_name -> d_engine.error.ErrorCode - 3, // 12: d_engine.client.RaftClientService.HandleClientWrite:input_type -> d_engine.client.ClientWriteRequest - 4, // 13: d_engine.client.RaftClientService.HandleClientRead:input_type -> d_engine.client.ClientReadRequest - 11, // 14: d_engine.client.RaftClientService.Watch:input_type -> d_engine.client.WatchRequest - 9, // 15: d_engine.client.RaftClientService.WatchMembership:input_type -> d_engine.client.WatchMembershipRequest - 6, // 16: d_engine.client.RaftClientService.HandleClientWrite:output_type -> d_engine.client.ClientResponse - 6, // 17: d_engine.client.RaftClientService.HandleClientRead:output_type -> d_engine.client.ClientResponse - 12, // 18: d_engine.client.RaftClientService.Watch:output_type -> d_engine.client.WatchResponse - 10, // 19: d_engine.client.RaftClientService.WatchMembership:output_type -> d_engine.client.MembershipSnapshot - 16, // [16:20] is the sub-list for method output_type - 12, // [12:16] is the sub-list for method input_type - 12, // [12:12] is the sub-list for extension type_name - 12, // [12:12] is the sub-list for extension extendee - 0, // [0:12] is the sub-list for field type_name + 16, // 0: d_engine.client.WriteCommand.insert:type_name -> d_engine.client.WriteCommand.Insert + 17, // 1: d_engine.client.WriteCommand.delete:type_name -> d_engine.client.WriteCommand.Delete + 18, // 2: d_engine.client.WriteCommand.compare_and_swap:type_name -> d_engine.client.WriteCommand.CompareAndSwap + 20, // 3: d_engine.client.WriteCommand.batch:type_name -> d_engine.client.WriteCommand.Batch + 2, // 4: d_engine.client.ClientWriteRequest.command:type_name -> d_engine.client.WriteCommand + 0, // 5: d_engine.client.ClientReadRequest.consistency_policy:type_name -> d_engine.client.ReadConsistencyPolicy + 21, // 6: d_engine.client.ClientResponse.error:type_name -> d_engine.error.ErrorCode + 5, // 7: d_engine.client.ClientResponse.write_result:type_name -> d_engine.client.WriteResult + 8, // 8: d_engine.client.ClientResponse.read_data:type_name -> d_engine.client.ReadResults + 11, // 9: d_engine.client.ClientResponse.scan_data:type_name -> d_engine.client.ScanResults + 22, // 10: d_engine.client.ClientResponse.metadata:type_name -> d_engine.error.ErrorMetadata + 7, // 11: d_engine.client.ReadResults.results:type_name -> d_engine.client.ClientResult + 9, // 12: d_engine.client.ScanResults.entries:type_name -> d_engine.client.KvEntry + 1, // 13: d_engine.client.WatchResponse.event_type:type_name -> d_engine.client.WatchEventType + 21, // 14: d_engine.client.WatchResponse.error:type_name -> d_engine.error.ErrorCode + 16, // 15: d_engine.client.WriteCommand.BatchOp.insert:type_name -> d_engine.client.WriteCommand.Insert + 17, // 16: d_engine.client.WriteCommand.BatchOp.delete:type_name -> d_engine.client.WriteCommand.Delete + 19, // 17: d_engine.client.WriteCommand.Batch.ops:type_name -> d_engine.client.WriteCommand.BatchOp + 3, // 18: d_engine.client.RaftClientService.HandleClientWrite:input_type -> d_engine.client.ClientWriteRequest + 4, // 19: d_engine.client.RaftClientService.HandleClientRead:input_type -> d_engine.client.ClientReadRequest + 10, // 20: d_engine.client.RaftClientService.HandleClientScan:input_type -> d_engine.client.ScanRequest + 14, // 21: d_engine.client.RaftClientService.Watch:input_type -> d_engine.client.WatchRequest + 12, // 22: d_engine.client.RaftClientService.WatchMembership:input_type -> d_engine.client.WatchMembershipRequest + 6, // 23: d_engine.client.RaftClientService.HandleClientWrite:output_type -> d_engine.client.ClientResponse + 6, // 24: d_engine.client.RaftClientService.HandleClientRead:output_type -> d_engine.client.ClientResponse + 6, // 25: d_engine.client.RaftClientService.HandleClientScan:output_type -> d_engine.client.ClientResponse + 15, // 26: d_engine.client.RaftClientService.Watch:output_type -> d_engine.client.WatchResponse + 13, // 27: d_engine.client.RaftClientService.WatchMembership:output_type -> d_engine.client.MembershipSnapshot + 23, // [23:28] is the sub-list for method output_type + 18, // [18:23] is the sub-list for method input_type + 18, // [18:18] is the sub-list for extension type_name + 18, // [18:18] is the sub-list for extension extendee + 0, // [0:18] is the sub-list for field type_name } func init() { file_proto_client_client_api_proto_init() } @@ -1149,20 +1548,26 @@ func file_proto_client_client_api_proto_init() { (*WriteCommand_Insert_)(nil), (*WriteCommand_Delete_)(nil), (*WriteCommand_CompareAndSwap_)(nil), + (*WriteCommand_Batch_)(nil), } file_proto_client_client_api_proto_msgTypes[2].OneofWrappers = []any{} file_proto_client_client_api_proto_msgTypes[4].OneofWrappers = []any{ (*ClientResponse_WriteResult)(nil), (*ClientResponse_ReadData)(nil), + (*ClientResponse_ScanData)(nil), + } + file_proto_client_client_api_proto_msgTypes[16].OneofWrappers = []any{} + file_proto_client_client_api_proto_msgTypes[17].OneofWrappers = []any{ + (*WriteCommand_BatchOp_Insert)(nil), + (*WriteCommand_BatchOp_Delete)(nil), } - file_proto_client_client_api_proto_msgTypes[13].OneofWrappers = []any{} type x struct{} out := protoimpl.TypeBuilder{ File: protoimpl.DescBuilder{ GoPackagePath: reflect.TypeOf(x{}).PkgPath(), RawDescriptor: unsafe.Slice(unsafe.StringData(file_proto_client_client_api_proto_rawDesc), len(file_proto_client_client_api_proto_rawDesc)), NumEnums: 2, - NumMessages: 14, + NumMessages: 19, NumExtensions: 0, NumServices: 1, }, diff --git a/d-engine-proto/go/client/client_api_grpc.pb.go b/d-engine-proto/go/client/client_api_grpc.pb.go index 70da120d..70f4f066 100644 --- a/d-engine-proto/go/client/client_api_grpc.pb.go +++ b/d-engine-proto/go/client/client_api_grpc.pb.go @@ -21,6 +21,7 @@ const _ = grpc.SupportPackageIsVersion9 const ( RaftClientService_HandleClientWrite_FullMethodName = "/d_engine.client.RaftClientService/HandleClientWrite" RaftClientService_HandleClientRead_FullMethodName = "/d_engine.client.RaftClientService/HandleClientRead" + RaftClientService_HandleClientScan_FullMethodName = "/d_engine.client.RaftClientService/HandleClientScan" RaftClientService_Watch_FullMethodName = "/d_engine.client.RaftClientService/Watch" RaftClientService_WatchMembership_FullMethodName = "/d_engine.client.RaftClientService/WatchMembership" ) @@ -31,6 +32,7 @@ const ( type RaftClientServiceClient interface { HandleClientWrite(ctx context.Context, in *ClientWriteRequest, opts ...grpc.CallOption) (*ClientResponse, error) HandleClientRead(ctx context.Context, in *ClientReadRequest, opts ...grpc.CallOption) (*ClientResponse, error) + HandleClientScan(ctx context.Context, in *ScanRequest, opts ...grpc.CallOption) (*ClientResponse, error) // Watch for changes to a specific key. // // Returns a stream of WatchResponse events whenever the watched key changes. @@ -88,6 +90,16 @@ func (c *raftClientServiceClient) HandleClientRead(ctx context.Context, in *Clie return out, nil } +func (c *raftClientServiceClient) HandleClientScan(ctx context.Context, in *ScanRequest, opts ...grpc.CallOption) (*ClientResponse, error) { + cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...) + out := new(ClientResponse) + err := c.cc.Invoke(ctx, RaftClientService_HandleClientScan_FullMethodName, in, out, cOpts...) + if err != nil { + return nil, err + } + return out, nil +} + func (c *raftClientServiceClient) Watch(ctx context.Context, in *WatchRequest, opts ...grpc.CallOption) (grpc.ServerStreamingClient[WatchResponse], error) { cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...) stream, err := c.cc.NewStream(ctx, &RaftClientService_ServiceDesc.Streams[0], RaftClientService_Watch_FullMethodName, cOpts...) @@ -132,6 +144,7 @@ type RaftClientService_WatchMembershipClient = grpc.ServerStreamingClient[Member type RaftClientServiceServer interface { HandleClientWrite(context.Context, *ClientWriteRequest) (*ClientResponse, error) HandleClientRead(context.Context, *ClientReadRequest) (*ClientResponse, error) + HandleClientScan(context.Context, *ScanRequest) (*ClientResponse, error) // Watch for changes to a specific key. // // Returns a stream of WatchResponse events whenever the watched key changes. @@ -175,6 +188,9 @@ func (UnimplementedRaftClientServiceServer) HandleClientWrite(context.Context, * func (UnimplementedRaftClientServiceServer) HandleClientRead(context.Context, *ClientReadRequest) (*ClientResponse, error) { return nil, status.Errorf(codes.Unimplemented, "method HandleClientRead not implemented") } +func (UnimplementedRaftClientServiceServer) HandleClientScan(context.Context, *ScanRequest) (*ClientResponse, error) { + return nil, status.Errorf(codes.Unimplemented, "method HandleClientScan not implemented") +} func (UnimplementedRaftClientServiceServer) Watch(*WatchRequest, grpc.ServerStreamingServer[WatchResponse]) error { return status.Errorf(codes.Unimplemented, "method Watch not implemented") } @@ -238,6 +254,24 @@ func _RaftClientService_HandleClientRead_Handler(srv interface{}, ctx context.Co return interceptor(ctx, in, info, handler) } +func _RaftClientService_HandleClientScan_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { + in := new(ScanRequest) + if err := dec(in); err != nil { + return nil, err + } + if interceptor == nil { + return srv.(RaftClientServiceServer).HandleClientScan(ctx, in) + } + info := &grpc.UnaryServerInfo{ + Server: srv, + FullMethod: RaftClientService_HandleClientScan_FullMethodName, + } + handler := func(ctx context.Context, req interface{}) (interface{}, error) { + return srv.(RaftClientServiceServer).HandleClientScan(ctx, req.(*ScanRequest)) + } + return interceptor(ctx, in, info, handler) +} + func _RaftClientService_Watch_Handler(srv interface{}, stream grpc.ServerStream) error { m := new(WatchRequest) if err := stream.RecvMsg(m); err != nil { @@ -275,6 +309,10 @@ var RaftClientService_ServiceDesc = grpc.ServiceDesc{ MethodName: "HandleClientRead", Handler: _RaftClientService_HandleClientRead_Handler, }, + { + MethodName: "HandleClientScan", + Handler: _RaftClientService_HandleClientScan_Handler, + }, }, Streams: []grpc.StreamDesc{ { diff --git a/d-engine-proto/proto/client/client_api.proto b/d-engine-proto/proto/client/client_api.proto index 2e7b444b..ed3a659b 100644 --- a/d-engine-proto/proto/client/client_api.proto +++ b/d-engine-proto/proto/client/client_api.proto @@ -8,11 +8,10 @@ option go_package = "github.com/deventlab/d-engine/proto/client"; // Write operation-specific command message WriteCommand { message Insert { + reserved 3; + reserved "ttl_secs"; bytes key = 1; bytes value = 2; - // Time-to-live in seconds. 0 means no expiration (default). - // Non-zero values specify expiration time in seconds from insertion. - uint64 ttl_secs = 3; } message Delete { bytes key = 1; diff --git a/d-engine-proto/src/exts/client_ext.rs b/d-engine-proto/src/exts/client_ext.rs index 52eeddf3..6e8f855b 100644 --- a/d-engine-proto/src/exts/client_ext.rs +++ b/d-engine-proto/src/exts/client_ext.rs @@ -61,28 +61,6 @@ impl WriteCommand { let cmd = write_command::Insert { key: key.into(), value: value.into(), - ttl_secs: 0, - }; - Self { - operation: Some(write_command::Operation::Insert(cmd)), - } - } - - /// Create write command for key-value pair with TTL - /// - /// # Parameters - /// - `key`: Byte array for storage key - /// - `value`: Byte array to be stored - /// - `ttl_secs`: Time-to-live in seconds - pub fn insert_with_ttl( - key: impl Into, - value: impl Into, - ttl_secs: u64, - ) -> Self { - let cmd = write_command::Insert { - key: key.into(), - value: value.into(), - ttl_secs, }; Self { operation: Some(write_command::Operation::Insert(cmd)), diff --git a/d-engine-proto/src/generated/d_engine.client.rs b/d-engine-proto/src/generated/d_engine.client.rs index 873e3c2f..f7a31ad8 100644 --- a/d-engine-proto/src/generated/d_engine.client.rs +++ b/d-engine-proto/src/generated/d_engine.client.rs @@ -15,10 +15,6 @@ pub mod write_command { pub key: ::prost::bytes::Bytes, #[prost(bytes = "bytes", tag = "2")] pub value: ::prost::bytes::Bytes, - /// Time-to-live in seconds. 0 means no expiration (default). - /// Non-zero values specify expiration time in seconds from insertion. - #[prost(uint64, tag = "3")] - pub ttl_secs: u64, } #[derive(serde::Serialize, serde::Deserialize)] #[derive(Clone, PartialEq, ::prost::Message)] diff --git a/d-engine-server/Cargo.toml b/d-engine-server/Cargo.toml index c3af3f9f..fd585e93 100644 --- a/d-engine-server/Cargo.toml +++ b/d-engine-server/Cargo.toml @@ -84,19 +84,11 @@ name = "state_machine" harness = false required-features = ["watch"] -[[bench]] -name = "ttl" -harness = false - [[bench]] name = "watch_overhead" harness = false required-features = ["watch", "rocksdb"] -[[bench]] -name = "lease_performance" -harness = false - [[bench]] name = "file_log_store" harness = false diff --git a/d-engine-server/benches/README.md b/d-engine-server/benches/README.md index eb0c349d..6d57da26 100644 --- a/d-engine-server/benches/README.md +++ b/d-engine-server/benches/README.md @@ -1,6 +1,6 @@ # D-Engine Performance Benchmarks -This directory contains Criterion-based performance benchmarks for the d-engine state machine and TTL functionality. +This directory contains Criterion-based performance benchmarks for the d-engine state machine and Watch mechanism. ## Overview @@ -15,28 +15,13 @@ We use [Criterion.rs](https://github.com/bheisler/criterion.rs) for benchmarking ### 1. State Machine Benchmarks (`state_machine.rs`) -Tests core state machine performance with focus on TTL overhead: +Tests core state machine performance and Watch overhead: -- **apply_without_ttl** - Baseline apply performance (target: < 10ns overhead) -- **apply_with_ttl** - Apply with TTL registration overhead -- **get_without_ttl** - Baseline read performance -- **get_with_ttl_check** - Read with TTL passive check (target: < 50ns overhead) -- **get_expired_ttl** - Cost of passive deletion on read +- **apply** - Baseline apply performance +- **get** - Baseline read performance - **batch_apply** - Scaling test for batch operations (10, 100, 1000 entries) -- **batch_apply_with_ttl** - Batch operations with TTL - -### 2. TTL Benchmarks (`ttl.rs`) - -Tests TTL-specific functionality: - -- **piggyback_cleanup** - Cleanup performance with varying expired key counts (target: < 1ms for 100 keys) -- **ttl_registration** - Cost of registering TTL entries -- **batch_ttl_registration** - TTL registration scaling (10, 100, 1000 entries) -- **mixed_ttl_workload** - Realistic mix of active and expired entries -- **piggyback_high_frequency** - Cost of frequent cleanup triggers -- **varying_ttl_durations** - Performance with different TTL values (1min, 1hr, 1day) -- **worst_case_all_expired** - All 1000 entries expired -- **best_case_no_expired** - No cleanup needed +- **apply_without_watch**, **apply_with_1_watcher**, **apply_with_10_watchers**, **apply_with_100_watchers** - Watch overhead on the apply path +- **watch_e2e_latency** - Watch notification latency ## Running Benchmarks @@ -48,12 +33,11 @@ cargo bench --package d-engine-server ### Run specific benchmark suite ```bash cargo bench --package d-engine-server --bench state_machine -cargo bench --package d-engine-server --bench ttl ``` ### Run specific test ```bash -cargo bench --package d-engine-server --bench state_machine apply_without_ttl +cargo bench --package d-engine-server --bench state_machine apply ``` ### Quick run (fewer samples) @@ -80,9 +64,7 @@ Based on our design goals: | Metric | Target | Benchmark | |--------|--------|-----------| -| No TTL overhead | < 10ns | `apply_without_ttl` vs `apply_with_ttl` | -| TTL passive check | < 50ns | `get_with_ttl_check` vs `get_without_ttl` | -| Piggyback cleanup (100 keys) | < 1ms | `piggyback_cleanup/100` | +| Watch overhead on apply path | < 10ns per watcher | `apply_with_*_watcher*` vs `apply_without_watch` | | Batch apply scaling | Linear | `batch_apply/*` | ## CI Integration @@ -169,4 +151,3 @@ fn bench_my_new_test(c: &mut Criterion) { - [Criterion.rs Documentation](https://bheisler.github.io/criterion.rs/book/) - [Rust Performance Book](https://nnethercote.github.io/perf-book/) -- [d-engine TTL Design Doc](../../d-engine-product-design/20251107/) diff --git a/d-engine-server/benches/lease_performance.rs b/d-engine-server/benches/lease_performance.rs deleted file mode 100644 index 1384dcac..00000000 --- a/d-engine-server/benches/lease_performance.rs +++ /dev/null @@ -1,206 +0,0 @@ -//! Lease Performance Benchmarks - Simplified Configuration -//! -//! This benchmark validates the performance of the simplified lease configuration: -//! - get() method: Background mode (no on-read overhead) -//! - apply_chunk TTL registration: optimized with cached flags -//! -//! These benchmarks compare: -//! 1. Background strategy vs No Lease overhead in get() -//! 2. TTL registration performance in apply_chunk() -//! 3. Batch operations with TTL - -use std::sync::Arc; -use std::sync::atomic::AtomicU64; -use std::sync::atomic::Ordering; - -use bytes::Bytes; -use criterion::BenchmarkId; -use criterion::Criterion; -use criterion::black_box; -use criterion::criterion_group; -use criterion::criterion_main; -use d_engine_core::ApplyEntry; -use d_engine_core::Command; -use d_engine_core::StateMachine; -use d_engine_server::storage::FileStateMachine; -use d_engine_server::storage::TtlLease; -use tempfile::TempDir; - -/// Create FileStateMachine with Background strategy (lease enabled) -async fn create_sm_background() -> (FileStateMachine, TempDir) { - let temp_dir = TempDir::new().expect("Failed to create temp dir"); - let lease_config = d_engine_core::config::LeaseConfig { - cleanup_interval_ms: 1000, - max_cleanup_duration_ms: 1, - }; - - let mut sm = FileStateMachine::new(temp_dir.path().to_path_buf()) - .await - .expect("Failed to create state machine"); - - let lease = Arc::new(TtlLease::new(lease_config)); - sm.set_lease(lease); - - (sm, temp_dir) -} - -/// Create FileStateMachine without lease (no TTL overhead) -async fn create_sm_no_lease() -> (FileStateMachine, TempDir) { - let temp_dir = TempDir::new().expect("Failed to create temp dir"); - let sm = FileStateMachine::new(temp_dir.path().to_path_buf()) - .await - .expect("Failed to create state machine"); - - (sm, temp_dir) -} - -/// Helper to create entry with TTL -fn create_entry_with_ttl( - index: u64, - term: u64, - key: &[u8], - value: &[u8], - ttl_secs: u64, -) -> ApplyEntry { - ApplyEntry { - index, - term, - command: Command::Insert { - key: Bytes::copy_from_slice(key), - value: Bytes::copy_from_slice(value), - ttl_secs: if ttl_secs > 0 { Some(ttl_secs) } else { None }, - }, - } -} - -/// Benchmark: get() performance with different strategies -fn bench_get_performance(c: &mut Criterion) { - let mut group = c.benchmark_group("get_performance"); - - // Scenario 1: No lease (baseline) - let rt = tokio::runtime::Builder::new_current_thread().enable_all().build().unwrap(); - let (sm, _temp) = rt.block_on(create_sm_no_lease()); - - // Insert key without TTL - rt.block_on(async { - let entry = create_entry_with_ttl(1, 1, b"bench_key", b"bench_value", 0); - sm.apply_chunk(&[entry]).await.unwrap(); - }); - - group.bench_function("no_lease_baseline", |b| { - b.iter(|| { - let result = sm.get(black_box(b"bench_key")).unwrap(); - black_box(result); - }); - }); - - drop(sm); - drop(_temp); - - // Scenario 2: Background strategy (lease enabled, no on-read checks) - let (sm_bg, _temp_bg) = rt.block_on(create_sm_background()); - - rt.block_on(async { - let entry = create_entry_with_ttl(1, 1, b"bg_key", b"bg_value", 3600); - sm_bg.apply_chunk(&[entry]).await.unwrap(); - }); - - group.bench_function("background_strategy", |b| { - b.iter(|| { - let result = sm_bg.get(black_box(b"bg_key")).unwrap(); - black_box(result); - }); - }); - - drop(sm_bg); - drop(_temp_bg); - - group.finish(); -} - -/// Benchmark: apply_chunk TTL registration performance -fn bench_ttl_registration(c: &mut Criterion) { - let mut group = c.benchmark_group("ttl_registration"); - - let rt = tokio::runtime::Builder::new_current_thread().enable_all().build().unwrap(); - - // Scenario 1: No TTL (baseline) - let (sm_no_ttl, _temp1) = rt.block_on(create_sm_background()); - - let idx1 = AtomicU64::new(1); - group.bench_function("no_ttl_baseline", |b| { - b.to_async(&rt).iter(|| async { - let i = idx1.fetch_add(1, Ordering::Relaxed); - let entry = create_entry_with_ttl(i, 1, b"key", b"value", 0); - sm_no_ttl.apply_chunk(&[entry]).await.unwrap(); - }); - }); - - drop(sm_no_ttl); - drop(_temp1); - - // Scenario 2: With TTL (lease enabled) - let (sm_ttl, _temp2) = rt.block_on(create_sm_background()); - - let idx2 = AtomicU64::new(1); - group.bench_function("with_ttl_background", |b| { - b.to_async(&rt).iter(|| async { - let i = idx2.fetch_add(1, Ordering::Relaxed); - let entry = create_entry_with_ttl(i, 1, b"key", b"value", 3600); - sm_ttl.apply_chunk(&[entry]).await.unwrap(); - }); - }); - - drop(sm_ttl); - drop(_temp2); - - group.finish(); -} - -/// Benchmark: Batch operations with TTL -fn bench_batch_ttl_operations(c: &mut Criterion) { - let mut group = c.benchmark_group("batch_ttl_operations"); - - let rt = tokio::runtime::Builder::new_current_thread().enable_all().build().unwrap(); - - for batch_size in [10, 100, 1000].iter() { - // Create fresh StateMachine for each batch size to avoid state pollution - let (sm_bg, _temp_bg) = rt.block_on(create_sm_background()); - let idx = AtomicU64::new(1); - - group.bench_with_input( - BenchmarkId::new("background", batch_size), - batch_size, - |b, &size| { - b.to_async(&rt).iter(|| async { - let start = idx.fetch_add(size as u64, Ordering::Relaxed); - let entries: Vec = (0..size) - .map(|i| { - create_entry_with_ttl( - start + i as u64, - 1, - format!("key_{i}").as_bytes(), - b"value", - 3600, - ) - }) - .collect(); - sm_bg.apply_chunk(&entries).await.unwrap(); - }); - }, - ); - - drop(sm_bg); - drop(_temp_bg); - } - - group.finish(); -} - -criterion_group!( - benches, - bench_get_performance, - bench_ttl_registration, - bench_batch_ttl_operations -); -criterion_main!(benches); diff --git a/d-engine-server/benches/state_machine.rs b/d-engine-server/benches/state_machine.rs index f7dc9b4a..61a86cec 100644 --- a/d-engine-server/benches/state_machine.rs +++ b/d-engine-server/benches/state_machine.rs @@ -1,11 +1,9 @@ //! State Machine Performance Benchmarks //! //! This benchmark suite measures the core performance characteristics of the state machine, -//! focusing on the overhead introduced by TTL functionality and Watch mechanism. +//! focusing on apply, read and the overhead introduced by the Watch mechanism. //! //! Performance Targets: -//! - Without TTL: < 10ns overhead per operation -//! - With TTL passive check: < 50ns overhead per read //! - Watch overhead on Apply path: < 0.01% (< 10ns per watcher) //! - End-to-end Watch notification latency: < 100µs //! - Batch operations: Linear scaling @@ -13,7 +11,6 @@ use std::sync::Arc; use std::sync::atomic::AtomicU64; use std::sync::atomic::Ordering; -use std::time::Duration; use bytes::Bytes; use criterion::BenchmarkId; @@ -35,18 +32,10 @@ use tokio::sync::mpsc; /// Helper to create a temporary state machine for benchmarking async fn create_test_state_machine() -> (FileStateMachine, TempDir) { let temp_dir = TempDir::new().expect("Failed to create temp dir"); - let mut sm = FileStateMachine::new(temp_dir.path().to_path_buf()) + let sm = FileStateMachine::new(temp_dir.path().to_path_buf()) .await .expect("Failed to create state machine"); - // Enable TTL for benchmarks that need it - let lease_config = d_engine_core::config::LeaseConfig { - cleanup_interval_ms: 1000, - max_cleanup_duration_ms: 1, - }; - let lease = Arc::new(d_engine_server::storage::TtlLease::new(lease_config)); - sm.set_lease(lease); - (sm, temp_dir) } @@ -96,8 +85,8 @@ fn register_watchers( handles } -/// Helper to create write entries without TTL -fn create_entries_without_ttl( +/// Helper to create write entries +fn create_entries( count: usize, start_index: u64, ) -> Vec { @@ -111,86 +100,43 @@ fn create_entries_without_ttl( command: Command::Insert { key: Bytes::from(key), value: Bytes::from(value), - ttl_secs: None, }, } }) .collect() } -/// Helper to create write entries with TTL -fn create_entries_with_ttl( - count: usize, - start_index: u64, - ttl_secs: u64, -) -> Vec { - (0..count) - .map(|i| { - let key = format!("key_ttl_{}", start_index + i as u64); - let value = format!("value_ttl_{}", start_index + i as u64); - ApplyEntry { - index: start_index + i as u64, - term: 1, - command: Command::Insert { - key: Bytes::from(key), - value: Bytes::from(value), - ttl_secs: if ttl_secs > 0 { Some(ttl_secs) } else { None }, - }, - } - }) - .collect() -} - -/// Benchmark: Apply operations WITHOUT TTL -/// Target: < 10ns overhead per operation -fn bench_apply_without_ttl(c: &mut Criterion) { +/// Benchmark: Apply a single entry +fn bench_apply(c: &mut Criterion) { let runtime = tokio::runtime::Builder::new_current_thread().enable_all().build().unwrap(); let (sm, _temp_dir) = runtime.block_on(async { create_test_state_machine().await }); let idx = AtomicU64::new(1); - c.bench_function("apply_without_ttl", |b| { + c.bench_function("apply", |b| { b.to_async(&runtime).iter(|| async { let start = idx.fetch_add(1, Ordering::Relaxed); - let entries = create_entries_without_ttl(1, start); + let entries = create_entries(1, start); sm.apply_chunk(&entries).await.unwrap(); black_box(()); }); }); } -/// Benchmark: Apply operations WITH TTL -/// This measures the overhead of registering TTL entries -fn bench_apply_with_ttl(c: &mut Criterion) { - let runtime = tokio::runtime::Builder::new_current_thread().enable_all().build().unwrap(); - - let (sm, _temp_dir) = runtime.block_on(async { create_test_state_machine().await }); - let idx = AtomicU64::new(1); - - c.bench_function("apply_with_ttl", |b| { - b.to_async(&runtime).iter(|| async { - let start = idx.fetch_add(1, Ordering::Relaxed); - let entries = create_entries_with_ttl(1, start, 3600); // 1 hour TTL - sm.apply_chunk(&entries).await.unwrap(); - black_box(()); - }); - }); -} - -/// Benchmark: Get operation WITHOUT TTL data +/// Benchmark: Get operation /// Baseline for read performance -fn bench_get_without_ttl(c: &mut Criterion) { +fn bench_get(c: &mut Criterion) { let runtime = tokio::runtime::Builder::new_current_thread().enable_all().build().unwrap(); // Setup state machine once before benchmark let (sm, _temp_dir) = runtime.block_on(async { let (sm, temp_dir) = create_test_state_machine().await; - let entries = create_entries_without_ttl(100, 1); + let entries = create_entries(100, 1); sm.apply_chunk(&entries).await.unwrap(); (sm, temp_dir) }); - c.bench_function("get_without_ttl", |b| { + c.bench_function("get", |b| { b.iter(|| { // Measure pure read performance (synchronous get) let key = b"key_50"; @@ -199,54 +145,6 @@ fn bench_get_without_ttl(c: &mut Criterion) { }); } -/// Benchmark: Get operation WITH TTL passive check -/// Target: < 50ns overhead compared to non-TTL reads -fn bench_get_with_ttl_check(c: &mut Criterion) { - let runtime = tokio::runtime::Builder::new_current_thread().enable_all().build().unwrap(); - - // Setup state machine once before benchmark - let (sm, _temp_dir) = runtime.block_on(async { - let (sm, temp_dir) = create_test_state_machine().await; - let entries = create_entries_with_ttl(100, 1, 3600); // Long TTL - sm.apply_chunk(&entries).await.unwrap(); - (sm, temp_dir) - }); - - c.bench_function("get_with_ttl_check", |b| { - b.iter(|| { - // Measure read with TTL check (synchronous get) - let key = b"key_ttl_50"; - black_box(sm.get(key).unwrap()); - }); - }); -} - -/// Benchmark: Get operation with EXPIRED TTL entry -/// This measures the cost of passive deletion -fn bench_get_expired_ttl(c: &mut Criterion) { - let runtime = tokio::runtime::Builder::new_current_thread().enable_all().build().unwrap(); - - // Setup state machine once before benchmark - let (sm, _temp_dir) = runtime.block_on(async { - let (sm, temp_dir) = create_test_state_machine().await; - let entries = create_entries_with_ttl(100, 1, 1); // 1 second TTL - sm.apply_chunk(&entries).await.unwrap(); - - // Wait for expiration - tokio::time::sleep(Duration::from_secs(2)).await; - - (sm, temp_dir) - }); - - c.bench_function("get_expired_ttl", |b| { - b.iter(|| { - // Measure read with expired entry (should trigger passive deletion) - let key = b"key_ttl_50"; - black_box(sm.get(key).unwrap()); - }); - }); -} - /// Benchmark: Batch apply operations (scaling test) /// Verify that performance scales linearly with batch size fn bench_batch_apply(c: &mut Criterion) { @@ -260,29 +158,7 @@ fn bench_batch_apply(c: &mut Criterion) { group.bench_with_input(BenchmarkId::from_parameter(size), size, |b, &size| { b.to_async(&runtime).iter(|| async { let start = idx.fetch_add(size as u64, Ordering::Relaxed); - let entries = create_entries_without_ttl(size, start); - sm.apply_chunk(&entries).await.unwrap(); - black_box(()); - }); - }); - } - - group.finish(); -} - -/// Benchmark: Batch apply with TTL (scaling test) -fn bench_batch_apply_with_ttl(c: &mut Criterion) { - let runtime = tokio::runtime::Builder::new_current_thread().enable_all().build().unwrap(); - let mut group = c.benchmark_group("batch_apply_with_ttl"); - - let (sm, _temp_dir) = runtime.block_on(async { create_test_state_machine().await }); - let idx = AtomicU64::new(1); - - for size in [10, 100, 1000].iter() { - group.bench_with_input(BenchmarkId::from_parameter(size), size, |b, &size| { - b.to_async(&runtime).iter(|| async { - let start = idx.fetch_add(size as u64, Ordering::Relaxed); - let entries = create_entries_with_ttl(size, start, 3600); + let entries = create_entries(size, start); sm.apply_chunk(&entries).await.unwrap(); black_box(()); }); @@ -304,7 +180,7 @@ fn bench_apply_without_watch(c: &mut Criterion) { c.bench_function("apply_without_watch", |b| { b.to_async(&runtime).iter(|| async { let start = idx.fetch_add(100, Ordering::Relaxed); - let entries = create_entries_without_ttl(100, start); + let entries = create_entries(100, start); sm.apply_chunk(&entries).await.unwrap(); black_box(()); }); @@ -327,7 +203,7 @@ fn bench_apply_with_1_watcher(c: &mut Criterion) { let _watchers = register_watchers(®istry, 1, "key_"); let start = idx.fetch_add(100, Ordering::Relaxed); - let entries = create_entries_without_ttl(100, start); + let entries = create_entries(100, start); for entry in &entries { match &entry.command { @@ -418,7 +294,7 @@ fn bench_apply_with_10_watchers(c: &mut Criterion) { let _watchers = register_watchers(®istry, 10, "key_"); let start = idx.fetch_add(100, Ordering::Relaxed); - let entries = create_entries_without_ttl(100, start); + let entries = create_entries(100, start); for entry in &entries { match &entry.command { @@ -509,7 +385,7 @@ fn bench_apply_with_100_watchers(c: &mut Criterion) { let _watchers = register_watchers(®istry, 100, "key_"); let start = idx.fetch_add(100, Ordering::Relaxed); - let entries = create_entries_without_ttl(100, start); + let entries = create_entries(100, start); for entry in &entries { match &entry.command { @@ -624,13 +500,9 @@ fn bench_watch_e2e_latency(c: &mut Criterion) { criterion_group!( benches, - bench_apply_without_ttl, - bench_apply_with_ttl, - bench_get_without_ttl, - bench_get_with_ttl_check, - bench_get_expired_ttl, + bench_apply, + bench_get, bench_batch_apply, - bench_batch_apply_with_ttl, bench_apply_without_watch, bench_apply_with_1_watcher, bench_apply_with_10_watchers, diff --git a/d-engine-server/benches/ttl.rs b/d-engine-server/benches/ttl.rs deleted file mode 100644 index 97d914b0..00000000 --- a/d-engine-server/benches/ttl.rs +++ /dev/null @@ -1,351 +0,0 @@ -//! TTL Manager Performance Benchmarks -//! -//! This benchmark suite focuses specifically on TTL management operations, -//! particularly background cleanup and TTL registration overhead. -//! -//! Performance Targets: -//! - Background cleanup: < 1ms for typical workloads -//! - TTL registration: < 100ns per entry -//! - Expired check: < 50ns per key - -use std::sync::atomic::AtomicU64; -use std::sync::atomic::Ordering; -use std::time::Duration; - -use bytes::Bytes; -use criterion::BenchmarkId; -use criterion::Criterion; -use criterion::black_box; -use criterion::criterion_group; -use criterion::criterion_main; -use d_engine_core::ApplyEntry; -use d_engine_core::Command; -use d_engine_core::Lease; -use d_engine_core::StateMachine; -use d_engine_server::storage::FileStateMachine; -use tempfile::TempDir; - -/// Helper to create a temporary state machine for benchmarking -async fn create_test_state_machine() -> (FileStateMachine, TempDir) { - use d_engine_server::storage::TtlLease; - - let temp_dir = TempDir::new().expect("Failed to create temp dir"); - // For TTL benchmarks, we need lease enabled - let lease_config = d_engine_core::config::LeaseConfig { - cleanup_interval_ms: 1000, - max_cleanup_duration_ms: 1, - }; - - let mut sm = FileStateMachine::new(temp_dir.path().to_path_buf()) - .await - .expect("Failed to create state machine"); - - // Manually inject lease for benchmarking - let lease = std::sync::Arc::new(TtlLease::new(lease_config)); - sm.set_lease(lease); - sm.load_lease_data().await.expect("Failed to load lease data"); - - (sm, temp_dir) -} - -/// Helper to create write entries with TTL -fn create_entries_with_ttl( - count: usize, - start_index: u64, - ttl_secs: u64, -) -> Vec { - (0..count) - .map(|i| { - let key = format!("key_ttl_{}", start_index + i as u64); - let value = format!("value_ttl_{}", start_index + i as u64); - ApplyEntry { - index: start_index + i as u64, - term: 1, - command: Command::Insert { - key: Bytes::from(key), - value: Bytes::from(value), - ttl_secs: if ttl_secs > 0 { Some(ttl_secs) } else { None }, - }, - } - }) - .collect() -} - -/// Helper to create a no-op entry (for triggering piggyback cleanup) -fn create_noop_entry(index: u64) -> ApplyEntry { - ApplyEntry { - index, - term: 1, - command: Command::Noop, - } -} - -/// Benchmark: Piggyback cleanup with varying numbers of expired keys -/// Target: < 1ms for typical workload (100 expired keys) -/// This directly benchmarks lease.on_apply() to isolate cleanup overhead -fn bench_piggyback_cleanup(c: &mut Criterion) { - use d_engine_server::storage::TtlLease; - - let mut group = c.benchmark_group("piggyback_cleanup"); - - // Test with different numbers of expired keys - for expired_count in [10, 50, 100, 500].iter() { - let lease_config = d_engine_core::config::LeaseConfig { - cleanup_interval_ms: 1000, - max_cleanup_duration_ms: 1, - }; - let lease = TtlLease::new(lease_config); - - // Register keys with very short TTL - for i in 0..*expired_count { - let key = format!("key_ttl_{i}"); - lease.register(bytes::Bytes::from(key), 1); // 1 second TTL - } - - // Wait for expiration - std::thread::sleep(Duration::from_secs(2)); - - group.bench_with_input( - BenchmarkId::from_parameter(expired_count), - expired_count, - |b, _| { - b.iter(|| { - // Directly measure cleanup logic without apply_chunk overhead - let expired_keys = lease.on_apply(); - black_box(expired_keys); - }); - }, - ); - } - - group.finish(); -} - -/// Benchmark: TTL registration overhead -/// Measures the cost of registering a TTL entry in the TTL manager -/// Target: < 100ns per registration -fn bench_ttl_registration(c: &mut Criterion) { - use d_engine_server::storage::TtlLease; - - let lease_config = d_engine_core::config::LeaseConfig { - cleanup_interval_ms: 1000, - max_cleanup_duration_ms: 1, - }; - let lease = TtlLease::new(lease_config); - - c.bench_function("ttl_registration", |b| { - let mut counter = 0u64; - b.iter(|| { - // Directly measure registration overhead - let key = format!("key_{counter}"); - lease.register(bytes::Bytes::from(key), 3600); - counter += 1; - black_box(()); - }); - }); -} - -/// Benchmark: Batch TTL registration -/// Verify that TTL registration scales well with batch size -fn bench_batch_ttl_registration(c: &mut Criterion) { - let runtime = tokio::runtime::Builder::new_current_thread().enable_all().build().unwrap(); - let mut group = c.benchmark_group("batch_ttl_registration"); - - let (sm, _temp_dir) = runtime.block_on(async { create_test_state_machine().await }); - - let idx = AtomicU64::new(1); - for size in [10, 100, 1000].iter() { - group.bench_with_input(BenchmarkId::from_parameter(size), size, |b, &size| { - b.to_async(&runtime).iter(|| async { - let start = idx.fetch_add(size as u64, Ordering::Relaxed); - let entries = create_entries_with_ttl(size, start, 3600); - sm.apply_chunk(&entries).await.unwrap(); - black_box(()); - }); - }); - } - - group.finish(); -} - -/// Benchmark: Mixed workload - active and expired TTL entries -/// Simulates realistic scenario with both active and expired entries -fn bench_mixed_ttl_workload(c: &mut Criterion) { - let runtime = tokio::runtime::Builder::new_current_thread().enable_all().build().unwrap(); - - let mut group = c.benchmark_group("mixed_ttl_workload"); - group.sample_size(10); - - // Setup: Create StateMachine with mixed entries (expired + active) - let (sm, _temp_dir) = runtime.block_on(async { - let (sm, temp_dir) = create_test_state_machine().await; - - // Insert 50 entries with short TTL (will expire) - let expired_entries = create_entries_with_ttl(50, 1, 1); - sm.apply_chunk(&expired_entries).await.unwrap(); - - // Insert 50 entries with long TTL (will remain active) - let active_entries = create_entries_with_ttl(50, 51, 3600); - sm.apply_chunk(&active_entries).await.unwrap(); - - // Wait for first batch to expire - tokio::time::sleep(Duration::from_secs(2)).await; - - (sm, temp_dir) - }); - - let noop_idx = AtomicU64::new(101); // setup applied 1..=100 - group.bench_function("cleanup_mixed", |b| { - b.to_async(&runtime).iter(|| async { - let i = noop_idx.fetch_add(1, Ordering::Relaxed); - let noop = create_noop_entry(i); - sm.apply_chunk(&[noop]).await.unwrap(); - black_box(()); - }); - }); - - group.finish(); -} - -/// Benchmark: Piggyback cleanup with expired entries -/// Measures the actual cleanup overhead when expired entries exist -fn bench_piggyback_high_frequency(c: &mut Criterion) { - let runtime = tokio::runtime::Builder::new_current_thread().enable_all().build().unwrap(); - - let mut group = c.benchmark_group("piggyback_high_frequency"); - group.sample_size(10); - - let (_temp_dir, sm) = runtime.block_on(async { - let (sm, temp_dir) = create_test_state_machine().await; - (temp_dir, sm) - }); - - let idx = AtomicU64::new(1); - group.bench_function("cleanup_with_expired_entries", |b| { - b.to_async(&runtime).iter(|| async { - let start = idx.fetch_add(101, Ordering::Relaxed); - let entries = create_entries_with_ttl(100, start, 1); - sm.apply_chunk(&entries).await.unwrap(); - tokio::time::sleep(Duration::from_secs(2)).await; - - let noop = create_noop_entry(start + 100); - sm.apply_chunk(&[noop]).await.unwrap(); - black_box(()); - }); - }); - - group.finish(); -} - -/// Benchmark: TTL with varying expiration times -/// Tests if different TTL durations affect performance -fn bench_varying_ttl_durations(c: &mut Criterion) { - let runtime = tokio::runtime::Builder::new_current_thread().enable_all().build().unwrap(); - let mut group = c.benchmark_group("varying_ttl_durations"); - - let (sm, _temp_dir) = runtime.block_on(async { create_test_state_machine().await }); - - // Test with different TTL durations - let idx = AtomicU64::new(1); - for ttl_secs in [60, 3600, 86400].iter() { - // 1 min, 1 hour, 1 day - group.bench_with_input( - BenchmarkId::from_parameter(ttl_secs), - ttl_secs, - |b, &ttl_secs| { - b.to_async(&runtime).iter(|| async { - let start = idx.fetch_add(100, Ordering::Relaxed); - let entries = create_entries_with_ttl(100, start, ttl_secs); - sm.apply_chunk(&entries).await.unwrap(); - black_box(()); - }); - }, - ); - } - - group.finish(); -} - -/// Benchmark: Worst case - all entries expired -/// Tests cleanup performance when all entries need to be removed -fn bench_worst_case_all_expired(c: &mut Criterion) { - let runtime = tokio::runtime::Builder::new_current_thread().enable_all().build().unwrap(); - - let mut group = c.benchmark_group("worst_case_all_expired"); - group.sample_size(10); - - // Setup: Create state machine with all expired entries - let (sm, _temp_dir) = runtime.block_on(async { - let (sm, temp_dir) = create_test_state_machine().await; - - // Insert 1000 entries with very short TTL - let entries = create_entries_with_ttl(1000, 1, 1); - sm.apply_chunk(&entries).await.unwrap(); - - // Wait for all to expire - tokio::time::sleep(Duration::from_secs(2)).await; - - (sm, temp_dir) - }); - - let idx = AtomicU64::new(1001); // setup applied 1..=1000 - group.bench_function("cleanup_all_expired", |b| { - b.to_async(&runtime).iter(|| async { - let start = idx.fetch_add(1001, Ordering::Relaxed); - let entries = create_entries_with_ttl(1000, start, 1); - sm.apply_chunk(&entries).await.unwrap(); - tokio::time::sleep(Duration::from_secs(2)).await; - - let noop = create_noop_entry(start + 1000); - sm.apply_chunk(&[noop]).await.unwrap(); - black_box(()); - }); - }); - - group.finish(); -} - -/// Benchmark: Best case - no expired entries -/// Tests that cleanup is fast when nothing needs to be cleaned -fn bench_best_case_no_expired(c: &mut Criterion) { - let runtime = tokio::runtime::Builder::new_current_thread().enable_all().build().unwrap(); - - let mut group = c.benchmark_group("best_case_no_expired"); - group.sample_size(10); - - // Setup: Create state machine with non-expired entries - let (sm, _temp_dir) = runtime.block_on(async { - let (sm, temp_dir) = create_test_state_machine().await; - - // Insert 1000 entries with very long TTL - let entries = create_entries_with_ttl(1000, 1, 86400); // 1 day - sm.apply_chunk(&entries).await.unwrap(); - - (sm, temp_dir) - }); - - let noop_idx = AtomicU64::new(1001); // setup applied 1..=1000 - group.bench_function("cleanup_no_expired", |b| { - b.to_async(&runtime).iter(|| async { - let i = noop_idx.fetch_add(1, Ordering::Relaxed); - let noop = create_noop_entry(i); - sm.apply_chunk(&[noop]).await.unwrap(); - black_box(()); - }); - }); - - group.finish(); -} - -criterion_group!( - benches, - bench_piggyback_cleanup, - bench_ttl_registration, - bench_batch_ttl_registration, - bench_mixed_ttl_workload, - bench_piggyback_high_frequency, - bench_varying_ttl_durations, - bench_worst_case_all_expired, - bench_best_case_no_expired, -); -criterion_main!(benches); diff --git a/d-engine-server/benches/watch_overhead.rs b/d-engine-server/benches/watch_overhead.rs index 7604bd74..8ff45ef2 100644 --- a/d-engine-server/benches/watch_overhead.rs +++ b/d-engine-server/benches/watch_overhead.rs @@ -65,7 +65,6 @@ fn create_test_entries( command: Command::Insert { key: Bytes::from(key.into_bytes()), value: Bytes::from(value.into_bytes()), - ttl_secs: None, }, }); } diff --git a/d-engine-server/src/api/embedded.rs b/d-engine-server/src/api/embedded.rs index 225d6f05..03c37a8e 100644 --- a/d-engine-server/src/api/embedded.rs +++ b/d-engine-server/src/api/embedded.rs @@ -204,7 +204,7 @@ impl EmbeddedEngine> { // one even checks this lock. let data_dir_lock = crate::node::DataDirLock::acquire(data_dir.as_path())?; - let (storage, mut sm) = if config.storage.unified_db { + let (storage, sm) = if config.storage.unified_db { let db_path = data_dir.unified_db_path()?; info!( "Starting embedded engine with unified RocksDB at {:?}", @@ -221,11 +221,6 @@ impl EmbeddedEngine> { (storage, sm) }; - let lease = Arc::new(crate::storage::TtlLease::new( - config.raft.state_machine.lease.clone(), - )); - sm.set_lease(lease); - Self::start_node_with_lock( data_dir.as_path().to_path_buf(), data_dir_lock, @@ -259,7 +254,7 @@ impl EmbeddedEngine> { let data_dir = crate::node::DataDir::new(data_dir.as_ref())?; let data_dir_lock = crate::node::DataDirLock::acquire(data_dir.as_path())?; - let (storage, mut sm) = if config.storage.unified_db { + let (storage, sm) = if config.storage.unified_db { let db_path = data_dir.unified_db_path()?; info!( "Starting embedded engine with unified RocksDB at {:?}", @@ -276,11 +271,6 @@ impl EmbeddedEngine> { (storage, sm) }; - let lease = Arc::new(crate::storage::TtlLease::new( - config.raft.state_machine.lease.clone(), - )); - sm.set_lease(lease); - // start_node_with_lock directly, not start_custom — config is // already loaded/validated here, and the lock is already held; // start_custom would re-parse config_path again for no benefit and diff --git a/d-engine-server/src/api/embedded_client.rs b/d-engine-server/src/api/embedded_client.rs index 1ea927f0..5164cea6 100644 --- a/d-engine-server/src/api/embedded_client.rs +++ b/d-engine-server/src/api/embedded_client.rs @@ -116,7 +116,6 @@ impl EmbeddedClient { command: Some(proto_convert::write_op_to_bytes(WriteOperation::Insert { key: Bytes::copy_from_slice(key.as_ref()), value: Bytes::copy_from_slice(value.as_ref()), - ttl_secs: None, })), }; @@ -585,48 +584,6 @@ impl ClientApi for EmbeddedClient { self.put(key, value).await } - async fn put_with_ttl( - &self, - key: impl AsRef<[u8]> + Send, - value: impl AsRef<[u8]> + Send, - ttl_secs: u64, - ) -> ClientApiResult<()> { - let request = ClientWriteRequest { - client_id: self.client_id, - command: Some(proto_convert::write_op_to_bytes(WriteOperation::Insert { - key: Bytes::copy_from_slice(key.as_ref()), - value: Bytes::copy_from_slice(value.as_ref()), - ttl_secs: Some(ttl_secs), - })), - }; - - let (resp_tx, resp_rx) = MaybeCloneOneshot::new(); - - self.read_handle - .cmd_tx - .send(d_engine_core::ClientCmd::propose(request, resp_tx)) - .await - .map_err(|_| channel_closed_error())?; - - let result = tokio::time::timeout(self.timeout, resp_rx) - .await - .map_err(|_| timeout_error(self.timeout))? - .map_err(|_| channel_closed_error())?; - - let response = - result.map_err(|status| server_error(format!("RPC error: {}", status.message())))?; - - if response.error != ErrorCode::Success { - return Err(map_error_response( - response.error, - response.leader_hint, - response.retry_after_ms, - )); - } - - Ok(()) - } - /// Atomically commits multiple write operations as a single Raft log entry. /// /// All operations succeed or none apply (all-or-nothing). diff --git a/d-engine-server/src/api/standalone.rs b/d-engine-server/src/api/standalone.rs index 0e15e6e3..7ad01719 100644 --- a/d-engine-server/src/api/standalone.rs +++ b/d-engine-server/src/api/standalone.rs @@ -41,7 +41,7 @@ impl StandaloneEngine { // one even checks this lock. let data_dir_lock = crate::node::DataDirLock::acquire(data_dir.as_path())?; - let (storage, mut sm) = if config.storage.unified_db { + let (storage, sm) = if config.storage.unified_db { let db_path = data_dir.unified_db_path()?; tracing::info!( "Starting standalone server with unified RocksDB at {:?}", @@ -58,11 +58,6 @@ impl StandaloneEngine { (storage, sm) }; - let lease = Arc::new(crate::storage::TtlLease::new( - config.raft.state_machine.lease.clone(), - )); - sm.set_lease(lease); - Self::start_node_with_lock( data_dir.as_path().to_path_buf(), data_dir_lock, @@ -95,7 +90,7 @@ impl StandaloneEngine { let data_dir = crate::node::DataDir::new(data_dir.as_ref())?; let data_dir_lock = crate::node::DataDirLock::acquire(data_dir.as_path())?; - let (storage, mut sm) = if config.storage.unified_db { + let (storage, sm) = if config.storage.unified_db { let db_path = data_dir.unified_db_path()?; tracing::info!( "Starting standalone server with unified RocksDB at {:?}", @@ -112,11 +107,6 @@ impl StandaloneEngine { (storage, sm) }; - let lease = Arc::new(crate::storage::TtlLease::new( - config.raft.state_machine.lease.clone(), - )); - sm.set_lease(lease); - // start_node_with_lock directly — config already loaded/validated, // lock already held. Self::start_node_with_lock( diff --git a/d-engine-server/src/api/standalone_test.rs b/d-engine-server/src/api/standalone_test.rs index d7517954..4fdb9df3 100644 --- a/d-engine-server/src/api/standalone_test.rs +++ b/d-engine-server/src/api/standalone_test.rs @@ -257,7 +257,6 @@ mod start_node_tests { use crate::RocksDBStateMachine; use crate::RocksDBStorageEngine; use crate::api::StandaloneEngine; - use crate::storage::TtlLease; /// Build a minimal valid `RaftNodeConfig`. fn make_config(node_id: u32) -> d_engine_core::RaftNodeConfig { @@ -277,7 +276,7 @@ mod start_node_tests { } } - /// Create RocksDB storage engine + state machine with lease initialized. + /// Create a RocksDB storage engine and state machine. fn create_rocksdb_se_and_sm( temp_dir: &tempfile::TempDir ) -> (Arc, Arc) { @@ -288,8 +287,7 @@ mod start_node_tests { let storage = Arc::new(RocksDBStorageEngine::new(&storage_path).expect("create storage engine")); - let mut sm = RocksDBStateMachine::new(&sm_path).expect("create state machine"); - sm.set_lease(Arc::new(TtlLease::new(Default::default()))); + let sm = RocksDBStateMachine::new(&sm_path).expect("create state machine"); let sm = Arc::new(sm); (storage, sm) @@ -379,8 +377,7 @@ mod start_node_tests { std::fs::create_dir_all(&sm_path).unwrap(); let storage = Arc::new(RocksDBStorageEngine::new(&storage_path).expect("create storage")); - let mut sm = RocksDBStateMachine::new(&sm_path).expect("create sm"); - sm.set_lease(Arc::new(TtlLease::new(Default::default()))); + let sm = RocksDBStateMachine::new(&sm_path).expect("create sm"); let sm = Arc::new(sm); let config = make_config(1); @@ -422,9 +419,8 @@ mod run_custom_tests { use crate::RocksDBStateMachine; use crate::RocksDBStorageEngine; use crate::api::StandaloneEngine; - use crate::storage::TtlLease; - /// Create RocksDB SE/SM with lease initialized. + /// Create a RocksDB storage engine and state machine. fn create_rocksdb_se_and_sm( temp_dir: &tempfile::TempDir ) -> (Arc, Arc) { @@ -435,8 +431,7 @@ mod run_custom_tests { let storage = Arc::new(RocksDBStorageEngine::new(&storage_path).expect("create storage engine")); - let mut sm = RocksDBStateMachine::new(&sm_path).expect("create state machine"); - sm.set_lease(Arc::new(TtlLease::new(Default::default()))); + let sm = RocksDBStateMachine::new(&sm_path).expect("create state machine"); let sm = Arc::new(sm); (storage, sm) diff --git a/d-engine-server/src/api/types.rs b/d-engine-server/src/api/types.rs index 5f867624..e6b2c7d8 100644 --- a/d-engine-server/src/api/types.rs +++ b/d-engine-server/src/api/types.rs @@ -36,8 +36,6 @@ pub enum WriteOperation { Insert { key: Bytes, value: Bytes, - /// `None` = no expiration. Proto encodes this as `ttl_secs = 0`. - ttl_secs: Option, }, Delete { key: Bytes, @@ -66,16 +64,8 @@ pub enum WriteOperation { impl From for WriteCommand { fn from(op: WriteOperation) -> Self { match op { - WriteOperation::Insert { - key, - value, - ttl_secs, - } => WriteCommand { - operation: Some(Operation::Insert(Insert { - key, - value, - ttl_secs: ttl_secs.unwrap_or(0), - })), + WriteOperation::Insert { key, value } => WriteCommand { + operation: Some(Operation::Insert(Insert { key, value })), }, WriteOperation::Delete { key } => WriteCommand { operation: Some(Operation::Delete(Delete { key })), @@ -96,11 +86,7 @@ impl From for WriteCommand { .into_iter() .map(|op| match op { BatchOp::Insert { key, value } => ProtoBatchOp { - op: Some(batch_op::Op::Insert(Insert { - key, - value, - ttl_secs: 0, - })), + op: Some(batch_op::Op::Insert(Insert { key, value })), }, BatchOp::Delete { key } => ProtoBatchOp { op: Some(batch_op::Op::Delete(Delete { key })), diff --git a/d-engine-server/src/api/types_test.rs b/d-engine-server/src/api/types_test.rs index d9fa8f86..f9e3b641 100644 --- a/d-engine-server/src/api/types_test.rs +++ b/d-engine-server/src/api/types_test.rs @@ -5,40 +5,23 @@ use d_engine_proto::client::write_command::Operation; use super::WriteOperation; -/// Verify Insert with TTL converts correctly to proto WriteCommand. +/// Verify Insert converts correctly to proto WriteCommand. #[test] -fn test_write_operation_insert_with_ttl() { +fn test_write_operation_insert() { let op = WriteOperation::Insert { key: Bytes::from("k"), value: Bytes::from("v"), - ttl_secs: Some(60), }; let cmd = WriteCommand::from(op); match cmd.operation { Some(Operation::Insert(ins)) => { assert_eq!(ins.key, Bytes::from("k")); assert_eq!(ins.value, Bytes::from("v")); - assert_eq!(ins.ttl_secs, 60); } other => panic!("expected Insert, got {:?}", other), } } -/// Verify Insert with no TTL maps ttl_secs=None → proto ttl_secs=0. -#[test] -fn test_write_operation_insert_no_ttl_maps_to_zero() { - let op = WriteOperation::Insert { - key: Bytes::from("k"), - value: Bytes::from("v"), - ttl_secs: None, - }; - let cmd = WriteCommand::from(op); - match cmd.operation { - Some(Operation::Insert(ins)) => assert_eq!(ins.ttl_secs, 0), - other => panic!("expected Insert, got {:?}", other), - } -} - /// Verify Delete converts correctly to proto WriteCommand. #[test] fn test_write_operation_delete() { @@ -110,7 +93,6 @@ fn test_write_operation_batch_mixed_ops() { Some(batch_op::Op::Insert(ins)) => { assert_eq!(ins.key, Bytes::from("k1")); assert_eq!(ins.value, Bytes::from("v1")); - assert_eq!(ins.ttl_secs, 0); } other => panic!("expected Insert, got {:?}", other), } diff --git a/d-engine-server/src/node/builder.rs b/d-engine-server/src/node/builder.rs index 3308f6ff..208e0df3 100644 --- a/d-engine-server/src/node/builder.rs +++ b/d-engine-server/src/node/builder.rs @@ -309,24 +309,9 @@ where ) })?; - // Note: Lease configuration should be injected BEFORE wrapping in Arc. - // See StandaloneServer::start() or EmbeddedEngine::with_rocksdb() for correct pattern. - // If state_machine is passed from user code, they are responsible for lease injection. - // Start state machine: flip flags and load persisted data state_machine.start().await?; - // Spawn lease background cleanup task (TTL is always active) - info!( - "Starting lease background cleanup worker (interval: {}ms)", - node_config.raft.state_machine.lease.cleanup_interval_ms - ); - let lease_cleanup_handle = Some(Self::spawn_background_cleanup_worker( - Arc::clone(&state_machine), - node_config.raft.state_machine.lease.cleanup_interval_ms, - self.shutdown_signal.clone(), - )); - // Handle storage engine initialization let storage_engine = self.storage_engine.take().ok_or_else(|| { SystemError::NodeStartFailed( @@ -648,7 +633,6 @@ where sm_worker, read_actor_handle: std::sync::Mutex::new(Some(read_actor_handle)), rpc_server_handle: std::sync::Mutex::new(None), - _lease_cleanup_handle: lease_cleanup_handle, shutdown_signal: self.shutdown_signal.clone(), read_lease, _data_dir: data_dir, @@ -659,61 +643,6 @@ where Ok(self) } - /// Spawn lease background cleanup task (if enabled). - /// - /// This task runs independently from Raft apply pipeline, avoiding any blocking. - /// Only spawns when cleanup_strategy = "background" (not "disabled" or "piggyback"). - /// - /// # Arguments - /// * `state_machine` - Arc reference to state machine for accessing lease manager - /// * `lease_config` - Lease configuration determining cleanup behavior - /// * `shutdown_signal` - Watch channel for graceful shutdown notification - /// - /// # Returns - /// * `Option` - Task handle if background cleanup is enabled, None otherwise - /// - /// # Design Principles - /// - **Zero overhead**: If disabled, returns None immediately (no task spawned) - /// - **One page visible**: All long-running tasks spawned here in builder.rs - /// - **Graceful shutdown**: Monitors shutdown signal for clean termination - fn spawn_background_cleanup_worker( - state_machine: Arc, - cleanup_interval_ms: u64, - mut shutdown_signal: watch::Receiver<()>, - ) -> tokio::task::JoinHandle<()> { - tokio::spawn(async move { - let mut interval = - tokio::time::interval(std::time::Duration::from_millis(cleanup_interval_ms)); - - loop { - tokio::select! { - _ = interval.tick() => { - // Call state machine's lease background cleanup - match state_machine.lease_background_cleanup().await { - Ok(deleted_keys) => { - if !deleted_keys.is_empty() { - debug!( - "Lease background cleanup: deleted {} expired keys", - deleted_keys.len() - ); - } - } - Err(e) => { - error!("Lease background cleanup failed: {:?}", e); - } - } - } - _ = shutdown_signal.changed() => { - info!("Lease background cleanup received shutdown signal"); - break; - } - } - } - - debug!("Lease background cleanup worker stopped"); - }) - } - /// Starts the gRPC server for cluster communication. /// /// # Panics @@ -743,9 +672,9 @@ where /// Builds and starts the Raft node. /// /// This is the primary method to initialize and start a node. It performs: - /// 1. State machine initialization (including lease injection if applicable) + /// 1. State machine initialization /// 2. Raft core construction - /// 3. Background task spawning (state machine worker, read actor, lease cleanup) + /// 3. Background task spawning (state machine worker, read actor) /// 4. gRPC server startup for cluster communication /// /// # Returns diff --git a/d-engine-server/src/node/mod.rs b/d-engine-server/src/node/mod.rs index 9cc0bae0..d8dd0d46 100644 --- a/d-engine-server/src/node/mod.rs +++ b/d-engine-server/src/node/mod.rs @@ -117,9 +117,6 @@ where /// long-lived streams (the Raft replication bidi stream never ends on its own) pub(crate) rpc_server_handle: std::sync::Mutex>>, - /// Lease cleanup task handle (background TTL cleanup) - pub(crate) _lease_cleanup_handle: Option>, - /// Shutdown signal for graceful termination pub(crate) shutdown_signal: watch::Receiver<()>, diff --git a/d-engine-server/src/proto_convert_test.rs b/d-engine-server/src/proto_convert_test.rs index 132c6e25..5aae97e0 100644 --- a/d-engine-server/src/proto_convert_test.rs +++ b/d-engine-server/src/proto_convert_test.rs @@ -33,7 +33,6 @@ fn test_to_core_write_req_insert_roundtrip() { operation: Some(Operation::Insert(Insert { key: Bytes::from("key"), value: Bytes::from("val"), - ttl_secs: 0, })), }), }; @@ -42,10 +41,7 @@ fn test_to_core_write_req_insert_roundtrip() { let bytes = core_req.command.expect("command must be Some"); let decoded = WriteCommand::decode(bytes).expect("must decode to WriteCommand"); assert!( - matches!( - decoded.operation, - Some(Operation::Insert(Insert { ttl_secs: 0, .. })) - ), + matches!(decoded.operation, Some(Operation::Insert(Insert { .. }))), "decoded command must be an Insert operation" ); } diff --git a/d-engine-server/src/storage/adaptors/file/file_state_machine.rs b/d-engine-server/src/storage/adaptors/file/file_state_machine.rs index 4b6bc761..873bbf80 100644 --- a/d-engine-server/src/storage/adaptors/file/file_state_machine.rs +++ b/d-engine-server/src/storage/adaptors/file/file_state_machine.rs @@ -9,7 +9,6 @@ //! //! - **`state.data`**: Serialized key-value store (persisted after each apply_chunk) //! - **`wal.log`**: Write-Ahead Log for crash recovery (cleared after successful persistence) -//! - **`ttl_state.bin`**: TTL manager state (persisted alongside state.data) //! - **`metadata.bin`**: Raft metadata (last_applied_index, last_applied_term) //! //! ## Write-Ahead Log (WAL) Design @@ -24,33 +23,11 @@ //! │ Key Length (8 bytes) │ Key Data (N bytes) │ //! ├─────────────────────────────────────────────────────────────────┤ //! │ Value Length (8 bytes)│ Value Data (M bytes, if present) │ -//! ├─────────────────────────────────────────────────────────────────┤ -//! │ Expire At (8 bytes, 0 = no TTL, >0 = UNIX timestamp in seconds) │ //! └─────────────────────────────────────────────────────────────────┘ //! ``` //! -//! ### TTL Semantics -//! -//! d-engine uses **absolute expiration time**: -//! -//! - When a key is created with TTL, the system calculates: `expire_at = now() + ttl_secs` -//! - WAL stores the **absolute expiration timestamp** (UNIX seconds since epoch) -//! - After crash recovery, expired keys are **not restored** (checked during replay) -//! - TTL does **not reset** on restart (crash-safe) -//! -//! **Example:** -//! ```text -//! T0: PUT key="foo", ttl=10s → expire_at = T0 + 10 = T10 (stored in WAL) -//! T5: CRASH -//! T12: RESTART -//! → Replay WAL: expire_at = T10 < T12 (already expired) -//! → Key is NOT restored (correctly expired) -//! ``` -//! -//! **Why absolute time in WAL:** -//! 1. Ensures expired keys stay expired after crash (durable expiration semantics) -//! 2. Passive expiration (in get()) is crash-safe without WAL writes -//! 3. No TTL reset on recovery (deterministic expiration) +//! A record cut short by a crash is dropped on replay: replay stops at the first +//! incomplete record. //! //! ### WAL Lifecycle //! @@ -71,26 +48,18 @@ //! On node startup (`new()`): //! 1. `load_metadata()` - Restore Raft state //! 2. `load_data()` - Load persisted key-value data -//! 3. `load_ttl_data()` - Load persisted TTL state -//! 4. `replay_wal()` - **Critical**: Replay uncommitted operations from WAL -//! - Restores keys AND their TTL metadata +//! 3. `replay_wal()` - **Critical**: Replay operations written after the last checkpoint +//! - Entries at or below `last_applied` are skipped //! - Ensures crash consistency (operations are idempotent) -//! -//! ## TTL Cleanup Strategy -//! -//! - **Background Cleanup**: Periodic async worker scans and deletes expired keys -//! - **Zero Overhead**: When TTL feature is disabled, no lease components are initialized use std::collections::HashMap; use std::collections::HashSet; use std::io::Write; use std::path::PathBuf; -use std::sync::Arc; use std::sync::Mutex; use std::sync::atomic::AtomicBool; use std::sync::atomic::AtomicU64; use std::sync::atomic::Ordering; -use std::time::SystemTime; use async_trait::async_trait; use bytes::Bytes; @@ -98,7 +67,6 @@ use d_engine_core::ApplyEntry; use d_engine_core::ApplyResult; use d_engine_core::Command; use d_engine_core::Error; -use d_engine_core::Lease; use d_engine_core::ScanResult; use d_engine_core::SnapshotApplyResult; use d_engine_core::SnapshotError; @@ -119,8 +87,6 @@ use tracing::error; use tracing::info; use tracing::warn; -use crate::storage::TtlLease; - type FileStateMachineDataType = RwLock>; /// WAL operation codes for fixed-size encoding @@ -179,39 +145,19 @@ fn encode_wal_entry( buf.push(WalOpCode::Noop as u8); buf.extend_from_slice(&0u64.to_be_bytes()); buf.extend_from_slice(&0u64.to_be_bytes()); - buf.extend_from_slice(&0u64.to_be_bytes()); } - Command::Insert { - key, - value, - ttl_secs, - } => { + Command::Insert { key, value } => { buf.push(WalOpCode::Insert as u8); buf.extend_from_slice(&(key.len() as u64).to_be_bytes()); buf.extend_from_slice(key); buf.extend_from_slice(&(value.len() as u64).to_be_bytes()); buf.extend_from_slice(value); - // expire_at is computed at WAL-encode time, not at apply time. Any delay - // between Raft commit and WAL write (e.g. slow I/O) causes a small under-count - // (key expires slightly earlier than intended). This is a known, accepted - // trade-off: TTL precision is best-effort, not a hard guarantee. - let expire_at_secs = if let Some(ttl) = ttl_secs { - let expire_at = std::time::SystemTime::now() + std::time::Duration::from_secs(*ttl); - expire_at - .duration_since(std::time::UNIX_EPOCH) - .map(|d| d.as_secs()) - .unwrap_or(0) - } else { - 0 - }; - buf.extend_from_slice(&expire_at_secs.to_be_bytes()); } Command::Delete { key } => { buf.push(WalOpCode::Delete as u8); buf.extend_from_slice(&(key.len() as u64).to_be_bytes()); buf.extend_from_slice(key); buf.extend_from_slice(&0u64.to_be_bytes()); // val_len = 0 - buf.extend_from_slice(&0u64.to_be_bytes()); // expire_at = 0 } Command::CompareAndSwap { key, value, .. } => { if cas_success { @@ -220,12 +166,10 @@ fn encode_wal_entry( buf.extend_from_slice(key); buf.extend_from_slice(&(value.len() as u64).to_be_bytes()); buf.extend_from_slice(value); - buf.extend_from_slice(&0u64.to_be_bytes()); // no TTL } else { buf.push(WalOpCode::CasFailed as u8); buf.extend_from_slice(&0u64.to_be_bytes()); // key_len = 0 buf.extend_from_slice(&0u64.to_be_bytes()); // val_len = 0 - buf.extend_from_slice(&0u64.to_be_bytes()); // expire_at = 0 } } Command::Batch { ops } => { @@ -243,14 +187,12 @@ fn encode_wal_entry( buf.extend_from_slice(key); buf.extend_from_slice(&(value.len() as u64).to_be_bytes()); buf.extend_from_slice(value); - buf.extend_from_slice(&0u64.to_be_bytes()); // no TTL in batch inserts } d_engine_core::BatchOp::Delete { key } => { buf.push(WalOpCode::Delete as u8); buf.extend_from_slice(&(key.len() as u64).to_be_bytes()); buf.extend_from_slice(key); buf.extend_from_slice(&0u64.to_be_bytes()); // val_len = 0 - buf.extend_from_slice(&0u64.to_be_bytes()); // expire_at = 0 } } } @@ -266,17 +208,11 @@ fn encode_wal_entry( /// - Write-ahead logging for crash consistency /// - Efficient snapshot handling with file-based storage /// - Thread-safe with minimal lock contention -/// - TTL support for automatic key expiration #[derive(Debug)] pub struct FileStateMachine { // Key-value storage with disk persistence data: FileStateMachineDataType, // (value, term) - // Lease management for automatic key expiration - // TtlLease is thread-safe internally (uses DashMap + Mutex) - // Injected by NodeBuilder after construction - lease: Option>, - // Raft state with disk persistence last_applied_index: AtomicU64, last_applied_term: AtomicU64, @@ -300,8 +236,6 @@ pub struct FileStateMachine { impl FileStateMachine { /// Creates a new file-based state machine with persistence /// - /// Lease will be injected by NodeBuilder after construction. - /// /// # Arguments /// * `data_dir` - Directory where data files will be stored /// @@ -313,7 +247,6 @@ impl FileStateMachine { let machine = Self { data: RwLock::new(HashMap::new()), - lease: None, last_applied_index: AtomicU64::new(0), last_applied_term: AtomicU64::new(0), last_snapshot_metadata: RwLock::new(None), @@ -329,21 +262,6 @@ impl FileStateMachine { Ok(machine) } - /// Sets the lease manager for this state machine. - /// - /// This is an internal method called by NodeBuilder during initialization. - /// The lease will also be restored from snapshot during `apply_snapshot_from_file()`. - /// Also available for testing and benchmarks. - pub fn set_lease( - &mut self, - lease: Arc, - ) { - self.lease = Some(lease); - } - - /// Injects lease configuration into this state machine. - /// - /// Framework-internal method: called by NodeBuilder::build() during initialization. /// Loads state machine data from disk files async fn load_from_disk(&self) -> Result<(), Error> { // Load last applied index and term from metadata file @@ -352,9 +270,6 @@ impl FileStateMachine { // Load key-value data from data file self.load_data().await?; - // Load TTL data from disk - self.load_ttl_data().await?; - // Replay write-ahead log for crash recovery self.replay_wal().await?; @@ -362,36 +277,6 @@ impl FileStateMachine { Ok(()) } - /// Loads TTL data from disk (if lease is configured) - async fn load_ttl_data(&self) -> Result<(), Error> { - // Lease will be injected by NodeBuilder later - // The lease data will be loaded after injection - // For now, just skip this step during construction - Ok(()) - } - - /// Loads TTL data into the configured lease - /// - /// Called after NodeBuilder injects the lease. - /// Also available for testing and benchmarks. - pub async fn load_lease_data(&self) -> Result<(), Error> { - let Some(ref lease) = self.lease else { - return Ok(()); // No lease configured - }; - - let ttl_path = self.data_dir.join("ttl_state.bin"); - if !ttl_path.exists() { - debug!("No TTL state file found"); - return Ok(()); - } - - let ttl_data = tokio::fs::read(&ttl_path).await?; - lease.reload(&ttl_data)?; - - info!("Loaded TTL state from disk: {} active TTLs", lease.len()); - Ok(()) - } - /// Loads metadata from disk async fn load_metadata(&self) -> Result<(), Error> { let metadata_path = self.data_dir.join("metadata.bin"); @@ -538,6 +423,8 @@ impl FileStateMachine { let last_applied = self.last_applied_index.load(Ordering::SeqCst); + // A crash can leave the last WAL record only partly written. The length checks below + // stop replay at the first incomplete record and drop it; records before it are replayed. while pos + 17 < buffer.len() { // Read entry index (8 bytes) let index = u64::from_be_bytes(buffer[pos..pos + 8].try_into().unwrap()); @@ -592,45 +479,14 @@ impl FileStateMachine { let value_len = u64::from_be_bytes(buffer[pos..pos + 8].try_into().unwrap()) as usize; pos += 8; - // Read value if present - let value = if value_len > 0 { - if pos + value_len > buffer.len() { - warn!("Incomplete value data at position {}, stopping replay", pos); - break; - } - let value_data = Bytes::from(buffer[pos..pos + value_len].to_vec()); - pos += value_len; - Some(value_data) - } else { - None - }; - - // Read absolute expiration time (8 bytes) - 0 means no TTL - // - // WAL Format Migration Path: - // - Old format (pre-v0.2.0): ttl_secs (u32, 4 bytes, relative time) - // - New format (v0.2.0+): expire_at_secs (u64, 8 bytes, absolute time) - // - // Backward Compatibility Strategy: - // Since this is a breaking change and d-engine has not been deployed to production, - // we do NOT support reading old WAL format. All WAL entries must use the new format. - // If upgrading from pre-v0.2.0, users must: - // 1. Gracefully stop the old version (persists state.data + ttl_state.bin) - // 2. Upgrade to v0.2.0+ - // 3. Start the new version (loads from persisted state, not WAL) - // Truncated tail: stop here rather than continuing with garbage bytes. - // A partial write at the WAL tail is expected on a crash; entries after - // the truncation point are discarded and will be re-applied via Raft. - if pos + 8 > buffer.len() { - warn!( - "Incomplete WAL entry at byte offset {} (expected expire_at field), truncating replay here", - pos - ); + // A zero length is an empty value, not a missing one. Delete and Noop records + // also write 0 here; replay ignores their value. + if pos + value_len > buffer.len() { + warn!("Incomplete value data at position {}, stopping replay", pos); break; } - let secs = u64::from_be_bytes(buffer[pos..pos + 8].try_into().unwrap()); - pos += 8; - let expire_at_secs = if secs > 0 { Some(secs) } else { None }; + let value = Bytes::from(buffer[pos..pos + value_len].to_vec()); + pos += value_len; if index <= last_applied { debug!( @@ -640,7 +496,7 @@ impl FileStateMachine { continue; } - operations.push((op_code, key, value, term, expire_at_secs)); + operations.push((op_code, key, value, term)); replayed_count += 1; } @@ -651,65 +507,17 @@ impl FileStateMachine { // Apply all collected operations with a single lock acquisition let mut applied_count = 0; - let mut skipped_expired = 0; - let now = std::time::SystemTime::now(); { let mut data = self.data.write(); - for (op_code, key, value, term, expire_at_secs) in operations { + for (op_code, key, value, term) in operations { match op_code { WalOpCode::Insert => { - if let Some(value_data) = value { - // Check if key is already expired (crash-safe TTL semantics) - let is_expired = if let Some(secs) = expire_at_secs { - let expire_at = - std::time::UNIX_EPOCH + std::time::Duration::from_secs(secs); - now >= expire_at - } else { - false - }; - - if is_expired { - // Skip restoring expired keys (durable expiration semantics) - debug!("Skipped expired key during WAL replay: key={:?}", key); - skipped_expired += 1; - continue; - } - - data.insert(key.clone(), (value_data, term)); - - // Restore TTL from WAL (if lease configured and has expiration) - if let Some(secs) = expire_at_secs { - if let Some(ref lease) = self.lease { - let expire_at = std::time::UNIX_EPOCH - + std::time::Duration::from_secs(secs); - let remaining = expire_at - .duration_since(now) - .map(|d| d.as_secs()) - .unwrap_or(0); - - if remaining > 0 { - lease.register(key.clone(), remaining); - debug!( - "Replayed INSERT with TTL: key={:?}, remaining={}s", - key, remaining - ); - } - } - } else { - debug!("Replayed INSERT: key={:?}", key); - } - - applied_count += 1; - } else { - warn!("INSERT operation without value"); - } + data.insert(key, (value, term)); + applied_count += 1; } WalOpCode::Delete => { data.remove(&key); - if let Some(ref lease) = self.lease { - lease.unregister(&key); - } applied_count += 1; debug!("Replayed DELETE: key={:?}", key); } @@ -718,13 +526,9 @@ impl FileStateMachine { // Current code writes CAS results as Insert (success) or Noop (failure), // so this branch is only reached when replaying WAL files created before // the fix. Apply new_value unconditionally as best-effort. - if let Some(new_value) = value { - data.insert(key.clone(), (new_value, term)); - applied_count += 1; - debug!("Replayed legacy CAS: key={:?}", key); - } else { - warn!("Legacy CAS WAL entry missing new_value"); - } + data.insert(key.clone(), (value, term)); + applied_count += 1; + debug!("Replayed legacy CAS: key={:?}", key); } WalOpCode::Noop | WalOpCode::Config | WalOpCode::CasFailed => { // No data modification needed @@ -736,8 +540,8 @@ impl FileStateMachine { } info!( - "WAL replay complete: {} operations replayed, {} applied, {} expired keys skipped", - replayed_count, applied_count, skipped_expired + "WAL replay complete: {} operations replayed, {} applied", + replayed_count, applied_count, ); // Unconditionally clear WAL after replay. load_data() already restored the last @@ -995,21 +799,6 @@ impl FileStateMachine { } } - /// Writes the lease table to `ttl_state.bin`. Once a checkpoint has cleared the WAL this - /// file is the only record of key expiry, so it is written to a temp file and renamed: a - /// crash mid-write cannot leave a truncated file. - fn persist_lease(&self) -> Result<(), Error> { - let Some(ref lease) = self.lease else { - return Ok(()); - }; - let tmp_path = self.data_dir.join("ttl_state.bin.tmp"); - let file = std::fs::File::create(&tmp_path)?; - std::io::Write::write_all(&mut &file, &lease.to_snapshot())?; - file.sync_all()?; - std::fs::rename(&tmp_path, self.data_dir.join("ttl_state.bin"))?; - Ok(()) - } - /// Batch WAL writes with proper durability guarantees. /// /// `cas_outcomes[i]` is consulted only when `entries[i].command` is @@ -1073,12 +862,6 @@ impl StateMachine for FileStateMachine { async fn start(&self) -> Result<(), Error> { self.running.store(true, Ordering::SeqCst); - // Load persisted lease data if configured - if self.lease.is_some() { - self.load_lease_data().await?; - debug!("Lease data loaded during state machine initialization"); - } - info!("File state machine started"); Ok(()) } @@ -1088,11 +871,9 @@ impl StateMachine for FileStateMachine { return Ok(()); // already closed } // The developer may keep an `Arc` to this state machine, so `Drop` may never run: - // persist everything a restart needs here. Both steps run; the first error is returned. + // persist everything a restart needs here. // No exclusive OS lock file to release. - let saved = self.save_hard_state(); - let leased = self.persist_lease(); - saved.and(leased) + self.save_hard_state() } fn is_running(&self) -> bool { @@ -1104,10 +885,6 @@ impl StateMachine for FileStateMachine { key_buffer: &[u8], ) -> Result, Error> { self.ensure_serving()?; - // Lazy cleanup: only check expiration in Lazy strategy - // Background strategy handles cleanup in dedicated async task - // Background cleanup strategy: expired keys are cleaned by background worker - // No on-read checks needed (simplifies get() hot path) let data = self.data.read(); Ok(data.get(key_buffer).map(|(value, _)| value.clone())) } @@ -1248,26 +1025,12 @@ impl StateMachine for FileStateMachine { Command::Noop => { results.push(ApplyResult::success(entry.index)); } - Command::Insert { - key, - value, - ttl_secs, - } => { + Command::Insert { key, value } => { data.insert(key.clone(), (value.clone(), entry.term)); - if let Some(ttl) = ttl_secs { - let lease = self - .lease - .as_ref() - .expect("lease always initialized by NodeBuilder"); - lease.register(key.clone(), *ttl); - } results.push(ApplyResult::success(entry.index)); } Command::Delete { key } => { data.remove(key.as_ref()); - if let Some(ref lease) = self.lease { - lease.unregister(key); - } results.push(ApplyResult::success(entry.index)); } Command::CompareAndSwap { @@ -1298,9 +1061,6 @@ impl StateMachine for FileStateMachine { } d_engine_core::BatchOp::Delete { key } => { data.remove(key.as_ref()); - if let Some(ref lease) = self.lease { - lease.unregister(key); - } } }); results.push(ApplyResult::success(entry.index)); @@ -1505,29 +1265,6 @@ impl StateMachine for FileStateMachine { new_data.insert(key, (value, term)); } - // Read and reload lease data if present - if pos + 8 <= buffer.len() { - let ttl_len_bytes = &buffer[pos..pos + 8]; - let ttl_len = u64::from_be_bytes([ - ttl_len_bytes[0], - ttl_len_bytes[1], - ttl_len_bytes[2], - ttl_len_bytes[3], - ttl_len_bytes[4], - ttl_len_bytes[5], - ttl_len_bytes[6], - ttl_len_bytes[7], - ]) as usize; - pos += 8; - - if pos + ttl_len <= buffer.len() { - let ttl_data = &buffer[pos..pos + ttl_len]; - if let Some(ref lease) = self.lease { - lease.reload(ttl_data)?; - } - } - } - // Atomically replace the data { let mut data = self.data.write(); @@ -1569,15 +1306,8 @@ impl StateMachine for FileStateMachine { }; // Batch serialize into a single buffer — eliminates per-entry async yield overhead. - let lease_snapshot = if let Some(ref lease) = self.lease { - lease.to_snapshot() - } else { - Vec::new() - }; let estimated: usize = - data_copy.iter().map(|(k, (v, _))| 8 + k.len() + 8 + v.len() + 8).sum::() - + 8 - + lease_snapshot.len(); + data_copy.iter().map(|(k, (v, _))| 8 + k.len() + 8 + v.len() + 8).sum::(); let mut buf = Vec::with_capacity(estimated); for (key, (value, term)) in &data_copy { @@ -1588,9 +1318,6 @@ impl StateMachine for FileStateMachine { buf.extend_from_slice(&term.to_be_bytes()); } - buf.extend_from_slice(&(lease_snapshot.len() as u64).to_be_bytes()); - buf.extend_from_slice(&lease_snapshot); - file.write_all(&buf).await?; file.flush().await?; @@ -1629,53 +1356,6 @@ impl StateMachine for FileStateMachine { self.reset().await } - async fn lease_background_cleanup(&self) -> Result, Error> { - let Some(ref lease) = self.lease else { - return Ok(vec![]); - }; - - // Fast path: no TTL keys ever registered — single atomic load (~10ns) - if !lease.has_lease_keys() { - return Ok(vec![]); - } - - let now = SystemTime::now(); - - // Fast path: sample first 10 entries — if none expired, skip full scan (~30ns) - if !lease.may_have_expired_keys(now) { - return Ok(vec![]); - } - - let expired_keys = lease.get_expired_keys(now); - - if expired_keys.is_empty() { - return Ok(vec![]); - } - - debug!( - "Lease background cleanup: found {} expired keys", - expired_keys.len() - ); - - // Delete expired keys from storage - { - let mut data = self.data.write(); - for key in &expired_keys { - data.remove(key); - } - } - - // Persist to disk after batch deletion; propagate error so caller can retry - self.persist_data_async().await?; - - info!( - "Lease background cleanup: deleted {} expired keys", - expired_keys.len() - ); - - Ok(expired_keys) - } - fn scan_prefix( &self, prefix: &[u8], diff --git a/d-engine-server/src/storage/adaptors/file/file_state_machine_test.rs b/d-engine-server/src/storage/adaptors/file/file_state_machine_test.rs index a0e5ef5e..ff1336b9 100644 --- a/d-engine-server/src/storage/adaptors/file/file_state_machine_test.rs +++ b/d-engine-server/src/storage/adaptors/file/file_state_machine_test.rs @@ -21,7 +21,7 @@ use crate::storage::adaptors::file::FileStateMachine; /// close() must mark the SM as not running. /// FileStateMachine has no exclusive OS lock, so close() is a lifecycle -/// signal and persists lease state. +/// signal and flushes state to disk. #[tokio::test] async fn test_file_sm_close_marks_not_running() { let dir = TempDir::new().unwrap(); @@ -50,7 +50,6 @@ async fn test_file_sm_reads_fail_after_close() { command: Command::Insert { key: Bytes::from_static(b"k"), value: Bytes::from_static(b"v"), - ttl_secs: None, }, }]) .await @@ -148,7 +147,6 @@ async fn test_wal_replay_after_crash() { command: Command::Insert { key: Bytes::from("key1"), value: Bytes::from("value1"), - ttl_secs: None, }, }, ApplyEntry { @@ -157,7 +155,6 @@ async fn test_wal_replay_after_crash() { command: Command::Insert { key: Bytes::from("key2"), value: Bytes::from("value2"), - ttl_secs: None, }, }, ]; @@ -176,7 +173,6 @@ async fn test_wal_replay_after_crash() { command: Command::Insert { key: Bytes::from("key3"), value: Bytes::from("value3"), - ttl_secs: None, }, }]; let dummy_outcomes = vec![false; crash_entries.len()]; // non-CAS entries: outcome unused @@ -203,6 +199,60 @@ async fn test_wal_replay_after_crash() { ); } +/// WAL replay stops at the first truncated entry and discards the tail. +/// +/// A truncated WAL tail is the expected outcome of a crash in the middle of a write. Replay must +/// stop at the first incomplete entry instead of reading garbage bytes. Complete entries before +/// the truncation point are applied; the truncated entry and anything after it are dropped. +#[tokio::test] +async fn test_wal_replay_truncated_tail_keeps_complete_entries() { + let temp_dir = tempfile::tempdir().unwrap(); + let data_dir = temp_dir.path().to_path_buf(); + + let sm = FileStateMachine::new(data_dir.clone()).await.unwrap(); + let entries = vec![ + ApplyEntry { + index: 1, + term: 1, + command: Command::Insert { + key: Bytes::from("key1"), + value: Bytes::from("value1"), + }, + }, + ApplyEntry { + index: 2, + term: 1, + command: Command::Insert { + key: Bytes::from("key2"), + value: Bytes::from("value2"), + }, + }, + ]; + // Write to the WAL only (memory and state.data untouched), then simulate a crash. + sm.append_to_wal(&entries, &[false, false]).await.unwrap(); + drop(sm); + + // Cut the last bytes off the WAL so the second entry is incomplete. + let wal_path = data_dir.join("wal.log"); + let wal = std::fs::OpenOptions::new().write(true).open(&wal_path).unwrap(); + let len = wal.metadata().unwrap().len(); + wal.set_len(len - 3).unwrap(); + drop(wal); + + let recovered = FileStateMachine::new(data_dir).await.unwrap(); + + assert_eq!( + recovered.get(b"key1").unwrap(), + Some(Bytes::from("value1")), + "complete entry before the truncation point must be replayed" + ); + assert_eq!( + recovered.get(b"key2").unwrap(), + None, + "truncated entry must be dropped" + ); +} + // ── CAS WAL crash-safety tests ──────────────────────────────────────────────── /// A failed CAS must NOT corrupt data when the node crashes and replays WAL. @@ -223,7 +273,6 @@ async fn test_cas_failure_wal_replay_does_not_corrupt_data() { command: Command::Insert { key: Bytes::from("k"), value: Bytes::from("original"), - ttl_secs: None, }, }]) .await @@ -279,7 +328,6 @@ async fn test_cas_in_same_chunk_as_preceding_insert() { command: Command::Insert { key: Bytes::from("k"), value: Bytes::from("v1"), - ttl_secs: None, }, }, ApplyEntry { @@ -312,7 +360,6 @@ async fn test_cas_success_wal_replay_applies_new_value() { command: Command::Insert { key: Bytes::from("k"), value: Bytes::from("v1"), - ttl_secs: None, }, }]) .await @@ -367,7 +414,6 @@ async fn test_replay_wal_unknown_opcode_returns_error() { command: Command::Insert { key: Bytes::from("k"), value: Bytes::from("v"), - ttl_secs: None, }, }]) .await @@ -376,16 +422,15 @@ async fn test_replay_wal_unknown_opcode_returns_error() { // Append a syntactically complete WAL entry whose opcode byte (99) is not // defined in WalOpCode. Format: index(8) + term(8) + opcode(1) + key_len(8) - // + val_len(8) + expire_at(8) = 41 bytes total (matches the Noop/CasFailed layout). + // + val_len(8) = 33 bytes total (matches the Noop/CasFailed layout). let wal_path = data_dir.join("wal.log"); let mut file = tokio::fs::OpenOptions::new().append(true).open(&wal_path).await.unwrap(); - let mut corrupt_entry = Vec::with_capacity(41); + let mut corrupt_entry = Vec::with_capacity(33); corrupt_entry.extend_from_slice(&2u64.to_be_bytes()); // index = 2 corrupt_entry.extend_from_slice(&1u64.to_be_bytes()); // term = 1 corrupt_entry.push(99u8); // unknown opcode corrupt_entry.extend_from_slice(&0u64.to_be_bytes()); // key_len = 0 corrupt_entry.extend_from_slice(&0u64.to_be_bytes()); // val_len = 0 - corrupt_entry.extend_from_slice(&0u64.to_be_bytes()); // expire_at = 0 file.write_all(&corrupt_entry).await.unwrap(); file.flush().await.unwrap(); drop(file); @@ -415,7 +460,6 @@ async fn test_file_sm_scan_prefix_returns_matching_keys() { command: Command::Insert { key: Bytes::from("/services/node1"), value: Bytes::from("10.0.0.1"), - ttl_secs: None, }, }, ApplyEntry { @@ -424,7 +468,6 @@ async fn test_file_sm_scan_prefix_returns_matching_keys() { command: Command::Insert { key: Bytes::from("/services/node2"), value: Bytes::from("10.0.0.2"), - ttl_secs: None, }, }, ApplyEntry { @@ -433,7 +476,6 @@ async fn test_file_sm_scan_prefix_returns_matching_keys() { command: Command::Insert { key: Bytes::from("/other/key"), value: Bytes::from("must_not_appear"), - ttl_secs: None, }, }, ]; @@ -466,7 +508,6 @@ async fn test_file_sm_scan_prefix_empty_namespace() { command: Command::Insert { key: Bytes::from("/other/key"), value: Bytes::from("v"), - ttl_secs: None, }, }]) .await @@ -507,7 +548,6 @@ async fn test_get_multi_file_sm_atomic_state_transitions() { command: Command::Insert { key: Bytes::from("/svc/addr"), value: Bytes::from("10.0.0.1"), - ttl_secs: None, }, }, ApplyEntry { @@ -516,7 +556,6 @@ async fn test_get_multi_file_sm_atomic_state_transitions() { command: Command::Insert { key: Bytes::from("/svc/version"), value: Bytes::from("v1"), - ttl_secs: None, }, }, ]) @@ -542,7 +581,6 @@ async fn test_get_multi_file_sm_atomic_state_transitions() { command: Command::Insert { key: Bytes::from("/svc/addr"), value: Bytes::from("10.0.0.2"), - ttl_secs: None, }, }, ApplyEntry { @@ -551,7 +589,6 @@ async fn test_get_multi_file_sm_atomic_state_transitions() { command: Command::Insert { key: Bytes::from("/svc/version"), value: Bytes::from("v2"), - ttl_secs: None, }, }, ]) @@ -597,7 +634,6 @@ async fn test_file_sm_scan_prefix_revision_reflects_applied_index() { command: Command::Insert { key: Bytes::from(format!("/s/{i}")), value: Bytes::from(format!("{i}")), - ttl_secs: None, }, }) .collect(); @@ -627,7 +663,6 @@ fn insert_at( command: Command::Insert { key: Bytes::from(key.to_vec()), value: Bytes::from(value.to_vec()), - ttl_secs: None, }, } } @@ -910,3 +945,101 @@ async fn test_replay_wal_skips_stale_entries_below_last_applied() { "last_applied must remain at the snapshot boundary after recovery" ); } + +/// An empty value is a real value, and WAL replay must restore it. +/// +/// Releasing a lock is written as a put or a CAS with an empty value. If replay treated an empty +/// value as "no value" and skipped the entry, the key would be missing after a crash even though +/// the write was acknowledged. +#[tokio::test] +async fn test_wal_replay_restores_empty_value() { + let temp_dir = tempfile::tempdir().unwrap(); + let data_dir = temp_dir.path().to_path_buf(); + + let sm = FileStateMachine::new(data_dir.clone()).await.unwrap(); + let entries = vec![insert_at(b"put_key", b"", 1), ApplyEntry { + index: 2, + term: 1, + command: Command::CompareAndSwap { + key: Bytes::from("cas_key"), + expected: None, + value: Bytes::new(), + }, + }]; + // Write to the WAL only (memory and state.data untouched), then simulate a crash. + // The second entry is a successful CAS. + sm.append_to_wal(&entries, &[false, true]).await.unwrap(); + drop(sm); + + let recovered = FileStateMachine::new(data_dir).await.unwrap(); + + assert_eq!( + recovered.get(b"put_key").unwrap(), + Some(Bytes::new()), + "empty-value insert must be replayed" + ); + assert_eq!( + recovered.get(b"cas_key").unwrap(), + Some(Bytes::new()), + "successful CAS with an empty value must be replayed" + ); +} + +// ── snapshot file format ────────────────────────────────────────────────────── + +/// `snapshot.bin` holds only `key_len | key | value_len | value | term` records. +/// +/// Nothing follows the last record. Guards the on-disk layout that +/// `apply_snapshot_from_file` parses until end of file. +#[tokio::test] +async fn test_file_sm_snapshot_contains_only_kv_records() { + let dir = TempDir::new().unwrap(); + let sm = FileStateMachine::new(dir.path().join("sm")).await.unwrap(); + sm.apply_chunk(&[insert_at(b"k", b"v", 1)]).await.unwrap(); + + let snapshot_dir = dir.path().join("snapshot"); + sm.generate_snapshot_data(snapshot_dir.clone(), LogId { term: 1, index: 1 }) + .await + .unwrap(); + + let len = std::fs::metadata(snapshot_dir.join("snapshot.bin")).unwrap().len(); + // key_len(8) + "k"(1) + value_len(8) + "v"(1) + term(8) + assert_eq!(len, 26, "snapshot.bin must contain exactly one KV record"); +} + +/// A snapshot of an empty state machine must install as an empty state machine. +/// +/// The receiver's older data has to be replaced, not kept, even though the snapshot file +/// contains no records. +#[tokio::test] +async fn test_file_sm_install_empty_snapshot_clears_existing_data() { + let dir = TempDir::new().unwrap(); + + let source = FileStateMachine::new(dir.path().join("source")).await.unwrap(); + let snapshot_dir = dir.path().join("snapshot"); + let checksum = source + .generate_snapshot_data(snapshot_dir.clone(), LogId { term: 1, index: 5 }) + .await + .unwrap(); + + let receiver = FileStateMachine::new(dir.path().join("receiver")).await.unwrap(); + receiver.apply_chunk(&[insert_at(b"stale", b"v", 1)]).await.unwrap(); + assert_eq!(receiver.get(b"stale").unwrap(), Some(Bytes::from("v"))); + + let metadata = SnapshotMetadata { + last_included: Some(LogId { term: 1, index: 5 }), + checksum: Bytes::copy_from_slice(&checksum), + }; + let result = receiver.apply_snapshot_from_file(&metadata, snapshot_dir).await.unwrap(); + + assert!( + matches!(result, SnapshotApplyResult::Applied { .. }), + "expected Applied, got: {result:?}" + ); + assert_eq!( + receiver.get(b"stale").unwrap(), + None, + "installing an empty snapshot must replace the receiver's data" + ); + assert_eq!(receiver.last_applied(), LogId { term: 1, index: 5 }); +} diff --git a/d-engine-server/src/storage/adaptors/rocksdb/rocksdb_state_machine.rs b/d-engine-server/src/storage/adaptors/rocksdb/rocksdb_state_machine.rs index d7f80dd0..f6806d15 100644 --- a/d-engine-server/src/storage/adaptors/rocksdb/rocksdb_state_machine.rs +++ b/d-engine-server/src/storage/adaptors/rocksdb/rocksdb_state_machine.rs @@ -3,7 +3,6 @@ use std::sync::Arc; use std::sync::atomic::AtomicBool; use std::sync::atomic::AtomicU64; use std::sync::atomic::Ordering; -use std::time::SystemTime; use arc_swap::ArcSwapOption; use async_trait::async_trait; @@ -12,7 +11,6 @@ use d_engine_core::ApplyEntry; use d_engine_core::ApplyResult; use d_engine_core::Command; use d_engine_core::Error; -use d_engine_core::Lease; use d_engine_core::ScanResult; use d_engine_core::SnapshotApplyResult; use d_engine_core::SnapshotError; @@ -41,16 +39,13 @@ use tracing::debug; use tracing::error; use tracing::info; use tracing::instrument; -use tracing::warn; use super::STATE_MACHINE_CF; use super::STATE_MACHINE_META_CF; -use crate::storage::TtlLease; const LAST_APPLIED_INDEX_KEY: &[u8] = b"last_applied_index"; const LAST_APPLIED_TERM_KEY: &[u8] = b"last_applied_term"; const SNAPSHOT_METADATA_KEY: &[u8] = b"snapshot_metadata"; -const TTL_STATE_KEY: &[u8] = b"ttl_state"; thread_local! { // SM WAL is redundant — Raft log WAL guarantees entry durability. @@ -85,7 +80,7 @@ struct CfExportFile { num_deletions: u64, } -/// RocksDB-based state machine implementation with lease support +/// RocksDB-based state machine implementation #[derive(Debug)] pub struct RocksDBStateMachine { // ArcSwapOption: close_db() stores None to release the RocksDB LOCK file immediately, @@ -95,11 +90,6 @@ pub struct RocksDBStateMachine { last_applied_index: AtomicU64, last_applied_term: AtomicU64, last_snapshot_metadata: RwLock>, - - // Lease management for automatic key expiration - // TtlLease is thread-safe internally (uses DashMap + Mutex) - // Injected by NodeBuilder after construction - lease: Option>, } /// Returns the lexicographic successor of `prefix` for use as an iterator upper bound. @@ -149,7 +139,6 @@ impl RocksDBStateMachine { last_applied_index: AtomicU64::new(last_applied_index), last_applied_term: AtomicU64::new(last_applied_term), last_snapshot_metadata: RwLock::new(last_snapshot_metadata), - lease: None, }) } @@ -167,22 +156,9 @@ impl RocksDBStateMachine { last_applied_index: AtomicU64::new(last_applied_index), last_applied_term: AtomicU64::new(last_applied_term), last_snapshot_metadata: RwLock::new(last_snapshot_metadata), - lease: None, }) } - /// Sets the lease manager for this state machine. - /// - /// This is an internal method called by NodeBuilder during initialization. - /// The lease will also be restored from snapshot during `apply_snapshot_from_file()`. - /// Also available for testing and benchmarks. - pub fn set_lease( - &mut self, - lease: Arc, - ) { - self.lease = Some(lease); - } - /// Replaces the underlying DB with `new_db`. Only available in `#[cfg(test)]`. /// Used to inject a read-only or otherwise broken DB to exercise error paths. #[cfg(test)] @@ -312,61 +288,6 @@ impl RocksDBStateMachine { }) } - fn persist_ttl_metadata(&self) -> Result<(), Error> { - let Some(ref lease) = self.lease else { - return Ok(()); - }; - let ttl_snapshot = lease.to_snapshot(); - self.with_db(|db| { - let cf = db.cf_handle(STATE_MACHINE_META_CF).ok_or_else(|| { - Error::System(d_engine_core::SystemError::Storage(StorageError::DbError( - "State machine meta CF not found".to_string(), - ))) - })?; - db.put_cf(&cf, TTL_STATE_KEY, &ttl_snapshot) - .map_err(|e| StorageError::DbError(e.to_string()))?; - debug!("Persisted TTL state to RocksDB"); - Ok(()) - }) - } - - /// Loads TTL state from RocksDB metadata after lease injection. - /// - /// Called after NodeBuilder injects the lease. - /// Also available for testing and benchmarks. - pub async fn load_lease_data(&self) -> Result<(), Error> { - let Some(ref lease) = self.lease else { - return Ok(()); // No lease configured - }; - self.with_db(|db| { - let cf = db.cf_handle(STATE_MACHINE_META_CF).ok_or_else(|| { - Error::System(d_engine_core::SystemError::Storage(StorageError::DbError( - "State machine meta CF not found".to_string(), - ))) - })?; - match db - .get_cf(&cf, TTL_STATE_KEY) - .map_err(|e| StorageError::DbError(e.to_string()))? - { - Some(ttl_data) => { - lease.reload(&ttl_data)?; - debug!("Loaded TTL state from RocksDB: {} active TTLs", lease.len()); - } - None => { - debug!("No TTL state found in RocksDB"); - } - } - Ok(()) - }) - } - - fn apply_batch( - &self, - batch: WriteBatch, - ) -> Result<(), Error> { - self.with_db(|db| db.write(&batch).map_err(|e| StorageError::DbError(e.to_string()).into())) - } - // ===== Snapshot restore helpers ===== /// Restore SM state from a snapshot (CF export format). @@ -382,18 +303,6 @@ impl RocksDBStateMachine { info!("Snapshot restore complete"); - // Restore lease if configured - if let Some(ref lease) = self.lease { - let ttl_path = snapshot_dir.join("ttl_state.bin"); - if ttl_path.exists() { - let ttl_data = tokio::fs::read(&ttl_path).await?; - lease.reload(&ttl_data)?; - self.persist_ttl_metadata()?; - } else { - warn!("No lease state found in snapshot"); - } - } - *self.last_snapshot_metadata.write() = Some(metadata.clone()); self.update_last_applied(last_included); @@ -589,7 +498,7 @@ impl RocksDBStateMachine { } /// Test-only: quiesce RocksDB's background compaction/flush WITHOUT flushing - /// MemTable, saving hard state, or persisting TTL metadata — simulates what a + /// MemTable or saving hard state — simulates what a /// real `kill -9` leaves behind on disk (no "dying gasp" persistence a real /// crash would never get). Callers copy the data directory to a fresh path /// immediately after calling this, then open the copy to simulate restart — @@ -617,17 +526,11 @@ impl StateMachine for RocksDBStateMachine { async fn start(&self) -> Result<(), Error> { self.is_serving.store(true, Ordering::SeqCst); - // Load persisted lease data if configured - if let Some(ref _lease) = self.lease { - self.load_lease_data().await?; - debug!("Lease data loaded during state machine initialization"); - } - info!("RocksDB state machine started"); Ok(()) } - /// Sequence: stop serving → persist TTL → flush → save hard state → cancel background + /// Sequence: stop serving → flush → save hard state → cancel background /// work → drop the DB handle (releasing the LOCK). A second call finds no DB and does nothing. fn close(&self) -> Result<(), Error> { if self.db.load().is_none() { @@ -645,7 +548,6 @@ impl StateMachine for RocksDBStateMachine { first_err.get_or_insert(e); } }; - record("persist TTL metadata", self.persist_ttl_metadata()); record("flush", self.flush()); record("save hard state", self.save_hard_state()); if let Some(db) = self.db.load().as_deref() { @@ -731,26 +633,13 @@ impl StateMachine for RocksDBStateMachine { debug!("Handling NOOP command at index {}", entry.index); results.push(ApplyResult::success(entry.index)); } - Command::Insert { - key, - value, - ttl_secs, - } => { + Command::Insert { key, value } => { batch.put_cf(&cf, key, value); - if let Some(ttl) = ttl_secs { - let lease = - self.lease.as_ref().expect("lease always initialized by NodeBuilder"); - lease.register(key.clone(), *ttl); - } - results.push(ApplyResult::success(entry.index)); } Command::Delete { key } => { batch.delete_cf(&cf, key); - if let Some(ref lease) = self.lease { - lease.unregister(key); - } results.push(ApplyResult::success(entry.index)); } Command::CompareAndSwap { @@ -796,9 +685,6 @@ impl StateMachine for RocksDBStateMachine { } d_engine_core::BatchOp::Delete { key } => { batch.delete_cf(&cf, key); - if let Some(ref lease) = self.lease { - lease.unregister(key); - } } }); results.push(ApplyResult::success(entry.index)); @@ -1012,13 +898,6 @@ impl StateMachine for RocksDBStateMachine { .await .map_err(Self::map_snapshot_join_error)??; - // Persist lease state alongside the export (if configured) - if let Some(ref lease) = self.lease { - let ttl_snapshot = lease.to_snapshot(); - let ttl_path = new_snapshot_dir.join("ttl_state.bin"); - tokio::fs::write(&ttl_path, ttl_snapshot).await?; - } - let checksum = compute_checksum_from_folder_path(&new_snapshot_dir).await?; info!("Snapshot generated at {:?}", new_snapshot_dir); @@ -1087,57 +966,6 @@ impl StateMachine for RocksDBStateMachine { Ok(()) } - async fn lease_background_cleanup(&self) -> Result, Error> { - let Some(ref lease) = self.lease else { - return Ok(vec![]); - }; - - // Fast path: no TTL keys ever registered — single atomic load (~10ns) - if !lease.has_lease_keys() { - return Ok(vec![]); - } - - let now = SystemTime::now(); - - // Fast path: sample first 10 entries — if none expired, skip full scan (~30ns) - if !lease.may_have_expired_keys(now) { - return Ok(vec![]); - } - - let expired_keys = lease.get_expired_keys(now); - - if expired_keys.is_empty() { - return Ok(vec![]); - } - - debug!( - "Lease background cleanup: found {} expired keys", - expired_keys.len() - ); - - // Delete expired keys from RocksDB - let batch = self.with_db(|db| { - let cf = db.cf_handle(STATE_MACHINE_CF).ok_or_else(|| { - Error::System(d_engine_core::SystemError::Storage(StorageError::DbError( - "State machine CF not found".to_string(), - ))) - })?; - let mut batch = WriteBatch::default(); - for key in &expired_keys { - batch.delete_cf(&cf, key); - } - Ok(batch) - })?; - self.apply_batch(batch)?; - - info!( - "Lease background cleanup: deleted {} expired keys", - expired_keys.len() - ); - - Ok(expired_keys) - } - fn scan_prefix( &self, prefix: &[u8], diff --git a/d-engine-server/src/storage/adaptors/rocksdb/rocksdb_state_machine_test.rs b/d-engine-server/src/storage/adaptors/rocksdb/rocksdb_state_machine_test.rs index 89533f38..c73aedf5 100644 --- a/d-engine-server/src/storage/adaptors/rocksdb/rocksdb_state_machine_test.rs +++ b/d-engine-server/src/storage/adaptors/rocksdb/rocksdb_state_machine_test.rs @@ -577,24 +577,6 @@ fn insert_at( command: Command::Insert { key: Bytes::copy_from_slice(key), value: Bytes::copy_from_slice(value), - ttl_secs: None, - }, - } -} - -fn insert_with_ttl( - key: &[u8], - value: &[u8], - ttl_secs: u64, - index: u64, -) -> ApplyEntry { - ApplyEntry { - index, - term: 1, - command: Command::Insert { - key: Bytes::copy_from_slice(key), - value: Bytes::copy_from_slice(value), - ttl_secs: if ttl_secs > 0 { Some(ttl_secs) } else { None }, }, } } @@ -639,7 +621,7 @@ async fn test_apply_chunk_cas_match_succeeds_and_writes_new_value() { let (_storage, sm) = RocksDBUnifiedEngine::open(dir.path()).unwrap(); // Pre-populate key - sm.apply_chunk(&[insert_with_ttl(b"k", b"old", 0, 1)]).await.unwrap(); + sm.apply_chunk(&[insert_at(b"k", b"old", 1)]).await.unwrap(); // CAS with matching expected value let results = sm.apply_chunk(&[cas_at(b"k", Some(b"old"), b"new", 2)]).await.unwrap(); @@ -658,7 +640,7 @@ async fn test_apply_chunk_cas_mismatch_returns_failure_and_preserves_value() { let dir = tempfile::TempDir::new().unwrap(); let (_storage, sm) = RocksDBUnifiedEngine::open(dir.path()).unwrap(); - sm.apply_chunk(&[insert_with_ttl(b"k", b"current", 0, 1)]).await.unwrap(); + sm.apply_chunk(&[insert_at(b"k", b"current", 1)]).await.unwrap(); let results = sm .apply_chunk(&[cas_at(b"k", Some(b"wrong_expected"), b"new", 2)]) @@ -694,7 +676,7 @@ async fn test_apply_chunk_cas_none_expected_on_existing_key_fails() { let dir = tempfile::TempDir::new().unwrap(); let (_storage, sm) = RocksDBUnifiedEngine::open(dir.path()).unwrap(); - sm.apply_chunk(&[insert_with_ttl(b"k", b"exists", 0, 1)]).await.unwrap(); + sm.apply_chunk(&[insert_at(b"k", b"exists", 1)]).await.unwrap(); let results = sm.apply_chunk(&[cas_at(b"k", None, b"new", 2)]).await.unwrap(); @@ -717,7 +699,7 @@ async fn test_apply_chunk_two_cas_same_key_second_must_see_first_write() { let dir = tempfile::TempDir::new().unwrap(); let (_storage, sm) = RocksDBUnifiedEngine::open(dir.path()).unwrap(); - sm.apply_chunk(&[insert_with_ttl(b"k", b"v0", 0, 1)]).await.unwrap(); + sm.apply_chunk(&[insert_at(b"k", b"v0", 1)]).await.unwrap(); // CAS1: v0 → v1 (should succeed) // CAS2: v0 → v2 (must fail — CAS1 already changed value to v1) @@ -747,7 +729,7 @@ async fn test_apply_chunk_delete_removes_key() { let dir = tempfile::TempDir::new().unwrap(); let (_storage, sm) = RocksDBUnifiedEngine::open(dir.path()).unwrap(); - sm.apply_chunk(&[insert_with_ttl(b"to_delete", b"v", 0, 1)]).await.unwrap(); + sm.apply_chunk(&[insert_at(b"to_delete", b"v", 1)]).await.unwrap(); sm.apply_chunk(&[delete_at(b"to_delete", 2)]).await.unwrap(); assert_eq!(sm.get(b"to_delete").unwrap(), None); @@ -768,7 +750,7 @@ async fn test_apply_chunk_returns_error_on_read_only_db() { let (_storage, sm) = RocksDBUnifiedEngine::open(&db_path).unwrap(); // Confirm the state machine works normally first - sm.apply_chunk(&[insert_with_ttl(b"k", b"v", 0, 1)]).await.unwrap(); + sm.apply_chunk(&[insert_at(b"k", b"v", 1)]).await.unwrap(); // Open the same DB as read-only and inject it — writes will now return NotSupported let ro_db = DB::open_cf_for_read_only( @@ -781,7 +763,7 @@ async fn test_apply_chunk_returns_error_on_read_only_db() { sm.swap_db_for_test(ro_db); // Write to a read-only DB must fail - let result = sm.apply_chunk(&[insert_with_ttl(b"k2", b"v2", 0, 2)]).await; + let result = sm.apply_chunk(&[insert_at(b"k2", b"v2", 2)]).await; assert!( result.is_err(), "apply_chunk should return Err when DB is read-only" diff --git a/d-engine-server/src/storage/adaptors/rocksdb/rocksdb_storage_engine_test.rs b/d-engine-server/src/storage/adaptors/rocksdb/rocksdb_storage_engine_test.rs index 77f1af7e..904f91cf 100644 --- a/d-engine-server/src/storage/adaptors/rocksdb/rocksdb_storage_engine_test.rs +++ b/d-engine-server/src/storage/adaptors/rocksdb/rocksdb_storage_engine_test.rs @@ -189,11 +189,7 @@ fn create_test_command_payload(index: u64) -> d_engine_proto::common::EntryPaylo let key = Bytes::from(format!("key_{index}").into_bytes()); let value = Bytes::from(format!("value_{index}").into_bytes()); - let insert = Insert { - key, - value, - ttl_secs: 0, - }; + let insert = Insert { key, value }; let operation = d_engine_proto::client::write_command::Operation::Insert(insert); let write_cmd = d_engine_proto::client::WriteCommand { operation: Some(operation), diff --git a/d-engine-server/src/storage/adaptors/rocksdb/rocksdb_unified_engine_test.rs b/d-engine-server/src/storage/adaptors/rocksdb/rocksdb_unified_engine_test.rs index 527d31ca..263be8f1 100644 --- a/d-engine-server/src/storage/adaptors/rocksdb/rocksdb_unified_engine_test.rs +++ b/d-engine-server/src/storage/adaptors/rocksdb/rocksdb_unified_engine_test.rs @@ -3,10 +3,8 @@ use std::sync::Mutex; use async_trait::async_trait; use d_engine_core::Error; -use d_engine_core::Lease; use d_engine_core::LogStore; use d_engine_core::StorageEngine; -use d_engine_core::config::LeaseConfig; use d_engine_core::state_machine_test::StateMachineBuilder; use d_engine_core::state_machine_test::StateMachineTestSuite; use d_engine_core::storage_engine_test::StorageEngineBuilder; @@ -17,7 +15,6 @@ use super::RocksDBStateMachine; use super::RocksDBStorageEngine; use super::RocksDBUnifiedEngine; use crate::StateMachine; -use crate::storage::TtlLease; // Both test suites call build() twice within a single persistence test (write data → drop → // build() again → verify data survived). Both builders therefore: @@ -168,74 +165,3 @@ fn test_concurrent_open_same_path_fails() { "second open on a live DB must fail with lock error" ); } - -// ── Lease round-trip tests ──────────────────────────────────────────────────── - -fn make_lease() -> Arc { - Arc::new(TtlLease::new(LeaseConfig { - cleanup_interval_ms: 1000, - max_cleanup_duration_ms: 10, - })) -} - -/// load_lease_data() is a no-op when no lease is configured (lease = None). -#[tokio::test] -async fn test_load_lease_data_without_lease_is_noop() { - let dir = TempDir::new().expect("temp dir"); - let (_storage, sm) = RocksDBUnifiedEngine::open(dir.path()).expect("open"); - - // SM was opened without a lease — load_lease_data must succeed silently. - let result = sm.load_lease_data().await; - assert!( - result.is_ok(), - "load_lease_data with no lease must not fail" - ); -} - -/// Registering a lease key, stopping the SM (which persists TTL state), then -/// reloading from the same DB must restore the lease entry. -#[tokio::test] -async fn test_persist_and_reload_lease_round_trip() { - let dir = TempDir::new().expect("temp dir"); - let lease = make_lease(); - - // --- Phase 1: write and persist --- - { - let (_storage, mut sm) = RocksDBUnifiedEngine::open(dir.path()).expect("open"); - sm.set_lease(Arc::clone(&lease)); - // Register a key with a long TTL so it survives the reload. - lease.register(bytes::Bytes::from("persistent_key"), 3600); - // close() calls persist_ttl_metadata() → writes TTL_STATE_KEY to RocksDB. - sm.close().unwrap(); - // Ensure DB is flushed before drop. - } - // Wait for RocksDB to release the file lock. - tokio::time::sleep(std::time::Duration::from_millis(100)).await; - - // --- Phase 2: reopen and reload --- - let new_lease = make_lease(); - let (_storage2, mut sm2) = RocksDBUnifiedEngine::open(dir.path()).expect("reopen"); - sm2.set_lease(Arc::clone(&new_lease)); - sm2.load_lease_data().await.expect("load_lease_data"); - - // The key registered in Phase 1 must be present in the reloaded lease. - assert!( - new_lease.has_lease_keys(), - "reloaded lease must contain the persisted key" - ); - assert_eq!(new_lease.len(), 1, "exactly one key should be restored"); -} - -/// load_lease_data() succeeds when TTL_STATE_KEY is absent in the DB (first start). -#[tokio::test] -async fn test_load_lease_data_with_empty_db_succeeds() { - let dir = TempDir::new().expect("temp dir"); - let (_storage, mut sm) = RocksDBUnifiedEngine::open(dir.path()).expect("open"); - - // Inject lease but do NOT persist anything first. - sm.set_lease(make_lease()); - - // load_lease_data on a fresh DB must succeed without error. - let result = sm.load_lease_data().await; - assert!(result.is_ok(), "load_lease_data on empty DB must succeed"); -} diff --git a/d-engine-server/src/storage/lease.rs b/d-engine-server/src/storage/lease.rs deleted file mode 100644 index f0ca91f2..00000000 --- a/d-engine-server/src/storage/lease.rs +++ /dev/null @@ -1,366 +0,0 @@ -//! Default lease implementation for d-engine. -//! -//! Provides high-performance lease management with single-index lock-free architecture. -//! The `Lease` trait is defined in d-engine-core for framework-level abstraction. -//! -//! # Architecture -//! -//! - **Single index**: DashMap (completely lock-free for register/unregister) -//! - **Cleanup**: O(N) iteration with time limit and shard read locks (rare operation) -//! -//! # Concurrency Model -//! -//! - **Register**: O(1) lock-free (single shard write lock) -//! - **Unregister**: O(1) lock-free (single shard write lock) -//! - **Cleanup**: O(N) with shard read locks (frequency: 1/1000 applies, duration: 1ms max) -//! - **No global Mutex** - Eliminates lock contention under high concurrency -//! -//! # Performance vs Dual-Index Design -//! -//! Old design (BTreeMap + Mutex): -//! - Register: O(log N) + Mutex lock → contention under concurrency -//! - Cleanup: O(K log N) + Mutex lock -//! -//! New design (DashMap only): -//! - Register: O(1) lock-free → zero contention -//! - Cleanup: O(N) with read locks → no write blocking, rare execution - -use std::collections::HashMap; -use std::sync::atomic::AtomicBool; -use std::sync::atomic::AtomicU64; -use std::sync::atomic::Ordering; -use std::time::Duration; -use std::time::SystemTime; - -use bytes::Bytes; -use d_engine_core::Lease; -use dashmap::DashMap; -use serde::Deserialize; -use serde::Serialize; - -use crate::Result; - -/// Default lease implementation with single-index lock-free architecture. -/// -/// # Performance Characteristics -/// -/// - `is_expired()`: O(1), ~10-20ns, lock-free -/// - `register()`: O(1), ~20ns, lock-free (single shard write lock) -/// - `unregister()`: O(1), ~20ns, lock-free (single shard write lock) -/// - `cleanup()`: O(N), time-limited, shard read locks (rare: 1/1000 applies) -/// -/// # Memory Usage -/// -/// - Per-key overhead: ~50 bytes (single DashMap entry) -/// - Expired keys are removed automatically during cleanup -#[derive(Debug)] -pub struct TtlLease { - /// Lease cleanup configuration (immutable after creation) - config: d_engine_core::config::LeaseConfig, - - /// Apply counter for piggyback cleanup frequency - apply_counter: AtomicU64, - - /// Single index: key → expiration_time (completely lock-free) - /// - Register/Unregister: O(1), single shard lock - /// - Cleanup: O(N) iteration with shard read locks - key_to_expiry: DashMap, - - /// Whether any lease has ever been registered - /// Once true, stays true forever (optimization flag) - has_keys: AtomicBool, -} - -impl TtlLease { - /// Creates a new default lease manager with the given configuration. - /// - /// # Arguments - /// * `config` - Lease cleanup strategy configuration - pub fn new(config: d_engine_core::config::LeaseConfig) -> Self { - Self { - config, - apply_counter: AtomicU64::new(0), - key_to_expiry: DashMap::new(), - has_keys: AtomicBool::new(false), - } - } - - /// Get expiration time for a specific key. - /// - /// # Performance - /// O(1) - DashMap lookup, lock-free - #[cfg(test)] - pub fn get_expiration( - &self, - key: &[u8], - ) -> Option { - self.key_to_expiry.get(key).map(|entry| *entry.value()) - } - - /// Restore from snapshot data (used during initialization). - /// - /// Filters out already-expired keys during restoration. - /// - /// # Arguments - /// * `data` - Serialized snapshot data - /// * `config` - Lease configuration to use - pub fn from_snapshot( - data: &[u8], - config: d_engine_core::config::LeaseConfig, - ) -> Self { - let snapshot: LeaseSnapshot = match bincode::deserialize(data) { - Ok(s) => s, - Err(_) => return Self::new(config), - }; - - let now = SystemTime::now(); - let manager = Self::new(config); - - // Rebuild single index, skipping expired keys - for (key, expire_at) in snapshot.key_to_expiry { - if expire_at > now { - let key_bytes = Bytes::from(key); - manager.key_to_expiry.insert(key_bytes, expire_at); - } - } - - // Restore has_keys flag if we have any keys - if !manager.key_to_expiry.is_empty() { - manager.has_keys.store(true, Ordering::Relaxed); - } - - manager - } - - /// Returns reference to the lease configuration. - /// - /// Used by StateMachine implementations to check cleanup strategy. - pub fn config(&self) -> &d_engine_core::config::LeaseConfig { - &self.config - } -} - -impl Lease for TtlLease { - /// Register a key with TTL (Time-To-Live). - /// - /// # TTL Semantics - /// - /// - **Absolute expiration time**: The expiration time is calculated as `expire_at = - /// SystemTime::now() + Duration::from_secs(ttl_secs)` and stored internally. - /// - **Crash-safe**: The absolute expiration time survives node restarts. After crash recovery, - /// expired keys remain expired (no TTL reset). - /// - **Persistent**: The expiration time is persisted to disk during snapshot generation and - /// graceful shutdown. - /// - /// # Example - /// - /// ```text - /// T0: register(key="foo", ttl=10) → expire_at = T0 + 10 = T10 - /// T5: CRASH - /// T12: RESTART - /// → WAL replay: expire_at = T10 < T12 (already expired) - /// → Key is NOT restored (correctly expired) - /// ``` - /// - /// # Arguments - /// - /// * `key` - The key to register expiration for - /// * `ttl_secs` - Time-to-live in seconds from now - /// - /// # Performance - /// - /// O(1) - Completely lock-free, only acquires single shard write lock - fn register( - &self, - key: Bytes, - ttl_secs: u64, - ) { - // Mark that lease is being used (lazy activation) - self.has_keys.store(true, Ordering::Relaxed); - - // Calculate absolute expiration time - let expire_at = SystemTime::now() + Duration::from_secs(ttl_secs); - - // Single index update (overwrites old value if exists) - // DashMap::insert is lock-free (only single shard write lock) - self.key_to_expiry.insert(key, expire_at); - } - - /// Unregister a key's TTL. - /// - /// # Performance - /// - /// O(1) - Completely lock-free, only acquires single shard write lock - fn unregister( - &self, - key: &[u8], - ) { - // Single index removal (lock-free) - self.key_to_expiry.remove(key); - } - - /// Check if a key has expired. - /// - /// # Performance - /// - /// O(1) - Lock-free DashMap lookup - fn is_expired( - &self, - key: &[u8], - ) -> bool { - if let Some(expire_at) = self.key_to_expiry.get(key) { - *expire_at <= SystemTime::now() - } else { - false - } - } - - /// Get all expired keys (without time limit). - /// - /// This method is rarely used directly. Most cleanup happens via `on_apply()`. - /// - /// # Performance - /// - /// O(N) - Iterates all keys with shard read locks - fn get_expired_keys( - &self, - now: SystemTime, - ) -> Vec { - // Phase 1: collect expired keys (read-only) - let to_remove: Vec = self - .key_to_expiry - .iter() - .filter(|entry| *entry.value() <= now) - .map(|entry| entry.key().clone()) - .collect(); - - // Phase 2: remove after dropping iter (avoids deadlock) - to_remove - .into_iter() - .filter_map(|key| self.key_to_expiry.remove_if(&key, |_, v| *v <= now).map(|(k, _)| k)) - .collect() - } - - /// Piggyback cleanup on apply operations (DEPRECATED - no longer used). - /// - /// This method is kept for backward compatibility but is no longer called. - /// Cleanup is now handled by: - /// - Lazy strategy: cleanup in get() method - /// - Background strategy: dedicated async task - /// - /// # Performance - /// - /// - Fast path: O(1) - always returns empty vec - fn on_apply(&self) -> Vec { - // No-op: piggyback cleanup removed to avoid blocking apply path - vec![] - } - - /// Check if any keys have been registered. - /// - /// # Performance - /// - /// O(1) - Single atomic load - fn has_lease_keys(&self) -> bool { - self.has_keys.load(Ordering::Relaxed) - } - - /// Quick check if there might be expired keys. - /// - /// This is a heuristic check - samples first 10 entries. - /// May return false negatives but never false positives. - /// - /// # Performance - /// - /// O(1) - Checks first few entries only - fn may_have_expired_keys( - &self, - now: SystemTime, - ) -> bool { - if !self.has_lease_keys() { - return false; - } - - // Quick check: iterate first 10 entries - // DashMap::iter().take(10) is cheap (early termination) - for entry in self.key_to_expiry.iter().take(10) { - if *entry.value() <= now { - return true; - } - } - - false - } - - /// Get total number of keys with active leases. - /// - /// # Performance - /// - /// O(1) - DashMap maintains internal count - fn len(&self) -> usize { - self.key_to_expiry.len() - } - - /// Serialize current lease state to snapshot. - /// - /// # Performance - /// - /// O(N) - Iterates all keys with shard read locks - fn to_snapshot(&self) -> Vec { - let snapshot = LeaseSnapshot { - key_to_expiry: self - .key_to_expiry - .iter() - .map(|entry| (entry.key().to_vec(), *entry.value())) - .collect(), - }; - bincode::serialize(&snapshot).unwrap_or_default() - } - - /// Reload lease state from snapshot. - /// - /// Filters out already-expired keys during restoration. - /// - /// # Performance - /// - /// O(N) - Rebuilds single index - fn reload( - &self, - data: &[u8], - ) -> Result<()> { - let snapshot: LeaseSnapshot = bincode::deserialize(data).map_err(|e| { - crate::Error::System(d_engine_core::SystemError::Storage( - d_engine_core::StorageError::StateMachineError(format!( - "Failed to deserialize lease snapshot: {e}" - )), - )) - })?; - - let now = SystemTime::now(); - - // Clear existing data - self.key_to_expiry.clear(); - self.apply_counter.store(0, Ordering::Relaxed); - - // Rebuild single index, skipping expired keys - for (key, expire_at) in snapshot.key_to_expiry { - if expire_at > now { - let key_bytes = Bytes::from(key); - self.key_to_expiry.insert(key_bytes, expire_at); - } - } - - // Update has_keys flag - if !self.key_to_expiry.is_empty() { - self.has_keys.store(true, Ordering::Relaxed); - } - - Ok(()) - } -} - -/// Snapshot-serializable lease state. -#[derive(Debug, Serialize, Deserialize)] -struct LeaseSnapshot { - key_to_expiry: HashMap, SystemTime>, -} diff --git a/d-engine-server/src/storage/lease_integration_test.rs b/d-engine-server/src/storage/lease_integration_test.rs deleted file mode 100644 index 02e6030a..00000000 --- a/d-engine-server/src/storage/lease_integration_test.rs +++ /dev/null @@ -1,1261 +0,0 @@ -#![allow(clippy::field_reassign_with_default)] -#![allow(clippy::uninlined_format_args)] - -//! Integration tests for Lease functionality across the full stack -//! -//! This module contains comprehensive tests for: -//! - TtlLease unit tests (moved from ttl_manager.rs) -//! - FileStateMachine Lease integration tests -//! - RocksDBStateMachine Lease integration tests - -mod lease_tests { - use std::thread::sleep; - use std::time::Duration; - - use bytes::Bytes; - use d_engine_core::Lease; - - use crate::storage::TtlLease; - - #[test] - fn test_register_and_get_expired() { - let config = d_engine_core::config::LeaseConfig::default(); - let manager = TtlLease::new(config); - - // Register keys with 1 second TTL - manager.register(Bytes::from("key1"), 1); - manager.register(Bytes::from("key2"), 1); - - assert_eq!(manager.len(), 2); - - // No keys expired yet - let expired = manager.get_expired_keys(std::time::SystemTime::now()); - assert_eq!(expired.len(), 0); - - // Wait for expiration - sleep(Duration::from_secs(2)); - - let expired = manager.get_expired_keys(std::time::SystemTime::now()); - assert_eq!(expired.len(), 2); - assert_eq!(manager.len(), 0); - } - - #[test] - fn test_unregister() { - let config = d_engine_core::config::LeaseConfig::default(); - let manager = TtlLease::new(config); - - manager.register(Bytes::from("key1"), 10); - assert_eq!(manager.len(), 1); - - manager.unregister(b"key1"); - assert_eq!(manager.len(), 0); - } - - #[test] - fn test_update_ttl() { - let config = d_engine_core::config::LeaseConfig::default(); - let manager = TtlLease::new(config); - - // Register with 10 seconds - manager.register(Bytes::from("key1"), 10); - - // Update to 20 seconds - manager.register(Bytes::from("key1"), 20); - - // Should only have one TTL entry (old one replaced) - assert_eq!(manager.len(), 1); - } - - #[test] - fn test_snapshot_roundtrip() { - let config = d_engine_core::config::LeaseConfig::default(); - let manager = TtlLease::new(config.clone()); - - manager.register(Bytes::from("key1"), 3600); - manager.register(Bytes::from("key2"), 7200); - - let snapshot = manager.to_snapshot(); - let restored = TtlLease::from_snapshot(&snapshot, config); - - assert_eq!(restored.len(), 2); - } - - #[test] - fn test_snapshot_filters_expired() { - let config = d_engine_core::config::LeaseConfig::default(); - let manager = TtlLease::new(config.clone()); - - // Register key with 1 second TTL - manager.register(Bytes::from("key1"), 1); - manager.register(Bytes::from("key2"), 3600); - - sleep(Duration::from_secs(2)); - - let snapshot = manager.to_snapshot(); - let restored = TtlLease::from_snapshot(&snapshot, config); - - // Only key2 should be restored - assert_eq!(restored.len(), 1); - } -} - -mod file_state_machine_tests { - use std::time::Duration; - - use bytes::Bytes; - use d_engine_core::ApplyEntry; - use d_engine_core::Command; - use d_engine_core::StateMachine; - use d_engine_core::file_io::compute_checksum_from_folder_path; - use tempfile::TempDir; - use tokio::time::sleep; - - use crate::storage::FileStateMachine; - use crate::storage::TtlLease; - - /// Helper to create a FileStateMachine with lease injected for testing - async fn create_file_state_machine_with_lease( - path: std::path::PathBuf, - lease_config: d_engine_core::config::LeaseConfig, - ) -> FileStateMachine { - let mut sm = FileStateMachine::new(path).await.unwrap(); - let lease = std::sync::Arc::new(TtlLease::new(lease_config)); - sm.set_lease(lease); - sm.load_lease_data().await.unwrap(); - sm - } - - fn create_insert_entry( - index: u64, - term: u64, - key: &[u8], - value: &[u8], - ttl_secs: u64, - ) -> ApplyEntry { - ApplyEntry { - index, - term, - command: Command::Insert { - key: Bytes::copy_from_slice(key), - value: Bytes::copy_from_slice(value), - ttl_secs: if ttl_secs > 0 { Some(ttl_secs) } else { None }, - }, - } - } - - #[tokio::test] - async fn test_ttl_expiration_after_apply() { - let temp_dir = TempDir::new().unwrap(); - let lease_config = d_engine_core::config::LeaseConfig { - cleanup_interval_ms: 1000, - max_cleanup_duration_ms: 1, - }; - let sm = - create_file_state_machine_with_lease(temp_dir.path().to_path_buf(), lease_config).await; - - // Insert key with 2 second TTL - let entry = create_insert_entry(1, 1, b"ttl_key", b"ttl_value", 2); - sm.apply_chunk(&[entry]).await.unwrap(); - - // Key should exist immediately - let value = sm.get(b"ttl_key").unwrap(); - assert_eq!(value, Some(Bytes::from("ttl_value"))); - - // Wait for expiration - sleep(Duration::from_secs(3)).await; - - // Background strategy: manually trigger cleanup (simulates background worker) - sm.lease_background_cleanup().await.unwrap(); - - // Key should be deleted by background cleanup - let value = sm.get(b"ttl_key").unwrap(); - assert_eq!(value, None); - } - - #[tokio::test] - async fn test_ttl_snapshot_persistence() { - let temp_dir = TempDir::new().unwrap(); - let lease_config = d_engine_core::config::LeaseConfig { - cleanup_interval_ms: 1000, - max_cleanup_duration_ms: 1, - }; - let sm = create_file_state_machine_with_lease( - temp_dir.path().to_path_buf(), - lease_config.clone(), - ) - .await; - - // Insert key with 3600 second TTL (won't expire during test) - let entry = create_insert_entry(1, 1, b"persistent_key", b"persistent_value", 3600); - sm.apply_chunk(&[entry]).await.unwrap(); - - // Create snapshot - let snapshot_dir = temp_dir.path().join("snapshot"); - sm.generate_snapshot_data(snapshot_dir.clone(), d_engine_proto::common::LogId { - index: 1, - term: 1, - }) - .await - .unwrap(); - - // Create new state machine with lease config and restore snapshot - let temp_dir2 = TempDir::new().unwrap(); - let sm2 = - create_file_state_machine_with_lease(temp_dir2.path().to_path_buf(), lease_config) - .await; - - let checksum = compute_checksum_from_folder_path(&snapshot_dir).await.unwrap(); - sm2.apply_snapshot_from_file( - &d_engine_proto::server::storage::SnapshotMetadata { - last_included: Some(d_engine_proto::common::LogId { index: 1, term: 1 }), - checksum: Bytes::copy_from_slice(&checksum), - }, - snapshot_dir, - ) - .await - .unwrap(); - - // Key should exist in restored state machine - let value = sm2.get(b"persistent_key").unwrap(); - assert_eq!(value, Some(Bytes::from("persistent_value"))); - } - - #[tokio::test] - async fn test_file_state_machine_ttl_persistence_across_restart() { - let temp_dir = TempDir::new().unwrap(); - let state_machine_path = temp_dir.path().to_path_buf(); - let lease_config = d_engine_core::config::LeaseConfig { - cleanup_interval_ms: 1000, - max_cleanup_duration_ms: 1, - }; - - // Phase 1: Create state machine and insert keys with TTL - { - let sm = create_file_state_machine_with_lease( - state_machine_path.clone(), - lease_config.clone(), - ) - .await; - - let entry1 = create_insert_entry(1, 1, b"short_ttl_key", b"value1", 2); - let entry2 = create_insert_entry(2, 1, b"long_ttl_key", b"value2", 3600); - let entry3 = create_insert_entry(3, 1, b"no_ttl_key", b"value3", 0); - - sm.apply_chunk(&[entry1, entry2, entry3]).await.unwrap(); - - // Verify all keys exist - assert_eq!( - sm.get(b"short_ttl_key").unwrap(), - Some(Bytes::from("value1")) - ); - assert_eq!( - sm.get(b"long_ttl_key").unwrap(), - Some(Bytes::from("value2")) - ); - assert_eq!(sm.get(b"no_ttl_key").unwrap(), Some(Bytes::from("value3"))); - - // Gracefully close state machine to persist TTL data - sm.close().unwrap(); - // State machine drops here - } - - // Phase 2: Restart - create new state machine from same directory - { - let sm = create_file_state_machine_with_lease(state_machine_path.clone(), lease_config) - .await; - - // Verify all keys still exist after restart - assert_eq!( - sm.get(b"short_ttl_key").unwrap(), - Some(Bytes::from("value1")) - ); - assert_eq!( - sm.get(b"long_ttl_key").unwrap(), - Some(Bytes::from("value2")) - ); - assert_eq!(sm.get(b"no_ttl_key").unwrap(), Some(Bytes::from("value3"))); - - // Wait for short TTL to expire - sleep(Duration::from_secs(3)).await; - - // Manually trigger background cleanup - sm.lease_background_cleanup().await.unwrap(); - - // short_ttl_key should be expired - assert_eq!(sm.get(b"short_ttl_key").unwrap(), None); - // long_ttl_key should still exist - assert_eq!( - sm.get(b"long_ttl_key").unwrap(), - Some(Bytes::from("value2")) - ); - // no_ttl_key should still exist - assert_eq!(sm.get(b"no_ttl_key").unwrap(), Some(Bytes::from("value3"))); - } - } - - #[tokio::test] - async fn test_ttl_update_cancels_previous() { - let temp_dir = TempDir::new().unwrap(); - let lease_config = d_engine_core::config::LeaseConfig { - cleanup_interval_ms: 1000, - max_cleanup_duration_ms: 1, - }; - let sm = - create_file_state_machine_with_lease(temp_dir.path().to_path_buf(), lease_config).await; - - // Insert key with 2 second TTL - let entry1 = create_insert_entry(1, 1, b"update_key", b"value1", 2); - sm.apply_chunk(&[entry1]).await.unwrap(); - - // Immediately update with longer TTL - let entry2 = create_insert_entry(2, 1, b"update_key", b"value2", 10); - sm.apply_chunk(&[entry2]).await.unwrap(); - - // Wait past original TTL - sleep(Duration::from_secs(3)).await; - - // Manually trigger cleanup - sm.lease_background_cleanup().await.unwrap(); - - // Key should still exist (new TTL not expired) - let value = sm.get(b"update_key").unwrap(); - assert_eq!(value, Some(Bytes::from("value2"))); - } - /// expiration times in the past are NOT restored to the state machine. - #[tokio::test] - async fn test_crash_safe_ttl_wal_replay_skips_expired() { - let temp_dir = TempDir::new().unwrap(); - let state_machine_path = temp_dir.path().to_path_buf(); - let lease_config = d_engine_core::config::LeaseConfig { - cleanup_interval_ms: 1000, - max_cleanup_duration_ms: 1, - }; - - // Phase 1: Write keys with TTL to WAL (without persisting to state.data) - { - let sm = create_file_state_machine_with_lease( - state_machine_path.clone(), - lease_config.clone(), - ) - .await; - - // Insert key with 1 second TTL (will expire quickly) - let entry1 = create_insert_entry(1, 1, b"expired_key", b"value1", 1); - // Insert key with long TTL (won't expire during test) - let entry2 = create_insert_entry(2, 1, b"valid_key", b"value2", 3600); - // Insert key with no TTL - let entry3 = create_insert_entry(3, 1, b"permanent_key", b"value3", 0); - - sm.apply_chunk(&[entry1, entry2, entry3]).await.unwrap(); - - // Verify all keys exist immediately after write - assert_eq!(sm.get(b"expired_key").unwrap(), Some(Bytes::from("value1"))); - assert_eq!(sm.get(b"valid_key").unwrap(), Some(Bytes::from("value2"))); - assert_eq!( - sm.get(b"permanent_key").unwrap(), - Some(Bytes::from("value3")) - ); - - // Wait for expired_key TTL to expire - sleep(Duration::from_secs(2)).await; - - // Manually trigger background cleanup - sm.lease_background_cleanup().await.unwrap(); - - // Verify expired_key is now expired - assert_eq!( - sm.get(b"expired_key").unwrap(), - None, - "Key should be expired after background cleanup" - ); - - // Now manually delete state.data to force WAL replay on next startup - // This simulates a crash where state.data was not synced to disk - let data_path = state_machine_path.join("state.data"); - if data_path.exists() { - std::fs::remove_file(&data_path).unwrap(); - } - - // Also clear TTL state to force clean replay - let ttl_path = state_machine_path.join("ttl_state.bin"); - if ttl_path.exists() { - std::fs::remove_file(&ttl_path).unwrap(); - } - - // Drop state machine (WAL remains with absolute expiration times) - drop(sm); - } - - // Phase 2: Restart and replay WAL - { - let sm = create_file_state_machine_with_lease(state_machine_path.clone(), lease_config) - .await; - - // WAL replay should skip expired_key (crash-safe behavior) - // The absolute expiration time in WAL is in the past, so it should not be restored - assert_eq!( - sm.get(b"expired_key").unwrap(), - None, - "Expired key should NOT be restored from WAL" - ); - - // Valid key should be restored with remaining TTL - assert_eq!( - sm.get(b"valid_key").unwrap(), - Some(Bytes::from("value2")), - "Valid key should be restored from WAL" - ); - - // Permanent key should be restored - assert_eq!( - sm.get(b"permanent_key").unwrap(), - Some(Bytes::from("value3")), - "Permanent key should be restored from WAL" - ); - - sm.close().unwrap(); - } - } - - /// WAL replay stops at the first truncated entry and discards the tail. - /// - /// A WAL tail truncation is the expected outcome of a crash mid-write. Replay stops at the - /// first incomplete entry rather than continuing with garbage bytes or silently loading a - /// partial entry with wrong TTL. Complete entries before the truncation point are applied; - /// the truncated entry and anything after it are dropped. - #[tokio::test] - async fn test_wal_replay_handles_incomplete_entries() { - let temp_dir = TempDir::new().unwrap(); - let state_machine_path = temp_dir.path().to_path_buf(); - - let wal_path = state_machine_path.join("wal.log"); - let mut wal_data = Vec::new(); - - // Complete entry — must survive replay - wal_data.extend_from_slice(&1u64.to_be_bytes()); // index - wal_data.extend_from_slice(&1u64.to_be_bytes()); // term - wal_data.push(1u8); // Insert - wal_data.extend_from_slice(&8u64.to_be_bytes()); // key_len - wal_data.extend_from_slice(b"complete"); // key - wal_data.extend_from_slice(&6u64.to_be_bytes()); // value_len - wal_data.extend_from_slice(b"value1"); // value - wal_data.extend_from_slice(&0u64.to_be_bytes()); // expire_at = 0 (no TTL) - - // Truncated entry: missing expire_at — must be discarded, replay stops here - wal_data.extend_from_slice(&2u64.to_be_bytes()); // index - wal_data.extend_from_slice(&1u64.to_be_bytes()); // term - wal_data.push(1u8); // Insert - wal_data.extend_from_slice(&10u64.to_be_bytes()); // key_len - wal_data.extend_from_slice(b"incomplete"); // key - wal_data.extend_from_slice(&6u64.to_be_bytes()); // value_len - wal_data.extend_from_slice(b"value2"); // value - // expire_at deliberately omitted — simulates crash mid-write - - std::fs::write(&wal_path, wal_data).unwrap(); - - // Replay must succeed (truncated tail is not a fatal error) - let sm = FileStateMachine::new(state_machine_path.clone()).await.unwrap(); - - // Complete entry is applied - assert_eq!(sm.get(b"complete").unwrap(), Some(Bytes::from("value1"))); - - // Truncated entry is dropped — replay stopped before applying it - assert_eq!( - sm.get(b"incomplete").unwrap(), - None, - "Truncated WAL tail must be discarded, not loaded with assumed TTL" - ); - } - - /// Test: WAL replay with only expired entries results in empty state - #[tokio::test] - async fn test_wal_replay_all_expired_empty_state() { - let temp_dir = TempDir::new().unwrap(); - let state_machine_path = temp_dir.path().to_path_buf(); - - // Create entries that are all expired (use very old timestamp) - let now = std::time::SystemTime::now(); - let expired_time = now - std::time::Duration::from_secs(100); - let expire_at_secs = expired_time.duration_since(std::time::UNIX_EPOCH).unwrap().as_secs(); - - // Manually create WAL with old expire_at times - let wal_path = state_machine_path.join("wal.log"); - let mut wal_data = Vec::new(); - - // Entry 1 - expired - wal_data.extend_from_slice(&1u64.to_be_bytes()); // index - wal_data.extend_from_slice(&1u64.to_be_bytes()); // term - wal_data.push(1u8); // Insert - wal_data.extend_from_slice(&4u64.to_be_bytes()); // key_len - wal_data.extend_from_slice(b"key1"); // key - wal_data.extend_from_slice(&6u64.to_be_bytes()); // value_len - wal_data.extend_from_slice(b"value1"); // value - wal_data.extend_from_slice(&expire_at_secs.to_be_bytes()); // expired timestamp - - // Entry 2 - expired - wal_data.extend_from_slice(&2u64.to_be_bytes()); // index - wal_data.extend_from_slice(&1u64.to_be_bytes()); // term - wal_data.push(1u8); // Insert - wal_data.extend_from_slice(&4u64.to_be_bytes()); // key_len - wal_data.extend_from_slice(b"key2"); // key - wal_data.extend_from_slice(&6u64.to_be_bytes()); // value_len - wal_data.extend_from_slice(b"value2"); // value - wal_data.extend_from_slice(&expire_at_secs.to_be_bytes()); // expired timestamp - - std::fs::write(&wal_path, wal_data).unwrap(); - - // Create new state machine to trigger replay - let sm = FileStateMachine::new(state_machine_path.clone()).await.unwrap(); - - // Both keys should be skipped (expired) - assert_eq!(sm.get(b"key1").unwrap(), None); - assert_eq!(sm.get(b"key2").unwrap(), None); - } - - /// Test: WAL replay with mixed expired and valid entries - #[tokio::test] - async fn test_wal_replay_mixed_expired_and_valid() { - let temp_dir = TempDir::new().unwrap(); - let state_machine_path = temp_dir.path().to_path_buf(); - - let now = std::time::SystemTime::now(); - let expired_time = now - std::time::Duration::from_secs(100); - let future_time = now + std::time::Duration::from_secs(3600); - - let expire_at_expired = - expired_time.duration_since(std::time::UNIX_EPOCH).unwrap().as_secs(); - let expire_at_future = future_time.duration_since(std::time::UNIX_EPOCH).unwrap().as_secs(); - - // Manually create WAL with mixed entries - let wal_path = state_machine_path.join("wal.log"); - let mut wal_data = Vec::new(); - - // Entry 1 - expired - wal_data.extend_from_slice(&1u64.to_be_bytes()); // index - wal_data.extend_from_slice(&1u64.to_be_bytes()); // term - wal_data.push(1u8); // Insert - wal_data.extend_from_slice(&11u64.to_be_bytes()); // key_len - wal_data.extend_from_slice(b"expired_key"); // key - wal_data.extend_from_slice(&6u64.to_be_bytes()); // value_len - wal_data.extend_from_slice(b"value1"); // value - wal_data.extend_from_slice(&expire_at_expired.to_be_bytes()); // expire_at - - // Entry 2 - valid with future TTL - wal_data.extend_from_slice(&2u64.to_be_bytes()); // index - wal_data.extend_from_slice(&1u64.to_be_bytes()); // term - wal_data.push(1u8); // Insert - wal_data.extend_from_slice(&9u64.to_be_bytes()); // key_len - wal_data.extend_from_slice(b"valid_key"); // key - wal_data.extend_from_slice(&6u64.to_be_bytes()); // value_len - wal_data.extend_from_slice(b"value2"); // value - wal_data.extend_from_slice(&expire_at_future.to_be_bytes()); // expire_at - - // Entry 3 - permanent (no TTL) - wal_data.extend_from_slice(&3u64.to_be_bytes()); // index - wal_data.extend_from_slice(&1u64.to_be_bytes()); // term - wal_data.push(1u8); // Insert - wal_data.extend_from_slice(&13u64.to_be_bytes()); // key_len - wal_data.extend_from_slice(b"permanent_key"); // key - wal_data.extend_from_slice(&6u64.to_be_bytes()); // value_len - wal_data.extend_from_slice(b"value3"); // value - wal_data.extend_from_slice(&0u64.to_be_bytes()); // no TTL - - std::fs::write(&wal_path, wal_data).unwrap(); - - // Create state machine to trigger replay - let sm = FileStateMachine::new(state_machine_path.clone()).await.unwrap(); - - // Expired key should not be loaded - assert_eq!(sm.get(b"expired_key").unwrap(), None); - - // Valid key with future TTL should be loaded - assert_eq!(sm.get(b"valid_key").unwrap(), Some(Bytes::from("value2"))); - - // Permanent key should be loaded - assert_eq!( - sm.get(b"permanent_key").unwrap(), - Some(Bytes::from("value3")) - ); - } - - /// Test: Background strategy manual cleanup - /// - /// Verifies that Background strategy can clean expired keys via manual trigger. - /// This simulates what the background worker task does periodically. - #[tokio::test] - async fn test_background_strategy_manual_cleanup() { - let temp_dir = TempDir::new().unwrap(); - let lease_config = d_engine_core::config::LeaseConfig { - cleanup_interval_ms: 1000, - max_cleanup_duration_ms: 50, - }; - let sm = - create_file_state_machine_with_lease(temp_dir.path().to_path_buf(), lease_config).await; - - // Insert 20 keys with 1 second TTL - for i in 0..20 { - let key = format!("bg_key_{}", i); - let entry = create_insert_entry(i + 1, 1, key.as_bytes(), b"value", 1); - sm.apply_chunk(&[entry]).await.unwrap(); - } - - // All keys should exist immediately - for i in 0..20 { - let key = format!("bg_key_{}", i); - assert_eq!(sm.get(key.as_bytes()).unwrap(), Some(Bytes::from("value"))); - } - - // Wait for expiration - tokio::time::sleep(Duration::from_secs(2)).await; - - // In Background mode, get() should NOT clean expired keys - // (they remain until background task runs) - for i in 0..20 { - let key = format!("bg_key_{}", i); - // Keys still exist because Background strategy doesn't check in get() - assert_eq!(sm.get(key.as_bytes()).unwrap(), Some(Bytes::from("value"))); - } - - // Manually trigger background cleanup (simulates what NodeBuilder's worker does) - let cleaned = sm.lease_background_cleanup().await.unwrap(); - assert_eq!( - cleaned.len(), - 20, - "Background cleanup should remove all 20 expired keys" - ); - - // Now all keys should be gone - for i in 0..20 { - let key = format!("bg_key_{}", i); - assert_eq!(sm.get(key.as_bytes()).unwrap(), None); - } - } - - /// Test: Background cleanup respects max_cleanup_duration_ms - /// - /// Verifies that cleanup cycle doesn't exceed configured time limit. - #[tokio::test] - async fn test_background_cleanup_respects_max_duration() { - let temp_dir = TempDir::new().unwrap(); - let lease_config = d_engine_core::config::LeaseConfig { - cleanup_interval_ms: 1000, - max_cleanup_duration_ms: 1, // Very short limit - }; - let sm = - create_file_state_machine_with_lease(temp_dir.path().to_path_buf(), lease_config).await; - - // Insert 100 keys with 1 second TTL (intentionally more than can be cleaned in 1ms) - for i in 0..100 { - let key = format!("duration_key_{}", i); - let entry = create_insert_entry(i + 1, 1, key.as_bytes(), b"value", 1); - sm.apply_chunk(&[entry]).await.unwrap(); - } - - // Wait for expiration - tokio::time::sleep(Duration::from_secs(2)).await; - - // Manually trigger cleanup with time limit - let start = std::time::Instant::now(); - let cleaned = sm.lease_background_cleanup().await.unwrap(); - let duration = start.elapsed(); - - // Should have cleaned some keys (partial cleanup due to time limit) - assert!( - !cleaned.is_empty(), - "Background cleanup should clean at least some expired keys" - ); - - // With max_cleanup_duration_ms=1, cleanup should be fast - // Allow some tolerance for test environment variance - assert!( - duration < Duration::from_millis(100), - "Cleanup should respect max duration (took {:?})", - duration - ); - } - - /// Fast path: cleanup is a no-op when no TTL keys have ever been registered. - /// - /// `has_lease_keys()` returns false → cleanup must return empty without - /// scanning storage. Regular (non-TTL) keys must be untouched. - #[tokio::test] - async fn test_background_cleanup_fast_path_no_ttl_keys_ever_registered() { - let temp_dir = TempDir::new().unwrap(); - let sm = create_file_state_machine_with_lease( - temp_dir.path().to_path_buf(), - d_engine_core::config::LeaseConfig::default(), - ) - .await; - - // Insert regular keys without TTL - for i in 0..5u64 { - let key = format!("plain_key_{i}"); - let entry = ApplyEntry { - index: i + 1, - term: 1, - command: Command::Insert { - key: Bytes::copy_from_slice(key.as_bytes()), - value: Bytes::from("value"), - ttl_secs: None, - }, - }; - sm.apply_chunk(&[entry]).await.unwrap(); - } - - // Cleanup must be a no-op: no TTL keys, nothing to scan or delete - let cleaned = sm.lease_background_cleanup().await.unwrap(); - assert!( - cleaned.is_empty(), - "cleanup must return empty when no TTL keys registered, got {:?}", - cleaned - ); - - // All regular keys must survive cleanup - for i in 0..5u64 { - let key = format!("plain_key_{i}"); - assert_eq!( - sm.get(key.as_bytes()).unwrap(), - Some(Bytes::from("value")), - "plain_key_{i} must not be touched by cleanup" - ); - } - } - - /// Fast path: cleanup is a no-op when TTL keys exist but none are expired. - /// - /// `has_lease_keys()` is true but `may_have_expired_keys()` is false → - /// cleanup must return empty and leave all keys intact. - #[tokio::test] - async fn test_background_cleanup_fast_path_unexpired_keys_preserved() { - let temp_dir = TempDir::new().unwrap(); - let sm = create_file_state_machine_with_lease( - temp_dir.path().to_path_buf(), - d_engine_core::config::LeaseConfig::default(), - ) - .await; - - // Insert keys with a very long TTL (3600s, will not expire during test) - for i in 0..5u64 { - let key = format!("ttl_key_{i}"); - let entry = create_insert_entry(i + 1, 1, key.as_bytes(), b"value", 3600); - sm.apply_chunk(&[entry]).await.unwrap(); - } - - // Cleanup must skip all keys: none are expired - let cleaned = sm.lease_background_cleanup().await.unwrap(); - assert!( - cleaned.is_empty(), - "cleanup must return empty when no keys are expired, got {:?}", - cleaned - ); - - // All TTL keys must still be readable - for i in 0..5u64 { - let key = format!("ttl_key_{i}"); - assert_eq!( - sm.get(key.as_bytes()).unwrap(), - Some(Bytes::from("value")), - "ttl_key_{i} must not be deleted by cleanup before expiry" - ); - } - } -} - -#[cfg(all(test, feature = "rocksdb"))] -mod rocksdb_state_machine_tests { - use std::time::Duration; - - use bytes::Bytes; - use d_engine_core::ApplyEntry; - use d_engine_core::Command; - use d_engine_core::StateMachine; - use tempfile::TempDir; - use tokio::time::sleep; - - use crate::storage::RocksDBStateMachine; - use crate::storage::RocksDBStorageEngine; - use crate::storage::RocksDBUnifiedEngine; - - /// Helper to create a RocksDBStateMachine with lease injected for testing. - /// Returns (storage, sm) — caller must keep storage alive alongside sm. - async fn create_rocksdb_state_machine_with_lease( - path: std::path::PathBuf, - lease_config: d_engine_core::config::LeaseConfig, - ) -> (RocksDBStorageEngine, RocksDBStateMachine) { - let (storage, mut sm) = RocksDBUnifiedEngine::open(&path).unwrap(); - let lease = std::sync::Arc::new(crate::storage::TtlLease::new(lease_config)); - sm.set_lease(lease); - sm.load_lease_data().await.unwrap(); - (storage, sm) - } - - fn create_insert_entry( - index: u64, - term: u64, - key: &[u8], - value: &[u8], - ttl_secs: u64, - ) -> ApplyEntry { - ApplyEntry { - index, - term, - command: Command::Insert { - key: Bytes::copy_from_slice(key), - value: Bytes::copy_from_slice(value), - ttl_secs: if ttl_secs > 0 { Some(ttl_secs) } else { None }, - }, - } - } - - fn create_delete_entry( - index: u64, - term: u64, - key: &[u8], - ) -> ApplyEntry { - ApplyEntry { - index, - term, - command: Command::Delete { - key: Bytes::copy_from_slice(key), - }, - } - } - - #[tokio::test] - async fn test_rocksdb_ttl_expiration_after_apply() { - let temp_dir = TempDir::new().unwrap(); - let ttl_config = d_engine_core::config::LeaseConfig { - cleanup_interval_ms: 1000, - max_cleanup_duration_ms: 1, - }; - let (_storage, sm) = - create_rocksdb_state_machine_with_lease(temp_dir.path().join("rocksdb"), ttl_config) - .await; - - // Insert key with 2 second TTL - let entry = create_insert_entry(1, 1, b"ttl_key", b"ttl_value", 2); - sm.apply_chunk(&[entry]).await.unwrap(); - - // Key should exist immediately - let value = sm.get(b"ttl_key").unwrap(); - assert_eq!(value, Some(Bytes::from("ttl_value"))); - - // Wait for expiration - sleep(Duration::from_secs(3)).await; - - // Manually trigger background cleanup - sm.lease_background_cleanup().await.unwrap(); - - // Key should be expired after cleanup - let value = sm.get(b"ttl_key").unwrap(); - assert_eq!(value, None); - } - - #[tokio::test] - #[ignore] // TODO: Fix RocksDB lock contention in snapshot restoration - async fn test_rocksdb_ttl_snapshot_persistence() { - let temp_dir = TempDir::new().unwrap(); - let ttl_config = d_engine_core::config::LeaseConfig { - cleanup_interval_ms: 1000, - max_cleanup_duration_ms: 1, - }; - let (_storage, sm) = create_rocksdb_state_machine_with_lease( - temp_dir.path().join("rocksdb"), - ttl_config.clone(), - ) - .await; - - // Insert key with 3600 second TTL (won't expire during test) - let entry = create_insert_entry(1, 1, b"persistent_key", b"persistent_value", 3600); - sm.apply_chunk(&[entry]).await.unwrap(); - - // Create snapshot - let snapshot_dir = temp_dir.path().join("snapshot"); - sm.generate_snapshot_data(snapshot_dir.clone(), d_engine_proto::common::LogId { - index: 1, - term: 1, - }) - .await - .unwrap(); - - // Verify TTL state file exists - let ttl_file = snapshot_dir.join("ttl_state.bin"); - assert!( - ttl_file.exists(), - "TTL state file should be created in snapshot" - ); - - // Create new state machine and restore snapshot - let temp_dir2 = TempDir::new().unwrap(); - let (_storage2, sm2) = - create_rocksdb_state_machine_with_lease(temp_dir2.path().join("rocksdb"), ttl_config) - .await; - - sm2.apply_snapshot_from_file( - &d_engine_proto::server::storage::SnapshotMetadata { - last_included: Some(d_engine_proto::common::LogId { index: 1, term: 1 }), - checksum: Bytes::new(), - }, - snapshot_dir, - ) - .await - .unwrap(); - - // TTL manager should be restored (we can't directly check the key in RocksDB - // since the checkpoint restoration is not fully implemented, but TTL state is restored) - } - - #[tokio::test] - async fn test_rocksdb_ttl_update_cancels_previous() { - let temp_dir = TempDir::new().unwrap(); - let ttl_config = d_engine_core::config::LeaseConfig { - cleanup_interval_ms: 1000, - max_cleanup_duration_ms: 1, - }; - let (_storage, sm) = - create_rocksdb_state_machine_with_lease(temp_dir.path().join("rocksdb"), ttl_config) - .await; - - // Insert key with 2 second TTL - let entry1 = create_insert_entry(1, 1, b"update_key", b"value1", 2); - sm.apply_chunk(&[entry1]).await.unwrap(); - - // Immediately update with longer TTL - let entry2 = create_insert_entry(2, 1, b"update_key", b"value2", 10); - sm.apply_chunk(&[entry2]).await.unwrap(); - - // Wait past original TTL - sleep(Duration::from_secs(3)).await; - - // Trigger expiration check - let entry3 = create_insert_entry(3, 1, b"trigger", b"trigger", 0); - sm.apply_chunk(&[entry3]).await.unwrap(); - - // Key should still exist (new TTL not expired) - let value = sm.get(b"update_key").unwrap(); - assert_eq!(value, Some(Bytes::from("value2"))); - } - - #[tokio::test] - async fn test_rocksdb_ttl_delete_unregisters() { - let temp_dir = TempDir::new().unwrap(); - let ttl_config = d_engine_core::config::LeaseConfig { - cleanup_interval_ms: 1000, - max_cleanup_duration_ms: 1, - }; - let (_storage, sm) = - create_rocksdb_state_machine_with_lease(temp_dir.path().join("rocksdb"), ttl_config) - .await; - - // Insert key with TTL - let entry1 = create_insert_entry(1, 1, b"delete_key", b"delete_value", 3600); - sm.apply_chunk(&[entry1]).await.unwrap(); - - // Delete the key - let entry2 = create_delete_entry(2, 1, b"delete_key"); - sm.apply_chunk(&[entry2]).await.unwrap(); - - // Key should not exist - let value = sm.get(b"delete_key").unwrap(); - assert_eq!(value, None); - - // Even after waiting, no expiration should occur (TTL was unregistered) - sleep(Duration::from_secs(2)).await; - let entry3 = create_insert_entry(3, 1, b"trigger", b"trigger", 0); - sm.apply_chunk(&[entry3]).await.unwrap(); - } - - #[tokio::test] - async fn test_rocksdb_multiple_keys_with_different_ttls() { - let temp_dir = TempDir::new().unwrap(); - let ttl_config = d_engine_core::config::LeaseConfig { - cleanup_interval_ms: 1000, - max_cleanup_duration_ms: 1, - }; - let (_storage, sm) = - create_rocksdb_state_machine_with_lease(temp_dir.path().join("rocksdb"), ttl_config) - .await; - - // Insert multiple keys with different TTLs - let entry1 = create_insert_entry(1, 1, b"key_1sec", b"value1", 1); - let entry2 = create_insert_entry(2, 1, b"key_5sec", b"value2", 5); - let entry3 = create_insert_entry(3, 1, b"key_no_ttl", b"value3", 0); - - sm.apply_chunk(&[entry1, entry2, entry3]).await.unwrap(); - - // All keys should exist initially - assert_eq!(sm.get(b"key_1sec").unwrap(), Some(Bytes::from("value1"))); - assert_eq!(sm.get(b"key_5sec").unwrap(), Some(Bytes::from("value2"))); - assert_eq!(sm.get(b"key_no_ttl").unwrap(), Some(Bytes::from("value3"))); - - // Wait 2 seconds - sleep(Duration::from_secs(2)).await; - - // Manually trigger background cleanup - sm.lease_background_cleanup().await.unwrap(); - - // key_1sec should be expired - assert_eq!(sm.get(b"key_1sec").unwrap(), None); - // key_5sec and key_no_ttl should still exist - assert_eq!(sm.get(b"key_5sec").unwrap(), Some(Bytes::from("value2"))); - assert_eq!(sm.get(b"key_no_ttl").unwrap(), Some(Bytes::from("value3"))); - - // Wait another 4 seconds (total 6 seconds) - sleep(Duration::from_secs(4)).await; - - // Manually trigger background cleanup again - sm.lease_background_cleanup().await.unwrap(); - - // key_5sec should now be expired too - assert_eq!(sm.get(b"key_5sec").unwrap(), None); - // key_no_ttl should still exist - assert_eq!(sm.get(b"key_no_ttl").unwrap(), Some(Bytes::from("value3"))); - } - - #[tokio::test] - async fn test_rocksdb_ttl_persistence_across_restart() { - let temp_dir = TempDir::new().unwrap(); - let state_machine_path = temp_dir.path().join("rocksdb"); - let ttl_config = d_engine_core::config::LeaseConfig { - cleanup_interval_ms: 1000, - max_cleanup_duration_ms: 1, - }; - - // Phase 1: Create state machine and insert keys with TTL - { - let (_storage, sm) = create_rocksdb_state_machine_with_lease( - state_machine_path.clone(), - ttl_config.clone(), - ) - .await; - - let entry1 = create_insert_entry(1, 1, b"short_ttl_key", b"value1", 2); - let entry2 = create_insert_entry(2, 1, b"long_ttl_key", b"value2", 3600); - let entry3 = create_insert_entry(3, 1, b"no_ttl_key", b"value3", 0); - - sm.apply_chunk(&[entry1, entry2, entry3]).await.unwrap(); - - // Verify all keys exist - assert_eq!( - sm.get(b"short_ttl_key").unwrap(), - Some(Bytes::from("value1")) - ); - assert_eq!( - sm.get(b"long_ttl_key").unwrap(), - Some(Bytes::from("value2")) - ); - assert_eq!(sm.get(b"no_ttl_key").unwrap(), Some(Bytes::from("value3"))); - - // Gracefully close state machine to persist TTL data - sm.close().unwrap(); - // State machine drops here - } - - // Phase 2: Restart - create new state machine from same directory - { - let (_storage, sm) = - create_rocksdb_state_machine_with_lease(state_machine_path.clone(), ttl_config) - .await; - - // Verify all keys still exist after restart - assert_eq!( - sm.get(b"short_ttl_key").unwrap(), - Some(Bytes::from("value1")) - ); - assert_eq!( - sm.get(b"long_ttl_key").unwrap(), - Some(Bytes::from("value2")) - ); - assert_eq!(sm.get(b"no_ttl_key").unwrap(), Some(Bytes::from("value3"))); - - // Wait for short TTL to expire - sleep(Duration::from_secs(3)).await; - - // Manually trigger background cleanup - sm.lease_background_cleanup().await.unwrap(); - - // short_ttl_key should be expired - assert_eq!(sm.get(b"short_ttl_key").unwrap(), None); - // long_ttl_key should still exist - assert_eq!( - sm.get(b"long_ttl_key").unwrap(), - Some(Bytes::from("value2")) - ); - // no_ttl_key should still exist - assert_eq!(sm.get(b"no_ttl_key").unwrap(), Some(Bytes::from("value3"))); - } - } - - #[tokio::test] - async fn test_rocksdb_reset_clears_ttl_manager() { - let temp_dir = TempDir::new().unwrap(); - let ttl_config = d_engine_core::config::LeaseConfig { - cleanup_interval_ms: 1000, - max_cleanup_duration_ms: 1, - }; - let (_storage, sm) = - create_rocksdb_state_machine_with_lease(temp_dir.path().join("rocksdb"), ttl_config) - .await; - - // Insert keys with TTL - let entry1 = create_insert_entry(1, 1, b"key1", b"value1", 3600); - let entry2 = create_insert_entry(2, 1, b"key2", b"value2", 7200); - sm.apply_chunk(&[entry1, entry2]).await.unwrap(); - - // Reset the state machine - sm.reset().await.unwrap(); - - // All data should be cleared - assert_eq!(sm.get(b"key1").unwrap(), None); - assert_eq!(sm.get(b"key2").unwrap(), None); - assert_eq!(sm.len(), 0); - } - - /// Fast path: cleanup is a no-op when no TTL keys have ever been registered. - /// - /// `has_lease_keys()` returns false → cleanup must return empty without - /// scanning storage. Regular (non-TTL) keys must be untouched. - #[tokio::test] - async fn test_rocksdb_background_cleanup_fast_path_no_ttl_keys_ever_registered() { - let temp_dir = TempDir::new().unwrap(); - let (_storage, sm) = create_rocksdb_state_machine_with_lease( - temp_dir.path().join("rocksdb"), - d_engine_core::config::LeaseConfig::default(), - ) - .await; - - // Insert regular keys without TTL - for i in 0..5u64 { - let key = format!("plain_key_{i}"); - let entry = ApplyEntry { - index: i + 1, - term: 1, - command: Command::Insert { - key: Bytes::copy_from_slice(key.as_bytes()), - value: Bytes::from("value"), - ttl_secs: None, - }, - }; - sm.apply_chunk(&[entry]).await.unwrap(); - } - - // Cleanup must be a no-op: no TTL keys, nothing to scan or delete - let cleaned = sm.lease_background_cleanup().await.unwrap(); - assert!( - cleaned.is_empty(), - "cleanup must return empty when no TTL keys registered, got {:?}", - cleaned - ); - - // All regular keys must survive cleanup - for i in 0..5u64 { - let key = format!("plain_key_{i}"); - assert_eq!( - sm.get(key.as_bytes()).unwrap(), - Some(Bytes::from("value")), - "plain_key_{i} must not be touched by cleanup" - ); - } - } - - /// Fast path: cleanup is a no-op when TTL keys exist but none are expired. - /// - /// `has_lease_keys()` is true but `may_have_expired_keys()` is false → - /// cleanup must return empty and leave all keys intact. - #[tokio::test] - async fn test_rocksdb_background_cleanup_fast_path_unexpired_keys_preserved() { - let temp_dir = TempDir::new().unwrap(); - let (_storage, sm) = create_rocksdb_state_machine_with_lease( - temp_dir.path().join("rocksdb"), - d_engine_core::config::LeaseConfig::default(), - ) - .await; - - // Insert keys with a very long TTL (3600s, will not expire during test) - for i in 0..5u64 { - let key = format!("ttl_key_{i}"); - let entry = create_insert_entry(i + 1, 1, key.as_bytes(), b"value", 3600); - sm.apply_chunk(&[entry]).await.unwrap(); - } - - // Cleanup must skip all keys: none are expired - let cleaned = sm.lease_background_cleanup().await.unwrap(); - assert!( - cleaned.is_empty(), - "cleanup must return empty when no keys are expired, got {:?}", - cleaned - ); - - // All TTL keys must still be readable - for i in 0..5u64 { - let key = format!("ttl_key_{i}"); - assert_eq!( - sm.get(key.as_bytes()).unwrap(), - Some(Bytes::from("value")), - "ttl_key_{i} must not be deleted by cleanup before expiry" - ); - } - } - - // #[tokio::test] - // async fn test_rocksdb_passive_deletion_on_get() { - // let temp_dir = TempDir::new().unwrap(); - // let ttl_config = d_engine_core::config::LeaseConfig { - // cleanup_interval_ms: 1000, - // max_cleanup_duration_ms: 1, - // }; - // let sm = - // create_rocksdb_state_machine_with_lease(temp_dir.path().join("rocksdb"), - // ttl_config) .await; - // - // // Insert key with 1 second TTL - // let entry = create_insert_entry(1, 1, b"passive_key", b"passive_value", 1); - // sm.apply_chunk(&[entry]).await.unwrap(); - // - // // Key should exist immediately - // let value = sm.get(b"passive_key").unwrap(); - // assert_eq!(value, Some(Bytes::from("passive_value"))); - // - // // Wait for expiration - // sleep(Duration::from_secs(2)).await; - // - // // Passive deletion: key should be deleted on get() without apply_chunk - // let value = sm.get(b"passive_key").unwrap(); - // assert_eq!(value, None); - // - // // Verify key is truly deleted (not just hidden) - // let value = sm.get(b"passive_key").unwrap(); - // assert_eq!(value, None); - // // } - // // - // // #[tokio::test] - // async fn test_rocksdb_lazy_activation_no_ttl_overhead() { - // let temp_dir = TempDir::new().unwrap(); - // let mut ttl_config = d_engine_core::config::LeaseConfig::default(); - // ttl_config.cleanup_strategy = "piggyback".to_string(); - // let sm = - // create_rocksdb_state_machine_with_lease(temp_dir.path().join("rocksdb"), - // ttl_config) .await; - // - // // Insert keys WITHOUT TTL - // for i in 0..10 { - // let key = format!("no_ttl_key_{i}"); - // let entry = create_insert_entry(i + 1, 1, key.as_bytes(), b"value", 0); - // sm.apply_chunk(&[entry]).await.unwrap(); - // } - // - // // Apply 150 more entries (should trigger piggyback cleanup check) - // for i in 10..160 { - // let entry = create_insert_entry(i + 1, 1, b"dummy", b"dummy", 0); - // sm.apply_chunk(&[entry]).await.unwrap(); - // } - // - // // All keys should still exist (no TTL, lazy activation should skip cleanup) - // for i in 0..10 { - // let key = format!("no_ttl_key_{i}"); - // let value = sm.get(key.as_bytes()).unwrap(); - // assert_eq!(value, Some(Bytes::from("value"))); - // } - // } -} diff --git a/d-engine-server/src/storage/lease_unit_test.rs b/d-engine-server/src/storage/lease_unit_test.rs deleted file mode 100644 index 550ea720..00000000 --- a/d-engine-server/src/storage/lease_unit_test.rs +++ /dev/null @@ -1,359 +0,0 @@ -//! Comprehensive unit tests for TtlLease implementation -//! -//! Tests cover: -//! - Basic registration, expiration, and cleanup -//! - Update and replacement scenarios -//! - Snapshot serialization and deserialization -//! - Piggyback cleanup configuration and behavior -//! - Edge cases and error conditions - -use std::thread::sleep; -use std::time::Duration; -use std::time::SystemTime; - -use bytes::Bytes; -use d_engine_core::Lease; - -use crate::storage::lease::TtlLease; - -fn default_config() -> d_engine_core::config::LeaseConfig { - d_engine_core::config::LeaseConfig { - cleanup_interval_ms: 1000, - max_cleanup_duration_ms: 1, - } -} - -// ============================================================================ -// Basic Registration and Expiration Tests -// ============================================================================ - -#[test] -fn test_register_and_is_expired() { - let lease = TtlLease::new(default_config()); - - lease.register(Bytes::from("key1"), 1); - assert!(!lease.is_expired(b"key1")); - - sleep(Duration::from_secs(2)); - assert!(lease.is_expired(b"key1")); -} - -#[test] -fn test_unregister() { - let lease = TtlLease::new(default_config()); - - lease.register(Bytes::from("key1"), 10); - assert_eq!(lease.len(), 1); - - lease.unregister(b"key1"); - assert_eq!(lease.len(), 0); - assert!(!lease.is_expired(b"key1")); -} - -#[test] -fn test_unregister_nonexistent_key() { - let lease = TtlLease::new(default_config()); - - // Should not panic on unregistering non-existent key - lease.unregister(b"nonexistent"); - assert_eq!(lease.len(), 0); -} - -#[test] -fn test_get_expired_keys() { - let lease = TtlLease::new(default_config()); - - lease.register(Bytes::from("key1"), 1); - lease.register(Bytes::from("key2"), 1); - - sleep(Duration::from_secs(2)); - - let expired = lease.get_expired_keys(SystemTime::now()); - assert_eq!(expired.len(), 2); - assert_eq!(lease.len(), 0); -} - -#[test] -fn test_is_empty() { - let lease = TtlLease::new(default_config()); - assert!(lease.is_empty()); - - lease.register(Bytes::from("key1"), 10); - assert!(!lease.is_empty()); - - lease.unregister(b"key1"); - assert!(lease.is_empty()); -} - -// ============================================================================ -// Has Lease Keys and Flags Tests -// ============================================================================ - -#[test] -fn test_has_lease_keys() { - let lease = TtlLease::new(default_config()); - assert!(!lease.has_lease_keys()); - - lease.register(Bytes::from("key1"), 10); - assert!(lease.has_lease_keys()); - - lease.unregister(b"key1"); - assert!(lease.has_lease_keys()); // Still true (once set, stays set) -} - -#[test] -fn test_may_have_expired_keys_false_when_no_keys() { - let lease = TtlLease::new(default_config()); - - // No keys registered, should return false immediately - assert!(!lease.may_have_expired_keys(SystemTime::now())); -} - -#[test] -fn test_may_have_expired_keys_false_when_all_valid() { - let lease = TtlLease::new(default_config()); - - lease.register(Bytes::from("key1"), 3600); - lease.register(Bytes::from("key2"), 7200); - - // All keys have future expiration, should return false - assert!(!lease.may_have_expired_keys(SystemTime::now())); -} - -#[test] -fn test_may_have_expired_keys_true_when_has_expired() { - let lease = TtlLease::new(default_config()); - - lease.register(Bytes::from("key1"), 1); - sleep(Duration::from_secs(2)); - - // Has expired keys, should return true - assert!(lease.may_have_expired_keys(SystemTime::now())); -} - -// ============================================================================ -// Get Expiration Tests -// ============================================================================ - -#[test] -fn test_get_expiration() { - let lease = TtlLease::new(default_config()); - - lease.register(Bytes::from("key1"), 3600); - - let expiration = lease.get_expiration(b"key1"); - assert!(expiration.is_some()); - assert!(expiration.unwrap() > SystemTime::now()); -} - -#[test] -fn test_get_expiration_not_found() { - let lease = TtlLease::new(default_config()); - - let expiration = lease.get_expiration(b"nonexistent"); - assert!(expiration.is_none()); -} - -// ============================================================================ -// Update and Replacement Tests -// ============================================================================ - -#[test] -fn test_register_updates_existing_key() { - let lease = TtlLease::new(default_config()); - - // Register with 10s TTL - lease.register(Bytes::from("key1"), 10); - assert_eq!(lease.len(), 1); - - let first_expiry = lease.get_expiration(b"key1").unwrap(); - - // Register same key with 20s TTL (should update) - sleep(Duration::from_millis(100)); - lease.register(Bytes::from("key1"), 20); - - let second_expiry = lease.get_expiration(b"key1").unwrap(); - - // New expiry should be later than first - assert!(second_expiry > first_expiry); - assert_eq!(lease.len(), 1); // Still only one key -} - -#[test] -fn test_multiple_keys_same_expiration() { - let lease = TtlLease::new(default_config()); - - let ttl = 5u64; - lease.register(Bytes::from("key1"), ttl); - lease.register(Bytes::from("key2"), ttl); - lease.register(Bytes::from("key3"), ttl); - - assert_eq!(lease.len(), 3); - - sleep(Duration::from_secs(ttl + 1)); - - let expired = lease.get_expired_keys(SystemTime::now()); - assert_eq!(expired.len(), 3); - assert_eq!(lease.len(), 0); -} - -#[test] -fn test_mixed_expired_and_valid_keys() { - let lease = TtlLease::new(default_config()); - - lease.register(Bytes::from("expire_soon"), 1); - lease.register(Bytes::from("expire_later"), 3600); - - sleep(Duration::from_secs(2)); - - let expired = lease.get_expired_keys(SystemTime::now()); - assert_eq!(expired.len(), 1); - assert_eq!(expired[0], Bytes::from("expire_soon")); - - // expire_later should still be there - assert_eq!(lease.len(), 1); - assert!(!lease.is_expired(b"expire_later")); -} - -// ============================================================================ -// Snapshot Tests -// ============================================================================ - -#[test] -fn test_snapshot_roundtrip() { - let config = default_config(); - let lease1 = TtlLease::new(config.clone()); - - lease1.register(Bytes::from("key1"), 3600); - lease1.register(Bytes::from("key2"), 7200); - - let snapshot = lease1.to_snapshot(); - let lease2 = TtlLease::from_snapshot(&snapshot, config); - - assert_eq!(lease2.len(), 2); - assert!(!lease2.is_expired(b"key1")); - assert!(!lease2.is_expired(b"key2")); -} - -#[test] -fn test_snapshot_roundtrip_filters_expired_keys() { - let config = default_config(); - let lease1 = TtlLease::new(config.clone()); - - lease1.register(Bytes::from("valid_key"), 3600); - lease1.register(Bytes::from("expired_key"), 1); - - sleep(Duration::from_secs(2)); - - let snapshot = lease1.to_snapshot(); - - // from_snapshot should filter out expired keys - let lease2 = TtlLease::from_snapshot(&snapshot, config); - - // Only valid_key should remain - assert_eq!(lease2.len(), 1); - assert!(!lease2.is_expired(b"valid_key")); -} - -#[test] -fn test_from_snapshot_with_invalid_data() { - let config = default_config(); - - // from_snapshot with invalid data should return empty lease - let lease = TtlLease::from_snapshot(&[0xFF, 0xFE], config); - assert_eq!(lease.len(), 0); - assert!(!lease.has_lease_keys()); -} - -#[test] -fn test_reload() { - let lease = TtlLease::new(default_config()); - - lease.register(Bytes::from("key1"), 3600); - let snapshot = lease.to_snapshot(); - - lease.register(Bytes::from("key2"), 100); - assert_eq!(lease.len(), 2); - - lease.reload(&snapshot).unwrap(); - assert_eq!(lease.len(), 1); - assert!(!lease.is_expired(b"key1")); - assert!(!lease.is_expired(b"key2")); -} - -#[test] -fn test_reload_invalid_data() { - let lease = TtlLease::new(default_config()); - - lease.register(Bytes::from("key1"), 3600); - assert_eq!(lease.len(), 1); - - // Reload with invalid data should fail gracefully - // Note: reload fails during deserialization, so it returns error - // and doesn't proceed to clear/rebuild (state remains unchanged) - let result = lease.reload(&[0xFF, 0xFE, 0xFD]); - assert!(result.is_err()); - - // State remains unchanged because deserialization failed before clear - assert_eq!(lease.len(), 1); - assert!(!lease.is_expired(b"key1")); -} - -#[test] -fn test_reload_clears_apply_counter() { - let config = d_engine_core::config::LeaseConfig { - cleanup_interval_ms: 1000, - max_cleanup_duration_ms: 1, - }; - - let lease = TtlLease::new(config.clone()); - - lease.register(Bytes::from("key1"), 3600); - - // Simulate some apply operations - let _ = lease.on_apply(); - let _ = lease.on_apply(); - let _ = lease.on_apply(); - - let snapshot = lease.to_snapshot(); - - // After reload, apply_counter should be reset to 0 - lease.reload(&snapshot).unwrap(); - - // Next on_apply should not trigger cleanup (counter starts at 0) - let result = lease.on_apply(); - assert!(result.is_empty()); -} - -// ============================================================================ -// Piggyback Cleanup Tests -// ============================================================================ - -#[test] -fn test_on_apply_piggyback_removed() { - let config = default_config(); - - let lease = TtlLease::new(config); - lease.register(Bytes::from("key1"), 1); - - sleep(Duration::from_secs(2)); - - // on_apply() is now deprecated and always returns empty - // Cleanup is handled by Lazy (in get) or Background (async task) - let result = lease.on_apply(); - assert!(result.is_empty()); - - // Expired key should still be in lease (on_apply no longer cleans up) - assert_eq!(lease.len(), 1); -} - -#[test] -fn test_custom_cleanup_interval() { - let config = d_engine_core::config::LeaseConfig { - cleanup_interval_ms: 5000, - max_cleanup_duration_ms: 1, - }; - - let lease = TtlLease::new(config); - assert_eq!(lease.config().cleanup_interval_ms, 5000); -} diff --git a/d-engine-server/src/storage/mod.rs b/d-engine-server/src/storage/mod.rs index 66e929a7..80863a68 100644 --- a/d-engine-server/src/storage/mod.rs +++ b/d-engine-server/src/storage/mod.rs @@ -9,7 +9,6 @@ //! - Providing an abstraction layer (`StorageEngine`) for persistence. //! - Supporting in-memory buffering and disk-backed storage (e.g., via Sled). //! - Coordinating state machine application and snapshot lifecycle. -//! - Managing key expiration through lease-based lifecycle management. //! //! This module is designed so developers can easily implement custom //! storage backends without changing the Raft protocol logic. @@ -19,14 +18,4 @@ /// This module contains pluggable storage backends including file-based /// and RocksDB-based state machines that implement the `StateMachine` trait. pub mod adaptors; -mod lease; - pub use adaptors::*; -// Re-export Lease trait from core for convenience -pub use d_engine_core::Lease; -pub use lease::TtlLease; - -#[cfg(test)] -mod lease_integration_test; -#[cfg(test)] -mod lease_unit_test; diff --git a/d-engine-server/src/test_utils/integration/mod.rs b/d-engine-server/src/test_utils/integration/mod.rs index 0a872a04..061798da 100644 --- a/d-engine-server/src/test_utils/integration/mod.rs +++ b/d-engine-server/src/test_utils/integration/mod.rs @@ -314,7 +314,6 @@ pub(crate) async fn insert_state_machine( command: Command::Insert { key: Bytes::from(id.to_be_bytes().to_vec()), value: Bytes::from(id.to_be_bytes().to_vec()), - ttl_secs: None, }, }); } diff --git a/d-engine-server/src/test_utils/mock/mock_node_builder.rs b/d-engine-server/src/test_utils/mock/mock_node_builder.rs index 2100d39e..71418506 100644 --- a/d-engine-server/src/test_utils/mock/mock_node_builder.rs +++ b/d-engine-server/src/test_utils/mock/mock_node_builder.rs @@ -365,7 +365,6 @@ impl MockBuilder { sm_worker, rpc_server_handle: std::sync::Mutex::new(None), read_actor_handle: std::sync::Mutex::new(None), - _lease_cleanup_handle: None, shutdown_signal, read_lease, _data_dir: data_dir, @@ -424,7 +423,6 @@ impl MockBuilder { sm_worker, rpc_server_handle: std::sync::Mutex::new(None), read_actor_handle: std::sync::Mutex::new(None), - _lease_cleanup_handle: None, shutdown_signal: shutdown.clone(), read_lease, _data_dir: data_dir, diff --git a/d-engine-server/tests/embedded_client/embedded_client_operations.rs b/d-engine-server/tests/embedded_client/embedded_client_operations.rs index 7e74ac02..b2112371 100644 --- a/d-engine-server/tests/embedded_client/embedded_client_operations.rs +++ b/d-engine-server/tests/embedded_client/embedded_client_operations.rs @@ -8,40 +8,11 @@ use std::time::Duration; use bytes::Bytes; -use d_engine_core::ClientApi; use d_engine_server::DefaultEmbeddedEngine; use tempfile::TempDir; use crate::common::get_available_ports; -/// Helper to create a test DefaultEmbeddedEngine without any lease config section. -/// -/// Used by regression tests that must verify behaviour with default (zero-config) -/// settings — notably #398 where omitting [raft.state_machine.lease] previously -/// caused a fatal crash when put_with_ttl was called. -async fn create_test_engine_default_config(test_name: &str) -> (DefaultEmbeddedEngine, TempDir) { - let temp_dir = tempfile::tempdir().expect("Failed to create temp dir"); - let db_path = temp_dir.path().join(test_name); - let config_path = temp_dir.path().join("d-engine.toml"); - let mut port_guard = get_available_ports(1).await; - port_guard.release_listeners(); - let port = port_guard.as_slice()[0]; - let config_content = format!( - r#" -[cluster] -listen_address = "127.0.0.1:{}" -single_node = true -"#, - port - ); - std::fs::write(&config_path, config_content).expect("Failed to write config"); - let engine = DefaultEmbeddedEngine::start_with(&db_path, config_path.to_str().unwrap()) - .await - .expect("Failed to start engine"); - engine.wait_ready(Duration::from_secs(5)).await.expect("Engine not ready"); - (engine, temp_dir) -} - /// Helper to create a test EmbeddedEngine async fn create_test_engine(test_name: &str) -> (DefaultEmbeddedEngine, TempDir) { let temp_dir = tempfile::tempdir().expect("Failed to create temp dir"); @@ -533,80 +504,6 @@ async fn test_concurrent_write_and_linearizable_read() { println!("✅ All 100 concurrent linearizable reads saw their own writes"); } -// ============================================================================= -// put_with_ttl -// ============================================================================= - -/// Verifies that put_with_ttl stores a value that is immediately readable. -/// -/// TTL expiry is not tested here (requires real time passage); this test -/// focuses on the write path: the entry must be committed and readable -/// via linearizable read immediately after the call returns. -#[tokio::test] -async fn test_put_with_ttl_readable_immediately() { - let (engine, _temp_dir) = create_test_engine("put_with_ttl").await; - let client = engine.client(); - - client - .put_with_ttl(b"ttl-key", b"ttl-value", 60) - .await - .expect("put_with_ttl should succeed"); - - let value = client - .get_linearizable(b"ttl-key") - .await - .expect("get_linearizable should succeed"); - - assert_eq!( - value, - Some(Bytes::from("ttl-value")), - "value written with TTL must be readable immediately after write completes" - ); -} - -/// Regression test for #398. -/// -/// put_with_ttl must succeed when no [raft.state_machine.lease] section is present -/// in the config (i.e. default/zero-config). Previously this caused a fatal crash: -/// the state machine returned an error which the SM Worker mapped to -/// InternalEvent::FatalError, shutting down the entire node. -/// -/// Verification steps: -/// 1. Engine starts with default config (no lease section) -/// 2. put_with_ttl returns Ok(()) — not a crash, not an error -/// 3. The written value is readable — entry was applied correctly -/// 4. A subsequent put/get succeeds — node is still alive after the TTL write -#[tokio::test] -async fn test_put_with_ttl_succeeds_with_default_config() { - let (engine, _temp_dir) = - create_test_engine_default_config("put_with_ttl_default_config").await; - let client = engine.client(); - - // Must not crash the node, must return Ok - client - .put_with_ttl(b"ttl-key", b"ttl-value", 30) - .await - .expect("put_with_ttl must succeed with default config (#398 regression)"); - - // Value must be readable immediately (entry was applied) - let value = client - .get_linearizable(b"ttl-key") - .await - .expect("get after put_with_ttl should succeed"); - assert_eq!(value, Some(Bytes::from("ttl-value"))); - - // Node must still be alive — a regular put/get must work after the TTL write - client - .put(b"probe", b"alive") - .await - .expect("node must still be alive after put_with_ttl (#398 regression)"); - let probe = client - .get_linearizable(b"probe") - .await - .expect("probe read should succeed after put_with_ttl"); - assert_eq!(probe, Some(Bytes::from("alive"))); -} - // ============================================================================= // get_with_consistency (LeaseRead) // ============================================================================= diff --git a/d-engine-server/tests/embedded_client/mod.rs b/d-engine-server/tests/embedded_client/mod.rs index 8044c44f..a0c47f10 100644 --- a/d-engine-server/tests/embedded_client/mod.rs +++ b/d-engine-server/tests/embedded_client/mod.rs @@ -6,7 +6,7 @@ //! //! ## Test Files //! -//! - `embedded_client_operations.rs` - EmbeddedClient API testing: put, put_with_ttl, get +//! - `embedded_client_operations.rs` - EmbeddedClient API testing: put, get //! (eventual/linearizable/lease), get_multi alignment, delete, concurrent ops, watch (put/delete //! events, key isolation) diff --git a/d-engine-server/tests/reliability/data_dir_lock_released_after_storage_init_failure_embedded.rs b/d-engine-server/tests/reliability/data_dir_lock_released_after_storage_init_failure_embedded.rs index 1eb1111e..a01230bc 100644 --- a/d-engine-server/tests/reliability/data_dir_lock_released_after_storage_init_failure_embedded.rs +++ b/d-engine-server/tests/reliability/data_dir_lock_released_after_storage_init_failure_embedded.rs @@ -11,7 +11,7 @@ use crate::common::get_available_ports; /// directory validation (which normally creates the subdirectory RocksDB /// needs) fails on it, since a file can't be written into like a directory. /// This failure happens after the lock is acquired but before RocksDB is -/// ever touched, exactly the window the expert review flagged. +/// ever touched, which is the window where a leaked lock would go unnoticed. #[tokio::test] async fn test_lock_released_after_storage_init_failure() { let temp_dir = tempfile::tempdir().unwrap(); diff --git a/d-engine/src/docs/client_guide/read-consistency.md b/d-engine/src/docs/client_guide/read-consistency.md index 8abe9386..50ee0153 100644 --- a/d-engine/src/docs/client_guide/read-consistency.md +++ b/d-engine/src/docs/client_guide/read-consistency.md @@ -132,7 +132,7 @@ allow_client_override = true # Allow client to override (default: true _Measured on 3-node cluster, AWS same-region_ -**Why 0 RTT for Eventual/LeaseRead (v0.2.5+)**: These policies are served by a dedicated ReadActor that reads directly from the state machine — bypassing the Raft command channel entirely. The leader's read lease is validated atomically; if it expires or the node steps down, ReadActor falls back to the Raft path automatically. +**Why 0 RTT for Eventual/LeaseRead (v0.3.0+)**: These policies are served by a dedicated ReadActor that reads directly from the state machine — bypassing the Raft command channel entirely. The leader's read lease is validated atomically; if it expires or the node steps down, ReadActor falls back to the Raft path automatically. **Important**: All policies return **correct** committed data. Difference is whether you get the **latest** or **slightly older** committed state. diff --git a/d-engine/src/docs/overview.md b/d-engine/src/docs/overview.md index d974c60c..3ece9784 100644 --- a/d-engine/src/docs/overview.md +++ b/d-engine/src/docs/overview.md @@ -57,7 +57,6 @@ Both expose identical methods — pick your mode, the API below applies to both. | Method | Description | | ------------------------------------ | ---------------------------------------------- | | [`EmbeddedClient::put`] | Write a key-value pair | -| [`EmbeddedClient::put_with_ttl`] | Write with expiry (key auto-expires after TTL) | | [`EmbeddedClient::delete`] | Delete a key | | [`EmbeddedClient::compare_and_swap`] | Atomic CAS | diff --git a/d-engine/src/docs/performance/throughput-optimization-guide.md b/d-engine/src/docs/performance/throughput-optimization-guide.md index 8f1ffb98..531a64ec 100644 --- a/d-engine/src/docs/performance/throughput-optimization-guide.md +++ b/d-engine/src/docs/performance/throughput-optimization-guide.md @@ -56,7 +56,7 @@ max_batch_size = 200 # default, suitable for most deployments > **Rule of thumb**: For embedded mode, optimal `max_batch_size ≈ concurrent_client_count`. For standalone, keep at 200 unless profiling shows cmd_rx consistently saturated. -## Read Fast Path Tuning (v0.2.5+) +## Read Fast Path Tuning (v0.3.0+) `Eventual` and `LeaseRead` bypass `cmd_rx` entirely — they never touch `max_batch_size` or `time_threshold_ms`. Two dedicated knobs: diff --git a/d-engine/src/docs/quick-start-5min.md b/d-engine/src/docs/quick-start-5min.md index ed4da8b2..c4ee9ab4 100644 --- a/d-engine/src/docs/quick-start-5min.md +++ b/d-engine/src/docs/quick-start-5min.md @@ -176,7 +176,7 @@ No gRPC overhead - direct function calls to embedded Raft core. - `get_eventual(key)` - Fastest local read (may be slightly stale) - `delete(key)` - Delete key -See docs for TTL, multi-key operations, and advanced consistency control. +See docs for multi-key operations and advanced consistency control. --- @@ -256,7 +256,7 @@ client.get_eventual(key: impl Into>) -> Result> client.delete(key: Vec) -> Result ``` -See [complete API documentation](https://docs.rs/d-engine/latest/d_engine/prelude/struct.EmbeddedClient.html) for all available methods including TTL operations, multi-key reads, and consistency control. +See [complete API documentation](https://docs.rs/d-engine/latest/d_engine/prelude/struct.EmbeddedClient.html) for all available methods including multi-key reads and consistency control. --- diff --git a/d-engine/src/docs/server_guide/customize-state-machine.md b/d-engine/src/docs/server_guide/customize-state-machine.md index d3815e5d..255b68a7 100644 --- a/d-engine/src/docs/server_guide/customize-state-machine.md +++ b/d-engine/src/docs/server_guide/customize-state-machine.md @@ -30,7 +30,7 @@ impl StateMachine for CustomStateMachine { } // close() has a default no-op implementation. - // Override it if your backend must persist something at shutdown (e.g. lease state) or + // Override it if your backend must persist something at shutdown (e.g. data, metadata) or // holds an exclusive OS resource (a lock file, an open DB handle). d-engine calls it once, // as the worker's last step: after every committed entry has been applied when the worker // drains cleanly, but shutdown continues without that guarantee if @@ -59,8 +59,8 @@ impl StateMachine for CustomStateMachine { let mut results = Vec::with_capacity(chunk.len()); for entry in chunk { let result = match &entry.command { - Command::Insert { key, value, ttl_secs } => { - self.backend.put(key, value, *ttl_secs)?; + Command::Insert { key, value } => { + self.backend.put(key, value)?; ApplyResult::success(entry.index) } Command::Delete { key } => { @@ -82,7 +82,7 @@ impl StateMachine for CustomStateMachine { for op in ops { match op { d_engine::BatchOp::Insert { key, value } => { - self.backend.put(key, value, None)?; + self.backend.put(key, value)?; } d_engine::BatchOp::Delete { key } => { self.backend.delete(key)?; diff --git a/d-engine/src/docs/use-cases.md b/d-engine/src/docs/use-cases.md index dacad77f..b5cbd3a7 100644 --- a/d-engine/src/docs/use-cases.md +++ b/d-engine/src/docs/use-cases.md @@ -54,7 +54,7 @@ simpler to operate, embeddable in Rust, but with a narrower API surface. | etcd feature | d-engine status | | --------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Lease with keepalive | ✗ Not supported. d-engine has `put_with_ttl`, but keys cannot be renewed. Patterns relying on lease expiry for crash detection (service discovery, auto-releasing locks) are not directly portable. | +| Lease with keepalive | ✗ Not supported. d-engine has no key expiry; a layer above d-engine must implement it. Patterns relying on lease expiry for crash detection (service discovery, auto-releasing locks) are not directly portable. | | Distributed lock (`Lock` API) | ✗ No built-in primitive. Can be approximated with `compare_and_swap` + `watch`, but without lease keepalive, safe auto-release on crash is not possible today. | | Multi-key transactions (`txn`) | Partial. d-engine has single-key `compare_and_swap`. Multi-key atomic operations are not supported. | | Watch from historical revision | ✗ Watches start from the current state only. Use `scan_prefix` to rebuild state at reconnect, then resume watching from current. If your logic depends on `start_revision` to replay exact event history, this pattern will not fully substitute. | diff --git a/deny.toml b/deny.toml index f19e57ff..a41ba9c5 100644 --- a/deny.toml +++ b/deny.toml @@ -216,8 +216,6 @@ skip = [ { crate = "socket2@0.5.10", reason = "tonic 0.13 pins socket2 0.5 internally" }, # ring and socket2 0.5 pull windows-sys 0.52 { crate = "windows-sys@0.52.0", reason = "transitive dep of ring 0.17 and socket2 0.5" }, - # bindgen (rocksdb build dep) pins itertools 0.13 - { crate = "itertools@0.13.0", reason = "bindgen (rocksdb build dep) pins itertools 0.13" }, # tempfile/uuid pull getrandom 0.4.x while rand_core 0.9 still requires 0.3.x { crate = "getrandom@0.4.2", reason = "tempfile 3.27 and uuid 1.x use getrandom 0.4; rand_core 0.9 pins 0.3" }, # getrandom 0.3.x→wasip2 and 0.4.x→wasip3 pull different wit-bindgen versions diff --git a/examples/service-discovery-embedded/server.rs b/examples/service-discovery-embedded/server.rs index 5fe9cf2d..47c17075 100644 --- a/examples/service-discovery-embedded/server.rs +++ b/examples/service-discovery-embedded/server.rs @@ -161,7 +161,7 @@ async fn main() -> Result<(), Box> { client.put(b"/services/payment/node2", b"10.0.0.2:9090").await?; tokio::time::sleep(Duration::from_millis(100)).await; - // Simulate node1 crashing (TTL expiry or explicit deregister) + // Simulate node1 leaving (explicit deregister) println!("\n--- node1 deregisters (crash / graceful shutdown) ---"); client.delete(b"/services/payment/node1").await?; tokio::time::sleep(Duration::from_millis(100)).await; diff --git a/examples/single-node-expansion/config/n1.toml b/examples/single-node-expansion/config/n1.toml index 1d961cb3..368ae09b 100644 --- a/examples/single-node-expansion/config/n1.toml +++ b/examples/single-node-expansion/config/n1.toml @@ -51,11 +51,6 @@ max_log_entries_before_snapshot = 5000 retained_log_entries = 100 cleanup_retain_count = 100 -# == TTL Lease Configuration == -[raft.state_machine.lease] -cleanup_interval_ms = 1000 -max_cleanup_duration_ms = 1 - # == Network Control Plane (voting, heartbeat, etc.) == [network.control] connection_window_size = 4_194_304 diff --git a/examples/sled-cluster/src/sled_engine_test.rs b/examples/sled-cluster/src/sled_engine_test.rs index 16698d38..4b13a3fa 100644 --- a/examples/sled-cluster/src/sled_engine_test.rs +++ b/examples/sled-cluster/src/sled_engine_test.rs @@ -166,9 +166,7 @@ fn create_test_command_payload(index: u64) -> EntryPayload { let insert = Insert { key, - value, - ttl_secs: 0, - }; + value, }; let operation = Operation::Insert(insert); let write_cmd = WriteCommand { operation: Some(operation), diff --git a/examples/sled-cluster/src/sled_state_machine.rs b/examples/sled-cluster/src/sled_state_machine.rs index 438b32ab..96fa0710 100644 --- a/examples/sled-cluster/src/sled_state_machine.rs +++ b/examples/sled-cluster/src/sled_state_machine.rs @@ -549,12 +549,6 @@ impl StateMachine for SledStateMachine { Ok(()) } - async fn lease_background_cleanup(&self) -> Result> { - // SledStateMachine example doesn't implement lease support - // Return empty vector (no cleanup performed) - Ok(vec![]) - } - fn scan_prefix( &self, prefix: &[u8], diff --git a/examples/sled-cluster/src/state_machine/sled_state_machine_test.rs b/examples/sled-cluster/src/state_machine/sled_state_machine_test.rs index 380e6777..52ddae2b 100644 --- a/examples/sled-cluster/src/state_machine/sled_state_machine_test.rs +++ b/examples/sled-cluster/src/state_machine/sled_state_machine_test.rs @@ -80,9 +80,7 @@ async fn test_snapshot_captures_all_current_data() { term: 1, command: Command::Insert { key: Bytes::from(format!("snap_key_{i}")), - value: Bytes::from(format!("snap_val_{i}")), - ttl_secs: None, - }, + value: Bytes::from(format!("snap_val_{i}")), }, }) .collect(); sm.apply_chunk(&entries).await.unwrap(); @@ -118,9 +116,7 @@ async fn test_snapshot_does_not_include_post_snapshot_entries() { term: 1, command: Command::Insert { key: Bytes::from_static(b"before"), - value: Bytes::from_static(b"v1"), - ttl_secs: None, - }, + value: Bytes::from_static(b"v1"), }, }]; sm.apply_chunk(&pre_entries).await.unwrap(); @@ -135,9 +131,7 @@ async fn test_snapshot_does_not_include_post_snapshot_entries() { term: 1, command: Command::Insert { key: Bytes::from_static(b"after"), - value: Bytes::from_static(b"v2"), - ttl_secs: None, - }, + value: Bytes::from_static(b"v2"), }, }]; sm.apply_chunk(&post_entries).await.unwrap(); @@ -168,9 +162,7 @@ async fn test_snapshot_install_content_survives_restart() { term: 1, command: Command::Insert { key: Bytes::from(format!("install_key_{i}")), - value: Bytes::from(format!("install_val_{i}")), - ttl_secs: None, - }, + value: Bytes::from(format!("install_val_{i}")), }, }) .collect(); source_sm.apply_chunk(&entries).await.unwrap(); @@ -246,9 +238,7 @@ async fn test_new_recovers_backup_after_interrupted_install() { term: 1, command: Command::Insert { key: Bytes::from_static(b"pre_crash_key"), - value: Bytes::from_static(b"pre_crash_val"), - ttl_secs: None, - }, + value: Bytes::from_static(b"pre_crash_val"), }, }]) .await .unwrap(); diff --git a/examples/three-nodes-standalone/Makefile b/examples/three-nodes-standalone/Makefile index 4c0670ff..03cd2989 100644 --- a/examples/three-nodes-standalone/Makefile +++ b/examples/three-nodes-standalone/Makefile @@ -99,7 +99,7 @@ start-cluster: # Isolates each node's storage on its own independent RAM disk volume — use # this to strip physical-disk latency out of a benchmark (e.g. to study # fsync/scheduling behavior in isolation), not as a general fix for -# unrelated flakiness. See tickets/milestones/v0.2.5/446-perf-batching-measurement-2026-09-13.md. +# unrelated flakiness. RAMDISK_SIZE_MB ?= 1024 RAMDISK_SECTORS := $(shell echo $$(( $(RAMDISK_SIZE_MB) * 2048 ))) RAMDISK_VOLUMES := RAMDisk1 RAMDisk2 RAMDisk3 diff --git a/examples/three-nodes-standalone/config/n1.toml b/examples/three-nodes-standalone/config/n1.toml index 0866c1bb..47fb4092 100644 --- a/examples/three-nodes-standalone/config/n1.toml +++ b/examples/three-nodes-standalone/config/n1.toml @@ -47,11 +47,6 @@ max_log_entries_before_snapshot = 5000 retained_log_entries = 100 cleanup_retain_count = 100 -# == TTL Lease Configuration == -[raft.state_machine.lease] -cleanup_interval_ms = 1000 -max_cleanup_duration_ms = 1 - # == Network Control Plane (voting, heartbeat, etc.) == [network.control] request_timeout_in_ms = 100 diff --git a/examples/three-nodes-standalone/config/n2.toml b/examples/three-nodes-standalone/config/n2.toml index cbb9e8b3..ea40dbe0 100644 --- a/examples/three-nodes-standalone/config/n2.toml +++ b/examples/three-nodes-standalone/config/n2.toml @@ -47,11 +47,6 @@ max_log_entries_before_snapshot = 5000 retained_log_entries = 100 cleanup_retain_count = 100 -# == TTL Lease Configuration == -[raft.state_machine.lease] -cleanup_interval_ms = 1000 -max_cleanup_duration_ms = 1 - # == Network Control Plane (voting, heartbeat, etc.) == [network.control] request_timeout_in_ms = 100 diff --git a/examples/three-nodes-standalone/config/n3.toml b/examples/three-nodes-standalone/config/n3.toml index 8688add7..416482a7 100644 --- a/examples/three-nodes-standalone/config/n3.toml +++ b/examples/three-nodes-standalone/config/n3.toml @@ -47,11 +47,6 @@ max_log_entries_before_snapshot = 5000 retained_log_entries = 100 cleanup_retain_count = 100 -# == TTL Lease Configuration == -[raft.state_machine.lease] -cleanup_interval_ms = 1000 -max_cleanup_duration_ms = 1 - # == Network Control Plane (voting, heartbeat, etc.) == [network.control] request_timeout_in_ms = 100 diff --git a/examples/three-nodes-standalone/docker/Dockerfile b/examples/three-nodes-standalone/docker/Dockerfile index 20589759..76b423b2 100644 --- a/examples/three-nodes-standalone/docker/Dockerfile +++ b/examples/three-nodes-standalone/docker/Dockerfile @@ -18,7 +18,7 @@ RUN apt-get update && \ && rm -rf /var/lib/apt/lists/* # Build context is the d-engine workspace root. -# Run: docker build -t demo:v0.2.4 -f examples/three-nodes-standalone/docker/Dockerfile . +# Run: docker build -t demo:v0.3.0 -f examples/three-nodes-standalone/docker/Dockerfile . WORKDIR /build # Disable sccache: .cargo/config.toml sets rustc-wrapper=sccache for local dev, diff --git a/examples/three-nodes-standalone/docker/README.md b/examples/three-nodes-standalone/docker/README.md index 8ffb07bd..3a92ded6 100644 --- a/examples/three-nodes-standalone/docker/README.md +++ b/examples/three-nodes-standalone/docker/README.md @@ -15,7 +15,7 @@ mkdir -p examples/three-nodes-standalone/docker/output/{logs,db} ```bash # Build core component (run from d-engine workspace root) -docker build -t demo:v0.2.4 -f examples/three-nodes-standalone/docker/Dockerfile . +docker build -t demo:v0.3.0 -f examples/three-nodes-standalone/docker/Dockerfile . # Build monitoring component docker build -t prometheus:1.0 \ diff --git a/examples/three-nodes-standalone/docker/config/n1.toml b/examples/three-nodes-standalone/docker/config/n1.toml index a5b007dc..ed372e18 100644 --- a/examples/three-nodes-standalone/docker/config/n1.toml +++ b/examples/three-nodes-standalone/docker/config/n1.toml @@ -46,11 +46,6 @@ enable = false max_log_entries_before_snapshot = 10000 retained_log_entries = 3 -# == TTL Lease Configuration == -[raft.state_machine.lease] -cleanup_interval_ms = 1000 -max_cleanup_duration_ms = 1 - # == Network Control Plane (voting, heartbeat, etc.) == [network.control] request_timeout_in_ms = 100 diff --git a/examples/three-nodes-standalone/docker/config/n2.toml b/examples/three-nodes-standalone/docker/config/n2.toml index 06c54f50..22319923 100644 --- a/examples/three-nodes-standalone/docker/config/n2.toml +++ b/examples/three-nodes-standalone/docker/config/n2.toml @@ -46,11 +46,6 @@ enable = false max_log_entries_before_snapshot = 10000 retained_log_entries = 3 -# == TTL Lease Configuration == -[raft.state_machine.lease] -cleanup_interval_ms = 1000 -max_cleanup_duration_ms = 1 - # == Network Control Plane (voting, heartbeat, etc.) == [network.control] request_timeout_in_ms = 100 diff --git a/examples/three-nodes-standalone/docker/config/n3.toml b/examples/three-nodes-standalone/docker/config/n3.toml index 2cfe4ad9..a49b6e55 100644 --- a/examples/three-nodes-standalone/docker/config/n3.toml +++ b/examples/three-nodes-standalone/docker/config/n3.toml @@ -46,11 +46,6 @@ enable = false max_log_entries_before_snapshot = 10000 retained_log_entries = 3 -# == TTL Lease Configuration == -[raft.state_machine.lease] -cleanup_interval_ms = 1000 -max_cleanup_duration_ms = 1 - # == Network Control Plane (voting, heartbeat, etc.) == [network.control] request_timeout_in_ms = 100 diff --git a/examples/three-nodes-standalone/docker/docker-compose.yml b/examples/three-nodes-standalone/docker/docker-compose.yml index 88dc7445..0cc98bb1 100644 --- a/examples/three-nodes-standalone/docker/docker-compose.yml +++ b/examples/three-nodes-standalone/docker/docker-compose.yml @@ -1,6 +1,6 @@ services: node1: &node-base - image: demo:v0.2.5 + image: demo:v0.3.0 cap_add: - NET_ADMIN # Add the NET_ADMIN capability environment: