Skip to content

Add dm-flakey power-loss harness + release 0.1.5 (#31) #35

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
robertoberto merged 4 commits into from
May 21, 2026
Merged

Add dm-flakey power-loss harness + release 0.1.5 (#31) #35

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
cd79bfa
feat(testing): add dm-flakey power-loss simulation harness (#31)
robertoberto May 21, 2026
46d3340
chore: release datawal 0.1.5
robertoberto May 21, 2026
2f74238
style: cargo fmt power_loss_workload.rs
robertoberto May 21, 2026
da88806
style: silence clippy on power-loss examples
robertoberto May 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,47 @@ and the crate uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
`WIRE_VERSION = 1`, public Rust API additions and small breaking
changes may still occur within `0.1.x` until `0.2.0`.

## [0.1.5] — 2026-05-21

`0.1.5` is a docs-and-testing release. It adds a Linux device-mapper
power-loss simulation harness and brings the README in line with the
post-alpha state of the crate. Public Rust API unchanged; wire
format (`WIRE_VERSION = 1`) unchanged; corpus fixtures unchanged.

### Added

- **Power-loss simulation harness (Linux, device-mapper)** (issue
[#31](https://github.com/deepcausa/datawal/issues/31)). New
examples `power_loss_workload` and `power_loss_validate`, plus
driver scripts `scripts/power_loss_dm_flakey.sh` and
`scripts/power_loss_cleanup.sh`, exercise `DataWal` under
`dm-flakey`-induced write loss on ext4. The workload appends
`put`/`delete` ops with `fsync` after every op and mirrors them
into an fsync-ordered JSONL oracle on a separate filesystem.
After the dm table is flipped to `error_writes` and the volume is
force-unmounted and remounted, the validator reopens the store
and checks that the keydir is a per-key prefix of the oracle
(invariants 3, 4, 5). Linux-only, root-only, **not** part of CI.
See [`docs/power-loss-testing.md`](docs/power-loss-testing.md) for
the harness contract and
[`docs/power-loss-results.md`](docs/power-loss-results.md) for a
sanitized record of a verified run.
- **`Durability evidence` section in `README.md`** linking the fuzz
targets, `proptest` invariants, SIGKILL crash-injection suite, and
the new `dm-flakey` harness.

### Changed

- **`README.md` aligned to post-alpha state.** Removed references to
"alpha crate", "alpha release", "alpha limits", "not
production-ready", and other wording inherited from
`0.1.0-alpha.1`. Updated "What is in" / "Limits" / "Evidence
stack" / "Formal models" to reflect `RecordLogReader`,
`scan_iter`, the `datawal` CLI, and the four TLA+ models
(`RecordLog`, `KeydirProjection`, `Compaction`, `ReadWhileWrite`).
Reframed status as "pre-1.0 crate suitable for local recoverable
logs with documented limits; not a general-purpose database".

## [0.1.4] — 2026-05-21

`0.1.4` is the first non-alpha release of `datawal`. It is suitable
Expand Down
4 changes: 2 additions & 2 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ exclude = [
]

[workspace.package]
version = "0.1.4"
version = "0.1.5"
edition = "2021"
license = "MIT OR Apache-2.0"
repository = "https://github.com/deepcausa/datawal"
Expand Down
124 changes: 79 additions & 45 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,12 @@
datawal is a local record store: a framed append-only `RecordLog` plus an
optional last-write-wins `DataWal` KV projection.

> `0.1.4` is the first non-alpha datawal release. It is suitable for
> local recoverable logs where JSONL would otherwise be used, with the
> documented limits in [`docs/canon.md`](docs/canon.md). `0.1.x` may
> still introduce small breaking changes before `0.2`; the on-disk
> wire format (`WIRE_VERSION = 1`) is frozen and locked by corpus
> fixtures.
> datawal is a pre-1.0 crate suitable for local recoverable logs where
> JSONL would otherwise be used, with the documented limits in
> [`docs/canon.md`](docs/canon.md). It is not a general-purpose
> database. `0.1.x` may still introduce small breaking changes before
> `0.2`; the on-disk wire format (`WIRE_VERSION = 1`) is frozen and
> locked by corpus fixtures.

**MSRV:** Rust 1.75.0

Expand Down Expand Up @@ -61,22 +61,28 @@ optional last-write-wins `DataWal` KV projection.

## Current status

**datawal is currently an alpha crate:** functional and model-checked at
the protocol level, but not production-ready. See
**datawal is a pre-1.0 crate suitable for local recoverable logs where
JSONL would otherwise be used, with documented limits.** It is not a
general-purpose database. `0.1.x` may still introduce small breaking
changes before `0.2`; the on-disk wire format (`WIRE_VERSION = 1`) is
frozen and locked by corpus fixtures. See
[`docs/roadmap.md`](docs/roadmap.md) for the exact release scope.

It is published on crates.io as an alpha release; the on-disk wire format
is frozen by a corpus fixture set, but workload coverage is still only at
the unit and integration level.

What is in:

- 58 tests green (`cargo test --workspace`).
- 3 TLA+ models model-checked with TLC 2.19.
- Wire-format corpus: 6 binary fixture directories, 11 corpus tests.
- 4 runnable examples.
- Real **CRC-32C** (Castagnoli, `0x1EDC6F41`) per record, pinned by a
known-vector test.
- Framed `RecordLog` with **CRC-32C** (Castagnoli, `0x1EDC6F41`) and
longest-valid-prefix recovery.
- `DataWal` bytes-in / bytes-out KV projection with tombstones and
`compact_to`.
- `RecordLogReader` snapshot-at-open reader API (no live tailing).
- `datawal` CLI for inspection and export
(`crates/datawal-cli/`).
- Wire-format corpus locked by binary fixtures.
- TLA+ models for `RecordLog`, `KeydirProjection`, `Compaction`,
and `ReadWhileWrite`.
- Fuzz targets, `proptest` invariants, crash-injection tests,
ENOSPC tests, soak harness, and `dm-flakey` power-loss harness.
- Criterion benchmarks with a reference run.
- **fs2 fd-based advisory lock**: held by a file descriptor, not by the
existence of the sentinel file. Released on `Drop` / process exit. A stale
`.lock` from a crashed previous process is not a problem.
Expand All @@ -95,28 +101,28 @@ What is not in:
- Multi-writer.
- Query / secondary indexes.
- In-place compaction.
- Reader API / concurrent reads.
- Group commit / configurable fsync policy.

## Limits

`datawal` is bytes-first, but not unbounded. Neither the `RecordLog` nor
the `DataWal` projection interprets the bytes — no JSON, no UTF-8, no
MessagePack parsing in the core. Current alpha limits:

| Limit | Value / status | Notes |
| ------------------ | ---------------------: | ---------------------------------------------------------------------------------------------------- |
| Max key size | 64 KiB | Per record. Larger keys are rejected. |
| Max payload size | 64 MiB | Per record. For larger values, use an external blob store and store references. |
| Writers | Single writer | Enforced with an advisory fd lock. No multi-writer semantics. |
| Readers | No reader API yet | `scan()` is the only read path; tracked in [#5](https://github.com/deepcausa/datawal/issues/5). |
| `scan()` memory | `Vec<Record>` | Not streaming yet; large logs need [#3](https://github.com/deepcausa/datawal/issues/3). |
| `DataWal` keydir | values in memory | Live values are held in the in-memory keydir; offset-based variant is [#4](https://github.com/deepcausa/datawal/issues/4). |
| Durability | explicit `fsync()` | `append()` is recoverable; `append() + fsync()` is durable under documented assumptions. |
| Compaction | `compact_to` only | Snapshot-style rebuild into a target directory. No in-place `compact()`. |
| CAS / blob | not included | Planned as a separate crate / layer; tracked in [#7](https://github.com/deepcausa/datawal/issues/7). |
| Compression | not included | `flags` must be zero in v0.1. |
| Query | not included | No SQL, indexes, joins, range scans, or planner. See [#13](https://github.com/deepcausa/datawal/issues/13). |
| Production status | alpha | Functional, tested, model-checked at the protocol level; not production-ready. |
MessagePack parsing in the core. Current limits:

| Limit | Value / status | Notes |
| ------------------ | ----------------------------: | -------------------------------------------------------------------------------------------------------------------------------------- |
| Max key size | 64 KiB | Per record. Larger keys are rejected. |
| Max payload size | 64 MiB | Per record. For larger values, use an external blob store and store references. |
| Writers | Single writer | Enforced with an advisory fd lock. No multi-writer semantics. |
| Readers | snapshot-at-open reader | `RecordLogReader` can inspect a store without taking the writer lock; no live tailing API. |
| `scan()` memory | eager `Vec<Record>` | Use `scan_iter()` for record-level lazy iteration; it is segment-buffered, not zero-copy. |
| `DataWal` keydir | offsets in memory | Live keys map to `RecordRef`; `get()` performs I/O and CRC verification. |
| Durability | explicit `fsync()` | `append()` is recoverable; `append() + fsync()` is durable under documented assumptions. |
| Compaction | `compact_to` only | Snapshot-style rebuild into a target directory. No in-place `compact()`. |
| CAS / blob | not included | Planned as a separate crate / layer; tracked in [#7](https://github.com/deepcausa/datawal/issues/7). |
| Compression | not included | `flags` must be zero in v0.1. |
| Query | not included | No SQL, indexes, joins, range scans, or planner. See [#13](https://github.com/deepcausa/datawal/issues/13). |
| Production status | scoped production use | Suitable for local single-writer recoverable logs; not a general-purpose database. |

What is **not** limited inside those bounds: the byte composition of
keys and payloads. Any sequence is legal, including all-zero, all-`0xFF`,
Expand Down Expand Up @@ -159,19 +165,44 @@ db.export_jsonl(Path::new("/tmp/my-kv.jsonl"))?;

The protocol has been validated at multiple levels:

| Layer | Evidence |
| ------------------- | ----------------------------------------------------- |
| Specification | `docs/canon.md` — 14 binding clauses; byte layout |
| Code | `crates/datawal-core/src/` — ~1900 LOC Rust |
| Unit + integration | 58 tests across `tests/*.rs` and embedded `#[test]`s |
| Wire-format corpus | 6 binary fixture dirs, 11 corpus tests |
| Formal models | 3 TLA+ models, model-checked with TLC 2.19 |
| Runnable examples | 4 examples under `crates/datawal-core/examples/` |
| Layer | Evidence |
| ------------------- | ---------------------------------------------------------------------------------------------- |
| Specification | `docs/canon.md`; documented byte layout and limits |
| Wire format | binary corpus fixtures locked by CI |
| Formal models | TLA+ models for `RecordLog`, `KeydirProjection`, `Compaction`, `ReadWhileWrite` |
| Parser robustness | `cargo-fuzz` targets and `proptest` invariants |
| Recovery behavior | crash-injection tests, ENOSPC tests, `dm-flakey` power-loss harness |
| Long-run behavior | soak harness |
| Performance | Criterion benchmarks and reference run |
| Operations | `datawal` CLI for inspection and export |

**Formal models wording:** model-checked under documented assumptions.
Not "formally verified". Models do not check the Rust implementation.
See `formal/README.md` for invariants and how to run TLC.

## Durability evidence

DataWal is exercised under several layers of failure-mode testing:

- Fuzz tests on the record decoder (see [Fuzzing](#fuzzing)).
- `proptest` invariants on append-then-recover sequences.
- A SIGKILL-based crash-injection suite in `tests/crash_injection.rs`
that spawns the test binary as a child, kills it at named points
(`append_no_fsync`, `append_fsync`, `rotate`, `compact_to`,
`export_jsonl`), then reopens the store and checks invariants.
- A `dm-flakey` power-loss simulation harness on Linux (root, not CI)
that routes ext4 over a device-mapper layer, flips the layer to
`error_writes`, force-unmounts, remounts the layer healthy, reopens
the store, and validates that the reopened state matches an
fsync-ordered oracle. See [`docs/power-loss-testing.md`](docs/power-loss-testing.md)
for the harness contract and prerequisites, and
[`docs/power-loss-results.md`](docs/power-loss-results.md) for a
sample verified run.

This is stricter than process-level crash testing but is not a
substitute for real power-cut testing on real hardware. DataWal trusts
the storage stack below it to honor `fsync`.

## Layout

```
Expand Down Expand Up @@ -220,6 +251,7 @@ cargo test --workspace
cargo run -p datawal --example record_log_demo
cargo run -p datawal --example datawal_kv_demo
cargo run -p datawal --example tail_recovery_demo
cargo run -p datawal-cli -- --help
cargo doc --workspace --no-deps
```

Expand Down Expand Up @@ -293,12 +325,14 @@ CI verifies the targets *compile* on nightly; it does not run them.

## Formal models

Three small TLA+ models live under `formal/` and are checked with
Four small TLA+ models live under `formal/` and are checked with
[TLC](https://github.com/tlaplus/tlaplus/) 2.19+:

- `RecordLog.tla` — append / fsync / crash; durable is a monotonic prefix.
- `KeydirProjection.tla` — last-write-wins keydir from a put/del log.
- `Compaction.tla` — `compact_to` preserves the live projection.
- `ReadWhileWrite.tla` — snapshot-at-open reader behavior under
concurrent writer progress.

**model-checked under documented assumptions** — not "formally verified",
does not check the Rust implementation. See `formal/README.md`.
Expand Down Expand Up @@ -331,7 +365,7 @@ from them.

- `docs/canon.md` — binding decisions and the byte-layout of a record.
- `docs/technical-decisions.md` — TD-NNN entries documenting choices.
- `docs/roadmap.md` — current alpha scope, what is frozen, and the tracked roadmap issues.
- `docs/roadmap.md` — current release scope, what is frozen, and the tracked roadmap issues.
- `formal/README.md` — the TLA+ models and how to run TLC.

## License
Expand Down
Loading
Loading