From 58f6976b0cba7a393b0c0316a54b28e066fcc22d Mon Sep 17 00:00:00 2001 From: Christophe Pettus Date: Wed, 1 Jul 2026 09:51:03 -0700 Subject: [PATCH 1/2] docs: split THEORY out of ARCHITECTURE, refresh all three docs for encryption MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Draw the ARCHITECTURE/THEORY boundary (previously blended into one file) and bring the three top-level docs current with the merged on-disk encryption feature. - ARCHITECTURE.md: compressed to the cold-start, act-correctly map — kept every on-disk byte-format table, the commit/recovery diagrams, the failure table, and the glossary; extracted the narrative "why" and the implementation history to THEORY.md. Fixed three content bugs found against the code: the stale module map (transaction.rs / superblock.rs were decomposed into directory modules; the crypto/ layer and handle.rs / lru.rs / spillway.rs were missing), the superblock 324..8184 reserved-bytes contradiction (now conditional on plaintext vs encrypted), and encryption shown as shipped. - THEORY.md (new): theory of operation for an engineer about to change the engine — the 18 load-bearing decisions with their rejected alternatives, the implementation history, and the benchmark methodology. Cross-references ARCHITECTURE's invariant enumeration instead of duplicating it. Seven rationale gaps are marked "not recovered from project sources" rather than invented. - README.md: added the encryption surface (Options.encryption_key / argon2_params, Key / Argon2Params, add_key / rotate_key / remove_key, the five operational error variants + fatal DecryptionFailed, MAJOR=2, the Python encryption_key kwarg) and corrected the toolchain commands (the Python flow needs hypothesis and --release; the clippy / msrv / test gates now match CI). --- ARCHITECTURE.md | 250 +++++++++++++++++++--------------------------- README.md | 80 +++++++++++++-- THEORY.md | 258 ++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 429 insertions(+), 159 deletions(-) create mode 100644 THEORY.md diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 1e45821..fe79ca5 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -1,12 +1,14 @@ # Chisel — Architecture and On-Disk Format -This document is for someone (human or AI) reading the Chisel codebase for the first time. It explains *how* Chisel is laid out, *why* the layers stack the way they do, and what every byte on disk means. For *what Chisel does* and how to use it, see [`README.md`](README.md). For the running decision log — open issues, closed issues, every design tradeoff with date-stamped rationale — see [`ISSUES.md`](ISSUES.md). +This document is the cold-start reference for someone (human or AI) picking up the Chisel codebase in a context-free session. It is a compressed map of the layers, the exact on-disk byte format, the load-bearing invariants, and the landmines. Read it whole at the start of a session. For *what Chisel does* and how to use it, see [`README.md`](README.md). For the running decision log — open issues, closed issues, every design tradeoff with date-stamped rationale — see [`ISSUES.md`](ISSUES.md). + +For the theory of operation and the rationale behind these decisions — why shadow-paging over WAL, the rejected alternatives, the implementation history — see [`THEORY.md`](THEORY.md). This is a living document; update it when the architecture changes. Decisions documented here should be supportable by the code at the time you read it — if a claim and the code disagree, trust the code and update the doc. ## Table of contents -1. [Design philosophy](#design-philosophy) +1. [Design commitments](#design-commitments) 2. [Commenting standards](#commenting-standards) 3. [Layer model](#layer-model) 4. [Commit protocol](#commit-protocol) @@ -14,21 +16,17 @@ This is a living document; update it when the architecture changes. Decisions do 6. [On-disk format](#on-disk-format) 7. [Cross-cutting concepts](#cross-cutting-concepts) - [On-disk encryption](#on-disk-encryption) -8. [Benchmark infrastructure](#benchmark-infrastructure) -9. [Implementation history](#implementation-history) -10. [Glossary](#glossary) +8. [Glossary](#glossary) --- -## Design philosophy - -Three commitments shape everything else: +## Design commitments -- **Single-writer, embedded.** Exactly one process owns the file at a time, enforced by `flock`. The Rust API is `&mut self` for every mutator; there is no internal locking, no concurrent transactions, no MVCC. This is *philosophical* (see the project memory note "Chisel single-client design is philosophical"), not a v1 simplification — the type system encodes it. -- **Shadow paging, not WAL.** Every write goes to a fresh page. The previously-committed superblock keeps pointing at the previous (intact) pages until commit swaps in a new superblock. Crash recovery is "pick the winning superblock" — there is no log to replay and no recovery procedure as such. -- **Durability over performance.** Every commit performs two `fsync` calls (data, then superblock). Every page on disk carries an XXH3 checksum validated on load. The poison model (see below) treats any `fsync` failure as terminal because Linux fsyncgate semantics make retry unsafe. +Three enforced facts shape everything else (rationale in THEORY.md): -These three together explain most of Chisel's other choices (poison model, per-module COW, exclusive `flock` even for readers). +- **Single-writer, embedded.** Exactly one process owns the file at a time, enforced by `flock`. The Rust API is `&mut self` for every mutator; no internal locking, no concurrent transactions, no MVCC. +- **Shadow paging, not WAL.** Every write goes to a fresh page; the previously-committed superblock keeps pointing at the previous intact pages until commit swaps in a new superblock. Recovery is "pick the winning superblock" — no log to replay. +- **Durability over performance.** Every commit performs two (three with the I28 pre-drain) `fsync` calls (data, then superblock). Every page carries an XXH3 checksum validated on load. Any `fsync` failure poisons the manager (fsyncgate makes retry unsafe). --- @@ -49,31 +47,40 @@ When a comment and the code disagree, the comment is stale by default. Update or ## Layer model -Chisel's modules form a strict bottom-up dependency graph: each layer only depends on layers below it, never sideways or upward. The diagram below is annotated with the responsibility of each module. +Chisel's modules form a strict bottom-up dependency graph: each layer only depends on layers below it, never sideways or upward. Read the codebase in dependency order and you never have to forward-reference. ```mermaid flowchart BT page["page.rs
constants, checksums, PageType"] error["error.rs
ChiselError, is_fatal()"] - superblock["superblock.rs
superblock layout, select()"] - page_io["page_io.rs
raw file I/O, flock
(only module touching FS)"] - page_cache["page_cache.rs
LRU cache, dirty tracking,
checksum validation on load"] + handle["handle.rs
public Handle/Tag newtypes"] + crypto["crypto/mod.rs
PageCipher, KDF envelope,
DEK wrap/unwrap, Key types"] + superblock["superblock/ (mod.rs + crypto_header.rs)
superblock layout, select(),
sealed body, key-slot table"] + page_io["page_io.rs
raw file I/O, flock, stride
(only module touching FS)"] + lru["lru.rs
O(1) intrusive LRU index"] + page_cache["page_cache.rs
LRU cache, dirty tracking,
checksum validation, cipher"] + spillway["spillway.rs
sidecar dirty-overflow file"] freemap["freemap.rs
bitmap free-page tracking
(single-page leaf primitive)"] freemap_tree["freemap_tree.rs
COW radix tree of bitmap leaves"] data_page["data_page.rs
slotted page (R1 packing)"] overflow["overflow.rs
large-value chains"] handle_table["handle_table.rs
radix tree, per-module COW"] membership_index["membership_index.rs
RadixU64 + two-level
MembershipIndex (tag→handles)"] - transaction["transaction.rs
TransactionManager:
orchestrates everything below"] + transaction["transaction/ (mod.rs + ~15 submodules)
TransactionManager:
orchestrates everything below"] defrag["defrag.rs
sparse-page consolidation"] stats["stats.rs
Stats snapshot type"] lib["lib.rs
Chisel: thin public API"] page --> page_io error --> page_io + crypto --> page_io page --> page_cache error --> page_cache + crypto --> page_cache page_io --> page_cache + lru --> page_cache + page_cache --> spillway + crypto --> superblock page --> freemap page --> data_page page --> overflow @@ -95,36 +102,41 @@ flowchart BT error --> transaction transaction --> defrag transaction --> stats + handle --> lib transaction --> lib ``` -Why bottom-up matters: it means you can read the codebase in dependency order and never have to forward-reference. It also means a parallel review pass (which Chisel has had three of) can split across layers cleanly — see the 2026-04-22 review pass that dispatched five agents, one per layer group. - ### Module responsibilities at a glance | Layer | Module | Responsibility | Key invariant | |---|---|---|---| | 1 | `page.rs` | Page size, type tags, header sizes, magic, format-version constants, XXH3 checksum primitives. | `PAGE_SIZE = 8192`; checksum lives in the last 8 bytes; little-endian on disk. | | 1 | `error.rs` | `ChiselError` enum, `is_fatal()` classifier (operational vs fatal). | Fatal variants poison the manager (I1). | -| 1 | `superblock.rs` | In-memory `Superblock` struct, `serialize`/`deserialize`, `select()` across N candidate slots. | Magic + checksum + `superblock_count ∈ 2..=16` filter slots before tie-breaking by `txn_counter`. | -| 2 | `page_io.rs` | Raw `pread`/`pwrite` of fixed-size pages, exclusive `flock`, `fsync`, in-memory `Vec` backing. Tracks cumulative successful `fsync_calls` count via a `Cell`. | The **only** module that touches the filesystem; everything else uses it through `PageCache`. | -| 3 | `page_cache.rs` | LRU cache over `PageIo`, dirty tracking, checksum validation on load, spillway overflow, and `CacheFull`/`SpillwayFull` errors. Owns three `Cell` engine-activity counters (cache hits/misses, pages allocated) and exposes `counters()` aggregating them with `PageIo::fsync_count`. | Soft eviction at `max_pages`; dirty overflow spills to a sidecar Spillway file (cap = `spillway_max_bytes`); `CacheFull` at strict `max_pages` when spillway is disabled (`spillway_max_bytes=0`); checksums verified on disk LOAD only. | +| 1 | `handle.rs` | Public `Handle` (`u64`) and `Tag` (`NonZeroU32`) newtypes for the API surface. | `#[repr(transparent)]` on `Handle` is load-bearing — the bench adapter transmutes `&[u64]` → `&[Handle]` (I120/I126). Lives ABOVE the engine; the engine stays raw-integer. | +| 1 | `crypto/mod.rs` | At-rest crypto core: XChaCha20-Poly1305 `PageCipher` (page + body seal/open), envelope KDF (HKDF-SHA256 raw / Argon2id passphrase), DEK wrap/unwrap, zeroizing `Key`/`Argon2Params`. | Standalone — touches no engine layer. Only vetted RustCrypto primitives; all randomness OS-sourced. `ENC_PAGE_SIZE = 8232`, `NONCE_LEN = 24`, `TAG_LEN = 16`, `DEK_LEN = 32`. | +| 1 | `superblock/` (`mod.rs` + `crypto_header.rs`) | In-memory `Superblock` struct, `serialize`/`deserialize`, `select()` across N candidate slots; `crypto_header.rs` holds the plaintext crypto-header + 8-slot key-slot table and the sealed-body decrypt path. | Magic + checksum + `superblock_count ∈ 2..=16` filter slots before tie-breaking by `txn_counter`. Encrypted DBs seal the sensitive body under the DEK. | +| 2 | `page_io.rs` | Raw `pread`/`pwrite` of fixed-**stride** pages, exclusive `flock`, `fsync`, in-memory `Vec` backing. Tracks cumulative successful `fsync_calls` via `Cell`. Stride = `PAGE_SIZE` (plaintext) or `ENC_PAGE_SIZE` (encrypted), set once via `set_stride`. | The **only** module that touches the filesystem; crypto-agnostic (moves `stride`-byte blobs at `page_id * stride`). | +| 2 | `lru.rs` | O(1) intrusive doubly-linked LRU index over `u64` page ids (`FxHashMap`-backed, I77). | Replaces the O(n) `VecDeque::retain` LRU; consumed only by `page_cache`. | +| 3 | `page_cache.rs` | LRU cache over `PageIo`, dirty tracking, checksum validation on load, `PageCipher` seal/open at the I/O boundary, spillway overflow, `CacheFull`/`SpillwayFull` errors. Owns three `Cell` engine-activity counters (cache hits/misses, pages allocated) and aggregates them with `PageIo::fsync_count` into `counters()`. | Soft eviction at `max_pages`; dirty overflow spills to a sidecar (cap = `spillway_max_bytes`); `CacheFull` at strict `max_pages` when spillway disabled (`spillway_max_bytes=0`); checksums verified on disk LOAD only. | +| 3 | `spillway.rs` | Sidecar `.spillway` file for dirty pages the LRU is forced to spill. Per-slot XXH3 over `page_id ‖ payload`; crypto-agnostic (plaintext 8192-byte page or sealed 8232-byte blob). | Never `fsync`ed; truncated at open/commit/rollback — its content is always discardable uncommitted state. | | 4 | `freemap.rs` | Single-page bitmap primitive: `allocate_first` / `mark_free` on one `[u8; PAGE_SIZE]` buffer. | Pure buffer manipulation; no cache or I/O. Composed into the multi-page tree by `freemap_tree.rs`. | | 4 | `freemap_tree.rs` | COW radix tree of FreeMap leaves; the full multi-page freemap. | All structural COW pages sourced out-of-band (never from the bitmap); session-COW dedup (one COW per node per commit). | | 4 | `data_page.rs` | Slotted page layout (R1): slot directory grows forward, packed value data grows backward, dead-slot tombstones reclaimed only by `compact()`. | Slot indices are stable until compact; compact returns an old→new mapping for callers to rewrite. | | 4 | `overflow.rs` | Singly-linked overflow chains for values > inline threshold. | `total_length` repeated on every chain page; `next_page == 0` terminates; cycle detection bounded by `total_length / OVERFLOW_PAYLOAD` (I14). | | 5 | `handle_table.rs` | Radix tree mapping `u64` handle → `(page_id, slot_index)`. Implements its own copy-on-write atop `PageCache::new_page`. | Capacity = `510 × 1021^depth`; `find_leaf` short-circuits on `handle ≥ capacity` to `Ok(None)` (I26). | | 5 | `membership_index.rs` | Reverse index `tag → {handles}` for chunk tags. A generic copy-on-write radix `RadixU64` (u64 key → u64 value, 0 = absent) used twice: an outer tree keyed by tag whose value bit-packs `(inner_depth \| inner_root)`, and per-tag inner trees keyed by handle. Returns the new root id after a COW mutation, like `handle_table`. | Fan-out 1021 per level; `0` is the absent sentinel; outer value packs `inner_root` in low 58 bits, `inner_depth` in top 6. | -| 6 | `transaction.rs` | `TransactionManager`: orchestrates begin/commit/rollback, savepoints, `persist_freemap`, the commit protocol, the poison flag. | Commit protocol step ordering is load-bearing — see next section. | +| 6 | `transaction/` (`mod.rs` + submodules) | `TransactionManager`: orchestrates begin/commit/rollback, savepoints, `persist_freemap`, the commit protocol, key-slot management, the poison flag. Submodules: `commit`, `lifecycle`, `mutate`, `read`, `savepoints`, `named_roots`, `staging` (I18 atomic allocate), `freemap` (structural-page recycle), `packing` (R1 `SlotPacker`), `keys` (key-slot ops), `recovery` (`create_new`/`open_existing`), `config`, `fault` (test-only injection), `stats`. | Commit protocol step ordering is load-bearing — see next section. | | 7 | `defrag.rs` | Sparse-page consolidation; runs inside an active transaction. | `pages_examined`/`pages_freed` are page-granular (I17). | | 7 | `stats.rs` | Two snapshot structs: `Stats` (`handle_count`, `total_pages`, `file_size_bytes`) and `ChiselCounters` (cache hits/misses, pages allocated, fsync calls — cumulative-from-open engine activity). | Both are point-in-time snapshots, not live views. `ChiselCounters` is `#[non_exhaustive]` so future counters can be added without a breaking change. | -| 8 | `lib.rs` | `Chisel` public API; thin wrapper over `TransactionManager`. | `&mut self` everywhere except `read`/`get_root_name`/`handles`/`stats`/`counters` (F3). | +| 8 | `lib.rs` | `Chisel` public API; thin wrapper over `TransactionManager`. Public surface includes the encryption additions: `Key`, `Argon2Params`, `Options::encryption_key` / `argon2_params`, and `add_key` / `rotate_key` / `remove_key`. | `&mut self` everywhere except `read`/`get_root_name`/`handles`/`stats`/`counters` (F3). | + +> `bench/` is a `default-members` workspace member (I58/I61), so a root `cargo test` runs its tests too. --- ## Commit protocol -The commit protocol is shadow paging's load-bearing part: the order of operations within `TransactionManager::commit` determines the crash-safety guarantee. Reordering any step changes what a recovering reader can observe. +The order of operations within `TransactionManager::commit` determines the crash-safety guarantee. Reordering any step changes what a recovering reader can observe. ```mermaid sequenceDiagram @@ -149,13 +161,13 @@ sequenceDiagram TM-->>U: Ok(()) ``` -### Why each step is in this order +### Step ordering (each step is load-bearing; rationale in THEORY.md) -1. **Pre-drain flush.** `persist_freemap` calls `allocate_data_page`, which can trip `maybe_evict`'s spill-or-error decision if every cached page is dirty (nothing evictable, spillway disabled or full). `CacheFull` is operational-by-design (caller recovers via commit/rollback) but the commit wrapper poisons on any error — so a `CacheFull` raised mid-commit would silently demote operational to fatal. Pre-draining clears every dirty pin so the strict cap is reachable via normal eviction. Cost: one extra `fsync`. (See I28.) -2. **`persist_freemap` allocates structural COW pages BEFORE merging.** The freed pages and the old freemap tree pages are both still referenced by the *currently-committed* superblock. Structural COW targets are sourced from an out-of-band recycle pool (dead freemap pages deferred one commit) or file extension, NEVER from the freemap's own bitmap — sourcing from the bitmap would clear a bit, COW a leaf, recurse, and never terminate. Allocating structural pages first guarantees the new page ids are either already-free in the committed state or freshly extended. Per-leaf, the allocate-before-merge (I18) ordering is preserved. (See I18.) -3. **Two separate fsyncs (data + superblock).** Linux's `fsync` does not order writes within itself — the OS may write the superblock to disk before the data pages it references. Splitting into two fsyncs enforces "all data durable BEFORE superblock durable." A crash between them leaves the previous (intact) superblock active. -4. **Round-robin write to `txn_counter % N`.** The "active" superblock is whichever slot has the highest valid counter. Writing to `txn_counter % N` always targets the stalest slot; the previously-active slot stays untouched. A torn write can damage the new superblock but cannot damage any of the N-1 last-known-good ones. Higher N (configurable 2..=16) trades disk space for survival of consecutive torn-write retries. (See R4.) -5. **Promote in-memory state LAST.** Until the superblock fsync returns, the transaction is not durable. Updating `committed_roots` before that point would make in-memory state lie about durability — a subsequent reader could see uncommitted handles. +1. **Pre-drain flush before `persist_freemap`** — clears every dirty pin so the strict cap is reachable via normal eviction, so a mid-commit `CacheFull` can't be silently promoted to fatal. Cost: one extra `fsync`. (I28) +2. **`persist_freemap` allocates structural COW pages BEFORE merging** — structural targets are sourced out-of-band (one-commit-deferred recycle pool or file extension), NEVER from the freemap's own bitmap (that would recurse without termination). (I18) +3. **Two separate fsyncs (data then superblock)** — enforces "all data durable BEFORE superblock durable"; `fsync` does not order writes within itself. +4. **Round-robin write to `txn_counter % N`** — always targets the stalest slot; a torn write cannot damage any of the N-1 last-known-good slots. N configurable 2..=16. (R4) +5. **Promote in-memory state LAST** — until the superblock fsync returns, the transaction is not durable; promoting earlier would make in-memory state lie about durability. ### What happens on failure at each step @@ -230,6 +242,8 @@ A Chisel file is a sequence of fixed-size 8 KB pages. The first N pages (where ` Every page ends with an 8-byte XXH3 checksum over bytes `0..CHECKSUM_OFFSET` (= bytes `0..8184`). `PageCache` validates the checksum on every cache miss; cache hits skip revalidation because the in-memory bytes are trusted between writes (the exclusive `flock` keeps any other Chisel-or-cooperating process from scribbling on the file). A checksum mismatch on load is fatal (`ChecksumMismatch`). +(For encrypted databases the on-disk stride is 8232 bytes per page; see [On-disk encryption](#on-disk-encryption).) + `flock` is POSIX-advisory: cooperating processes (any other Chisel instance, or any tool that honours advisory locks) respect it; a tool that bypasses advisory locking — `cp` during a transaction, naive backup scripts, some sync utilities — can still corrupt the file even with Chisel holding the lock. The single-writer model assumes external respect for the lock; see README's "Platform support" section for the user-facing version of this caveat. ### Common page header @@ -282,10 +296,15 @@ bytes | field | type 308..312 | superblock_count | u32 LE (= N, in 2..=16) 312..320 | root_membership_index_page | u64 LE (PAGE_ID_NONE if no index) 320..324 | freemap_depth | u32 LE (0 = single-page/depth-0) -324..8184 | reserved (zeroed for forward compat)| [u8; ~7860] +324..8184 | reserved / crypto region | see below 8184..8192 | XXH3 checksum | u64 LE ``` +Bytes **324..8184** are conditional on whether the database is encrypted: + +- **Plaintext DB:** the whole range is reserved (zeroed for forward compat), and every sensitive body field above (root pointers, `total_pages`, `next_handle`, `freemap_depth`, `named_roots`) is stored in plaintext at its offset. +- **Encrypted DB:** bytes **324..1356** hold the plaintext crypto-header + 8-slot key-slot table (8-byte prefix at 324..332, slots at 332..1356). The sensitive body fields are NOT plaintext — they are **sealed under the DEK** as a `nonce ‖ tag ‖ ciphertext` sub-blob. The plaintext portion (magic, format version, txn counter, page size, superblock count, crypto-header) retains its XXH3 checksum so `select()` still works before any decryption. See the [On-disk encryption](#on-disk-encryption) subsection for the exact key-slot record layout. + `Superblock::select` reads up to `MAX_SUPERBLOCKS` (= 16) candidate pages, calls `deserialize` on each (which fails fast on bad checksum / wrong magic / out-of-range `superblock_count`), and `max_by_key`s on `txn_counter`. Ties break by lowest slot index (deterministic but rare in practice — only seen during the `create_new` seeding window before the first user commit). The `superblock_count` field being **in every slot** is what lets `open_existing` discover N at recovery time without out-of-band metadata: read the first MAX_SUPERBLOCKS pages blindly, the winning slot tells you N. Higher N (3..16) trades 8 KB per slot for survival of consecutive torn writes (see R4). @@ -479,7 +498,7 @@ bytes | field | type 8184..8192 | XXH3 checksum | u64 LE ``` -`allocate_first` returns the lowest free page id by scanning bytes for non-zero values and using `trailing_zeros` to find the bit. (An `allocate_near(target)` radius-scan variant existed but was removed in the PR #46 dead-code sweep — no caller used it.) +`allocate_first` returns the lowest free page id by scanning bytes for non-zero values and using `trailing_zeros` to find the bit. ```text FreeMapInterior page (PageType = 0x07) @@ -507,21 +526,19 @@ The tree is consumed during commit's `persist_freemap`: pages freed during the t ### Handle stability -A handle is a `u64` returned by `allocate()`. Handles are assigned monotonically from `next_handle` (a counter in the superblock) and **never reused** within a database's lifetime, even after delete. Delete writes a tombstone (`HandleFlags::Deleted`) into the leaf entry; the slot stays allocated, the page stays valid, but `lookup` reports `Ok(None)` and the user-facing API returns `InvalidHandle`. This permanent-burn policy is what makes handles safe to embed in long-lived references (e.g., from another data structure or another database) without worrying about a stale handle pointing at unrelated data after a delete-and-realloc cycle. +A handle is a `u64` returned by `allocate()`. Handles are assigned monotonically from `next_handle` (a counter in the superblock) and **never reused** within a database's lifetime, even after delete. Delete writes a tombstone (`HandleFlags::Deleted`) into the leaf entry; the slot stays allocated, the page stays valid, but `lookup` reports `Ok(None)` and the user-facing API returns `InvalidHandle`. This permanent-burn policy makes handles safe to embed in long-lived references without a stale handle later pointing at unrelated data after a delete-and-realloc cycle. The radix-tree indirection means values can move freely on disk — `update()` to a larger value, `defrag()` consolidation, future page-format upgrades — without changing the handle the caller holds. -Within-session iteration stability follows from that same handle identity. `handles()` and `handles_with_tag()` walk arithmetic radix trees in a structure-only traversal, so within one open instance repeated scans return an identical `Vec` — same handles, same order — as long as the live set is unchanged and no `defrag` has run. This is a *repeatability* guarantee only: the order itself is unspecified (it is not promised to be sorted, and may differ after a reopen or `defrag`, or across versions), which keeps the index internals free to change. The guarantee is deliberately scoped to a single session and does not survive reopen or `defrag`; it rests on the radix-depth re-derivation invariant (see [In-memory radix depth is re-derived from the root](#in-memory-radix-depth-is-re-derived-from-the-root-never-stored)) — a rolled-back grow must restore depth or a later scan would mis-enumerate. +Within-session iteration stability follows from that same handle identity. `handles()` and `handles_with_tag()` walk arithmetic radix trees in a structure-only traversal, so within one open instance repeated scans return an identical `Vec` — same handles, same order — as long as the live set is unchanged and no `defrag` has run. This is a *repeatability* guarantee only: the order itself is unspecified (not promised sorted, may differ after a reopen or `defrag`, or across versions). The guarantee is scoped to a single session and does not survive reopen or `defrag`; it rests on the radix-depth re-derivation invariant (see [In-memory radix depth is re-derived from the root](#in-memory-radix-depth-is-re-derived-from-the-root-never-stored)). ### Per-module copy-on-write -Chisel does not have a centralized COW abstraction. Each layer-4 / layer-5 module that mutates pages (handle_table, freemap_tree during persist) implements COW by allocating fresh pages, writing the new state into them, and returning the new root id to the caller. The previously-committed page is left untouched on disk; it remains valid and reachable through the previously-committed superblock for the entire duration of the new transaction. - -This per-module pattern is deliberate. A monolithic COW abstraction was considered and rejected — it would have forced every page-type module to express its mutations through a uniform interface, and the modules' actual COW shapes are different enough (handle table walks a tree; freemap_tree COWs the touched leaf+spine of a radix tree; data pages reuse the same page across multiple commits via `claim_page`) that a generic interface would have leaked detail. +Chisel does not have a centralized COW abstraction. Each layer-4 / layer-5 module that mutates pages (handle_table, freemap_tree during persist) implements COW by allocating fresh pages, writing the new state into them, and returning the new root id to the caller. The previously-committed page is left untouched on disk; it remains valid and reachable through the previously-committed superblock for the entire duration of the new transaction. (Why per-module rather than a monolithic COW abstraction: rationale in THEORY.md.) ### In-memory radix depth is re-derived from the root, never stored -Both radix trees — the handle table and the membership index's outer tree — keep their current depth as an in-memory field (`HandleTable.depth`, `MembershipIndex.outer_depth`) that is NOT carried in `Roots` and so not in the superblock; it is derivable by walking the left spine from the root (each `grow()` installs the old root at child 0). `RadixU64::recover_depth` / `HandleTable::recover_depth` are those walks. Every path that restores a root must re-derive the depth or the in-memory descent depth disagrees with the page it descends: on OPEN (seed both depths from the roots) and on ROLLBACK / rollback_to (after `current_roots` rewinds, re-derive both from the restored roots — a rolled-back `grow()` shrinks the tree by a level; a stale-deep depth would mis-descend and return `InvalidHandle` for committed handles, or mis-enumerate a tag). This was a real silent-corruption bug: it surfaced first in the membership index during chunk-tags development and was recognized as the same root cause in the handle table. The handle-table half is **I99**, the membership half **C1**; both fixes extract the open-time spine walk into a reusable `recover_depth` called from both rollback paths. +Both radix trees — the handle table and the membership index's outer tree — keep their current depth as an in-memory field (`HandleTable.depth`, `MembershipIndex.outer_depth`) that is NOT carried in `Roots` and so not in the superblock; it is derivable by walking the left spine from the root (each `grow()` installs the old root at child 0). `RadixU64::recover_depth` / `HandleTable::recover_depth` are those walks. **Invariant:** every path that restores a root must re-derive the depth, or the in-memory descent depth disagrees with the page it descends. This means: on OPEN (seed both depths from the roots) and on ROLLBACK / rollback_to (after `current_roots` rewinds, re-derive both from the restored roots — a rolled-back `grow()` shrinks the tree by a level; a stale-deep depth would mis-descend and return `InvalidHandle` for committed handles, or mis-enumerate a tag). The handle-table half is **I99**, the membership half **C1**. (Bug history in THEORY.md.) ### Chunk tags (the membership index in use) @@ -553,7 +570,7 @@ The client byte is a single opaque `u8` stored in entry byte `[15]`. Chisel stor The page cache enforces a strict cap (`Options::cache_max_bytes`). When the cache is full and every entry is dirty (so nothing is evictable), overflow dirty pages spill to a sidecar file `.spillway` rather than returning `CacheFull`. The spillway is bounded by `Options::spillway_max_bytes` (default `1024 × cache_max_bytes` = 8 GiB at the 8 MiB cache default); `SpillwayFull { limit_bytes }` fires when both the cache and the spillway are exhausted. Setting `spillway_max_bytes = 0` disables the spillway and restores `CacheFull`-at-cap semantics. -Spillway slots carry their own per-slot XXH3 checksum over `page_id || page_bytes`, distinct from the main-file page checksum, so a corrupt spillway slot is detected on rehydrate. The spillway is never `fsync`ed — its content does not need to survive a crash; it's truncated at open and at every commit/rollback. A crash with a non-empty spillway just discards its contents on the next open, which is correct because anything in the spillway was uncommitted dirty state. +Spillway slots carry their own per-slot XXH3 checksum over `page_id ‖ page_bytes`, distinct from the main-file page checksum, so a corrupt spillway slot is detected on rehydrate. The spillway is never `fsync`ed — its content does not need to survive a crash; it's truncated at open and at every commit/rollback. A crash with a non-empty spillway just discards its contents on the next open, which is correct because anything in the spillway was uncommitted dirty state. The no-spill commit cost is **3 fsyncs**: pre-drain flush (I28) + main-pages flush + superblock. The pre-drain handles a subtle interaction in the commit protocol (see [Commit protocol](#commit-protocol) step 1). @@ -585,15 +602,13 @@ The fixed table size (8 entries × 32-byte slots) is intentional: it keeps the s ### Defragmentation -`defrag()` consolidates sparse data pages: it identifies pages whose live-slot count falls below a threshold and re-inserts their live values, freeing the source pages for reclamation. Defrag runs *inside* an active transaction so it composes with other work and is atomic on commit — this is intentional, not an oversight; the alternative ("auto-begin / auto-commit") would have made defrag impossible to schedule alongside a larger maintenance batch. +`defrag()` consolidates sparse data pages: it identifies pages whose live-slot count falls below a threshold and re-inserts their live values, freeing the source pages for reclamation. Defrag runs *inside* an active transaction so it composes with other work and is atomic on commit. The cap parameter (`DefragOptions::max_pages`) bounds the number of *values* relocated in one pass, despite the legacy name (kept for API stability; see C4 in ISSUES.md). ### Poison model -On any fatal error — an `IoError` from `fsync`, a `ChecksumMismatch` on a page load, a `CorruptSuperblock` on open, any error raised after the commit protocol has begun — the `TransactionManager` becomes **poisoned**. Every subsequent call returns `ChiselError::Poisoned`, including reads. The only legal recovery is to drop the `Chisel` handle and call `Chisel::open` again; the shadow-paging recovery path then returns the database to its last-durable state. - -This mirrors `std::sync::Mutex` poisoning. It is mandatory because Linux fsyncgate (post-2018) makes retrying a failed `fsync()` unsafe — the kernel may have discarded the dirty pages already, and a subsequent successful `fsync()` does not mean earlier data is durable. The reopen-to-recover idiom exercises the same code path as crash recovery, which has the side benefit of testing the recovery path on every real-world poison event. (See I1 for the full design and I29 for what `UnsupportedFormatVersion` means under the packed scheme.) +On any fatal error — an `IoError` from `fsync`, a `ChecksumMismatch` on a page load, a `CorruptSuperblock` on open, a `DecryptionFailed`, any error raised after the commit protocol has begun — the `TransactionManager` becomes **poisoned**. Every subsequent call returns `ChiselError::Poisoned`, **including reads**. The only legal recovery is to drop the `Chisel` handle and call `Chisel::open` again; the shadow-paging recovery path then returns the database to its last-durable state. (See I1; rationale — fsyncgate, the Mutex analogy — in THEORY.md.) ### Engine-activity counters @@ -603,143 +618,79 @@ Three semantic conventions matter: - **Counters reset on close + reopen**, because `PageCache` and `PageIo` are reconstructed. There is no persistent counter state on disk — the in-memory `Cell` is the entire record. - **Misses, allocations, and hit increments record *attempts*, not successes.** `cache_misses` is incremented before `load_page` (so a checksum-mismatch error still records the miss); `pages_allocated` is incremented before `maybe_evict` (so a `CacheFull` allocation still records the attempt). `fsync_calls` is the asymmetric exception: it counts only *successful* fsyncs, because a failed fsync poisons the engine (I1) and the counter on a poisoned engine has no defined further meaning. -- **Reads via `Chisel::counters()` are `&self`** and do not mutate. The bench harness reads counters before and after a measurement and reports the delta; that's the primary intended consumer, but the counters are also useful for ad-hoc debugging ("how many cache misses did this query cause?"). +- **Reads via `Chisel::counters()` are `&self`** and do not mutate. The bench harness reads counters before and after a measurement and reports the delta; that's the primary intended consumer, but the counters are also useful for ad-hoc debugging. -The counter set is fixed at four for v1 of the instrumentation (PR 1 of the bench-suite series). `#[non_exhaustive]` on `ChiselCounters` keeps the door open for adding a fifth counter later without a breaking change. +The counter set is fixed at four for v1 of the instrumentation. `#[non_exhaustive]` on `ChiselCounters` keeps the door open for adding a fifth counter later without a breaking change. ### Format versioning (two-tier) Chisel versions its on-disk format at two levels. -- **File level** (I29): the superblock carries a packed `format_version` u32 — upper 16 bits MAJOR, lower 16 bits MINOR. Open-time gate compares MAJOR only. Any same-major file opens regardless of minor; a different-major file is rejected with `UnsupportedFormatVersion`. This is what makes the README's "sacred within a major version" promise enforceable. -- **Page level** (I31): each non-superblock page carries a one-byte `page_format_version` in its header (byte 1 for Data/Overflow/FreeMap; byte 2 for HandleTable, where byte 1 holds the FLAG byte). This lets individual page layouts evolve within a major without a file-wide bump. The current value is `0` everywhere. The post-1.0 upgrade plan is lazy migration: reads *will* dispatch on the version byte (the per-module decode helpers and `page::page_format_version()` exist but are **dormant today** — `PageCache::load_page` validates only the XXH3 checksum and nothing reads the version byte yet), writes always stamp the current version, and an opt-in eager upgrader (deferred) sweeps remaining old pages. +- **File level** (I29): the superblock carries a packed `format_version` u32 — upper 16 bits MAJOR, lower 16 bits MINOR. Open-time gate compares MAJOR only. Any same-major file opens regardless of minor; a different-major file is rejected with `UnsupportedFormatVersion`. This is what makes the README's "sacred within a major version" promise enforceable. (Plaintext DBs stamp MAJOR = 1; encrypted DBs stamp MAJOR = 2 — see [On-disk encryption](#on-disk-encryption).) +- **Page level** (I31): each non-superblock page carries a one-byte `page_format_version` in its header (byte 1 for Data/Overflow/FreeMap/Membership; byte 2 for HandleTable, where byte 1 holds the FLAG byte). This lets individual page layouts evolve within a major without a file-wide bump. The current value is `0` everywhere. The post-1.0 upgrade plan is lazy migration: reads *will* dispatch on the version byte (the per-module decode helpers and `page::page_format_version()` exist but are **dormant today** — `PageCache::load_page` validates only the XXH3 checksum and nothing reads the version byte yet), writes always stamp the current version, and an opt-in eager upgrader (deferred) sweeps remaining old pages. -Both schemes leave reserved space for forward compatibility — the superblock has bytes 324..8184 reserved (after the `freemap_depth` field at 320..324), and every non-superblock page has bytes 8..16 reserved (8 bytes / 64 bits) for future common-header fields. +Both schemes leave reserved space for forward compatibility — the superblock has bytes 324..8184 reserved (after `freemap_depth` at 320..324; used by the crypto-header on encrypted DBs), and every non-superblock page has bytes 8..16 reserved (8 bytes / 64 bits) for future common-header fields. ### On-disk encryption -Chisel supports optional authenticated encryption of database files. An encrypted database is indistinguishable from random bytes to a reader without the key; each page is individually authenticated, so corruption (accidental or deliberate) is detected before any plaintext is returned. +Chisel supports optional authenticated encryption of database files (shipped; MAJOR = 2). An encrypted database is indistinguishable from random bytes without the key; each page is individually authenticated, so corruption (accidental or deliberate) is detected before any plaintext is returned. (Threat-model discussion, nonce-reuse-negligibility argument, and DEK/KEK envelope rationale live in THEORY.md.) -**Cipher.** XChaCha20-Poly1305 (IETF extended-nonce variant). Each page write generates a fresh random 192-bit nonce; the extended nonce space (2¹⁹²) makes nonce reuse under shadow-paging page reassignment negligible in practice (spec §2.1). The on-disk layout per encrypted page is `ciphertext(8192) ‖ tag(16) ‖ nonce(24)` = 8232 bytes (`ENC_PAGE_SIZE`). The additional data (AAD) for each page is the 8-byte little-endian `page_id`, which binds ciphertext to its slot and prevents a valid block from being relocated to another page position without detection. +**Cipher.** XChaCha20-Poly1305 (IETF extended-nonce variant). Each page write generates a fresh random 192-bit nonce. The on-disk layout per encrypted page is `ciphertext(8192) ‖ tag(16) ‖ nonce(24)` = 8232 bytes (`ENC_PAGE_SIZE`). The additional data (AAD) for each page is the 8-byte little-endian `page_id`, which binds ciphertext to its slot and prevents a valid block from being relocated to another page position without detection. -**On-disk stride.** Encrypted databases use a uniform 8232-byte stride for every page including the superblock slots. Plaintext databases continue to use the 8192-byte stride; the two are mutually exclusive and the stride is recorded in the superblock's plaintext crypto-header so the engine reads the correct number of bytes before attempting any operation. The `page_io` layer is stride-agnostic: callers set the stride once (via `PageIo::set_stride`) and all subsequent raw reads and writes use it. +**On-disk stride.** Encrypted databases use a uniform 8232-byte stride for every page including the superblock slots. Plaintext databases use the 8192-byte stride; the two are mutually exclusive and the stride is recorded in the superblock's plaintext crypto-header (bytes 325..329) so the engine reads the correct number of bytes before any operation. The `page_io` layer is stride-agnostic: callers set the stride once (via `PageIo::set_stride`) and all subsequent raw reads/writes use it. -**Envelope (key hierarchy).** A random 256-bit per-database encryption key (DEK) encrypts all page content. The DEK itself is never stored in plaintext: it is wrapped under a key-encryption key (KEK) and the wrapped form is held in a plaintext key-slot table inside the superblock's reserved region (bytes 324..1356; 8 slots × 128 bytes each, preceded by an 8-byte prefix (1-byte algorithm id, 4-byte stride, 3 reserved bytes); the key-slot table begins at byte 332). Each slot stores the KDF identity, KDF parameters, salt, wrap nonce, wrapped DEK, and wrap tag. There are two KEK derivation paths: +**Envelope (key hierarchy).** A random 256-bit per-database DEK encrypts all page content. The DEK is never stored in plaintext: it is wrapped under a key-encryption key (KEK) and the wrapped form is held in a plaintext key-slot table inside the superblock's reserved region. The crypto-header + key-slot table occupy bytes **324..1356**: + +```text +Crypto-header (in the superblock's reserved region) + +bytes | field +-----------|----------------------------------------------------- +324..325 | algorithm (u8; 1 = XChaCha20-Poly1305, 0 = plaintext) +325..329 | stride (u32 LE; 8232 for encrypted) +329..332 | reserved (zero) +332..1356 | 8 key-slot records × 128 bytes each + +Key-slot record (128 bytes; trailing bytes reserved/zero) +0 | state (u8; 1 = active, 0 = empty) +1 | kdf_id (u8; 1 = HKDF, 2 = Argon2id) +2..14 | argon2 params: m_cost(u32) | t_cost(u32) | p_cost(u32) (zero for HKDF) +14..30 | salt (16) +30..54 | wrap_nonce (24) +54..86 | wrapped_dek (32) +86..102 | wrap_tag (16) +102..128 | reserved +``` + +Two KEK derivation paths: - **Raw key** (`Key::Raw`): KEK = HKDF-SHA256(ikm=key material, salt=slot salt, info=`"chisel-kek"`). - **Passphrase** (`Key::Passphrase`): KEK = Argon2id(password, salt, m/t/p from the slot's stored parameters). The DEK wrapping uses detached XChaCha20-Poly1305 with AAD bound to the slot's KDF metadata, so an attacker cannot swap a slot's KDF parameters to force mis-derivation without breaking the tag. -**Superblock body protection.** The superblock's sensitive body — root pointers (`root_handle_table`, `root_freemap_page`, `root_tag_map_page`), `total_pages`, `next_handle`, `freemap_depth`, and the `named_roots` name table — is sealed under the DEK as a `nonce ‖ tag ‖ ciphertext` sub-blob whose AAD binds it to the superblock's identity. The plaintext portion of the superblock (magic, format version, txn counter, page size, superblock count, crypto-header) retains its XXH3 checksum so the A/B torn-write selector (`select()`) still works before any decryption. +**Superblock body protection.** The sensitive body — root pointers (`root_handle_table`, `root_freemap_page`, `root_membership_index_page`), `total_pages`, `next_handle`, `freemap_depth`, and the `named_roots` table — is sealed under the DEK as a `nonce ‖ tag ‖ ciphertext` sub-blob whose AAD binds it to the superblock's identity. The plaintext portion (magic, format version, txn counter, page size, superblock count, crypto-header) retains its XXH3 checksum so the A/B torn-write selector (`select()`) still works before any decryption. -**Format version.** Encrypted databases stamp file-level **MAJOR = 2, MINOR = 0**. Plaintext databases remain at MAJOR = 1. The existing open-time gate (which rejects any file whose MAJOR differs from the compiled-in `FORMAT_MAJOR_VERSION`) therefore hard-rejects an encrypted database on an encryption-unaware binary with `UnsupportedFormatVersion`, preventing ciphertext from being silently misread as page data. No per-page (I31) format change is needed — the logical page image is unchanged. +**Format version.** Encrypted databases stamp file-level **MAJOR = 2, MINOR = 0**; plaintext stays MAJOR = 1. The open-time gate hard-rejects an encrypted database on an encryption-unaware binary with `UnsupportedFormatVersion`, preventing ciphertext from being silently misread as page data. No per-page (I31) change is needed — the logical page image is unchanged. -**Key management.** Credential rotation is O(1) and crash-safe — it never re-encrypts any page: +**Key management.** Credential rotation is O(1) and crash-safe — it never re-encrypts any page. Each operation is a normal superblock commit through the A/B + fsync protocol: - `add_key(old_key, new_key)`: derives a new KEK, wraps the same DEK into a free slot, then commits. - `rotate_key(old_key, new_key)`: `add_key` followed by clearing the old slot in the same commit. - `remove_key(key)`: clears the matching slot, refusing to clear the last active slot (which would make the database permanently unreadable). -Each operation is a normal superblock commit through the A/B + fsync protocol. Bulk DEK rotation (re-encrypting every page under a fresh DEK — relevant only when the DEK itself is believed compromised) is deferred; see I142. - -**Spillway.** For encrypted databases the in-memory spillway carries sealed blobs: pages are encrypted exactly once on eviction from the page cache (`seal` on evict-to-spillway) and copied verbatim — without decryption or re-encryption — on drain to the main file. Rehydration from the spillway decrypts the blob back into the cache. This means no plaintext page content is ever written to disk by an encrypted database, even during spill. - -**Threat-model boundary.** Provided: confidentiality of all user data and sensitive metadata at rest; AEAD tamper-detection per page and per superblock body (any modification surfaces as the fatal `DecryptionFailed` error, which poisons the engine); anti-relocation (AAD = `page_id` prevents transplanting a ciphertext block to a different slot). Not provided: rollback/replay resistance (an attacker who substitutes a wholly older, validly-signed database image cannot be detected without an external monotonic trust anchor such as a TPM); in-memory protection beyond `zeroize`-on-drop for the DEK and page plaintext; traffic-analysis resistance (file size, page count, and access patterns are visible). - ---- - -## Benchmark infrastructure - -The `bench/` subcrate is a workspace member (listed in `members` and `default-members`, so a root `cargo test` runs it) that path-deps on the root `chisel` crate. It provides three measurement layers comparing Chisel against [redb](https://github.com/cberner/redb) and SQLite: - -1. **Cross-engine equivalence tests** — five scenarios × three engines × snapshot/restore checks, asserting that all three engines produce identical observable state for the same workload. Catches semantic divergence in the workload-replay machinery before it contaminates measurement. -2. **Criterion micro-grid** — six rows of small-scoped operations (single-tx allocate, point-read, single-tx update at small batch sizes), 165 cells of wall-clock + file-size + Chisel-internal-counter metrics. Drives the `Engine` trait through tight loops. -3. **YCSB-style scenario tier** — four end-to-end workloads (YCSB-A 50/50 read/update Zipfian; YCSB-B 95/5 read-heavy Zipfian; Mutation Log 25/25/25/25 alloc/read/update/delete uniform; Document Store 70/20/10 read/alloc/update with log-normal sizes). Timed with `Instant::now()` rather than Criterion — Criterion's many-samples-per-bench model exceeds the 1-6 minute scenario budget. - -A post-processor (`chisel-bench-summarize`) reads scenario metrics + Criterion archive data and emits three artifacts: per-cell `summary.md`, flat `results.json` (composite-key schema for the CI diff binary), and `cross-engine.md` (a per-metric Chisel/redb/SQLite comparison: throughput, p99 latency, file size). A diff binary (`chisel-bench-diff`) consumes two `results.json` files and posts a sticky regression-report comment on each PR. - -### macOS fsync semantics - -On macOS, Chisel calls `fcntl(F_FULLFSYNC)` via Rust's `sync_all` — durable through the disk's write cache. SQLite's default `fsync()` on macOS only flushes to the disk's write cache without `F_FULLFSYNC`, so unmodified `SqliteEngine` runs ~3 orders of magnitude faster than `ChiselEngine` on `Strict` durability. The bench harness closes this gap by issuing `PRAGMA fullfsync=ON` in `SqliteEngine::open_file` for `Strict` mode (no `#[cfg(target_os)]` gate — Linux ignores the pragma). With the fix, both engines pay the same per-commit `F_FULLFSYNC` cost on macOS, and Linux runs are unchanged. - -Without the fix, comparing chisel-strict vs sqlite-strict on macOS measures Apple-vs-Apple disk-cache semantics, not engine performance. With the fix, the comparison reflects the engines themselves. - -### Counter-driven measurement - -Engine activity is observable via `Chisel::counters()` (cumulative-from-open: cache hits/misses, pages allocated, fsyncs). The micro-grid records counter snapshots before/after each cell so the post-processor can attribute throughput differences to fsync count, cache pressure, or page-allocation rate. This is why `ChiselCounters` is `#[non_exhaustive]` — the bench harness reads these via the public API, but additional counters can be added without a breaking change. - -The asymmetry "fsync_calls counts only successes; everything else counts attempts" matters here: a `CacheFull` allocation still bumps `pages_allocated`, but a failed fsync poisons the engine and stops counter increments. The bench harness handles poisoning by aborting that cell rather than fudging the numbers. - ---- - -## Implementation history - -This section is a date-stamped narrative of the larger pieces of work that landed in the engine and the bench harness. The intent is to give a future reader (human or AI) the context to understand *why* the code looks the way it does — the running decision log lives in `ISSUES.md`, but the prose context for each major thrust lives here. - -### Benchmark suite (PRs 1–8) - -The bench-suite series ran from 2026-04-30 through 2026-05-04 and landed in eight PRs against `main`: - -- **PR 1 (2026-04-30)** — counter instrumentation. Added the four `Chisel::counters()` fields (`cache_hits`, `cache_misses`, `pages_allocated`, `fsync_calls`) as `Cell` increments at the site of each operation. `ChiselCounters` is `#[non_exhaustive]` so future counters can be added without a breaking change. Documented in [Engine-activity counters](#engine-activity-counters). -- **PR 2 (2026-04-30)** — `bench/` subcrate + `Engine` trait + `ChiselEngine`. The `bench/` directory was a sibling subcrate at this point (I61 later made it a `default-members` workspace member); it path-deps on the root `chisel` crate. -- **PR 3 (2026-04-30)** — `RedbEngine` + `SqliteEngine` + cross-engine equivalence tests (five scenarios × three engines = 15 tests). SQLite snapshot-restore required `Engine::flush_for_snapshot()` (default no-op; SQLite override does `journal_mode=DELETE`) because WAL mode leaves committed data in the `-wal` sibling between explicit checkpoints — `std::fs::copy` of the main `.db` alone otherwise yields "database disk image is malformed" on reopen. -- **PR 4a (2026-04-30)** — workload data layer. `Operation` / `Workload` types plus six seeded generators in `bench/src/workload.rs`, ChaCha8Rng-pinned for cross-version reproducibility. -- **PR 4b (2026-05-01)** — Runner machinery + 6-row Criterion micro grid in `bench/src/runner.rs` + `bench/benches/micro_grid.rs`. Produces 165 cells of wall-clock + file-size + Chisel-internal-counter metrics into `target/criterion/...` and `bench/results/aux_metrics.jsonl`. The original PR 4 from the master spec was split into 4a + 4b once it became clear ~600 LOC in one PR was less reviewable than two smaller PRs. - - The 4b grid is 6 rows, not the 9 the master spec called for: three 1000-per-tx variants (update, delete, delete_many) were dropped during implementation because 1000 random ops over the prepopulated DB pin a working set of dirty pages exceeding Chisel's pre-spillway 2048-page cache ceiling. The dropped row functions remain in `micro_grid.rs` (with `#[allow(dead_code)]`) so they can be re-enabled in a future PR with a configurable larger cache. - -- **PR 5 (2026-05-03)** — markdown summary post-processor. Binary `chisel-bench-summarize` in `bench/src/bin/summarize.rs` plus a library module under `bench/src/summary/`. Reads Criterion's `sample.json` per cell plus `bench/results/aux_metrics.jsonl` and emits three artifacts under `bench/results//`: `summary.md` (per-row markdown tables with magnitude-adaptive units), `results.json` (flat composite-key schema for PR 7's CI diff), and `raw/` (archival copy of estimates.json + sample.json per cell). Percentiles are computed directly from `sample.json` per-iteration times via numpy-style linear interpolation (consistent p50/p95/p99 semantics rather than mixing Criterion's bootstrap median with a CI proxy). -- **PR 6 (2026-05-03)** — scenario tier. Four YCSB-style end-to-end workloads in `bench/src/scenarios.rs` + `bench/benches/scenarios.rs`, driven by `run_scenario_cell` in `bench/src/runner.rs`. YCSB-A (50/50 read/update, Zipfian θ=0.99), YCSB-B (95/5), Mutation Log (25/25/25/25 alloc/read/update/delete uniform), Document Store (70/20/10 read/alloc/update with lognormal sizes, Zipfian θ=0.7). Each runs once per strict durability mode → 12 cells. Inline `Instant::now()` timing rather than Criterion (the master-spec budget of 1–6 minutes per full tier rules out Criterion's many-samples-per-bench model). - - Three latent bugs surfaced at PR 6's end-to-end acceptance gate that no per-task unit test caught: (1) `run_scenario_cell` originally did one-allocate-per-tx during prepop (100K fsyncs on chisel-strict ≈ 12 min/cell on macOS APFS); fixed by mirroring PR 4b's byte-accumulator chunking. (2) `gen_mutation_log` generated Read/Update/Delete on indices without tracking which had been deleted; replaced with a state-aware walk maintaining a live-set `Vec`. (3) `discover_cells` errored `NoCellsFound` when the criterion dir was empty even with scenarios present; `summarize.rs` now lets the unified `cells.is_empty() && scenarios.is_empty()` gate decide. - - Runtime caveat: the spec target was 1–6 minutes / 10 minutes ceiling. On macOS that ceiling is unreachable — Chisel uses Rust's `sync_all` which calls `fcntl(F_FULLFSYNC)` (durable through the disk cache), while SQLite by default uses plain `fsync()` (which on macOS only flushes to the disk's write cache without `F_FULLFSYNC`). Result: chisel-strict cells are fsync-bound at ~5–10 ms per commit while sqlite-strict cells run ~3 orders of magnitude faster. Full 12-cell grid takes ~70–90 minutes on macOS APFS; Linux CI runners are much faster. -- **PR 7 (2026-05-04)** — CI integration. `chisel-bench-diff` binary at `bench/src/bin/diff.rs` plus `.github/workflows/bench.yml` that runs the scenario tier on each PR, diffs against `main`'s baseline, and posts a sticky regression-report comment. Two-checkout strategy: build + run scenarios on `main`, build + run on PR HEAD, summarize both, run the diff binary, post via `peter-evans/find-comment` + `create-or-update-comment` keyed on the marker ``. Thresholds: throughput + p50 at 5%, p95 + p99 at 10%, worse-direction only, no absolute time floor in v1. Pinned to `ubuntu-latest` per the PR 6 macOS fsync caveat. Signal-only — never blocks merge. - - PR 7's first acceptance gate caught a real environmental issue: `origin/main` was 76 commits behind local `main` because PRs 4–6 were merged locally but never pushed to GitHub. Fix was a single `git push origin main`. Pattern worth remembering: any workflow that does `Checkout main` + build assumes `origin/main` is current. - -- **PR 8 (2026-05-04)** — cross-engine comparison report + macOS-fsync fairness fix. `chisel-bench-summarize` now emits `cross-engine.md` alongside `summary.md` and `results.json` (three per-metric tables: throughput, p99 latency, file size) over the four PR 6 scenarios in strict mode. `SqliteEngine::open_file` issues `PRAGMA fullfsync=ON` for `DurabilityMode::Strict` — no `#[cfg(target_os)]` gate (Linux ignores it; macOS uses `fcntl(F_FULLFSYNC)`), one extra PRAGMA exec at open time. Closes the cross-engine fairness gap that was pre-existing from PR 3's `SqliteEngine` wrapper. - - PR 8's first-run bench-diff signal is a useful calibration point for GitHub-runner variance on the scenario tier: two `document-store` p50 cells flagged as "regressed" (redb-strict +9.4%, chisel-strict +5.6%) while throughput on both was within ±1% (genuine noise on microsecond-scale measurements). Future bench-diff readers should treat ≤±15% deltas on the scenario tier as plausible runner noise rather than real perf signals; the diff binary's job is to surface them, not to gate merges. - -A small followup landed alongside PR 8: `bench.yml` now uploads the PR-side `summarize` output (`cross-engine.md`, `summary.md`, `results.json`) as a workflow artifact `bench-results-pr-` with 90-day retention. Retrieve via: - -``` -gh run download --repo pgexperts/chisel --name bench-results-pr- -``` - -Get `` from `gh run list --branch ` or the PR checks page. The `raw/` Criterion archive is intentionally absent from scenario-tier output — scenarios use `Instant::now()` timing rather than Criterion. Main-side output is not uploaded; for absolute README/release-notes numbers, run on dedicated hardware rather than the shared CI runner. - -Master design spec at `docs/superpowers/specs/2026-04-25-chisel-benchmark-suite-design.md` covers PRs 1–7; PR 8 has its own spec/plan pair at `docs/superpowers/specs/2026-05-04-chisel-bench-cross-engine-design.md` and `docs/superpowers/plans/2026-05-04-chisel-bench-cross-engine.md`. - -### Spillway feature (2026-05-04) - -The spillway landed out-of-band from the bench-suite series on the same day as PR 8. It adds `src/spillway.rs` plus integration across `PageCache` (spill on dirty overflow, rehydrate on miss, drain under the existing fsync, truncate on rollback) and the public API (`Chisel::set_cache_max_bytes` / `set_spillway_max_bytes` / `set_drain_insertion`). - -Breaking changes: -- `Options::cache_size: usize` (page count) → `Options::cache_max_bytes: u64` (bytes); default unchanged at 8 MiB. -- New `Options::spillway_max_bytes` (default `1024 × cache_max_bytes` = 8 GiB; 0 disables the spillway and restores legacy `CacheFull`-at-cap semantics). -- New `Options::drain_insertion` (`LruTail` default | `Mru`). -- The pre-existing 8× `HARD_CEILING_MULTIPLIER` elasticity is removed. - -The bench engine (`bench/src/chisel_engine.rs`) was updated mid-PR to enable the spillway by default. The original "spillway disabled for cross-engine fairness" reasoning was backwards: SQLite uses a temp file for transaction overflow, redb uses on-disk btrees; disabling Chisel's spillway makes Chisel the only engine that fails on big transactions, which is the unfair config. - -Spec/plan at `docs/superpowers/specs/2026-05-03-chisel-spillway-design.md` + `docs/superpowers/plans/2026-05-04-chisel-spillway.md`. Engine-side description in [Cross-cutting concepts → Spillway](#spillway). - -### Lessons captured during the spillway rollout +Bulk DEK rotation (re-encrypting every page under a fresh DEK) is deferred; see I142. -Three engineering lessons surfaced during the spillway PR that are worth remembering for future cross-cutting work: +**Spillway.** For encrypted databases the in-memory spillway carries sealed blobs: pages are encrypted exactly once on eviction (`seal` on evict-to-spillway) and copied verbatim — no decryption or re-encryption — on drain to the main file. Rehydration decrypts the blob back into the cache. No plaintext page content is ever written to disk by an encrypted database, even during spill. -1. **Per-task `cargo test` from the repo root did NOT run the bench subcrate's tests (at the time).** Bench was a sibling crate, not a workspace member. `cd bench && cargo test` was documented separately, but per-task gates skipped it. The final whole-PR review caught the missed bench test failures. Tracked as I58 in ISSUES.md and since RESOLVED: I58/I61 made `bench/` a `default-members` workspace member, so a root `cargo test` now covers it. -2. **A breaking change in cache discipline ripples to every consumer that papered over a different limitation.** The bench engine had been quietly relying on the 8× elasticity as a substitute for proper transaction-overflow handling. Removing the elasticity exposed the missing config; the right fix was to give Chisel the spillway (production parity), not to keep it disabled and lower other budgets. -3. **No-spill commit cost is 3 fsyncs, not 2.** I28 pre-drain flush + main-pages flush + superblock. The spillway spec called it "two-fsync" because the spec author was thinking only of the spillway's contribution (zero); the actual baseline was already 3. The `no_spill_workload_preserves_two_fsync_commit` test now pins to `== 3` with documentation of the protocol so a future reader knows what each fsync covers. +**Tamper detection.** Any modification to a page or the superblock body surfaces as the fatal `DecryptionFailed` error, which poisons the engine. --- ## Glossary - **COW (copy-on-write)** — every mutation writes to a fresh page rather than modifying an existing one. The previously-committed page stays valid until the superblock swap promotes the new state. +- **DEK / KEK** — Data Encryption Key (encrypts all pages) and Key Encryption Key (wraps the DEK). The DEK is stored only in wrapped form in the superblock's key-slot table. See [On-disk encryption](#on-disk-encryption). - **Handle** — a stable `u64` returned by `allocate()`. Survives `update()`, `defrag()`, and reopen. - **HandleEntry** — the 16-byte record in a handle-table leaf describing one handle's `(page_id, slot_index, flags)`. - **Inline value** — a value small enough to live in a data-page slot directly. Larger values overflow. @@ -747,12 +698,13 @@ Three engineering lessons surfaced during the spillway PR that are worth remembe - **Operational error** — a `ChiselError` variant indicating the caller made a mistake or hit a transient condition; the database is fine. `is_fatal()` returns false. - **Overflow chain** — a singly-linked sequence of overflow pages holding one large value. Owned exclusively by one handle. - **PAGE_ID_NONE** — `u64::MAX`. Sentinel meaning "not yet allocated" for root pointers (handle-table root, freemap root). -- **PageType** — the 1-byte tag at offset 0 of every non-superblock page. Values: `0x01` HandleTable, `0x02` Data, `0x03` Overflow, `0x04` FreeMap, `0x05` MembershipInterior, `0x06` MembershipLeaf. `0x00` is reserved so a zeroed page cannot masquerade as a valid type. +- **PageType** — the 1-byte tag at offset 0 of every non-superblock page. Values: `0x01` HandleTable, `0x02` Data, `0x03` Overflow, `0x04` FreeMap, `0x05` MembershipInterior, `0x06` MembershipLeaf, `0x07` FreeMapInterior. `0x00` is reserved so a zeroed page cannot masquerade as a valid type. - **Poison** — the state a `TransactionManager` enters after any fatal error. Every subsequent call returns `Poisoned` until the handle is dropped and the database reopened. - **Shadow paging** — the durability technique Chisel uses: writes go to new pages; commit swaps a superblock pointer; old state stays intact for crash recovery. - **Slot packing (R1)** — multiple values per data page. Each value occupies one slot; the slot directory grows forward and value data grows backward from the page's checksum. - **Slot tombstone** — a slot directory entry with `SLOT_FLAG_DEAD`. Reclaimed by `compact()`, not reused by `insert()`. -- **Superblock** — the page (one of N slots at the file head) that names the current handle-table root, freemap root, membership-index root, named roots, and durability metadata. Picked by `Superblock::select` on open. The membership-index root is a fourth root that swaps atomically with the others on each commit. +- **Stride** — the on-disk unit size read/written per page: `PAGE_SIZE` (8192) plaintext, `ENC_PAGE_SIZE` (8232) encrypted. Recorded in the crypto-header; set on `PageIo` via `set_stride`. +- **Superblock** — the page (one of N slots at the file head) that names the current handle-table root, freemap root, membership-index root, named roots, and durability metadata. Picked by `Superblock::select` on open. - **Tombstone (handle)** — a `HandleEntry` with `HandleFlags::Deleted`. The slot stays allocated; the handle is permanently retired (never reused). See "permanent-burn policy" in [Handle stability](#handle-stability). - **txn_counter** — monotonically-increasing u64 in every committed superblock. Used by `select` to pick the winner across slots and by the round-robin to decide which slot to write next. - **Watermark rollback (I3)** — the rollback strategy: cache + file are truncated to `committed_roots.total_pages`. Pages allocated during the transaction (id ≥ watermark) get dropped; freemap-reused pages (id < watermark) get their dirty cache entries discarded. No undo log. Rollback also re-derives the handle-table and membership-index depths from the restored roots (those in-memory radix depths are not part of the snapshot; I99 / C1). diff --git a/README.md b/README.md index 40bf358..0a51b1d 100644 --- a/README.md +++ b/README.md @@ -19,6 +19,7 @@ Pre-1.0. Current release: `0.1.0`. The API is stable-by-intent but subject to re - **Named roots** — a small fixed table in the superblock mapping string names to handles. Survives commit / rollback transactionally. - **Defragmentation** — explicit `defrag()` consolidates sparse pages and returns a count-based stats record. - **In-memory mode** — same engine, `Vec`-backed I/O, no file and no lock. For tests, benchmarks, and ephemeral work. +- **At-rest encryption** — optional, off by default. Every page is sealed with XChaCha20-Poly1305 AEAD under a client-supplied key (raw 32-byte or Argon2id passphrase). An 8-slot envelope table wraps the data-encryption key, so credential rotation is O(1) — no bulk re-encryption. - **Poison model** — any fatal error (I/O failure, checksum mismatch, commit-protocol failure) poisons the handle; recovery is drop-and-reopen. Mirrors `std::sync::Mutex` poisoning. - **Single-writer** — exclusive `flock` at the filesystem level; `&mut self` on every mutating method. @@ -75,12 +76,14 @@ Cargo workspace with three members: the root `chisel` engine, the `python/` PyO3 ```bash cargo build -cargo test -cargo clippy -- -D warnings +cargo test # engine + bench (default-members); do NOT use --lib +cargo clippy --workspace -- -D warnings # --workspace also lints the python/ binding cargo fmt -- --check ``` -CI runs the Rust checks above plus a Python matrix (CPython 3.11 and 3.13 × Linux/macOS) that builds the PyO3 binding via `maturin develop` and runs `pytest` in `python/tests`. A separate workflow builds abi3 wheels on tagged releases. +`cargo test` from the workspace root covers the engine and the bench crate (`default-members = [".", "bench"]`). Do **not** use `cargo test --lib` — it skips the `tests/` integration suite. MSRV is verified with a library-only build under the pinned toolchain: `cargo +1.82 build -p chisel`. + +CI runs the Rust checks above plus a Python matrix (CPython 3.11 and 3.13 × Linux/macOS) that builds the PyO3 binding via `maturin develop --release` and runs `pytest -v` in `python/tests`. A separate workflow builds abi3 wheels on tagged releases. ### Python binding @@ -89,9 +92,9 @@ The `python/` subcrate is a PyO3 wrapper (`chisel-py` → `_chisel.abi3.so`) wit ```bash cd python python -m venv .venv && source .venv/bin/activate -pip install maturin pytest -maturin develop -pytest +pip install maturin pytest hypothesis # the `test` extra requires pytest>=8 and hypothesis>=6 +maturin develop --release +pytest -v ``` See [`python/README.md`](python/README.md) for usage. @@ -215,6 +218,34 @@ let mut db = Chisel::open_in_memory()?; For tuned options (cache size, superblock count), use `Chisel::open_in_memory_with_options(options)`. +### Encryption + +Encryption is opt-in and driven entirely through `Options::encryption_key`. A key is supplied as a `chisel::Key`: + +- `Key::Raw(bytes)` — high-entropy key material (32 bytes), stretched to a key-encryption key via HKDF-SHA256. +- `Key::Passphrase(string)` — a human secret, stretched via Argon2id (memory-hard). Cost comes from `Options::argon2_params` on create, or the stored slot's params on reopen. + +Both variants zeroize their material on drop. + +```rust +use chisel::{Chisel, Key, Options}; +use std::path::Path; +use zeroize::Zeroizing; + +// Create (or reopen) an encrypted database. +let key = Key::Passphrase(Zeroizing::new("correct horse battery staple".into())); +let mut db = Chisel::open( + Path::new("secret.db"), + Options::default().encryption_key(key), +)?; +``` + +On create, Chisel generates a random data-encryption key (DEK), encrypts every page under it with XChaCha20-Poly1305, and wraps the DEK under a key-encryption key derived from your supplied key. On reopen, the supplied key must unwrap one of the on-disk key slots or `open` fails with `InvalidEncryptionKey`. Supplying a key to a plaintext DB returns `EncryptionNotSupported`; omitting it on an encrypted DB returns `NoEncryptionKey`. + +The wrapped DEK lives in an **8-slot key table**. Because the DEK itself never changes, credential rotation only re-wraps the DEK in a slot — it is O(1), independent of database size. `add_key` stages a second credential (both open the DB), `rotate_key` replaces one credential in place, and `remove_key` retires one (refusing the last remaining slot with `LastKeySlot`). A full table returns `NoFreeKeySlot`. + +See [ARCHITECTURE.md#on-disk-encryption](ARCHITECTURE.md#on-disk-encryption) for the on-disk layout (crypto header, key slots, per-page nonce stride) and [THEORY.md](THEORY.md) for the rationale behind the envelope scheme and the shadow-paging nonce discipline (with [ISSUES.md](ISSUES.md) as the dated decision log). + ## API reference | Method | Purpose | @@ -248,7 +279,11 @@ For tuned options (cache size, superblock count), use `Chisel::open_in_memory_wi | `handles()` | Enumerate all live handles; repeatable within a session, order unspecified (takes `&self`) | | `stats()` | Handle count, page count, file size (takes `&self`) | | `counters()` | Engine-activity counters: cache hits/misses, fsync calls, pages allocated (takes `&self`) | +| `file_size_bytes()` | Physical size of the database file in bytes (takes `&self`) | | `defrag(options)` | Consolidate sparse pages | +| `add_key(existing, new)` | Stage a second credential; both `existing` and `new` then open the DB. `Result<()>` | +| `rotate_key(old, new)` | Replace credential `old` with `new` in place. `Result<()>` | +| `remove_key(key)` | Retire credential `key`; `LastKeySlot` if it is the only one. `Result<()>` | ## Options @@ -262,26 +297,34 @@ let options = Options { create_if_missing: true, read_only: false, superblock_count: 2, // 2..=16; only consulted on create + encryption_key: None, // Some(key) to create/open an encrypted DB + argon2_params: None, // None = OWASP defaults; only used on create }; ``` +`Options` is `#[non_exhaustive]`, so build a customized value with the chained setters rather than a struct literal from another crate: `Options::default().cache_max_bytes(N).encryption_key(key)`. Every field has a matching setter, including `Options::encryption_key(key)` and `Options::argon2_params(params)`. + `cache_max_bytes` is a strict cap on the in-memory LRU cache. When the cache is full and a dirty page cannot be evicted, overflow dirty pages spill to a sidecar `Spillway` file rather than returning an error. The spillway file is bounded by `spillway_max_bytes` (default 8 GiB). Setting `spillway_max_bytes = 0` disables the spillway entirely, restoring the pre-spillway `CacheFull` semantics at the strict cache cap: the operational error `CacheFull` fires when the cache is full and no eviction is possible. With the spillway enabled, exhausting both the cache and the spillway returns `SpillwayFull { limit_bytes }` (also operational; caller recovers by committing or rolling back). `read_only = true` still acquires an exclusive `flock` — it only suppresses writes at the application layer. Two read-only opens cannot coexist on the same file. This is a deliberate choice: even a reader must block concurrent writers to keep the shadow-paging invariants intact. `superblock_count` is set at create time and stored on disk; reopening discovers it from the winning superblock. Higher N increases durability against consecutive torn writes at the cost of N × 8 KB of file space: N = 3 survives one torn commit plus a torn retry, N = 4 survives two retries. +`encryption_key` defaults to `None` (plaintext). `Some(key)` creates a new encrypted database (sealing a fresh random data-encryption key under the supplied key) or reopens one (unwrapping the stored key). `argon2_params` defaults to `None`, which uses the OWASP-recommended Argon2id cost (19 MiB / t=2 / p=1) when deriving a key from a `Key::Passphrase` on create; it is ignored for raw keys and on reopen (the stored slot carries its own params). See the [Encryption](#encryption) concept below. + ## Error handling `ChiselError` splits into two conceptual tiers. **Operational errors** — the database is healthy; the caller made a mistake. Catch and continue. -`InvalidHandle`, `TagMismatch`, `NoActiveTransaction`, `TransactionAlreadyActive`, `TransactionInProgress`, `SavepointNotFound`, `DuplicateSavepoint`, `ReadOnlyMode`, `FileNotFound`, `InvalidRootName`, `RootNameTableFull`, `InvalidSuperblockCount`, `CacheFull`, `SpillwayFull`. +`InvalidHandle`, `TagMismatch`, `NoActiveTransaction`, `TransactionAlreadyActive`, `TransactionInProgress`, `SavepointNotFound`, `DuplicateSavepoint`, `ReadOnlyMode`, `FileNotFound`, `InvalidRootName`, `RootNameTableFull`, `InvalidSuperblockCount`, `CacheFull`, `SpillwayFull`, `NoEncryptionKey`, `InvalidEncryptionKey`, `EncryptionNotSupported`, `NoFreeKeySlot`, `LastKeySlot`. **Fatal errors** — storage integrity is in question. Drop the handle and reopen. -`IoError`, `ChecksumMismatch`, `CorruptSuperblock`, `FileSizeMismatch`, `LockFailed`, `UnsupportedFormatVersion`, `CorruptPage`, `InvalidPageId`, `Poisoned`. +`IoError`, `ChecksumMismatch`, `CorruptSuperblock`, `FileSizeMismatch`, `LockFailed`, `UnsupportedFormatVersion`, `UnsupportedPageSize`, `CorruptPage`, `InvalidPageId`, `DecryptionFailed`, `Poisoned`. + +`DecryptionFailed { page_id }` is fatal: an AEAD authentication failure while decrypting an already-read page means the ciphertext or session key can no longer be trusted, so it poisons the handle exactly like `ChecksumMismatch` (see the poison model below). It is distinct from the operational `InvalidEncryptionKey`, which fires at open time when the supplied key unwraps no key slot — before any data page is served. Use `ChiselError::is_fatal()` to classify at runtime. @@ -313,13 +356,15 @@ Versioning is two-tiered. **File level** — each superblock carries a packed `format_version` u32: upper 16 bits = MAJOR, lower 16 bits = MINOR. The open-time gate compares MAJOR only. A 1.3 binary opens a 1.7 file cleanly, but a 1.3 binary rejects a 2.0 file. Minor bumps within a major are reserved for additive changes, so older binaries can safely *read* newer-minor files. The chunk-tags feature is the first such additive minor bump (MINOR 0 → 1): it adds a per-chunk tag and a membership index, and pre-tag files open cleanly with every chunk untagged. +Encryption introduces the second major: an encrypted database is stamped MAJOR = 2 (`FORMAT_MAJOR_VERSION_ENCRYPTED`), because its on-disk layout carries a crypto header and every page is ciphertext — an encryption-unaware or older binary cannot make sense of it. The MAJOR-only open gate therefore hard-rejects a MAJOR = 2 file with `UnsupportedFormatVersion` rather than misreading it, exactly as it rejects any future incompatible major. Plaintext databases stay at MAJOR = 1. + **Page level** — each non-superblock page carries a one-byte `page_format_version` in its header, letting individual page layouts evolve within a major without a file-wide format bump. The post-1.0 upgrade story is lazy migration: on read, the page-type module dispatches on its page's declared version; on write, it always produces the current version; cold pages stay in the old layout until an opt-in `db.upgrade()` sweep rewrites them. An additional 8 bytes are reserved in every non-superblock page header for future common-header fields. Write safety across minors is a narrower guarantee: a binary at MINOR = *m* opening a file at MINOR = *m' > m* cannot safely commit without risking overwriting fields it doesn't know about. The open gate is MAJOR-only by design, so minor variants coexist — same-major files of any minor open successfully, and the chunk-tags MINOR = 1 variant is the first such case. The write-refusal arm (refuse writes when file MINOR > binary MINOR, leaving the newer-minor file read-only) is not yet wired up; it lands with the first post-1.0 minor bump that makes the direction observable. The post-1.0 cross-minor read-compatibility guarantee is absolute; write-compatibility requires binary MINOR ≥ file MINOR. ### Pre-1.0 caveat -Until Chisel reaches 1.0, the on-disk format may change between pre-release builds without a major-version bump. Any such pre-1.0 change will be called out in release notes. The first 1.0 release freezes MAJOR at 1 for the entire 1.x line. +Until Chisel reaches 1.0, the on-disk format may change between pre-release builds without a major-version bump. Any such pre-1.0 change will be called out in release notes. The first 1.0 release freezes the plaintext format at MAJOR = 1 for the entire 1.x line; encrypted databases carry MAJOR = 2 (see above), and each major's on-disk format is sacred within that major. Files written by prior development builds (pre-1.0 flat `format_version`, which decodes as MAJOR = 0) are rejected at open time — recreate the database. No production-grade migration is provided for pre-release files. @@ -348,9 +393,24 @@ A PyO3 wrapper lives in the `python/` subdirectory and will be published to PyPI The Python API mirrors the Rust one but adds context managers for transactions and savepoints. +Encryption is exposed through the `open()` `encryption_key` keyword (default `None`): pass `bytes` for a raw 32-byte key or `str` for a passphrase. The three key-management methods take the same `bytes | str` credential vocabulary: + +```python +import chisel + +db = chisel.open("secret.db", encryption_key="correct horse battery staple") + +db.add_key(existing="correct horse battery staple", new=b"\x00" * 32) # stage a 2nd credential +db.rotate_key(old=b"\x00" * 32, new="new passphrase") # replace in place +db.remove_key("correct horse battery staple") # retire one +``` + +`add_key` / `rotate_key` raise `NoFreeKeySlotError` when the 8-slot table is full; `remove_key` raises `LastKeySlotError` rather than leaving the DB with no usable credential. + ## Design documents -- [`ARCHITECTURE.md`](ARCHITECTURE.md) — living architecture overview: layer model, commit protocol, recovery, full on-disk format byte-by-byte, and cross-cutting concepts. Start here if you're reading the codebase for the first time. +- [`ARCHITECTURE.md`](ARCHITECTURE.md) — living architecture overview: layer model, commit protocol, recovery, full on-disk format byte-by-byte, and cross-cutting concepts. Start here if you're reading the codebase to *act* on it. +- [`THEORY.md`](THEORY.md) — theory of operation: *why* the design is what it is — the load-bearing decisions, the rejected alternatives, and the implementation history. Read this to build a durable model before changing the engine. - [`ISSUES.md`](ISSUES.md) — running decision log: open issues, closed issues, and every design tradeoff with date-stamped rationale. ## License diff --git a/THEORY.md b/THEORY.md new file mode 100644 index 0000000..993adfe --- /dev/null +++ b/THEORY.md @@ -0,0 +1,258 @@ +# Chisel — Theory of Operation + +This document is for a human engineer who is about to *change* Chisel. It is the "why" companion to two siblings: + +- [`ARCHITECTURE.md`](ARCHITECTURE.md) — the cold-start map: the layer model, the exact on-disk byte format, the enumerated load-bearing invariants, the commit-protocol step list. When this document needs to point at a layout or an invariant, it cross-references ARCHITECTURE rather than restating it. +- [`README.md`](README.md) — what Chisel does and how to use it. + +Read this one **once, slowly**. It is narrative, not reference. It exists to build a durable mental model — the shape of the decisions and the tradeoffs behind them — so that when you later open the code you already understand why it is the way it is. After that first read, come back to it only when you need to remember *why* something is the way it is; use ARCHITECTURE for *what* it is. + +A note on the division of labor: wherever you feel the urge to check "what exactly is at byte 324" or "what are all the fatal-error variants," that belongs in ARCHITECTURE and you will find it there. This document deliberately does not repeat those tables, because duplicated content drifts and a cross-reference cannot. + +--- + +## What Chisel is + +Chisel is an **embedded, single-writer, copy-on-write (shadow-paging) transactional slot store** written in Rust. You hand it a byte value, it gives you back a stable `u64` handle; you hand it the handle later, it gives you back the value. Values can be updated, deleted, tagged into groups, and defragmented, all inside ACID transactions that survive a crash. There is no server, no network, no query language — it is a library that owns one file. + +One through-line explains most of the design: **the recovery path *is* the normal path.** Chisel never writes a log that must be replayed on crash. Instead, every mutation writes fresh pages and leaves the last-good state fully intact on disk; a commit atomically swaps a small pointer (the superblock) to the new state. Opening a database — any database, crashed or clean — runs exactly the same "pick the winning superblock" procedure. There is no separate crash-recovery subsystem to get wrong, because there is nothing to recover: the file is always in a consistent state, either the new one (if the commit's final fsync landed) or the previous one (if it didn't). + +Three enforced commitments follow from that and shape everything downstream. They are stated as facts in [ARCHITECTURE.md#design-commitments](ARCHITECTURE.md#design-commitments); the rest of *this* document is their justification. + +- **Single-writer, embedded** — one process, `&mut self` mutators, no internal locking. +- **Shadow paging, not WAL** — fresh pages plus a superblock swap; recovery is superblock selection. +- **Durability over performance** — fsync on every commit, a checksum on every page, and a poison model that treats any fsync failure as terminal. + +If you internalize only one thing: Chisel spends disk space and write amplification to buy *provability*. You can convince yourself of its crash-safety by inspection, without simulating a log replay in your head. Almost every "why not the faster thing?" answer below reduces to "the faster thing costs us that provability." + +--- + +## The load-bearing decisions + +What follows is the spine of the document: the eighteen design decisions that, if reversed, would make Chisel a different system. Each one names what was **chosen**, what was **rejected** (usually the more instructive half), and **why**. Sources are cited as `ADR-N`, spec dates, or `I` issue numbers so you can go deeper. + +### Durability model: shadow paging over WAL (ADR-1) + +**Chosen.** Every mutation allocates a fresh page. Commit writes the new pages, fsyncs, writes a new superblock into a *different* slot, and fsyncs again. Recovery is `Superblock::select` scanning N candidate slots and picking the highest valid `txn_counter`. + +**Rejected.** A write-ahead log with in-place updates — the PostgreSQL/SQLite standard. WAL recovery is a substantial subsystem in its own right: a replay state machine, checkpoint handling, log truncation. That machinery adds a risk surface comparable to *the entire rest of Chisel*. A hybrid (WAL for small changes, shadow for large) was also rejected, as combining the worst of both — the recovery paths multiply and the mode boundary becomes another correctness obligation nobody asked for. + +**Why.** Shadow paging trades disk space (you hold the live and previous version of every mutated page until commit) for code simplicity. There is no log to replay; the "is this database openable?" check *is* the crash-recovery path; crash safety is provable by inspection. The known cost is accepted with eyes open: write amplification (a one-byte change costs a full 8 KB page) and the eventual need for `defrag.rs` to reclaim sparse pages over time. See [ARCHITECTURE.md#commit-protocol](ARCHITECTURE.md#commit-protocol) for the mechanism. + +### Single-writer, enforced by the type system (ADR-2) + +**Chosen.** One process, one writer. The invariant is enforced at three levels at once: an OS exclusive `flock` in `page_io.rs`, `&mut self` on every mutating API, and an explicit statement that this is philosophical, not a v1 shortcut. Reads take `&self` because the cache lives behind a `RefCell`, not a `Mutex`. + +**Rejected.** Multi-writer with an internal `RwLock`/`Mutex` plus conflict detection and deadlock handling — roughly doubles engine complexity. MVCC — adds version chains, garbage collection, and snapshot-isolation semantics, all out of scope for an embedded store. And crucially, "single-writer now, multi-writer later" was rejected: relaxing `&mut self` after the fact is a 2.0 breaking change for *every* consumer, not a minor bump, so deferring the decision doesn't actually defer the cost. + +**Why.** With `&mut self`, "two concurrent transactions" is impossible to even *express* — so there is no test for it, because there is no API for it. The type system encodes the invariant in a way internal locking never could. Choosing `RefCell` over `Mutex` is a direct consequence: with no concurrency to guard against, a `Mutex` would be pure overhead and a lie about the model. (See the memory note `project_chisel_single_client_design` — this really is a design stance, not a limitation to be lifted.) + +### Per-module copy-on-write, no central abstraction (ADR-3) + +**Chosen.** Each module implements its own COW. `handle_table.rs` clones the root-to-leaf path; the freemap tree allocates fresh structural pages during `persist_freemap`; `data_page.rs` mutates in place via `claim_page` and reuses the same page across commits. There is no `trait Cow`, no `enum CowStrategy`. + +**Rejected.** A centralized COW trait spanning all page-type modules. It was rejected because the modules' actual COW shapes differ enough that a uniform interface would be either too generic (losing the useful per-module detail) or too specific (sprouting variants that serve exactly one module). A generic page-mutation wrapper around a strategy object hit the same wall, plus it would leak generics into the public API. + +**Why.** Co-locating COW with each page-type's logic lets each evolve independently — the handle table grew its `grow()` and short-circuit optimizations without touching the freemap or the data pages. The accepted cost is real: some repeated boilerplate across three or four modules, and an onboarding surprise. New readers look for "the COW abstraction" and there isn't one — deliberately. If you are that reader right now: this is the answer. See [ARCHITECTURE.md#per-module-copy-on-write](ARCHITECTURE.md#per-module-copy-on-write). + +### N rotating superblocks for atomic commit (ADR-4) + +**Chosen.** N superblock slots (`Options::superblock_count`, range `2..=16`, default 2) occupy file offsets `0..N`. Commit writes to slot `txn_counter % N`; recovery validates all N (magic, checksum, count-in-range) and picks the highest valid `txn_counter`. + +**Rejected.** A single superblock overwritten each commit, protected by a PostgreSQL-style double-write buffer. Rejected because shadow paging *already* double-protects the data pages, so only the superblock itself needs torn-write protection, and N rotating slots are simpler than standing up a separate buffer area. A write-side mini-WAL just for the superblock fell to the same complexity argument as ADR-1. + +**Why.** A single superblock overwritten in place is vulnerable to a torn write leaving the file unrecoverable. Rotating slots give trivial torn-write recovery — a torn slot fails its checksum and is ignored — need no separate journal (the slots *are* the journal), and offer a configurable space/durability tradeoff: higher N survives more *consecutive* torn writes. The cost is N pages of overhead at the file head and the fact that changing N for an existing file needs migration. The commit always writes the *stalest* slot, so a torn write can only damage a slot you were about to discard anyway; see the round-robin reasoning in [ARCHITECTURE.md#commit-protocol](ARCHITECTURE.md#commit-protocol). + +### Poison model on fatal errors (ADR-6, I1) + +**Chosen.** Any fatal error — a commit-path `IoError`, a `ChecksumMismatch`, a `CorruptSuperblock`, a `DecryptionFailed`, anything raised after the commit protocol has begun (see `ChiselError::is_fatal()`) — sets a poison flag on the `TransactionManager`. Every subsequent call, *including reads*, returns `Poisoned`. The only legal recovery is to drop the handle and `Chisel::open` again. + +**Rejected.** Retrying the failed fsync — rejected outright per fsyncgate (below). Poisoning writes but letting reads continue — rejected because reads share the page cache with writes, so a corrupt page may already have been served, and there is no way to know *which* reads are tainted. Auto-reopen on poison — rejected because reopen is a substantial state transition (file descriptors, locks, cache) that the caller must orchestrate; forcing it silently would hide a boundary the caller needs to see. + +**Why.** This is the fsyncgate decision, and it is worth understanding rather than memorizing. On Linux since 2018, a failed `fsync()` cannot be safely retried: the kernel reports the error on the *next* fsync and then *clears* its error state, so a later "successful" fsync does **not** mean the earlier data is durable — those dirty pages may already have been dropped from the page cache. PostgreSQL responds by PANICking on fsync failure for exactly this reason. Chisel takes the same stance, and it costs almost nothing here because shadow paging plus embedded single-writer makes reopen cheap — and reopen exercises the *same* recovery code path a real crash would, which is a testing win. The design mirrors `std::sync::Mutex` poisoning: once the invariant might be broken, refuse to pretend otherwise. See the memory note `project_chisel_i1_poison_decision` and [ARCHITECTURE.md#poison-model](ARCHITECTURE.md#poison-model). + +### Spillway sidecar over a hard cache ceiling (ADR-5) + +**Chosen.** A sidecar file `.spillway` absorbs LRU-tail dirty pages when the cache is full of dirty (un-evictable) pages, turning the cache into a strict bound rather than an elastic one. It is bounded by `Options::spillway_max_bytes` (default `1024 × cache_max_bytes`). Each slot carries its own XXH3 over `page_id ‖ page_bytes`; it is *never* fsynced and is truncated at open and at every commit/rollback. Setting `spillway_max_bytes = 0` disables it and restores `CacheFull`-at-cap. + +**Rejected.** The pre-2026-05-04 design had a `HARD_CEILING_MULTIPLIER = 8` — the cache could balloon to 8× its nominal size before erroring. That was rejected because real workloads legitimately need to dirty far more than 8× the cache (a document-store benchmark with log-normal value sizes can dirty 100×). Simply *raising* the multiplier only delays the same failure. Unbounded growth was rejected as an OOM path that violates the memory budget. Spilling into a reserved area of the *main* file was rejected because that becomes a permanent on-disk artifact needing its own checksum and durability management. + +**Why.** The key insight — the reason the spillway is *simple* — is that its contents are, by definition, uncommitted. So it never needs to be fsynced: a crash just discards it, and the prior committed superblock stays active. That single fact ("never fsync the spillway") is what makes the whole feature correct and cheap. The accepted cost: a no-spill commit is now three fsyncs (the I28 pre-drain, the main-pages flush, and the superblock), not two, plus a new `SpillwayFull` error. The public-API break was `Options::cache_size` (a page count) becoming `cache_max_bytes` (a byte budget). See [ARCHITECTURE.md#spillway](ARCHITECTURE.md#spillway) and spec `2026-05-03-chisel-spillway-design`. + +### In-memory mode via a `Vec`-backed `PageIo` (ADR-8) + +**Chosen.** `Chisel::open_in_memory` runs the *full* engine against a `Vec`-backed `PageIo` — no filesystem, no `flock` — exposed as `chisel.open(None)` in Python. Same code path, same guarantees, except durability. + +**Rejected.** A tmpfs or OS-level mock filesystem — still incurs kernel-call overhead and real FS semantics, whereas a `Vec` removes the OS from the loop entirely and is faster. A separate `MemChisel` type — rejected because it would duplicate every method and drift from the real one. A public `Backend` trait — rejected because the internal `Backing` enum (File / Memory) already *is* that, and exposing it through a constructor keeps the public API smaller. + +**Why.** Tests, benchmarks, and ephemeral workloads shouldn't pay for disk I/O or touch the filesystem — but running them down the *same* code path means an in-memory test catches bugs that would also bite on disk. The one cost worth remembering: counters reset on close+reopen, because a `Vec` doesn't persist. + +### Counter instrumentation via `Chisel::counters()` (ADR-9) + +**Chosen.** Four cumulative-from-open counters — `cache_hits`, `cache_misses`, `pages_allocated`, `fsync_calls` — each a `Cell` at its increment site, aggregated into a `#[non_exhaustive] ChiselCounters` read via `&self`. Misses, allocations, and hits count *attempts*; `fsync_calls` counts only *successes*. + +**Rejected.** Internal logging/tracing — rejected because it forces log-parsing on the consumer, whereas counters give a structured, allocation-free read. Latency histograms — out of scope for v1 (a histogram-crate dependency and per-op overhead); the bench harness computes distributions externally instead. A configurable counter set — rejected as complexity for marginal benefit, with `#[non_exhaustive]` keeping future additions cheap anyway. + +**Why.** Wall-clock time is too noisy for *component-level* analysis — you cannot tell a cache-hit-rate regression from an fsync-rate regression from a timer. The bench harness reads counters before and after each scenario cell and reports the delta, giving per-cell attribution. `fsync_calls` counts only successes for a principled reason: a failed fsync poisons the engine, and a counter on a poisoned engine has no defined further meaning. See [ARCHITECTURE.md#engine-activity-counters](ARCHITECTURE.md#engine-activity-counters). + +### Chunk tags and the reverse membership index (ADR-12) + +**Chosen.** An optional immutable `u32` tag per chunk, fixed at allocation (`tag == 0` is the untagged sentinel). The *forward* map (handle → tag) lives in four of the `HandleEntry`'s five reserved bytes, so `tag(handle)` is O(1) with no extra page read. The *reverse* map (tag → {handles}) is a two-level COW radix (`MembershipIndex` over `RadixU64`): an outer tree keyed by tag whose leaf bit-packs `(inner_depth | inner_root)`, and per-tag inner trees keyed by handle. `delete_with_tag(tag, max)` is the bounded relation-drop primitive. + +**Rejected.** Storing the group id inside the value and scanning all chunks — O(all chunks) per scan or drop, the exact cost the feature exists to avoid. A single flat radix over a packed `(tag:handle)` key — rejected for the two-level form, which reuses the u64-radix shape *and* makes "enumerate the distinct tags" O(T). A `u64` tag — rejected because it wouldn't fit the reserved entry bytes, and fitting there is what makes forward storage *free*. Mutable tags — deferred, because immutability removes the retag path entirely (retagging becomes allocate-new-plus-delete-old). Bitmap inner sets — deferred, since per-tag sets are expected sparse, making a radix the right default. + +**Why.** The relational client needs three operations without an O(all chunks) pass: scan a relation, drop a relation, and delete-a-chunk-and-remove-it-from-its-set. Forward storage in the reserved entry bytes is *free* — no side table, no extra page reads — which is precisely why `u32` (which fits) beat `u64` (which doesn't). The reverse map reuses the existing radix machinery, COW discipline, superblock-anchored root, freemap reclaim, and poison model, adding *no* new commit-protocol surface. Untagged chunks cost nothing and are invisible to non-users. See [ARCHITECTURE.md#chunk-tags-the-membership-index-in-use](ARCHITECTURE.md#chunk-tags-the-membership-index-in-use) and spec `2026-06-02-chunk-tags-design`. + +### Within-session iteration-stability contract (ADR-13) + +**Chosen.** Promote an already-true property to a *documented, tested* guarantee: within a single open instance, repeated `handles()` / `handles_with_tag(tag)` calls return an identical `Vec` as long as the live set is unchanged and no `defrag` ran. The order itself stays **unspecified** — this is a repeatability guarantee, not an ordering one. No production code changed to add it. + +**Rejected.** Guaranteeing a *specific* order (ascending handle, which the radix walk already happens to produce) — rejected because it would commit the public contract to a particular order and constrain future index internals. Snapshot/MVCC isolation across mutation — out of scope (single-writer, and the requirement explicitly excludes mutation between scans). A wider scope surviving reopen or defrag — rejected in favor of single-session, which constrains internals the least. An internal `debug_assert` sortedness canary — rejected because it would couple internal code to the very ascending behavior the contract deliberately declines to promise. + +**Why.** The relational client wants to scan a relation, do work, and scan again expecting identical results — to re-drive a query, resume a pass, cross-check — without defensively sorting or snapshotting. "Repeatable but opaque" serves that exactly while leaving the radix order, the reopen layout, and defrag reordering all free to change. The guarantee rests on the radix-depth re-derivation invariant below; a rolled-back `grow()` that failed to restore depth would make a later scan mis-enumerate. See [ARCHITECTURE.md#handle-stability](ARCHITECTURE.md#handle-stability) and spec `2026-06-04-stable-chunk-iteration-design`. + +### Client byte — spending the last reserved entry byte (ADR-14) + +**Chosen.** A mutable per-chunk `u8` "client byte" in `HandleEntry` byte `[15]`: `set_client_byte` / `client_byte`, default 0, fully opaque (no search, filter, or index), carried forward across value `update()` and reverted on rollback. No on-disk format change; `FORMAT_MINOR_VERSION` stays 1. + +**Rejected.** Storing the byte with the value on a data page — rejected because it forces a full value rewrite per change and re-couples metadata to value bytes. Making it immutable/set-at-allocation like the tag — rejected because the client needs in-place change. A richer `Handle { id, tag, client_byte }` return type — rejected as a breaking change to every handle-returning signature. A MINOR bump 1→2 for record-keeping — rejected because the layout is byte-identical and there is nothing for a reader to gate on. + +**Why.** Byte `[15]` has *always* been part of the 16-byte entry and always written as 0, so activating a reserved byte is not a versioned change — there is genuinely nothing for a reader to distinguish. Entry-resident storage makes a flip cost one handle-table leaf COW, independent of the value's size. This decision refines ADR-7's versioning rule: reserved bytes are part of the format from creation, and only *new structures or semantics a reader must gate on* warrant a version bump. The one accepted caveat: a pre-feature binary hardcodes `[15] = 0`, so rewriting an entry under an old binary silently clears the byte — acceptable pre-1.0. See [ARCHITECTURE.md#client-byte](ARCHITECTURE.md#client-byte). + +### Two-tier format versioning: file MAJOR/MINOR plus per-page byte (ADR-7, I29/I31) + +**Chosen.** The superblock carries a packed `format_version` u32 (upper 16 bits MAJOR, lower 16 MINOR); the open-time gate compares MAJOR only. Every non-superblock page carries a one-byte `page_format_version`. Version dispatch is per-module and decode-only — the reader branches on the byte; writes always stamp `page::current_version`. A file whose MINOR exceeds the binary's is forced read-only rather than rejected. + +**Rejected.** File-level versioning only — rejected because then every per-page tweak forces a file-wide migration. Page-level only — rejected because it loses the clean "this binary simply cannot read this file" failure. A full schema-migration system (migration scripts, version-jump testing) — out of scope for embedded. + +**Why.** Compatibility genuinely has two granularities — the file's overall layout and any single page-type's layout — and conflating them forces a file-wide bump for a change that touches one page type. Chisel has four-plus page types likely to evolve at different rates. The MAJOR check gives a crisp incompatibility failure; the per-page byte lets one page-type's layout move within a major without migrating the others, with lazy COW-on-write upgrade as the default path. A refinement worth noting (and the seed of ADR-14): a zero-default *additive* field needs no version bump at all — the per-page byte exists only to disambiguate absent-versus-zero where zero is a legitimate value. The read-side dispatch is **dormant today** (nothing reads the byte yet); see [ARCHITECTURE.md#format-versioning-two-tier](ARCHITECTURE.md#format-versioning-two-tier) and spec `2026-06-21-per-page-format-versioning-design`. + +### Multi-page freemap: a COW radix of bitmap leaves (spec 2026-06-22) + +**Chosen.** Generalize the single-page freemap into a COW radix tree of bitmap leaves — a *third* radix alongside the handle table and the membership index. The leaf is today's FreeMap page (`0x04`), unchanged; the interior is a new `FreeMapInterior` (`0x07`); **depth 0 is exactly the current single-page format**. Depth is stored explicitly in the superblock (byte 320), the freemap moved into the page cache, and an in-memory "lowest free" hint accelerates find-first-free. + +**Rejected.** A linked chain of freemap pages — O(n) to scan for a free page in a high range. A fixed two-level directory — re-introduces a (merely higher) ceiling. A `FreeMapFull` guard *instead of* the real feature — rejected: guarding the symptom isn't fixing it. Spine-walk depth recovery (the trick the other two radixes use) — rejected *here specifically* because a sparse freemap makes it ambiguous: a zero pointer near the root looks like a shallow tree, so depth must be stored, not derived. Per-interior "subtree-has-free" summary bits — deferred (YAGNI for v1) in favor of the in-memory hint. + +**Why.** This one has a *bug* behind it. Past 65,344 pages (~512 MB at 8 KB pages) the single-page `FreeMap::mark_free` silently no-ops — reclamation just *stops*, and every freed page leaks forever with **no error surfaced**. A radix-of-bitmaps removes the ceiling entirely (coverage `65,344 × 1021^depth`, growing as `log_1021`) and is the lowest-surprise fix available, because the engine already has two proven COW radix trees to imitate. Moving the freemap into the cache makes per-transaction cost independent of database size. + +There is one **critical, revised** sub-decision that is easy to get wrong and worth carrying in your head: the freemap's *own* structural COW pages must recycle **out-of-band** — from an in-memory one-commit-deferred pool plus file extension — and **never** from the bitmap itself. An earlier draft wrongly assumed bitmap reclamation. The reason it cannot work: sourcing a structural page from a free bit *clears* that bit, which COWs a leaf, which needs another structural page, which clears another bit — an unbounded recursion. Crash-orphaned recycle entries (the pool is in-memory, so a crash loses it) are swept back by a defrag orphan-scan, chosen over persisting the recycle list or accepting a permanent per-crash leak. See [ARCHITECTURE.md#freemap-reclamation](ARCHITECTURE.md#freemap-reclamation) and the memory note `project_chisel_multipage_freemap`. + +### Encryption cipher: XChaCha20-Poly1305 with a random 192-bit nonce (ADR-15) + +**Chosen.** XChaCha20-Poly1305 AEAD, a fresh random 192-bit nonce per page write stored alongside the ciphertext, with AAD = `page_id` (anti-relocation). + +**Rejected.** AES-256-XTS — the FDE standard, length-preserving, zero overhead, no format change — rejected because it provides confidentiality *only*, with no cryptographic tamper-detection; the client wanted authenticated encryption and the one-time format change is free (there are no production databases yet). A deterministic nonce from `(page_id, counter)` — rejected as **unsafe under shadow paging** (see below). AES-256-GCM — rejected because its 96-bit nonce forces either the deterministic-nonce hazard or a stored counter, and it leans on AES-NI for constant time. + +**Why.** Two forces decide this. First, AEAD upgrades integrity from the *forgeable* non-cryptographic XXH3 to an *unforgeable* Poly1305 tag — you get tamper-detection, not just secrecy. Second, and decisively, the 192-bit extended nonce lets random nonces be used *safely*, which matters enormously under Chisel's specific machinery: a crashed transaction discards its writes and returns page_ids to the freemap **while the durable counter does not advance**. So a deterministic `(page_id, counter)` nonce would write *different plaintext* under the *same* `(key, nonce)` after a crash-and-retry — catastrophic keystream reuse. Random nonces have nothing to persist and cannot hit that crash-reuse class at all. ChaCha is also constant-time in portable software with no AES-NI dependence — the right default for an embedded library shipping to unknown hardware — and since crypto throughput sits far below fsync latency, AES-GCM's hardware edge is irrelevant here. This nonce-reuse-under-shadow-paging reasoning is the single most important thing to understand about the crypto layer. See [ARCHITECTURE.md#on-disk-encryption](ARCHITECTURE.md#on-disk-encryption) and spec `2026-06-29-on-disk-encryption-design` §2/§2.1. + +### Encryption keys: envelope DEK/KEK with an 8-slot table, O(1) rotation (ADR-15) + +**Chosen.** A random per-database 256-bit DEK (from `OsRng` at create) seals every page and the sensitive superblock body. The DEK is never stored bare — it is wrapped under a KEK derived from the client key (HKDF-SHA256 for raw keys, Argon2id for passphrases) and held in an 8-slot key-slot table in the superblock's plaintext reserved region. `add_key` / `rotate_key` / `remove_key` re-wrap the *stable* DEK — O(1), no data re-encryption. `rotate_key` stages the new slot before revoking the old (no zero-key window); `remove_key` refuses to clear the last active slot (brick prevention). A successful unwrap *is* proof the client key is correct — there is no separate password verifier. + +**Rejected.** Encrypting only the data pages and leaving the superblock plaintext — rejected because `named_roots` holds user-chosen names, which are real user data a plaintext body would leak. Full DEK rotation (re-encrypting every page under a fresh DEK) — deferred (I142) as a heavy O(total_pages) whole-file operation reserved for "the DEK itself is compromised"; credential rotation is the far more common need and is O(1). + +**Why.** Envelope encryption makes credential rotation O(1) — you re-wrap the DEK — instead of O(database size). The per-slot KDF choice matches input entropy: HKDF is fast and correct for high-entropy keys, while Argon2id is memory-hard to resist brute-forcing low-entropy passphrases (its params are recorded per slot). And every rotation op is an ordinary superblock A/B + fsync commit, so it reuses the existing crash-safe protocol wholesale: a metadata-only `rewrite_crypto_header` commit persists a rotated slot table atomically (write the inactive slot, fsync, promote), so a crash mid-rotation leaves the old table intact. + +Two threat-model boundaries are documented rather than solved, and you should know them before you rely on this: there is **no rollback/replay resistance** (an attacker who substitutes a wholly older, validly-signed image is undetectable without an external trust anchor like a TPM), and the DEK sits in plaintext in process memory during a session (mitigated by zeroize-on-drop, not by encryption). See spec `2026-06-29` §3/§5/§9 and ISSUES.md I142. + +### Encryption page format: 8232-byte stride, logical page stays 8192, MAJOR 1→2 (ADR-15) + +**Chosen.** Encrypted databases use an 8232-byte on-disk stride (8192 ciphertext + 16 tag + 24 nonce) uniformly from birth, including superblock slots (zero-padded). The *logical* page stays 8192, so freemap, data-page, and handle-table geometry are untouched. Encryption is a transform at the page-I/O seam: `PageCache` owns a `PageCipher` and seals once (on flush *or* evict); `page_io` is stride-aware but crypto-agnostic; the spillway holds sealed blobs. Encrypted DBs stamp MAJOR = 2; plaintext stays MAJOR = 1 and byte-identical. + +**Rejected.** Shrinking the logical content to fit a 40-byte trailer *inside* 8192 — rejected because it makes every geometry constant encryption-dependent, an invasive and bug-prone refactor of the engine's core page math. Whole-page-sealing the superblock — rejected because page 0 must stay plaintext-bootstrappable (magic, format version, txn counter, page size, superblock count, and the crypto-header/key-slot table) so the engine can learn the stride and find the key material before anything can be decrypted. A per-page (I31) format change — not required, because the logical 8192 image is unchanged; encryption lives entirely *below* it. + +**Why.** A larger stride is the smallest possible blast radius. The page cache, the freemap, the data pages, the handle table, and the entire transaction layer keep producing and consuming byte-identical 8192-byte pages; only the I/O stride and the seal/open transform change, at a cost of 0.49% larger files. Sealing once (drain is a verbatim byte copy of the already-sealed blob — both spill and main-file writes use `AAD = page_id`, so no re-seal is needed) avoids a crypto round-trip on drain and keeps exactly one ciphertext per write. MAJOR = 2 is the *first real exercise* of ADR-7's MAJOR tier: an encryption-unaware old binary, gating on `MAJOR == 1`, refuses a MAJOR = 2 file rather than misreading ciphertext as plaintext. The inner XXH3 checksum, now redundant with the AEAD tag, is deliberately *kept* so the upper layers stay untouched — a cheap inner sanity check after open. See spec `2026-06-29` §4/§5/§8. + +### MSRV pinned at 1.82 with tilde-pins on edition2024 deps (Cargo.toml, I55/I61/I110) + +**Chosen.** `rust-version = 1.82` in all three subcrates; a CI `msrv` job builds `-p chisel` library-only against `dtolnay/rust-toolchain@1.82`. Dependencies are tilde-pinned to hold that floor: `zeroize ~1.8` (with the derive feature dropped entirely — only `Zeroizing` is used) and the transitive `base64ct ~1.6` (via `argon2 0.5`). The RustCrypto deps are unconditional (seal/open is always compiled) but meaningfully reach the dependency tree only when a database is opened with a key. + +**Rejected.** A floating MSRV on stable — rejected because an unannounced 1.x bump can land silently (README's "Rust stable" is not a pinned promise). Running the msrv job with `--tests` — tried and reverted, because proptest's transitive `getrandom` needs edition2024 (1.85+); the MSRV promise is about the *published library* (no getrandom/proptest in its tree), so the job stays lib-only. Adopting a real Cargo workspace — deferred (I61): members would share edition/rust-version/feature resolution, too restrictive for the PyO3 abi3 binding, and the bench floats its floor faster than the engine wants. + +**Why.** 1.82 is a conservative pin driven by actual language usage (`is_none_or` is 1.82+; the true stdlib floor is nearer 1.74). Pinning gives a stable, published MSRV promise you bump only with a release-notes call-out. The tilde pins are what *hold* that floor: `zeroize 1.9` and `base64ct 1.7+` both adopted edition2024 (Rust 1.85+), which would drag the library MSRV above 1.82; the pinned lines are the last edition-2021 releases, and the un-exercised code paths (no `#[derive(Zeroize)]`, never emitting PHC hash strings) make the pins safe. + +--- + +## Implementation history + +This section is history, framed to explain *why the code looks the way it does* when you open it. It is not a changelog; it is the story that leaves fingerprints on the structure. + +### The benchmark suite, PRs 1–8 + +The benchmark infrastructure was built incrementally across eight PRs, and its shape is deliberate. It lives in a `bench/` subcrate that is a *sibling* to `python/`, not a plain workspace member drawn into the engine's own build in a way that would auto-run its 10–25 minutes of tests on every `cargo test`. (This is also the concrete reason plain `cargo test` versus `cargo test --lib` matters in this repo — see the memory note `feedback_cargo_test_full`.) The subcrate depends on `redb` and `rusqlite` for cross-engine comparison; keeping it a sibling is what keeps the *engine's* dependency graph minimal — the storage engine itself needs neither. + +The suite has three layers, and they exist because no single tool fits all three jobs: + +1. **Cross-engine equivalence tests** — the same operations run against Chisel, redb, and SQLite, asserting identical *results*, so a performance comparison is never comparing engines that secretly disagree on behavior. +2. **A Criterion micro-grid** — 165 cells of small, isolated operations. Criterion's many-samples statistical model is exactly right for micro-measurement. +3. **A YCSB-style scenario tier** — realistic mixed workloads, timed with `Instant::now()` rather than Criterion, because Criterion's sampling model *exceeds* the 1–6 minute scenario budget. This is why you'll see a hand-rolled timing hybrid there instead of "just use Criterion everywhere." + +That three-layer split (ADR-10) and the sibling-crate decision together are why the bench directory looks structurally unlike the rest of the repo. + +### Cross-engine fairness and the macOS fsync problem (ADR-11) + +A subtle fairness bug drove ADR-11. SQLite's default `fsync` on macOS does *not* actually flush to stable storage the way Chisel's does — Chisel uses `F_FULLFSYNC`. If you benchmark them naively, macOS numbers flatter SQLite by comparing Apple's default fsync semantics against Chisel's stricter durability. The fix: `SqliteEngine` issues `PRAGMA fullfsync=ON` for Strict durability, with *no* `cfg(target_os)` gate — Linux simply ignores the pragma, and macOS matches Chisel's `F_FULLFSYNC`. The absence of the platform gate is intentional and is the whole point: it makes the macOS numbers reflect *engine behavior*, not Apple's default fsync. + +### The counter-driven measurement idea + +The instrumentation counters (ADR-9, above) exist *because* the benchmark suite needed them. Wall-clock time can tell you a scenario got slower but not *which component* regressed. The counters — hits, misses, allocations, fsyncs — let the harness read a before/after delta per scenario cell and attribute the change: a rise in `fsync_calls` is a durability-cost story, a fall in the hit rate is a cache story. The counters and the bench harness are two halves of one measurement design. + +### The spillway feature + +The spillway (ADR-5, above) is the clearest case of a benchmark *finding a real ceiling*. The document-store scenario with log-normal value sizes could legitimately dirty ~100× the cache, and the old `HARD_CEILING_MULTIPLIER = 8` turned that into a `CacheFull` error on a workload that was doing nothing wrong. The spillway replaced the elastic ceiling with a strict cap plus a discardable sidecar. When you read the commit protocol and wonder why there are three fsyncs instead of the two the durability story implies, the extra one (the I28 pre-drain) traces directly back to this feature's interaction with `persist_freemap`. + +### Lessons captured + +A few hard-won lessons are worth carrying forward because they recur: + +- **The radix-depth silent-corruption bug.** Both the handle table and the membership index keep their tree depth as an *in-memory* field that is never serialized — it's re-derived by walking the left spine from the root, since each `grow()` installs the old root at child 0. The bug: any path that *restores* a root (open, and especially rollback) must re-derive that depth, or the in-memory descent depth disagrees with the page it descends. A rolled-back `grow()` shrinks the tree by a level; a stale-deep depth then mis-descends and returns `InvalidHandle` for *committed* handles, or mis-enumerates a tag — silently. It surfaced first in the membership index during chunk-tags work and was then recognized as the *same* root cause in the handle table. The two fixes (I99 for the handle table, C1 for the membership half) extract the open-time spine walk into a reusable `recover_depth` called from both rollback paths. This is why iteration stability (ADR-13) rests on that invariant, and why [ARCHITECTURE.md#in-memory-radix-depth-is-re-derived-from-the-root-never-stored](ARCHITECTURE.md#in-memory-radix-depth-is-re-derived-from-the-root-never-stored) flags it as load-bearing. +- **The commit-protocol ordering is not incidental.** Every step in `TransactionManager::commit` is placed where it is for a crash-window reason. The most non-obvious is the I28 pre-drain: `persist_freemap` can trip the cache's spill-or-error path, and a `CacheFull` raised *mid-commit* would be silently promoted from operational to fatal by the commit wrapper's poison-on-any-error rule. Pre-draining every dirty pin makes the strict cap reachable via normal eviction, at the cost of one extra fsync. Reordering these steps changes what a recovering reader can observe; treat the order as a contract. +- **Structural freemap pages must never come from the bitmap.** Restated here because it cost a design draft: an earlier multi-page-freemap plan assumed the freemap's own COW pages could be reclaimed from free bits. They can't — it recurses without termination. Out-of-band recycling is the only correct source. + +--- + +## Benchmark methodology + +The measurement design is worth a section of its own because its choices are principled, not arbitrary, and because someone will eventually need to add a benchmark and should follow the same rules. + +**Measurement layers.** As above, there are three tiers, and the split is about matching the tool to the timescale: Criterion's many-samples model for the micro-grid (where each op is sub-millisecond and statistical rigor is achievable and cheap), and a hand-rolled `Instant::now()` hybrid for the YCSB-style scenarios (where a single run is 1–6 minutes and Criterion's sampling would blow the budget). Cross-engine equivalence tests underpin both, so no comparison is ever between engines that secretly disagree on behavior. + +**macOS `F_FULLFSYNC` fairness.** The most important methodological rule is that durability must be compared like-for-like. Chisel always uses `F_FULLFSYNC` on macOS; a naive SQLite comparison would pit Chisel's true flush against Apple's weaker default and make Chisel look slow for being *more* correct. The `PRAGMA fullfsync=ON` fix (ADR-11), ungated by platform, is what makes the numbers honest. If you add a new comparison engine, this is the trap to check first. + +**Counter-driven measurement.** Prefer the engine's own counters over wall-clock whenever you want to know *why* something changed. Time tells you a scenario regressed; the delta in `cache_hits` / `cache_misses` / `pages_allocated` / `fsync_calls` tells you which subsystem did it. The harness reads counters before and after each scenario cell and reports deltas; this is the intended primary consumer of `Chisel::counters()`. + +**A CI caution (from the project's standing guidance).** Benchmark numbers from shared CI runners are too noisy to gate a pipeline on — a throttled or noisy-neighbor VM can spike an allocation-heavy bench 2×+ on byte-identical code. Keep perf/benchmark CI steps report-only (non-blocking); let only correctness (build, test, lint, clippy) gate the pipeline. Benchmark baselines exist for *trend-tracking*, not pass/fail, and should be re-recorded when a trend shows a persistent shift. Gate on perf only on consistent, dedicated hardware. + +--- + +## Rationale not recovered + +The following choices are real and load-bearing, but the project's ADRs, specs, and issue log do **not** record *why* the specific value or option was picked over its neighbors. They are listed here honestly rather than back-filled with a plausible-sounding invention — a fabricated rationale in a "why" document is worse than an acknowledged gap, because a future reader would trust it. + +- **Why XXH3 specifically** (over CRC32C, BLAKE3, or another non-cryptographic checksum) for the page and spillway checksum. XXH3 is used everywhere, but no source records the comparison; only the I77 FxHash-over-SipHash swap for *internal maps* has recorded rationale. + + > Rationale not recovered from project sources. + +- **Why 8192-byte pages** (over 4096 or 16384). `PAGE_SIZE = 8192` is pervasive in the geometry and treated as a given; no ADR, spec, or issue captures the size choice. + + > Rationale not recovered from project sources. + +- **Why 8 key-slots** (over 4 or 16). Spec §3.2 shows that 8 × 128 bytes fits comfortably in the reserved region, but nothing explains why 8 is the right *operational* number of credentials. + + > Rationale not recovered from project sources. + +- **Why `superblock_count` default = 2** (rather than 3), and why the upper bound is 16. ADR-4 explains N rotating slots and that N is configurable, but not the specific default or ceiling. + + > Rationale not recovered from project sources. + +- **The chosen `Argon2Params` defaults** (m_cost / t_cost / p_cost) for create-time passphrase slots, and the security/latency tradeoff behind them. The slot codec stores them and the API exposes `Argon2Params`, but the default cost values are unexplained. + + > Rationale not recovered from project sources. + +- **Why `spillway_max_bytes` default = 1024× `cache_max_bytes`** (8 GiB at the 8 MiB cache default). ADR-5 states the multiplier but does not justify 1024 specifically as the ceiling. + + > Rationale not recovered from project sources. + +- **The membership-tree depth-6 bound** beyond the fan-out arithmetic. The freemap spec derives its fan-out math, but the membership tree's depth-6 `MAX_DEPTH` is only cross-referenced, not independently justified in the harvested sources. + + > Rationale not recovered from project sources. From 019e422663e2a1048a0ddb8bb8f6cf9387fe8f50 Mon Sep 17 00:00:00 2001 From: Christophe Pettus Date: Wed, 1 Jul 2026 09:56:32 -0700 Subject: [PATCH 2/2] docs(theory): fill two rationale gaps from the engine author MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two of the seven "Rationale not recovered" gaps in THEORY.md are answered directly by the engine author and move up into the decision sections: - Page size 8 KB: chosen to align with the underlying storage block size so a page maps cleanly onto the device's block granularity (new subsection next to the durability decision). - superblock_count default = 2: a deliberate integrity/performance balance — 3 is defensible, but larger N costs considerable performance for little added safety (folded into the ADR-4 decision). Both are attributed as recorded from the engine author rather than a written ADR. Five genuine gaps remain, still marked "not recovered" rather than invented. --- THEORY.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/THEORY.md b/THEORY.md index 993adfe..aa1c454 100644 --- a/THEORY.md +++ b/THEORY.md @@ -39,6 +39,12 @@ What follows is the spine of the document: the eighteen design decisions that, i **Why.** Shadow paging trades disk space (you hold the live and previous version of every mutated page until commit) for code simplicity. There is no log to replay; the "is this database openable?" check *is* the crash-recovery path; crash safety is provable by inspection. The known cost is accepted with eyes open: write amplification (a one-byte change costs a full 8 KB page) and the eventual need for `defrag.rs` to reclaim sparse pages over time. See [ARCHITECTURE.md#commit-protocol](ARCHITECTURE.md#commit-protocol) for the mechanism. +### Page size: 8 KB, aligned to the storage block + +**Chosen.** `PAGE_SIZE = 8192`, fixed for the life of a database and pervasive in every geometry calculation (slot directories, radix fan-out, freemap coverage). + +**Why.** 8 KB was chosen to align with the underlying storage block size, so that a page maps cleanly onto the device's own block granularity rather than straddling it. *(Recorded directly from the engine author; there is no written ADR for the page-size choice.)* + ### Single-writer, enforced by the type system (ADR-2) **Chosen.** One process, one writer. The invariant is enforced at three levels at once: an OS exclusive `flock` in `page_io.rs`, `&mut self` on every mutating API, and an explicit statement that this is philosophical, not a v1 shortcut. Reads take `&self` because the cache lives behind a `RefCell`, not a `Mutex`. @@ -63,6 +69,8 @@ What follows is the spine of the document: the eighteen design decisions that, i **Why.** A single superblock overwritten in place is vulnerable to a torn write leaving the file unrecoverable. Rotating slots give trivial torn-write recovery — a torn slot fails its checksum and is ignored — need no separate journal (the slots *are* the journal), and offer a configurable space/durability tradeoff: higher N survives more *consecutive* torn writes. The cost is N pages of overhead at the file head and the fact that changing N for an existing file needs migration. The commit always writes the *stalest* slot, so a torn write can only damage a slot you were about to discard anyway; see the round-robin reasoning in [ARCHITECTURE.md#commit-protocol](ARCHITECTURE.md#commit-protocol). +**Why the default is 2.** The default `superblock_count` of 2 is a deliberate integrity/performance balance: two slots already give clean torn-write recovery, and 3 is defensible, but larger N carries considerable performance cost for little additional safety — each extra slot only buys survival of one more *consecutive* torn write, already a vanishingly rare event. The `2..=16` range leaves headroom for a deployment that wants more redundancy without imposing it by default. *(The specific default was recorded from the engine author; ADR-4 covered the rotating-slot mechanism but not the chosen value.)* + ### Poison model on fatal errors (ADR-6, I1) **Chosen.** Any fatal error — a commit-path `IoError`, a `ChecksumMismatch`, a `CorruptSuperblock`, a `DecryptionFailed`, anything raised after the commit protocol has begun (see `ChiselError::is_fatal()`) — sets a poison flag on the `TransactionManager`. Every subsequent call, *including reads*, returns `Poisoned`. The only legal recovery is to drop the handle and `Chisel::open` again. @@ -227,24 +235,16 @@ The measurement design is worth a section of its own because its choices are pri ## Rationale not recovered -The following choices are real and load-bearing, but the project's ADRs, specs, and issue log do **not** record *why* the specific value or option was picked over its neighbors. They are listed here honestly rather than back-filled with a plausible-sounding invention — a fabricated rationale in a "why" document is worse than an acknowledged gap, because a future reader would trust it. +The following choices are real and load-bearing, but the project's ADRs, specs, and issue log do **not** record *why* the specific value or option was picked over its neighbors. They are listed here honestly rather than back-filled with a plausible-sounding invention — a fabricated rationale in a "why" document is worse than an acknowledged gap, because a future reader would trust it. As gaps are answered by the people who made the calls, they move up into the decision sections above: the **8 KB page size** (aligned to the storage block) and the **`superblock_count` default of 2** (an integrity/performance balance) were recorded from the engine author and now live with their decisions. - **Why XXH3 specifically** (over CRC32C, BLAKE3, or another non-cryptographic checksum) for the page and spillway checksum. XXH3 is used everywhere, but no source records the comparison; only the I77 FxHash-over-SipHash swap for *internal maps* has recorded rationale. > Rationale not recovered from project sources. -- **Why 8192-byte pages** (over 4096 or 16384). `PAGE_SIZE = 8192` is pervasive in the geometry and treated as a given; no ADR, spec, or issue captures the size choice. - - > Rationale not recovered from project sources. - - **Why 8 key-slots** (over 4 or 16). Spec §3.2 shows that 8 × 128 bytes fits comfortably in the reserved region, but nothing explains why 8 is the right *operational* number of credentials. > Rationale not recovered from project sources. -- **Why `superblock_count` default = 2** (rather than 3), and why the upper bound is 16. ADR-4 explains N rotating slots and that N is configurable, but not the specific default or ceiling. - - > Rationale not recovered from project sources. - - **The chosen `Argon2Params` defaults** (m_cost / t_cost / p_cost) for create-time passphrase slots, and the security/latency tradeoff behind them. The slot codec stores them and the API exposes `Argon2Params`, but the default cost values are unexplained. > Rationale not recovered from project sources.