From 615044316c41e3c1e9fb4a527368edfe439cb6c2 Mon Sep 17 00:00:00 2001 From: Christophe Pettus Date: Tue, 4 Aug 2026 20:19:54 -0700 Subject: [PATCH] =?UTF-8?q?feat:=20bulk=20DEK=20rotation=20=E2=80=94=20re-?= =?UTF-8?q?encrypt=20a=20whole=20database=20under=20a=20fresh=20data=20key?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes #140 (I142). Credential rotation and data-key rotation answer different questions, and only one of them was answerable. `add_key`/`rotate_key`/`remove_key` re-wrap the SAME DEK under a different KEK: O(1), superblock only, and the right tool when a passphrase leaks. None of them help when the DEK ITSELF leaks — a process memory dump, a core file, an attached debugger — because the key they re-wrap is the thing that leaked. The CRYPTO-1 fix sharpened the point by documenting it on `rotate_key`; this supplies the operation that sentence pointed at. `Chisel::rekey(path, key, argon2_params)` generates a fresh DEK, re-seals every page under it at the SAME page id (so no pointer in any tree changes, only the bytes beneath it), rebuilds the superblock bank, and replaces the file. Two design points, both recorded in ADR 0018. CRASH SAFETY IS COPY-THEN-RENAME, and it has to be. Shadow paging protects writes that go to NEW pages; this rewrites pages where they already are. A crash halfway through an in-place rotation leaves some pages under the new DEK and some under the old, with the surviving superblock naming one of them — an unrecoverable mix, and exactly the half-state the engine otherwise never produces. The replacement is built in a scratch file beside the original, fsynced, then published with an atomic rename, so the database path only ever names a complete file. IT TAKES A PATH, NOT A HANDLE. The rename leaves any previously-opened descriptor pointing at the original, now-unlinked inode; reads and writes through it would silently target a deleted file. Returning a live handle would be returning that hazard, so there is none to misuse. It collapses the key-slot table to the single supplied credential. That is forced, not chosen: a slot's KEK is derived from its own credential, so wrapping the new DEK for a slot requires the credential that slot belongs to. It is also the safer default — if you are rotating because the DEK leaked, quietly preserving every credential that could reach it is not the goal. `add_key` restores the others, and a test pins that remedy. Exposed through the PyO3 binding as a module-level `chisel.rekey(path, key)`, mirroring `chisel.open()` for the same reason: it names a database by path rather than assuming one is open. The GIL is released for the duration (I156 — this is whole-file I/O plus an Argon2id derivation). Adversarial review found four real defects in the first version, all fixed: * IT FAILED ON HEALTHY DATABASES CARRYING CRASH DEBRIS. Requiring every unit to decrypt looked strict and was actually an availability trap: a process killed mid-commit leaves a TORN unit at a page id the freemap marks free, which shadow paging recovers from by never reading it. Such a database opens and works perfectly — but rekey would refuse it, deterministically, at the one moment it is needed. Undecryptable units are now copied through verbatim. That cannot lose committed data (a live page must decrypt or the database would not open) and preserves the file geometry. * NO TEST PROVED THE DEK CHANGED. `seal` draws a fresh nonce per call, so re-sealing under the SAME key also changes every byte — the whole-file comparison caught a no-op but not a same-key reseal. The new test asserts the old cipher can no longer open a page, and fails when the seal is pointed back at `old_cipher`. * PASSPHRASE ROTATION SILENTLY DOWNGRADED ARGON2 COST. `wrap_into` falls back to the OWASP defaults, so a deliberately hardened database was re-wrapped weaker, with no signal, during the operation you run because you were compromised. Cost parameters are now inherited from the slot the key unlocks unless explicitly overridden. * The scratch-path reclamation unlinked whatever occupied a predictable path. Now matched to the spillway's ownership discipline: remove only a plain file this uid owns with a single link, and refuse anything else. Also corrected ARCHITECTURE.md and THEORY.md, which still said bulk rotation was deferred — five lines from the new text describing it. --- ARCHITECTURE.md | 4 +- README.md | 15 +- THEORY.md | 6 +- ...8-bulk-dek-rotation-via-copy-and-rename.md | 109 +++ docs/adr/README.md | 1 + python/chisel/__init__.py | 3 +- python/chisel/__init__.pyi | 30 + python/src/db.rs | 36 + python/tests/test_rekey.py | 155 ++++ src/lib.rs | 65 ++ src/rekey.rs | 719 ++++++++++++++++++ src/transaction/mod.rs | 10 +- 12 files changed, 1142 insertions(+), 11 deletions(-) create mode 100644 docs/adr/0018-bulk-dek-rotation-via-copy-and-rename.md create mode 100644 python/tests/test_rekey.py create mode 100644 src/rekey.rs diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index a37f5ea..4c69f37 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -712,9 +712,11 @@ The DEK wrapping uses detached XChaCha20-Poly1305 with AAD bound to the slot's K - `add_key(old_key, new_key)`: derives a new KEK, wraps the same DEK into a free slot, then commits. - `rotate_key(old_key, new_key)`: `add_key` followed by clearing the old slot in the same commit. +- Every key-slot rewrite also scrubs the key-slot region of the OTHER superblock slots (`CryptoHeader::overwrite_slot_table`). Without that, a revoked credential's wrapped DEK survived verbatim in a sibling slot of the live file — cleartext at bytes 332..1356, wrapping a DEK that never changes — so read access plus the revoked credential was enough to recover the current DEK. Only the table region is patched, so each sibling keeps its own counter, roots and sealed body; the body's AAD covers magic/format_version/txn_counter/superblock_count, not the crypto header. +- `Chisel::rekey(path, key, argon2)` (ADR 0018) is the bulk alternative: a fresh DEK, every page re-sealed, the file replaced by atomic rename. It is an associated function on a path rather than a method because the rename invalidates any descriptor opened beforehand, and it collapses the key-slot table to the supplied credential because a slot's KEK cannot be re-derived without its own credential. - `remove_key(key)`: clears the matching slot, refusing to clear the last active slot (which would make the database permanently unreadable). -Bulk DEK rotation (re-encrypting every page under a fresh DEK) is deferred; see I142. +Bulk DEK rotation is implemented as `Chisel::rekey` (ADR 0018) — the operation for a compromised DEK, as opposed to a compromised credential. It is offline and takes a path, builds the rotated database in a scratch file, and publishes it with an atomic rename; it collapses the key-slot table to the single supplied credential, because a slot's KEK cannot be re-derived without its own credential. **Spillway.** For encrypted databases the in-memory spillway carries sealed blobs: pages are encrypted exactly once on eviction (`seal` on evict-to-spillway) and copied verbatim — no decryption or re-encryption — on drain to the main file. Rehydration decrypts the blob back into the cache. No plaintext page content is ever written to disk by an encrypted database, even during spill. diff --git a/README.md b/README.md index d643d84..6b6e793 100644 --- a/README.md +++ b/README.md @@ -25,7 +25,7 @@ Chisel is designed for single-writer embedded use: one process holds the file vi - **Named roots** — a small fixed table in the superblock mapping string names to handles. Survives commit / rollback transactionally. - **Defragmentation** — explicit `defrag()` consolidates sparse pages and returns a count-based stats record. - **In-memory mode** — same engine, `Vec`-backed I/O, no file and no lock. For tests, benchmarks, and ephemeral work. -- **At-rest encryption** — optional, off by default. Every page is sealed with XChaCha20-Poly1305 AEAD under a client-supplied key (raw 32-byte or Argon2id passphrase). An 8-slot envelope table wraps the data-encryption key, so credential rotation is O(1) — no bulk re-encryption. +- **At-rest encryption** — optional, off by default. Every page is sealed with XChaCha20-Poly1305 AEAD under a client-supplied key (raw 32-byte or Argon2id passphrase). An 8-slot envelope table wraps the data-encryption key, so credential rotation is O(1) — no bulk re-encryption. `rekey()` performs the heavy alternative when the data key itself is compromised: a crash-safe whole-file re-encryption under a fresh key. - **Poison model** — any fatal error (I/O failure, checksum mismatch, commit-protocol failure) poisons the handle; recovery is drop-and-reopen. Mirrors `std::sync::Mutex` poisoning. - **Single-writer** — exclusive `flock` at the filesystem level; `&mut self` on every mutating method. @@ -293,6 +293,12 @@ On create, Chisel generates a random data-encryption key (DEK), encrypts every p The wrapped DEK lives in an **8-slot key table**. Because the DEK itself never changes, credential rotation only re-wraps the DEK in a slot — it is O(1), independent of database size. `add_key` stages a second credential (both open the DB), `rotate_key` replaces one credential in place, and `remove_key` retires one (refusing the last remaining slot with `LastKeySlot`). A full table returns `NoFreeKeySlot`. +Rotating a *credential* is not the same as rotating the *data key*. Because the DEK never changes, `rotate_key` and `remove_key` deny a credential a way in — they do not cut it off from data it has already seen, and they cannot help if the DEK itself leaked (a process memory dump, a core file). `Chisel::rekey(path, key, argon2_params)` is the operation for that case: it generates a fresh DEK, re-encrypts every page, and replaces the file atomically. + +`rekey` takes a **path, not an open handle**, and that is deliberate: it replaces the file by `rename`, so a descriptor opened beforehand would afterwards refer to the original, now-unlinked inode. Close the database, rekey, reopen. It is crash-safe — the rotated database is built in a scratch file beside the original and published atomically, so the path only ever names a complete file — and it costs O(total_pages) I/O plus a transient second copy on disk. + +It also **collapses the key-slot table to the single credential you supply**. That is forced rather than chosen: each slot's KEK is derived from its own credential, so wrapping the new DEK for a slot requires the credential that slot belongs to. Re-add the others with `add_key` afterwards. It is the safer default anyway — if you are rotating because the DEK leaked, preserving every credential that could reach it is not the goal. + See [ARCHITECTURE.md#on-disk-encryption](ARCHITECTURE.md#on-disk-encryption) for the on-disk layout (crypto header, key slots, per-page nonce stride) and [THEORY.md](THEORY.md) for the rationale behind the envelope scheme and the shadow-paging nonce discipline (with [`docs/adr/`](docs/adr/) as the dated decision log). ## API reference @@ -332,6 +338,7 @@ See [ARCHITECTURE.md#on-disk-encryption](ARCHITECTURE.md#on-disk-encryption) for | `defrag(options)` | Consolidate sparse pages | | `add_key(existing, new)` | Stage a second credential; both `existing` and `new` then open the DB. `Result<()>` | | `rotate_key(old, new)` | Replace credential `old` with `new` in place. `Result<()>` | +| `Chisel::rekey(path, key, argon2)` | Re-encrypt the whole database under a fresh DEK. Associated function, not a method — takes a path. `Result<()>` | | `remove_key(key)` | Retire credential `key`; `LastKeySlot` if it is the only one. `Result<()>` | ## Options @@ -469,6 +476,12 @@ db.remove_key("correct horse battery staple") # retire `add_key` / `rotate_key` raise `NoFreeKeySlotError` when the 8-slot table is full; `remove_key` raises `LastKeySlotError` rather than leaving the DB with no usable credential. +```python +chisel.rekey("my.db", key=b"\x00" * 32) # whole-file re-encryption under a fresh data key +``` + +`chisel.rekey` is a module-level function, like `chisel.open`, because it names a database by path rather than assuming one is open — it replaces the file, so any handle held across the call would be stale. Every credential other than the one supplied is revoked; re-add them with `add_key`. + ## Design documents - [`ARCHITECTURE.md`](ARCHITECTURE.md) — living architecture overview: layer model, commit protocol, recovery, full on-disk format byte-by-byte, and cross-cutting concepts. Start here if you're reading the codebase to *act* on it. diff --git a/THEORY.md b/THEORY.md index 08676db..0c82217 100644 --- a/THEORY.md +++ b/THEORY.md @@ -155,13 +155,13 @@ There is one **critical, revised** sub-decision that is easy to get wrong and wo ### Encryption keys: envelope DEK/KEK with an 8-slot table, O(1) rotation (ADR-15) -**Chosen.** A random per-database 256-bit DEK (from `OsRng` at create) seals every page and the sensitive superblock body. The DEK is never stored bare — it is wrapped under a KEK derived from the client key (HKDF-SHA256 for raw keys, Argon2id for passphrases) and held in an 8-slot key-slot table in the superblock's plaintext reserved region. `add_key` / `rotate_key` / `remove_key` re-wrap the *stable* DEK — O(1), no data re-encryption. `rotate_key` stages the new slot before revoking the old (no zero-key window); `remove_key` refuses to clear the last active slot (brick prevention). A successful unwrap *is* proof the client key is correct — there is no separate password verifier. +**Chosen.** A random per-database 256-bit DEK (from `OsRng` at create) seals every page and the sensitive superblock body. The DEK is never stored bare — it is wrapped under a KEK derived from the client key (HKDF-SHA256 for raw keys, Argon2id for passphrases) and held in an 8-slot key-slot table in the superblock's plaintext reserved region. `add_key` / `rotate_key` / `remove_key` re-wrap the *stable* DEK — O(1), no data re-encryption. `rotate_key` stages the new slot before revoking the old (no zero-key window); `remove_key` refuses to clear the last active slot (brick prevention). A successful unwrap *is* proof the client key is correct — there is no separate password verifier. Rotating a credential is not rotating the data key: the DEK is unchanged, so revocation denies entry rather than re-keying, and anyone who captured the DEK while the credential was valid keeps reading. `rekey` (ADR 0018) is the answer when the DEK itself is what leaked — a whole-file re-encryption published by atomic rename, since shadow paging cannot make an in-place rewrite crash-safe. -**Rejected.** Encrypting only the data pages and leaving the superblock plaintext — rejected because `named_roots` holds user-chosen names, which are real user data a plaintext body would leak. Full DEK rotation (re-encrypting every page under a fresh DEK) — deferred (I142) as a heavy O(total_pages) whole-file operation reserved for "the DEK itself is compromised"; credential rotation is the far more common need and is O(1). +**Rejected.** Encrypting only the data pages and leaving the superblock plaintext — rejected because `named_roots` holds user-chosen names, which are real user data a plaintext body would leak. Full DEK rotation (re-encrypting every page under a fresh DEK) — implemented separately as `rekey` (ADR 0018) rather than folded into the credential path, because the two answer different questions: credential rotation is the far more common need and is O(1), while a whole-file re-encryption is reserved for "the DEK itself is compromised" and is O(total_pages). **Why.** Envelope encryption makes credential rotation O(1) — you re-wrap the DEK — instead of O(database size). The per-slot KDF choice matches input entropy: HKDF is fast and correct for high-entropy keys, while Argon2id is memory-hard to resist brute-forcing low-entropy passphrases (its params are recorded per slot). And every rotation op is an ordinary superblock A/B + fsync commit, so it reuses the existing crash-safe protocol wholesale: a metadata-only `rewrite_crypto_header` commit persists a rotated slot table atomically (write the inactive slot, fsync, promote), so a crash mid-rotation leaves the old table intact. -Two threat-model boundaries are documented rather than solved, and you should know them before you rely on this: there is **no rollback/replay resistance** (an attacker who substitutes a wholly older, validly-signed image is undetectable without an external trust anchor like a TPM), and the DEK sits in plaintext in process memory during a session (mitigated by zeroize-on-drop, not by encryption). See spec `2026-06-29` §3/§5/§9 and [issue #140](https://github.com/pgexperts/chisel/issues/140) (the deferred bulk DEK rotation, formerly I142). +Two threat-model boundaries are documented rather than solved, and you should know them before you rely on this: there is **no rollback/replay resistance** (an attacker who substitutes a wholly older, validly-signed image is undetectable without an external trust anchor like a TPM), and the DEK sits in plaintext in process memory during a session (mitigated by zeroize-on-drop, not by encryption). See spec `2026-06-29` §3/§5/§9. Note that `rekey` (ADR 0018) does bear on the first boundary: it invalidates every page image sealed under the old DEK, so it ends an attacker's ability to splice in stale pages captured beforehand — it does not, however, make the engine detect such splicing, which remains open as [issue #142](https://github.com/pgexperts/chisel/issues/142). ### Encryption page format: 8232-byte stride, logical page stays 8192, MAJOR 1→2 (ADR-15) diff --git a/docs/adr/0018-bulk-dek-rotation-via-copy-and-rename.md b/docs/adr/0018-bulk-dek-rotation-via-copy-and-rename.md new file mode 100644 index 0000000..7bb3604 --- /dev/null +++ b/docs/adr/0018-bulk-dek-rotation-via-copy-and-rename.md @@ -0,0 +1,109 @@ +--- +id: 0018 +title: Bulk DEK rotation as an offline copy-then-rename operation on a path +date: 2026-08-04 +status: Accepted +summary: rekey() re-encrypts the whole database under a fresh DEK by building a replacement file and renaming it into place; it takes a path rather than a handle, and collapses the key-slot table to the single supplied credential. +--- + +# 0018. Bulk DEK rotation as an offline copy-then-rename operation on a path + +## Context + +ADR [0015](0015-on-disk-encryption-xchacha20-poly1305-envelope-keys.md) chose an +envelope scheme: one per-database DEK seals every page, and up to eight key +slots each wrap that DEK under a KEK derived from a client credential. That +makes *credential* rotation O(1) — `add_key`/`rotate_key`/`remove_key` touch +only the superblock, and no page is re-encrypted. It was recorded at the time +that full DEK rotation was deferred. + +Deferred is not the same as unnecessary. The two operations answer different +questions, and only one of them was answerable: + +- a credential leaked → deny that credential a way in → credential rotation +- **the DEK itself leaked** (process memory dump, core file, attached debugger) + → the data must be re-sealed under a key the attacker does not have + +Credential rotation cannot help with the second case at all, because the DEK it +re-wraps is the very thing that leaked. The fix for CRYPTO-1 sharpened the point +by documenting it in `rotate_key`: revocation denies entry, it does not +re-key. Without a bulk rotation there was no operation in the crate that did. + +Two things made the design non-obvious. + +**Crash safety cannot lean on shadow paging.** Shadow paging protects writes +that go to *new* pages; a DEK rotation rewrites every page where it already is. +A crash halfway through leaves some pages sealed under the new DEK and some +under the old, with the surviving superblock naming one of them — an +unrecoverable mix, and precisely the kind of half-state the engine otherwise +never produces. + +**The operation invalidates its own handle.** Any strategy that replaces the +file by rename leaves a previously-opened file descriptor pointing at the +original, now-unlinked inode. Reads and writes through it would silently target +a deleted file. + +## Decision + +We will implement `Chisel::rekey(path, key, argon2_params)` as an **offline +operation on a path**, which builds the rotated database in a scratch file +beside the original and publishes it with an atomic `rename`. + +Concretely: open normally (validating the key and taking the exclusive flock), +generate a fresh DEK, write every page into `.rekey-tmp` — superblock slots +rebuilt from the winning superblock under a new crypto header, every other page +opened under the old DEK and re-sealed under the new one *at the same page id* — +`fsync`, `rename`, `fsync` the directory, then drop the handle. + +The rotated database carries **exactly one key slot**: the credential supplied. + +## Alternatives considered + +- **In-place two-pass rotation.** Rejected: not crash-safe for the reason + above, and making it so would need a journal — a second durability mechanism + in an engine whose entire premise (ADR + [0001](0001-shadow-paging-not-wal.md)) is that it does not have one. + +- **`rekey(&mut self)` on a live handle.** Rejected: it would hand back a + handle whose file descriptor names a deleted inode. Making that safe means + swapping the descriptor and re-acquiring the flock underneath a live cache — + real complexity to preserve an ergonomic nicety on an operation that rewrites + the entire file and is not something anyone runs in a loop. + +- **`rekey(self)` consuming the handle.** Closer, and it does dispose of the + stale-descriptor hazard, but `Chisel` does not retain its path, so the caller + would have to pass it back in and could pass the wrong one. A path-taking + associated function has one source of truth and reads as what it is. + +- **Re-wrapping the new DEK into all currently-active slots.** Not + implementable, not merely declined: each slot's KEK is derived from *its own* + credential, and only one was supplied. There is no way to produce a valid wrap + for a credential you do not hold. It is also the safer default — if you are + rotating because the DEK leaked, silently preserving every credential that + could reach it is not what you want. `add_key` restores the others. + +- **Keeping the older superblock slots' historical roots.** Declined. Their + sealed bodies are under the old DEK, so preserving them means re-sealing + states that are already unreachable. Every slot in the new file carries the + winning superblock's roots at staggered counters, exactly as `create_new` + seeds a fresh bank. + +## Consequences + +- Rotation costs O(total_pages) I/O and, transiently, a second copy of the + database on disk. It is maintenance, not routine operation. +- The scratch file is created `O_EXCL | O_NOFOLLOW` mode 0600, matching the + hardening the database and spillway received, so a planted file at the + predictable scratch path cannot be adopted or followed. +- The directory `fsync` after the rename is best-effort: the file contents are + already durable, so a failure costs rename durability across a crash rather + than correctness. +- `rekey` must live outside `transaction/`, since it reaches across an open + handle and the filesystem at once. It is its own module. +- Because it is a free function on a path, the PyO3 binding exposes it as a + module-level `chisel.rekey(path, key)` mirroring `chisel.open()` rather than + as a method — the same reasoning carried across the boundary. +- Rotation does not defend against per-page temporal replay + ([#142](https://github.com/pgexperts/chisel/issues/142)); it does, however, + invalidate every previously-sealed page image, so it is the operation that + ends an attacker's ability to splice stale pages sealed under the old DEK. diff --git a/docs/adr/README.md b/docs/adr/README.md index 058ebca..6361ac8 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -22,3 +22,4 @@ | [0015](0015-on-disk-encryption-xchacha20-poly1305-envelope-keys.md) | On-disk encryption (XChaCha20-Poly1305, envelope keys) | Accepted | 2026-06-30 | | | [0016](0016-swift-binding-via-uniffi.md) | Swift binding via UniFFI | Accepted | 2026-07-19 | The iOS/macOS Swift binding is a UniFFI-generated FFI over an Arc> wrapper crate, with the engine crate left unchanged. | | [0017](0017-github-issues-replace-tracked-issues-md.md) | Track issues in GitHub, not a tracked ISSUES.md | Accepted | 2026-08-03 | The 1868-line ISSUES.md decision log was retired; open entries were migrated to GitHub issues and the file deleted. | +| [0018](0018-bulk-dek-rotation-via-copy-and-rename.md) | Bulk DEK rotation as an offline copy-then-rename operation on a path | Accepted | 2026-08-04 | rekey() re-encrypts the whole database under a fresh DEK by building a replacement file and renaming it into place; it takes a path rather than a handle, and collapses the key-slot table to the single supplied credential. | diff --git a/python/chisel/__init__.py b/python/chisel/__init__.py index cb61a4f..a51fb1e 100644 --- a/python/chisel/__init__.py +++ b/python/chisel/__init__.py @@ -17,6 +17,7 @@ Savepoint, DrainInsertion, open, + rekey, ChiselError, OperationalError, FatalError, @@ -130,7 +131,7 @@ class DefragStats: __all__ = [ "__version__", - "Chisel", "Transaction", "Savepoint", "DrainInsertion", "open", + "Chisel", "Transaction", "Savepoint", "DrainInsertion", "open", "rekey", "Stats", "Counters", "DefragOptions", "DefragStats", "ChiselError", "OperationalError", "FatalError", "InvalidHandleError", "NoActiveTransactionError", diff --git a/python/chisel/__init__.pyi b/python/chisel/__init__.pyi index a3ea61c..4d98448 100644 --- a/python/chisel/__init__.pyi +++ b/python/chisel/__init__.pyi @@ -142,6 +142,36 @@ def open( ) -> Chisel: ... +def rekey(path: str | os.PathLike[str], key: bytes | str) -> None: + """Re-encrypt an entire database under a freshly generated data key. + + Takes a path rather than an open handle, and mirrors :func:`open` for that + reason: it rewrites the whole file and replaces it by rename, so a handle + opened beforehand would afterwards refer to the original, now-unlinked + inode. Close any open database first, call this, then reopen. + + Use this only when the DATA key itself is believed compromised. If a + credential leaked, :meth:`Chisel.rotate_key` is the right tool -- it is + O(1) and touches only the superblock. + + Every credential other than ``key`` is revoked, because each key slot's + wrapping key is derived from its own credential and only ``key`` was + supplied. Add the others back with :meth:`Chisel.add_key` afterwards. + + For a passphrase credential the Argon2 cost parameters are INHERITED from + the slot ``key`` currently unlocks, so a deliberately hardened database is + not silently downgraded to the defaults by the rotation. + + Crash-safe: the rotated database is built beside the original and published + with an atomic rename, so the path only ever names a complete file. + + Raises: + DatabaseFileNotFoundError: no database at ``path``. + EncryptionNotSupportedError: the database is not encrypted. + InvalidEncryptionKeyError: ``key`` unlocks no key slot. + """ + + class Chisel: @property def is_poisoned(self) -> bool: ... diff --git a/python/src/db.rs b/python/src/db.rs index 15ddc02..a23dc5d 100644 --- a/python/src/db.rs +++ b/python/src/db.rs @@ -681,9 +681,45 @@ fn closed_err() -> PyErr { crate::errors::ClosedError::new_err("database handle has been closed") } +/// Bulk DEK rotation — re-encrypt an entire database under a fresh data key. +/// +/// A module-level function rather than a `Chisel` method because it operates on +/// a PATH, not an open handle: it rewrites the whole file and replaces it by +/// rename, so any handle opened beforehand would afterwards refer to the +/// original, now-unlinked inode. Mirrors `chisel.open()` in shape for that same +/// reason — both name a database by path rather than assuming one is open. +/// +/// I156: the GIL is released for the duration. This is O(total_pages) I/O plus +/// an Argon2id derivation for a passphrase credential; holding the GIL across +/// it would freeze every other Python thread for the length of a whole-file +/// rewrite, which is the worst case of exactly the problem I156 describes. +#[pyfunction] +#[pyo3(signature = (path, key))] +pub fn rekey(py: Python<'_>, path: Py, key: Py) -> PyResult<()> { + // Coerce both arguments under the GIL, before detaching, so a bad type + // raises a synchronous Python TypeError — same discipline as open(). + let path_buf: PathBuf = { + let bound = path.bind(py); + let s: String = if let Ok(py_str) = bound.cast::() { + py_str.to_str()?.to_owned() + } else { + let os = py.import("os")?; + os.call_method1("fspath", (path,))?.extract()? + }; + PathBuf::from(s) + }; + // Zeroizing on coercion, before any engine call, so key material never + // reaches an error message, traceback, or repr. + let key = py_key(key.bind(py))?; + + py.detach(|| chisel::Chisel::rekey(&path_buf, &key, None)) + .map_err(to_py_err) +} + pub fn register(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_class::()?; m.add_class::()?; m.add_function(wrap_pyfunction!(open, m)?)?; + m.add_function(wrap_pyfunction!(rekey, m)?)?; Ok(()) } diff --git a/python/tests/test_rekey.py b/python/tests/test_rekey.py new file mode 100644 index 0000000..defadf2 --- /dev/null +++ b/python/tests/test_rekey.py @@ -0,0 +1,155 @@ +"""Tests for chisel.rekey() — bulk DEK rotation through the binding. + +`rekey` is the heavy sibling of rotate_key: rotate_key re-wraps the SAME data +key under a different credential (O(1), superblock only), while rekey generates +a fresh data key and re-encrypts every page. + +It is a module-level function rather than a Chisel method because it operates +on a PATH: it rewrites the whole file and replaces it by rename, so a handle +opened beforehand would afterwards refer to the original, now-unlinked inode. + +Covers: + - values, named roots and large (overflow-chained) values survive + - the on-disk bytes actually change (a silent no-op would pass everything else) + - the same credential still opens the database afterwards + - credentials NOT supplied to rekey are revoked, and can be re-added + - a wrong key is refused without modifying the file + - a plaintext database is refused + - a missing file is refused + - the database is still writable afterwards + - no scratch file is left beside the database + - bad argument types raise TypeError before anything is touched +""" + +import pathlib + +import chisel +import pytest + +KEY = bytes([0xA1]) * 32 +OTHER = bytes([0xB2]) * 32 +BIG = b"\xc7" * (8192 * 3) + + +def _seed(path: pathlib.Path, key: bytes | str) -> tuple[int, int]: + """Create a database with enough structure that a rotation has real work: + many small values, one overflow-chained value, and a named root.""" + with chisel.open(path, encryption_key=key) as db: + with db.transaction() as txn: + small = txn.allocate(b"small value") + big = txn.allocate(BIG) + db.set_root_name("primary", small) + return small, big + + +def test_rekey_preserves_data_and_changes_the_ciphertext(tmp_path): + path = tmp_path / "rk.db" + small, big = _seed(path, KEY) + before = path.read_bytes() + + chisel.rekey(path, KEY) + + after = path.read_bytes() + assert len(before) == len(after), "the page count must not change" + assert before != after, "every page is sealed under a new data key" + + # Same credential, same data. + with chisel.open(path, encryption_key=KEY) as db: + assert db.read(small) == b"small value" + assert db.read(big) == BIG + assert db.get_root_name("primary") == small + + +def test_rekey_revokes_credentials_it_was_not_given(tmp_path): + # The documented consequence: each key slot's wrapping key is derived from + # its own credential, so a credential that was not supplied cannot have the + # new data key wrapped for it. Asserting it keeps the doc honest. + path = tmp_path / "rk.db" + small, _ = _seed(path, KEY) + with chisel.open(path, encryption_key=KEY) as db: + db.add_key(KEY, OTHER) + + # Both work beforehand. + with chisel.open(path, encryption_key=OTHER) as db: + assert db.read(small) == b"small value" + + chisel.rekey(path, KEY) + + with chisel.open(path, encryption_key=KEY) as db: + assert db.read(small) == b"small value" + with pytest.raises(chisel.InvalidEncryptionKeyError): + chisel.open(path, encryption_key=OTHER) + + # The documented remedy. + with chisel.open(path, encryption_key=KEY) as db: + db.add_key(KEY, OTHER) + with chisel.open(path, encryption_key=OTHER) as db: + assert db.read(small) == b"small value" + + +def test_rekey_with_a_passphrase_credential(tmp_path): + path = tmp_path / "rk.db" + small, _ = _seed(path, "correct horse battery staple") + chisel.rekey(path, "correct horse battery staple") + with chisel.open(path, encryption_key="correct horse battery staple") as db: + assert db.read(small) == b"small value" + + +def test_rekey_refuses_a_wrong_key_without_touching_the_file(tmp_path): + path = tmp_path / "rk.db" + small, _ = _seed(path, KEY) + before = path.read_bytes() + + with pytest.raises(chisel.InvalidEncryptionKeyError): + chisel.rekey(path, OTHER) + + assert path.read_bytes() == before, "a refused rekey must not modify a byte" + with chisel.open(path, encryption_key=KEY) as db: + assert db.read(small) == b"small value" + + +def test_rekey_refuses_a_plaintext_database(tmp_path): + path = tmp_path / "plain.db" + with chisel.open(path) as db: + with db.transaction() as txn: + txn.allocate(b"cleartext") + with pytest.raises(chisel.EncryptionNotSupportedError): + chisel.rekey(path, KEY) + + +def test_rekey_refuses_a_missing_file(tmp_path): + with pytest.raises(chisel.DatabaseFileNotFoundError): + chisel.rekey(tmp_path / "nope.db", KEY) + + +def test_database_is_writable_after_a_rotation(tmp_path): + # The freemap, handle table and next_handle all come through the superblock + # body that was re-sealed under the new key, so a rotated database that + # reads but cannot write would be a real (and quiet) failure. + path = tmp_path / "rk.db" + small, _ = _seed(path, KEY) + + chisel.rekey(path, KEY) + + with chisel.open(path, encryption_key=KEY) as db: + with db.transaction() as txn: + fresh = txn.allocate(b"written after the rotation") + with chisel.open(path, encryption_key=KEY) as db: + assert db.read(fresh) == b"written after the rotation" + assert db.read(small) == b"small value" + + +def test_rekey_leaves_no_scratch_file(tmp_path): + path = tmp_path / "rk.db" + _seed(path, KEY) + chisel.rekey(path, KEY) + assert not (tmp_path / "rk.db.rekey-tmp").exists() + + +def test_rekey_rejects_bad_argument_types(tmp_path): + path = tmp_path / "rk.db" + _seed(path, KEY) + # Coercion happens under the GIL before any engine call, so this is a + # synchronous TypeError and the file is never opened. + with pytest.raises(TypeError): + chisel.rekey(path, 12345) diff --git a/src/lib.rs b/src/lib.rs index 928b04f..6899bd9 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -60,6 +60,7 @@ pub(crate) mod overflow; pub(crate) mod page; pub(crate) mod page_cache; pub(crate) mod page_io; +pub(crate) mod rekey; mod spillway; pub(crate) mod stats; pub(crate) mod superblock; @@ -1108,6 +1109,70 @@ impl Chisel { pub fn remove_key(&mut self, key: &crypto::Key) -> Result<()> { self.txm.remove_key(key) } + + /// Re-encrypt the entire database under a freshly generated data-encryption + /// key. The heavy sibling of [`Chisel::rotate_key`]. + /// + /// Reach for this only when the DEK ITSELF is believed compromised — a + /// process memory dump, a core file, a debugger session. If what leaked was + /// a *credential*, [`Chisel::rotate_key`] is the right tool: it is O(1), + /// touches only the superblock, and is the common operational need + /// (password change, key rollover, adding a second credential). + /// + /// # Why this is an offline operation on a path + /// + /// It takes a path rather than `&mut self` because it rewrites the whole + /// file and replaces it by `rename`. After a successful rotation, any file + /// descriptor opened before the call refers to the ORIGINAL, now-unlinked + /// inode — reads and writes through it would silently target a deleted + /// file. Handing back a live handle would be handing back that hazard, so + /// there is no handle to misuse: close any open `Chisel` first, call this, + /// then [`Chisel::open`] again. + /// + /// The exclusive `flock` is held for the whole rotation, so no other + /// process can open the database while it is in flight. + /// + /// # Crash safety + /// + /// The rotated database is built in a scratch file beside the original and + /// published with an atomic `rename`. The database path therefore only ever + /// names a complete file — the fully-rotated one, or the untouched + /// original. A crash at any point leaves one or the other, never a mix. + /// + /// In-place rotation would NOT be safe, and shadow paging does not rescue + /// it: shadow paging protects writes that go to *new* pages, while this + /// rewrites pages where they are. A crash halfway would leave some pages + /// under the new DEK and some under the old, with no way to tell which. + /// + /// # This collapses the key-slot table to `key` alone + /// + /// Every other credential is revoked. That is not a shortcut — it is + /// forced: each slot's KEK is derived from its own credential, so re-wrapping + /// the new DEK for a slot requires the credential that slot belongs to, and + /// only `key` was supplied. Add the others back with [`Chisel::add_key`] + /// afterwards. + /// + /// It is also the safer default for the situation this exists for. If you + /// are rotating because the DEK leaked, quietly preserving every credential + /// that could reach it is not what you want. + /// + /// # Cost + /// + /// O(total_pages) I/O and, transiently, a second copy of the database on + /// disk. Not something to schedule routinely. + /// + /// # Errors + /// `FileNotFound` if `path` does not exist; `EncryptionNotSupported` if the + /// database is plaintext; `InvalidEncryptionKey` if `key` unlocks no slot; + /// `IoError` for any failure building or publishing the replacement — in + /// which case the original file is untouched and the scratch file removed. + pub fn rekey( + path: &Path, + key: &crypto::Key, + argon2_params: Option, + ) -> Result<()> { + rekey::rekey(path, key, argon2_params) + } } #[cfg(test)] diff --git a/src/rekey.rs b/src/rekey.rs new file mode 100644 index 0000000..e5ff070 --- /dev/null +++ b/src/rekey.rs @@ -0,0 +1,719 @@ +// rekey.rs — bulk DEK rotation: re-encrypt an entire database under a fresh +// data-encryption key. +// +// Role in the system: this is the heavy sibling of the O(1) credential rotation +// in `transaction::keys`. Those operations (`add_key`/`rotate_key`/`remove_key`) +// re-wrap the SAME DEK under a different KEK — they change who can get in, and +// touch only the superblock. This one changes the key the data itself is sealed +// with, so it must rewrite every page in the file. +// +// Which one you want depends on what leaked: +// +// * a credential (passphrase, raw key) -> credential rotation, O(1) +// * the DEK itself (process memory dump) -> this, O(total_pages) +// +// Layer: sits above the engine rather than inside it. It opens a normal +// `Chisel` handle to validate the key and take the exclusive flock, reads +// through that handle's `PageIo`, and writes a whole new file — so it depends +// on the public open path plus crate-internal access to the live cipher. +// +// Crash safety is copy-then-atomic-rename, and it has to be. The obvious +// in-place alternative is NOT safe under shadow paging: shadow paging protects +// writes that go to NEW pages, and this rewrites pages in place. A crash +// halfway through leaves some pages sealed under the new DEK and some under the +// old, with the surviving superblock naming whichever DEK it names — an +// unrecoverable mix. `rename(2)` is atomic within a filesystem, so the database +// path only ever names a complete file: the fully-rotated one, or the original. + +use std::path::Path; + +use crate::crypto::{Key, PageCipher}; +use crate::error::{ChiselError, Result}; +use crate::page::PAGE_SIZE; +use crate::superblock::{ + CryptoHeader, KeySlot, Superblock, ALGO_XCHACHA20POLY1305, KEY_SLOT_COUNT, +}; +use crate::{Chisel, Options}; + +/// Suffix for the scratch file the rotated database is built in. It lives +/// beside the database so `rename` stays within one filesystem — a rename +/// across filesystems is not atomic and would defeat the whole strategy. +const TMP_SUFFIX: &str = ".rekey-tmp"; + +/// Re-encrypt every page of the database at `path` under a freshly generated +/// DEK, then atomically replace the original. +/// +/// See [`Chisel::rekey`] for the caller-facing contract; this is its body. +pub(crate) fn rekey( + path: &Path, + key: &Key, + argon2_params: Option, +) -> Result<()> { + // Open normally. This does all the validation we would otherwise duplicate + // — file exists, superblock selects, format version and page size accepted, + // algorithm known, key unlocks a slot — and takes the exclusive flock for + // the duration. `create_if_missing(false)` because rotating the key of a + // database that does not exist is a caller error, not a create. + let mut db = Chisel::open( + path, + Options::default() + .encryption_key(key.clone()) + .create_if_missing(false), + )?; + + let old_cipher = match db.txm.cipher.clone() { + Some(c) => c, + // A plaintext database has no DEK to rotate. Operational: the caller + // asked for something that does not apply, nothing is damaged. + None => return Err(ChiselError::EncryptionNotSupported), + }; + + // Snapshot the geometry we are about to reproduce. `page_count` is in + // stride units, and the stride is ENC_PAGE_SIZE for every encrypted file. + let (page_count, superblock_count, stride) = { + let mut cache = db.txm.cache.borrow_mut(); + let io = cache.io_mut(); + (io.page_count()?, db.txm.superblock_count, io.stride()) + }; + debug_assert_eq!(stride, crate::crypto::ENC_PAGE_SIZE); + + // The winning superblock, which every slot of the new file will carry. Its + // roots are the committed state; older slots' roots are history we + // deliberately do not preserve, because their sealed bodies are under the + // OLD DEK and re-sealing states we cannot reach buys nothing. + let winner = current_superblock(&mut db)?; + + // Fresh DEK, and a crypto header holding exactly ONE credential: the one + // supplied. See `Chisel::rekey`'s doc — re-wrapping the new DEK for the + // other active slots is impossible, because each of those slots' KEK is + // derived from a credential we were not given. + // Cost parameters: an explicit override wins, otherwise INHERIT what the + // slot this key just unlocked was using. + // + // Defaulting to `Argon2Params::default()` here would silently DOWNGRADE a + // database whose owner had deliberately hardened its KDF cost — during the + // operation you run because you were compromised, with no signal. The old + // slot's parameters are right there in the header we are replacing, and + // "keep what you had" is the only defensible default for a key rotation. + let effective_params = match argon2_params { + Some(p) => Some(p), + None => inherited_argon2(&db, key), + }; + + let new_dek = crate::crypto::random_dek(); + let mut new_header = CryptoHeader { + algorithm: ALGO_XCHACHA20POLY1305, + stride: crate::crypto::ENC_PAGE_SIZE as u32, + slots: [KeySlot::EMPTY; KEY_SLOT_COUNT], + }; + new_header.wrap_into(0, key, &new_dek, effective_params)?; + let new_cipher = PageCipher::new(new_dek); + + let tmp_path = { + let mut p = path.as_os_str().to_owned(); + p.push(TMP_SUFFIX); + std::path::PathBuf::from(p) + }; + + // Build the replacement, cleaning up the scratch file on any failure so a + // botched rotation does not leave debris that the next attempt trips over + // (the create is O_EXCL — see `create_scratch`). + let build = build_rotated_file( + &mut db, + &tmp_path, + page_count, + superblock_count, + &winner, + &new_header, + &old_cipher, + &new_cipher, + ); + if let Err(e) = build { + let _ = std::fs::remove_file(&tmp_path); + return Err(e); + } + + // The linearization point. Before this the database path names the + // original; after it, the rotated file. There is no in-between state a + // reader can observe, which is the entire reason for the copy. + std::fs::rename(&tmp_path, path).map_err(ChiselError::IoError)?; + + // A rename is durable only once the DIRECTORY entry is durable. Without + // this, a crash right after a successful rename can resurrect the old name + // binding on some filesystems — the file contents are safe (they were + // fsynced) but the path could still point at the original inode. + if let Some(dir) = path.parent() { + // An empty parent means `path` was relative with no directory + // component; the current directory is the right target then. + let dir = if dir.as_os_str().is_empty() { + Path::new(".") + } else { + dir + }; + if let Ok(f) = std::fs::File::open(dir) { + // Best-effort: some platforms refuse to fsync a directory handle. + // The data itself is already durable, so a failure here costs + // rename durability across a crash, not correctness of the file. + let _ = f.sync_all(); + } + } + + // Drop the handle only now, so the flock is held across the rename and no + // other process can open the database mid-rotation. The handle's file + // descriptor refers to the ORIGINAL (now unlinked) inode, which is exactly + // why `rekey` does not hand it back: any further use would silently read + // and write a deleted file. + drop(db); + Ok(()) +} + +/// The Argon2 cost parameters the supplied key's CURRENT slot uses, if any. +/// +/// `None` when the database has no crypto header, when the key unlocks no slot +/// (unreachable — `Chisel::open` already proved it does), or when the slot is +/// an HKDF slot, whose params are zero by format definition and meaningless to +/// carry forward. In every one of those cases `wrap_into` falls back to its own +/// default, which is correct: a raw key never reads these fields at all. +fn inherited_argon2(db: &Chisel, key: &Key) -> Option { + let header = db.txm.crypto_header.as_ref()?; + let (idx, _dek) = header.unlock(key).ok()?; + let slot = &header.slots[idx]; + if slot.kdf_id == crate::crypto::KdfId::Argon2id as u8 { + Some(slot.argon2) + } else { + None + } +} + +/// Re-read the winning superblock through the live handle. +/// +/// `open_existing` already selected and decrypted it, but the decrypted result +/// lives inside the manager rather than being handed back, so read slot +/// `txn_counter % superblock_count` — the slot the last commit wrote — and +/// decrypt it with the session cipher. +fn current_superblock(db: &mut Chisel) -> Result { + let txn_counter = db.txm.txn_counter; + let n = db.txm.superblock_count as u64; + let mut cache = db.txm.cache.borrow_mut(); + let cipher = db + .txm + .cipher + .as_ref() + .ok_or(ChiselError::EncryptionNotSupported)?; + + // Try the expected slot first, then every other, so a torn slot that + // `select` already routed around does not defeat this. + let mut unit = [0u8; crate::crypto::ENC_PAGE_SIZE]; + let expected = txn_counter % n; + let order = std::iter::once(expected).chain((0..n).filter(|s| *s != expected)); + for slot in order { + cache.io_mut().read_page_unit_into(slot, &mut unit)?; + let mut image = [0u8; PAGE_SIZE]; + image.copy_from_slice(&unit[..PAGE_SIZE]); + if let Some(mut sb) = Superblock::deserialize(&image) { + if sb.txn_counter == txn_counter && sb.decrypt_body(cipher, &image).is_ok() { + return Ok(sb); + } + } + } + // The manager opened from one of these slots moments ago, so failing here + // means the file changed underneath a held flock. + Err(ChiselError::CorruptSuperblock { + defects: Vec::new(), + }) +} + +/// Create the scratch file with the same hardening the database itself gets: +/// mode 0600, `O_EXCL` so we never adopt a file we did not create, and +/// `O_NOFOLLOW` so a planted symlink at the predictable scratch path cannot +/// redirect the write. +fn create_scratch(tmp_path: &Path) -> Result { + let mut opts = std::fs::OpenOptions::new(); + opts.read(true).write(true).create_new(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt; + opts.mode(0o600).custom_flags(libc::O_NOFOLLOW); + } + match opts.open(tmp_path) { + Ok(f) => Ok(f), + Err(e) if e.kind() == std::io::ErrorKind::AlreadyExists => { + // Debris from a crashed earlier attempt, or someone squatting on a + // predictable path. Same discipline as the spillway sidecar: remove + // it ONLY if it is a plain file this uid owns with a single link, + // and never unlink another user's entry. Then retry, still + // exclusively, so a re-plant between the two syscalls loses. + reclaim_scratch(tmp_path)?; + opts.open(tmp_path).map_err(ChiselError::IoError) + } + Err(e) => Err(ChiselError::IoError(e)), + } +} + +/// Remove a pre-existing scratch entry, but only when it is plausibly our own +/// debris. Mirrors `spillway::reclaim_stale_sidecar` — same threat (a +/// predictable path in a directory another local user may write), same answer. +#[cfg(unix)] +fn reclaim_scratch(path: &Path) -> Result<()> { + use std::os::unix::fs::MetadataExt; + let md = std::fs::symlink_metadata(path).map_err(ChiselError::IoError)?; + // SAFETY: geteuid() is a pure read of process credentials; it cannot fail + // and touches no memory we own. + let ours = + md.file_type().is_file() && md.nlink() == 1 && md.uid() == unsafe { libc::geteuid() }; + if !ours { + return Err(ChiselError::IoError(std::io::Error::new( + std::io::ErrorKind::PermissionDenied, + "rekey scratch path is occupied by an entry this process did not create \ + (symlink, foreign owner, or extra hard link) — refusing to remove it", + ))); + } + std::fs::remove_file(path).map_err(ChiselError::IoError) +} + +#[cfg(not(unix))] +fn reclaim_scratch(path: &Path) -> Result<()> { + std::fs::remove_file(path).map_err(ChiselError::IoError) +} + +/// Write the fully-rotated database into `tmp_path` and fsync it. +/// +/// Page classes are handled differently, and the distinction is the crux of +/// the whole operation: +/// +/// * superblock slots are NOT `PageCipher`-sealed — their sensitive body is +/// sealed by `serialize_encrypted`, and their crypto header is cleartext. +/// They are rebuilt from `winner` under the new header and new DEK. +/// * every other page IS `PageCipher`-sealed with AAD = page_id, so it is +/// opened under the old DEK and re-sealed under the new one at the same +/// page id. Keeping the page id fixed is what lets the roots stay valid: +/// no pointer in the tree changes, only the bytes under each pointer. +#[allow(clippy::too_many_arguments)] +fn build_rotated_file( + db: &mut Chisel, + tmp_path: &Path, + page_count: u64, + superblock_count: u32, + winner: &Superblock, + new_header: &CryptoHeader, + old_cipher: &PageCipher, + new_cipher: &PageCipher, +) -> Result<()> { + use std::io::Write; + + let mut out = create_scratch(tmp_path)?; + let mut cache = db.txm.cache.borrow_mut(); + + // Superblock bank. Every slot carries the same roots; counters descend from + // the winner's in rotation order, so the winner still wins selection and + // the next commit's `(counter + 1) % N` target is still the slot it would + // have been. Staggering rather than writing one counter N times keeps + // `select`'s tie-break from having to choose between identical slots. + let n = superblock_count as u64; + let top = winner.txn_counter; + for k in 0..n { + let slot = (top + n - k) % n; + let mut sb = winner.clone(); + sb.txn_counter = top.saturating_sub(k); + sb.encryption = Some(*new_header); + let image = sb.serialize_encrypted(new_cipher); + let mut unit = [0u8; crate::crypto::ENC_PAGE_SIZE]; + unit[..image.len()].copy_from_slice(&image); + out.seek_write_unit(slot, &unit)?; + } + + // Data pages. + let mut unit = [0u8; crate::crypto::ENC_PAGE_SIZE]; + for page_id in n..page_count { + cache.io_mut().read_page_unit_into(page_id, &mut unit)?; + // A unit that does not decrypt under the current DEK is copied through + // verbatim rather than failing the rotation, and the distinction + // matters more than it looks. + // + // Two kinds of unit legitimately do not decrypt in a HEALTHY database: + // + // * never-written space — the file grew past a page id that was never + // filled, which POSIX zero-fills; and + // * crash debris — a process killed mid-commit can leave a TORN unit + // (a partial write of the 8232-byte blob) at a page id the winning + // superblock's freemap marks free. Shadow paging recovers by simply + // never reading those pages, which is why the database opens and + // works perfectly afterwards. + // + // Refusing to rotate on either would make this operation fail + // deterministically on a database that is completely fine — and fail at + // precisely the moment it exists for, since you reach for a DEK + // rotation when something has already gone wrong. That is the worst + // possible time to discover an availability trap. + // + // Copying verbatim cannot lose committed data: a page that is part of + // the committed state MUST decrypt under the current DEK, or the + // database could not be read at all. So anything landing here is by + // construction not live. What it does give up is using rekey as a + // corruption detector — a genuinely damaged LIVE page would be carried + // across rather than reported. That is the right trade: rekey is a key + // rotation, not an fsck, and a damaged live page already surfaces as + // DecryptionFailed on the read path where it belongs. + // + // Preserving the bytes (rather than zeroing) also keeps the file + // byte-length and page geometry identical, which the tests assert. + match old_cipher.open(page_id, &unit) { + Ok(plaintext) => { + let sealed = new_cipher.seal(page_id, &plaintext); + out.seek_write_unit(page_id, &sealed)?; + } + Err(_) => out.seek_write_unit(page_id, &unit)?, + } + } + + out.flush().map_err(ChiselError::IoError)?; + // Durable before the rename, not after: the rename publishes this file, and + // publishing bytes that are not yet on disk is exactly the crash window the + // copy strategy exists to avoid. + out.sync_all().map_err(ChiselError::IoError)?; + Ok(()) +} + +/// Small helper so the two write sites above read as "put this unit at this +/// page id" rather than repeating the offset arithmetic. +trait SeekWriteUnit { + fn seek_write_unit( + &mut self, + page_id: u64, + unit: &[u8; crate::crypto::ENC_PAGE_SIZE], + ) -> Result<()>; +} + +impl SeekWriteUnit for std::fs::File { + fn seek_write_unit( + &mut self, + page_id: u64, + unit: &[u8; crate::crypto::ENC_PAGE_SIZE], + ) -> Result<()> { + use std::io::{Seek, SeekFrom, Write}; + self.seek(SeekFrom::Start( + page_id * crate::crypto::ENC_PAGE_SIZE as u64, + )) + .map_err(ChiselError::IoError)?; + self.write_all(unit).map_err(ChiselError::IoError) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::Handle; + use tempfile::TempDir; + use zeroize::Zeroizing; + + fn raw(b: u8) -> Key { + Key::Raw(Zeroizing::new(vec![b; 32])) + } + + /// Build a database with enough content to span several pages, including an + /// overflow chain and a tagged chunk, so the rotation has real structure to + /// preserve rather than one page of nothing. + fn seed(path: &Path, key: &Key) -> Vec<(Handle, Vec)> { + let mut db = Chisel::open(path, Options::default().encryption_key(key.clone())).unwrap(); + let mut out = Vec::new(); + db.begin().unwrap(); + for i in 0..40u8 { + let v = vec![i; 300]; + out.push((db.allocate(&v).unwrap(), v)); + } + // Larger than a page: exercises the overflow chain, whose pages are + // sealed exactly like any other and must survive the rotation. + let big = vec![0xC7u8; PAGE_SIZE * 3]; + out.push((db.allocate(&big).unwrap(), big)); + db.set_root_name("primary", out[0].0).unwrap(); + db.commit().unwrap(); + drop(db); + out + } + + fn assert_intact(path: &Path, key: &Key, expected: &[(Handle, Vec)]) { + let db = Chisel::open(path, Options::default().encryption_key(key.clone())).unwrap(); + for (h, v) in expected { + assert_eq!(&db.read(*h).unwrap(), v, "handle {h:?} did not survive"); + } + assert_eq!(db.get_root_name("primary").unwrap(), Some(expected[0].0)); + } + + #[test] + fn rekey_preserves_every_value_under_a_new_dek() { + let dir = TempDir::new().unwrap(); + let path = dir.path().join("rk.db"); + let k = raw(0x31); + let expected = seed(&path, &k); + + // Capture the on-disk bytes of a data page so we can prove they really + // changed — a rotation that silently no-ops would otherwise pass every + // behavioural assertion here. + let before = std::fs::read(&path).unwrap(); + + Chisel::rekey(&path, &k, None).unwrap(); + + let after = std::fs::read(&path).unwrap(); + assert_eq!(before.len(), after.len(), "page count must not change"); + assert_ne!( + before, after, + "every page is sealed under a new DEK, so the ciphertext must differ" + ); + + // The SAME credential still opens it: rekey rotates the data key, not + // the credential. + assert_intact(&path, &k, &expected); + } + + #[test] + fn rekey_actually_changes_the_data_key() { + // The byte-comparison in the test above does NOT prove this, and it is + // worth being explicit about why: `PageCipher::seal` draws a fresh + // random nonce on every call, so re-sealing under the SAME DEK also + // changes every byte in the file. A bug that reused `old_cipher` for + // sealing, or wrapped the OLD DEK into the new header, would sail past + // every other assertion here. + // + // So assert the property directly: after the rotation, the old cipher + // must no longer be able to open a data page. + let dir = TempDir::new().unwrap(); + let path = dir.path().join("rk.db"); + let k = raw(0x91); + seed(&path, &k); + + // Recover the pre-rotation DEK the same way an attacker holding the + // credential would: unwrap it from the on-disk key slot. + let old_cipher = { + let db = Chisel::open(&path, Options::default().encryption_key(k.clone())).unwrap(); + db.txm.cipher.clone().expect("encrypted fixture") + }; + // A page that is definitely a sealed data page (past the superblock bank). + let probe = crate::DEFAULT_SUPERBLOCK_COUNT as u64; + let read_unit = |p: &Path, id: u64| { + use std::io::{Read, Seek, SeekFrom}; + let mut f = std::fs::File::open(p).unwrap(); + let mut u = [0u8; crate::crypto::ENC_PAGE_SIZE]; + f.seek(SeekFrom::Start(id * crate::crypto::ENC_PAGE_SIZE as u64)) + .unwrap(); + f.read_exact(&mut u).unwrap(); + u + }; + assert!( + old_cipher.open(probe, &read_unit(&path, probe)).is_ok(), + "fixture must have a decryptable data page at {probe} before the rotation" + ); + + Chisel::rekey(&path, &k, None).unwrap(); + + assert!( + old_cipher.open(probe, &read_unit(&path, probe)).is_err(), + "the old DEK must no longer open any page — if it does, the data key \ + was not actually rotated and only the nonces changed" + ); + } + + #[test] + fn rekey_inherits_the_argon2_cost_of_the_slot_it_unlocks() { + // Silently re-wrapping at Argon2Params::default() would DOWNGRADE a + // database whose owner hardened its KDF — during the operation you run + // because you were compromised, with no signal at all. + use crate::Argon2Params; + let dir = TempDir::new().unwrap(); + let path = dir.path().join("rk.db"); + let pass = || Key::Passphrase(Zeroizing::new("a passphrase".to_string())); + // Distinct from the OWASP default (19456/2/1) in every field, and cheap + // enough to keep the test fast. + let hardened = Argon2Params { + m_cost: 8192, + t_cost: 3, + p_cost: 2, + }; + { + let mut db = Chisel::open( + &path, + Options::default() + .encryption_key(pass()) + .argon2_params(hardened), + ) + .unwrap(); + db.begin().unwrap(); + db.allocate(b"payload").unwrap(); + db.commit().unwrap(); + } + + Chisel::rekey(&path, &pass(), None).unwrap(); + + let db = Chisel::open(&path, Options::default().encryption_key(pass())).unwrap(); + let header = db.txm.crypto_header.expect("encrypted"); + let (idx, _) = header.unlock(&pass()).unwrap(); + assert_eq!( + header.slots[idx].argon2, hardened, + "rekey must carry the existing cost parameters forward, not reset them" + ); + } + + #[test] + fn rekey_survives_an_undecryptable_page() { + // A crash mid-commit can leave a TORN unit at a page id the winning + // superblock's freemap marks free. Shadow paging recovers by never + // reading it, so the database is healthy — but a rekey that insisted + // every unit decrypt would fail deterministically on that healthy + // database, at exactly the moment the operation is needed. + use std::io::{Seek, SeekFrom, Write}; + let dir = TempDir::new().unwrap(); + let path = dir.path().join("rk.db"); + let k = raw(0xA5); + let expected = seed(&path, &k); + + // Grow the file by one unit of garbage: a page id past everything the + // committed state references, holding bytes that are neither zeros nor + // valid ciphertext. This is the shape crash debris takes. + let stray = { + let mut f = std::fs::OpenOptions::new().write(true).open(&path).unwrap(); + let len = f.metadata().unwrap().len(); + let id = len / crate::crypto::ENC_PAGE_SIZE as u64; + f.seek(SeekFrom::Start(id * crate::crypto::ENC_PAGE_SIZE as u64)) + .unwrap(); + f.write_all(&[0x5Au8; crate::crypto::ENC_PAGE_SIZE]) + .unwrap(); + f.sync_all().unwrap(); + id + }; + + Chisel::rekey(&path, &k, None).expect("crash debris must not block a rotation"); + + assert_intact(&path, &k, &expected); + // The debris is carried across verbatim, so the file geometry is + // unchanged — nothing was dropped or zero-filled behind our back. + let len = std::fs::metadata(&path).unwrap().len(); + assert_eq!(len / crate::crypto::ENC_PAGE_SIZE as u64, stray + 1); + } + + #[test] + fn rekey_leaves_no_scratch_file_behind() { + let dir = TempDir::new().unwrap(); + let path = dir.path().join("rk.db"); + let k = raw(0x32); + seed(&path, &k); + Chisel::rekey(&path, &k, None).unwrap(); + + let scratch = dir.path().join("rk.db.rekey-tmp"); + assert!( + !scratch.exists(), + "the scratch file must be renamed away, not left beside the database" + ); + } + + #[test] + fn rekey_collapses_the_key_slot_table_to_the_supplied_credential() { + // The documented consequence, and the one most likely to surprise: the + // other credentials cannot be carried across, because each slot's KEK + // is derived from a credential we were not given. Asserting it here + // makes the doc a tested claim rather than a promise. + let dir = TempDir::new().unwrap(); + let path = dir.path().join("rk.db"); + let first = raw(0x41); + let second = raw(0x42); + let expected = seed(&path, &first); + { + let mut db = + Chisel::open(&path, Options::default().encryption_key(first.clone())).unwrap(); + db.add_key(&first, &second).unwrap(); + } + // Both work before. + assert_intact(&path, &first, &expected); + assert_intact(&path, &second, &expected); + + Chisel::rekey(&path, &first, None).unwrap(); + + assert_intact(&path, &first, &expected); + assert!( + Chisel::open(&path, Options::default().encryption_key(second.clone())).is_err(), + "a credential not supplied to rekey cannot survive it" + ); + // ...and can be added back, which is the documented remedy. + { + let mut db = + Chisel::open(&path, Options::default().encryption_key(first.clone())).unwrap(); + db.add_key(&first, &second).unwrap(); + } + assert_intact(&path, &second, &expected); + } + + #[test] + fn rekey_refuses_a_wrong_key_without_touching_the_file() { + let dir = TempDir::new().unwrap(); + let path = dir.path().join("rk.db"); + let k = raw(0x51); + let expected = seed(&path, &k); + let before = std::fs::read(&path).unwrap(); + + match Chisel::rekey(&path, &raw(0x52), None) { + Err(ChiselError::InvalidEncryptionKey) => {} + other => panic!("expected InvalidEncryptionKey, got {other:?}"), + } + + assert_eq!( + before, + std::fs::read(&path).unwrap(), + "a refused rekey must not modify a single byte" + ); + assert_intact(&path, &k, &expected); + } + + #[test] + fn rekey_refuses_a_plaintext_database() { + let dir = TempDir::new().unwrap(); + let path = dir.path().join("plain.db"); + { + let mut db = Chisel::open(&path, Options::default()).unwrap(); + db.begin().unwrap(); + db.allocate(b"cleartext").unwrap(); + db.commit().unwrap(); + } + // The open inside rekey supplies a key to a plaintext DB, which the + // open path itself rejects — the error the caller sees names the real + // mismatch either way. + match Chisel::rekey(&path, &raw(0x61), None) { + Err(ChiselError::EncryptionNotSupported) => {} + other => panic!("expected EncryptionNotSupported, got {other:?}"), + } + } + + #[test] + fn rekey_refuses_a_missing_file() { + let dir = TempDir::new().unwrap(); + match Chisel::rekey(&dir.path().join("nope.db"), &raw(0x71), None) { + Err(ChiselError::FileNotFound) => {} + other => panic!("expected FileNotFound, got {other:?}"), + } + } + + #[test] + fn the_database_stays_writable_after_a_rotation() { + // The rotated file must be a fully functional database, not just a + // readable one: the freemap, handle table and next_handle all come + // through the superblock body that was re-sealed under the new DEK. + let dir = TempDir::new().unwrap(); + let path = dir.path().join("rk.db"); + let k = raw(0x81); + let expected = seed(&path, &k); + + Chisel::rekey(&path, &k, None).unwrap(); + + let mut db = Chisel::open(&path, Options::default().encryption_key(k.clone())).unwrap(); + db.begin().unwrap(); + let fresh = db.allocate(b"written after the rotation").unwrap(); + db.delete(expected[1].0).unwrap(); + db.commit().unwrap(); + drop(db); + + let db = Chisel::open(&path, Options::default().encryption_key(k)).unwrap(); + assert_eq!(db.read(fresh).unwrap(), b"written after the rotation"); + assert!( + db.read(expected[1].0).is_err(), + "the delete must have stuck" + ); + assert_eq!(db.read(expected[0].0).unwrap(), expected[0].1); + } +} diff --git a/src/transaction/mod.rs b/src/transaction/mod.rs index 5f0e875..c3560dc 100644 --- a/src/transaction/mod.rs +++ b/src/transaction/mod.rs @@ -154,7 +154,7 @@ pub struct TransactionManager { // `borrow_mut()`; reborrowing for downstream `&mut PageCache` parameters // (e.g., handle_table methods) is done via `&mut *cache` on a single // RefMut held for the duration of the operation. - cache: RefCell, + pub(crate) cache: RefCell, // Roots that match the superblock currently on disk. Safe to read at any time. committed_roots: Roots, // Roots under construction. Equals committed_roots when no txn is active; @@ -167,14 +167,14 @@ pub struct TransactionManager { // Monotonically increasing. Written into each new superblock; the higher value // wins on recovery. Also used to pick the inactive slot on commit via // `txn_counter % superblock_count`. - txn_counter: u64, + pub(crate) txn_counter: u64, // Number of superblock slots occupying pages 0..superblock_count // (ISSUES.md R4). Set at open time from the winning superblock's // own `superblock_count` field; cached here so commit doesn't have // to re-fetch it. Must equal every slot's self-reported value in a // healthy database; divergence would indicate mid-flight reconfig // or corruption. - superblock_count: u32, + pub(crate) superblock_count: u32, active_txn: bool, savepoints: Vec, // Pages whose contents are no longer reachable from the new roots. @@ -221,12 +221,12 @@ pub struct TransactionManager { /// seal/open. The manager's copy is also threaded through CommitCtx for the /// superblock body seal on every commit. The DEK inside PageCipher is /// zeroizing and is cleared on drop. - cipher: Option, + pub(crate) cipher: Option, /// The crypto-header (algorithm id + key-slot table) for an encrypted database. /// Written verbatim into every committed superblock. `None` for plaintext DBs. /// The key-slot contents never change after create or open: the slots hold the /// DEK wrapped under KEKs from each user key and are opaque to the commit path. - crypto_header: Option, + pub(crate) crypto_header: Option, // Test-only fault injection consolidated off the production type (review // 2026-06-22 SMELL #4): the four BUG#2 atomic-staging arming flags live in