feat: bulk DEK rotation — re-encrypt a whole database under a fresh data key (#140)

ARCHITECTURE.md

@@ -712,9 +712,11 @@ The DEK wrapping uses detached XChaCha20-Poly1305 with AAD bound to the slot's K
 
 - `add_key(old_key, new_key)`: derives a new KEK, wraps the same DEK into a free slot, then commits.
 - `rotate_key(old_key, new_key)`: `add_key` followed by clearing the old slot in the same commit.
+- Every key-slot rewrite also scrubs the key-slot region of the OTHER superblock slots (`CryptoHeader::overwrite_slot_table`). Without that, a revoked credential's wrapped DEK survived verbatim in a sibling slot of the live file — cleartext at bytes 332..1356, wrapping a DEK that never changes — so read access plus the revoked credential was enough to recover the current DEK. Only the table region is patched, so each sibling keeps its own counter, roots and sealed body; the body's AAD covers magic/format_version/txn_counter/superblock_count, not the crypto header.
+- `Chisel::rekey(path, key, argon2)` (ADR 0018) is the bulk alternative: a fresh DEK, every page re-sealed, the file replaced by atomic rename. It is an associated function on a path rather than a method because the rename invalidates any descriptor opened beforehand, and it collapses the key-slot table to the supplied credential because a slot's KEK cannot be re-derived without its own credential.
 - `remove_key(key)`: clears the matching slot, refusing to clear the last active slot (which would make the database permanently unreadable).
 
-Bulk DEK rotation (re-encrypting every page under a fresh DEK) is deferred; see I142.
+Bulk DEK rotation is implemented as `Chisel::rekey` (ADR 0018) — the operation for a compromised DEK, as opposed to a compromised credential. It is offline and takes a path, builds the rotated database in a scratch file, and publishes it with an atomic rename; it collapses the key-slot table to the single supplied credential, because a slot's KEK cannot be re-derived without its own credential.
 
 **Spillway.** For encrypted databases the in-memory spillway carries sealed blobs: pages are encrypted exactly once on eviction (`seal` on evict-to-spillway) and copied verbatim — no decryption or re-encryption — on drain to the main file. Rehydration decrypts the blob back into the cache. No plaintext page content is ever written to disk by an encrypted database, even during spill.
 

README.md

@@ -25,7 +25,7 @@ Chisel is designed for single-writer embedded use: one process holds the file vi
 - **Named roots** — a small fixed table in the superblock mapping string names to handles. Survives commit / rollback transactionally.
 - **Defragmentation** — explicit `defrag()` consolidates sparse pages and returns a count-based stats record.
 - **In-memory mode** — same engine, `Vec<u8>`-backed I/O, no file and no lock. For tests, benchmarks, and ephemeral work.
-- **At-rest encryption** — optional, off by default. Every page is sealed with XChaCha20-Poly1305 AEAD under a client-supplied key (raw 32-byte or Argon2id passphrase). An 8-slot envelope table wraps the data-encryption key, so credential rotation is O(1) — no bulk re-encryption.
+- **At-rest encryption** — optional, off by default. Every page is sealed with XChaCha20-Poly1305 AEAD under a client-supplied key (raw 32-byte or Argon2id passphrase). An 8-slot envelope table wraps the data-encryption key, so credential rotation is O(1) — no bulk re-encryption. `rekey()` performs the heavy alternative when the data key itself is compromised: a crash-safe whole-file re-encryption under a fresh key.
 - **Poison model** — any fatal error (I/O failure, checksum mismatch, commit-protocol failure) poisons the handle; recovery is drop-and-reopen. Mirrors `std::sync::Mutex` poisoning.
 - **Single-writer** — exclusive `flock` at the filesystem level; `&mut self` on every mutating method.
 
@@ -293,6 +293,12 @@ On create, Chisel generates a random data-encryption key (DEK), encrypts every p
 
 The wrapped DEK lives in an **8-slot key table**. Because the DEK itself never changes, credential rotation only re-wraps the DEK in a slot — it is O(1), independent of database size. `add_key` stages a second credential (both open the DB), `rotate_key` replaces one credential in place, and `remove_key` retires one (refusing the last remaining slot with `LastKeySlot`). A full table returns `NoFreeKeySlot`.
 
+Rotating a *credential* is not the same as rotating the *data key*. Because the DEK never changes, `rotate_key` and `remove_key` deny a credential a way in — they do not cut it off from data it has already seen, and they cannot help if the DEK itself leaked (a process memory dump, a core file). `Chisel::rekey(path, key, argon2_params)` is the operation for that case: it generates a fresh DEK, re-encrypts every page, and replaces the file atomically.
+
+`rekey` takes a **path, not an open handle**, and that is deliberate: it replaces the file by `rename`, so a descriptor opened beforehand would afterwards refer to the original, now-unlinked inode. Close the database, rekey, reopen. It is crash-safe — the rotated database is built in a scratch file beside the original and published atomically, so the path only ever names a complete file — and it costs O(total_pages) I/O plus a transient second copy on disk.
+
+It also **collapses the key-slot table to the single credential you supply**. That is forced rather than chosen: each slot's KEK is derived from its own credential, so wrapping the new DEK for a slot requires the credential that slot belongs to. Re-add the others with `add_key` afterwards. It is the safer default anyway — if you are rotating because the DEK leaked, preserving every credential that could reach it is not the goal.
+
 See [ARCHITECTURE.md#on-disk-encryption](ARCHITECTURE.md#on-disk-encryption) for the on-disk layout (crypto header, key slots, per-page nonce stride) and [THEORY.md](THEORY.md) for the rationale behind the envelope scheme and the shadow-paging nonce discipline (with [`docs/adr/`](docs/adr/) as the dated decision log).
 
 ## API reference
@@ -332,6 +338,7 @@ See [ARCHITECTURE.md#on-disk-encryption](ARCHITECTURE.md#on-disk-encryption) for
 | `defrag(options)` | Consolidate sparse pages |
 | `add_key(existing, new)` | Stage a second credential; both `existing` and `new` then open the DB. `Result<()>` |
 | `rotate_key(old, new)` | Replace credential `old` with `new` in place. `Result<()>` |
+| `Chisel::rekey(path, key, argon2)` | Re-encrypt the whole database under a fresh DEK. Associated function, not a method — takes a path. `Result<()>` |
 | `remove_key(key)` | Retire credential `key`; `LastKeySlot` if it is the only one. `Result<()>` |
 
 ## Options
@@ -469,6 +476,12 @@ db.remove_key("correct horse battery staple")                          # retire
 
 `add_key` / `rotate_key` raise `NoFreeKeySlotError` when the 8-slot table is full; `remove_key` raises `LastKeySlotError` rather than leaving the DB with no usable credential.
 
+```python
+chisel.rekey("my.db", key=b"\x00" * 32)   # whole-file re-encryption under a fresh data key
+```
+
+`chisel.rekey` is a module-level function, like `chisel.open`, because it names a database by path rather than assuming one is open — it replaces the file, so any handle held across the call would be stale. Every credential other than the one supplied is revoked; re-add them with `add_key`.
+
 ## Design documents
 
 - [`ARCHITECTURE.md`](ARCHITECTURE.md) — living architecture overview: layer model, commit protocol, recovery, full on-disk format byte-by-byte, and cross-cutting concepts. Start here if you're reading the codebase to *act* on it.

THEORY.md

@@ -155,13 +155,13 @@ There is one **critical, revised** sub-decision that is easy to get wrong and wo
 
 ### Encryption keys: envelope DEK/KEK with an 8-slot table, O(1) rotation (ADR-15)
 
-**Chosen.** A random per-database 256-bit DEK (from `OsRng` at create) seals every page and the sensitive superblock body. The DEK is never stored bare — it is wrapped under a KEK derived from the client key (HKDF-SHA256 for raw keys, Argon2id for passphrases) and held in an 8-slot key-slot table in the superblock's plaintext reserved region. `add_key` / `rotate_key` / `remove_key` re-wrap the *stable* DEK — O(1), no data re-encryption. `rotate_key` stages the new slot before revoking the old (no zero-key window); `remove_key` refuses to clear the last active slot (brick prevention). A successful unwrap *is* proof the client key is correct — there is no separate password verifier.
+**Chosen.** A random per-database 256-bit DEK (from `OsRng` at create) seals every page and the sensitive superblock body. The DEK is never stored bare — it is wrapped under a KEK derived from the client key (HKDF-SHA256 for raw keys, Argon2id for passphrases) and held in an 8-slot key-slot table in the superblock's plaintext reserved region. `add_key` / `rotate_key` / `remove_key` re-wrap the *stable* DEK — O(1), no data re-encryption. `rotate_key` stages the new slot before revoking the old (no zero-key window); `remove_key` refuses to clear the last active slot (brick prevention). A successful unwrap *is* proof the client key is correct — there is no separate password verifier. Rotating a credential is not rotating the data key: the DEK is unchanged, so revocation denies entry rather than re-keying, and anyone who captured the DEK while the credential was valid keeps reading. `rekey` (ADR 0018) is the answer when the DEK itself is what leaked — a whole-file re-encryption published by atomic rename, since shadow paging cannot make an in-place rewrite crash-safe.
 
-**Rejected.** Encrypting only the data pages and leaving the superblock plaintext — rejected because `named_roots` holds user-chosen names, which are real user data a plaintext body would leak. Full DEK rotation (re-encrypting every page under a fresh DEK) — deferred (I142) as a heavy O(total_pages) whole-file operation reserved for "the DEK itself is compromised"; credential rotation is the far more common need and is O(1).
+**Rejected.** Encrypting only the data pages and leaving the superblock plaintext — rejected because `named_roots` holds user-chosen names, which are real user data a plaintext body would leak. Full DEK rotation (re-encrypting every page under a fresh DEK) — implemented separately as `rekey` (ADR 0018) rather than folded into the credential path, because the two answer different questions: credential rotation is the far more common need and is O(1), while a whole-file re-encryption is reserved for "the DEK itself is compromised" and is O(total_pages).
 
 **Why.** Envelope encryption makes credential rotation O(1) — you re-wrap the DEK — instead of O(database size). The per-slot KDF choice matches input entropy: HKDF is fast and correct for high-entropy keys, while Argon2id is memory-hard to resist brute-forcing low-entropy passphrases (its params are recorded per slot). And every rotation op is an ordinary superblock A/B + fsync commit, so it reuses the existing crash-safe protocol wholesale: a metadata-only `rewrite_crypto_header` commit persists a rotated slot table atomically (write the inactive slot, fsync, promote), so a crash mid-rotation leaves the old table intact.
 
-Two threat-model boundaries are documented rather than solved, and you should know them before you rely on this: there is **no rollback/replay resistance** (an attacker who substitutes a wholly older, validly-signed image is undetectable without an external trust anchor like a TPM), and the DEK sits in plaintext in process memory during a session (mitigated by zeroize-on-drop, not by encryption). See spec `2026-06-29` §3/§5/§9 and [issue #140](https://github.com/pgexperts/chisel/issues/140) (the deferred bulk DEK rotation, formerly I142).
+Two threat-model boundaries are documented rather than solved, and you should know them before you rely on this: there is **no rollback/replay resistance** (an attacker who substitutes a wholly older, validly-signed image is undetectable without an external trust anchor like a TPM), and the DEK sits in plaintext in process memory during a session (mitigated by zeroize-on-drop, not by encryption). See spec `2026-06-29` §3/§5/§9. Note that `rekey` (ADR 0018) does bear on the first boundary: it invalidates every page image sealed under the old DEK, so it ends an attacker's ability to splice in stale pages captured beforehand — it does not, however, make the engine detect such splicing, which remains open as [issue #142](https://github.com/pgexperts/chisel/issues/142).
 
 ### Encryption page format: 8232-byte stride, logical page stays 8192, MAJOR 1→2 (ADR-15)
 

docs/adr/0018-bulk-dek-rotation-via-copy-and-rename.md

@@ -0,0 +1,109 @@
+---
+id: 0018
+title: Bulk DEK rotation as an offline copy-then-rename operation on a path
+date: 2026-08-04
+status: Accepted
+summary: rekey() re-encrypts the whole database under a fresh DEK by building a replacement file and renaming it into place; it takes a path rather than a handle, and collapses the key-slot table to the single supplied credential.
+---
+
+# 0018. Bulk DEK rotation as an offline copy-then-rename operation on a path
+
+## Context
+
+ADR [0015](0015-on-disk-encryption-xchacha20-poly1305-envelope-keys.md) chose an
+envelope scheme: one per-database DEK seals every page, and up to eight key
+slots each wrap that DEK under a KEK derived from a client credential. That
+makes *credential* rotation O(1) — `add_key`/`rotate_key`/`remove_key` touch
+only the superblock, and no page is re-encrypted. It was recorded at the time
+that full DEK rotation was deferred.
+
+Deferred is not the same as unnecessary. The two operations answer different
+questions, and only one of them was answerable:
+
+- a credential leaked → deny that credential a way in → credential rotation
+- **the DEK itself leaked** (process memory dump, core file, attached debugger)
+  → the data must be re-sealed under a key the attacker does not have
+
+Credential rotation cannot help with the second case at all, because the DEK it
+re-wraps is the very thing that leaked. The fix for CRYPTO-1 sharpened the point
+by documenting it in `rotate_key`: revocation denies entry, it does not
+re-key. Without a bulk rotation there was no operation in the crate that did.
+
+Two things made the design non-obvious.
+
+**Crash safety cannot lean on shadow paging.** Shadow paging protects writes
+that go to *new* pages; a DEK rotation rewrites every page where it already is.
+A crash halfway through leaves some pages sealed under the new DEK and some
+under the old, with the surviving superblock naming one of them — an
+unrecoverable mix, and precisely the kind of half-state the engine otherwise
+never produces.
+
+**The operation invalidates its own handle.** Any strategy that replaces the
+file by rename leaves a previously-opened file descriptor pointing at the
+original, now-unlinked inode. Reads and writes through it would silently target
+a deleted file.
+
+## Decision
+
+We will implement `Chisel::rekey(path, key, argon2_params)` as an **offline
+operation on a path**, which builds the rotated database in a scratch file
+beside the original and publishes it with an atomic `rename`.
+
+Concretely: open normally (validating the key and taking the exclusive flock),
+generate a fresh DEK, write every page into `<db>.rekey-tmp` — superblock slots
+rebuilt from the winning superblock under a new crypto header, every other page
+opened under the old DEK and re-sealed under the new one *at the same page id* —
+`fsync`, `rename`, `fsync` the directory, then drop the handle.
+
+The rotated database carries **exactly one key slot**: the credential supplied.
+
+## Alternatives considered
+
+- **In-place two-pass rotation.** Rejected: not crash-safe for the reason
+  above, and making it so would need a journal — a second durability mechanism
+  in an engine whose entire premise (ADR
+  [0001](0001-shadow-paging-not-wal.md)) is that it does not have one.
+
+- **`rekey(&mut self)` on a live handle.** Rejected: it would hand back a
+  handle whose file descriptor names a deleted inode. Making that safe means
+  swapping the descriptor and re-acquiring the flock underneath a live cache —
+  real complexity to preserve an ergonomic nicety on an operation that rewrites
+  the entire file and is not something anyone runs in a loop.
+
+- **`rekey(self)` consuming the handle.** Closer, and it does dispose of the
+  stale-descriptor hazard, but `Chisel` does not retain its path, so the caller
+  would have to pass it back in and could pass the wrong one. A path-taking
+  associated function has one source of truth and reads as what it is.
+
+- **Re-wrapping the new DEK into all currently-active slots.** Not
+  implementable, not merely declined: each slot's KEK is derived from *its own*
+  credential, and only one was supplied. There is no way to produce a valid wrap
+  for a credential you do not hold. It is also the safer default — if you are
+  rotating because the DEK leaked, silently preserving every credential that
+  could reach it is not what you want. `add_key` restores the others.
+
+- **Keeping the older superblock slots' historical roots.** Declined. Their
+  sealed bodies are under the old DEK, so preserving them means re-sealing
+  states that are already unreachable. Every slot in the new file carries the
+  winning superblock's roots at staggered counters, exactly as `create_new`
+  seeds a fresh bank.
+
+## Consequences
+
+- Rotation costs O(total_pages) I/O and, transiently, a second copy of the
+  database on disk. It is maintenance, not routine operation.
+- The scratch file is created `O_EXCL | O_NOFOLLOW` mode 0600, matching the
+  hardening the database and spillway received, so a planted file at the
+  predictable scratch path cannot be adopted or followed.
+- The directory `fsync` after the rename is best-effort: the file contents are
+  already durable, so a failure costs rename durability across a crash rather
+  than correctness.
+- `rekey` must live outside `transaction/`, since it reaches across an open
+  handle and the filesystem at once. It is its own module.
+- Because it is a free function on a path, the PyO3 binding exposes it as a
+  module-level `chisel.rekey(path, key)` mirroring `chisel.open()` rather than
+  as a method — the same reasoning carried across the boundary.
+- Rotation does not defend against per-page temporal replay
+  ([#142](https://github.com/pgexperts/chisel/issues/142)); it does, however,
+  invalidate every previously-sealed page image, so it is the operation that
+  ends an attacker's ability to splice stale pages sealed under the old DEK.

docs/adr/README.md

@@ -22,3 +22,4 @@
 | [0015](0015-on-disk-encryption-xchacha20-poly1305-envelope-keys.md) | On-disk encryption (XChaCha20-Poly1305, envelope keys) | Accepted | 2026-06-30 |  |
 | [0016](0016-swift-binding-via-uniffi.md) | Swift binding via UniFFI | Accepted | 2026-07-19 | The iOS/macOS Swift binding is a UniFFI-generated FFI over an Arc<Mutex<Chisel>> wrapper crate, with the engine crate left unchanged. |
 | [0017](0017-github-issues-replace-tracked-issues-md.md) | Track issues in GitHub, not a tracked ISSUES.md | Accepted | 2026-08-03 | The 1868-line ISSUES.md decision log was retired; open entries were migrated to GitHub issues and the file deleted. |
+| [0018](0018-bulk-dek-rotation-via-copy-and-rename.md) | Bulk DEK rotation as an offline copy-then-rename operation on a path | Accepted | 2026-08-04 | rekey() re-encrypts the whole database under a fresh DEK by building a replacement file and renaming it into place; it takes a path rather than a handle, and collapses the key-slot table to the single supplied credential. |

python/chisel/__init__.py

@@ -17,6 +17,7 @@
     Savepoint,
     DrainInsertion,
     open,
+    rekey,
     ChiselError,
     OperationalError,
     FatalError,
@@ -130,7 +131,7 @@ class DefragStats:
 
 __all__ = [
     "__version__",
-    "Chisel", "Transaction", "Savepoint", "DrainInsertion", "open",
+    "Chisel", "Transaction", "Savepoint", "DrainInsertion", "open", "rekey",
     "Stats", "Counters", "DefragOptions", "DefragStats",
     "ChiselError", "OperationalError", "FatalError",
     "InvalidHandleError", "NoActiveTransactionError",