docs: split THEORY out of ARCHITECTURE, refresh all three docs for encryption - #86
Merged
Conversation
…cryption Draw the ARCHITECTURE/THEORY boundary (previously blended into one file) and bring the three top-level docs current with the merged on-disk encryption feature. - ARCHITECTURE.md: compressed to the cold-start, act-correctly map — kept every on-disk byte-format table, the commit/recovery diagrams, the failure table, and the glossary; extracted the narrative "why" and the implementation history to THEORY.md. Fixed three content bugs found against the code: the stale module map (transaction.rs / superblock.rs were decomposed into directory modules; the crypto/ layer and handle.rs / lru.rs / spillway.rs were missing), the superblock 324..8184 reserved-bytes contradiction (now conditional on plaintext vs encrypted), and encryption shown as shipped. - THEORY.md (new): theory of operation for an engineer about to change the engine — the 18 load-bearing decisions with their rejected alternatives, the implementation history, and the benchmark methodology. Cross-references ARCHITECTURE's invariant enumeration instead of duplicating it. Seven rationale gaps are marked "not recovered from project sources" rather than invented. - README.md: added the encryption surface (Options.encryption_key / argon2_params, Key / Argon2Params, add_key / rotate_key / remove_key, the five operational error variants + fatal DecryptionFailed, MAJOR=2, the Python encryption_key kwarg) and corrected the toolchain commands (the Python flow needs hypothesis and --release; the clippy / msrv / test gates now match CI).
Two of the seven "Rationale not recovered" gaps in THEORY.md are answered directly by the engine author and move up into the decision sections: - Page size 8 KB: chosen to align with the underlying storage block size so a page maps cleanly onto the device's block granularity (new subsection next to the durability decision). - superblock_count default = 2: a deliberate integrity/performance balance — 3 is defensible, but larger N costs considerable performance for little added safety (folded into the ADR-4 decision). Both are attributed as recorded from the engine author rather than a written ADR. Five genuine gaps remain, still marked "not recovered" rather than invented.
🚦 Bench results: PR vs main
Per-scenario detail (4 metrics × cells)document-store
mutation-log
ycsb-a
ycsb-b
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Refreshes the three top-level docs now that on-disk encryption has landed (#85), and draws the ARCHITECTURE / THEORY boundary that was never separated — the two were blended into a single
ARCHITECTURE.md.The split is by read pattern, not detail:
ARCHITECTURE.md (compressed 759 → 710, 3 content bugs fixed)
Kept every byte-format table, the commit/recovery diagrams, the failure table, and the glossary. Extracted the narrative "why" and the entire implementation-history / benchmark-methodology narrative to THEORY.md. Fixed three bugs surfaced by re-checking against the code:
transaction.rs/superblock.rs, but the code decomposed these intotransaction/(~14 submodules) andsuperblock/(+crypto_header.rs); the wholecrypto/layer andhandle.rs/lru.rs/spillway.rswere absent. Map and mermaid graph now match the code, including the encryption public surface.324..8184"reserved" while the encryption section put the key-slot table at324..1356. Now correctly conditional: plaintext DBs keep it reserved; encrypted DBs place the crypto-header + 8-slot key table there and seal the sensitive body under the DEK.THEORY.md (new)
The 18 load-bearing decisions, each as chosen / rejected-alternative / why, harvested from the ADR graph,
docs/specs/, and ISSUES.md — shadow-paging over WAL, single-writer, per-module COW, the poison model, the spillway, chunk tags, the encryption cipher/keys/format, the MSRV pins. Plus the implementation history and benchmark methodology. It cross-references ARCHITECTURE's invariant enumeration rather than restating it (verified: zero byte-tables in THEORY vs 31 in ARCHITECTURE; all cross-reference anchors resolve).Seven rationale gaps are marked
> Rationale not recovered from project sourcesrather than invented (why XXH3, why 8 KB pages, why 8 key-slots, why superblock default = 2, the Argon2 default costs, the spillway 1024× multiplier, the membership depth-6 bound). If any of these reasons are known, they're worth filling in.README.md (updated)
Added the encryption surface (
Options.encryption_key/argon2_params,Key/Argon2Params,add_key/rotate_key/remove_key, the five operational error variants + fatalDecryptionFailed, MAJOR=2, the Pythonencryption_keykwarg). Corrected the toolchain commands againstCargo.toml/pyproject.toml/ci.yml— the Python flow was missinghypothesisand used debug builds where CI uses--release; the clippy / msrv / test gates now match CI. Human-authored prose preserved.Verification
Docs-only; no code touched. All internal cross-references resolve; ARCHITECTURE's own TOC has no dangling anchors after the section moves; the documented error variants and byte offsets were checked against
src/.