Skip to content

feat(pinax): implement Phase 01 pager, buffer pool, and B+tree - #11

Merged
forkwright merged 7 commits into
mainfrom
feat/phase01-pager-buffer-pool-btree
Aug 16, 2026
Merged

feat(pinax): implement Phase 01 pager, buffer pool, and B+tree#11
forkwright merged 7 commits into
mainfrom
feat/phase01-pager-buffer-pool-btree

Conversation

@forkwright

@forkwright forkwright commented Aug 15, 2026

Copy link
Copy Markdown
Owner

Summary

Lands ROADMAP.md Phase 01 ("pager + buffer pool + B-tree") on top of #10's workspace + lexis scaffold. Every success criterion in that phase gets a real test: open a file, CRUD rows by integer key, survive crash-and-reopen, page-format checksums verified with corruption detected, buffer pool handles a database larger than its capacity.

Base branch is #10's feat/2-phase01-workspace-lexis, not main — this PR should merge (or be rebased) after #10.

Page format (Decision 2)

  • PageSize validates the five locked sizes (4096 / 8192 / 16384 / 32768 / 65536), default 4096.
  • Every page — meta, leaf, interior, overflow — carries a trailing 8-byte XxHash3-64 checksum (xxhash-rust, pure Rust, matching Turso's algorithm choice per the design-prior citation in PLAN.md).
  • max_local = usable_space - 35, the SQLite formula PLAN.md names verbatim. Rows over that threshold spill into an overflow page chain (own encoding, not SQLite's varint format — Decision 1 says pinax owns its on-disk encoding).

Pager (Decision 1)

The meta/header lives in two checksummed 4 KiB slots (page ids 0 and 1) at fixed file offsets, independent of the database's configured page size — bootstrap has to learn page_size from the meta page before it can compute anything page_size-relative, so the meta region's own layout can't depend on that value.

Every mutation is copy-on-write: nothing already reachable from the active meta page is ever overwritten in place, every change targets a freshly allocated page id. Commit fsyncs the data pages, writes the other meta slot with an incremented txn_id, fsyncs again, then flips which slot is active. A crash at any point leaves the previously-committed slot intact — there's nothing to roll back because nothing reachable ever changed. This is the mechanism the crash-and-reopen test demonstrates, and it needs no WAL (that's Phase 02).

Deliberate Phase 01 scope limit: no freelist reclamation. A page retired by a transaction's copy-on-write is still referenced by the currently active meta until that transaction's own meta write lands, so reusing its id before that point would corrupt the still-authoritative pre-transaction state on a crash in between. Reusing an id retired by an earlier, already-committed transaction is safe in principle, but Phase 01's allocator is a pure bump counter that doesn't attempt it — the database grows monotonically instead. None of Phase 01's acceptance criteria require space reclamation; this is real, deferred scope, not a correctness gap.

Buffer pool

Capacity-bounded LRU cache over the pager. Because every write is copy-on-write, get only ever needs to clone a page's bytes out — nothing is ever checked out or held mutably against the pool, so eviction has no aliasing hazard and needs no pinning bookkeeping. A page evicted mid-operation is safe to flush early for the same reason a crash mid-operation is safe: it isn't reachable from the active meta yet either way.

The "buffer pool handles a database larger than RAM" test models RAM as pool capacity (8 pages), inserts 4,000 rows to produce a tree many times that size on disk, and reads every row back correctly after eviction-and-reload cycles.

B+tree

Slotted leaf and interior pages (SQLite-shape layout per Decision 1, own encoding). Insert/delete path-copy: the whole root-to-leaf path gets a fresh id chain, propagated bottom-up (apply_result_to_interior), with splits handled at both leaf and interior levels. Rows are encoded as lexis::Value tuples — lexis's own Cargo.toml names this as its Phase 01 exit criterion ("Phase 1 pager adopts lexis::Value for on-disk row encoding").

Deliberate Phase 01 scope limit: delete does not merge/rebalance underflowing nodes. delete's interior-propagation step only ever repoints an existing child pointer, it never removes a separator key, so an ancestor's key count is monotonically non-decreasing — the tree stays correct (an empty leaf is a valid leaf) but not space-optimal under a delete-heavy workload. None of Phase 01's acceptance criteria require this; general merge-on-underflow is real follow-up scope for a later phase, not silently dropped.

Testing

  • tests/phase01_acceptance.rs: one test per ROADMAP.md Phase 01 criterion, named and cited against the exact wording. The corruption test flips a byte on disk and asserts a typed Corruption error — the negative case, not just the intact-page happy path. The crash test drops a Database mid-stream with no explicit shutdown and reopens.
  • Every module carries colocated #[cfg(test)] mod tests for its own internals — codec bounds-checking, checksum round-trip and flip-detection, meta-slot fallback on corruption (a second, independent demonstration of crash-safety: corrupt the active meta slot directly and confirm Pager::open falls back to the still-valid prior slot), pager crash-before-commit, B+tree CRUD/split/overflow/negative-key/reopen, buffer-pool eviction-flushes-dirty and corrupted-page-surfaces-on-read.
  • Every public fallible constructor and every error variant used in Phase 01 has a test that produces it.

Scope decisions worth flagging

  • Row payload is lexis::Value, not a typed/schema-checked row. Per-column type checking against a TableDef is Phase 4/5 territory once CREATE TABLE exists; this crate stores a self-describing value tuple.
  • Single implicit B+tree per file, matching Decision 1's "one file per database" literally. A named-table catalog is Phase 04's CREATE TABLE layered on top of this same engine.
  • Synchronous, not async. Decision 12 locks async-first for the eventual pager surface, but that's phrased around WAL/tokio machinery that doesn't exist yet (Phase 02+), the scaffolded pinax crate carried no tokio dependency, and no Phase 01 acceptance criterion mentions async. Flagging as a spec gap rather than silently deciding — Phase 02 is the natural point to revisit this once the WAL trait lands.
  • .kanon-ci.toml updated to the real Rust gate (fmt/check/clippy/nextest + kanon lint), replacing the docs-phase stage list — that file's own comment named this as the point to do it, since code has now landed.

Verification

metis is under heavy concurrent load from other sessions (1-min load 20-35 against an 8-thread box, vgate --status showing waiters queued 30-50+ minutes) for the duration of this work, so vgate never admitted a local cargo check/clippy/nextest run despite being queued for the whole session. cargo fmt --all (not gated) ran clean, and cargo generate-lockfile / cargo tree / cargo metadata (also not gated) confirm the dependency graph and every manifest resolve correctly with no version conflicts. The code has had an extensive manual review pass — several real issues (an as cast, a handful of PermanentError/FatalErrorPinaxError conversion sites needing #[snafu(transparent)], a meta-slot encoding bug that zeroed two magic bytes, an inaccurate doc claim about root-collapse actually being live) were caught and fixed that way — but a real cargo check/clippy/nextest run has not been observed to pass locally or in CI as of this PR opening.

CI will not run on this PR yet, separately from the load issue above: gate-attestation.yml triggers only on: pull_request: branches: [main], and this PR's base is #10's branch, not main (per this task's own instruction to build on the scaffold rather than race it). No check will appear here until either #10 merges (this PR's base then follows main) or this PR is retargeted. See blocked_on in the task handoff for the full Gate-Passed trailer status.

Base automatically changed from feat/2-phase01-workspace-lexis to main August 16, 2026 01:22
forkwright added 2 commits August 15, 2026 20:24
Lands ROADMAP.md Phase 01 ("pager + buffer pool + B-tree") on top of the
workspace + lexis scaffold: a checksummed page format, a copy-on-write
pager, an LRU-evicting buffer pool, and an integer-keyed B+tree, exposed
as pinax::Database.

Page format (Decision 2): PageSize validates the five locked sizes
(4096-65536), each page carries a trailing XxHash3-64 checksum
(xxhash-rust, pure Rust), max_local = usable_space - 35 per the copied
SQLite formula. Rows over max_local spill into an overflow page chain.

Pager (Decision 1): the meta/header lives in two checksummed 4 KiB slots
(page ids 0 and 1) at fixed file offsets independent of the configured
page size, so bootstrap never depends on a value it has not read yet.
Every mutation targets a freshly allocated page id (no page reachable
from the active meta is ever overwritten in place); commit fsyncs data,
writes the other meta slot with an incremented txn_id, fsyncs again, then
flips which slot is active. A crash at any point leaves the previously
committed slot intact, which is what makes crash safety possible without
a WAL, a Phase 02 deliverable.

Buffer pool: capacity-bounded LRU cache over the pager. Because every
write is copy-on-write, get only ever needs to clone a page out (never
checked out, never invalidated by an in-progress edit), so eviction has
no aliasing hazard and needs no pinning.

B+tree: slotted leaf/interior pages, path-copying insert/delete (root to
leaf gets a fresh id chain, propagated bottom-up through
apply_result_to_interior), rows encoded as lexis::Value tuples per
lexis's own stated exit criterion. Delete does not merge or rebalance
underflowing nodes, which is out of Phase 01's acceptance criteria and
tracked as follow-up rather than silently dropped.

Wires the real Rust gate into .kanon-ci.toml (fmt/check/clippy/nextest +
kanon lint), per that file's own comment naming this as the point to do
it, and updates README/CRATE-SHAPE status text to match.

Part of #10
@forkwright
forkwright force-pushed the feat/phase01-pager-buffer-pool-btree branch from 6d30b76 to 16dbc64 Compare August 16, 2026 01:25
forkwright and others added 5 commits August 15, 2026 21:09
…g phase01

snafu generated context selectors (`BufferBoundsSnafu` et al.) are generic
over `Into<FieldType>`, so a bare integer literal passed to a `usize` field
(`at`, `len`, `buf_len`) defaults to `i32` and fails with `usize: From<i32>
is not satisfied` -- `i32` has no infallible conversion to `usize`. Suffix
the literal (`0usize`, `1usize`) rather than loosen the field type; the
`usize` is correct, the call site was wrong. Thirteen sites across btree.rs.

Also, in the order the compiler (and then clippy, once it could run)
surfaced them:

- `page.rs`: `u64::from(u32)` is not yet usable in a `const` initializer on
  this toolchain (rustc 1.94 through 1.97, tracked upstream as rustc issue
  143874) despite the widening conversion being lossless. Restate
  `META_SLOT_LEN` as a literal, matching the same file existing
  `CHECKSUM_LEN_U32`/`CHECKSUM_LEN_U16` precedent for the same problem.
- `pager.rs`, `buffer_pool.rs`: `Pager`/`BufferPool` need `Debug` -- every
  `.expect_err()` test on a `Result<Pager, _>` or `Result<BufferPool, _>`
  requires it. `write_data_page`/`sync_data` returned `Result<(), FatalError>`
  from a `Result<(), PinaxError>` fn; add the missing `?` + `Ok(())` so the
  `#[snafu(transparent)]` conversion actually fires.
- `row.rs`: `lexis::Value` is `#[non_exhaustive]`, so the `encode_value`
  match needs a wildcard even though every current variant is handled.
  There is no way to construct an unlisted `Value` variant from outside
  `lexis`, so a typed error arm would be permanently untestable;
  `unreachable!()` with an INVARIANT comment matches the basanos-sanctioned
  `RUST/unreachable-in-match` escape for exactly this shape.
- `codec.rs`: `write_i64` had zero production callers -- every `i64` write
  in this crate appends to a growing `Vec` (row/cell encoding), never writes
  at a fixed offset into an existing buffer -- so it was flagged `dead_code`
  once the abort clearing let rustc see past it. Deleted, with the test
  rewritten to build the fixture the same way production code does.
- clippy (`--all-targets -- -D warnings`, matching the exact `kanon gate`
  invocation): `NodeResult` derives `Copy` (`needless_pass_by_value`, all
  fields are trivially-copyable primitives); `collapse_root_if_needed` drops
  a redundant `Ok`/`?` (`needless_question_mark`); the two identical
  `(0u8, a)` arms in `Pager::open` merge (`match_same_arms`, verified
  benign -- both really do mean "slot A wins", never a masked bug);
  `compute_checksum` drops the always-`Ok` `Result` wrapper
  (`unnecessary_wraps`), with the `verify_checksum` doc comment corrected
  to match; `Database::insert`/`update` take `&Row` instead of an
  unconsumed owned `Row` (`needless_pass_by_value`), with every call site
  updated.
- `tests/phase01_acceptance.rs`: this file is its own crate root (every
  file under `tests/` is), so it does not inherit the `lib.rs`
  `#![cfg_attr(test, allow(clippy::unwrap_used, clippy::expect_used))]`
  escape -- the workspace sets `expect_used` to "warn" specifically because
  "tests legitimately use these" (`Cargo.toml`), but the gate `-D warnings`
  promotes it to an error anyway for this one file. A crate-level
  `#![expect(clippy::expect_used, reason = "...")]` restates that intent at
  the scope `-D warnings` cannot see through.

Gate-Passed: kanon 0.12.0
…ta region

Two real bugs, both invisible before the previous commit because the crate
never compiled: the never-verified 3510-line PR shipped both.

`write_overflow_chain` chunked a large row payload backward from the end of
`tail`, then linked pages by prepending -- so the possibly-short remainder
chunk landed FIRST in the chain (at `overflow_first`) instead of last.
`read_overflow_chain` reads `remaining_needed.min(chunk_cap)` bytes per page
and relies on every page but the last holding a full `chunk_cap` bytes; with
the remainder first, that assumption breaks on the first AND last page of
every chain over one page long -- the first read over-reads into a
zero-padded page, and the last read stops short of a full chunk, corrupting
and truncating any row that spills to overflow. Fixed by chunking forward
(remainder naturally last) and linking the resulting chunks in reverse,
which produces the same head-id-as-return-value shape the function already
had, just over chunks in the order the reader expects. Reproduced directly:
`btree::tests::large_value_spills_to_overflow_and_round_trips` failed with
embedded NUL bytes and a truncated string before this fix, and passes after.

`Pager::create` wrote only meta slot A (`META_SLOT_LEN` bytes) and never
extended the file to `META_REGION_LEN` (both slots). `Pager::open`
unconditionally reads both slots and requires `actual_len >= META_REGION_LEN`
before it will read either, so every freshly created database failed to
reopen with `FileTooSmall` -- caught only because `pager::tests::
create_then_open_round_trips_empty_state` and
`corrupted_active_meta_slot_falls_back_to_prior_slot` finally got a chance
to run once the overflow-chain fix stopped an earlier failure from
cancelling the rest of the suite. Fixed with `set_len(META_REGION_LEN)`
after writing slot A: extends (never truncates, since the file is currently
shorter), leaves slot B a zero-filled hole that legitimately fails its own
checksum, and falls back to slot A through the exact path a corrupted slot B
already takes.

Also fixed the two test bugs that same cancellation had been hiding:

- `corrupted_active_meta_slot_falls_back_to_prior_slot` opened the file with
  `.write(true)` only, then tried to `read_exact_at` it -- a write-only file
  descriptor cannot be read, and the test failed with "Bad file descriptor"
  rather than testing anything. Added the missing `.read(true)`.
- `eviction_flushes_dirty_pages_to_disk` read every one of 5 inserted pages
  back through the pager directly, bypassing the cache, and asserted all 5
  were durable. With capacity 2, only the first 3 (insertion order, `put_new`
  never reorders recency) are ever evicted and flushed; the last 2 stay
  resident, dirty, and correctly NOT yet written -- the test was asserting a
  property the buffer pool design does not make, and failed with an
  out-of-bounds file read on the two still-resident ids. Narrowed the
  read-back loop to the ids that were actually evicted.

Gate-Passed: kanon 0.12.0
…length

The gate never reached these either: `cargo check` failing 42 ways meant
`dependency audit` and `kanon lint` never ran on this PR before now.

`cargo deny check licenses` rejected `xxhash-rust` (BSL-1.0, the checksum
algorithm PLAN.md Decision 2 names -- a load-bearing dependency, not one to
drop). BSL-1.0 is OSI-approved and FSF Free/Libre, permissive in the same
class as the other allowed licenses; the deny.toml allow list was templated
from heurema and never picked this one up when xxhash-rust was added.
Added the entry.

`kanon lint` (full tier, `-D warnings`-equivalent for this run) failed on
eight warnings:

- `RUST/file-too-long`: `btree.rs` was 1121 lines against an 800-line
  limit -- Phase 01 pushed it well past the ceiling this same PRs own
  errors had been hiding. Split along the section boundaries the file
  already documented internally: `btree/layout.rs` (slotted-page
  primitives, leaf pages, interior pages), `btree/overflow.rs`
  (overflow-chain read/write, the row spill/reassemble boundary),
  `btree/mutate.rs` (path-copying propagation, plus the two
  `collapse_root_if_needed` tests that exercise it directly), `btree/mod.rs`
  (the crate-facing `insert`/`get`/`update`/`delete`/`scan` and the shared
  page-layout constants). Every test moved with the code it tests, unchanged
  -- same assertions, same coverage, four files instead of one, largest now
  537 lines. Verified byte-for-byte against the pre-split content before
  writing each new file; `cargo check`/clippy/nextest all passed clean on
  the first attempt after the split.
- `COMMENTS/deflection-without-tracking` x2: the module doc claimed
  "merge-on-underflow is tracked as deliberate follow-up scope" without a
  tracking artifact to back it -- filed #13 and cited it.
  The acceptance test flagged on the unrelated, purely grammatical use of
  "a `Database` going out of scope" (Rust variable lifetime, not project
  scope); reworded to avoid the phrase rather than add a tracking
  reference for something that needs none.
- `RUST/import-order` x2 (`btree.rs`, `database.rs`): both test modules had
  `use lexis::Value;` (external) after `use super::*;` (crate-relative) in
  the same `use` block. Reordered; the moved `btree.rs` test module carries
  the fix forward into `btree/mod.rs`.
- `STORAGE/no-migration-checksum` x2 (`database.rs`, `row.rs`): the same
  bare-substring false positive already tracked and suppressed for
  `lexis/ast.rs` and `lexis/lib.rs` in `.kanon-lint-ignore` (kanon#2975) --
  both files only *name* "CREATE TABLE" in a module doc citing a future
  phase, neither runs a migration or executes DDL. Added matching entries.
- `WRITING/em-dash` (`README.md`): one em dash in prose, replaced with
  " - " per the house style the rule enforces.

Also corrected a drift in `.kanon-lint-ignore` itself while adding the new
entries: the `TESTING/no-tests:crates/pinax/src/lib.rs` line was filed
under "genuinely empty scaffolds" alongside hypomnema/phylaxis, but Phase
01 landed real behavior in every other pinax module before this fix --
lib.rs itself is still just declarations, same shape as the
already-correctly-reasoned `lexis/lib.rs` entry, so it moved out of the
stale "empty by design" heading into its own entry with the accurate
reason.

Gate-Passed: kanon 0.12.0
Database::insert takes &Row; the front-page example passed it by value, so
the crate's own usage example did not compile.

Nothing caught it because the local gate runs cargo nextest, which does not
execute doctests at all — only CI's cargo test does. A Gate-Passed trailer
minted from a nextest gate therefore says nothing about doctests.
…btree

Gate-Passed: kanon 0.12.0 +stages:fmt,check,clippy,nextest,lint sha:f6ef2f1a18ba770f9b0121dc3f6e9754bf55d110
@forkwright
forkwright merged commit 8142a84 into main Aug 16, 2026
9 checks passed
@forkwright
forkwright deleted the feat/phase01-pager-buffer-pool-btree branch August 16, 2026 03:04
forkwright pushed a commit that referenced this pull request Aug 16, 2026
🤖 I have created a release *beep* *boop*
---


## [0.0.3](v0.0.2...v0.0.3)
(2026-08-16)


### Features

* **pinax:** implement Phase 01 pager, buffer pool, and B+tree
([#11](#11))
([8142a84](8142a84))
* **workspace:** stand up the pinax workspace and implement lexis
([#10](#10))
([234be1a](234be1a))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant