Skip to content

Gap stage 1: append-only blocks for inode values and binding hints, schema v2 (#91) - #595

Merged
yifanxuaaa merged 7 commits into
layerfs_v2_0_0from
opt/91-gap-blocks
Nov 6, 2026
Merged

yifanxuaaa merged 7 commits into
layerfs_v2_0_0from
opt/91-gap-blocks

Conversation

@yifanxuaaa

Copy link
Copy Markdown
Contributor

inode_value and binding_hint now store many entries per row: append-only ≤4,000 B blocks, stored PLAIN with a CRC32C, the same block format on SQLite and PostgreSQL. This is schema v2. On fs-storage-quick, Q1 is −5.5% (SQLite) / −13.7% (PostgreSQL) and Q6 is −3.1% / −8.4%. D10 (diagnostic) goes from 52.9 to 49.8 MB on SQLite and from 70.4 to 61.0 MB on PostgreSQL.

Gap stage 1: fixes Ephemeral-AI-Lab/layerfs-project#91 (parent Ephemeral-AI-Lab/layerfs-project#84). Base: layerfs_v2_0_0 @ 8b701f42f0. It must merge together with the stacked #90 PR (schema v2 on PostgreSQL). Merging #91 alone would force #90 to a v3.

Commits

  • b973426ee0 Blocks:
    • inode-value blocks with implicit ordinals, a per-block mode table, varint/zig-zag fields, and an exact 73-byte canonical round trip, so no object id changes;
    • binding-hint blocks;
    • schema v2 on both backends (v1 is refused);
    • tools/store_blocks.py for the optional offline SQLite migration and DIAGNOSTIC attribution.
    • LOC 139405 → 139919 (+514).
  • 54486095d6 File digests: file_digest is kept only for chunked files. Small files are whole-file objects found by id at wave admission. +38.
  • 017507ccf2 Save value memo: now keyed on the whole 73-byte value. This fixes a pre-existing bug: values differing only in mtime low bytes or nanoseconds got duplicate ordinals in one Save (D10: 953 extra rows). It wasted space but never returned wrong values. +3.

Codec frozen: field packing only. There is no zstd, LZ, dictionary or TOAST compression on these rows, and bodies are STORAGE PLAIN.

Measured and not done:

  • Cross-Save ordinal reuse: needs a value index costing about what it saves.
  • Small-pack coalescing: at most 0.8%, and needs a pack-grammar change.

Multi-writer contract (#84 gap design R1–R9)

Rule How it holds
R1 Blocks are written inside the publication statement, so they commit with their packs (postgresql/objects/publish.rs:46–79). Hints are append-only.
R2 One statement per inode_values call (metadata/reads.rs:100–122); one json_each statement on SQLite.
R3 Location path unchanged.
R4 Lost-race and Collision unchanged. An overlapping block returns NULL, which becomes Integrity.
R5 Blocks are encoded before the lane (publish.rs:117); still one statement.
R6 INSERT … WHERE NOT EXISTS only, never UPDATE. Hint seq+1 under the lane.
R7 No compaction.
R8 Ordinals and groups never enter hashed ids.
R9 Same statement count; hint writes use fewer statements.

Tests

222 test executables pass on the stacked head (persistence incl. PostgreSQL, storage, daemon, store-service, workspace, project). clippy -D warnings, fmt, the boundary guard and tools/test_store_blocks.py all pass.

fs-storage-quick (qgap-c2b, against the Phase 2 baseline 02133596ae; 12/12 PASS, no regressions)

Case SQLite net PostgreSQL net
Q1 4,796,416 → 4,534,272 (−5.5%) 6,119,424 → 5,283,840 (−13.7%)
Q3 same 942,080 → 753,664 (−20.0%)
Q4 same 1,343,488 → 1,155,072 (−14.0%)
Q6 24,399,872 → 23,650,304 (−3.1%) 28,999,680 → 26,550,272 (−8.4%)

On D10 PostgreSQL, inode_value goes 9.39 → 2.84 MB and binding_hint 3.26 → 0.43 MB.

Speed gate (fs-bench-quick, Docker, interleaved control/candidate/candidate/control, quiet host)

Migration

  • SQLite v1 Stores: refused at open with Refused, naming tools/store_blocks.py migrate. The migration is optional and offline: it writes a checked v2 copy from a closed v1 copy.
  • PostgreSQL v1 schemas: refused, with no conversion tool. Ship it for new Stores.

🤖 Generated with Claude Code

yifanxuaaa and others added 6 commits November 6, 2026 18:11
… schema v2)

Gap stage 1 of #84. inode_value and binding_hint kept one database row per
entry; PostgreSQL pays ~28 B of tuple header and line pointer plus a B-tree
entry for each. Schema version 2 keeps them in immutable, append-only blocks
of at most 4,000 bytes (one per 4 KiB SQLite page, two per 8 KiB PostgreSQL
page, bytea STORAGE PLAIN):

- inode_value_block(first_ordinal, last_ordinal, body): consecutive ordinals,
  implicit inside the block; per entry the value's byte 0 as tag, LEB128
  reference count, content root, and either the attribute root or a
  per-block mode-table index, zig-zag mtime delta and nanoseconds; any other
  73 bytes are stored whole. Decoding rebuilds the exact canonical bytes.
- binding_hint_block(grp, seq, body): one write's hints of the 64 serials of
  group child >> 6, child/parent deltas and shared name prefixes; readers
  merge a group's blocks and drop repeats.
- Every body ends with a CRC32C checked on every decode.

Field packing only, no compression (codec frozen). Blocks are encoded before
the statement that holds the lane; the publication stays one statement on
PostgreSQL. An insert whose ordinals a stored block holds inserts nothing
and fails the publication (a stored value is never replaced). Reads stay one
statement: per run of ordinals, the last block starting at or before it and
the blocks to its end.

Schema version 2: PostgreSQL SCHEMA_VERSION 2; SQLite user_version 36-38
(42-44 with acquisition tables). Version 1 Stores (SQLite 16-18/22-24,
PostgreSQL 1) are refused at open. tools/store_blocks.py adds the diagnostic
decoder (attribution from a closed SQLite copy or exported PostgreSQL rows)
and an offline SQLite v1 -> v2 migration of a closed copy; the
merge-forms-root.sqlite fixture was converted with it. Golden block vectors
are asserted by the engine tests and the tool's tests.

Tests: new binding_hint_blocks.rs; sqlite_inode_values.rs reworked for
blocks (exact round trip of every value shape, block bounds, checksum,
golden vector); versions updated in sqlite_group_rows*.rs,
postgresql_open_check.rs (a version 1 schema is refused),
sqlite_ordinal_lookahead.rs and old_format_open.rs. No test deleted.

Production LOC: 139405 -> 139919 (delta +514) (tools/production_loc.py combined
scope: core/engine/connect/api src + runtime SQL; parent tree vs staged tree)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A file below the Store's small_file_threshold_bytes is one whole-file
object: building it again yields the same object id, which wave admission
finds stored and does not encode again. Its file_digest row cost more than
it saved (D10: 15,524 of 15,535 rows were small). record_file_digests now
drops rows below the threshold and file_roots answers none for a smaller
length, with no statement when a window asks only small lengths. The
threshold comes from store_policy at open (both backends).

Tests: daemon file_digests asserts the floor (a small file is offered,
not kept; relink reuses only the chunked files); postgresql_identical_ids
checks the floor on SQLite and PostgreSQL.

Production LOC: 139919 -> 139957 (delta +38) (tools/production_loc.py
combined, parent vs commit tree, rebased on 8b701f4)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The memo filter mixed only bytes 0..8, 9..17 and 41..49 of a value, so two
values that differ in mtime low bytes or nanoseconds shared one entry: the
first hid the second, and each later copy of the second in the same Save
took a new ordinal (D10: 953 of 16,801 repeated values). The filter now
mixes all 73 bytes; candidates are still checked byte for byte.

Test: stored_value_reads one_save_stores_values_that_differ_only_in_mtime_once_each
(fails on the old filter).

Production LOC: 139957 -> 139960 (delta +3) (tools/production_loc.py
combined, parent vs commit tree, rebased on 8b701f4)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…x index (#90)

object_location becomes pack_id bigint, canonical_length integer, role,
group_number and record_number smallint, then object_id bytea PLAIN, with
no unique key on the id: a B-tree object_location_prefix on the id's first
eight bytes (signed big-endian bigint) finds candidates and the full id is
compared in the heap. A location is inserted only when its id is absent
(INSERT ... WHERE NOT EXISTS) inside the one publication statement that
already holds the store_lane row, so exactly one location per id is chosen
under the lane; the lost-race list and the core's Collision read-back are
unchanged. The content_signature foreign key needed a unique key: the same
statement now refuses a signature that names neither a location of its
publication nor a stored one. Locate and digest_roots join through the
prefix (one statement each, as before).

Bounded bytea/text columns of the small tables (and the acquisition
tables) are STORAGE PLAIN, so those tables get no TOAST relation.

Measured and not taken: explicit 8,100 B PLAIN pack chunk rows (D10 packs
44.0 MB against 43.2 MB with EXTERNAL TOAST: one chunk per page plus a
partly filled tail page per pack); dropping the composite UNIQUEs (they
back the same-stack composite foreign keys); file_digest narrowing (the
#91 digest floor already removes almost all its rows).

Tests: postgresql_narrow_locations (prefix-sharing ids, lost race,
repeated id refused, signature check, TOAST set, prefix index used);
postgresql_open_check refuses a widened location column on the full check.

Production LOC: 139960 -> 139999 (delta +39) (tools/production_loc.py
combined, parent vs staged tree)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on named (#91)

A SQLite Store of user_version 16 to 18 or 22 to 24 (one row per inode
value and binding hint) was refused at open as Malformed. It is now
Refused with a status that says what it is and names the optional offline
conversion (tools/store_blocks.py migrate on a closed copy). Other unknown
versions stay Malformed. The README states that conversion is optional and
that a PostgreSQL schema version 1 Store has no conversion tool.

Test: sqlite_inode_values refuses versions 16-18 and 22-24 with that
status, writable and read-only.

Production LOC: 139999 -> 140008 (delta +9) (tools/production_loc.py
combined, parent vs staged tree)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…index (#90)

With no unique key on object_id, the planner may answer a large batch
(thousands of ids against a table of tens of thousands of rows) with a
hash join or hash anti join that scans the whole table and computes the
prefix expression for every row. DIAGNOSTIC on a copy of D10's 51,742
locations, 4,000 stored ids: the insert-if-absent took 22.7 ms that way,
against 4.9 ms as one correlated index probe per id and 8.5 ms for the
schema version 1 ON CONFLICT on the 32-byte primary key. OFFSET 0 in the
location subqueries (publication insert and signature check, locate,
digest_roots) keeps each one a probe per id through object_location_prefix:
3 buffers per id.

Production LOC: 140008 -> 140008 (delta +0) (tools/production_loc.py
combined, parent vs staged tree)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yhcedpn yhcedpn added this to the LayerFS v2.0.0 milestone Nov 6, 2026
Gap stage 2: narrow PostgreSQL object_location under an 8-byte-prefix index (layerfs-project #90), stacked on #91; lands with #595 (schema v2).

Validated together in the #84 final integration 7691ac2 (layerfs-benchmark#11) and with round 2 in the quick verification (layerfs-benchmark#10). Backup checks B1-B4 PASS (schema v1 backups refused at open naming tools/store_blocks.py migrate).

Production LOC: 139960 -> 140008 (delta +48)
Scope: combined first-party production source, tools/production_loc.py --json on git archive trees, first parent 017507c (opt/91-gap-blocks). Reference absent (retired R9c; last counted 65,417 at tree 498dd19).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yifanxuaaa
yifanxuaaa merged commit e20c6a2 into layerfs_v2_0_0 Nov 6, 2026
2 of 5 checks passed
yifanxuaaa added a commit that referenced this pull request Nov 6, 2026
Brings layerfs-project #84's storage changes (#92 directory/extent page
deltas, pack framing v26; #91 inode-value blocks and #90 narrow PostgreSQL
rows, schema v2) under round 2. Conflict only in
core/crates/layerfs-storage/tests/stored_value_reads.rs: both sides
appended a test; both kept. Tree 87f25ed, as verified in the quick
verification (layerfs-benchmark#10).

Production LOC: 140178 -> 141111 (delta +933)
Scope: combined first-party production source, tools/production_loc.py
--json on git archive / checkout-index trees, first parent 048d604.
Second parent e20c6a2 (140338). Reference absent (retired R9c; last
counted 65,417 at tree 498dd19).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
yifanxuaaa added a commit that referenced this pull request Nov 6, 2026
PostgreSQL round 2 (layerfs-project #61: #94-#98, #64, #66): fewer connections and round trips in create, mount, Commit and reads, now on top of #84's storage changes (#594, #595/#596).

Quick verification on this exact tree dd191ba (layerfs-benchmark#10): backup B1-B4 51/51; fs-bench-quick q84-2 + q84-2x 36/36 PASS with PostgreSQL counts equal to the round-2 control except C16 Commit 251 -> 224 statements, no aggregate or per-row timing flag; fs-storage-quick Q1/Q5/Q6 x SQLite/PostgreSQL 6/6 PASS, every metric 'same' as the #84 final. Clippy -D warnings and cargo fmt --check clean.

Production LOC: 140338 -> 141118 (delta +780)
Scope: combined first-party production source, tools/production_loc.py --json on git archive trees, first parent e20c6a2. +773 round 2 (#94-#98, #64) and +7 from a rustfmt-only commit on #92 files. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yifanxuaaa yifanxuaaa added the new feature A new user-facing capability label Nov 13, 2026
@yifanxuaaa
yifanxuaaa deleted the opt/91-gap-blocks branch November 18, 2026 08:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

new feature A new user-facing capability

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants