Skip to content

feat(persistence): export a Store's changed rows as bundles for replicas (2.0 port of #495) - #669

Open
rexjz wants to merge 2 commits into
Ephemeral-AI-Lab:layerfs_v2_0_0from
rexjz:feat/incremental-store-replication-2.0
Open

rexjz wants to merge 2 commits into
Ephemeral-AI-Lab:layerfs_v2_0_0from
rexjz:feat/incremental-store-replication-2.0

Conversation

@rexjz

@rexjz rexjz commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

This is #495 ported to layerfs_v2_0_0 (03e4c541a), as an alternative to #630. #630 restores #495's original two commits. Those were written against main before schema version 2, so they conflict with 2.0 in four files, and their triggers name inode_value and binding_hint, which 2.0 no longer has. This PR has the same two commits and the same API, with the changes 2.0 needs (listed below). If you would rather keep #630, its branch can take these two commits as they are.

What

A new module, layerfs_persistence::replication, keeps a second copy of a Store up to date by moving only the rows that changed:

  • mark_base(store, token) makes an offline Store the start of an export chain. A copy of the closed file is then a replica at the returned cursor.
  • export_changes(store, since, bundle) writes every row changed after since into a new SQLite bundle file, from one read snapshot of the live Store. It takes no write lock and stops no writer.
  • apply_changes(replica, bundle) installs bundles in order into the offline replica, one transaction each. The replica then holds exactly the live Store's rows and opens through the normal path.

A bundle is a local file. Nothing here moves it anywhere: the caller decides whether it goes to another disk, another machine or object storage. The design is the same as in #495 (chained states, BLAKE3 digest, refusal before any install, publish by hard link, checkpoint report). See that PR for the details. Its open questions still apply.

What changed for 2.0

  1. Schema version 2 blocks (v0.1.4 phase 2: benchmark against v0.1.3 and repair severe regressions #91). The triggers and the replicated table list now log inode_value_block (key first_ordinal) and binding_hint_block (keys grp, seq) instead of inode_value and binding_hint. There are still 18 logged tables, plus pack_unit in the group-row layouts.
  2. The Doctor's type check. schema_diagnostic::type_checks compares each column's stored values with the column's declared type. change_log's key columns are declared ANY, which a STRICT table allows. ANY names no storage class, so every Store failed store.database.types, and about 20 diagnostic tests failed. The check now skips ANY columns (3 lines). The round-trip test also runs the Doctor on the live Store and on the replica, and expects no failure.
  3. schema.rs. 2.0 made scripts() visible to the diagnostic module. This PR keeps that, and adds the change log script for each layout as before. The open-time migration add_change_log is unchanged. A Store written by 2.0 before this PR gains the two tables at its first writable open, as with the lineage, merge and digest tables.
  4. Docs. docs/architecture/engine/store/sqlite.md describes the two tables and has a short "Change export" section. Doc 84 keeps its paragraph.
  5. The tests use SqliteHandles and the block tables.

The second commit (apply no longer logs the rows it installs) is unchanged. On 2.0, without that commit, the free-page test still finds 22 free pages.

Not changed

  • No change to object formats, the pack layout, history semantics, the publication lane, the PostgreSQL backend, or any layerfs-api, SDK or CLI surface. Nothing outside the module's tests calls it yet.
  • Cargo.lock is unchanged: blake3 =1.8.5 moves from the dev-dependencies to the dependencies of layerfs-persistence.

Tests

tests/change_export.rs has 9 tests:

  • the replica round trip, now with the Doctor on both sides
  • every pack layout
  • eight changed bundles, each refused, leaving the replica unchanged
  • a written replica and a changed bundle
  • a reader blocking the checkpoint
  • a locked replica
  • a replica outside the Disposable profile
  • an unmarked Store
  • the free-page test

lineage_backfill.rs, old_format_open.rs and sqlite_publication.rs are updated for the two new tables.

Checked on trees without each 2.0 change: without the ANY skip, about 20 tests fail in diagnostic_inventory, diagnostic_schema, diagnostic_snapshot and store_diagnostic. Without the second commit, the free-page test finds 22 free pages.

The loop again, on 2.0

I reran the two-machine loop from #495 with the 2.0 CLI built from this branch. Machine A writes through layerfs and exports one bundle per round. The bundle moves through a shared folder, and machine B applies it. B was restored from A's MinIO backup at r0. After each round, all 23 Store tables on A and B are compared row by row, and the same CLI reads run on both machines.

Round Writes on A (CLI) Bundle Rows / deletes Tables equal Reads equal
r1 one file apply: write, mkdir, mv, cp, rm, edit 41,984 B 31 / 0 23/23 12/12
r2 30 single-file Commits, publish 94,208 B 383 / 0 23/23 16/16
r3 500 + 200 files on two Branches, three-way merge 205,824 B 1,446 / 0 23/23 24/24
r4 64 MiB text + 16 MiB random 29,955,072 B 4,409 / 0 23/23 29/29
r5 second Project shop/web: init + write 157,696 B 290 / 0 23/23 29/31 (gap)
r6 delete a file and a folder, project remove shop/web 33,792 B 24 / 11 23/23 35/35
r7 4 parallel writers × 15 Commits 145,408 B 787 / 0 23/23 43/43
r9 one Commit + publish (r8 is the same, exported before r9) 33,792 B 23 / 0 23/23 49/49
  • The 9 bundles total 30.7 MB, of which r4 is 30.0 MB. A whole-file push after each round would move about 1.0 GB (the Store was 91–122 MB). Export took 2.4–58.5 ms and apply took 2.5–79.3 ms per round.
  • Bundles carry fewer rows than on 0b8ae8af5: r1 went from 137 to 31 rows, r3 from 3,735 to 1,446, and r5 from 3,260 to 290. As far as I can tell, this is because inode values and hints now move as blocks.
  • A skipped bundle and a replayed bundle are refused (Cursor). A replica with its own Commit is refused (Diverged). In all three cases, the replica's 23 table digests did not change.
  • Every write on A succeeded, including all 60 parallel writes in r7.
  • The gap from feat(persistence): export a Store's changed rows as bundles for replicas #495 is unchanged. In r5, the bundle carries the new Project's rows, but B's CLI answers NoSuchProject. 2.0 keeps Project records in the home, and apply_changes changes only the Store file.
Loop setup
  • Linux aarch64 Docker containers (Debian bookworm) running release builds with Rust 1.85.1. A small driver binary, not part of this PR, calls mark_base, export_changes, apply_changes and read_state.
  • A and B each have their own LAYERFS_HOME on ext4. At r0, A ran backup create and B ran backup restore, through MinIO (the pinned image from minio_service.sh). The snapshot carries the change state, so B starts at A's cursor.
  • The Project is a git archive of 0b8ae8af5: 5,631 files, 132 MB.
  • After each round: a digest of every row in all 23 tables on A and on B, ordered by primary key; the same CLI reads on both machines (file read/stat/list/glob/grep, branch list/get, commit list), compared byte by byte; and B's read_state, checked against the after cursor of A's export.
Checks run

CI is off for pull requests into layerfs_v2_0_0, so I ran these locally. Setup: Linux aarch64 (Debian bookworm container on a macOS host), Rust 1.85.1, cargo-nextest 0.9.146. I used the commands from engine/AGENTS.md "Checks and completion", ci.yml and nightly.yml, with every test command under a timeout. I ran the same commands on this branch and on 03e4c541a.

Check This branch 03e4c541a
cargo fmt --all -- --check fails in 2 files this PR does not touch same
check_product_boundary.py, with and without --skip-sdk 2 findings in a file this PR does not touch same
check_fuser_integrity.py, tools unit tests pass
clippy --locked --workspace --exclude layerfs-node --all-targets -D warnings pass
clippy -p layerfs-persistence --features postgresql --all-targets -D warnings pass
cargo test --locked --all-targets --no-run pass
fast lane (ci_tests.py targets/filter fast) 1901 run, 1901 passed, 26 skipped 1892 run, 1892 passed, 26 skipped
nightly lane (ci_tests.py targets/filter nightly) 146 passed, 11 skipped
mount lane (nightly.yml: each binary in its own container with /dev/fuse) 45 binaries, 116 passed, 3 ignored same
every layerfs-persistence target 180 passed, 2 skipped
  • The fmt and boundary findings are in control/crates/layerfs-store/src/lease.rs and api/crates/layerfs-cli/tests/concurrent_reads.rs, from 14f62997b.
  • The fast lane ran as a non-root user with TMPDIR on ext4, as on the runner. As root, with TMPDIR on the container's overlay, both trees fail the same 59 tests: the kernel guard refuses overlayfs, and the daemon refuses uid 0.

Not run: x86_64, macOS, the Durable profile and layerfs-node.

Production LOC: 159967 -> 160998 (delta +1031): engine 74419 -> 75450, all in layerfs-persistence 17267 -> 18298; core, control and api unchanged. Per commit: +1030 and +1. Counted with tools/production_loc.py --json (blob 35e4a6d601) on git archive extracts of 03e4c541a and of each commit's tree.

@yifanxuaaa

Copy link
Copy Markdown
Contributor

Review of the 2.0 port, read against 00bdc77887 on 03e4c541a. This was a code reading only: I did not build the branch or run its tests. #630 still carries #495's original commits (they conflict with 2.0 and name inode_value and binding_hint), so this PR is the one to review, and #630 can be closed in its favour.

Findings, most serious first

1. A marked Store's change log grows without bound. After mark_base, every write to the 19 logged tables appends a change_log row, one per object in object_location. Nothing on the live Store ever removes them: the only reset is mark_base again, which breaks every existing replica (they then need a full copy). There is also no way to turn logging off, and a Store restored from a backup of a marked Store keeps logging. Neither architecture doc mentions this. It needs a trim up to the oldest cursor still in use, or an unmark call, or at least a documented limit.

2. No measurement of the cost on Commit. Every Store now carries 57 triggers, and each written row checks whether the Store is marked. When it is marked, the writer's transaction also inserts log rows. AGENTS §3 rule 9 asks for both speed and storage numbers. The PR times export and apply, but not Commit or Save with logging off and on.

3. Every Store's schema changes for an API nothing calls yet. The first writable open by this binary adds the two tables and the triggers, after which an older binary (for example one pinned in a VM image or a worker) may no longer open that Store. Nothing outside the module's tests calls the API, and the review on #495 asked to agree the order with #506 and #533 in the 2.0 roadmap first. Whether this lands in 2.0 now is a call for the maintainers.

4. Nothing stops mark_base or apply_changes on a Store a daemon is using. The docs say both are for offline Stores, but they open their own SQLite connection with no busy handler and do not take the Store's writer lock (backend/sqlite/lane/writer_admission.rs). Run against a Store a daemon is writing to, the daemon's Commit can fail with Busy, which that lock exists to prevent under the no-retry rule. They should take that lock, or refuse when they cannot get it.

5. A bundle does not check that both sides have the same schema. history_meta, which records the schema, is copied like any other row. Apply only checks that each bundled table exists, then inserts with SELECT *, matching columns by position. If a later additive migration runs on the live Store after it was marked, the next bundle overwrites the replica's recorded schema without bringing the new table, and the replica then fails validation at open. A schema hash in bundle_meta, compared before installing, would catch this.

6. The Project-records gap (already disclosed). 2.0 keeps Project records in the home, outside the Store file, so after a bundle that creates a Project the replica's CLI answers NoSuchProject. Replication works within existing Projects, but not for adding one.

Checked and correct

  • Store totals on the replica come out right. Commit and layer rows are only inserted or deleted, never updated, and any change to them also updates store_totals on the live Store, so the bundle always carries the totals and apply installs them last.
  • No foreign key uses ON DELETE CASCADE, so apply's delete-then-insert cannot remove other rows. Deferring foreign-key checks to commit is correct.
  • The refusals for a skipped, replayed or diverged bundle look right.
  • The second commit (pause logging while installing, then restore the marker) is correct.
  • Skipping ANY columns in the Doctor's type check is a small, reasonable fix.

mark_base makes an offline Store an export base: it adds a change log
that records the key of every row written to the replicated tables.
export_changes writes the rows changed after a cursor into a small SQLite
bundle, from one read snapshot of the live Store: it takes no lock and
stops no writer. apply_changes installs bundles in order into an offline
copy of the base. Each bundle carries a BLAKE3 digest of its rows and
deletions, the identity of its Store's schema and a chained state, so an
out-of-order, repeated, changed or diverged bundle, or one exported from a
Store with another schema, is refused before anything is installed.

- Only an export base has a change log. mark_base adds the change_state
  and change_log tables and AFTER INSERT/UPDATE/DELETE triggers on the 18
  replicated tables (plus pack_unit in the group-row layouts) and records
  the schema with them, so the Store still opens and passes the Doctor.
  unmark_base drops them and records the schema of a Store that was never
  marked. A Store that is never marked keeps the 2.0 schema unchanged: no
  migration, no trigger, nothing logged, and a build without this module
  opens it. Inode values and binding hints are logged as the schema
  version 2 blocks they are stored in (inode_value_block by first_ordinal,
  binding_hint_block by grp and seq).
- trim_changes moves the base forward to the cursor of the least advanced
  replica still followed and deletes the log up to it, so the log of a
  live base stays bounded. sqlite.md states that a base logs every write
  until it is trimmed or unmarked, and that a copy of a base, such as a
  backup or a Store restored from one, is a base too.
- mark_base, apply_changes, trim_changes and unmark_base take the Store's
  writer lane (writer_admission::Writer::try_acquire) in one non-blocking
  attempt and return ChangeError::Busy while another writer holds it. They
  then begin with BEGIN IMMEDIATE and no busy handler, so a connection
  outside the lane that holds SQLite's write lock also makes them Busy.
  export_changes reads one snapshot and takes no lane.
- bundle_meta.schema is a BLAKE3 hash of the source Store's user_version
  and every table, index and trigger definition, read in the export's
  snapshot. apply_changes compares it with the replica's and returns
  ChangeError::Schema on a mismatch, then compares each bundled table's
  definition with the replica's before it installs a row.
- The Doctor's storage-class check (schema_diagnostic::type_checks) skips
  a STRICT table's ANY columns, which declare no storage class. The change
  log's keys are ANY.
- The module (src/replication/) keeps marking, the log, export and apply
  in their own files. Store connections run the Disposable profile
  (profile::apply: WAL, synchronous OFF); a non-WAL Store is refused as
  ChangeError::Store with the profile's own error.
- An export writes the bundle under a temporary name and hard-links it
  into place, so the bundle path names a complete file or nothing, also
  after a process crash. Bundles use 1 KiB pages and carry only the tables
  with logged changes. apply_changes parses the bundle's meta row without
  trusting the bundle's own constraints (one row, value types and lengths,
  sequence order).
- mark_base and apply_changes checkpoint after they commit. A checkpoint
  that a reader blocks, or that fails, returns ChangeError::NotCheckpointed:
  the change committed, and the Store file alone does not hold it yet.
  trim_changes and unmark_base commit without a checkpoint.
- New tests/change_export.rs (12 tests): the replica round trip, which
  also runs the Doctor on the live Store and the replica; refused bundles;
  a replica with another schema; a replica locked outside the lane; every
  write refused while another writer holds the lane; trimming; unmarking,
  which restores the schema of a never-marked Store and passes the Doctor;
  a non-WAL replica; a reader blocking the checkpoint; all three pack
  layouts; an unmarked Store. docs/architecture/engine/store/sqlite.md
  describes the change export.

Production LOC: 171524 -> 172719 (delta +1195)
  engine 76947 -> 78142 (+1195): layerfs-persistence 17288 -> 18483; core
  39795, control 48436 and api 6346 unchanged. Counted with
  tools/production_loc.py --json (blob 35e4a6d) on git archive extracts
  of the parent (layerfs_v2_0_0 5742338, tree f65d18f) and the
  committed tree 0dcaefe.
apply_changes installed a bundle's rows while the replica's change_state
row was present, so every trigger logged them, and the apply then deleted
that log again. The deleted log left free pages in the replica file, and
every copy or backup of the replica carried them.

- install deletes change_state before it installs rows and inserts it
  again with the bundle's ending state, in the same transaction, so a
  failed apply leaves the replica as it was.
- New test: applying a 1500-object bundle leaves only the sequence marker
  in change_log and no free pages. Without the change it leaves 22 free
  pages.

Production LOC: 172719 -> 172720 (delta +1)
  engine 78142 -> 78143 (+1): layerfs-persistence 18483 -> 18484; core
  39795, control 48436 and api 6346 unchanged. Counted with
  tools/production_loc.py --json (blob 35e4a6d) on git archive extracts
  of the parent tree 0dcaefe and the committed tree e37a195.
@rexjz
rexjz force-pushed the feat/incremental-store-replication-2.0 branch from 00bdc77 to 59e9726 Compare November 17, 2026 08:04
@rexjz

rexjz commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the careful read. I reworked the port for findings 1, 3, 4 and 5 and rebased it on 57423382f (#727): 11e8d32cd (the feature) and 59e9726ab (the second commit, unchanged apart from its LOC line).

1. The change log grows without bound. New trim_changes(path, oldest) moves the base forward to the cursor of the least advanced replica still followed and deletes the log up to it. It checks oldest as an export checks its cursor, and afterwards an export from a trimmed-away cursor is refused with Cursor. New unmark_base(path) drops the log and its triggers. docs/architecture/engine/store/sqlite.md now states that a base logs every write until it is trimmed or unmarked, and that a copy of a base, such as a backup or a Store restored from one, is a base too.

3. Every Store's schema changes. The change log is no longer part of the shipped schema or an open-time migration: add_change_log is gone, and store/open.rs, old_format_open.rs, sqlite_publication.rs and lineage_backfill.rs are back to 2.0's. mark_base adds the two tables and the triggers to that one Store and records the schema with them (schema_source gains the change-log script and schema_definition is recomputed), so open validation and the Doctor accept a marked Store. unmark_base restores the never-marked schema. A Store that is never marked keeps the 2.0 schema and runs no trigger. Whether this goes into 2.0 is still your call.

4. No writer lock. mark_base, apply_changes, trim_changes and unmark_base take the writer lane with Writer::try_acquire (one attempt) and return the new ChangeError::Busy while another writer holds it. Then they run BEGIN IMMEDIATE with no busy handler, so a connection outside the lane that holds SQLite's write lock also gets Busy. The lane is released after the connection closes, so the checkpoint at close runs under it. export_changes still reads one snapshot and takes no lane.

5. No schema check. bundle_meta.schema is a BLAKE3 hash of the source Store's user_version and every sqlite_schema row, read in the export's snapshot. Apply compares it with the replica's and returns the new ChangeError::Schema before installing anything, then compares the CREATE text of each bundled table with the replica's.

2 and 6 are not addressed in this push. With the change for 3, Commit on an unmarked Store runs against the 2.0 schema with no trigger. The cost on a marked Store is still unmeasured. 6 is unchanged.

Evidence, on 59e9726ab:

  • macOS (aarch64), cargo +1.85.1 test --locked -p layerfs-persistence --all-targets: 260 passed, 2 ignored. tests/change_export.rs now has 13 tests. The new ones cover a replica with another schema, every write refused while another holder has the lane, trimming, and unmarking (the schema then equals a never-marked Store's, and the Doctor passes).
  • Linux container (uid 1000, TMPDIR on ext4), same commands on this branch and on 57423382f:
    • clippy --workspace --exclude layerfs-node --all-targets -D warnings: clean.
    • Persistence lane: 234/235 here, 222/222 on the base.
    • Fast lane: 2261/2262 here, 2247/2249 on the base.
    • The failure here is concurrent_first_installs::two_writers_saving_the_same_new_content_both_complete. It fails intermittently with Integrity("dependency cycle") on both: 3 of 200 runs here, 5 of 200 on 57423382f.
    • machine_files::the_table_lists_the_call_quiet_with_a_path_and_its_options fails on both, and cargo fmt --check reports the same four upstream files on both.
  • Two containers and MinIO, with release CLIs built from 57423382f and from this branch:
    • Unmarked Store. A Store written by the 57423382f CLI and then by this branch keeps an identical schema: same hash of user_version, every sqlite_schema row, schema_source and schema_definition. The 57423382f CLI still reads and writes it, and its Doctor passes all 57 checks.
    • Marked Store. After mark_base, this branch's Doctor passes. The 57423382f CLI refuses the Store (Store open: physical Store I/O: pack persistence: Malformed) and leaves every table unchanged.
    • Replication with live writes. Backup, restore, then two export/apply rounds. Next, 1500 CLI writes on the live Store overlapped 40 rounds of export, apply and trim_changes to the replica's cursor. No write failed. 7 trims returned Busy and succeeded on a later attempt, and 29,814 log rows were trimmed. Every table but change_log and change_state then holds the same rows on the replica as on the live Store, and both CLIs read the same bytes. After the trim, an export from an earlier cursor is refused.
    • Unmark. unmark_base on both Stores gives the identical never-marked schema. The 57423382f CLI then reads, writes and passes the Doctor again, and mark_base starts a new chain.

@rexjz

rexjz commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

Updated description for 59e9726ab. The PR description above is for 00bdc7788 on 03e4c541a. Since then I reworked the branch for the review and rebased it on 57423382f (#727). What follows describes the current head; my reply lists the changes finding by finding. #630 still carries #495's original commits, which conflict with 2.0.

What

A new module, layerfs_persistence::replication, keeps a second copy of a Store up to date by moving only the rows that changed:

Call What it does Lock
mark_base(path, token) Makes the Store an export base: adds the change log and returns the starting cursor. A copy of the closed file is then a replica at that cursor. writer lane
export_changes(path, since, bundle) Writes every row changed after since into a new SQLite bundle file, from one read snapshot of the live Store none
apply_changes(replica, bundle) Installs one bundle into an offline replica, in one transaction writer lane
trim_changes(path, oldest) Deletes the log up to the cursor of the least advanced replica still followed writer lane
unmark_base(path) Drops the change log; the Store's schema is again that of a Store never marked writer lane
read_state(path) The Store's cursor, or None if it is not a base none

A bundle is a local file. Nothing here moves it anywhere: the caller decides whether it goes to another disk, another machine or object storage. Nothing outside the module's tests calls the API yet.

How it works

  • Only a base has a change log. The shipped schema and the open path are 2.0's: no migration, no trigger, and an older binary opens a Store that was never marked. mark_base adds change_state, change_log and AFTER INSERT/UPDATE/DELETE triggers on the 18 replicated tables (plus pack_unit in the group-row layouts), and records the schema with them, so open validation and the Doctor accept the base. An older binary refuses a base. unmark_base restores the never-marked schema.
  • The log stays bounded only if it is trimmed. A base logs every write until it is trimmed or unmarked. A copy of a base, such as a backup or a Store restored from one, is a base too. docs/architecture/engine/store/sqlite.md says both.
  • Writers. mark_base, apply_changes, trim_changes and unmark_base take the writer lane with Writer::try_acquire, one attempt, and return ChangeError::Busy while another writer holds it. Then they begin with BEGIN IMMEDIATE and no busy handler, so a connection outside the lane holding SQLite's write lock also gets Busy. No call waits or retries.
  • Refusals before any install. Each bundle carries a chained state, a BLAKE3 digest of its rows and deletions, and the identity of its Store's schema (a BLAKE3 hash of user_version and every sqlite_schema row, read in the export's snapshot). apply_changes refuses a skipped or replayed bundle (Cursor), a replica with its own writes (Diverged), a changed bundle (Integrity) and a replica with another schema (Schema), and compares each bundled table's CREATE text with the replica's. A refused bundle leaves the replica unchanged.
  • Files. An export writes the bundle under a temporary name and hard-links it into place, so the bundle path names a complete file or nothing. Store connections run the Disposable profile; a non-WAL replica is refused. mark_base and apply_changes checkpoint after they commit and report a blocked checkpoint as NotCheckpointed.
  • 2.0 specifics. Inode values and binding hints are logged as the schema version 2 blocks they are stored in (inode_value_block by first_ordinal, binding_hint_block by grp, seq). The Doctor's storage-class check (schema_diagnostic::type_checks) skips a STRICT table's ANY columns, which declare no storage class; the change log's keys are ANY.

The second commit stops apply_changes from logging the rows it installs on the replica. Without it, a 1500-object bundle leaves 22 free pages in the replica.

Not changed, not done

  • No change to object formats, the pack layout, history semantics, the publication lane, the PostgreSQL backend, or any layerfs-api, SDK or CLI surface. Cargo.lock is unchanged: blake3 =1.8.5 moves from the dev-dependencies to the dependencies of layerfs-persistence.
  • Not measured: the cost of a Commit on a marked Store. A Store that is never marked runs 2.0's schema with no trigger.
  • Still open from feat(persistence): export a Store's changed rows as bundles for replicas #495: 2.0 keeps Project records in the home, outside the Store file, so after a bundle that creates a Project the replica's CLI answers NoSuchProject.

Tests

tests/change_export.rs has 13 tests: the replica round trip (with the Doctor on both sides); every pack layout; changed bundles, each refused; a written replica; a replica with another schema; a replica locked outside the lane; every write refused while another writer holds the lane; a reader blocking the checkpoint; a non-WAL replica; an unmarked Store; trimming; unmarking (the schema then equals a never-marked Store's, and the Doctor passes); and the free-page test.

The loop, on this head

I reran the two-machine loop from #495 with the release CLI and a small driver built from 59e9726ab. Machine A writes through layerfs and exports one bundle per round. The bundle moves through a shared folder, and machine B applies it. At r0, A was marked and backed up to MinIO, and B was restored from that backup. After each round, all 23 Store tables on A and B are compared row by row, and the same CLI reads run on both machines.

Round Writes on A (CLI) Bundle Rows / deletes Tables equal Reads equal
r1 one file apply: write, mkdir, mv, cp, rm, edit 41,984 B 31 / 0 23/23 12/12
r2 one Commit of four folders, 30 single-file Commits, publish 93,184 B 383 / 0 23/23 16/16
r3 500 + 200 files on two Branches, three-way merge 205,824 B 1,446 / 0 23/23 24/24
r4 64 MiB text + 16 MiB random 29,946,880 B 4,393 / 0 23/23 29/29
r5 second Project shop/web: init + write 157,696 B 290 / 0 23/23 29/31 (the Project gap)
r6 delete a file and a folder, project remove shop/web 33,792 B 24 / 11 23/23 35/35
r7 4 parallel writers × 15 Commits 146,432 B 787 / 0 23/23 43/43
r9 one Commit + publish (r8 is the same, exported before r9) 33,792 B 23 / 0 23/23 49/49
  • The 9 bundles total 30.7 MB, of which r4 is 29.9 MB. A whole-file push after each round would move about 1.0 GB (the Store was 91–122 MB). Export took 2.5–66.5 ms and apply took 2.5–82.7 ms per round.
  • A skipped bundle and a replayed bundle are refused (Cursor), and a replica with a Commit of its own is refused (Diverged). In all three cases, the replica's table digests did not change.
  • Every write on A succeeded, including all 60 parallel writes in r7.
Loop setup
  • Linux aarch64 Docker containers (Debian bookworm) running release builds with Rust 1.85.1. The driver, not part of this PR, calls mark_base, export_changes, apply_changes and read_state here, and also trim_changes and unmark_base in the run below.
  • A and B each have their own LAYERFS_HOME on ext4. MinIO runs the pinned image from minio_service.sh. The backup carries the change state, so B starts at A's cursor.
  • The Project is a git archive of 0b8ae8af5: 5,631 files, 132 MB.
  • After each round: a digest of every row in all 23 tables on A and on B, ordered by primary key; the same CLI reads on both machines (file read/stat/list/glob/grep, branch list/get, commit list), compared byte by byte; and B's read_state, checked against the after cursor of A's export.

Two binaries, live writers, trim and unmark

Two containers and MinIO, with release CLIs built from 57423382f and from this branch:

  • A Store written by the 57423382f CLI and then by this branch keeps an identical schema (same hash of user_version, every sqlite_schema row, schema_source and schema_definition). The 57423382f CLI still reads and writes it, and its Doctor passes all 57 checks.
  • After mark_base, this branch's Doctor passes. The 57423382f CLI refuses the Store (Store open: physical Store I/O: pack persistence: Malformed) and leaves every table unchanged.
  • 1500 CLI writes on the live Store overlapped 40 rounds of export, apply and trim_changes to the replica's cursor. No write failed. 7 trims returned Busy and succeeded on a later attempt; 29,814 log rows were trimmed. Every table but change_log and change_state then holds the same rows on both machines, and both CLIs read the same bytes. An export from a trimmed-away cursor is refused (Cursor).
  • unmark_base on both Stores gives the never-marked schema byte for byte; the 57423382f CLI then reads, writes and passes the Doctor again, and mark_base starts a new chain.
Checks run

CI is off for pull requests into layerfs_v2_0_0, so I ran these locally, on this branch and on 57423382f. Linux: aarch64 Debian bookworm container on a macOS host, uid 1000, TMPDIR on ext4, Rust 1.85.1, cargo-nextest, commands from engine/AGENTS.md "Checks and completion" and ci.yml.

Check This branch 57423382f
cargo fmt --all -- --check 4 diffs in 3 files this PR does not touch same 4
check_product_boundary.py, with and without --skip-sdk; tools unit tests pass pass
clippy --locked --workspace --exclude layerfs-node --all-targets -D warnings (the PostgreSQL backend is always built on 2.0) pass pass
persistence lane 234 of 235 passed 222 of 222 passed
fast lane (ci_tests.py targets/filter fast) 2261 of 2262 passed, 26 skipped 2247 of 2249 passed, 26 skipped
every layerfs-persistence target, macOS aarch64 260 passed, 2 ignored
  • The persistence-lane failure here is concurrent_first_installs::two_writers_saving_the_same_new_content_both_complete. It fails intermittently with Integrity("dependency cycle") on both trees: 3 of 200 runs here, 5 of 200 on 57423382f.
  • The fast-lane failure here, machine_files::the_table_lists_the_call_quiet_with_a_path_and_its_options, also fails on 57423382f. The other failure there, doctor_host::concurrent_management_record_rewrites_leave_consistency_incomplete, passed on this branch.
  • The fmt findings are in api/crates/layerfs-cli/tests/concurrent_reads.rs, control/crates/layerfs-sandbox-backend/src/backend/namespaces/mod.rs and control/crates/layerfs-sandbox-backend/tests/namespaces_lifecycle.rs.

Not rerun on this head: the nightly and mount lanes, x86_64, the Durable profile and layerfs-node.

Production LOC: 171524 -> 172720 (delta +1196): engine 76947 -> 78143, all in layerfs-persistence 17288 -> 18484; core, control and api unchanged. Per commit: +1195 and +1. Counted with tools/production_loc.py --json (blob 35e4a6d601) on git archive extracts of 57423382f and of each commit's tree.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants