Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 48 additions & 0 deletions docs/architecture/engine/store/sqlite.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ The SQLite backend of the [Store](README.md): one WAL file every opener shares t
store_totals id=1 file_digest
lineage: layer_lineage, commit_lineage, absorbed, reach
merge: merge_operation, merge_record
export: change_state, change_log (export base only)
Init: init_operation, init_entry, init_host_file (42-44 only)
```

Expand Down Expand Up @@ -203,6 +204,53 @@ The import algorithm is in [Project Init](../project-init/README.md).
| Spilled | Then a write-back buffer of `BUFFER_BYTES` (256 KiB), flushed in one write | same |
| Lease | `MERGE_LEASE` 600 s; writable open sweeps expired owners | same |

## Change export

```text
closed Store live Store (writers continue) offline replica
mark_base(path) export_changes(path, cursor, apply_changes(
-> base cursor bundle) -> next cursor replica, bundle)
copy = replica one read snapshot, no lane one transaction
trim_changes(path, oldest)
unmark_base(path)
```

Only an export base has a change log. `mark_base` adds `change_state`,
`change_log` and one trigger per logged table and write kind, and records the
schema with them; `unmark_base` drops them and records the schema of a Store
that was never marked, which a build without this module opens. A Store that
was never marked has that schema, runs no trigger and logs nothing. The
triggers cover every physical and history table except `merge_*` and `init_*`
(19 tables in layouts 37–38, 18 without `pack_unit`) and log each inserted,
updated or deleted row's key in the single writer's commit order; `pack` and
`pack_unit` log inserts only.

| Function | What it does | Source |
| --- | --- | --- |
| `mark_base(path, token)` | Starts a chain on an offline Store: adds the change log when the Store has none, records a fresh state and the current sequence; a copy of the closed file is a replica at the returned cursor. Refuses a Store whose recorded schema is not this build's | [`base.rs`](../../../../engine/crates/layerfs-persistence/src/replication/base.rs), [`log.rs`](../../../../engine/crates/layerfs-persistence/src/replication/log.rs) |
| `export_changes(path, after, bundle)` | Writes every row changed after `after`, at its current value, the deleted keys and the identity of the Store's schema into a new bundle file (SQLite, the logged tables' exact definitions); written under a temporary name and linked into place | [`export.rs`](../../../../engine/crates/layerfs-persistence/src/replication/export.rs) |
| `apply_changes(replica, bundle)` | Checks the bundle's meta, schema identity, table definitions, starting state, digest and ending state, then installs it in one transaction with foreign keys checked at commit; rows it installs are not logged on the replica | [`apply.rs`](../../../../engine/crates/layerfs-persistence/src/replication/apply.rs) |
| `trim_changes(path, oldest)` | Moves the base forward to the cursor of the least advanced replica still followed and deletes the log up to it; later exports from an earlier cursor are refused | [`base.rs`](../../../../engine/crates/layerfs-persistence/src/replication/base.rs) |
| `unmark_base(path)` | Ends the chain: drops the change log and its triggers | same |

While a Store is a base, every write appends to its log, which grows until
`trim_changes` or `unmark_base`. A copy of a base, such as a backup or a Store
restored from one, is itself a base and logs every write until it is unmarked.
The schema identity covers `user_version` and every table, index and trigger
definition, so a replica migrated on one side only refuses a bundle with
`Schema` before installing a row.

`mark_base`, `apply_changes`, `trim_changes` and `unmark_base` take the
Store's writer lane in one non-blocking attempt and return `Busy` while another
writer holds it; a connection that bypasses the lane and holds SQLite's write
lock also makes them `Busy`. `export_changes` reads one snapshot and takes no
lane. Each bundle's ending state derives from its starting state and its
digest, so a bundle applies only to the state it was exported from, and two
replicas that diverge never share a later state. The digest detects damage and
misordering; it is not a signature. `mark_base` and `apply_changes` checkpoint
after they commit and return `NotCheckpointed` when another reader blocks it;
`trim_changes` and `unmark_base` commit without a checkpoint.

## Related

- [Store](README.md): the seam, admission, history calls and errors.
Expand Down
4 changes: 1 addition & 3 deletions engine/crates/layerfs-persistence/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ layerfs-storage = { path = "../../../core/crates/layerfs-storage" }
layerfs-history = { path = "../../../core/crates/layerfs-history", default-features = false }
layerfs-content = { path = "../../../core/crates/layerfs-content" }
layerfs-telemetry = { path = "../../../core/crates/layerfs-telemetry" }
blake3 = { version = "=1.8.5", default-features = false }
rusqlite = { version = "=0.40.2", features = ["cache", "limits", "blob"] }

# PostgreSQL Store engine (#506), always built beside SQLite. Owner-authorized new
Expand All @@ -23,9 +24,6 @@ native-tls = "=0.2.18"
tokio = { version = "=1.54.1", features = ["rt", "net", "time"] }
tokio-postgres = "=0.7.18"

[dev-dependencies]
blake3 = { version = "=1.8.5", default-features = false }

[target.'cfg(any(target_os = "macos", target_os = "linux"))'.dependencies]
nix = { version = "=0.31.3", default-features = false, features = ["fs"] }

Expand Down
134 changes: 134 additions & 0 deletions engine/crates/layerfs-persistence/sql/sqlite/change_log.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
CREATE TABLE change_state (
id INTEGER PRIMARY KEY CHECK(id=1),
state BLOB NOT NULL CHECK(length(state)=32),
seq INTEGER NOT NULL CHECK(seq>=0)
) STRICT;
CREATE TABLE change_log (
seq INTEGER PRIMARY KEY CHECK(seq>0),
tbl INTEGER NOT NULL CHECK(tbl BETWEEN 0 AND 19),
k1 ANY,
k2 ANY,
k3 ANY,
CHECK((tbl=0)=(k1 IS NULL))
) STRICT;
CREATE TRIGGER change_pack_insert AFTER INSERT ON pack WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(1,new.pack_id); END;
CREATE TRIGGER change_object_location_insert AFTER INSERT ON object_location WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(3,new.object_id); END;
CREATE TRIGGER change_object_location_update AFTER UPDATE ON object_location WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(3,new.object_id);
INSERT INTO change_log(tbl,k1) SELECT 3,old.object_id WHERE old.object_id IS NOT new.object_id; END;
CREATE TRIGGER change_object_location_delete AFTER DELETE ON object_location WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(3,old.object_id); END;
CREATE TRIGGER change_inode_value_block_insert AFTER INSERT ON inode_value_block WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(4,new.first_ordinal); END;
CREATE TRIGGER change_inode_value_block_update AFTER UPDATE ON inode_value_block WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(4,new.first_ordinal);
INSERT INTO change_log(tbl,k1) SELECT 4,old.first_ordinal WHERE old.first_ordinal IS NOT new.first_ordinal; END;
CREATE TRIGGER change_inode_value_block_delete AFTER DELETE ON inode_value_block WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(4,old.first_ordinal); END;
CREATE TRIGGER change_content_signature_insert AFTER INSERT ON content_signature WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(5,new.slot); END;
CREATE TRIGGER change_content_signature_update AFTER UPDATE ON content_signature WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(5,new.slot);
INSERT INTO change_log(tbl,k1) SELECT 5,old.slot WHERE old.slot IS NOT new.slot; END;
CREATE TRIGGER change_content_signature_delete AFTER DELETE ON content_signature WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(5,old.slot); END;
CREATE TRIGGER change_store_policy_insert AFTER INSERT ON store_policy WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(6,new.id); END;
CREATE TRIGGER change_store_policy_update AFTER UPDATE ON store_policy WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(6,new.id);
INSERT INTO change_log(tbl,k1) SELECT 6,old.id WHERE old.id IS NOT new.id; END;
CREATE TRIGGER change_store_policy_delete AFTER DELETE ON store_policy WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(6,old.id); END;
CREATE TRIGGER change_store_totals_insert AFTER INSERT ON store_totals WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(7,new.id); END;
CREATE TRIGGER change_store_totals_update AFTER UPDATE ON store_totals WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(7,new.id);
INSERT INTO change_log(tbl,k1) SELECT 7,old.id WHERE old.id IS NOT new.id; END;
CREATE TRIGGER change_store_totals_delete AFTER DELETE ON store_totals WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(7,old.id); END;
CREATE TRIGGER change_history_meta_insert AFTER INSERT ON history_meta WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(8,new.id); END;
CREATE TRIGGER change_history_meta_update AFTER UPDATE ON history_meta WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(8,new.id);
INSERT INTO change_log(tbl,k1) SELECT 8,old.id WHERE old.id IS NOT new.id; END;
CREATE TRIGGER change_history_meta_delete AFTER DELETE ON history_meta WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(8,old.id); END;
CREATE TRIGGER change_layer_stack_insert AFTER INSERT ON layer_stack WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(9,new.layer_stack_id); END;
CREATE TRIGGER change_layer_stack_update AFTER UPDATE ON layer_stack WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(9,new.layer_stack_id);
INSERT INTO change_log(tbl,k1) SELECT 9,old.layer_stack_id WHERE old.layer_stack_id IS NOT new.layer_stack_id; END;
CREATE TRIGGER change_layer_stack_delete AFTER DELETE ON layer_stack WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(9,old.layer_stack_id); END;
CREATE TRIGGER change_layer_insert AFTER INSERT ON layer WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(10,new.layer_id); END;
CREATE TRIGGER change_layer_update AFTER UPDATE ON layer WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(10,new.layer_id);
INSERT INTO change_log(tbl,k1) SELECT 10,old.layer_id WHERE old.layer_id IS NOT new.layer_id; END;
CREATE TRIGGER change_layer_delete AFTER DELETE ON layer WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(10,old.layer_id); END;
CREATE TRIGGER change_commit_insert AFTER INSERT ON "commit" WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(11,new.commit_id); END;
CREATE TRIGGER change_commit_update AFTER UPDATE ON "commit" WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(11,new.commit_id);
INSERT INTO change_log(tbl,k1) SELECT 11,old.commit_id WHERE old.commit_id IS NOT new.commit_id; END;
CREATE TRIGGER change_commit_delete AFTER DELETE ON "commit" WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(11,old.commit_id); END;
CREATE TRIGGER change_branch_insert AFTER INSERT ON branch WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(12,new.branch_id); END;
CREATE TRIGGER change_branch_update AFTER UPDATE ON branch WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(12,new.branch_id);
INSERT INTO change_log(tbl,k1) SELECT 12,old.branch_id WHERE old.branch_id IS NOT new.branch_id; END;
CREATE TRIGGER change_branch_delete AFTER DELETE ON branch WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(12,old.branch_id); END;
CREATE TRIGGER change_scope_allocator_insert AFTER INSERT ON scope_allocator WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(13,new.scope_id); END;
CREATE TRIGGER change_scope_allocator_update AFTER UPDATE ON scope_allocator WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(13,new.scope_id);
INSERT INTO change_log(tbl,k1) SELECT 13,old.scope_id WHERE old.scope_id IS NOT new.scope_id; END;
CREATE TRIGGER change_scope_allocator_delete AFTER DELETE ON scope_allocator WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(13,old.scope_id); END;
CREATE TRIGGER change_binding_hint_block_insert AFTER INSERT ON binding_hint_block WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1,k2) VALUES(14,new.grp,new.seq); END;
CREATE TRIGGER change_binding_hint_block_update AFTER UPDATE ON binding_hint_block WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1,k2) VALUES(14,new.grp,new.seq);
INSERT INTO change_log(tbl,k1,k2) SELECT 14,old.grp,old.seq WHERE old.grp IS NOT new.grp OR old.seq IS NOT new.seq; END;
CREATE TRIGGER change_binding_hint_block_delete AFTER DELETE ON binding_hint_block WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1,k2) VALUES(14,old.grp,old.seq); END;
CREATE TRIGGER change_layer_lineage_insert AFTER INSERT ON layer_lineage WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(15,new.layer_id); END;
CREATE TRIGGER change_layer_lineage_update AFTER UPDATE ON layer_lineage WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(15,new.layer_id);
INSERT INTO change_log(tbl,k1) SELECT 15,old.layer_id WHERE old.layer_id IS NOT new.layer_id; END;
CREATE TRIGGER change_layer_lineage_delete AFTER DELETE ON layer_lineage WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(15,old.layer_id); END;
CREATE TRIGGER change_commit_lineage_insert AFTER INSERT ON commit_lineage WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(16,new.commit_id); END;
CREATE TRIGGER change_commit_lineage_update AFTER UPDATE ON commit_lineage WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(16,new.commit_id);
INSERT INTO change_log(tbl,k1) SELECT 16,old.commit_id WHERE old.commit_id IS NOT new.commit_id; END;
CREATE TRIGGER change_commit_lineage_delete AFTER DELETE ON commit_lineage WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(16,old.commit_id); END;
CREATE TRIGGER change_absorbed_insert AFTER INSERT ON absorbed WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1,k2) VALUES(17,new.run,new.depth); END;
CREATE TRIGGER change_absorbed_update AFTER UPDATE ON absorbed WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1,k2) VALUES(17,new.run,new.depth);
INSERT INTO change_log(tbl,k1,k2) SELECT 17,old.run,old.depth WHERE old.run IS NOT new.run OR old.depth IS NOT new.depth; END;
CREATE TRIGGER change_absorbed_delete AFTER DELETE ON absorbed WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1,k2) VALUES(17,old.run,old.depth); END;
CREATE TRIGGER change_reach_insert AFTER INSERT ON reach WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1,k2) VALUES(18,new.key,new.x); END;
CREATE TRIGGER change_reach_update AFTER UPDATE ON reach WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1,k2) VALUES(18,new.key,new.x);
INSERT INTO change_log(tbl,k1,k2) SELECT 18,old.key,old.x WHERE old.key IS NOT new.key OR old.x IS NOT new.x; END;
CREATE TRIGGER change_reach_delete AFTER DELETE ON reach WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1,k2) VALUES(18,old.key,old.x); END;
CREATE TRIGGER change_file_digest_insert AFTER INSERT ON file_digest WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1,k2) VALUES(19,new.digest,new.length); END;
CREATE TRIGGER change_file_digest_update AFTER UPDATE ON file_digest WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1,k2) VALUES(19,new.digest,new.length);
INSERT INTO change_log(tbl,k1,k2) SELECT 19,old.digest,old.length WHERE old.digest IS NOT new.digest OR old.length IS NOT new.length; END;
CREATE TRIGGER change_file_digest_delete AFTER DELETE ON file_digest WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1,k2) VALUES(19,old.digest,old.length); END;
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
CREATE TRIGGER change_pack_unit_insert AFTER INSERT ON pack_unit WHEN EXISTS(SELECT 1 FROM change_state) BEGIN
INSERT INTO change_log(tbl,k1) VALUES(2,new.unit_id); END;
Loading