Repository navigation
perf(gc): faster store.compact on SQLite and PostgreSQL; fix settle lane and refused seal - #734
Merged
Merged
Conversation
…#41) Mark needs one fact about a file payload: its delta base. It found it by reading and hashing each live pack whole. It now reads the group's record directory and each record's first 41 bytes. Changes - core delta/record.rs: header_base parses a payload record's base from its header alone and accepts and refuses exactly what stored_base does. - core delta/edge.rs (new): RecordDirectory validates and parses a raw group's directory once; sparse_directory and sparse_edge answer from the bytes a caller holds and name the next range they lack. - core delta/read.rs: stored_edge is ChainBases::edge over caller-owned caches, so the walk's decoded-group cache is shared and a compressed group is decompressed once, not once per record. - engine history/objects/edges.rs (new): per pack and per wave, bounded acquisitions of directories and headers; group_directories for the waste accounting. - engine history/objects.rs: the walk takes header edges first and keeps the decoding reader for compressed groups and the Ordinary lane; kept waste reads group directories instead of group bodies; inode values use a map; the wave sort computes each key once. - engine backend/sqlite/objects/read.rs: the pack prefix read uses incremental BLOB I/O. `substr(body,1,?)` made SQLite load every pack body whole to return its first bytes. Quadratic steps removed: a record lookup rescanned its group's directory (records squared per group, in kept waste and in the first header reader); inode values were scanned once per ordinal. Trade-off: header reads are not authenticated, so the mark no longer hashes uncompressed payload packs as a side effect. The spec's verify step never required it (no sha256 of either file); spec section 6.2 now says so. Measured (gc_bench, Disposable profile, warm cache, macOS, one invocation per binary, load 5 to 8; receipts not committed) - small, debug, 3 runs per arm, 704 live objects: mark 564.0 -> 50.9 ms (none), 578.4 -> 61.2 ms (rebuild); same 3,133,440 bytes freed. - medium, release, 1 run per arm, 9.789 GB live in 83,559 packs: mark 45.17 -> 4.79 s (none), 24.68 -> 1.53 s (rebuild); rebuild wall 66.58 -> 46.16 s; same 1,095,524,352 bytes freed. Tests - New core tests/sparse_edges.rs (5): header reader equivalence with stored_base including truncations, sparse walk bounds, broken directories, and a full 8,191-record group answered in one pass. - Pass: layerfs-storage (122); persistence kept_objects, kept_node, compact_metrics, sqlite_compact, sqlite_compact_crash, sqlite_compact_matrix, sqlite_compact_units, space_reclamation, acquisition_reads, acquisition_batching, sqlite_group_rows; PostgreSQL gc_mark, compact, compact_fixtures, compact_units, identical_ids, gc_fence against a local PostgreSQL 17. - No test deleted. Production LOC: 187394 -> 187909 (delta +515) Scope and method: tools/production_loc.py at this commit, first parent 6bca861 against the committed tree; core 40408 -> 40589, engine 89386 -> 89720, control 51073 -> 51073, api 6527 -> 6527; reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…roject#41) Finding where a mark or a compaction spends its time needed more than the aggregate counters: which statement, how often, how long. - session/thread_work.rs: profile_statements() starts a per-thread profile keyed by statement text (whitespace collapsed, first 110 characters); take_statement_profile() ends it and returns it costliest first. Off, a statement pays one thread-local read. - session/prepared.rs: a statement lease keeps its text only while its thread profiles, and notes each preparation and execution. - examples/gc_profile.rs: on a Store gc_bench kept, runs N marks and optionally one compaction, printing wall time, this thread's SQL work counters and the costliest statements. - tools/ci/gc-bench.md: how to use it and what it has shown. No test added: the example is a diagnostic with no receipt. No test deleted. Production LOC: 187909 -> 187965 (delta +56) Scope and method: tools/production_loc.py at this commit, first parent d19d98f against the committed tree; core 40589 -> 40589, engine 89720 -> 89776, control 51073 -> 51073, api 6527 -> 6527; reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…(layerfs-project#41) The profile of a mark over 8,816 packs showed two statements and two BLOB handles per pack: one acquisition for the record headers and one prefix read for the kept-waste pass. - backend/seam.rs: Mechanics::pack_prefixes reads the bounded prefixes of several packs in one call (default: one pack_prefix each). - backend/sqlite: one BLOB handle moved from row to row, so a pack costs no statement and no handle of its own. The single prefix read shares the same head reader. - history/objects/edges.rs: a wave reads the first 1,024 bytes of each of its packs, 256 packs per call. A pack whose directory and record headers sit in that head is answered from it; a wider directory is read again, and only a pack that needs more ranges takes a scoped acquisition. - history/objects.rs: the walk keeps each pack's group extents and the kept-waste pass uses them instead of reading the directory again; the scope hash used for metadata objects is computed once per walk. PostgreSQL keeps the default pack_prefixes (one statement per pack); a batched statement there is a follow-up. Measured (gc_profile, release, warm cache, macOS, load about 4, one run; a 1.13 GB gc_bench Store of 8,913 objects in 8,816 packs): mark wall 88.8 -> 36.3 ms; statements 17,740 -> 298; BLOB handles 17,633 -> 229; BLOB reads 35,267 -> 9,104; bytes read 40.4 -> 9.0 MB. The same binary on the 28 MB small Store (737 objects in 158 packs of up to six groups): mark 6.8 ms, 172 statements, 150 handles; packs of several groups still take one acquisition each. Tests: kept_objects, kept_node, compact_metrics, sqlite_compact, sqlite_compact_crash, sqlite_compact_matrix, space_reclamation, postgresql_gc_mark, postgresql_compact, postgresql_compact_fixtures pass. No test added or deleted. Production LOC: 187965 -> 188037 (delta +72) Scope and method: tools/production_loc.py at this commit, first parent against the committed tree; core 40589 -> 40589, engine 89776 -> 89848, control 51073 -> 51073, api 6527 -> 6527; reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…layerfs-project#41) The §7 build writes the fresh Store one page per pwrite. A CPU sample of the 1.1 GB rebuild put 70% of the copy in pwrite: on APFS a page write that also moves the end of the file costs about four times one that lands inside a sized, allocated file (1 GiB of 4 KiB writes: 1.05 s extending, 0.58 s into a sized hole, 0.28 s sized and preallocated; ext4 in the Docker VM showed no difference between the three). The run already knows the retained payload (the mark's packs and their lengths) before it copies a byte, so the build now sizes the fresh file for it once, after the schema exists and before the load: - compact/space.rs `reserve`: macOS only. One advisory F_PREALLOCATE and one set_len, rounded down to whole pages, never shrinking. A refused allocation leaves a sparse file of the same size; a load that is really out of space still meets its own Capacity refusal. - compact/space.rs `trim`: after the checked close the file is cut to the pages its header counts. The size is a lower bound of the load today (a pack body needs at least its length in pages), so nothing is cut; a build that writes less than the mark's packs (a repack of partly dead packs) would otherwise install zero pages past the database's end. - build.rs: `BuildInput.payload_bytes` carries the retained pack bytes from run.rs; the page count is read once the schema is complete. No statement is added to the copy; one PRAGMA page_count is added to the build. The installed file is byte-for-byte the size it was (1,134,886,912 bytes on the 1.1 GB shape, before and after). Measured, release, macOS APFS, one sample each, gc_bench --runs 1: - small (28 MB): rebuild 122.7 -> 106.3 ms, build 51.9 -> 36.9 ms (1-minute load 3.3 before, 4.8 after). - 1.1 GB (GC_BENCH_GROUPS=96 ... OBJECT_BYTES=131071): rebuild 1749.9 -> 1039.2 ms, build 1550.7 -> 833.2 ms; the pack copy statement alone 1305 -> 548 ms (load 3.4 before, 4.8 after). A cold source is not helped by this: its copy is bound by reads. Tests: new sqlite_compact_units `a_sized_fresh_file_loads_and_is_cut_back_to_its_page_count`. Pass: sqlite_compact (33), sqlite_compact_crash (15), sqlite_compact_matrix (1), sqlite_compact_units (8), compact_metrics (5), space_reclamation (3), kept_objects (7), kept_node (13); sqlite_compact_capacity has no macOS test. No test deleted. Production LOC: 187,909 -> 187,965 (delta +56) core 40,589 -> 40,589 (0) engine 89,720 -> 89,776 (+56) layerfs-persistence +56 control 51,073 -> 51,073 (0) api 6,527 -> 6,527 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd19) Method: tools/production_loc.py --detail on parent d19d98f and the staged tree Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…rfs-project#41) One INSERT ... SELECT over every retained pack reads the sealed file in the order SQLite stored it: a pack's row, then its payload pages a few dozen pages behind that row, then the next row. No kernel read-ahead follows that pattern, so a source that is not in memory is read one pack per device round trip. On a 1.26 GB Store written past the buffer cache (F_NOCACHE) the pack copy took 3.4 to 3.5 s, against 0.55 s for the same Store in memory; a plain sequential read of that file runs at 4.4 GB/s cold. The 9.8 GB bench shape was in this state (its first mark ran 3x slower than its second), which is where its 225 MB/s build came from. - compact/window.rs `windows`: cuts the retained packs into runs of 64 MiB of retained payload, ended early by a dead run over 8 MiB. Packs are appended in id order, so a pack's payload starts no earlier than the summed lengths of the packs before it and no later than that plus every byte of the file that is not pack payload; a window's file range is that bound over its packs. Both inputs (every pack's length, the mark's packs) were already in memory in run.rs. - compact/window.rs `ReadAhead`: before each window, asks the kernel for the window's range through a second read-only descriptor, each byte once. macOS: F_RDADVISE, which returns when the range is in memory (about 5 GB/s cold, 4 ms per GB warm). Linux: POSIX_FADV_WILLNEED in 128 KiB steps (the kernel caps one request at the device read-ahead size). A hint only: no byte and no result changes, and a file laid out in another order only loses the benefit. The sealed file is attached immutable=1, so SQLite holds no lock on it that closing the descriptor could drop. - build.rs `copy_packs`: the payload table (`pack`, or `pack_unit` in the group-row layouts) is copied one window per execution of one prepared statement, `WHERE pack_id BETWEEN ?1 AND ?2 AND <retained>`. The retained filter and every other table's copy are unchanged. Statements: the pack copy is one prepared statement run once per 64 MiB of retained payload (17 on the 1.1 GB shape, 1 on the small one) instead of one statement run once. Rows, bytes and the installed file are identical. Measured, release, macOS APFS, one sample each. Cold = the Store copy was written with F_NOCACHE just before the run (the cache state is declared by method, not verified); rebuild_probe is a local diagnostic, the same compaction call gc_bench makes. - 1.1 GB cold: pack copy 3483 -> 690 ms, build 3710 -> 865 ms, rebuild wall 5054 -> 2071 ms (load 6.1 and 6.8). The mark is 0.8 to 0.9 s of what remains: it is bound by the same one-pack-per-round-trip reads. - small (28 MB) cold: pack copy 5.9 -> 6.2 ms, no change (one window). - 1.1 GB warm, gc_bench --runs 1: rebuild 1039 -> 1638 ms in a sample taken at load 5.9 to 7.2 in which the untouched mark also ran 96 -> 175 ms; the instrumented warm run beside it (load 6.6) gave pack copy 583 ms in 17 windows against 548 ms as one statement, build 734 ms, rebuild 1082 ms. Small warm, gc_bench: 106 -> 146 ms in the same loaded sample (mark 17 -> 22 ms, none path 25 -> 51 ms). Linux was type-checked for aarch64-unknown-linux-musl and x86_64-unknown-linux-gnu in a scratch crate (the package itself does not cross-check from this host); it was not run or measured. Tests: new sqlite_compact `retained_packs_on_both_sides_of_a_long_dead_run_are_all_copied` (a two window rebuild) and sqlite_compact_units `the_pack_copy_is_cut_into_windows_in_pack_order`. The generated matrix report counts the new sqlite_compact test (34 -> 35). Pass: sqlite_compact (34), sqlite_compact_crash (15), sqlite_compact_matrix (1), sqlite_compact_units (9), compact_metrics (5), space_reclamation (3), kept_objects (7), kept_node (13). No test deleted. Production LOC: 187,965 -> 188,160 (delta +195) core 40,589 -> 40,589 (0) engine 89,776 -> 89,971 (+195) layerfs-persistence +195 control 51,073 -> 51,073 (0) api 6,527 -> 6,527 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd19) Method: tools/production_loc.py --detail on parent 7b2f9ac and the staged tree Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ory sync (layerfs-project#41) Two steps of a SQLite rebuild changed no outcome: - After the mark the run rewrote `<store>.compact-op` (write, fsync, rename, directory fsync) only to add the mark's tuple and digest. No reader uses them: a later call judges a leftover by the run, generation and owner the admission record already names, and `store.status` reads the admission phase as `mark`. The values go out with the next write (`build`). A stop point still writes its own record with them. - After the build the run synchronized the Store's directory "before the verification reads them back". Verification reads through the page cache, the fresh file is synchronized before the rename and the directory after it, and the record written next is synchronized itself. Round trips per rebuild: 6 op record writes -> 5, 16 fsync calls -> 13 (each is F_FULLFSYNC on macOS). The `none` path is unchanged. docs/architecture/engine/store/sqlite.md gains a "Compaction rebuild" section. It also covers the two previous commits (the fresh file's size, the pack copy's windows and the read-ahead), which changed the build's steps without a page to update. Measured, release, macOS APFS, gc_bench --runs 1, one sample each (1-minute load 3.0 to 3.5): - small (28 MB): rebuild 106.3 -> 95.1 ms against the last sample taken at a comparable load (the first commit of this series); mark phase 17.0 -> 8.2 ms, which is the removed record rewrite. - 1.1 GB: rebuild 1039.2 -> 1138.3 ms; build 833.2 -> 800.7 ms, verify 22.4 -> 155.5 ms. The device flush that follows the 1.1 GB load is not removed by this commit, only moved: it was paid by the directory sync inside the build phase and is now paid by the first record write of the verify phase. This shape gains nothing here. Tests: no new test (an fsync is not observable through the public API). Pass: sqlite_compact (34), sqlite_compact_crash (15), sqlite_compact_matrix (1), sqlite_compact_units (9), compact_metrics (5), space_reclamation (3), kept_objects (7), kept_node (13). No test deleted. Production LOC: 188,160 -> 188,152 (delta -8) core 40,589 -> 40,589 (0) engine 89,971 -> 89,963 (-8) layerfs-persistence -8 control 51,073 -> 51,073 (0) api 6,527 -> 6,527 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd19) Method: tools/production_loc.py --detail on parent d463879 and the staged tree Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…project#41) NEEDS THE OWNER'S DECISION: spec §7 step 7 reads "chmod 0600; sync_all; rename; fsync the directory" without naming a profile. This commit keeps that sequence for a Store whose profile synchronizes (Normal, Durable) and skips the synchronization for a Disposable one. It is one flag; reverting this commit restores the unconditional calls. A SQLite rebuild made 13 fsync calls of its own (16 before the previous commit): two per op record write, the fresh file before the rename, the directory after it and after the settle. Each is F_FULLFSYNC on macOS. On the 28 MB shape they were about two thirds of the whole rebuild, and three of the `none` path's work was the same calls. A Disposable Store (WAL, synchronous=OFF) survives a process crash and claims nothing after a power loss: its own commits and checkpoints are never synchronized, so a compaction that synchronizes its record and its install protects a Store that the same power loss may already have torn. The run now takes `durable = profile.synchronous != 0` once: - op.rs `RunOp.durable`: a record write is still temp file plus rename (what a process crash needs); the file and directory fsync run only when durable. A record read back from disk is written synchronized, and `mark_returned` takes the run's flag. - install.rs `prepare_target(target, durable)`: chmod always, sync_all only when durable. `settle` synchronizes the directory only when durable. - run.rs: the directory fsync after the rename and at the end of the `none` path run only when durable. The abort paths are unchanged. A Normal Store's compaction is unchanged by this commit: the same 13 calls. The product opens Stores Normal since 2026-11-17 while GC is measured under Disposable (#42 item 42), so from here the Disposable bench numbers exclude synchronization the product profile still pays. Measured on the same fixtures opened Normal (rebuild_probe, a local diagnostic; one sample, load 3.9): 28 MB rebuild 142 ms against 46 ms Disposable; 1.1 GB rebuild 1100 ms against 963 ms. Measured, release, macOS APFS, gc_bench --runs 1 (Disposable), one sample each, 1-minute load 4.9: - small (28 MB): rebuild 95.1 -> 32.7 ms (build 31.8 -> 20.8, verify 10.5 -> 1.0, install 34.2 -> 2.1 ms); `none` 24 -> 9 ms. - 1.1 GB: rebuild 1138.3 -> 982.5 ms (build 800.7 -> 782.2, verify 155.5 -> 11.3, install 78.7 -> 94.1 ms); `none` 115 -> 93 ms. What is left of install is the rename, which frees the old file's 1.26 GB. Tests: no new test (an fsync is not observable through the public API). Pass: sqlite_compact (34), sqlite_compact_crash (15), sqlite_compact_matrix (1), sqlite_compact_units (9), compact_metrics (5), space_reclamation (3), kept_objects (7), kept_node (13). No test deleted. Production LOC: 188,152 -> 188,168 (delta +16) core 40,589 -> 40,589 (0) engine 89,963 -> 89,979 (+16) layerfs-persistence +16 control 51,073 -> 51,073 (0) api 6,527 -> 6,527 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd19) Method: tools/production_loc.py --detail on parent 7f5bb75 and the staged tree Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…project#41) After the batched head read, a pack that needed more than its head still took scoped acquisitions of its own: a statement and a BLOB handle per pack and per round on SQLite, and on PostgreSQL, where mark's time follows round trips, one or more round trips per pack. - backend/seam.rs: Mechanics::pack_ranges returns selected byte ranges of several packs in one call. Its default is one scoped acquisition per pack, which is what PostgreSQL uses until it overrides it. - backend/sqlite: one BLOB handle moved from row to row; a pack's ranges ascend, so its overflow chain is followed once. The GroupRows layouts keep the scoped read. - history/objects/edges.rs: rounds are per wave batch, not per pack. Each round resolves every pack of the batch against the bytes held and reads what all of them still need in bounded calls (1,024 ranges, 512 KiB). The per-pack range plan and acquisition are gone; the kept-waste directory read uses the same call. Measured (gc_profile, release, warm cache, macOS, load about 3, one run): - 28 MB small Store (737 objects in 158 packs of up to six groups): mark 6.8 -> 5.4 ms; statements 172 -> 58; BLOB handles 150 -> 37; reads 996 -> 768. - 1.13 GB Store (8,913 objects in 8,816 single-group packs): unchanged, 37.0 ms, 298 statements, 229 handles. Tests: kept_objects, kept_node, compact_metrics, sqlite_compact, sqlite_compact_crash, space_reclamation, sqlite_group_rows, sqlite_group_rows_indexed, postgresql_gc_mark, postgresql_compact, postgresql_compact_fixtures pass. sqlite_compact_matrix failed once and passed on the next run with no source change: its AC-49 arm races a Commit thread against the rebuild and unwraps the Save inside that thread (tests/sqlite_compact_matrix.rs:149). Reported, not changed here. No test added or deleted. Production LOC: 188037 -> 188192 (delta +155) Scope and method: tools/production_loc.py at this commit, first parent against the committed tree; core 40589 -> 40589, engine 89848 -> 90003, control 51073 -> 51073, api 6527 -> 6527; reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…layerfs-project#41) Counts, not time: none of these is visible in a one-sample wall time on the bench shapes. They remove statements, table passes and connection opens that produced nothing the run did not already have. - compact_visit.rs: the mark ran two inserts per reached object, one into `visited` and one `INSERT OR IGNORE` into `live_pack`. The retained packs are the visited rows' packs, so `finish` now derives them with one ordered `INSERT ... SELECT DISTINCT`. Statements per mark: 2N -> N + 1 (N = reached objects; about 9,000 on the 1.1 GB shape, so about 18,000 -> 9,000). - verify.rs `verify`: four count statements and one totals statement became one statement. Passes over the fresh file: `pack` 3 -> 1, `object_location` 2 -> 1, `store_totals` 5 reads -> 1 (the tuple already reads the row). Own statements 8 -> 4. Every check, its error and the order of the checks around the `DuringVerify` stop are unchanged. - verify.rs `sealed_state`: the sealed file's tuple and live-heads digest are read through one `immutable=1` connection instead of two (`immutable_tuple` and `immutable_digest` are removed). Connections opened per rebuild: 7 -> 6 besides the session. - build.rs: the totals rewrite reads `pack` once for `stored` and `metadata` instead of twice. Measured, release, macOS APFS, gc_bench --runs 1 (Disposable), one sample each, 1-minute load 3.7 and 3.4: - small (28 MB): rebuild 32.7 -> 34.6 ms (mark 7.3 -> 7.6, build 20.8 -> 22.2, verify 0.96 -> 0.94, install 2.06 -> 2.03 ms); `none` 9 -> 9 ms. - 1.1 GB: rebuild 982.5 -> 927.5 ms (mark 92.3 -> 88.5, build 782.2 -> 730.4, verify 11.3 -> 10.2, install 94.1 -> 96.2 ms); `none` 93 -> 92 ms. The build difference is sample noise in the pack copy, not this change. Tests: pass, unchanged: sqlite_compact (34), sqlite_compact_crash (15), sqlite_compact_matrix (1), sqlite_compact_units (9), compact_metrics (5), space_reclamation (3), kept_objects (7), kept_node (13). The rebuild tests cover `live_pack`: verification refuses a fresh file whose pack count is not the mark's. No test deleted. Production LOC: 188,168 -> 188,162 (delta -6) core 40,589 -> 40,589 (0) engine 89,979 -> 89,973 (-6) layerfs-persistence -6 control 51,073 -> 51,073 (0) api 6,527 -> 6,527 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd19) Method: tools/production_loc.py --detail on parent 8e7acb3 and the staged tree Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ayerfs-project#41) Five commits by a worktree agent, merged without conflict onto the mark work: size the fresh file before the bulk load, copy packs in 64 MiB windows with read-ahead on the sealed Store, drop the post-mark record rewrite and the post-build directory sync, make a Disposable Store's compaction ask for no fsync, and merge repeated statements and scans. Owner decision pending: 8e7acb3 departs from spec section 7 step 7 as written (it names sync_all and the directory fsync without naming a profile). Normal, the product profile, still syncs. An independent crash-safety review of the five commits is in progress and is not reflected here. Measured on the merged tree (gc_bench --scale medium, release, Disposable, macOS, one run per arm, load 6.6 at start with other agents working; 9.789 GB live in 83,559 packs; receipt not committed): none mark 7.42 s wall 7.42 s rebuild mark 0.92 s build 21.69 s verify 0.30 s install 0.50 s wall 23.41 s Same bytes freed as every earlier run: 1,095,524,352. Earlier runs of the same shape: at 6bca861 none 45.19 s and rebuild 66.58 s (build 38.45 s); at d19d98f none 4.82 s and rebuild 46.16 s (build 43.93 s). The none arm is the first mark after the fixture is built and varies with the page cache: 4.79 s then, 7.42 s now. Tests on the merged tree: kept_objects 7, kept_node 13, compact_metrics 5, sqlite_compact 34, sqlite_compact_crash 15, sqlite_compact_matrix 1, sqlite_compact_units 9, space_reclamation 3, sqlite_group_rows 7 pass. PostgreSQL tests were not rerun for this merge: it changes SQLite files only. Production LOC: 188192 -> 188445 (delta +253) Scope and method: tools/production_loc.py, first parent c1c2f98 against the merged tree; core 40589 -> 40589, engine 90003 -> 90256, control 51073 -> 51073, api 6527 -> 6527; reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This reverts commit 8e7acb3. A review of the merged rebuild commits found that skipping every fsync on a Disposable Store changes what a power loss can do during a compaction: - after the rename, the fresh file's pages may not be on disk, so the whole Store can be lost, not only the compaction; - an operation record that was never synchronized can come back unreadable, and the leftover files then make `store.compact` refuse until someone deletes them by hand; - the synchronization before the rename was also the last point where a deferred write error of the fresh file could stop the install. The spec's behaviour (section 8: synchronize the fresh file, the record and the directory around the rename) stays until the owner decides otherwise. Cost on the 1.1 GB bench shape, release, one sample, load 4.4: install 71.6 ms of a 1.045 s rebuild. Tests: sqlite_compact (34), sqlite_compact_crash (15), sqlite_compact_units (9), compact_metrics (5), space_reclamation (3), sqlite_group_rows (7), kept_objects (7) pass. sqlite_compact_matrix failed once and passed once with no source change between: its AC-49 arm unwraps a Save inside the racing Commit thread (tests/sqlite_compact_matrix.rs:149), which panics when the rename lands during that Save. The test is unchanged on this branch; not fixed here. Production LOC: 188445 -> 188429 (delta -16) Scope: core, engine, control and api product source; counted with tools/production_loc.py on the first parent and the committed tree. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`copy_packs` copies retained packs one 64 MiB window of pack ids at a time. The range sat on the source table only, and SQLite drives that statement from `scratch.live_pack`, so every window walked every retained pack of the Store and seeked the source for each: windows times retained packs seeks. A review measured 200 to 250 ms per window at 800,000 retained packs against 0.1 ms with the range inside the subquery. The range now bounds the `live_pack` scan itself, so a window touches only its own packs and the copy is linear in retained packs. 1.1 GB bench shape, release, one sample, load 4.4: rebuild 1.045 s (mark 71.7 ms, build 791.6 ms, verify 97.1 ms, install 71.6 ms). That Store has too few windows to show the difference; the count argument above is the evidence. Tests: as the parent commit (same build), including the sqlite_compact_matrix note there. Production LOC: 188429 -> 188429 (delta 0) Scope: core, engine, control and api product source; counted with tools/production_loc.py on the first parent and the committed tree. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The mark loads every `object_location` row once. The query asked for pack order, which no index supplies, so each mark made the engine sort the whole table (`USE TEMP B-TREE FOR ORDER BY` on SQLite, a sort node on PostgreSQL). The order bought nothing: `object_id` is the table's key, so an object has one row and there is no winner to pick, and both maps the rows fill are keyed. The query now reads the table in its own key order on both backends. gc_profile on the kept 1.1 GB bench Store (8,913 objects, 8,816 packs), release, warm second mark, one sample, load 6.7: engine steps of a mark 217,406 -> 128,273 sorts 6 -> 5 the statement 3.87 ms -> 1.88 ms Tests: kept_objects (7), kept_node (13), sqlite_compact (34), sqlite_compact_crash (15), compact_metrics (5), space_reclamation (3), postgresql_gc_mark (4), postgresql_compact (30), postgresql_statement_schema (2) pass; Clippy and fmt clean. Production LOC: 188,429 -> 188,428 (delta -1) core 40,589 -> 40,589 (0) engine 90,240 -> 90,239 (-1) layerfs-persistence -1 control 51,073 -> 51,073 (0) api 6,527 -> 6,527 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd19) Method: tools/production_loc.py --detail on parent cb0a97b and the staged tree Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The mark asks each pack for its first bytes: the directory and, for the
common one-record pack, the record header that names its delta base. It
asked for 1,024 bytes. SQLite keeps only the first part of a large row
on the row's own page (at least 489 bytes of a row too large for a
4,096-byte page) and the rest in a chain of further pages, so most of
those reads went one page past the row: a second page load per pack,
and a random read on a cold cache.
The first read is now 432 bytes. The columns before the body take at
most 49 bytes, so up to 440 bytes of body are always on the page the
row lookup has already loaded. A pack with a wider directory is read
again, as before.
gc_profile on the kept 1.1 GB bench Store (8,913 objects, 8,816 packs,
every pack at the 489-byte minimum), release, one run of three marks
per size:
first read time in pack reads warm mark wall 1-min load
1,024 bytes 8.2 to 8.6 ms 38.8 to 40.8 ms 4.9
448 bytes 8.9 to 9.4 ms 39.9 to 43.9 ms 6.8
440 bytes 0.2 ms 28.9 to 34.7 ms 6.5
432 bytes 0.2 ms 32.7 to 40.4 ms 9.2
Read calls are unchanged at 9,104; bytes read fall from 8,972,342 to
3,857,332. The wall times were taken under different load, so the
pack-read time is the comparison to rely on.
Tests: kept_objects (7), kept_node (13), sqlite_compact (34),
sqlite_compact_crash (15), sqlite_group_rows (7), compact_metrics (5),
space_reclamation (3), postgresql_gc_mark (4), postgresql_compact (30)
pass; Clippy and fmt clean.
Production LOC: 188,428 -> 188,428 (delta 0)
core 40,589 -> 40,589 (0)
engine 90,239 -> 90,239 (0)
control 51,073 -> 51,073 (0)
api 6,527 -> 6,527 (0)
reference absent (retired R9c; last counted 65,417 at tree 498dd19)
Method: tools/production_loc.py --detail on parent c15826e and the staged tree
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A CPU sample of warm marks (after the pack reads became cheap) put more than half of a mark in map bookkeeping: the walk and the kept-waste pass ask ordered maps keyed by 32-byte identity about every object several times, and each question pays a 32-byte comparison per tree level. Those maps are now hash maps (`history/objects/ids.rs`). The standard hasher was tried first and cost as much as the comparisons it replaced, because it runs its permutation over all 32 bytes. An identity is a BLAKE3 digest, so `IdState` takes its first eight bytes and mixes them with a mask and multiplier drawn per map from the standard random source. The key is there because identities come from stored content: a writer could otherwise choose content whose identities share table slots. Results do not depend on it: the two lists a mark returns in identity order (`KeptObjects::objects`, the delta base references) are sorted after collection, and the walk still orders each wave by locator. The per-pack maps (`read_bytes`, `layouts`) are hashed too. The pack inventory keeps its ordered maps. `MarkGroups` moves to `history/objects/plan.rs`, unchanged, so `objects.rs` stays under the 999-line ceiling (958 -> 895 lines). 100 warm marks of the kept 1.1 GB bench Store (8,913 objects, 8,816 packs), release, two runs per build, interleaved, 1-min load 7.6 to 7.8: ordered maps user 3.83 s, 3.92 s mean mark 43.1 ms, 42.0 ms standard hasher user 3.50 s, 3.18 s (load 9.0 to 9.2; not kept) this commit user 2.82 s, 2.86 s mean mark 30.7 ms, 31.3 ms Tests: kept_objects (7), kept_node (13), sqlite_compact (34), sqlite_compact_crash (15), sqlite_group_rows (7), compact_metrics (5), space_reclamation (3), postgresql_gc_mark (4), postgresql_compact (30) pass; Clippy, fmt and the product boundary guard clean. Production LOC: 188,428 -> 188,488 (delta +60) core 40,589 -> 40,589 (0) engine 90,239 -> 90,299 (+60) layerfs-persistence +60 control 51,073 -> 51,073 (0) api 6,527 -> 6,527 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd19) Method: tools/production_loc.py --detail on parent c1d48b6 and the staged tree Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…t (layerfs-project#41)
Neither the Store's nor the scratch `object_location` has a `pack_id` index,
so every Phase A batch scanned the whole source table and the whole visited
set three times and the growing scratch table four times (EXPLAIN ANALYZE on
PostgreSQL 17.11: a batch holding 16,161 of 290,561 rows still read all
290,561 source rows and all 258,561 visited rows per statement). The copy cost
batches times objects.
Phase A now makes one scan of the source into a session temporary table of the
visited location rows (`pg_temp.gc_copy`), indexed on `pack_id`. Each batch
reads its own range through that index and takes its row counts from the two
INSERTs. The copy set must hold exactly one row per visited object, and the
rows copied by all batches must add up to it, so a visited object in a pack
outside the live list is still refused. The final scratch check
(count = count(DISTINCT object_id) = |visited|, pack count) is unchanged and is
now one statement. The retained canonical total is read from the copy set
instead of a second semi-join scan of the source.
Kept as before: one transaction per batch, the 1 GiB batch bound, and the
own-range DELETE of spec section 8 step 3. That DELETE still scans the scratch
`object_location` once per batch (it has no `pack_id` index and must not gain
one, because it becomes the Store's table); it is the only per-batch scan left.
Build phase, one sample per case, local PostgreSQL 17.11 in Docker,
1-minute load 4.0-4.5, fixture of 1 KiB whole files (4 projects):
72,641 location rows, 73.9 MB of packs
1 batch (product bound) 349.3 -> 326.2 ms 45 -> 38 statements 92 -> 80 round trips
4 batches (16 MiB bound) 408.9 -> 336.4 ms 75 -> 53 statements 155 -> 113 round trips
290,561 location rows, 295.5 MB of packs
1 batch (product bound) 1886.1 -> 1134.3 ms 45 -> 38 statements 92 -> 80 round trips
16 batches (16 MiB bound) 2654.9 -> 1240.0 ms 195 -> 113 statements 407 -> 245 round trips
Per additional batch: 10 -> 5 statements, 21 -> 11 round trips, and at 290,561
rows +51.3 -> +7.0 ms. Four times the rows and batches cost 6.5x before and
3.7x after.
Limits of these numbers. The multi-batch rows were taken with the batch bound
lowered to 16 MiB in an uncommitted build, because no fixture I may build has
more than one 1 GiB batch; no Store with several real batches was measured.
For one batch the wall time is not a demonstrated gain: a second baseline
sample of the 290,561-row case read 1338.3 ms, so the baseline's own spread is
as large as the difference. The statement and round-trip counts are exact.
Cost: the copy set is temporary space the run holds until its session ends,
about 92 to 97 bytes per retained object row (6.3 MB for 64,641 retained rows
and 23.8 MB for 258,561). It is added to the reported peak temp.
Test: `a_duplicate_retained_location_row_refuses_the_build` pins that a second
location row for a retained object is an `Integrity` refusal before the swap,
with the Store left at G and the op row naming the leftover scratch schema. It
passes on the parent too (there the refusal is "compaction batch count", here
"compaction copy set"), so it characterizes kept behaviour; it is not a
failing-on-parent regression test. No test deleted.
Checks before the rebase onto c15826e (this change on c1c2f98): the ten PostgreSQL suites (postgresql_compact 31,
_compact_fixtures 7, _compact_units 8, _gc_mark 4, _gc_fence 19,
_identical_ids 3, _statement_schema 2, _commit_round_trips 1, _acquisition 4,
_acquisition_compaction 4) pass; cargo fmt --check, clippy --all-targets
-D warnings for layerfs-persistence and tools/check_product_boundary.py pass.
Production LOC: 188,428 -> 188,405 (delta -23)
core 40,589 -> 40,589 (0)
engine 90,239 -> 90,216 (-23) layerfs-persistence -23
control 51,073 -> 51,073 (0)
api 6,527 -> 6,527 (0)
reference absent (retired R9c; last counted 65,417 at tree 498dd19)
Method: tools/production_loc.py --detail on git archive of parent c1c2f98
and of this commit, recounted after the rebase onto c1c2f98. The
measurements above were taken before the rebases, on parent d19d98f.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…yerfs-project#41) The mark asks for the first 1,024 bytes of up to 256 packs per call (`Mechanics::pack_prefixes`, 962acd4). PostgreSQL had only the default, which sends one statement per pack. `objects::read::prefixes` answers the call with one statement: the ids as an array, joined to `pack` and returned in the array's order with each pack's declared length. A pack that is absent is `Missing`, and a head whose size disagrees with the declared length is `Malformed`, as for a single prefix. Mark walk, one sample per case, local PostgreSQL 17.11 in Docker, fixtures of 1 KiB whole files (99 per Layer, 4 projects): 72,641 location rows, 1,280 live packs (640 payload) load 5.26 -> 3.53 statements 3,850 -> 3,212 round trips 3,865 -> 3,228 pack bytes received 131,893,084 -> 131,893,084 wall 2,064.8 -> 1,846.7 ms 290,561 location rows, 5,120 live packs (2,560 payload) load 5.00 -> 3.53 statements 15,370 -> 12,820 round trips 15,385 -> 12,836 pack bytes received 527,732,824 -> 527,732,824 wall 8,430.4 -> 7,779.5 ms The counts are exact: one head statement per payload pack becomes one per 256 packs (640 -> 3 and 2,560 -> 10). The wall figures were taken at a lower host load than their baselines and are one sample each, so they are not evidence of a speed-up by themselves. The bytes do not move: the heads are 1 KiB each, and what dominates is the 256 KiB head a scoped read fetches with every descriptor. These figures were taken on parent 962acd4, before the mark moved its payload ranges to `pack_ranges` (c1c2f98). Test: `a_mark_reaches_every_record_of_payload_packs_larger_than_its_head` marks a Store of four trees whose payload packs (99 records, about 102 KiB, several groups) are far larger than a head, and requires every saved object and nothing else. It passes on the parent as well; it fails when the heads come back in another order (checked by reversing the statement's ORDER BY: the closure is refused as `Integrity`). It asserts no statement count, because the mark's per-pack statements are set by `history/objects/edges.rs`, which is still changing. No test deleted. Checks before the rebase onto c15826e (this change on c1c2f98): the ten PostgreSQL suites pass (postgresql_compact 31, _compact_fixtures 7, _compact_units 8, _gc_mark 5, _gc_fence 19, _identical_ids 3, _statement_schema 2, _commit_round_trips 1, _acquisition 4, _acquisition_compaction 4); cargo fmt --check, clippy --all-targets -D warnings for layerfs-persistence and tools/check_product_boundary.py pass. Production LOC: 188,405 -> 188,451 (delta +46) core 40,589 -> 40,589 (0) engine 90,216 -> 90,262 (+46) layerfs-persistence +46 control 51,073 -> 51,073 (0) api 6,527 -> 6,527 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd19) Method: tools/production_loc.py --detail on git archive of the parent and of this commit, recounted after the rebase onto c1c2f98 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ayerfs-project#41) The mark's payload path asks for the group directories and record headers of up to 256 packs per call (`Mechanics::pack_ranges`, c1c2f98). PostgreSQL had only the default, one scoped acquisition per pack, and a scoped acquisition here fetches the first 256 KiB of the body with its descriptor, so every payload pack of the fixtures below still crossed the wire whole. `objects::read::ranges` answers one call with one statement: the pack ids, offsets and lengths as three arrays, joined to `pack`, one `substring` per range, returned in request order. The body is stored out of line and uncompressed, so each `substring` fetches only the chunks it covers. A pack's ranges must ascend without overlap (`Malformed` otherwise); an absent pack is `Missing`; a range that reaches past its body comes back short and is `Malformed`. Mark walk, one sample per case, local PostgreSQL 17.11 in Docker, fixtures of 1 KiB whole files (99 per Layer, 4 projects), against the parent commit: 72,641 location rows, 1,280 live packs (640 payload) load 5.54 -> 4.14 statements 3,262 -> 1,996 round trips 3,278 -> 2,013 pack bytes received 136,989,034 -> 4,611,164 wall 2,518.0 -> 1,529.4 ms 290,561 location rows, 5,120 live packs (2,560 payload) load 5.25 -> 4.20 statements 13,020 -> 7,952 round trips 13,036 -> 7,969 pack bytes received 548,116,624 -> 18,605,144 wall 10,746.3 -> 6,721.3 ms The counts are exact. The wall figures are one sample each and the baseline ran at a higher host load, so the size of the speed-up is not established; that both statements and bytes fall is. What is left of the walk is per metadata pack, three statements each (small fixture, 640 metadata packs, from the server's statement log): 640 scoped reads through the decoding reader, 640 `inode_value_block` reads and 639 single-pack prefix reads of the kept-waste pass, against 64 range statements and 3 head statements for all payload packs. Those callers are in `history/objects.rs`, outside this change. Test: `a_mark_reaches_every_record_of_payload_packs_larger_than_its_head` now runs through this statement. It fails when the ranges come back in another order (checked by reversing the ORDER BY: the closure is refused as `Integrity`). The missing-pack and short-range refusals are not reachable through a public call and have no test. No test deleted. Checks before the rebase onto c15826e (this change on c1c2f98): the ten PostgreSQL suites pass (postgresql_compact 31, _compact_fixtures 7, _compact_units 8, _gc_mark 5, _gc_fence 19, _identical_ids 3, _statement_schema 2, _commit_round_trips 1, _acquisition 4, _acquisition_compaction 4); cargo fmt --check, clippy --all-targets -D warnings for layerfs-persistence and tools/check_product_boundary.py pass. Production LOC: 188,451 -> 188,518 (delta +67) core 40,589 -> 40,589 (0) engine 90,262 -> 90,329 (+67) layerfs-persistence +67 control 51,073 -> 51,073 (0) api 6,527 -> 6,527 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd19) Method: tools/production_loc.py --detail on git archive of the parent and git checkout-index of the staged tree Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ck (layerfs-project#41)
The swap transaction took ACCESS EXCLUSIVE on every Store table and then
scanned the moved object table three times (count, count DISTINCT, canonical
sum) and the pack table three times, and read the live-heads digest twice.
Every reader and writer of the Store waited for those scans.
The row counts and sums of spec section 8 step 9 are now read in one statement
from the scratch tables, inside the same transaction and before the Store-wide
lock. The two scratch tables are locked first (ACCESS EXCLUSIVE, waiting), so
the rows counted are the rows SET SCHEMA moves. Both sides of each comparison
come from the mark: the mark's object and pack counts, and the mark's totals
minus the dead totals, which the tuple re-check under the lock then pins. The
digest is read once under the lock; the lock is held from that read to COMMIT.
`store_totals.logical` is no longer re-read: the tuple re-check has just
compared it, so the history rows are summed against the mark's value. The two
`gc_op` reads are one statement.
Still under the lock: the tuple and digest re-check, the ACL read and replay,
the drop and move, the table fingerprint, the logical sum over the history
rows, the trigger/function contract check, the relation size, both updates,
the scratch drop and the standby check.
Measured on a local PostgreSQL 17 (Docker, loopback), one sample each,
synchronous_commit=local, lock hold timed in the client from the LOCK TABLE
statement to the end of COMMIT:
72,641 object rows, 1,360 packs (1-minute load 3.26 -> 3.04)
lock hold 57.4 ms -> 28.5 ms
statements under it 35 -> 25
swap phase 38 statements, 64 round trips, 64.8 ms
-> 30 statements, 49 round trips, 58.5 ms
290,561 object rows, 5,440 packs (1-minute load 3.08 -> 3.20)
lock hold 154.4 ms -> 49.7 ms
statements under it 35 -> 25
swap phase 38 statements, 64 round trips, 160.3 ms
-> 30 statements, 49 round trips, 150.9 ms
At the larger size the COMMIT itself is 17 ms of the remaining hold. The
count DISTINCT sort (77 ms at 290,561 rows) still runs, outside the lock.
Changed behaviour to know about: a count or sum mismatch on the scratch tables
is now refused as Integrity before the tuple is re-read, so it is reported
even when the Store also moved after the mark (before, Conflict came first).
The scratch-table lock waits without NOWAIT; before this change the same wait
happened at ALTER TABLE ... SET SCHEMA while the Store-wide lock was held.
No test reaches the count or sum refusals (none did before); the phases run
back to back inside one call and there is no hook between build and swap.
No test deleted.
Checks before the rebase onto c15826e (this change on c1c2f98): the ten PostgreSQL suites pass (postgresql_compact 31,
_compact_fixtures 7, _compact_units 8, _gc_mark 5, _gc_fence 19,
_identical_ids 3, _statement_schema 2, _commit_round_trips 1, _acquisition 4,
_acquisition_compaction 4); cargo fmt --check, clippy --all-targets
-D warnings for layerfs-persistence and tools/check_product_boundary.py pass.
Production LOC: 188,518 -> 188,502 (delta -16)
core 40,589 -> 40,589 (0)
engine 90,329 -> 90,313 (-16) layerfs-persistence -16
control 51,073 -> 51,073 (0)
api 6,527 -> 6,527 (0)
reference absent (retired R9c; last counted 65,417 at tree 498dd19)
Method: tools/production_loc.py --detail on git archive of the parent and
git checkout-index of the staged tree
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…rfs-project#41) The mark's history closure (`kept_node_mark`) ran the text both engines share. Its recursive step joins each round's new nodes to a view of every history edge, and PostgreSQL plans that as a hash of the new nodes probed by a scan of all edges. A Commit that only its parent chain reaches arrives one round at a time, so a Branch with n unpublished Commits cost n scans of the edge set: 4,000 rounds read 5.4 million buffers (EXPLAIN ANALYZE, BUFFERS). PostgreSQL now runs its own text, registered in `backend/postgresql/history/statements.rs`. The roots and the final insert are unchanged. The recursive step reads each new node's own row by primary key and emits its references from a VALUES list. Where the shared text joined to `commit` or `layer` to learn which kind a reference names, this text emits both kinds; a name that is no row of that kind has no row to follow and the final insert's join drops it. The shared file is not edited and SQLite still runs it. Measured on a local PostgreSQL 17 (Docker, loopback), one sample each, the closure statement's server time inside a `store.compact` mark: one Branch, 999 unpublished Commits 584.9 ms -> 6.5 ms (load 5.72, 6.28) one Branch, 3,999 unpublished Commits 9,386.3 ms -> 20.8 ms (load 5.72, 6.17) mark walk of the larger Store 14,752.2 ms -> 4,486.6 ms Four times the chain cost 16 times before and 3.2 times now. It is slower where every Commit is published as a Layer, because every Layer of a live Stack is a root and the old text finished in two rounds: 2 Stacks x 1,000 Layers, 2,001 kept nodes 19.9 ms -> 258.9 ms 2 Stacks x 2,400 Layers, 4,801 kept nodes 132.4 ms -> 251.1 ms About 240 ms of each new figure is the server's JIT compiler, not the statement's work: the planner's estimate for the new text passes `jit_above_cost` at about 550 root nodes, and EXPLAIN ANALYZE shows 241 ms of JIT against 10 ms of execution. The shared text passes the same threshold on larger Stores (45 ms of its 132.4 ms above). The Store session does not set `jit`; that is in `session/profile.rs`, outside this change. Test: `the_postgresql_closure_keeps_the_nodes_the_shared_statement_keeps` builds a live project (a published Commit, a fork, a merge whose second parent is a Layer, 30 unpublished Commits) and a deleted project, and requires the same `kept_node` rows from both texts. It fails when the parent-Commit edge is removed from the new text. It does not fail when the second-parent, fork or source-Branch edge is removed: every Layer of a live Stack and every live Branch is already a root in this fixture. No fixture here has a `merged_from_id` or `displaced_id` reference. The two texts also gave the same node set (count and md5 of the sorted ids) on the four measurement Stores. No test deleted. Checks before the rebase onto c15826e (this change on c1c2f98): the ten PostgreSQL suites pass (postgresql_compact 31, _compact_fixtures 7, _compact_units 8, _gc_mark 6, _gc_fence 19, _identical_ids 3, _statement_schema 2, _commit_round_trips 1, _acquisition 4, _acquisition_compaction 4); cargo fmt --check, clippy --all-targets -D warnings for layerfs-persistence and tools/check_product_boundary.py pass. Production LOC: 188,502 -> 188,569 (delta +67) core 40,589 -> 40,589 (0) engine 90,313 -> 90,380 (+67) layerfs-persistence +67 control 51,073 -> 51,073 (0) api 6,527 -> 6,527 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd19) Method: tools/production_loc.py --detail on git archive of the parent and git checkout-index of the staged tree Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The mark's dead-set insert selected `('digest', fd.root)` for every
`file_digest` row whose root is unvisited. `file_digest` is keyed by
`(digest, length)`, so two rows can name one root, and `gc_dead` is keyed by
`(kind, id)`: the second copy of the root broke the key and `store.compact`
failed in the mark with `Integrity("persistence")`. Phase C already expects
such rows (F50-2 deletes by root "even when a different digest names it").
The digest arm now selects each root once. Every other arm of the insert
selects its own table's key and cannot repeat.
Test: `two_digest_rows_of_one_dead_root_are_both_collected` records two
digest rows for one dead root through `record_file_digests`, compacts and
requires both rows gone and the op settled. Before this change it failed at
the `compact` call with `Integrity("persistence")`. No test deleted.
Checks before the rebase onto c15826e (this change on c1c2f98): the ten PostgreSQL suites pass (postgresql_compact 31,
_compact_fixtures 8, _compact_units 8, _gc_mark 6, _gc_fence 19,
_identical_ids 3, _statement_schema 2, _commit_round_trips 1, _acquisition 4,
_acquisition_compaction 4); cargo fmt --check, clippy --all-targets
-D warnings for layerfs-persistence and tools/check_product_boundary.py pass.
Production LOC: 188,569 -> 188,569 (delta 0)
core 40,589 -> 40,589 (0)
engine 90,380 -> 90,380 (0)
control 51,073 -> 51,073 (0)
api 6,527 -> 6,527 (0)
reference absent (retired R9c; last counted 65,417 at tree 498dd19)
Method: tools/production_loc.py --detail on git archive of the parent and
git checkout-index of the staged tree
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…-project#41)
Phase C deleted `file_digest` rows whose root is in the dead set 1,000 at a
time. Each batch joined `gc_dead` to `file_digest` on `root`, which has no
index, once to pick the batch and once to delete it: two scans of the whole
table per 1,000 rows. Four times the dead rows cost 17.8 times the server
time (EXPLAIN shows a hash join over a sequential scan of `file_digest` in
both halves).
One scan now lists the keys of the rows to delete in a numbered session
table. Each batch deletes its own range of that list by `(digest, length)`,
the table's primary key, and checks the root against `gc_dead` again, so
membership in the persisted dead set is still the filter. Batches are still
1,000 rows, one transaction each. A crash loses the session table; the
resumed call lists what is left.
Measured on a local PostgreSQL 17 (Docker, loopback), one sample each,
synchronous_commit=local. Server time of the digest statements, and the
whole cleanup phase:
dead rows digest statements cleanup phase load
50,000 710.0 ms -> 201.2 ms 1,078.3 -> 483.7 ms 4.42, 7.29
200,000 12,622.2 ms -> 2,443.5 ms 14,284.4 -> 3,984.1 ms 4.42, 7.29
800,000 not run -> 7,965.7 ms not run -> 11,468.6 ms 6.68
Statements and round trips are nearly unchanged (200,000 rows: 227 -> 229
statements, 672 -> 677 round trips); the gain is in what each batch reads.
Per batch: 3.4 ms at 50,000 rows, 11.4 ms at 200,000, 9.3 ms at 800,000.
Up to 200,000 rows the planner still answers a batch with one scan of
`file_digest` (cheaper by its own cost model than 1,000 index probes), so
the cost per batch grows until the table is large enough that it switches
to the primary key; from there it is level. The old text at 800,000 rows
was not run (its measured slope puts it far past the two-minute limit).
What is left of the phase at 200,000 rows: 1,175 ms in 211 COMMITs, one per
batch. Larger batches would cut that; the batch size is unchanged here.
The session table holds one row per dead digest row (about 70 bytes plus
its index) until the last batch. It is not counted in `peakTempBytes`,
which the swap records before Phase C starts.
A `file_digest` row with a dead root that is inserted after the list is
made is now left in place; the old loop would have deleted it in a later
batch. Such a row can only come from a save at the new generation.
Tests: `dead_digest_rows_beyond_one_batch_are_all_collected` records 2,500
dead digest rows and one retained row, compacts and requires only the
retained row left; it fails when a batch drops its last row (checked with
`b.n < $2`). The existing F50-2 case (a dead-root row inserted before a
resumed cleanup) passes unchanged. No test deleted.
Checks before the rebase onto c15826e (this change on c1c2f98): the ten PostgreSQL suites pass (postgresql_compact 31,
_compact_fixtures 9, _compact_units 8, _gc_mark 6, _gc_fence 19,
_identical_ids 3, _statement_schema 2, _commit_round_trips 1, _acquisition 4,
_acquisition_compaction 4); cargo fmt --check, clippy --all-targets
-D warnings for layerfs-persistence and tools/check_product_boundary.py pass.
The timings were taken on this cleanup.rs one commit earlier (before the
dead-root dedup); the file is byte-identical.
Production LOC: 188,569 -> 188,590 (delta +21)
core 40,589 -> 40,589 (0)
engine 90,380 -> 90,401 (+21) layerfs-persistence +21
control 51,073 -> 51,073 (0)
api 6,527 -> 6,527 (0)
reference absent (retired R9c; last counted 65,417 at tree 498dd19)
Method: tools/production_loc.py --detail on git archive of the parent and
git checkout-index of the staged tree
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…erfs-project#41) `load_visited` first copied every visited object id into a `Vec<Vec<u8>>` (one heap allocation per object) and the live pack ids into a second `Vec`, then streamed those to COPY. The copy lived until the load finished, beside the mark's own set. Each COPY row is now written straight from the mark's ids. The statements, their order and the rows sent are unchanged. Not measured: the saving is one 24-byte `Vec` header and one 32-byte heap block per visited object (arithmetic, before allocator overhead), and the fixtures at hand are too small to show it in peak memory. The change is confirmed by reading and by the suites below, not by a memory measurement. No test added or deleted. Checks before the rebase onto c15826e (this change on c1c2f98): the ten PostgreSQL suites pass (postgresql_compact 31, _compact_fixtures 9, _compact_units 8, _gc_mark 6, _gc_fence 19, _identical_ids 3, _statement_schema 2, _commit_round_trips 1, _acquisition 4, _acquisition_compaction 4); cargo fmt --check, clippy --all-targets -D warnings for layerfs-persistence and tools/check_product_boundary.py pass. Production LOC: 188,590 -> 188,584 (delta -6) core 40,589 -> 40,589 (0) engine 90,401 -> 90,395 (-6) layerfs-persistence -6 control 51,073 -> 51,073 (0) api 6,527 -> 6,527 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd19) Method: tools/production_loc.py --detail on git archive of the parent and git checkout-index of the staged tree Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The kept-waste pass reads the directory prefix of every retained pack the walk did not lay out itself: the metadata packs, which the walk reaches through the decoding reader. It asked for them one pack per call: on SQLite a length statement and a BLOB handle each, on PostgreSQL one statement and round trip each (2,399 of the 7,219 statements of a mark over 5,440 packs in the PostgreSQL agent's count). `edges::Prefixes` hands the same prefixes out in pack order and reads them through `Mechanics::pack_prefixes`, 256 packs per call: one BLOB handle reused on SQLite, one statement on PostgreSQL. gc_profile on the kept 1.1 GB bench Store (8,913 objects, 8,816 packs, 96 metadata packs), release, warm second mark, one sample, load 3.8: statements of a mark 298 -> 203 BLOB handles opened 229 -> 135 wall 30.2 ms (31 ms mean before, under load 7.6) Tests: kept_objects (7), kept_node (13), sqlite_compact (34), sqlite_compact_crash (15), sqlite_group_rows (7), compact_metrics (5), space_reclamation (3), postgresql_gc_mark (4), postgresql_compact (30) pass; Clippy, fmt and the product boundary guard clean. Production LOC: 188,488 -> 188,527 (delta +39) core 40,589 -> 40,589 (0) engine 90,299 -> 90,338 (+39) layerfs-persistence +39 control 51,073 -> 51,073 (0) api 6,527 -> 6,527 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd19) Method: tools/production_loc.py --detail on parent d600429 and the staged tree Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…act (layerfs-project#41) Eight commits from the PostgreSQL rebuild agent, each with its own measurements (PostgreSQL 17 on loopback, one sample per case): 97169f0 Phase A reads each batch from an indexed copy set: per extra batch 10 -> 5 statements, 21 -> 11 round trips 11242b2 a wave's pack heads in one statement 51944d2 a wave's pack ranges in one statement: mark walk 13,020 -> 7,952 statements, 548.1 MB -> 18.6 MB received, 10,746 -> 6,721 ms at 290,561 rows 0bb8989 row checks before the Store-wide lock: lock hold 154.4 -> 49.7 ms at 290,561 rows dc6734b PostgreSQL-only history closure by primary key: 9,386 -> 20.8 ms for 3,999 unpublished Commits 305007f fix: two digest rows naming one dead root made store.compact fail with Integrity("persistence") 86f5a4a dead digest rows deleted by primary key: 12,622 -> 2,444 ms for 200,000 dead rows 9dcecc8 visited set loaded without a second copy An independent read-only review of the range found no case that keeps too little and found the new closure statement equivalent to the shared one on all 16 edge kinds. Open items it and the author raised, none fixed here: - the closure equivalence test exercises 2 of the 16 edges, so the PostgreSQL text can drift from the shared one unnoticed; - the new closure is slower when every Commit is published (about 20 -> 250 ms at 2,000 Layers; the server's JIT compiler); - swap row counts are now taken by scratch-table name before the move, and a scratch mismatch is reported as Integrity before a Conflict; - `pg_temp.gc_copy` lives until the session ends; - `peakTempBytes` now includes the copy set; - spec section 8 and one module comment do not describe the new order. Checked on this merged tree (the agent's own runs were before its rebase): postgresql_compact (31), postgresql_compact_fixtures (9), postgresql_compact_units (8), postgresql_gc_mark (6), postgresql_gc_fence (19), postgresql_identical_ids (3), postgresql_statement_schema (2), postgresql_commit_round_trips (1), postgresql_acquisition (4), postgresql_acquisition_compaction (4), kept_objects (7), kept_node (13), sqlite_compact (34), sqlite_compact_crash (15), sqlite_compact_units (9), sqlite_group_rows (7), compact_metrics (5), space_reclamation (3) pass; Clippy, fmt and the product boundary guard clean. No test deleted. Production LOC: 188,527 -> 188,683 (delta +156), first parent 9da6038 core 40,589 -> 40,589 (0) engine 90,338 -> 90,494 (+156) layerfs-persistence +156 control 51,073 -> 51,073 (0) api 6,527 -> 6,527 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd19) Method: tools/production_loc.py --detail on parent 9da6038 and the staged tree Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The review of the PostgreSQL rebuild commits found two texts that no longer described the code: - Spec section 8, Phase B: the swap now locks the two scratch tables and takes the row counts and sums before the Store-wide lock (0bb8989). An "as built" note says so, including that a scratch mismatch is reported before a Conflict. - `backend/postgresql/history/statements.rs`: the module comment said only statements with SQLite-only syntax are replaced. The history closure is replaced too; the comment now says why and that both files must follow the same edges. No behaviour change. Production LOC: 188,683 -> 188,683 (delta 0) core 40,589 -> 40,589 (0) engine 90,494 -> 90,494 (0) control 51,073 -> 51,073 (0) api 6,527 -> 6,527 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd19) Method: tools/production_loc.py --detail on parent f22faa7 and the staged tree Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`gc_bench` and `gc_profile` are SQLite only, so PostgreSQL GC had no
committed way to repeat a measurement. `pg_gc_profile` is the probe the
PostgreSQL rebuild agent measured its commits with, cleaned up:
pg_gc_profile build DB a Store in its own database
pg_gc_profile mark DB WORK clone, run the mark, count its SQL work
pg_gc_profile run DB WORK clone, run store.compact, count it
pg_gc_profile seed|sizes|drop
It prints wall time, the compaction's phases and the session's
statements, round trips, rows and bound bytes. A diagnostic: no receipt
and no claim. Documented in tools/ci/gc-bench.md.
First use, PostgreSQL 17 in Docker on loopback, release, one sample
each, on a Store of 72,641 objects in 1,360 packs (4 Stacks x 160
Layers x 99 files of 1 KiB, 8,000 dead objects):
6bca861 58aa062 (parent)
mark 3,875.3 ms 1,727.2 ms
mark statements 4,489 1,359
store.compact 5,574.0 ms 2,936.0 ms
compact statements 4,627 1,484
1-min load 4.2 to 4.3 7.3
An example and a doc only: no product source changes.
Production LOC: 188,683 -> 188,683 (delta 0)
core 40,589 -> 40,589 (0)
engine 90,494 -> 90,494 (0)
control 51,073 -> 51,073 (0)
api 6,527 -> 6,527 (0)
reference absent (retired R9c; last counted 65,417 at tree 498dd19)
Method: tools/production_loc.py --detail on parent 58aa062 and the staged tree
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
After the earlier commits a PostgreSQL mark still sent 1,359 statements
over a Store of 1,360 packs. The server log split them three ways:
- the history closure, 415 ms. EXPLAIN ANALYZE on the same Store: 409 ms
with the server's JIT compiler on, 10.5 ms with it off.
- one scoped pack read per metadata pack the walk decodes (640);
- one inode value read per inode leaf (640).
The last two are cheap on the server (11 ms and 123 ms in all) but each
is a round trip.
Three changes, the first PostgreSQL only and the other two behind new
`Mechanics` calls whose defaults change nothing, so SQLite is as before:
1. The mark's snapshot transaction sets `jit` off (`snapshot_begin`).
2. `expect_scoped_packs`: before acquiring a pack the walk announces the
next 128 metadata packs of its wave. PostgreSQL reads, in one
statement, the rows `read_scoped_pack` would fetch for those of up to
32 KiB and keeps them on the transaction; each scoped read then uses
its row once. A longer pack, or one not announced, is read alone as
before. The rows are the same rows, so the plan, the authentication
and `mark_bytes` are unchanged.
3. `inode_values_ahead`: PostgreSQL returns every value of the blocks
holding the asked ordinals and up to 4,096 ordinals past them; the
walk keeps at most 65,536 values and asks again only for a leaf it
cannot answer.
pg_gc_profile, PostgreSQL 17 in Docker on loopback, release, one sample
each, the Store of the parent commit's message (72,641 objects, 1,360
packs):
6bca861 parent this commit
mark 3,875.3 ms 1,727.2 ms 577.4 ms
mark statements 4,489 1,359 100
store.compact 5,574.0 ms 2,936.0 ms 1,483.3 ms
compact statements 4,627 1,484 225
1-min load 4.2 to 4.3 7.3 5.3
Objects kept (64,641), packs kept (1,280), mark_bytes (3,818,542) and
the receipt (dead bound 8,235,840, shrink 10,149,888) are identical in
all three.
New test `a_mark_sends_the_same_statements_for_twice_the_layers`
(postgresql_gc_mark): 8 and 16 Layers must cost the same statements and
reach every saved object. On the parent's source it fails with
[28, 44]; here it passes.
Not done: the same closure statement also runs in project removal's
write transaction, which keeps JIT on.
Tests: postgresql_compact (31), postgresql_compact_fixtures (9),
postgresql_compact_units (8), postgresql_gc_mark (7),
postgresql_gc_fence (19), postgresql_identical_ids (3),
postgresql_statement_schema (2), postgresql_commit_round_trips (1),
postgresql_acquisition (4), postgresql_acquisition_compaction (4),
postgresql_narrow_locations (3), postgresql_position_columns (4),
kept_objects (7), kept_node (13), sqlite_compact (34),
sqlite_compact_crash (15), sqlite_compact_units (9),
sqlite_group_rows (7), compact_metrics (5), space_reclamation (3) pass;
Clippy, fmt and the product boundary guard clean. No test deleted.
Production LOC: 188,683 -> 188,813 (delta +130)
core 40,589 -> 40,589 (0)
engine 90,494 -> 90,624 (+130) layerfs-persistence +130
control 51,073 -> 51,073 (0)
api 6,527 -> 6,527 (0)
reference absent (retired R9c; last counted 65,417 at tree 498dd19)
Method: tools/production_loc.py --detail on parent e180a41 and the staged tree
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
After the mark, the rebuild loads the kept object ids into a session temp table. It used a binary COPY and handed the driver one row at a time: 257.9 ms on the server's clock for 64,641 ids, about four microseconds a row, nearly all of it waiting for the next row. The ids now go as `unnest($1::bytea[])`, 8,192 per statement (about 300 KiB), and the pack ids as `unnest($1::bigint[])`, 32,768 per statement. The ids are already ascending, so the key's index is built by appending. Server log (log_min_duration_statement=0), the 72,641-object Store of the two commits before, one sample: visited ids load 257.9 ms, 1 statement -> 43.6 ms, 8 statements mark phase after the walk (server) 305.8 ms -> 86.9 ms pg_gc_profile, release, one sample each, 1-min load 10.8 (higher than the earlier rows, so read the direction, not the ratio): mark 577.4 ms -> 433.2 ms (100 statements both) store.compact 1,483.3 ms -> 962.0 ms (225 -> 232 statements) The receipt is unchanged (dead bound 8,235,840, shrink 10,149,888). Tests: postgresql_compact (31), postgresql_compact_fixtures (9), postgresql_compact_units (8), postgresql_gc_mark (7), postgresql_gc_fence (19), postgresql_identical_ids (3), postgresql_acquisition_compaction (4) pass; Clippy and fmt clean. Production LOC: 188,813 -> 188,808 (delta -5) core 40,589 -> 40,589 (0) engine 90,624 -> 90,619 (-5) layerfs-persistence -5 control 51,073 -> 51,073 (0) api 6,527 -> 6,527 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd19) Method: tools/production_loc.py --detail on parent ba6cba3 and the staged tree Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…dges The PostgreSQL history closure (dc6734b) is a second text of the shared statement. The review of that commit found its data test reaches 2 of the 16 edge kinds, so an edge added to one file and not the other, or a dropped VALUES row, would go unnoticed, and on that backend a rebuild would then drop history the other keeps. `the_postgresql_closure_follows_exactly_the_shared_edges` compares the two texts: the same roots, the same sixteen (from kind, to kind, column) edges and the same final insert. It needs no server. Checked once by removing `('commit',c.merged_from_id)` from the PostgreSQL text: the test fails; restored, it passes. It compares text, not behaviour: a rewrite of either statement into another shape will need this test rewritten with it. Tests: postgresql_gc_mark (8) pass. Production LOC: 188,808 -> 188,808 (delta 0) core 40,589 -> 40,589 (0) engine 90,619 -> 90,619 (0) control 51,073 -> 51,073 (0) api 6,527 -> 6,527 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd19) Method: tools/production_loc.py --detail on parent ca82dbc and the staged tree; a test-only commit Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ark-header-edges Brings in 3fc66bc (#733: PostgreSQL Stores run on any machine; store.update replaces attach/detach) so the GC speed work is verified on the tree it will merge into. No conflict; one file merged textually (docs/architecture/engine/store/postgresql.md). Production LOC: 188,808 -> 188,960 (delta +152), first parent 032b212 core 40,589 -> 40,589 (0) engine 90,619 -> 90,619 (0) control 51,073 -> 51,230 (+157) api 6,527 -> 6,522 (-5) reference absent (retired R9c; last counted 65,417 at tree 498dd19) Method: tools/production_loc.py --detail on parent 032b212 and the staged tree; the whole delta is #733's Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
After the rename a rebuild lets go of the writer lane and settles: it removes its artifacts, then clears its admission intent and the kept_node rows in the installed Store through a raw connection. That transaction did not take the lane. A writer admitted in that window (a Commit that had been waiting for the rebuild) has no busy handler, so it met the settle's write lock and failed with SQLITE_BUSY. The settle now takes the lane for that one transaction and waits its turn. A lane that cannot be taken keeps the op record for the next run, like a Store that could not be cleared. The order of the settle, the hooks and the leftover rules are unchanged; the abort paths already hold the run's lane. This is the cause of the acceptance matrix's AC-49 arm failing at sqlite_compact_matrix.rs:149/:744 (also at 6bca861, before this branch). With the fix it passed 8 of 8 samples. Test: the_settle_waits_for_a_writer_that_holds_the_lane takes the lane in the settle window and holds it 400 ms; the compaction must return after the release. It fails on the parent's install.rs ("the settle wrote to the Store while another writer held the lane"). The matrix report is regenerated: its test counts follow the new test here (sqlite_compact 36) and the one added with the PostgreSQL closure check (postgresql_compact 31). Checks: sqlite_compact (35 run on macOS), sqlite_compact_matrix (1), sqlite_compact_crash (15), sqlite_compact_fixtures (1) pass; fmt clean. Production LOC: 188,960 -> 188,967 (delta +7) core 40,589 -> 40,589 (0) engine 90,619 -> 90,626 (+7) layerfs-persistence +7 control 51,230 -> 51,230 (0) api 6,522 -> 6,522 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd19) Method: tools/production_loc.py --detail on parent 9971d6e and the staged tree Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`Session::seal_holding` kept the writer lane in a parameter and the open connection in the session state. Every error exit before the checked close (a hard-linked or symlinked Store on macOS, an obstructed checkpoint, a failed close) dropped the state first. A writable close waits for the lane like any write, so it waited for the lane the same call held: the seal never returned. `store.compact` seals through this path, so a rebuild refused at its seal hung as well. The seal now makes its lane the connection's owner while the connection is open. An error exit closes under that hold and then frees the lane; after the checked close the lane goes back to the caller as before. This was already so at 6bca861: seal_allocation's hard-link and symlink refusal tests ran past 60 s there and now finish in 0.08 s. Test: writer_admission::a_refused_seal_returns_and_frees_the_lane (Linux and macOS) obstructs the checkpoint with a reader's older snapshot. The seal must fail Busy within 20 s and the next open must write. On the parent's seal.rs it fails with "the refused seal returns: Timeout". Checks: writer_admission (15) and seal_allocation (5) pass; fmt clean. Production LOC: 188,967 -> 188,972 (delta +5) core 40,589 -> 40,589 (0) engine 90,626 -> 90,631 (+5) layerfs-persistence +5 control 51,230 -> 51,230 (0) api 6,522 -> 6,522 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd19) Method: tools/production_loc.py --detail on parent 4c359ef and the staged tree Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The fixture asserted that a new selection's synchronous_commit is `off`. The product default became `local` with Normal mode (6bca861, #732), so every test in postgresql_doctor_access, postgresql_doctor_cancellation and postgresql_doctor_physical that built the fixture panicked at pg_doctor/mod.rs:56. The fixture is a Disposable Store; it now selects `off` itself and keeps the assertion. Checks: postgresql_doctor_access (10), postgresql_doctor_cancellation (9), postgresql_doctor_physical (16) pass against PostgreSQL 17. Production LOC: 188,972 -> 188,972 (delta 0) core 40,589 -> 40,589 (0) engine 90,631 -> 90,631 (0) control 51,230 -> 51,230 (0) api 6,522 -> 6,522 (0) reference absent (retired R9c; last counted 65,417 at tree 498dd19) Method: tools/production_loc.py --detail on parent and the staged tree; tests only Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Makes
store.compactfaster on both backends and fixes two defects found while verifying it. No API, CLI, SDK or Store format change.Speed
One sample per arm, release build, macOS host, 1-minute load 5 to 10 unless noted.
SQLite, 9.8 GB Store (9.789 GB live, 83,559 packs; every run frees 1,095,524,352 bytes):
6bca86167f)PostgreSQL 17 (Docker on loopback, back to back at load 25 to 27, identical receipts):
How
gc_profile(SQLite, per-statement) andpg_gc_profile(PostgreSQL, counted statements). Seetools/ci/gc-bench.md.Defects fixed
Both were already present at
6bca86167f.SQLITE_BUSY. The rebuild's settle cleared its intent in the installed Store without taking the writer lane. It now takes the lane for that one transaction (4c359ef8ca). This was why the acceptance matrix's AC-49 arm failed most runs; it passed 8 of 8 samples with the fix.store.compactseals through the same path (3190e6c593).Also: the PostgreSQL doctor test fixture now selects
synchronous_commit=offitself; it had asserted the old default and failed since Normal mode (#732).Checks
At
c32a6dcbab, with PostgreSQL 17 for the PostgreSQL tests:6bca86167f(sqlite_compact_matrix,seal_allocation, threepostgresql_doctor_*).-D warnings, fmt,check_product_boundary.pyand the tool unit tests: clean.a_mark_sends_the_same_statements_for_twice_the_layers,the_postgresql_closure_follows_exactly_the_shared_edges,the_settle_waits_for_a_writer_that_holds_the_lane,a_refused_seal_returns_and_frees_the_lane.Not covered: Linux was not run locally (macOS host); the speed numbers are macOS only.
Production LOC
187,546 → 188,972 (delta +1,426) against
layerfs_v2_0_0at3fc66bc406: core 40,408 → 40,589 (+181), engine 89,386 → 90,631 (+1,245), control and api unchanged. Counted withtools/production_loc.py --detail. Each commit records its own comparison.Not in this PR
claude/gc-opt-repack(rewriting half-dead packs) stays out: it is an owner decision.🤖 Generated with Claude Code