Skip to content

feat(gc): LayerFS 2.0 GC implementation - #711

Merged
yifanxuaaa merged 175 commits into
layerfs_v2_0_0from
gc-2.0-final
Nov 17, 2026
Merged

yifanxuaaa merged 175 commits into
layerfs_v2_0_0from
gc-2.0-final

Conversation

@mengmengmengqiang

@mengmengmengqiang mengmengmengqiang commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Single PR for the LayerFS 2.0 GC implementation, consolidated from the staged
PRs for review. Tracks the GC design in Ephemeral-AI-Lab/layerfs-project#41
and sub-issues #42–#52; the contract is
core/docs/issues/project-41/LayerFS-GC-spec.md and the open decisions are
core/docs/issues/project-41/gc-42-decisions.md.

Area What shipped
Schema/migrations marker tables (branch_retired, layer_stack_retired, path_removed), project_epoch, gc_intent, gc_op, gc_dead, kept_node, serials, trace_salt; SQLite + PostgreSQL additive steps
Removals branch.remove, project.delete (home/worker, sidecar, retry), project.remove_file (visibility-only whiteouts)
Admission gc_intent, MountsActive/Busy/Unresolved, PG advisory keys, project-epoch writer fence; a SQLite compaction now refuses Busy on another process's open Store descriptions, and removals order concurrent runs on the project_epoch row
Mark/closure history closure, object/pack closure, keptWaste, live-heads digest
Compaction SQLite: seal, step 1.5, the fresh-copy rebuild (copy only the retained rows into a fresh Store — no whole-Store VACUUM INTO), install fence, crash/kill recovery. PostgreSQL: Phase M/A/B/C, gc_op/gc_dead, scratch, swap, totals, standby/lost reply
Report/status §11 report, store.status (SQLite + PG), bounded metrics/SLO
API/MCP/CLI/SDK 85 calls in twelve families (the GC calls, the 6 #712 monitoring calls, machine.space, and the base's home.init/home.get), with the GC admin exceptions; docs, pages and count tests at 85/12
Tests/CI/docs SQLite + PG suites, real SIGKILL / CLI-process kill, the 10 GB capacity receipts and the newer R730 medium run, the ext4 CI lane (--quick/--full), runbook/rollout/capacity docs; the nine Round-2, seven Round-3, five Round-4, six Round-5 and five Round-6 fix branches, the #720/#725/#726 and #730 bases and the Round-6 base sync sit on the Round-1 head

Fresh-copy SQLite rebuild (D2)

The SQLite rebuild no longer stages a whole-Store copy and rewrites it. It
creates <store>.compact with the sealed Store's schema, copies only the rows
the mark retained (pack-as-unit), builds secondary indexes and triggers after
the bulk load, verifies the fresh file and renames it.

The two committed 10 GB receipts are not measurements of this head: both
were taken on e824e8a7, before #49 round 4 moved the rebuild onto the head8
build path. That earlier path had no 256 MiB src.mmap_size source window and
its verify still ran quick_check, so their phase split is the pre-head8 build
and verify path. They stay in engine/crates/layerfs-persistence/tests/fixtures/
as the recorded evidence, not as a claim about the merged tree.

  • The gc_bench receipt is three runs per arm on a generated fixture
    (9.789 GB live, 83,559 packs): none at 8.8% dead, rebuild at 10.0% dead
    after the dead-object add. rebuild 133.076 s median wall (range
    132.274–134.601 s, 1.75% relative range): mark 82.050 s, build
    23.469 s, verify 6.060 s, install 21.011 s; peak temp
    9.836 GB and physical shrink 1.093 GB. none is 82.683 s median
    on the same fixture.
  • The fresh-copy receipt's rebuild is 118.440 s median against the
    superseded whole-Store VACUUM INTO build's 246.371 s on the same
    9.987 GB fixture
    — 2.08×, same fixture on both sides. The earlier
    1.85× compared two different generated fixtures and is withdrawn.
  • A 10 GB medium run on the R730 (newer than either receipt, warm page
    cache) measured rebuild median 122.077 s (range
    116.804–126.820 s): mark 80.711 s, build 19.881 s, verify
    0.418 s, install 22.337 s. A verify this short is the head8 path,
    which no longer runs quick_check. That host's harness emitted invalid
    JSON ("name": ""sda"", doubled quotes); fixr6-benchjson fixes
    json_host(), so a real R730 receipt is valid JSON again.
  • Qualifiers: synthetic generated fixture, overlayfs temp filesystem,
    warm page cache (drop_caches refused), one declared rotational-media
    host class. These are qualification numbers on that host, not a cross-host
    guarantee; no 50 GB arm was run.

Production LOC

Measured with the tree's own tools/production_loc.py on the final head.
Nonblank, non-comment first-party product lines; tests, docs, benches,
fixtures and generated artifacts excluded.

Scope main 329a20be base 57423382f7 head Δ vs base
core 40,408 39,795 40,408 +613
engine 83,269 76,947 89,299 +12,352
control 44,141 48,436 51,132 +2,696
api 6,186 6,346 6,527 +181
total 174,004 171,524 187,366 +15,842 (+9.24%)

reference and dsh are 0 on both sides. The #727 base sync alone moves the
head from 184,030 to 185,983 (+1,953 product lines). The Round-4 branches are
test/doc-only except #43/#49/#50's small product changes; the Round-5
branches add 47 product lines (control +12, engine +35) through #43's
Init/delete naming, #50's PostgreSQL standby/holder/Phase M/status arms and
#49's install/identity fence, while #42/#52 are docs-only. The PR-only delta
is therefore the fresh-copy rebuild and the GC implementation itself. The
fresh-copy rebuild is engine-only (+389 lines over the 11c1896 snapshot:
the compact/ phase modules, the fresh-file build and its verification).

Integration

Round-6 fix branches on 7702bb21 (five branches)

7702bb2114 (pr711-head9, LayerFS commit 12d00e1c3102) is the Round-5
fixed head. Review Round 6 produced five fix branches, each forked from that
LayerFS commit. Each was exported and re-committed with parent 7702bb21
(.git and .lww dropped), then merged with --no-ff in this order:
42 → 43 → 50 → 41 → benchjson. The merges are 91dd691e70, e61da64d5e,
e5fdf821be, fbd074988b and fbf9864131.

Order Branch LayerFS head → export commit Merge Round-6 findings fixed
1 fixr6-42 12001542967a → 52823ff5c8 91dd691e70 #42 round-6 (docs) A11's non-GC close list now names store.pull (through the shared transfer's settle) and backup.restore beside store.push, store.remove and store.disconnect, and says the list is illustrative, not exhaustive; the A8 mark's op_id is named as the Store compaction op (GcIntentKind::Compaction with GcScopeKind::Store in postgresql/compact/run.rs), not "the project op". Carries the identical InUse-list wording into compact.rs's doc comment.
2 fixr6-43 12915e9797e4 → a920b98316 e61da64d5e F43R6-1 an Init reserve_inodes failure whose outcome is unknown is retained by backing.settle as ProjectError::Uncertain { cause }; add_failure now unwraps that arm to lookup::history_in, so the call still answers Unknown naming the project, the init phase and the durable op — never Failed with a null detail. A definite history cause inside Uncertain keeps its own code (the new arm asserts Busy with its operation entity).
3 fixr6-50 12bc4acf63e6 → b971061e33 e5fdf821be F50R6-1 §11: an intent-only Phase-M projection is unresolved, not busy, when the recorded process owner is provably dead (OwnerView::owner_dead, extracted from dead), so a dead owner cannot keep a run "in flight" merely because its intent is still live; a run whose intent is uncertain is never in flight. F50R6-2 §8 step 10: the legacy synchronous_standby_names parser reads FIRST/ANY as keywords only as whole tokens (first_standby, anybody are names) and accepts PostgreSQL's FIRST 2(s1,s2) spacing with no space before the list.
4 fixr6-41 125d1497b4b3 → ca5a95287a fbd074988b F41-1 an installed G + 1 leftover answers the original Rebuild report from the totals the run recorded (a new installed_* block in .compact-op), instead of re-running §3 on the already rebuilt Store and reporting none; the receipt keeps the recorded generation_before. F41-2 the runbook's "known gaps" become the current pre-GC Store upgrade, naming the open-time additive steps on both backends. F41-3 SQLite maps HistoryError::BusyOp to CompactRefusal::Busy in the metrics. The AC-47 row and the report's sqlite_compact count (34) follow.
5 fixr6-benchjson 12d8d24aecd9 → 99d90eb120 fbf9864131 gc_bench JSON json_host() re-quoted the values json_string() already quotes, so a host with a detected disk printed "name": ""sda"" and the R730 receipt was invalid JSON; the doubled quotes are gone and the R730 host block parses.

Conflict resolution. Every branch shares 7702bb21 as its three-way base,
and the five source merges are clean: 42 touches only gc-42-decisions.md
and compact.rs's doc comment, 43 touches project/add.rs,
import/init.rs and its test, 50 touches the PostgreSQL sync.rs/status.rs
and admission.rs plus its tests, 41 touches the SQLite
compact/op.rs/install.rs/run.rs/receipt.rs, the runbook/report and its
tests, and benchjson touches only examples/gc_bench.rs and its
[[example]] test = true manifest line. No two branches touch one file, so
the union keeps every change; the sqlite_compact_matrix test asserts the
merged report is current (it is), and cargo fmt/clippy fixups on the R6
trees are in b78c076252.

Base sync to layerfs_v2_0_0 39a25138ea (#730)

The base moved 24 commits past the 57423382f7 this head was based on: #730
adds home.init/home.get, machine platforms for setup, doctor host checks
and the native unmount reap. The merge e9586546aa keeps both sides — the
table becomes 85 calls in twelve families (the two home calls join;
project.delete + project.remove_file replace the base's project.remove,
and branch.remove stays), the profile buckets are 38 work / 40 admin
only / 6 ui / 1 no tool so --profile admin lists 78, and every count
test, SDK page, manual page and the generated coverage block say 85/12.
Thirteen files conflicted (the call table and its counts, the MCP profiles and
their tests, the Node and Python tests, and the SDK, calls and front-end
docs); each was resolved by keeping the base's home/machine/doctor/unmount
additions beside the PR's GC calls, and the counts were then made consistent.
The base's engine, daemon and machine work is kept beside this PR's changes.

Round-5 fix branches on b357463b (six branches)

b357463b (pr711-head8) is the Round-4 fixed head. Review Round 5 produced
six fix branches, each forked from LayerFS commit 12020f69bea9 whose tree is
b357463b (only the .lww differs). Each was exported from the LayerFS
project and re-committed with parent b357463b (.git and the .lww
dropped), then merged with --no-ff in this order: 42 → 43 → 49 → 50 → 51 → 52.
The merges are dba16b1f80, cca723ac36, 93853740eb, 5654607e3d,
d05fe297b1, c3632887e6; c07297303b moves 43's listed_unknown helper
out of project/delete.rs after the added lines took that file over the
999-line product boundary.

Order Branch LayerFS head → export commit Merge Round-5 findings fixed
1 fixr5-42 12cea9b21d7a → f48dd2c24a dba16b1f80 #42 round-5 (docs/comments, F42R5-1…4) the decisions note adds A12 as a reading, not a twelfth amendment: a rebuild stages gc_intent empty and inserts only the run's own op_id row, so "never copied" and §5.1's "lives in <store>.compact" agree; the preamble/sign-off say A8–A12 and the heading reads "Contract amendments (11) and one reading". The A9 refusal text names store.push (not store.move) as the non-GC call that can still report the cache's InUse; the intent-sidecar doc lists the optional op_id; the two 10 GB receipts in the evidence basis carry the vintage caveat (F52R5-2 wording: both predate #49 round 4's mmap window and quick_check removal).
2 fixr5-43 1266b28b9200 → bc4f5c8156 cca723ac36 F43R5-1 an Init history answer lost after admission names the project and init phase: init_unknown_state is built before request.name moves and is attached to record_new_bindings, initialize_layerstack, the Store-name-taken check and the fork (a bare UnknownOutcome reaches the client as Unknown with {kind: "project", name}, phase: "init", no durable op — a definite BusyOp stays Busy). F43R5-2 a project.delete Store-listing failure that cannot be determined names the project too (remote::listed_stack wraps both the empty-map UnknownOutcome and the poisoned-Store Unknown with listed_unknown), so it never stays a bare Unknown {}; the new project_add_unknown and project_delete arms pin both.
3 fixr5-49 124b565f447c → b2d0757a97 93853740eb #49 round-5 (install fence) §7 step 7's install re-applies the sealed (dev, ino) identity; a mismatch is now Conflict("the sealed Store identity changed") with no writes, not StaleStore. A Store replaced between seal and install is therefore Conflict; StaleStore stays the §10 session/transaction fence for a handle that opens or transacts after the install. The test is renamed a_store_replaced_between_seal_and_install_is_conflict, and the AC-46/AC-59 rows, the regenerated report and the identity/SDK Store docs follow.
4 fixr5-50 12da9f6ffe2d → d88d4b3293 5654607e3d F50R5-1 a legacy bare synchronous_standby_names list is FIRST 1, not one synchronous standby per listed name (unit cases for "s1, s2" and "s1"). F50R5-2 the swap 55P03 holder query reads any granted table lock on the Store base tables, so an ordinary AccessShareLock reader is named (pid/client_addr/backend_start) with a reader-blocker test. F50R5-3 store.status projects the registered Phase M gc_intent while gc_op does not exist yet (id/name/mark phase, started, updated_ms, busy); a test parks the run inside Phase M on gc_dead. F50R5-4 A8 wording: only the after size is sampled inside the swap transaction; the before size is sampled before it opens.
5 fixr5-51 121751691772 → 1494a9e01c d05fe297b1 F51R5-1 the MCP admin-profile test now expects the documented 76 tools (83 calls − the doctor family's one − the six monitoring calls a UI polls), not table.length - 1 = 82; the check compares against the profile's own admin count. Test-only, no source change.
6 fixr5-52 1251cf43c9c4 → 780421de3d c3632887e6 F52R5-1 gc-42-decisions.md says seven gc_op columns (A8 and the PostgreSQL schema already carry seven); 42 carries the identical word change. F52R5-2 tools/ci/gc-bench.md §13 records the 10 GB receipts' vintage — both predate #49 round 4's 256 MiB src.mmap_size source window and its quick_check removal — and marks the ~4 s quick_check cost derived, not measured; the matrix runner's AC-57 note, the report body and the E3 bullet say the same, and the report is regenerated. No 10 GB re-run: the receipts are not replaced. F52R5-3 the store-ext4.yml quick-lane comment now says the api suite runs four targets instead of "one or two targets per suite".

Conflict resolution. Every branch shares b357463b as its three-way base,
and the six merges are clean. The overlapping files were read semantically:

  • 42 + 52 on gc-42-decisions.md. Both make the identical
    "six → seven gc_op columns" change, so the merge applies it once; 52
    then has no net change there. 42's A12 reading, store.push and sidecar
    lines stay.
  • 42 + 50 on the same file. 50's A8 paragraph (before size sampled
    before the swap, after size inside it) is a different region from 42's
    A12 insertion and request paragraph; the union keeps both.
  • 49 + 52 on the T12.3a report and the matrix generator. 49 edits
    the AC-46/AC-59 evidence names and notes; 52 edits the §13 vintage text
    and the AC-57 row, and regenerates the report. The union keeps both, and the
    merged runner regenerates the merged report (the sqlite_compact_matrix
    test asserts it).
  • 42 + 43 on project/delete.rs. Different regions (the intent-sidecar
    doc and the new listing helper); the merge is clean, but the added helper
    took the file to 1004 lines over the 999-line boundary, so c07297303b
    moves it to project/delete/remote.rs and re-exports
    project::delete::listed_unknown for the F43R5-2 test. The product boundary
    is green again (1526 files).

The frozen LayerFS-GC-spec.md is still byte-identical to ed14ed53b
(sha256 84e16978d92d14ee65eb10d38f8d2f5347455b84c2c5b2e49fd28f60282415cf,
84,894 bytes) on the merged head.

Round-4 fix branches on a536f920 (five branches)

a536f920 (pr711-head7) is the Round-3 fixed head. Review Round 4 produced
five fix branches, each forked from LayerFS commit 3230a02565d5 whose tree is
a536f920 (only the .lww differs). Each was exported from the LayerFS
project and re-committed with parent a536f920 (.git and the .lww
dropped), then merged with --no-ff in this order: 42 → 43 → 49 → 50 → 52.
The merges are dfdd055503, 3370046f3f, bc811e91ea, 19679fd104,
7b58d4eaa1.

Order Branch LayerFS head → export commit Merge Round-4 findings fixed
1 fixr4-42 122df99487a9 → 0242ff8ed0 dfdd055503 F52R4-1 the gc-42-decisions.md preamble counts are consistent: sixteen places (A1–A11 + D1–D5), the note adds A8–A11, and the schema paragraph's "four gc_op columns" becomes six before #50's physical_after (the merged tree says seven). F52R4-2 A11's "Where" names the moved unit tests in control/crates/layerfs-store/tests/compact_in_process_call.rs, not src/compact.rs (which keeps gc_refusal).
2 fixr4-43 129ccd5e1e57 → 355ff7d183 3370046f3f F43R4-1 named_unknown also fills an empty-map Unknown detail (a bare UnknownOutcome renders SdkValue::map::<0>([]), not Null), so project.get/project.list(store) and the worker project.delete.release path name the Store/entity and operation instead of returning Unknown {}. F43R4-2 project.init's registration/release UnknownOutcome reaches the client as Unknown naming {entity: {kind: "project", name}, phase: "init", op} (both the InitError::Project(ProjectError::History) and InitError::History arms), not Failed with a null detail; a definite BusyOp stays Busy with its operation detail.
3 fixr4-49 1261ccde6d5c → 21b401573e bc811e91ea #49 round-4 (verify, mmap) §7 step 6's verify no longer runs PRAGMA quick_check: the pass is schema identity, foreign keys, counts, live-heads digests, the allocator tuple and the recomputed totals, and quick_check/the doctor stay separate commands, so it reads no payload. The T12.3a text and AC-57 record the earlier ~4 s page scan as the then-present quick_check (removed here) rather than a current verify cost. §7 step 2's bulk copy applies PRAGMA src.mmap_size = 268,435,456 to the attached sealed source (the session default is no mmap) and returns the size SQLite applied.
4 fixr4-50 12c9ab78bfd2 → 3a747da6d9 19679fd104 #50 round-4 (PG status/sync/physical_after) PostgreSQL's gc_op gains physical_after, filled from the swap transaction's own pg_total_relation_size(pack, object_location) sample, so a save after the swap (AC-28) cannot inflate a resumed G+1 report's physicalShrink; Swapped.physical_after now carries it. store.status reports busy only while the op's gc_intent is present and the recorded (pid, backend_start, usename) is alive, and unresolved once the intent is released (PostgreSQL's form of SQLite's returned) or the owner is gone; the state != 'idle' test is dropped, so an idle but live owner of a released op is not busy. effective_rank no longer floors the swap's sync strength at local, so two off settings stay off (and the §8 step number is corrected to step 10).
5 fixr4-52 12780a4178a8 → 057e1ffb0e 7b58d4eaa1 F52R4-3 the untraceable 92–104 s superseded-build range is gone: the T12.3a text, AC-57 and the matrix generator record the 94.317 s median only and the recorded missing-range deviation, pointing at tools/ci/gc-bench.md §13. F52R4-4 the ci.yml/format.yml comments name the store-ext4 exception (PRs into main or layerfs_v2_0_0), and store-ext4.yml's quick-lane comment counts five cargo groups and the api group's four targets. F52R4-5 test_store_ext4.sh --list prints the commands before the LAYERFS_TEST_TMPDIR check, so the documented verbatim command exits 0 without the variable.

Conflict resolution. Every branch shares a536f920 as its three-way base.
Only three files overlapped, and each was read semantically:

  • 42 + 52 on gc-42-decisions.md. The two blobs are identical: 42
    fixes F52R4-1/F52R4-2 and 52 carries the same text, so the merge takes
    42's version and 52 changes nothing there.
  • 42 + 50 on the same file. 50 adds physical_after and the "both
    relation sizes sampled in the swap transaction" sentence; 42 had already
    moved kept_waste into the mark and re-wrapped the paragraph. The union
    keeps both: the paragraph now says seven columns, the mark writes four
    (op_id, mark_fingerprint, mark_visited, kept_waste) and the swap
    writes three (physical_before, physical_after, peak_temp).
  • 49 + 52 on the T12.3a report and its generator. Union: the merged
    generator carries both 49's removed-quick_check wording and 52's
    superseded-build missing-range deviation, and the report is generated from
    it, so AC-57 states both. 43's new unit tests had landed in product source
    (src/project/add.rs, src/unknown.rs), which the product-boundary guard
    forbids: 6efec405ba moves them to layerfs-store/tests/project_add_unknown.rs
    (unknown::named_unknown is #[doc(hidden)] pub), and formats 52's matrix
    generator. 983cbb4ca1 replaces 50's .map(format!).collect::<String>()
    artifact ids with one hex helper so clippy's format_collect stays green,
    and the T12.3a report is regenerated on the merged tree (the extra [v0.1.3][Phase 2] Fix CDC locality scaling #43
    project_delete_worker test moves its count from 7 to 8).

Base sync to layerfs_v2_0_0 57423382f7 (#727)

The integrated head is merged with the current layerfs_v2_0_0 base
57423382f7 (#727, the lp73 integration: the NFS AppleDouble codec, spill and
per-inode companion layers, batched READDIRPLUS/LOOKUP facts and macOS mount
options) as commit b41ebabc93, a plain merge with no rebase and no force.
dd4a2cc40f is the merge base and the base moves 44 files. Its only file in
common with this PR is Cargo.lock (the base adds libc to layerfs-daemon,
the PR adds postgres to layerfs-persistence), and the merge is clean.

The previous base sync already removed the postgresql build feature (#726),
so the PostgreSQL engine is always compiled; no gate uses a feature flag.

Against the new base 57423382f7 the PR is 146 commits, 481 files,
+62,710 / −1,426
.

Round-3 fix branches on d5cc150377 (seven branches)

d5cc150377 (pr711-head6) is the Round-2 fixed head. Review Round 3 produced
seven fix branches, each forked from LayerFS commit 1280f9b1df18 whose tree
is d5cc150377 (only the .lww differs). Each was exported from the LayerFS
project and re-committed with parent d5cc150377 (.git and the .lww
dropped), then merged with --no-ff in this order: 43 → 44 → 45 → 46 → 49 →
51 → 52. The merges are fee2ebcc49, 5ef2ce0153, 636469bfb1, 87418c8920,
30da9172f1, eddf7534ec, 02dca44dc1.

Order Branch LayerFS head → export commit Merge Round-3 findings fixed
1 fixr3-43 12e834134847 → 61212f265e70 fee2ebcc49 #43 a worker project.list reply whose answer is lost is Unknown, and the caller now names the Store it listed: project.family's worker listing, project.get and project.list of a named Store attach {entity: {kind: "store", name}, phase, op} through unknown::named_unknown. A shared intent's failed gc_register/gc_release keeps its SharedIntent scope, project and phase and reports Unknown through lookup::history_in with the operation id, instead of a bare refusal.
2 fixr3-44 122e15d7d904 → 05bf7e00d467 5ef2ce0153 F44R3-1 branch_any, published_layer and node_history_serial ran their one statement in autocommit with no search_path and qualified() served no {s} body for them, so PostgreSQL answered Integrity("persistence"). Added the three {s}-qualified bodies and postgresql_statement_schema: a guard that discovers every read_statement body in the crate, pins the keys it runs, fails when a body or key is added without a {s} body, and runs all of them against a real server (it also reads a brace body's { correctly, not a closure argument's parenthesis).
3 fixr3-45 128e61a33007 → a9cb4c374852 636469bfb1 #45 a sidecar in marked is already past the marker: its retry takes no admission, writes no marker, removes the worker's own entry through the new project.delete.entry worker op and then the caller's home entry, recording done. A record or listing that names a different Stack than the sidecar is a re-created project: the stale intent is discarded and the call is NoSuchProject, so the new project's entry is never touched. Listing/read refusals keep their History wire code through lookup::published (identity, project.family, project.listing) instead of failed's generic code; the 2.0 manual and worker-op docs follow.
4 fixr3-46 127e6f7e49e5 → 671eef261ebc 87418c8920 F46R3-1 remove_unreachable deleted a Stack's unreachable Layer and Commit rows but store_totals.logical is maintained by AFTER INSERT triggers alone, so the deleted rows' logical bytes stayed counted. The prune now retracts exactly this Stack's unreachable logical bytes in the same write transaction, while the rows the statement reads still exist (history_remove_unreachable_totals.sql), with kept_node and sqlite_compact regressions.
5 fixr3-49 125c76ce3fe7 → fe0f790e2b55 30da9172f1 #49 returned/BusyOp a <store>.compact-op record gains the returned marker written before the call replies, so a run that already answered is a leftover the next call judges: store.status stays unresolved for its record even while the recorded pid lives, and busy only for an in-flight live owner. A live in-flight op is refused with BusyOp naming the record's entity, phase and operation. Install/run/op, the report/status projection, the sqlite_compact returned-run regressions and the docs follow.
6 fixr3-51 12ebf0439213 → ad5640140a77 eddf7534ec F51R3-1 store.compact's open-cache refusal for a call still running on the Store in this process is the GC vocabulary's Busy naming the Store with a null phase/op, not the cache's InUse (which stays for the non-GC store.move/store.remove/store.disconnect); the store.compact error row and focused tests pin it. F50R3-2 AC-44's evidence and matrix text gain the three Round-2 postgresql_compact tests (ended owner without the Store key, live key holder Busy, no pg_signal_backend).
7 fixr3-52 122cdc91330d → d3b967de7b5d 02dca44dc1 F52R3-1 LayerFS-GC-spec.md is byte-identical to ed14ed53b again: D1/D2/A10 stay only in gc-42-decisions.md, and the runbook states the frozen text is unchanged (the D2 peak/journal deviation is recorded there). F52R3-2 the report-only store-ext4 lane is documented for PRs into main or layerfs_v2_0_0, not "any base". F52R3-3 / F42R3-1 / F50R3-1 the nonexistent postgresql build feature is removed from the Store, project, front-end, versioned and doctor docs (always built, selected by the connection string). F42R3-2 the logicalDiscarded/backendReusable comments follow A8's semantics.

Conflict resolution. Every branch shares d5cc150377 as its three-way base.
The overlapping files auto-merged to the stated ownership except the generated
matrix report, and each overlap was then read semantically:

  • 51 + 52 on LayerFS-GC-spec.md. 52 restores the whole frozen file
    to ed14ed53b; 51 adds five §10 lines. A plain merge kept both, so the
    final tree takes 52's file byte-for-byte and 51's rule moves to
    gc-42-decisions.md as A11 (A. Contract amendments (11)), with 51's
    store.compact docs row and its unit/matrix tests kept. The spec diff against
    ed14ed53b is empty.
  • 46 + 49 + 51 on the T12.3a report. Every branch regenerates the
    report from its own test sources and each merge conflicted there; the report
    is regenerated from the merged test sources (33 sqlite_compact,
    7 project_delete_worker, 17 project_delete, 6
    compact_report_status), so the derived counts match the merged tree.
  • 43 + 45 on project/family.rs. Union: 45's lookup::published
    import and live_projects code with 43's named_unknown wrapping on the
    worker reply, project.get and the named-Store project.list.
  • 49 + 52 on store.md and 51 + 52 on store.md union: 49's
    returned-run status paragraph and 51's Busy row survive 52's docs
    corrections; 52's removed postgresql-feature sentences win.

Integration fixups.

  • 719b5b1651 restores the frozen spec, adds A11 and regenerates the T12.3a
    report on the merged tree.
  • 3dc3453173 moves the two Round-3 source unit tests out of product source
    (the boundary guard forbids #[cfg(test)]/#[test] there):
    51's compact::gc_refusal mapping tests to layerfs-store
    tests/compact_in_process_call.rs (gc_refusal is #[doc(hidden)] pub),
    and 45's published() test to the store-service history_wire_codes
    target where the other lookup wire-code tests live; it also runs
    cargo fmt --all over 44's unformatted PostgreSQL statement-schema test.
  • a536f920fa55 finishes the A11 references in gc-42-decisions.md (the
    header count and the [v0.1.3][Phase 2] Fix cross-file CAS deduplication scaling #42 sign-off request name A1–A11 / A8–A11).

Round-2 fix branches on 43e72cf5 (nine branches)

43e72cf5 (pr711-head5) is the Round-1 fixed head. Review Round 2 produced
nine fix branches, each forked from LayerFS commit 126d2796b7dd whose tree is
43e72cf5 (only the .lww differs). Each was exported from the LayerFS
project and re-committed with parent 43e72cf5 (.git and the .lww
dropped), then merged with --no-ff in this order: 43 → 44 → 45 → 46 → 47 →
49 → 50 → 51 → 52. The merges are 5868fb3cef, ebcf856041, 11f05dc72c,
df5d31ac10, 837e04a54d, 157f62c285, a956794f2a, 06621c7869,
aaeed8f900; the integration head before the base sync is 3b1aa01ab3.

Order Branch LayerFS head → export commit Merge Round-2 findings fixed
1 fixr2-43 120794668f2c → 270998cfefc4 5868fb3cef F43R2-1 (§10 A6) a lost worker reply now names the entity and the operation: HistoryError::UnknownOutcomeOp carries UnknownOutcomeState {scope_kind, entity, phase, op}, with_unknown attaches it at the call boundary, and unknown::named_unknown fills the {entity:{kind,name}, phase, op} detail (branch/project calls, compact, project.delete home and worker, remote). F43R2-2 gc_intent_ops_{branch,project,store}.sql see a live Init and merge as well as the removal/compaction subset (never another mount), and the visibility-guard allow-list reason says so.
2 fixr2-44 126f9243c75c → e238710a9164 ebcf856041 F44R2-1 the PostgreSQL read side is generation-fenced: the expected incarnation rides as a trailing column on each autocommit read's statement and in the BEGIN batch of a transaction (the AccessShareLock it holds blocks a swap), run_observed keeps the diagnostic observational; retired backup heads — read_heads filters branch_retired/layer_stack_retired, so a backup digest names the visible history; the inline visibility guard gains the statement-set tests.
3 fixr2-45 12fbfc773896 → 8f20720ebc98 11f05dc72c F45R2-2/3 a fresh project.delete resolves the project only through the Stack its Store lists now: a retired project is NoSuchProject (never finished from a stale home record), and a fresh call that meets Retired/Absent removes its sidecar and keeps the entry; §10 Busy detail names the recorded Sandbox; F45R2-2/4 a stale record without a sidecar is NoSuchProject; inventory and the 2.0 manual say project.delete.
4 fixr2-46 12d9639ebc5e → 6d36cc25f2cf df5d31ac10 F46R2 the unreachable-history mark and its prune run in one write transaction (kept.rs, catalog.rs, history_unreachable_count.sql); a cleared mark's prune is a post-compaction no-op; kept_node tests.
5 fixr2-47 125f89fab84e → 98e45d499641 837e04a54d F47R2 only-symlink strictness is restored: an intermediate symlink is ill-typed and refused, a regular-file intermediate is merely absent; marker existence is lenient; project.remove_file is project-wide — it checks the path's grammar against every visible tree (main's head and every Branch head), so a Branch-only or never-existing path is accepted.
6 fixr2-49 1232319559eb → 1b7aae1ca1fd 157f62c285 F49R2-1 a <store>.compact-op record whose owner is provably live is not a leftover: the engine asks the caller's process facts (CompactRequest.owner_live), so a run that released the writer lane but has not settled stays Busy instead of being adopted; the settle removes the record only while it still names the operation being settled (install/run/op/types), with a before_settle_hook test window.
7 fixr2-50 123a2ff9b23e → 5de0eaf88746 a956794f2a F42R2-1 (also #42) the §8 lost-COMMIT reply is resolved by the next store.compact on a second connection: a recorded owner that no longer holds the exclusive advisory key is terminated (pg_terminate_backend), one that still holds it stays Busy; A8's gc_op columns are completed (op_id, mark_fingerprint, mark_visited); Phase C deletes each project's lineage/derived closure in its per-project transaction and records the cleanup phase before the first batch; postgresql_compact grows the lost-reply, lineage and phase tests.
8 fixr2-51 12a9510a10a9 → ec6620a36c4f 06621c7869 F51R2-1/2/3 api/node/src/index.d.ts and the Python _client.pyi/aio.pyi are regenerated from the 83-call table; docs/architecture/api/calls/coverage.md is regenerated; store.md's zero_reasons wording is fixed.
9 fixr2-52 124cd7bf9388 → 31faeb43c8ee aaeed8f900 F52R2-1..4 / F48R2-1 the T12.3a report's "Verified here" counts are derived from the test sources (a stale count now fails the matrix test), the 2.08x whole-Store-to-fresh-copy comparison is cited with the whole-Store receipt, A10 records deadBytesRefined == deadBytesBound on every path, the 50 GB receipt is named in the runbook/report, and the store-ext4 lane is wired in CI.

Conflict resolution. Every branch shares 43e72cf5 as its three-way base,
and the overlapping files auto-merged to the stated ownership; each overlap was
then read semantically:

  • 43 + 45 on control/.../project/delete.rs: 45's fresh/retry split and
    fail-closed NoSuchProject keep 43's lookup::history_in naming on the
    admission and marker-error paths (and 43's remote.rs/admit detail).
  • 43 + 44 on tests/visibility_guard.rs: 44 owns the statement-set tests;
    43's only change there is the gc_intent_ops_project.sql allow-list reason,
    which matches 43's widened SQL, so the union is the correct text.
  • 43 + 47 on engine/.../methods/project.rs: 47's project-wide
    visible_trees check with 43's history_in on gc_remove_path.
  • 43 + 49 on control/.../store/compact.rs: 49's owner_live field with
    43's history_in(..., Store, store, None, Some(&op_id)).
  • 49 + 50 on engine/.../history/admission.rs: 49's pub(crate) current_namespace with 50's store_key_held_by (the lost-COMMIT guard).
  • 49 + 52 on sqlite/compact/{mod,types}.rs: 49's OwnerLiveness /
    owner_live / before_settle_hook with 52's A10 module and field docs.
  • 50 + 52 docs: the A8 six-column gc_op paragraph (50) and A10 (52)
    are both kept; gc-runbook.md carries 50's automatic lost-COMMIT
    resolution and 52's A10 paragraph and 50 GB/whole-Store receipts; the
    matrix report's AC-52/AC-57 rows carry both.
  • 46's kept.rs/catalog.rs and 52's matrix/report/receipts do not
    overlap; 51's typings/coverage regenerate from the unchanged 83-call table.

Integration fixups. 3b1aa01ab3 moves 45's new fresh argument into the
existing Finish struct: the added positional argument pushed
mark_and_finish to eight arguments and failed
clippy::too_many_arguments under -D warnings. d5cc150377 regenerates the
T12.3a report on the merged tree (the derived counts are sqlite_compact 30,
sdk_path_hidden_semantics 3, sdk_path_removal 5 and postgresql_compact
29).

Base sync to layerfs_v2_0_0 dd4a2cc40f (#720/#725/#726)

The integrated head is merged with the current layerfs_v2_0_0 base
dd4a2cc40f as commit ae0a4c4c2f, a plain merge with no rebase and no
force. The new base carries #720 (a namespaces command cancelled before its
handle was told ends with SIGKILL, not SIGTERM), #725 (per-Workspace
CPU/memory/pids in sandbox.get; the Sandbox group holds no process) and #726
(the PostgreSQL engine is always compiled beside SQLite, removing the
postgresql feature from every crate).

The merge conflicts are all the PR's own additions meeting #726's cfg
removal:

  • engine/crates/layerfs-persistence/src/lib.rs: the feat(gc): LayerFS 2.0 GC implementation #711 exports
    (compact_metrics, internals, report, the sqlite compact and
    kept/mark re-exports, OwnerLiveness) are kept without the removed
    postgresql gates.
  • engine/crates/layerfs-persistence/src/diagnostic/mod.rs: pub(crate) mod postgresql_contract is kept without gates.
  • engine/crates/layerfs-store-service/tests/postgresql_open_cost.rs: the
    URL-or-keyword conninfo::with_host proxy fix is kept without the
    #![cfg(feature = "postgresql")] header.

Because #726 removes the feature, the PR-side code and harness that still
assumed an opt-in build are adapted in the same merge: the 61 remaining
cfg(feature = "postgresql") gates
on #711 code and tests are removed, the
three not(feature) stubs (store compact_postgresql, store
compact_status, admission's postgres() fallback) are deleted or flattened,
tools/ci/test_store_ext4.sh stops injecting --features postgresql, and the
store-ext4 workflow comment, testing.md, limitations.md,
examples/README.md, parallel-worktrees.md and the matrix report generator
describe the always-built engine. --features layerfs-persistence/postgresql
is consequently no longer a meaningful gate; every check below runs without
it.

The ancestry below this head is unchanged: the nine Round-2 merges above, the
earlier base-snap (main 329a20be) export, the eight
fix711-{contract,pg,life,rm,sqlite,api,perf,hyg} merges, the
fix711-{spec,bench,pgtest,perf2} merges, the in-place SQLite/PostgreSQL
compaction split (7223b8d3f2), the eleven Round-1 fix branches and the four
e824e8a7 fix branches all remain in history.

Against the current base dd4a2cc40f the PR is 114 commits, 465 files,
+60,349 / −1,409
; the final head is d5cc150377.

Round-1 final integration (eight branches on 11c1896a01)

This head integrates the eight final fix branches onto the #711 head
11c1896a01 (LayerFS commit 1201e45276…). Every branch is forked from that
head; each was exported from the LayerFS project and re-committed with the PR
head as its parent (.git/.lww dropped), then merged in order:

Order Branch Head Merge
1 impl-freshcopy 12b4547a932f 0b41b16d08
2 fix711-none 1244bbb8334c 687612b39e
3 fix711-report2 129c4b59c7b6 1cb3751423
4 fix711-conc 12d812591a5d f52e932aca
5 fix711-remote 1287d209cd48 ec1c018588
6 fix711-docs 12955309f113 0f5b1b4f1c
7 fix711-bench3 12c1bcb00afb b315ac91ec
8 fix711-install 12f9403e810d c80de3bc02

Conflict resolutions are semantic, since all eight share the PR head as their
three-way base:

  • Fresh copy wins on engine/.../sqlite/compact/build.rs (the rebuild is
    the fresh-copy load); fix711-report2's Built.free_bytes removal is kept
    with it, because backendReusable is now the source Store's pre-mark
    free-page bytes.
  • fix711-conc wins on history/admission.rs (open-description admission
    and epoch-row removal locks), preserving fix711-none's register_in
    in-transaction call from the mark.
  • fix711-report2 wins on the PostgreSQL report (canonical
    logicalDiscarded, source backendReusable, resumed-run report), with
    fix711-remote's markBytes metric kept.
  • fix711-none wins on sqlite/compact/run.rs (the none path's mark and
    intent are one rolled-back transaction), with fix711-conc's Busy probe
    and fresh-copy's build integration around it.
  • fix711-remote and fix711-docs win on their docs (control/architecture
    docs and the call manual respectively).
  • gc-42-decisions.md is a union: A8 (fix711-docs), D5 (fix711-remote),
    G7 (fix711-install), the fresh-copy D1/D2 rewrite (impl-freshcopy), the
    rollback-only D3 (fix711-none) and the fixture-basis note
    (fix711-bench3); the count is now thirteen places = eight A + five D.
    gc-runbook.md, the T12.3a matrix report and tools/ci/gc-bench.md carry
    the same union (D5 row, fresh-copy and old-build capacity numbers, fixture
    shape/cold-cache caveat, scratch sizing).
  • Fixups after the merges: the runbook names D5 with the other deviations;
    rustfmt on the fresh-copy build and matrix text; the six report/receipt
    unit checks moved out of product source into
    tests/{sqlite,postgresql}_compact_units.rs (the product-boundary guard).

Base sync to layerfs_v2_0_0 4a8efe63b3 (#718/#719/#724)

The head is then merged with the current layerfs_v2_0_0 base 4a8efe63b3
(#718 a forced native stop cancels running commands first, #719 each exec's
helper and command inside their cgroups, #724 machine.space) as commit
5e73795d98, a plain merge with no rebase and no force. The resolution keeps
the GC implementation, the fresh-copy rebuild and the remote/docs work, and
takes the new machine, Sandbox and monitoring code with its docs, CI and
tools/check_product_boundary.py audit-note updates. The only files both
sides changed are the call-count files: the merged table is 83 calls in
eleven families
(76 GC + six #712 monitoring + machine.space). The
work/admin buckets stay 38/38 (the MCP server lists 76) and the UI-only
set becomes six; calls_table, the generated MCP profile table, the coverage
matrix and the 2.0.0 manual are at 83. Follow-ups 6daca0bc39 and
89fdbde155 keep the machine.space manual rows and the 83-call counts in
docs/versioned/2.0.0/calls.md, the calls page, the SDK page inventory and
the calls module. Against base 4a8efe63b3 the PR is 68 commits, 430 files,
+54,843 / −1,302.

The ancestry below this head is unchanged: the earlier base-snap (main
329a20be) export, the eight fix711-{contract,pg,life,rm,sqlite,api,perf,hyg}
merges, the fix711-{spec,bench,pgtest,perf2} merges, and the in-place
SQLite/PostgreSQL compaction split (7223b8d3f2) all remain in history.

Final fix branches on e824e8a7 (docs4 → clippy4 → opencost4)

e824e8a7 is a test-only fix on top of 89fdbde155: the PG concurrent-remove
fence now counts live Branches through the branch_retired marker instead
of raw count(*) FROM branch, so the long-standing postgresql_gc_fence
expectation matches the soft-removal design (§4.4).

The last three branches fork from pr711-head4 (LayerFS 12d276e2, whose
tree is byte-identical to e824e8a7 apart from the LayerFS .lww). Each was
exported and re-committed with parent e824e8a7 (.git and .lww
dropped), then merged in order; no conflicts — docs vs code vs one test:

Order Branch Export commit Merge
1 fix711-docs4 5be3f41c07 cf6e107804
2 fix711-clippy4 8b2cf7748e dcacc2ddac
3 fix711-opencost4 a9b1042da1 9ab2869cf2
  • fix711-docs4 brings the SDK/observe skills and the versioned references
    to 83 calls in eleven families (the new doctor and logs pages,
    branch.commits/get_commit/remove, file.revision, sandbox.files,
    the machine and store/backup tables) plus the docs/plugins and
    docs/versioned/2.0.0 README/migration/SDK pages.
  • fix711-clippy4 clears the workspace clippy warnings under -D warnings
    across 10 files: test assertions, needless format!/return, the
    advisory_holder type alias, the write! owner token, doc-comment
    formatting.
  • fix711-opencost4 makes postgresql_open_cost form-agnostic: it takes the
    upstream host:port from postgres::Config and rewrites the endpoint with
    conninfo::with_host, so the proxy accepts a URL or a libpq keyword
    connection string.

Round-1 fix branches on e824e8a7 (11 branches)

Final counts after this integration: 93 commits (25 on top of 9ab2869c),
450 files, +57,881 / −1,358 against the base 4a8efe63b3; final head
43e72cf538.

Review Round 1 produced eleven fix branches. Each was exported from its LayerFS
branch (.git and .lww dropped) and re-committed with parent
e824e8a7
, then merged with --no-ff in this order: 42 → 43 → 44 → 45 → 47
→ 48 → 49 → 50 → 51 → 52 → pginit.

Order Branch Export commit Merge Round-1 findings fixed
1 fixr1-42 5411baa2 cab24792 F42-1 only the definite insufficient-resource SQLSTATEs are Capacity (53100/53200/53400/54000); 53300 is Busy, while 54001/54011/54023 keep a statement verdict. F42-2 StorageError::CapacityExceeded is a §10 capacity refusal with an explicit PersistenceError::Refused arm. F42-3 PG Phase C and the none settle clear the transient §6.1 kept_node closure. F42-4 the SQLite rebuild copies store_policy by explicit column list.
2 fixr1-43 b798a383 da8e19da F43-1 never settle an uncertain intent whose owner is dead (PG and per-holder pre-scans); a dead certain owner is still adopted. F43-2 branch.remove registers its retire-phase intent inside the marker transaction, so the intent is visible to the holder check. F43-3 a retried op replaces its predecessor's owner evidence.
3 fixr1-44 6a29ecbb bf7fb3d1 F44 one universal §4.2 visibility guard over every shipped branch/layer_stack statement, with a documented, proved allow-list; the §4.4 name comment; A9 records that the project-delete retry reads the marker plus the filtered by-id lookup instead of the unshipped layerstack_stack_by_id_retired.
4 fixr1-45 0ec97429 f2a5b876 F45-1 project.delete admission detail carries {kind:project,name} (the name flows through Effects::admit and DeleteAdmitFrame). F45-2 the same universal guard from the other branch; the merge keeps one test. F45-3 risk:delete names project/delete.rs, and the dead project/remove.rs paths and architecture links are gone.
5 fixr1-47 b4d67c91 f2730663 project.remove_file resolves against the Store tree and refuses a non-directory; the read path refuses a non-directory component; symlink and hidden-leaf path families are covered through the SDK.
6 fixr1-48 3a62729a e9ac1984 F48-1 the kept-waste reference cap is recorded at construction, so the §11 {value, cap} shape reaches the wire. F48-2 the PG rebuild decision uses the mark tuple's stored total (takes_rebuild_path), so a post-mark publication cannot flip none to rebuild. F48-3/F48-4 signature-stamp and digest-root cleanup (F50's batched Phase C subsumes these; see below). F48-5 a retained Raw group whose inventory no longer explains its extent is read exactly after a rebuild.
7 fixr1-49 ff438425 b1534fcf F49-1 clear Apple's persistent-WAL flag before the staged WAL switch. F49-2 settle keeps the op record while the artifact set is not empty, and the staged request stays distinct so the recovery run reopens.
8 fixr1-50 6d05e2f3 c75c2fcf F50-2..11: root-keyed file_digest and exact (slot, stamp) content signatures; the Phase B ownership precheck; the gc_op table fingerprint with the C/I and T/F swap contract; batched Phase C over the persisted gc_dead set; per-test bindings; the catalog-derived scratch name and gc_op.op_id status decoding; 1 GiB pack-id batches; ANALYZE on the visited temp tables; scratch+WAL peakTempBytes.
9 fixr1-51 bb360f4c 0f03b428 machine.space is named in the machine manual rows; the call tables are rebuilt; store.status phases and errors are documented; the aio stub is named.
10 fixr1-52 4317fa89 e2c1a99c F52-1..7: the T12.3a matrix is regenerated to executed 8 / covered 30 / partial 16 / deferred 6; AC-44 (PG privilege + lost COMMIT), AC-01 (9 sigkill_at! arms + 8 CLI SIGKILL arms), AC-38 and AC-47 (whole artifact set) are named; the 10 GB capacity_measure and gc_bench receipts are committed; the store-ext4 PR base set is stated in AGENTS and the review policy.
11 fixr1-pginit f5720d8a b42ab420 F50-1: Phase C never deletes a live Init's init_operation, init_entry or init_host_file rows (acquisition release owns that cleanup); the regression parks Phase C with a statement trigger and proves the Init survives and completes.

Conflict resolution.

  • fixr1-44 and fixr1-45 both add
    engine/crates/layerfs-persistence/tests/visibility_guard.rs. The merge
    keeps fixr1-45's version (the superset allow-list wording and the
    clippy-clean test), not 44's copy.
  • fixr1-48 vs fixr1-50: F50's batched Phase C (compact/{op,cleanup}.rs,
    the persisted gc_dead set, exact (slot, stamp) signatures and root-keyed
    digests) subsumes F48-3/F48-4, so the merge keeps F50's op/cleanup and
    does not re-apply 48's signature/digest fixes. F48-1, F48-2 and F48-5 are
    kept, and the tests are unioned.
  • fixr1-pginit keeps Phase C's three Init tables untouched on top of F50's
    batching: the settle batch drops only its dead set, binding_hint_block and
    the transient kept_node.
  • Report/docs: the union keeps A8, A9, D5 and G7 in
    gc-42-decisions.md; the T12.3a report/matrix carries the regenerated
    8/30/16/6 counts, 44's AC-27 visibility_guard evidence and 52's committed
    10 GB receipts.
  • The 9ab2869c docs4/clippy4/opencost4 work is preserved: the two
    project.delete clippy hunks, the advisory_holder type alias, the
    standby_satisfied assert and the URL-or-keyword postgresql_open_cost
    all remain.
  • Two post-merge integration fixes: cargo fmt --all reflow (06e965ad), and
    e629a522 splits history/objects.rs's inherent impls into
    history/objects/impls.rs (the merged file was 1046 lines, over the
    999-line product boundary) and points pginit's test at F50's catalog-derived
    scratch name.

Docs in this PR

  • core/docs/issues/project-41/gc-42-decisions.md gains A8 (§11 report
    semantics: logicalDiscarded = canonical bytes of the collected objects,
    shared once, orphans included; backendReusable = source-Store pre-mark
    free-page bytes; zero vocabulary adds no-free-pages and no-shrink; the
    pre-release gc_op columns with no additive step), A9 (§4.2: the
    project-delete retry resolves the caller's recorded Stack with the
    layerstack_deleted_ms marker plus the filtered layerstack_stack_by_id, so
    no unfiltered layerstack_stack_by_id_retired is shipped), D5 (store.compact
    is local-only; a sqlite Store pushed to another machine is Unsupported
    before any effect and is compacted after store.push back to native) and
    G7 (the install-only lane stays deferred; the prototype and its 10 GiB
    measurements live in tools/bench/installlane/ and change no product code).
    D1, D3 and D4 remain deviations; D2 is now the settled fresh-copy build
    decision; D5 is the new local-only deviation.
  • A10 (§6.4/§11: deadBytesRefined equals deadBytesBound on every
    path, not only on none: the mark resolves the whole retained set and §7
    step 4 runs no second payload sweep, so no refinement exists to report; the
    field and its snake_case wire key stay, and a report never names a
    different refined figure on either backend).
  • docs/versioned/2.0.0/calls.md and
    docs/architecture/api/calls/coverage.md are reconciled to the table:
    calls 85 in twelve families (main 329a20be 76/10 → 85/12; the
    base 39a25138ea adds the home family); the manual's family counts and
    the missing call rows are added, with the feat(monitoring): machine and Sandbox logs, files and observed values on one bounded read route (layerfs-project#179) #712/feat(monitoring): machine and Sandbox metrics — processor counters, per-Sandbox rows, memory and pids controllers (layerfs-project#182) #717/feat(monitoring): machine.space — where a machine's data disk goes, and what is not LayerFS's #724 monitoring
    sections (processor counters, per-Sandbox usage rows and
    machine.space) and a home section. The coverage matrix was re-derived
    from the API call table and its generated block now has 85 rows in table
    order.
  • The API/control architecture docs name D5 (local-only compaction) and the
    runbook's deviation table and disk-headroom section carry the fresh-copy
    rebuild (old Store + about one new Store; no 2.9× peak).
  • The .agents/skills/layerfs-js-sdk and layerfs-sdk-observe guides are
    reconciled to the same 85 calls / twelve families table, with the
    doctor and logs family pages added and the branch, file, machine,
    sandbox and workspace pages extended to their current arguments.

Test evidence

Final head 53dac0441b (base layerfs_v2_0_0 39a25138ea; the Round-6
merges 91dd691e70 … fbf9864131, the base merge e9586546aa, and the
integration fixups b78c076252, b28e74a24f, c91c8f0b18 and
53dac0441b) was gated on the final tree:

  • cargo check --workspace exit 0; cargo fmt --all -- --check exit 0;
    cargo clippy --workspace --all-targets --no-deps -- -D warnings exit 0;
    python3 tools/check_product_boundary.py --skip-sdk PASS (1533 production
    Rust/SQL/JavaScript files). The postgresql build feature no longer exists
    (build(store): always compile the PostgreSQL engine beside SQLite #726), so no check passes a feature flag.
  • Default three crates (layerfs-persistence, layerfs-store-service,
    layerfs-store): 800 passed / 0 failed / 18 ignored (Round-5 head 791).
    The Round-6 branches add [Follow-up to #49] Recover remaining Exec overhead after shared-live-owner migration #50's dead-owner and uncertain-intent status arms
    and [v0.1.3][Phase 2] Fix CDC locality scaling #43's retained-Uncertain arm.
  • compact_metrics 5 + sqlite_compact 34 + sqlite_compact_crash 24 +
    sqlite_compact_fixtures 1 + sqlite_compact_matrix 1 +
    sqlite_compact_units 7 = 72 passed / 0 failed. sqlite_compact is 34
    because [v0.1.3][Phase 2] Fix payload creation scaling #41 adds
    an_installed_leftover_reports_the_same_rebuild_as_a_completed_run; the
    T12.3a report is current on the merged tree (the matrix test asserts it) and
    no regeneration was needed after the base merge: executed 8 / covered 30 /
    partial 16 / deferred 6 over the 60 rows.
  • calls_table 21 + gc_calls 8 + gc_trace 5 = 34 passed;
    publish_expected_base 9 passed.
  • The whole layerfs-api crate (77 targets, including the base's
    home_calls and machine_platforms_setup): 229 passed / 0 failed / 2
    ignored
    . The whole layerfs-control crate (including the base's
    home_get, home_layout and every doctor_* suite): 153 passed / 13
    failed / 2 ignored
    ; the 13 failures are the privileged kernel-mount cases
    (mount fusectl denied, kernel attach Device: No such file or directory)
    in kernel_application, kernel_mount_routes, observed_application,
    observed_control and r5_1_actual_daemon_commit — the same environment
    limit Round 5 recorded. Every home and doctor suite passes on this host.
  • PostgreSQL 17.12 at 127.0.0.1:55433 (Docker, trust auth), keyword
    conninfo host=127.0.0.1 port=55433 user=postgres dbname=postgres sslmode=disable, --test-threads=1: persistence, store, store-service and
    project suites = 824 passed, 0 failed, 18 ignored (Round-5 head 815).
    The new postgresql_status dead-owner/uncertain-intent arms and the
    synchronous_standby_names spacing units run.
  • tools/ci/test_store_ext4.sh --quick with LAYERFS_TEST_TMPDIR on ext4:
    5/5 suites (83 passed, 0 failed, 1 ignored).
    --full (PostgreSQL cases NOT_RUN, as in the earlier evidence): 6/6
    suites (888 passed, 0 failed, 19 ignored)
    . --list runs with no
    LAYERFS_TEST_TMPDIR and exits 0 (F52R4-5). The lanes cover the CLI
    process-kill arms, which now follow F41-1.
  • gc_bench example tests (cargo test -p layerfs-persistence --example gc_bench): 2 passed — the R730 host block with one detected disk is
    valid JSON and its quotes/backslashes are escaped (the fixr6-benchjson
    fix).
  • Typings against the freshly built native addons: node api/scripts/gen-node-types.mjs --check exit 0, python3 api/python/tools/gen_stubs.py --check exit 0, and
    api/python/test/test_stubs.py 8 tests, OK (6 passed / 2 skipped; mypy
    absent)
    .
  • MCP npm test (api/mcp, Node 24.22.2): 17 passed / 0 failed; the
    admin profile lists 78 tools (85 calls − the doctor family's one − the
    six monitoring calls a UI polls), every generated tool validates, and the
    stdout purity and native-addon paths pass. The api/node suite with the
    built addon and command: 63 tests, 58 passed, 5 skipped, 0 failed,
    including the 85-call table, the SDK-page inventory and the regenerated
    coverage.md.
  • Capacity: the two committed 10 GB receipts predate [v0.1.3] Redesign FUSE Exec around one shared live Workspace core #49 round 4 (see
    "Fresh-copy SQLite rebuild"); the newer 10 GB medium run on the R730 is
    rebuild median 122.077 s, and fixr6-benchjson makes an R730 host block
    valid JSON. The 50 GB arm was not run and no 50 GB Store was created on
    this head.
  • GitHub CI on the pushed head: store ext4 (--quick) success; the store ext4 (--full, PostgreSQL 17) lane is skipped by the workflow, and ci.yml does not run for pull requests into layerfs_v2_0_0, so the local gates above are the evidence for the rest.

Known limitations

  • The 50 GB capacity arm was not run for this integration and no 50 GB
    Store was created on this head. The owner-final 50 GB receipt in the
    runbook is the earlier owner measurement; the 30% arm remains [E].
  • The two committed 10 GB receipts predate [v0.1.3] Redesign FUSE Exec around one shared live Workspace core #49 round 4: both were taken
    on e824e8a7, the pre-head8 build path (no 256 MiB src.mmap_size source
    window; verify still ran quick_check), so their 133.076 s / 118.440 s
    medians are not the merged tree's numbers. They are qualification numbers
    on a synthetic generated fixture on overlayfs, with a warm page
    cache on one declared rotational-media host class, not a cross-host
    guarantee. The fresh-copy path is 2.08× faster than the old whole-Store
    VACUUM INTO on the same fixture (118.440 s vs 246.371 s); the earlier
    1.85× mixed two different generated fixtures and is withdrawn. The 10 GB
    medium run on the R730
    (warm cache) is the newer measurement — rebuild
    median 122.077 s — and fixr6-benchjson fixes the harness JSON bug
    that made that host's receipt invalid.
  • The whole layerfs-control crate was run on this head: its home and all
    doctor_* suites pass, and its 13 failures are the privileged
    kernel-mount cases (mount fusectl denied, no kernel attach device) that
    need a privileged FUSE host the integration machine is not; Round 5's
    limitation is unchanged.
  • PG runtime evidence is a local PostgreSQL 17.12 run on this head
    (127.0.0.1:55433, Docker, trust auth); the CI store-ext4 full lane is
    the gate. Both postgresql_open_cost conninfo forms pass, and
    postgresql_gc_fence::two_concurrent_branch_removes_in_one_project_are_ordered
    counts live Branches through the marker, so the two earlier PG failures
    are closed here.
  • Acceptance: current v0.3.0: Cloud native integration and S3-compatible persistence #52 dispositions are executed 8 / covered 30 /
    partial 16 / deferred 6 over the 60 rows, with no pending row and no
    SQLite gap. The 6 deferred rows are the PostgreSQL halves (AC-28, AC-35,
    AC-41, AC-42, AC-43, AC-60). Partial rows are named in
    core/docs/issues/project-41/gc-t12.3a-sqlite-matrix-report.md.
  • [v0.1.3][Phase 2] Fix cross-file CAS deduplication scaling #42 stays open: A1–A11 and the A12 reading, with D1–D5, are recorded as
    a proposal for owner sign-off; the frozen spec is byte-identical. [v0.1.3][Phase 2] Optimize tiny_file_churn Exec and Commit as the first shared recovery step #46 (pruning) is deferred
    entirely; v0.3.0: Cloud native integration and S3-compatible persistence #52 stays open until the deferred PG rows and the 30%/50 GB arms
    run.

Related

Commits

Suggested final commit message:

GC 2.0: soft removal and Store compaction (#41, #42-#52)

Schema/markers, logical removals (branch.remove, project.delete,
project.remove_file), admission, mark/closure, SQLite and PostgreSQL
compaction, §11 report/status, API/MCP/CLI/SDK, qualification and the
fresh-copy build.

Integrates the nine Round-2 fix branches on the Round-1 fixed head 43e72cf5,
the seven Round-3 fix branches on d5cc1503, the five Round-4 fix branches on
a536f920, the six Round-5 fix branches on b357463b and the five Round-6 fix
branches on 7702bb21, then merges the layerfs_v2_0_0 base 39a25138ea (#730:
the home family, machine platforms, doctor host checks and the native unmount
reap; #727, the lp73 NFS integration, came earlier). The table is 85 calls in
twelve families after the base's home.init/home.get. Docs add A10
(deadBytesRefined == deadBytesBound on every path) and A12 (a rebuild carries
its own gc_intent row) beside A8, A9, D5 and G7.

Production LOC vs the base 57423382f7: +15,842 (171,524 -> 187,366).
Default suites: 800 passed / 0 failed / 18 ignored;
targeted SQLite 72 (34 sqlite_compact + 24 crash + 1 fixtures + 1 matrix + 7
units + 5 compact_metrics); calls_table/gc_calls/gc_trace 34;
publish_expected_base 9; layerfs-api 229; clippy
--workspace --all-targets -D warnings green; boundary PASS; ext4 --quick 5/5
(83 passed) and --full 6/6 (888 passed).
PostgreSQL 17.12 serial: 824 passed / 0 failed / 18 ignored. Node and Python
typings checks exit 0; MCP npm test 17/17 and api/node 58 passed / 5 skipped.
The 50 GB arm was not run; both committed 10 GB receipts predate #49 round 4,
the fresh-copy path is 2.08x the whole-Store VACUUM INTO on the same fixture,
and the newer R730 10 GB medium run is 122.077 s.

@yifanxuaaa yifanxuaaa left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of #711 (GC for layerfs-project#41)

Verdict: request changes. This is a read of the code at d26ae548e1 against the base spec (cf6e268b3b); nothing was built or run. Paths are at the PR head. Items marked ✔ were re-checked by hand in the source.

The scope is impressive and the SQLite safety core is solid: I found no path that drops an object still reachable from retained history, and a crash leaves either the old or the new Store. The blockers are repo process, admission correctness and PostgreSQL data safety.

What is done right

  • SQLite admission is atomic. A removal's intent scan and its marker run in one BEGIN IMMEDIATE; the mount insert re-validates the Branch/project with filtered statements, so a mount and a removal cannot both succeed. Bind registers its intent before the whiteout, so a marker landing in between is still applied.
  • Closure is complete. kept_node_mark.sql follows every §6.1 edge (including merged_from_id/displaced_id, retired Branches reached through commit.branch_id / layer.source_branch_id). The object closure is Store-wide (cross-project sharing survives) and follows each stored delta base to a fixpoint. inode_value_block is kept whole (item 37); packs with any live object are kept whole; pack_unit is pruned only for dead packs.
  • SQLite install is crash-safe. Sidecar checks on both files, sealed tuple/size/mtime/change counter re-checked before install, sync_all, atomic rename, directory fsync; the op record is temp-file + rename. Verify checks foreign keys, counts, live-heads digest, allocator tuple and totals; sqlite_sequence and the immutable triggers are restored.
  • Rules followed: no automatic retry loops, busy_timeout 0, the Store opens Disposable explicitly; the three removal calls take identity only; project.remove is retired; markers insert once with ON CONFLICT DO NOTHING; NameTaken comes from the unfiltered name checks (item 15).
  • PostgreSQL: schema names are validated and identifiers quoted (no injection); the swap including DROP SCHEMA is one transaction; the scratch schema is recorded in gc_op before CREATE SCHEMA; the totals_applied gate prevents double subtraction on resume; autovacuum stays enabled.

Blockers: process and repo rules

  1. ✔ The PR silently reverts #694. All five AGENTS.md files, docs/general/parallel-worktrees.md and docs/research/README.md are byte-identical to pre-#694 versions, and docs/research/history/agents-guides/README.md is deleted. This is also why the PR is CONFLICTING (on AGENTS.md). Please drop these files from the diff.
  2. ✔ The frozen spec is rewritten to match the code. LayerFS-GC-spec.md (+1003/−233) changes Status to "Frozen … approved by the owner", adds a sign-off block and pins a baseline ed14ed53b that is not in this repo. Contract changes with no #42 decision entry: §6.3 drops "bounded memory"; §11 redefines logicalDiscarded / backendReusable; §8 moves dead-row deletes from Phase B to Phase C; §5 adds the gc_intent / sidecar / retry design; item 16 switches remove_file to visibility-only. (Item 16 permits that as a fallback, but it has to be declared as the fallback. The S11 PostgreSQL relaxation is backed by item 34.) Contract changes need a #42 entry first, not an edit inside the implementation PR.
  3. ✔ 999-physical-line ceiling exceeded: engine/crates/layerfs-persistence/src/backend/sqlite/compact.rs (2260) and .../postgresql/compact.rs (1560).
  4. ✔ No commit records production LOC. With the base branch's tools/production_loc.py: core 39,795 → 40,416 (+621), engine 75,029 → 83,504 (+8,475), control 42,777 → 44,171 (+1,394), api 6,084 → 6,193 (+109); combined 163,685 → 174,284 (+10,599).
  5. ✔ Twelve new docs under core/docs/issues/project-41/ (runbook, rollout, capacity, reports). The documentation policy's exception covers only LayerFS-GC-spec.md; issue records belong in the issue/PR.
  6. Unexplained core change: core/crates/layerfs-storage/src/encoding/delta/select.rs and save/pooled.rs change delta-base selection on every normal Save. The selection walks bypass the per-Save DepthCache and re-walk the whole stored chain per candidate, and some candidates are now stored FULL. There is no rationale in the commit and no Save speed/storage measurement. Please split it into its own PR.
  7. The item 44 benchmark home (examples/gc_bench.rs, gc_fixture.rs) is missing.

Correctness: high

  1. ✔ PostgreSQL has no generation fence outside compaction. Nothing in the PostgreSQL backend outside compact.rs checks catalog_incarnation or produces StaleStore (session/profile.rs is unchanged). A one-shot commit on another machine publishes or dedups objects at G; compaction marks them dead and swaps; the history publish lands at G+1 referencing dropped location rows. That is committed data lost.
  2. ✔ The PostgreSQL mark runs inside the publication lane. mark_gc is self.write(mark_gc) (history/objects.rs:191), with one locate round trip and a full decode per object. Every writer on every machine waits on the lane, then gets Busy, for the whole mark; one long transaction also holds back vacuum. Tree-sized work in the lane is a defect under the Store concurrency model.
  3. ✔ Admission rows of a crashed holder block removals forever. GcIntent::new sets owner_runtime/owner_boot to None (core/crates/layerfs-history/src/contract/records.rs:806-828) and no intent builder fills them in (store-service/src/history/admission.rs). classify cannot tell a dead owner from a live one. After a killed daemon, a crashed one-shot private bind or merge, or a crash during a PostgreSQL mark (the compaction intent is registered at postgresql/compact.rs:1518 before gc_op at :1283, and settle_leftover returns None without gc_op), every later remove/delete/compact on that scope is MountsActive/Busy, with no recovery call. project.delete also registers its removal row (control/crates/layerfs-store/src/project/delete.rs:469) before the sidecar write, so a crash between them leaks an unrecorded row.
  4. ✔ PostgreSQL project.delete takes the Store key exclusively. advisory_keys always starts with Store key 1 and register locks every key with the intent's own sharedness (engine/crates/layerfs-persistence/src/history/admission.rs:315-328, 435-452). A delete is Busy while any mount is live anywhere in the Store, and during it mounts in unrelated projects fail MountsActive. The spec wants shared kind 1 + exclusive kind 2.
  5. settle_leftover can terminate live sessions (postgresql/compact.rs:815-856). It runs before the exclusive key is taken and treats idle as dead. A leftover gc_op from a swap that ended Busy/Conflict leads the next compaction to pg_terminate_backend a healthy worker/SDK connection, or a running compaction that is idle between transactions.

Correctness: medium

  • ✔ remove_file visibility fails open. store-service/src/read/visibility.rs:57-72 turns any marker-read error into "no markers" (.ok() / unwrap_or_default()), so read/stat/grep/glob/blame/compare/export (project/export/walk.rs:79) return the removed bytes. The mount whiteout (removal.rs:90-106) handles this correctly.
  • ✔ PostgreSQL ACL copy grants on the old table. grant_of(line, schema, table) emits GRANT … ON TABLE {schema}.{table} for the live table just before it is dropped (postgresql/compact.rs:1041-1094 vs :917), so non-owner roles lose access after the swap. PUBLIC and grant options are also dropped.
  • ✔ Unfinished Init/merge rows are wiped without refusal. sqlite/compact.rs:1127-1146 deletes merge_* and init_* unconditionally (the PostgreSQL Phase C does the same); §5 requires refusing while they exist.
  • ✔ Memory grows with Store size. The mark holds visited/widths/bases/logical/queued as BTree maps (objects.rs:368-440); object_location is loaded twice; PostgreSQL binds every visited id as one bytea[] (postgresql/compact.rs:290-309), which exceeds the 1 GB field limit around 30M objects. keptWaste.references is uncapped (S7).
  • SQLite rebuild uses about 3× the disk: full VACUUM INTO (dead packs included), a DELETE with a rollback journal, then an in-place VACUUM. The spec allows old + live + visited. Your own 50 GB receipt shows 143 GB temp and a 71-minute build.
  • Open Sandboxes are no longer checked: can_remove_project was deleted, so an open Sandbox with nothing mounted, or one on an unreachable machine, no longer refuses with Busy.
  • Report fields: logicalDiscarded sums the history-tree logical of collected Commits/Layers (shared trees counted once per Commit, 0 for orphan objects) instead of canonical bytes of collected objects; backendReusable is read from the new file (about 0); a resumed PostgreSQL G+1 run reports physicalShrink = 0 / "no-waste".
  • synchronous_commit floor lost: effective_sync_rank reads current_setting after BEGIN already ran SET LOCAL with the Store setting, so the swap commits with the default off (postgresql/compact.rs:898-900).
  • Transaction poolers are not refused for mounts/compaction (item 34 requires a direct or session-pooled connection); session advisory locks behind a transaction pooler leak or silently fail.
  • SQLite exclusion is detected late: another open client leaves -shm, and the seal fails with Conflict only after the full mark (sqlite/compact.rs:600-610), not Busy at admission.

Lower

  • Fork from a Branch removed in a race: fork re-reads the source inside the transaction with unfiltered branch_any (history/branch.rs:317).
  • Two PostgreSQL branch.remove calls in one project can deadlock (40P01 surfaces as a generic failure).
  • Merge/Init holders are reported as MountsActive, not Busy; a failed shared lock is always MountsActive.
  • project.stats (read/tree_stats.rs) counts entries hidden by remove_file.
  • The SQLite compaction releases the lane before settle, so a concurrent compactor's op record can be removed.

Benchmark receipts

The 10 GB / 50 GB SQLite capacity numbers are useful as diagnostics (50 GB: none 27.2 min, rebuild 85.5 min, 5.10 GB freed, 143 GB temp), but cannot qualify anything:

  • the 10 GB arm reports a median of three runs (one sample per case);
  • one run used Durable (WAL/FULL), which is disabled;
  • no cache drop between arms, and the 50 GB none mark ran right after the fixture build with the data in page cache (the doc itself shows a 2.1× cache spread);
  • the fixture is only whole-file 131 KB objects, with no delta chains, cross-project sharing or history depth;
  • the 147.6 GB peak-temp figure is arithmetic, not a re-measurement;
  • PostgreSQL is not measured at all.

Suggested path

  1. Drop the docs reverts and rebase.
  2. Move spec changes to #42 decision entries, or back them out.
  3. Split the delta-selection change into its own PR.
  4. Fix items 8–12 and the fail-open visibility filter.
  5. Split both compact.rs files under the ceiling and add LOC records.
  6. Re-measure with a representative fixture.

Happy to re-review after that.

🤖 Generated with Claude Code

@mengmengmengqiang

Copy link
Copy Markdown
Contributor Author

Spec-based review (8 LayerFS workspaces)

Reviewed against core/docs/issues/project-41/LayerFS-GC-spec.md, one isolated workspace per area. Tests: ~280 scoped tests, 0 failures (real PG 17.12 for #50). Not merge-ready: 8 High/blocking + contract gaps.

Blocking

# Area Problem
1 #50 PG ACL replay happens before the old table is dropped; grants go to the dropped table, new table ACL is empty (reproduced on PG17). backend/postgresql/compact.rs:913-923
2 #50 PG standby precheck only looks at streaming > 0; ignores synchronous_standby_names and matching sync_state, so swap COMMIT can block. compact.rs:690-695
3 #43 PG remove_path reads frontier before bumping epoch; a commit in between is not covered by the whiteout. admission.rs:661/669
4 #43 epoch fence opens after commit/merge/layerstack pre-reads; a removal in the window is invisible and a Commit can land on a retired Branch. commit.rs:91/100, layerstack.rs:344/357, merge.rs:49/82
5 #43 mount intents have no owner evidence (gc_mark_uncertain unused); Unresolved unreachable; sandbox.close/machine.stop/daemon exit never release intents → permanent MountsActive.
6 #44/#45 project.delete retry: a stale sidecar can delete a re-created project; a retry with a live row removes the entry; a lost admit reply leaves no sidecar. project/delete.rs:415-423
7 #42 tools/store_blocks.py not updated for additive position columns; real 1.0→2.0 offline migration fails to match and produces an unopenable Store.
8 #49 S5/§7 step 2/4 not implemented: whole-Store VACUUM INTO + DELETE instead of copying retained rows; peak ≈ 2.9× the old Store. sqlite/compact.rs:997,1076

Contract gaps (P1)

  • gc_op.usename missing on both backends.
  • Busy never carries {entity,phase,op} or PG pid/client_addr/backend_start; client_addr absent from the tree.
  • retained_roots / get_retained_roots / remove_unreachable remain Unsupported stubs.

Notable Medium

  • keyed trace hash not implemented (trace_key unused; trace_salt write-only)
  • merge Removed refusal for unmaterializable whiteout missing
  • SQLite none path mutates kept_node/gc_intent, no visited scratch, peakTempBytes=0
  • content_signature cleanup lacks stamp <= mark_stamp
  • G+1 resume does not take the exclusive Store intent/key
  • PG privilege precheck misses table ownership / pg_signal_backend
  • no pg_attribute/attstorage fingerprint; Phase C single transaction; live build/cleanup phases never written
  • SQLite SQLITE_NOMEM misclassified as Unknown; missing fsync; .compact-op.tmp residue
  • acceptance-matrix PG evidence path mismatch; markBytes missing; PR CI not triggered for this base

Test evidence

Fixes are being prepared in batches; this PR should stay open until the blocking items are resolved or explicitly waived.

@yifanxuaaa

Copy link
Copy Markdown
Contributor

Follow-up: plan, compaction speed and test size

This follows the review above. All numbers are estimates derived from this PR's own 50 GB receipt (mark 784 s, build 4,254 s, verify 86 s, install 7 s); none is a measurement.

Speed against the design

Spec §13 and layerfs-project#42 item 24 estimate about 2–3 minutes of downtime for a 50 GB SQLite rebuild. This PR measures 85.5 min for rebuild and 27.2 min for none (which also holds the writer lane the whole time). Against the proposed targets: about 3.1× live bytes written (target ≤ 1.1×), peak disk old + 143 GB (target ≤ old + 1.1 × live), the build is about 3× a plain VACUUM INTO by construction (target ≤ +10%), and memory is unbounded.

The gap comes from three things the design assumed and the PR does not do:

  1. The mark decodes every live object. It resolves full bytes, runs FinalizedObject::admit and re-reads the chain for each object (history/objects.rs:380-440), at about 62 MB/s. That is doctor-grade work moved into the mark.
  2. The build moves about 141 GB to keep 45.8 GB: VACUUM INTO of the whole Store, a journaled DELETE, then an in-place VACUUM.
  3. Verify reads payloads back.

Suggested speed-ups, in order

# Change Estimated effect at 50 GB Contract
1 Mark reads metadata only: decode only roots, directories and chunk manifests; take delta bases from record headers; drop admission and hash checks; read in pack order with batched lookups; keep the visited set in a temp-file table mark 13–27 min → about 10–60 s; bounded memory already §6.2/§6.3, item 22
2 Cheap verify (foreign keys, counts, digests, totals; no payload readback) 86 s → seconds item 23
3 Build by clone + delete + incremental vacuum: std::fs::copy of the sealed file (copy_file_range, a reflink on xfs/btrfs), delete dead rows in the clone with journal_mode=OFF, incremental_vacuum until the free list is empty, verify, then rename. Stores made by Project Init already use auto_vacuum=INCREMENTAL (session/profile.rs:53-57), and lifecycle/reclamation.rs already wraps incremental_vacuum. Fall back to a single ordered live-row copy otherwise. build 71 min → about 2–6 min (xfs/btrfs), about 4–30 min (ext4); peak disk about 5–10 GB on reflink filesystems; cost follows dead bytes, not live bytes needs a #42 entry (S5/§3 wording); the single-copy fallback does not
4 Mark on a read snapshot outside the writer lane; take the lane only for build and install after the tuple check writers blocked by none: 27 min → 0 needs a #42 entry
5 PostgreSQL mark on a read-only snapshot outside the publication lane; = ANY batches of about 1,000 IDs; no payload transfer; visited set loaded by batched COPY remote mark: tens of minutes or hours → seconds or minutes; also fixes the lane blocker mostly already in the spec
6 PostgreSQL build trims: pack primary key after load, higher maintenance_work_mem, batches of about 1 GB modest —

Estimated total for a 50 GB rebuild: steps 1+2 alone save about 14 min (the build is 83% of the time). With step 3 the total is about 2.5–7 min on xfs/btrfs (about 78–83 min saved) or about 4.5–31 min on ext4. Writers are blocked for about 2–6 min instead of 85.5. A none run takes about 10–60 s.

Test size: use 10 GB, not 50 GB

The 50 GB runs cost hours per sample (fixture build 12.7 min, none 27 min, rebuild 85 min) and are not needed for development. Item 44 already splits this: agents run the small and medium fixtures within the measurement budgets, and the owner runs the 20–50 GB fixture, once, at the final identity.

  • Iterate on a small fixture (seconds per run), and qualify the medium arm at about 10 GB.
  • Report per-GB rates (downtime per GB live and per GB freed, bytes written per GB, peak disk ratio; T-11). Scaling to 50 GB is then explicit arithmetic, marked as an estimate.
  • Declare and enforce a cold cache for the 10 GB arm. Your own capacity doc shows the 10 GB Store fitted in page cache while the 50 GB one did not, which is why the 50 GB build was 9× slower per GB. Use a memory limit smaller than the Store (for example, a cgroup or container memory limit) or drop caches before each arm, and record which. One sample per arm, no Durable arm.
  • Fixture shape matters more than size. Add delta chains, content shared across projects and real history depth; whole-file 131 KB leaves alone cannot show the mark's real cost.

Suggested plan

  1. Drop the docs reverts and spec edits; rebase.
  2. Split along the build order: removals + admission → reachability → SQLite compaction → PostgreSQL → API surfaces; the delta/select.rs Save change in its own PR with measurements.
  3. Fix the correctness blockers first: PostgreSQL fence, admission owner identity, PostgreSQL Store key, fail-closed visibility, settle_leftover/ACL, Init/merge refusal and the open-Sandbox check.
  4. Prototype steps 1 and 3 on the small and 10 GB fixtures against a plain VACUUM INTO ([v0.1.3][Phase 2] Fix cross-file CAS deduplication scaling #42 item 24) before rebuilding the rest of compaction.

🤖 Generated with Claude Code

layerfs-integration added 20 commits November 17, 2026 09:29
Move fixr5-43's listed_unknown helper from project/delete.rs (1004 lines,
over the product-source boundary) into project/delete/remote.rs, the only
module that calls it, and re-export it at project::delete::listed_unknown
so the F43R5-2 unit test keeps its path. The product boundary is green
again (1526 files); fmt, check and the project_delete test pass.
The merged tree has fixr5-43's extra project_delete test and fixr5-50's
extra PostgreSQL compact test, so the runner-derived 'Verified here'
counts move project_delete 17 -> 18 and postgresql_compact 29 -> 30.
Regenerated with UPDATE_REPORT=1; the matrix test then passes without it.
…tore, and the mark records the Store compaction op
…ctor setup and native unmount reap

The upstream base advanced 24 commits past the 5742338 merge base while
#711 was in review. Merged into the R6 integration, keeping both sides:

- the new `home` family (`home.init`, `home.get`) joins the table: twelve
  families, 85 calls (equalling the 83 of the PR head plus the two home
  calls); the profile buckets are 38 work / 40 admin-only / 6 ui / 1 no
  tool, so `--profile admin` lists 78 tools, and every generated page and
  test says 85;
- `project.delete`/`project.remove_file` (this PR) replace the base's
  `project.remove`, and `branch.remove` stays, so the admin exception list
  and the SDK pages name both;
- the base's machine-platforms, doctor host checks, retained-child reaper
  and native unmount work are untouched by the PR's engine changes.
…der clippy, and count 85 calls

- `cargo fmt` on the R6 code (fixr6-41's compact op/install/receipt/run and
  its `sqlite_compact` arms, fixr6-43's `add_failure`).
- fixr6-43's `acquire` grew the run's `op` id for the retained Unknown state
  (F43R6-1) and crossed clippy's eight-argument bar; it is allowed with the
  reason, as the other wide recovery paths already are.
- the base's non-macOS `daemon_value` stub took `&mut Vec` where a slice
  does, which `-D warnings` refuses on Linux.
- `check_product_boundary.py` named `src/home/` a retired layerfs-api folder
  while the base re-added the `home` call family there; the family is no
  longer treated as retired.
- the base's `home.init`/`home.get` make the table 85 calls in twelve
  families: the API, glossary, flow, plugin, skill and 2.0.0 manual pages say
  so, `calls.md` gains its `home` family section, and the generated
  `coverage.md` block is regenerated (Node/Python/aio/CLI 85, MCP 78).
layerfs-integration added 5 commits November 17, 2026 12:02
The base's `home.init` is the table's first entry and takes no positionals,
so the mutation test's `python.table[0].args[0]` was undefined. Pick the first
call with an argument and find its Python entry by `family.call` instead.

Ran the whole api/node suite with the built addon and command:
63 tests, 58 passed, 5 skipped, 0 failed.
…ds, so a real host block parses; the example gains its two tests
fixr6-41 (F41-1) makes an installed `G + 1` leftover answer the totals the
run recorded instead of re-deciding to `none` on the already rebuilt Store.
The in-process crash matrix follows that, but `sqlite_compact_process_kill`'s
`AfterInstall` and `DuringSettle` arms still asserted `path = none` with
`G + 1 -> G + 1`; both now assert `rebuild` with `G -> G + 1` and, for an
installed leftover, a non-zero collected `dead_bytes_bound`. Caught by the
ext4 `--quick` lane; all 8 arms pass.

Also: merge fixr6-benchjson (gc_bench json_host doubled quotes + its two
example tests).
…he Store answers NoSuchBranch

`scope_references` asserted `InvalidInput` for the two fork references the
scope composes into its namespace (`from_commit = "other"`,
`from_branch = "other"`); the composed names (`agent.other`) are inside the
namespace and name no Branch, so the Store's answer is `NoSuchBranch`. The
other five references are still refused by the scope with
`InvalidInput: out of scope: branch`, and no Branch appears.

Reached at `7702bb21` too: the test is older than the base sync and no evidence
run of the earlier rounds included it. The rest of the crate is green
(229 passed / 0 failed / 2 ignored across its 77 targets).
@mengmengmengqiang mengmengmengqiang changed the title feat(gc): LayerFS 2.0 GC implementation (single PR) feat(gc): LayerFS 2.0 GC implementation Nov 17, 2026
@yifanxuaaa
yifanxuaaa merged commit 9203d99 into layerfs_v2_0_0 Nov 17, 2026
2 checks passed
@mengmengmengqiang

Copy link
Copy Markdown
Contributor Author

10 GB SQLite compaction — measured performance

Test: engine/crates/layerfs-persistence/tests/sqlite_compact_capacity.rs#capacity_measure (opt-in, CAPACITY_MEASURE=1), branch gc-2.0-final @ 7702bb211.
Fixture A: 701 x 99 x 131071 B + 6770 dead = 9,986,795,475 B (9.987 GB), 76,870 packs, 8.88% dead (none) / 10.00% dead (rebuild), 3 runs per arm.

Results (median of 3 runs)

Path Mac native (M1, 8C, 16 GB, APFS) Dell native (Xeon 88T, 125 GB, HDD RAID, ext4) Mac container (OrbStack, 8 GB) Recorded receipt (135 GB Xeon)
none 51.41 s 73.02 s 65.42 s 88.44 s
rebuild 61.44 s (warm) / 112.83 s (cold run 1) 110.96 s 126.28 s 118.44 s

Phase breakdown (rebuild)

Phase Mac native (warm) Dell native Receipt
mark 18.0–19.0 s 73.03 s ~74.5–75.7 s
build 40.9 s 17.36 s ~18.2–19.2 s
verify 0.91–1.11 s 0.21 s ~4.3 s
install 0.45–0.55 s 20.40 s ~19.2–23.7 s
build+install 41.3 s 37.76 s ~38–43 s

Dead bytes 1.011 GB (10.00%); physical shrink ~1.018 GB; peakTempBytes 9.144 GB; concurrent working set ~19.3 GB; packs 77,812 -> 70,100.

What the numbers say

  • Single-threaded and I/O-bound. Dell ran at 97% of one core, 541 MiB RSS; its 88 threads and 125 GB RAM were never engaged. Raising RAM is not the lever: three mark passes were flat within 0.2%.
  • Mark is per-pack access bound, not byte bound. Mac (SSD) 18–19 s vs Dell (HDD-backed RAID) 73 s -- ~4x; ~1 ms per pack access on rotational storage.
  • Build+install is the fair write comparison: Dell 37.76 s vs Mac 41.3 s (Dell ~1.1x).
  • Mac install is under-measured. 0.45 s native / 0.69 s container for a 9.14 GB sync_all + rename implies 13–20 GB/s of durable writeback, which is impossible; the filesystem did not force the target to media. Do not quote Mac per-phase install; compare build+install.
  • The 8 GB container RAM cap did not distort (working set 541 MiB; the 10 GB Store streams from disk). The distortion was virtualization/fsync, not memory.

Bottom line

  • Same performance class: both hosts land within +/-12% on the arm that matters (rebuild).
  • Mac native is fastest overall here because mark runs on SSD; Dell native is faster than the recorded receipt on both arms (+17.4% none, +6.3% rebuild) and its phase split matches the receipt, which validates the run.
  • Peak temporary disk 9.144 GB either way; total working set ~19.3 GB.

Caveats: the Mac run used rustc 1.99.1 and a macOS-patched test (3-line cfg / mem_bytes / /dev/fd change); Dell used rustc 1.87.0 on ext4 HDD RAID. Both used the same fixture bytes at the same commit. Dell logs: /home/dell/lf-gc10-logs/ (receipt-A.json, phases-A.txt, run-A.log).

@yifanxuaaa
yifanxuaaa deleted the gc-2.0-final branch November 18, 2026 08:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants