Skip to content

feat(gc): LayerFS 2.0 GC implementation (single PR) [fork mirror of #711] - #71

Open
mengmengmengqiang wants to merge 175 commits into
layerfs_v2_0_0from
gc-2.0-final
Open

mengmengmengqiang wants to merge 175 commits into
layerfs_v2_0_0from
gc-2.0-final

Conversation

@mengmengmengqiang

Copy link
Copy Markdown
Collaborator

Summary

Single PR for the LayerFS 2.0 GC implementation, consolidated from the staged
PRs for review. Tracks the GC design in Ephemeral-AI-Lab/layerfs-project#41
and sub-issues #42–#52; the contract is
core/docs/issues/project-41/LayerFS-GC-spec.md and the open decisions are
core/docs/issues/project-41/gc-42-decisions.md.

Area What shipped
Schema/migrations marker tables (branch_retired, layer_stack_retired, path_removed), project_epoch, gc_intent, gc_op, gc_dead, kept_node, serials, trace_salt; SQLite + PostgreSQL additive steps
Removals branch.remove, project.delete (home/worker, sidecar, retry), project.remove_file (visibility-only whiteouts)
Admission gc_intent, MountsActive/Busy/Unresolved, PG advisory keys, project-epoch writer fence; a SQLite compaction now refuses Busy on another process's open Store descriptions, and removals order concurrent runs on the project_epoch row
Mark/closure history closure, object/pack closure, keptWaste, live-heads digest
Compaction SQLite: seal, step 1.5, the fresh-copy rebuild (copy only the retained rows into a fresh Store — no whole-Store VACUUM INTO), install fence, crash/kill recovery. PostgreSQL: Phase M/A/B/C, gc_op/gc_dead, scratch, swap, totals, standby/lost reply
Report/status §11 report, store.status (SQLite + PG), bounded metrics/SLO
API/MCP/CLI/SDK 83 calls in eleven families (76 GC + 6 Ephemeral-AI-Lab#712 monitoring + machine.space), with the GC admin exceptions; docs and count tests at 83/11
Tests/CI/docs SQLite + PG suites, real SIGKILL / CLI-process kill, the 10 GB final capacity measurement, the ext4 CI lane (--quick/--full), runbook/rollout/capacity docs; the nine Round-2, seven Round-3, five Round-4 and six Round-5 fix branches and the Ephemeral-AI-Lab#720/Ephemeral-AI-Lab#725/Ephemeral-AI-Lab#726 base sit on the Round-1 head

Fresh-copy SQLite rebuild (D2)

The SQLite rebuild no longer stages a whole-Store copy and rewrites it. It
creates <store>.compact with the sealed Store's schema, copies only the rows
the mark retained (pack-as-unit), builds secondary indexes and triggers after
the bulk load, verifies the fresh file and renames it.

The final 10 GB measurement on e824e8a7 is three runs per arm on a
generated fixture (9.789 GB live, 83,559 packs): none at 8.8% dead,
rebuild at 10.0% dead after the dead-object add.

  • rebuild 133.076 s median wall (range 132.274–134.601 s, 1.75%
    relative range): mark 82.050 s, build 23.469 s, verify 6.060 s,
    install 21.011 s. Peak temp 9.836 GB (new file 9.832 GB +
    4.26 MB visited scratch) and physical shrink 1.093 GB.
  • none is 82.683 s median on the same fixture.
  • 1.85× faster than the superseded whole-Store VACUUM INTO build
    (246.371 s median on the earlier 9.987 GB fixture: mark 149.089 s,
    build 94.317 s, verify 0.312 s, install 1.916 s, ~2.9× peak). The 23.5 s
    fresh build is ~4× shorter than the old 94.3 s; the added time is the page
    scan (verify 6.1 s) and the sync_all of the freshly written 9.8 GB
    (install 21.0 s), both inside the 133 s total.
  • Qualifiers: synthetic generated fixture, overlayfs temp filesystem,
    warm page cache (drop_caches refused 6/6, so it was not dropped), one
    declared rotational-media host class. These are qualification numbers on
    that host, not a cross-host guarantee; no 50 GB arm was run.

Production LOC

Measured with the tree's own tools/production_loc.py on the final head.
Nonblank, non-comment first-party product lines; tests, docs, benches,
fixtures and generated artifacts excluded.

Scope main 329a20be base 57423382f7 head Δ vs base
core 40,408 39,795 40,408 +613
engine 83,269 76,947 88,897 +11,950
control 44,141 48,436 50,269 +1,833
api 6,186 6,346 6,456 +110
total 174,004 171,524 186,030 +14,506 (+8.46%)

reference and dsh are 0 on both sides. The Ephemeral-AI-Lab#727 base sync alone moves the
head from 184,030 to 185,983 (+1,953 product lines). The Round-4 branches are
test/doc-only except #43/#49/#50's small product changes; the Round-5
branches add 47 product lines (control +12, engine +35) through #43's
Init/delete naming, #50's PostgreSQL standby/holder/Phase M/status arms and
#49's install/identity fence, while #42/#52 are docs-only. The PR-only delta
is therefore the fresh-copy rebuild and the GC implementation itself. The
fresh-copy rebuild is engine-only (+389 lines over the 11c1896 snapshot:
the compact/ phase modules, the fresh-file build and its verification).

Integration

Round-5 fix branches on b357463b (six branches)

b357463b (pr711-head8) is the Round-4 fixed head. Review Round 5 produced
six fix branches, each forked from LayerFS commit 12020f69bea9 whose tree is
b357463b (only the .lww differs). Each was exported from the LayerFS
project and re-committed with parent b357463b (.git and the .lww
dropped), then merged with --no-ff in this order: 42 → 43 → 49 → 50 → 51 → 52.
The merges are dba16b1f80, cca723ac36, 93853740eb, 5654607e3d,
d05fe297b1, c3632887e6; c07297303b moves 43's listed_unknown helper
out of project/delete.rs after the added lines took that file over the
999-line product boundary.

Order Branch LayerFS head → export commit Merge Round-5 findings fixed
1 fixr5-42 12cea9b21d7a → f48dd2c24a dba16b1f80 #42 round-5 (docs/comments, F42R5-1…4) the decisions note adds A12 as a reading, not a twelfth amendment: a rebuild stages gc_intent empty and inserts only the run's own op_id row, so "never copied" and §5.1's "lives in <store>.compact" agree; the preamble/sign-off say A8–A12 and the heading reads "Contract amendments (11) and one reading". The A9 refusal text names store.push (not store.move) as the non-GC call that can still report the cache's InUse; the intent-sidecar doc lists the optional op_id; the two 10 GB receipts in the evidence basis carry the vintage caveat (F52R5-2 wording: both predate #49 round 4's mmap window and quick_check removal).
2 fixr5-43 1266b28b9200 → bc4f5c8156 cca723ac36 F43R5-1 an Init history answer lost after admission names the project and init phase: init_unknown_state is built before request.name moves and is attached to record_new_bindings, initialize_layerstack, the Store-name-taken check and the fork (a bare UnknownOutcome reaches the client as Unknown with {kind: "project", name}, phase: "init", no durable op — a definite BusyOp stays Busy). F43R5-2 a project.delete Store-listing failure that cannot be determined names the project too (remote::listed_stack wraps both the empty-map UnknownOutcome and the poisoned-Store Unknown with listed_unknown), so it never stays a bare Unknown {}; the new project_add_unknown and project_delete arms pin both.
3 fixr5-49 124b565f447c → b2d0757a97 93853740eb #49 round-5 (install fence) §7 step 7's install re-applies the sealed (dev, ino) identity; a mismatch is now Conflict("the sealed Store identity changed") with no writes, not StaleStore. A Store replaced between seal and install is therefore Conflict; StaleStore stays the §10 session/transaction fence for a handle that opens or transacts after the install. The test is renamed a_store_replaced_between_seal_and_install_is_conflict, and the AC-46/AC-59 rows, the regenerated report and the identity/SDK Store docs follow.
4 fixr5-50 12da9f6ffe2d → d88d4b3293 5654607e3d F50R5-1 a legacy bare synchronous_standby_names list is FIRST 1, not one synchronous standby per listed name (unit cases for "s1, s2" and "s1"). F50R5-2 the swap 55P03 holder query reads any granted table lock on the Store base tables, so an ordinary AccessShareLock reader is named (pid/client_addr/backend_start) with a reader-blocker test. F50R5-3 store.status projects the registered Phase M gc_intent while gc_op does not exist yet (id/name/mark phase, started, updated_ms, busy); a test parks the run inside Phase M on gc_dead. F50R5-4 A8 wording: only the after size is sampled inside the swap transaction; the before size is sampled before it opens.
5 fixr5-51 121751691772 → 1494a9e01c d05fe297b1 F51R5-1 the MCP admin-profile test now expects the documented 76 tools (83 calls − the doctor family's one − the six monitoring calls a UI polls), not table.length - 1 = 82; the check compares against the profile's own admin count. Test-only, no source change.
6 fixr5-52 1251cf43c9c4 → 780421de3d c3632887e6 F52R5-1 gc-42-decisions.md says seven gc_op columns (A8 and the PostgreSQL schema already carry seven); 42 carries the identical word change. F52R5-2 tools/ci/gc-bench.md §13 records the 10 GB receipts' vintage — both predate #49 round 4's 256 MiB src.mmap_size source window and its quick_check removal — and marks the ~4 s quick_check cost derived, not measured; the matrix runner's AC-57 note, the report body and the E3 bullet say the same, and the report is regenerated. No 10 GB re-run: the receipts are not replaced. F52R5-3 the store-ext4.yml quick-lane comment now says the api suite runs four targets instead of "one or two targets per suite".

Conflict resolution. Every branch shares b357463b as its three-way base,
and the six merges are clean. The overlapping files were read semantically:

  • 42 + 52 on gc-42-decisions.md. Both make the identical
    "six → seven gc_op columns" change, so the merge applies it once; 52
    then has no net change there. 42's A12 reading, store.push and sidecar
    lines stay.
  • 42 + 50 on the same file. 50's A8 paragraph (before size sampled
    before the swap, after size inside it) is a different region from 42's
    A12 insertion and request paragraph; the union keeps both.
  • 49 + 52 on the T12.3a report and the matrix generator. 49 edits
    the AC-46/AC-59 evidence names and notes; 52 edits the §13 vintage text
    and the AC-57 row, and regenerates the report. The union keeps both, and the
    merged runner regenerates the merged report (the sqlite_compact_matrix
    test asserts it).
  • 42 + 43 on project/delete.rs. Different regions (the intent-sidecar
    doc and the new listing helper); the merge is clean, but the added helper
    took the file to 1004 lines over the 999-line boundary, so c07297303b
    moves it to project/delete/remote.rs and re-exports
    project::delete::listed_unknown for the F43R5-2 test. The product boundary
    is green again (1526 files).

The frozen LayerFS-GC-spec.md is still byte-identical to ed14ed53b
(sha256 84e16978d92d14ee65eb10d38f8d2f5347455b84c2c5b2e49fd28f60282415cf,
84,894 bytes) on the merged head.

Round-4 fix branches on a536f920 (five branches)

a536f920 (pr711-head7) is the Round-3 fixed head. Review Round 4 produced
five fix branches, each forked from LayerFS commit 3230a02565d5 whose tree is
a536f920 (only the .lww differs). Each was exported from the LayerFS
project and re-committed with parent a536f920 (.git and the .lww
dropped), then merged with --no-ff in this order: 42 → 43 → 49 → 50 → 52.
The merges are dfdd055503, 3370046f3f, bc811e91ea, 19679fd104,
7b58d4eaa1.

Order Branch LayerFS head → export commit Merge Round-4 findings fixed
1 fixr4-42 122df99487a9 → 0242ff8ed0 dfdd055503 F52R4-1 the gc-42-decisions.md preamble counts are consistent: sixteen places (A1–A11 + D1–D5), the note adds A8–A11, and the schema paragraph's "four gc_op columns" becomes six before #50's physical_after (the merged tree says seven). F52R4-2 A11's "Where" names the moved unit tests in control/crates/layerfs-store/tests/compact_in_process_call.rs, not src/compact.rs (which keeps gc_refusal).
2 fixr4-43 129ccd5e1e57 → 355ff7d183 3370046f3f F43R4-1 named_unknown also fills an empty-map Unknown detail (a bare UnknownOutcome renders SdkValue::map::<0>([]), not Null), so project.get/project.list(store) and the worker project.delete.release path name the Store/entity and operation instead of returning Unknown {}. F43R4-2 project.init's registration/release UnknownOutcome reaches the client as Unknown naming {entity: {kind: "project", name}, phase: "init", op} (both the InitError::Project(ProjectError::History) and InitError::History arms), not Failed with a null detail; a definite BusyOp stays Busy with its operation detail.
3 fixr4-49 1261ccde6d5c → 21b401573e bc811e91ea #49 round-4 (verify, mmap) §7 step 6's verify no longer runs PRAGMA quick_check: the pass is schema identity, foreign keys, counts, live-heads digests, the allocator tuple and the recomputed totals, and quick_check/the doctor stay separate commands, so it reads no payload. The T12.3a text and AC-57 record the earlier ~4 s page scan as the then-present quick_check (removed here) rather than a current verify cost. §7 step 2's bulk copy applies PRAGMA src.mmap_size = 268,435,456 to the attached sealed source (the session default is no mmap) and returns the size SQLite applied.
4 fixr4-50 12c9ab78bfd2 → 3a747da6d9 19679fd104 #50 round-4 (PG status/sync/physical_after) PostgreSQL's gc_op gains physical_after, filled from the swap transaction's own pg_total_relation_size(pack, object_location) sample, so a save after the swap (AC-28) cannot inflate a resumed G+1 report's physicalShrink; Swapped.physical_after now carries it. store.status reports busy only while the op's gc_intent is present and the recorded (pid, backend_start, usename) is alive, and unresolved once the intent is released (PostgreSQL's form of SQLite's returned) or the owner is gone; the state != 'idle' test is dropped, so an idle but live owner of a released op is not busy. effective_rank no longer floors the swap's sync strength at local, so two off settings stay off (and the §8 step number is corrected to step 10).
5 fixr4-52 12780a4178a8 → 057e1ffb0e 7b58d4eaa1 F52R4-3 the untraceable 92–104 s superseded-build range is gone: the T12.3a text, AC-57 and the matrix generator record the 94.317 s median only and the recorded missing-range deviation, pointing at tools/ci/gc-bench.md §13. F52R4-4 the ci.yml/format.yml comments name the store-ext4 exception (PRs into main or layerfs_v2_0_0), and store-ext4.yml's quick-lane comment counts five cargo groups and the api group's four targets. F52R4-5 test_store_ext4.sh --list prints the commands before the LAYERFS_TEST_TMPDIR check, so the documented verbatim command exits 0 without the variable.

Conflict resolution. Every branch shares a536f920 as its three-way base.
Only three files overlapped, and each was read semantically:

  • 42 + 52 on gc-42-decisions.md. The two blobs are identical: 42
    fixes F52R4-1/F52R4-2 and 52 carries the same text, so the merge takes
    42's version and 52 changes nothing there.
  • 42 + 50 on the same file. 50 adds physical_after and the "both
    relation sizes sampled in the swap transaction" sentence; 42 had already
    moved kept_waste into the mark and re-wrapped the paragraph. The union
    keeps both: the paragraph now says seven columns, the mark writes four
    (op_id, mark_fingerprint, mark_visited, kept_waste) and the swap
    writes three (physical_before, physical_after, peak_temp).
  • 49 + 52 on the T12.3a report and its generator. Union: the merged
    generator carries both 49's removed-quick_check wording and 52's
    superseded-build missing-range deviation, and the report is generated from
    it, so AC-57 states both. 43's new unit tests had landed in product source
    (src/project/add.rs, src/unknown.rs), which the product-boundary guard
    forbids: 6efec405ba moves them to layerfs-store/tests/project_add_unknown.rs
    (unknown::named_unknown is #[doc(hidden)] pub), and formats 52's matrix
    generator. 983cbb4ca1 replaces 50's .map(format!).collect::<String>()
    artifact ids with one hex helper so clippy's format_collect stays green,
    and the T12.3a report is regenerated on the merged tree (the extra [upstream issue #365] Benchmark bundle: pjdfstest, fsx, fsstress, smallfile, mdtest, fio on the SDK route (v1.0.0) #43
    project_delete_worker test moves its count from 7 to 8).

Base sync to layerfs_v2_0_0 57423382f7 (Ephemeral-AI-Lab#727)

The integrated head is merged with the current layerfs_v2_0_0 base
57423382f7 (Ephemeral-AI-Lab#727, the lp73 integration: the NFS AppleDouble codec, spill and
per-inode companion layers, batched READDIRPLUS/LOOKUP facts and macOS mount
options) as commit b41ebabc93, a plain merge with no rebase and no force.
dd4a2cc40f is the merge base and the base moves 44 files. Its only file in
common with this PR is Cargo.lock (the base adds libc to layerfs-daemon,
the PR adds postgres to layerfs-persistence), and the merge is clean.

The previous base sync already removed the postgresql build feature (Ephemeral-AI-Lab#726),
so the PostgreSQL engine is always compiled; no gate uses a feature flag.

Against the new base 57423382f7 the PR is 146 commits, 481 files,
+62,710 / −1,426
.

Round-3 fix branches on d5cc150377 (seven branches)

d5cc150377 (pr711-head6) is the Round-2 fixed head. Review Round 3 produced
seven fix branches, each forked from LayerFS commit 1280f9b1df18 whose tree
is d5cc150377 (only the .lww differs). Each was exported from the LayerFS
project and re-committed with parent d5cc150377 (.git and the .lww
dropped), then merged with --no-ff in this order: 43 → 44 → 45 → 46 → 49 →
51 → 52. The merges are fee2ebcc49, 5ef2ce0153, 636469bfb1, 87418c8920,
30da9172f1, eddf7534ec, 02dca44dc1.

Order Branch LayerFS head → export commit Merge Round-3 findings fixed
1 fixr3-43 12e834134847 → 61212f265e70 fee2ebcc49 #43 a worker project.list reply whose answer is lost is Unknown, and the caller now names the Store it listed: project.family's worker listing, project.get and project.list of a named Store attach {entity: {kind: "store", name}, phase, op} through unknown::named_unknown. A shared intent's failed gc_register/gc_release keeps its SharedIntent scope, project and phase and reports Unknown through lookup::history_in with the operation id, instead of a bare refusal.
2 fixr3-44 122e15d7d904 → 05bf7e00d467 5ef2ce0153 F44R3-1 branch_any, published_layer and node_history_serial ran their one statement in autocommit with no search_path and qualified() served no {s} body for them, so PostgreSQL answered Integrity("persistence"). Added the three {s}-qualified bodies and postgresql_statement_schema: a guard that discovers every read_statement body in the crate, pins the keys it runs, fails when a body or key is added without a {s} body, and runs all of them against a real server (it also reads a brace body's { correctly, not a closure argument's parenthesis).
3 fixr3-45 128e61a33007 → a9cb4c374852 636469bfb1 #45 a sidecar in marked is already past the marker: its retry takes no admission, writes no marker, removes the worker's own entry through the new project.delete.entry worker op and then the caller's home entry, recording done. A record or listing that names a different Stack than the sidecar is a re-created project: the stale intent is discarded and the call is NoSuchProject, so the new project's entry is never touched. Listing/read refusals keep their History wire code through lookup::published (identity, project.family, project.listing) instead of failed's generic code; the 2.0 manual and worker-op docs follow.
4 fixr3-46 127e6f7e49e5 → 671eef261ebc 87418c8920 F46R3-1 remove_unreachable deleted a Stack's unreachable Layer and Commit rows but store_totals.logical is maintained by AFTER INSERT triggers alone, so the deleted rows' logical bytes stayed counted. The prune now retracts exactly this Stack's unreachable logical bytes in the same write transaction, while the rows the statement reads still exist (history_remove_unreachable_totals.sql), with kept_node and sqlite_compact regressions.
5 fixr3-49 125c76ce3fe7 → fe0f790e2b55 30da9172f1 #49 returned/BusyOp a <store>.compact-op record gains the returned marker written before the call replies, so a run that already answered is a leftover the next call judges: store.status stays unresolved for its record even while the recorded pid lives, and busy only for an in-flight live owner. A live in-flight op is refused with BusyOp naming the record's entity, phase and operation. Install/run/op, the report/status projection, the sqlite_compact returned-run regressions and the docs follow.
6 fixr3-51 12ebf0439213 → ad5640140a77 eddf7534ec F51R3-1 store.compact's open-cache refusal for a call still running on the Store in this process is the GC vocabulary's Busy naming the Store with a null phase/op, not the cache's InUse (which stays for the non-GC store.move/store.remove/store.disconnect); the store.compact error row and focused tests pin it. F50R3-2 AC-44's evidence and matrix text gain the three Round-2 postgresql_compact tests (ended owner without the Store key, live key holder Busy, no pg_signal_backend).
7 fixr3-52 122cdc91330d → d3b967de7b5d 02dca44dc1 F52R3-1 LayerFS-GC-spec.md is byte-identical to ed14ed53b again: D1/D2/A10 stay only in gc-42-decisions.md, and the runbook states the frozen text is unchanged (the D2 peak/journal deviation is recorded there). F52R3-2 the report-only store-ext4 lane is documented for PRs into main or layerfs_v2_0_0, not "any base". F52R3-3 / F42R3-1 / F50R3-1 the nonexistent postgresql build feature is removed from the Store, project, front-end, versioned and doctor docs (always built, selected by the connection string). F42R3-2 the logicalDiscarded/backendReusable comments follow A8's semantics.

Conflict resolution. Every branch shares d5cc150377 as its three-way base.
The overlapping files auto-merged to the stated ownership except the generated
matrix report, and each overlap was then read semantically:

  • 51 + 52 on LayerFS-GC-spec.md. 52 restores the whole frozen file
    to ed14ed53b; 51 adds five §10 lines. A plain merge kept both, so the
    final tree takes 52's file byte-for-byte and 51's rule moves to
    gc-42-decisions.md as A11 (A. Contract amendments (11)), with 51's
    store.compact docs row and its unit/matrix tests kept. The spec diff against
    ed14ed53b is empty.
  • 46 + 49 + 51 on the T12.3a report. Every branch regenerates the
    report from its own test sources and each merge conflicted there; the report
    is regenerated from the merged test sources (33 sqlite_compact,
    7 project_delete_worker, 17 project_delete, 6
    compact_report_status), so the derived counts match the merged tree.
  • 43 + 45 on project/family.rs. Union: 45's lookup::published
    import and live_projects code with 43's named_unknown wrapping on the
    worker reply, project.get and the named-Store project.list.
  • 49 + 52 on store.md and 51 + 52 on store.md union: 49's
    returned-run status paragraph and 51's Busy row survive 52's docs
    corrections; 52's removed postgresql-feature sentences win.

Integration fixups.

  • 719b5b1651 restores the frozen spec, adds A11 and regenerates the T12.3a
    report on the merged tree.
  • 3dc3453173 moves the two Round-3 source unit tests out of product source
    (the boundary guard forbids #[cfg(test)]/#[test] there):
    51's compact::gc_refusal mapping tests to layerfs-store
    tests/compact_in_process_call.rs (gc_refusal is #[doc(hidden)] pub),
    and 45's published() test to the store-service history_wire_codes
    target where the other lookup wire-code tests live; it also runs
    cargo fmt --all over 44's unformatted PostgreSQL statement-schema test.
  • a536f920fa55 finishes the A11 references in gc-42-decisions.md (the
    header count and the [upstream issue #379] A Commit refused for disk Capacity leaves its packs in the Store, and every later write is refused Capacity #42 sign-off request name A1–A11 / A8–A11).

Round-2 fix branches on 43e72cf5 (nine branches)

43e72cf5 (pr711-head5) is the Round-1 fixed head. Review Round 2 produced
nine fix branches, each forked from LayerFS commit 126d2796b7dd whose tree is
43e72cf5 (only the .lww differs). Each was exported from the LayerFS
project and re-committed with parent 43e72cf5 (.git and the .lww
dropped), then merged with --no-ff in this order: 43 → 44 → 45 → 46 → 47 →
49 → 50 → 51 → 52. The merges are 5868fb3cef, ebcf856041, 11f05dc72c,
df5d31ac10, 837e04a54d, 157f62c285, a956794f2a, 06621c7869,
aaeed8f900; the integration head before the base sync is 3b1aa01ab3.

Order Branch LayerFS head → export commit Merge Round-2 findings fixed
1 fixr2-43 120794668f2c → 270998cfefc4 5868fb3cef F43R2-1 (§10 A6) a lost worker reply now names the entity and the operation: HistoryError::UnknownOutcomeOp carries UnknownOutcomeState {scope_kind, entity, phase, op}, with_unknown attaches it at the call boundary, and unknown::named_unknown fills the {entity:{kind,name}, phase, op} detail (branch/project calls, compact, project.delete home and worker, remote). F43R2-2 gc_intent_ops_{branch,project,store}.sql see a live Init and merge as well as the removal/compaction subset (never another mount), and the visibility-guard allow-list reason says so.
2 fixr2-44 126f9243c75c → e238710a9164 ebcf856041 F44R2-1 the PostgreSQL read side is generation-fenced: the expected incarnation rides as a trailing column on each autocommit read's statement and in the BEGIN batch of a transaction (the AccessShareLock it holds blocks a swap), run_observed keeps the diagnostic observational; retired backup heads — read_heads filters branch_retired/layer_stack_retired, so a backup digest names the visible history; the inline visibility guard gains the statement-set tests.
3 fixr2-45 12fbfc773896 → 8f20720ebc98 11f05dc72c F45R2-2/3 a fresh project.delete resolves the project only through the Stack its Store lists now: a retired project is NoSuchProject (never finished from a stale home record), and a fresh call that meets Retired/Absent removes its sidecar and keeps the entry; §10 Busy detail names the recorded Sandbox; F45R2-2/4 a stale record without a sidecar is NoSuchProject; inventory and the 2.0 manual say project.delete.
4 fixr2-46 12d9639ebc5e → 6d36cc25f2cf df5d31ac10 F46R2 the unreachable-history mark and its prune run in one write transaction (kept.rs, catalog.rs, history_unreachable_count.sql); a cleared mark's prune is a post-compaction no-op; kept_node tests.
5 fixr2-47 125f89fab84e → 98e45d499641 837e04a54d F47R2 only-symlink strictness is restored: an intermediate symlink is ill-typed and refused, a regular-file intermediate is merely absent; marker existence is lenient; project.remove_file is project-wide — it checks the path's grammar against every visible tree (main's head and every Branch head), so a Branch-only or never-existing path is accepted.
6 fixr2-49 1232319559eb → 1b7aae1ca1fd 157f62c285 F49R2-1 a <store>.compact-op record whose owner is provably live is not a leftover: the engine asks the caller's process facts (CompactRequest.owner_live), so a run that released the writer lane but has not settled stays Busy instead of being adopted; the settle removes the record only while it still names the operation being settled (install/run/op/types), with a before_settle_hook test window.
7 fixr2-50 123a2ff9b23e → 5de0eaf88746 a956794f2a F42R2-1 (also #42) the §8 lost-COMMIT reply is resolved by the next store.compact on a second connection: a recorded owner that no longer holds the exclusive advisory key is terminated (pg_terminate_backend), one that still holds it stays Busy; A8's gc_op columns are completed (op_id, mark_fingerprint, mark_visited); Phase C deletes each project's lineage/derived closure in its per-project transaction and records the cleanup phase before the first batch; postgresql_compact grows the lost-reply, lineage and phase tests.
8 fixr2-51 12a9510a10a9 → ec6620a36c4f 06621c7869 F51R2-1/2/3 api/node/src/index.d.ts and the Python _client.pyi/aio.pyi are regenerated from the 83-call table; docs/architecture/api/calls/coverage.md is regenerated; store.md's zero_reasons wording is fixed.
9 fixr2-52 124cd7bf9388 → 31faeb43c8ee aaeed8f900 F52R2-1..4 / F48R2-1 the T12.3a report's "Verified here" counts are derived from the test sources (a stale count now fails the matrix test), the 2.08x whole-Store-to-fresh-copy comparison is cited with the whole-Store receipt, A10 records deadBytesRefined == deadBytesBound on every path, the 50 GB receipt is named in the runbook/report, and the store-ext4 lane is wired in CI.

Conflict resolution. Every branch shares 43e72cf5 as its three-way base,
and the overlapping files auto-merged to the stated ownership; each overlap was
then read semantically:

  • 43 + 45 on control/.../project/delete.rs: 45's fresh/retry split and
    fail-closed NoSuchProject keep 43's lookup::history_in naming on the
    admission and marker-error paths (and 43's remote.rs/admit detail).
  • 43 + 44 on tests/visibility_guard.rs: 44 owns the statement-set tests;
    43's only change there is the gc_intent_ops_project.sql allow-list reason,
    which matches 43's widened SQL, so the union is the correct text.
  • 43 + 47 on engine/.../methods/project.rs: 47's project-wide
    visible_trees check with 43's history_in on gc_remove_path.
  • 43 + 49 on control/.../store/compact.rs: 49's owner_live field with
    43's history_in(..., Store, store, None, Some(&op_id)).
  • 49 + 50 on engine/.../history/admission.rs: 49's pub(crate) current_namespace with 50's store_key_held_by (the lost-COMMIT guard).
  • 49 + 52 on sqlite/compact/{mod,types}.rs: 49's OwnerLiveness /
    owner_live / before_settle_hook with 52's A10 module and field docs.
  • 50 + 52 docs: the A8 six-column gc_op paragraph (50) and A10 (52)
    are both kept; gc-runbook.md carries 50's automatic lost-COMMIT
    resolution and 52's A10 paragraph and 50 GB/whole-Store receipts; the
    matrix report's AC-52/AC-57 rows carry both.
  • 46's kept.rs/catalog.rs and 52's matrix/report/receipts do not
    overlap; 51's typings/coverage regenerate from the unchanged 83-call table.

Integration fixups. 3b1aa01ab3 moves 45's new fresh argument into the
existing Finish struct: the added positional argument pushed
mark_and_finish to eight arguments and failed
clippy::too_many_arguments under -D warnings. d5cc150377 regenerates the
T12.3a report on the merged tree (the derived counts are sqlite_compact 30,
sdk_path_hidden_semantics 3, sdk_path_removal 5 and postgresql_compact
29).

Base sync to layerfs_v2_0_0 dd4a2cc40f (Ephemeral-AI-Lab#720/Ephemeral-AI-Lab#725/Ephemeral-AI-Lab#726)

The integrated head is merged with the current layerfs_v2_0_0 base
dd4a2cc40f as commit ae0a4c4c2f, a plain merge with no rebase and no
force. The new base carries Ephemeral-AI-Lab#720 (a namespaces command cancelled before its
handle was told ends with SIGKILL, not SIGTERM), Ephemeral-AI-Lab#725 (per-Workspace
CPU/memory/pids in sandbox.get; the Sandbox group holds no process) and Ephemeral-AI-Lab#726
(the PostgreSQL engine is always compiled beside SQLite, removing the
postgresql feature from every crate).

The merge conflicts are all the PR's own additions meeting Ephemeral-AI-Lab#726's cfg
removal:

  • engine/crates/layerfs-persistence/src/lib.rs: the feat(gc): LayerFS 2.0 GC implementation Ephemeral-AI-Lab/layerfs#711 exports
    (compact_metrics, internals, report, the sqlite compact and
    kept/mark re-exports, OwnerLiveness) are kept without the removed
    postgresql gates.
  • engine/crates/layerfs-persistence/src/diagnostic/mod.rs: pub(crate) mod postgresql_contract is kept without gates.
  • engine/crates/layerfs-store-service/tests/postgresql_open_cost.rs: the
    URL-or-keyword conninfo::with_host proxy fix is kept without the
    #![cfg(feature = "postgresql")] header.

Because Ephemeral-AI-Lab#726 removes the feature, the PR-side code and harness that still
assumed an opt-in build are adapted in the same merge: the 61 remaining
cfg(feature = "postgresql") gates
on Ephemeral-AI-Lab#711 code and tests are removed, the
three not(feature) stubs (store compact_postgresql, store
compact_status, admission's postgres() fallback) are deleted or flattened,
tools/ci/test_store_ext4.sh stops injecting --features postgresql, and the
store-ext4 workflow comment, testing.md, limitations.md,
examples/README.md, parallel-worktrees.md and the matrix report generator
describe the always-built engine. --features layerfs-persistence/postgresql
is consequently no longer a meaningful gate; every check below runs without
it.

The ancestry below this head is unchanged: the nine Round-2 merges above, the
earlier base-snap (main 329a20be) export, the eight
fix711-{contract,pg,life,rm,sqlite,api,perf,hyg} merges, the
fix711-{spec,bench,pgtest,perf2} merges, the in-place SQLite/PostgreSQL
compaction split (7223b8d3f2), the eleven Round-1 fix branches and the four
e824e8a7 fix branches all remain in history.

Against the current base dd4a2cc40f the PR is 114 commits, 465 files,
+60,349 / −1,409
; the final head is d5cc150377.

Round-1 final integration (eight branches on 11c1896a01)

This head integrates the eight final fix branches onto the Ephemeral-AI-Lab#711 head
11c1896a01 (LayerFS commit 1201e45276…). Every branch is forked from that
head; each was exported from the LayerFS project and re-committed with the PR
head as its parent (.git/.lww dropped), then merged in order:

Order Branch Head Merge
1 impl-freshcopy 12b4547a932f 0b41b16d08
2 fix711-none 1244bbb8334c 687612b39e
3 fix711-report2 129c4b59c7b6 1cb3751423
4 fix711-conc 12d812591a5d f52e932aca
5 fix711-remote 1287d209cd48 ec1c018588
6 fix711-docs 12955309f113 0f5b1b4f1c
7 fix711-bench3 12c1bcb00afb b315ac91ec
8 fix711-install 12f9403e810d c80de3bc02

Conflict resolutions are semantic, since all eight share the PR head as their
three-way base:

  • Fresh copy wins on engine/.../sqlite/compact/build.rs (the rebuild is
    the fresh-copy load); fix711-report2's Built.free_bytes removal is kept
    with it, because backendReusable is now the source Store's pre-mark
    free-page bytes.
  • fix711-conc wins on history/admission.rs (open-description admission
    and epoch-row removal locks), preserving fix711-none's register_in
    in-transaction call from the mark.
  • fix711-report2 wins on the PostgreSQL report (canonical
    logicalDiscarded, source backendReusable, resumed-run report), with
    fix711-remote's markBytes metric kept.
  • fix711-none wins on sqlite/compact/run.rs (the none path's mark and
    intent are one rolled-back transaction), with fix711-conc's Busy probe
    and fresh-copy's build integration around it.
  • fix711-remote and fix711-docs win on their docs (control/architecture
    docs and the call manual respectively).
  • gc-42-decisions.md is a union: A8 (fix711-docs), D5 (fix711-remote),
    G7 (fix711-install), the fresh-copy D1/D2 rewrite (impl-freshcopy), the
    rollback-only D3 (fix711-none) and the fixture-basis note
    (fix711-bench3); the count is now thirteen places = eight A + five D.
    gc-runbook.md, the T12.3a matrix report and tools/ci/gc-bench.md carry
    the same union (D5 row, fresh-copy and old-build capacity numbers, fixture
    shape/cold-cache caveat, scratch sizing).
  • Fixups after the merges: the runbook names D5 with the other deviations;
    rustfmt on the fresh-copy build and matrix text; the six report/receipt
    unit checks moved out of product source into
    tests/{sqlite,postgresql}_compact_units.rs (the product-boundary guard).

Base sync to layerfs_v2_0_0 4a8efe63b3 (Ephemeral-AI-Lab#718/Ephemeral-AI-Lab#719/Ephemeral-AI-Lab#724)

The head is then merged with the current layerfs_v2_0_0 base 4a8efe63b3
(Ephemeral-AI-Lab#718 a forced native stop cancels running commands first, Ephemeral-AI-Lab#719 each exec's
helper and command inside their cgroups, Ephemeral-AI-Lab#724 machine.space) as commit
5e73795d98, a plain merge with no rebase and no force. The resolution keeps
the GC implementation, the fresh-copy rebuild and the remote/docs work, and
takes the new machine, Sandbox and monitoring code with its docs, CI and
tools/check_product_boundary.py audit-note updates. The only files both
sides changed are the call-count files: the merged table is 83 calls in
eleven families
(76 GC + six Ephemeral-AI-Lab#712 monitoring + machine.space). The
work/admin buckets stay 38/38 (the MCP server lists 76) and the UI-only
set becomes six; calls_table, the generated MCP profile table, the coverage
matrix and the 2.0.0 manual are at 83. Follow-ups 6daca0bc39 and
89fdbde155 keep the machine.space manual rows and the 83-call counts in
docs/versioned/2.0.0/calls.md, the calls page, the SDK page inventory and
the calls module. Against base 4a8efe63b3 the PR is 68 commits, 430 files,
+54,843 / −1,302.

The ancestry below this head is unchanged: the earlier base-snap (main
329a20be) export, the eight fix711-{contract,pg,life,rm,sqlite,api,perf,hyg}
merges, the fix711-{spec,bench,pgtest,perf2} merges, and the in-place
SQLite/PostgreSQL compaction split (7223b8d3f2) all remain in history.

Final fix branches on e824e8a7 (docs4 → clippy4 → opencost4)

e824e8a7 is a test-only fix on top of 89fdbde155: the PG concurrent-remove
fence now counts live Branches through the branch_retired marker instead
of raw count(*) FROM branch, so the long-standing postgresql_gc_fence
expectation matches the soft-removal design (§4.4).

The last three branches fork from pr711-head4 (LayerFS 12d276e2, whose
tree is byte-identical to e824e8a7 apart from the LayerFS .lww). Each was
exported and re-committed with parent e824e8a7 (.git and .lww
dropped), then merged in order; no conflicts — docs vs code vs one test:

Order Branch Export commit Merge
1 fix711-docs4 5be3f41c07 cf6e107804
2 fix711-clippy4 8b2cf7748e dcacc2ddac
3 fix711-opencost4 a9b1042da1 9ab2869cf2
  • fix711-docs4 brings the SDK/observe skills and the versioned references
    to 83 calls in eleven families (the new doctor and logs pages,
    branch.commits/get_commit/remove, file.revision, sandbox.files,
    the machine and store/backup tables) plus the docs/plugins and
    docs/versioned/2.0.0 README/migration/SDK pages.
  • fix711-clippy4 clears the workspace clippy warnings under -D warnings
    across 10 files: test assertions, needless format!/return, the
    advisory_holder type alias, the write! owner token, doc-comment
    formatting.
  • fix711-opencost4 makes postgresql_open_cost form-agnostic: it takes the
    upstream host:port from postgres::Config and rewrites the endpoint with
    conninfo::with_host, so the proxy accepts a URL or a libpq keyword
    connection string.

Round-1 fix branches on e824e8a7 (11 branches)

Final counts after this integration: 93 commits (25 on top of 9ab2869c),
450 files, +57,881 / −1,358 against the base 4a8efe63b3; final head
43e72cf538.

Review Round 1 produced eleven fix branches. Each was exported from its LayerFS
branch (.git and .lww dropped) and re-committed with parent
e824e8a7
, then merged with --no-ff in this order: 42 → 43 → 44 → 45 → 47
→ 48 → 49 → 50 → 51 → 52 → pginit.

Order Branch Export commit Merge Round-1 findings fixed
1 fixr1-42 5411baa2 cab24792 F42-1 only the definite insufficient-resource SQLSTATEs are Capacity (53100/53200/53400/54000); 53300 is Busy, while 54001/54011/54023 keep a statement verdict. F42-2 StorageError::CapacityExceeded is a §10 capacity refusal with an explicit PersistenceError::Refused arm. F42-3 PG Phase C and the none settle clear the transient §6.1 kept_node closure. F42-4 the SQLite rebuild copies store_policy by explicit column list.
2 fixr1-43 b798a383 da8e19da F43-1 never settle an uncertain intent whose owner is dead (PG and per-holder pre-scans); a dead certain owner is still adopted. F43-2 branch.remove registers its retire-phase intent inside the marker transaction, so the intent is visible to the holder check. F43-3 a retried op replaces its predecessor's owner evidence.
3 fixr1-44 6a29ecbb bf7fb3d1 F44 one universal §4.2 visibility guard over every shipped branch/layer_stack statement, with a documented, proved allow-list; the §4.4 name comment; A9 records that the project-delete retry reads the marker plus the filtered by-id lookup instead of the unshipped layerstack_stack_by_id_retired.
4 fixr1-45 0ec97429 f2a5b876 F45-1 project.delete admission detail carries {kind:project,name} (the name flows through Effects::admit and DeleteAdmitFrame). F45-2 the same universal guard from the other branch; the merge keeps one test. F45-3 risk:delete names project/delete.rs, and the dead project/remove.rs paths and architecture links are gone.
5 fixr1-47 b4d67c91 f2730663 project.remove_file resolves against the Store tree and refuses a non-directory; the read path refuses a non-directory component; symlink and hidden-leaf path families are covered through the SDK.
6 fixr1-48 3a62729a e9ac1984 F48-1 the kept-waste reference cap is recorded at construction, so the §11 {value, cap} shape reaches the wire. F48-2 the PG rebuild decision uses the mark tuple's stored total (takes_rebuild_path), so a post-mark publication cannot flip none to rebuild. F48-3/F48-4 signature-stamp and digest-root cleanup (F50's batched Phase C subsumes these; see below). F48-5 a retained Raw group whose inventory no longer explains its extent is read exactly after a rebuild.
7 fixr1-49 ff438425 b1534fcf F49-1 clear Apple's persistent-WAL flag before the staged WAL switch. F49-2 settle keeps the op record while the artifact set is not empty, and the staged request stays distinct so the recovery run reopens.
8 fixr1-50 6d05e2f3 c75c2fcf F50-2..11: root-keyed file_digest and exact (slot, stamp) content signatures; the Phase B ownership precheck; the gc_op table fingerprint with the C/I and T/F swap contract; batched Phase C over the persisted gc_dead set; per-test bindings; the catalog-derived scratch name and gc_op.op_id status decoding; 1 GiB pack-id batches; ANALYZE on the visited temp tables; scratch+WAL peakTempBytes.
9 fixr1-51 bb360f4c 0f03b428 machine.space is named in the machine manual rows; the call tables are rebuilt; store.status phases and errors are documented; the aio stub is named.
10 fixr1-52 4317fa89 e2c1a99c F52-1..7: the T12.3a matrix is regenerated to executed 8 / covered 30 / partial 16 / deferred 6; AC-44 (PG privilege + lost COMMIT), AC-01 (9 sigkill_at! arms + 8 CLI SIGKILL arms), AC-38 and AC-47 (whole artifact set) are named; the 10 GB capacity_measure and gc_bench receipts are committed; the store-ext4 PR base set is stated in AGENTS and the review policy.
11 fixr1-pginit f5720d8a b42ab420 F50-1: Phase C never deletes a live Init's init_operation, init_entry or init_host_file rows (acquisition release owns that cleanup); the regression parks Phase C with a statement trigger and proves the Init survives and completes.

Conflict resolution.

  • fixr1-44 and fixr1-45 both add
    engine/crates/layerfs-persistence/tests/visibility_guard.rs. The merge
    keeps fixr1-45's version (the superset allow-list wording and the
    clippy-clean test), not 44's copy.
  • fixr1-48 vs fixr1-50: F50's batched Phase C (compact/{op,cleanup}.rs,
    the persisted gc_dead set, exact (slot, stamp) signatures and root-keyed
    digests) subsumes F48-3/F48-4, so the merge keeps F50's op/cleanup and
    does not re-apply 48's signature/digest fixes. F48-1, F48-2 and F48-5 are
    kept, and the tests are unioned.
  • fixr1-pginit keeps Phase C's three Init tables untouched on top of F50's
    batching: the settle batch drops only its dead set, binding_hint_block and
    the transient kept_node.
  • Report/docs: the union keeps A8, A9, D5 and G7 in
    gc-42-decisions.md; the T12.3a report/matrix carries the regenerated
    8/30/16/6 counts, 44's AC-27 visibility_guard evidence and 52's committed
    10 GB receipts.
  • The 9ab2869c docs4/clippy4/opencost4 work is preserved: the two
    project.delete clippy hunks, the advisory_holder type alias, the
    standby_satisfied assert and the URL-or-keyword postgresql_open_cost
    all remain.
  • Two post-merge integration fixes: cargo fmt --all reflow (06e965ad), and
    e629a522 splits history/objects.rs's inherent impls into
    history/objects/impls.rs (the merged file was 1046 lines, over the
    999-line product boundary) and points pginit's test at F50's catalog-derived
    scratch name.

Docs in this PR

  • core/docs/issues/project-41/gc-42-decisions.md gains A8 (§11 report
    semantics: logicalDiscarded = canonical bytes of the collected objects,
    shared once, orphans included; backendReusable = source-Store pre-mark
    free-page bytes; zero vocabulary adds no-free-pages and no-shrink; the
    pre-release gc_op columns with no additive step), A9 (§4.2: the
    project-delete retry resolves the caller's recorded Stack with the
    layerstack_deleted_ms marker plus the filtered layerstack_stack_by_id, so
    no unfiltered layerstack_stack_by_id_retired is shipped), D5 (store.compact
    is local-only; a sqlite Store pushed to another machine is Unsupported
    before any effect and is compacted after store.push back to native) and
    G7 (the install-only lane stays deferred; the prototype and its 10 GiB
    measurements live in tools/bench/installlane/ and change no product code).
    D1, D3 and D4 remain deviations; D2 is now the settled fresh-copy build
    decision; D5 is the new local-only deviation.
  • A10 (§6.4/§11: deadBytesRefined equals deadBytesBound on every
    path, not only on none: the mark resolves the whole retained set and §7
    step 4 runs no second payload sweep, so no refinement exists to report; the
    field and its snake_case wire key stay, and a report never names a
    different refined figure on either backend).
  • docs/versioned/2.0.0/calls.md and
    docs/architecture/api/calls/coverage.md are reconciled to the table:
    calls 83 in eleven families (main 329a20be 76/10 → 83/11; the
    new base 4a8efe63b3 80/11 → 83/11); the manual's family counts and the
    nine missing call rows are added, with the feat(monitoring): machine and Sandbox logs, files and observed values on one bounded read route (layerfs-project#179) Ephemeral-AI-Lab/layerfs#712/feat(monitoring): machine and Sandbox metrics — processor counters, per-Sandbox rows, memory and pids controllers (layerfs-project#182) Ephemeral-AI-Lab/layerfs#717/feat(monitoring): machine.space — where a machine's data disk goes, and what is not LayerFS's Ephemeral-AI-Lab/layerfs#724 monitoring
    sections (processor counters, per-Sandbox usage rows and
    machine.space). The coverage matrix was re-derived from the API call
    table and its generated block now has 83 rows in table order.
  • The API/control architecture docs name D5 (local-only compaction) and the
    runbook's deviation table and disk-headroom section carry the fresh-copy
    rebuild (old Store + about one new Store; no 2.9× peak).
  • The .agents/skills/layerfs-js-sdk and layerfs-sdk-observe guides are
    reconciled to the same 83 calls / eleven families table, with the
    doctor and logs family pages added and the branch, file, machine,
    sandbox and workspace pages extended to their current arguments.

Test evidence

Final head 7702bb2114 (base layerfs_v2_0_0 57423382f7, still current; the
Round-2 head d5cc150377, the seven Round-3 merges fee2ebcc49 …
02dca44dc1, the five Round-4 merges dfdd055503 … 7b58d4eaa1, the six
Round-5 merges dba16b1f80 … c3632887e6, the base sync b41ebabc93 and the
integration fixups 6efec405ba, 983cbb4ca1, 66ff210e83, b357463b6c,
c07297303b and 7702bb2114) was gated on the final tree:

  • cargo check --workspace exit 0; cargo fmt --all -- --check exit 0;
    cargo clippy --workspace --all-targets --no-deps -- -D warnings exit 0;
    python3 tools/check_product_boundary.py --skip-sdk PASS (1526 production
    Rust/SQL/JavaScript files). The postgresql build feature no longer exists
    (build(store): always compile the PostgreSQL engine beside SQLite Ephemeral-AI-Lab/layerfs#726), so no check passes a feature flag.
  • Default three crates (layerfs-persistence, layerfs-store-service,
    layerfs-store): 791 passed / 0 failed / 18
    ignored
    (Round-4 head 786). The Round-5 branches add [upstream issue #365] Benchmark bundle: pjdfstest, fsx, fsstress, smallfile, mdtest, fio on the SDK route (v1.0.0) #43's named-Unknown
    arms (project_add_unknown, project_delete) and [upstream pr #504] Use VecDeque for Sandbox log tails #50's PostgreSQL
    standby/holder/Phase M/status cases; [upstream pr #505] Area P batch 1: P2, P14, P1, P7, P8 and commit cost accounting fix #49 renames one test and [upstream pr #498] feat(sandbox): 统一 Sandbox provider 接口,收编 Docker / Linux 原生 / macOS 原生(B1/B2/B3,叠在 #427 上) #52
    regenerates the report.
  • sqlite_compact 33 + sqlite_compact_crash 24 + sqlite_compact_fixtures
    1 + sqlite_compact_matrix 1 + sqlite_compact_units 7 = 66 passed / 0
    failed / 0 ignored
    . The T12.3a report is regenerated on the merged tree
    (7702bb2114): its "Verified here" list names project_delete (18) and
    postgresql_compact (30), the added F43R5-2 and F50R5-2 cases, and its
    derived dispositions are executed 8 / covered 30 / partial 16 / deferred 6
    over the 60 rows.
  • calls_table 21 + gc_calls 8 + gc_trace 5 = 34 passed;
    publish_expected_base 9 passed; compact_report_status is inside the
    default layerfs-store run.
  • PostgreSQL 17.12 at 127.0.0.1:55433 (Docker, trust auth), keyword
    conninfo host=127.0.0.1 port=55433 user=postgres dbname=postgres sslmode=disable, --test-threads=1: persistence, store, store-service and
    project suites = 815 passed, 0 failed, 18
    ignored
    . The new postgresql_status Phase M/reader-holder arms, the
    legacy synchronous_standby_names FIRST-1 units and the swap
    physical_before sampling wording all run.
  • tools/ci/test_store_ext4.sh --quick with LAYERFS_TEST_TMPDIR on ext4:
    5/5 suites (83 passed, 0 failed, 1 ignored).
    --full (PostgreSQL cases NOT_RUN, as in the earlier evidence): 6/6
    suites (879 passed, 0 failed, 19 ignored)
    . --list runs with no LAYERFS_TEST_TMPDIR and exits 0
    (F52R4-5).
  • Typings against the freshly built native addons: node api/scripts/gen-node-types.mjs --check exit 0, python3 api/python/tools/gen_stubs.py --check exit 0, and
    api/python/test/test_stubs.py 8 tests, OK (6 passed / 2 skipped; mypy
    absent)
    .
  • MCP npm test (api/mcp, Node 24.22.2): 17 passed / 0 failed; the
    admin profile lists 76 tools, not 82 (F51R5-1), every generated tool
    validates, and the stdout purity and native-addon paths pass.
  • Capacity: the 10 GB fresh-copy and gc_bench receipts are committed in
    engine/crates/layerfs-persistence/tests/fixtures/ (fresh-copy 118.440 s vs
    whole-Store 246.371 s on the same fixture = 2.08x). Both receipts'
    pre-head8 build/verify vintage is recorded in gc-bench.md §13 and AC-57
    (F52R5-2); the 50 GB arm was not run and no 50 GB Store was created on
    this head.
  • GitHub CI on the pushed head: store ext4 (--quick) success; the
    store ext4 (--full, PostgreSQL 17) lane is skipped by the workflow. The
    full layerfs-control crate was not rerun in Round 5
    (its only known failure is the unprivileged FUSE mount fusectl case; the
    touched layerfs-store crate is in the default gate).

Known limitations

Related

Commits

Suggested final commit message:

GC 2.0: soft removal and Store compaction (#41, #42-#52)

Schema/markers, logical removals (branch.remove, project.delete,
project.remove_file), admission, mark/closure, SQLite and PostgreSQL
compaction, §11 report/status, API/MCP/CLI/SDK, qualification and the
fresh-copy build.

Integrates the nine Round-2 fix branches on the Round-1 fixed head 43e72cf5,
the seven Round-3 fix branches on d5cc1503, the five Round-4 fix branches on
a536f920 and the six Round-5 fix branches on b357463b, then merges the
layerfs_v2_0_0 base 57423382f7 (#727, the lp73 NFS integration; PostgreSQL
was already always built beside SQLite after #720/#725/#726). Docs add A10
(deadBytesRefined == deadBytesBound on every path) and A12 (a rebuild carries
its own gc_intent row) beside A8, A9, D5 and G7.

Production LOC vs the base 57423382f7: +14,506 (171,524 -> 186,030).
Default suites: 791 passed / 0 failed / 18 ignored;
targeted SQLite 66 (33 sqlite_compact + 24 crash + 1 fixtures + 1 matrix + 7
units); calls_table/gc_calls/gc_trace 34; publish_expected_base 9; clippy
--workspace --all-targets -D warnings green; boundary PASS; ext4 --quick 5/5
(83 passed) and --full 6/6 (879 passed).
PostgreSQL 17.12 serial: 815 passed / 0 failed /
18 ignored. Node and Python typings checks exit 0; MCP
npm test 17/17. The 50 GB arm was not run; the 10 GB fresh-copy median is
118.440 s against the whole-Store 246.371 s (2.08x).

mengmengmengqiang and others added 30 commits November 15, 2026 23:54
…ab#711

Merge base-snap keeping the PR head tree, so later merges of the
fix711 branches use the main export as their three-way base while
the Ephemeral-AI-Lab#692/Ephemeral-AI-Lab#693 publish delta is preserved.
Integrates the fix711-contract branch onto the Ephemeral-AI-Lab#692/Ephemeral-AI-Lab#693 publish base as
part of PR Ephemeral-AI-Lab#711.
…fixes

Combines fix711-contract's holder diagnostics (OpConflict backend detail,
scope intents, session/transaction refusal) with fix711-pg's per-key
advisory modes, note_backend pooling proof and non-terminating owner
settlement. Conflicts: postgresql/compact.rs (standby Replica evaluation,
poll_owner_gone over terminate_owner), history/admission.rs (per-key
shared mode + session_lock_refusal).
Keeps lifecycle's unfinished-Init/merge guard in postgres compact and its
extra resume/phase tests, on top of pg's exclusive Store claim ordering.
Shared admission diagnostics keep the per-key advisory mode and scope_kind.
Conflicts: history error docs, gc_advisory_holder.sql, postgresql/compact.rs
(standby BusyOp, owner settlement, single Store claim), admission.rs
(duplicate EMPTY_OP / doc wording), postgresql_compact.rs (backend detail,
test ids, lifecycle tests).
Takes fix711-rm's sidecar-before-admission project.delete recovery and the
'retire' marker phase for branch.remove, keeping fix711-life's related
Sandbox guard and release_or_mark. Conflicts: project/delete.rs
(mark_and_finish recovery body), store-service history/admission.rs
(branch removal marker phase).
…ixes

Takes fix711-sqlite's compact.rs as authoritative for the SQLite intent
lifecycle (intent survives build/install, cleared at cleanup settle; abort
takes Option<&op_id>), its keep-through-install crash expectations, and its
transient kept_node clear, combined with fix711-contract's retained-roots
helpers. Conflicts: backend/sqlite/compact.rs, history/kept.rs,
tests/sqlite_compact_crash.rs.
Keeps fix711-rm's sidecar-before-admission delete body and lifecycle's
Related.can_remove_project in the tests, plus fix711-sqlite's capped
delta-base report imports. Adapts api's new tree_stats visibility filter
to fix711-rm's fallible Visibility::for_snap. Conflicts:
project/delete.rs, store_transfer_recovery.rs, restore_install.rs,
backend/sqlite/compact.rs.
Takes fix711-hyg's deliberate revert of the delta cycle-detection change
(out of this PR), its AGENTS/docs restructure, and its removal of the 12
implementation notes from core/docs/issues/project-41. Conflicts:
layerfs-storage delta/select.rs (cycle-detection revert),
modify/delete of the moved implementation notes.
…or import

Both fix711-pg and fix711-life added the same standby Replica test helper;
the merge kept both. Drop the duplicate and an import fix711-api no longer
uses in the SQLite compact test module.
- acquisition_layout: a live acquisition is unfinished work, so compaction
  refuses (fix711-life §5 guard) and the test releases the row first.
- branch_remove: Busy entity.kind reports the resource scope (branch), as
  fix711-contract's OpConflict now specifies.
- sdk_methods golden: ReservedName carries the spec-gap G1 empty detail.
- regenerate the SQLite acceptance matrix report (fix711-hyg had removed the
  generated evidence the matrix suite requires).
…intent test

fix711-contract's gc_intent schema CHECK requires a 32-byte Store scope_id;
the existing 16-byte placeholder fails registration with Integrity.
…e GC branch

Merges the new 2.0 base (Ephemeral-AI-Lab#712: machine/Sandbox logs and files, observed
values, doctor.fix, the hardened worker) into gc-2.0-final, keeping the GC
implementation (#42-#52, Ephemeral-AI-Lab#692, Ephemeral-AI-Lab#693, Ephemeral-AI-Lab#694).

Conflict resolutions keep both sides:
- worker_server.rs is Ephemeral-AI-Lab#712's accept loop plus connection/fault/ops modules;
  ops.rs carries the GC project.delete, project.delete.admit,
  project.delete.release and project.list operations (project.list added to
  the read-only set) beside Ephemeral-AI-Lab#712's doctor.fix and inspect.
- MCP profiles: the GC admin exceptions (project.delete, project.remove_file,
  branch.remove, store.compact) plus Ephemeral-AI-Lab#712's ui and none buckets.
- The call table is 82 calls in eleven families (GC +3: project.delete,
  branch.remove, store.compact; Ephemeral-AI-Lab#712 +6: sandbox.files, machine.stats,
  machine.files, logs.machine, logs.sandbox, doctor.fix).
- Docs, Node/Python call-count tests and the SDK pages take 82/11.
layerfs-integration added 30 commits November 17, 2026 08:11
…tion

SRC_MMAP_BYTES is a const, so the bare assert! folded to assert!(true)
and clippy's assertions_on_constants rejected it under -D warnings; the
check now reads the constant through std::hint::black_box.
…refusal owner, seven gc_op columns, receipt vintage caveat
Move fixr5-43's listed_unknown helper from project/delete.rs (1004 lines,
over the product-source boundary) into project/delete/remote.rs, the only
module that calls it, and re-export it at project::delete::listed_unknown
so the F43R5-2 unit test keeps its path. The product boundary is green
again (1526 files); fmt, check and the project_delete test pass.
The merged tree has fixr5-43's extra project_delete test and fixr5-50's
extra PostgreSQL compact test, so the runner-derived 'Verified here'
counts move project_delete 17 -> 18 and postgresql_compact 29 -> 30.
Regenerated with UPDATE_REPORT=1; the matrix test then passes without it.
…tore, and the mark records the Store compaction op
…ctor setup and native unmount reap

The upstream base advanced 24 commits past the 5742338 merge base while
Ephemeral-AI-Lab#711 was in review. Merged into the R6 integration, keeping both sides:

- the new `home` family (`home.init`, `home.get`) joins the table: twelve
  families, 85 calls (equalling the 83 of the PR head plus the two home
  calls); the profile buckets are 38 work / 40 admin-only / 6 ui / 1 no
  tool, so `--profile admin` lists 78 tools, and every generated page and
  test says 85;
- `project.delete`/`project.remove_file` (this PR) replace the base's
  `project.remove`, and `branch.remove` stays, so the admin exception list
  and the SDK pages name both;
- the base's machine-platforms, doctor host checks, retained-child reaper
  and native unmount work are untouched by the PR's engine changes.
…der clippy, and count 85 calls

- `cargo fmt` on the R6 code (fixr6-41's compact op/install/receipt/run and
  its `sqlite_compact` arms, fixr6-43's `add_failure`).
- fixr6-43's `acquire` grew the run's `op` id for the retained Unknown state
  (F43R6-1) and crossed clippy's eight-argument bar; it is allowed with the
  reason, as the other wide recovery paths already are.
- the base's non-macOS `daemon_value` stub took `&mut Vec` where a slice
  does, which `-D warnings` refuses on Linux.
- `check_product_boundary.py` named `src/home/` a retired layerfs-api folder
  while the base re-added the `home` call family there; the family is no
  longer treated as retired.
- the base's `home.init`/`home.get` make the table 85 calls in twelve
  families: the API, glossary, flow, plugin, skill and 2.0.0 manual pages say
  so, `calls.md` gains its `home` family section, and the generated
  `coverage.md` block is regenerated (Node/Python/aio/CLI 85, MCP 78).
The base's `home.init` is the table's first entry and takes no positionals,
so the mutation test's `python.table[0].args[0]` was undefined. Pick the first
call with an argument and find its Python entry by `family.call` instead.

Ran the whole api/node suite with the built addon and command:
63 tests, 58 passed, 5 skipped, 0 failed.
…ds, so a real host block parses; the example gains its two tests
fixr6-41 (F41-1) makes an installed `G + 1` leftover answer the totals the
run recorded instead of re-deciding to `none` on the already rebuilt Store.
The in-process crash matrix follows that, but `sqlite_compact_process_kill`'s
`AfterInstall` and `DuringSettle` arms still asserted `path = none` with
`G + 1 -> G + 1`; both now assert `rebuild` with `G -> G + 1` and, for an
installed leftover, a non-zero collected `dead_bytes_bound`. Caught by the
ext4 `--quick` lane; all 8 arms pass.

Also: merge fixr6-benchjson (gc_bench json_host doubled quotes + its two
example tests).
…he Store answers NoSuchBranch

`scope_references` asserted `InvalidInput` for the two fork references the
scope composes into its namespace (`from_commit = "other"`,
`from_branch = "other"`); the composed names (`agent.other`) are inside the
namespace and name no Branch, so the Store's answer is `NoSuchBranch`. The
other five references are still refused by the scope with
`InvalidInput: out of scope: branch`, and no Branch appears.

Reached at `7702bb21` too: the test is older than the base sync and no evidence
run of the earlier rounds included it. The rest of the crate is green
(229 passed / 0 failed / 2 ignored across its 77 targets).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant