diff --git a/.agents/skills/layerfs-js-sdk/references/methods/branch.md b/.agents/skills/layerfs-js-sdk/references/methods/branch.md index 0cc8d154ea..df913adea1 100644 --- a/.agents/skills/layerfs-js-sdk/references/methods/branch.md +++ b/.agents/skills/layerfs-js-sdk/references/methods/branch.md @@ -125,14 +125,10 @@ reply, never out of the merge. For an incomplete summary, page ## `branch.publish` -`branch.publish(project, branch, commit?, { expected_base, expected_head, force })` +`branch.publish(project, branch, commit?)` Makes `main` show the Branch's head, or the given Commit, by adding a Layer -to `main`. It refuses (`HeadMoved`) when `main` gained a Layer since your -Branch forked, so another agent's work is never dropped by accident. -`expected_base` (a Layer id) and `expected_head` (a Commit id) refuse unless -`main` and your Branch are still where you read them; `force: true` -overwrites `main` on purpose. +to `main`. ```bash layerfs branch publish shop/api work @@ -147,29 +143,29 @@ layerfs branch publish shop/api work | `kind` | Meaning | | --- | --- | -| `Added` | `main` now shows that Commit; `published_layer` is the new Layer; `overwrote` lists `main` Layers that a `force` publish skipped | -| `UpToDate` | That Commit is already published (`{ kind, layer }`), even if `main` moved on since | -| `NoChanges` | `main` already holds that tree (`{ kind, head_layer }`) | +| `Added` | `main` now shows that Commit; `published_layer` is the new Layer; `overwrote` lists `main` Layers that were skipped | +| `UpToDate` | `main` already shows it (`{ kind, layer }`) | +| `NoChanges` | Nothing to publish (1.1.0 shape; not seen in this pass) | | Code | Meaning | Do | | --- | --- | --- | | `NoCommit` | The Branch has no Commit yet | Commit something first | | `NoSuchBranch` | No such Branch | Check the name | -| `HeadMoved` | `main` moved since your Branch forked (or since `expected_base`), or the Branch head is not `expected_head`; `detail` has `expectedBase`/`actualBase` or `expectedHead`/`actualHead` | Merge with `main` into a new Branch (below), then publish that | -| `InvalidInput` | `commit` is not a full Commit id, or an expected id names the wrong kind | Pass a full id | +| `HeadMoved` | `main` moved during the call | Merge `main` in (below), then publish again | +| `InvalidInput` | `commit` is not a full Commit id | Pass a full id | -**Publishing replaces `main`'s view** with your Commit, so it is refused when -`main` has Layers your Commit doesn't contain. Publishing your own Branch -again at milestones is fine. If someone else published since you forked, -merge `main` in (a merge makes a new Branch), check the result, then publish -the merged Branch: +**Publishing replaces `main`'s view** with your Commit. Layers published to +`main` after your Branch's base that your Commit doesn't contain are skipped: +they stay in history and `overwrote` lists them. If someone else may have +published since you forked, merge `main` in first, check the result, then +publish the merged Branch: ```js const P = "shop/api"; const m = await layerfs.branch.merge(P, "work-release", "work", "main", { message: "merge main before publish" }); // mount m.branch in a Sandbox and run the tests here, then: const p = await layerfs.branch.publish(P, m.branch); -// a merge with main publishes as a fast-forward: nothing is skipped +if (p.kind === "Added" && p.overwrote.length) console.log("warning: skipped", p.overwrote); ``` ## `branch.diff` diff --git a/AGENTS.md b/AGENTS.md index b76053d386..fb2f6b79ca 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,392 +2,450 @@ > **Status:** Current general guide. -Instructions for agents working in LayerFS. Read this file, then the guide of -each layer you change: [core](core/AGENTS.md), [engine](engine/AGENTS.md), -[control](control/AGENTS.md), [API](api/AGENTS.md). A layer guide adds only -what differs in its layer. Current owner instructions and documented -supersessions govern; a closed handoff or historical receipt is not a new -assignment. Earlier wording of these guides is in the -[archived guide history](docs/research/history/agents-guides/README.md). - -## Rules you must not break - -| Rule | Where | -| --- | --- | -| Disposable / WAL / synchronous=OFF is the only SQLite Store profile; Durable execution is `NOT_RUN` | [Persistence profile](#active-persistence-profile--owner-direction-2026-10-07) | -| One attempted operation: no automatic retry, busy handler, refresh or replay | [§4](#4-code-build-and-docs) | -| Build tests with `--no-run` first; every test command stops at 120 s, and reaching it is a hang | [§4](#4-code-build-and-docs) | -| CI reports and never gates; verify your own change and report checks and gaps | [§4](#4-code-build-and-docs) | -| No new dependency an existing crate covers; no third-party patch, fork or vendor except the fuser Git fork | [§4](#4-code-build-and-docs) | -| Refactors delete tests that are tricky to adapt; never weaken a surviving assertion | [§4](#4-code-build-and-docs) | -| Architecture docs change in the same commit as the code | [§4](#4-code-build-and-docs) | -| Every commit records production LOC, split by core, engine, control and api | [Production LOC](#production-loc-comparison-for-every-commit) | -| A warm cache never credits a measured phase; one sample per case | [Measurements](#measurements) | -| Retired stays retired: the v0.1.6 reference, `layerfs-server`, standalone UI pages, removed platforms; fs-bench-pro is frozen | [Retired and frozen](#retired-and-frozen) | - -## Layers - -| Layer | Crates (root manifest) | Guide | +Repository-wide instructions for LayerFS. Also read the applicable [core](core/AGENTS.md), [engine](engine/AGENTS.md), +[control](control/AGENTS.md) and [API](api/AGENTS.md) guides for implementation in those layers. Current owner instructions and documented +supersessions govern; a closed handoff or historical receipt is not a new assignment. + +## Product scope and current workstreams + +**Implement both workstreams through the core, engine and API layers.** The v0.1.6 +reference implementation that lived under root `crates/` was retired on +2026-10-10 (R9c) and is absent at HEAD. It was never a dependency, source +include or fallback for core, and must not return as a second product. Its last +source is at local tag `reference-v0.1.6-final` (`d296981ac`, `crates` tree `498dd1917812`); +see the [R9c record](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/R9C-COMPLETION-20261010.md). +Historical evidence, receipts and verdicts are retained unchanged in Git +history: `core/docs/` (issue records, numbered papers, receipts) was removed +by #671 and stays readable at commit `af2ad92bce`. Link it there by pinned +URL; do not restore it to the tree. The +retirement is a legacy retirement in the source-size comparison, not a +simplification. + +| Workstream | Current scope | Entry point | | --- | --- | --- | -| core | `layerfs-content`, `layerfs-history`, `layerfs-storage`, `layerfs-telemetry` | [core](core/AGENTS.md) | -| engine | `layerfs-bridge`, `layerfs-daemon`, `layerfs-fuse`, `layerfs-mount-ports`, `layerfs-nfs`, `layerfs-overlay`, `layerfs-persistence`, `layerfs-project`, `layerfs-store-service`, `layerfs-workspace` | [engine](engine/AGENTS.md) | -| control | entities `layerfs-home`, `layerfs-machine`, `layerfs-store`, `layerfs-sandbox` (with its `workspace/` module); drivers `layerfs-platform`, `layerfs-bucket`, `layerfs-sandbox-backend`; and `layerfs-control` | [control](control/AGENTS.md) | -| api | the call surface `layerfs-api`, the `layerfs` command `layerfs-cli`, the Node binding `layerfs-node`, the PyO3 module `layerfs-python`; the JavaScript and Python packages under `api/` | [API](api/AGENTS.md) | - -- **Dependencies point down:** `core < engine < control < api`, including dev - dependencies and test includes; nothing in engine depends on control. The - control layer (`control/crates`, renamed from `connect/` by the owner - 2026-11-06, layerfs-project#101 D-6; #535 D4) holds the management code and - the crates that reach a machine or the network. -- **[Cargo.toml](Cargo.toml) decides membership**; a directory or old test does - not establish a built replacement. -- **Two workstreams, one set of layers.** Cluster one is the core algorithms - (Content, Storage, History, Telemetry) with engine Persistence and Project - Init: start with the [cluster-one handbook](docs/general/cluster_one_handbook.md) - and the [CAS/CDC/delta handbook](docs/general/cas_cdc_deltaencoding_handbook.md). - Cluster two is engine Workspace, FUSE, daemon and Sandbox with API execution - and Commit integration: start with the - [architecture guide](docs/architecture/README.md) and its - [flows](docs/architecture/flows/README.md). C1/C2/C5 in older documents mean - content/storage/history. -- **What establishes what.** Source and public contracts establish implemented - behaviour. Owner requirements and proposals establish target behaviour; - research establishes neither capability nor qualification. Preserve source - pins, limits, failures and open prerequisites. - -## Retired and frozen - -- **The v0.1.6 reference** under root `crates/` was retired on 2026-10-10 (R9c) - and is absent. It was never a dependency, source include or fallback for - core and must not return as a second product; its last source is at local - tag `reference-v0.1.6-final` (`d296981ac`, `crates` tree `498dd1917812`); see - the [R9c record](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/R9C-COMPLETION-20261010.md). - Its retirement is a legacy retirement in the source-size comparison, not a - simplification. -- **`core/docs/`** (issue records, numbered papers, receipts) was removed by - #671 and stays readable at commit `af2ad92bce`; link it there by pinned URL - and do not restore it. Historical evidence, receipts and verdicts stay - unchanged. -- **`layerfs-server` and the host-mediated SDK runtime** are retired, not - extended; build no host/server adapter in the data path. -- **The LayerFS UI is the dsh LayerFS plugin** (owner, 2026-10-19), the LayerFS - tab in DeepSeek Harness, in its own repository - [dsh-layerfs](https://github.com/Ephemeral-AI-Lab/dsh-layerfs) (moved out on - 2026-10-23, #428), which pins the LayerFS commit it builds on; this - repository holds no dsh code. Build no standalone HTML or TUI pages. The - `dsh-layerfs-demo` bundle is disabled permanently (owner, 2026-10-21): it - loads nothing, is not released and no guide installs it. -- **Firecracker, Microsandbox and SSH machines are removed** (owner, - 2026-11-08); see [Where calls run](#where-calls-run). -- **fs-bench-pro is frozen and left broken** (owner, 2026-10-18). Do not edit, - build, run or repair it, and leave it out of workspace builds and checks. - Its call sites broken by #352 F-020 (`ProjectApi` fork and commits, - `ForkRequest`, a `SandboxCreate` without `connect_host`) stay broken; a - product change never adapts it. Only an explicit owner instruction reopens - it. Benchmarks, harnesses and released data live in - [layerfs-benchmark](https://github.com/Ephemeral-AI-Lab/layerfs-benchmark) - (moved out of `benchmark/` on 2026-10-23; routing is its - [fs-bench-pro/AGENTS.md](https://github.com/Ephemeral-AI-Lab/layerfs-benchmark/blob/main/fs-bench-pro/AGENTS.md)). - Older documents' `benchmark/` paths are that repository's paths without the - prefix; the root harness for the retired reference is at tag - `reference-v0.1.6-final`. -- **Old prompts and research.** Completed Stage 0–6 prompts are historical. - The former #303 contracts and the - [optimization investigation](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/303/fuse-optimization-investigation.md) - are archived research that selects no algorithm, profile or capability. Do - not resume old loops, reopen closed rows or run old campaigns unless the - task selects that work, and do not choose authority by issue number alone. +| Cluster one | Core Content/Storage/History/Telemetry algorithms; Engine Persistence and Project Init | [Cluster-one handbook](docs/general/cluster_one_handbook.md), [CAS/CDC/delta handbook](docs/general/cas_cdc_deltaencoding_handbook.md) | +| Cluster two | Engine Workspace/FUSE/daemon/Sandbox; API execution and Commit/runtime integration | [Architecture guide](docs/architecture/README.md): engine and control topics and the [flows](docs/architecture/flows/README.md) | +| Legacy reference | v0.1.6, retired from `main` by R9c; read at the recovery tag only | [R9c record](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/R9C-COMPLETION-20261010.md) | + +**The LayerFS UI is the dsh LayerFS plugin** (owner, 2026-10-19): the LayerFS tab in +DeepSeek Harness. It lives in its own repository, +[dsh-layerfs](https://github.com/Ephemeral-AI-Lab/dsh-layerfs) (moved out of `dsh/` +on 2026-10-23, #428), which pins the LayerFS commit it builds on; this repository +holds no dsh code. +Do not build or revive standalone HTML or TUI pages. **The `dsh-layerfs-demo` +bundle (the LayerFS mock model and its demo workspace) is disabled permanently** +(owner, 2026-10-21): it loads nothing, is not released and no guide installs it. + +Cluster names are workstreams. C1/C2/C5 in older component documents mean +content/storage/history. Check [Cargo.toml](Cargo.toml) for active +members: a directory or old test does not establish a built replacement. +Source and public contracts establish implemented behavior. Owner requirements +and proposals establish target behavior; research establishes neither capability +nor qualification. Preserve source pins, limits, failures and open prerequisites. ## Active persistence profile — owner direction 2026-10-07 -- **Disposable / WAL / synchronous=OFF is the only permitted profile for a - SQLite Store.** Durable execution is disabled indefinitely, until the owner - explicitly allows it again; this supersedes earlier instructions to exercise - both profiles or resume Durable. A milestone, old benchmark selection, - default argument or passing build does not reauthorize Durable. -- Select Disposable explicitly for every new Store, application run, test, - diagnostic and measurement; never execute Durable or fall back to it. - Retained Durable code may compile; report its execution as - `NOT_RUN — disabled by owner until explicit reauthorization`. Historical - Durable receipts keep their verdicts. This policy does not claim the Durable - API was removed. -- Disposable's COMMIT does not checkpoint (`wal_autocheckpoint=0`); its writing - session's own thread checkpoints the WAL off the commit path (owner approval, - 2026-11-07; [SQLite checkpoints](docs/architecture/engine/store/sqlite.md#checkpoints)). -- A PostgreSQL Store has no profile; its write setting is `synchronous_commit` - (default `off`), recorded by `store.connect`. -- Each Workspace's daemon overlay is a separate MEMORY/OFF/EXCLUSIVE database - file. +**Disposable / WAL / synchronous=OFF is the only permitted global Store +profile. Durable execution is disabled indefinitely, until the owner explicitly +allows it again.** This supersedes earlier instructions to exercise both profiles +or to resume Durable after Disposable development. A milestone, old benchmark +selection, default argument or passing build does not reauthorize Durable. + +Select Disposable explicitly for every new Store, application run, test, +diagnostic and measurement. Do not execute Durable or fall back to it. Retained +Durable implementation may compile, but report its execution as +`NOT_RUN — disabled by owner until explicit reauthorization`. Preserve historical +Durable receipts and their original verdicts. This execution policy does not +claim that the Durable API has been removed. Each Workspace's daemon overlay +remains a separate MEMORY/OFF/EXCLUSIVE database file. +Disposable's COMMIT does not checkpoint (`wal_autocheckpoint=0`); its writing +session's own thread checkpoints the WAL off the commit path (owner approval, +2026-11-07; [SQLite checkpoints](docs/architecture/engine/store/sqlite.md#checkpoints)). ## LayerFS mental model -- **A Workspace is a complete mutable filesystem view** over an immutable - committed root, including `.git` and its index, ignored files, - dependencies, symlinks, caches and output; Git ignore rules do not filter - LayerFS state. -- **Granularity is free.** Per-tool-call orchestration is the common case and - needs fast bootstrap; per-task works too. A Workspace can serve many - sequential or concurrent calls, last long and Commit incrementally. - Workspace granularity, Exec duration and Commit cadence are independent. - Ordinary Bash has no automatic timeout, command-specific restore/install, - implicit Commit or automatic unmount; only explicit terminal unmount closes - local ownership. -- **Filesystem access needs no Exec.** Any process with mount visibility and - permission, including Bash launched by an external executor, gets the same - Workspace semantics through ordinary filesystem calls. The Sandbox or the - external executor owns command launch, streams, exit status and - cancellation; the filesystem daemon supervises no commands. See - [execution ownership](engine/AGENTS.md#filesystem-access-and-execution-ownership). -- **Mount is cheap.** Initial full-root acquisition is explicit. Mount binds - the complete root with no whole-tree scan, copy, materialization or - dependency restore; the only file it creates is the Workspace's overlay - database, copied from a template built before readiness and deleted at - terminal unmount. Demand-loaded metadata and content still pay real I/O. -- **Workspaces share nothing mutable.** A Workspace's metadata, payload and - operation records live only in its overlay file; its open-file, +- A Workspace is a complete mutable filesystem view over an immutable committed + root. Include `.git` and its index, ignored files, dependencies, symlinks, + caches and output; Git ignore rules do not filter LayerFS state. +- Support both per-tool-call and per-task orchestration. Per-tool-call is the + expected common case and needs fast bootstrap. A Workspace can serve multiple + sequential/concurrent calls, last a long time and Commit incrementally. +- Workspace granularity, Exec duration and Commit cadence are independent. + Commands may be short or long-lived in either mode. Ordinary Bash has no + automatic runtime timeout, command-specific restore/install, implicit Commit + or automatic unmount. Only explicit terminal unmount closes local ownership. +- Filesystem access does not require `workspace_api.exec` or an Exec registration. + Any process with mount visibility and permission, including Bash launched by an + external executor, uses the same Workspace semantics through ordinary filesystem + calls. Sandbox or the external executor owns command launch, streams, exit + status and explicit cancellation. The filesystem daemon supervises no commands; + optional `WorkspaceApi.exec` only delegates ordinary Sandbox execution. + Keep this boundary explicit in implementation + and proofs; see [execution ownership](engine/AGENTS.md#filesystem-access-and-execution-ownership). +- Initial full-root acquisition is explicit. Repeated mount binds the complete + root without a whole-tree scan/copy/materialization or dependency restoration. + The only file it creates is the Workspace's own overlay database, copied from + the template. Demand-loaded metadata/content still pay real I/O. +- Each mounted Workspace has its own local overlay SQLite file, copied at mount + from a template built before readiness and deleted at terminal unmount. Its + metadata, payload and operation records live only in that file; its open-file, directory-position and kernel-reference state lives in daemon memory. Workspaces share no overlay connection, writer, page cache or memory budget; - the daemon shares only committed immutable content and the global Store. - One lock guards a Workspace's connection: hold it for one short atomic job, - never across Store I/O, a whole Exec or a whole Commit. -- **Commit captures the published frontier**, which may hold several calls' - changes however their processes were launched; never filter by Exec - registration or completion. Retain stable input without a bulk overlay copy. - A known install advances the base while preserving later mutations and the - live view. UpToDate need not create a new Commit; invocation auditing is - separate. -- **Outcomes are not inferred.** A lost reply does not remove a published - mutation. Bash exit does not prove that descendants, descriptors, dirty - mappings or requests are finished. -- **FUSE** preserves permissions, stable identity, coherent caches and exact - request/open/lookup ownership; qualify both frequent fresh mounts and - sustained same-mount calls and Commits. Kernel writeback stays off. -- **No artificial caps.** Large files, huge namespaces and mutation bursts use - indexed or backed state, bounded windows, streaming and backpressure; - resident containers and transport frames impose no total file, edit, - Workspace, Commit, flow or time cap. Physical capacity, platform and format - limits and explicit admission remain. The one resident exception is a - Workspace's open state, which grows with open files and the files the - kernel references. -- **Reclamation is automatic and bounded**, after last-owner release, during - activity and idle time; a Commit's own reclamation follows its answer - (owner approval, 2026-11-07). Terminal unmount drains requests, closes the + the daemon shares only committed immutable content and the global Store. One + lock guards a Workspace's connection: hold it for one short atomic job, never + across Store I/O, a whole Exec or a whole Commit. +- Commit captures the shared locally published Workspace frontier, potentially + including several calls' changes regardless of how their processes were + launched. Do not filter captured changes by Exec registration or completion. + Retain stable existing input without a bulk overlay copy. Construct/save/publish + it; known install advances the base while preserving later active mutations and + the effective live view. +- Namespace Init and captured Workspace construction reuse the same Content, + Storage and Persistence libraries. Project adapts native files and metadata; + Workspace adapts the captured immutable base plus Overlay changes. Keep + canonical construction in Content, object encoding/packing in Storage and + database writes in Persistence. Reuse these ports when completing daemon + construction; do not add a second encoder or route it through host Init. See + [shared construction boundaries](engine/AGENTS.md#shared-construction-and-completion-boundaries) + for implemented entrypoints and the remaining S10 namespace integration. +- `Save::finish()` completes saved objects while the Store remains open; + History publication follows. `SqliteHandles::seal()` consumes the sole Store + session for host/offline handoff, checkpoints and closes it. Its macOS unused + extent cleanup belongs to that lifecycle. A live daemon keeps the shared + Store open across Saves and Commits; it does not seal it per file or Commit. +- A lost reply does not remove an already published mutation. Bash exit is not + proof that descendants, descriptors, dirty mappings or requests are finished. + UpToDate need not create a new Commit; invocation auditing is separate. +- FUSE must preserve permissions, stable identity, coherent caches and exact + request/open/lookup ownership. Qualify both frequent fresh mounts and sustained + same-mount calls/Commits. Kernel writeback remains off under the target contract. +- Large files, huge namespaces and mutation bursts require indexed/backed state, + bounded windows, streaming and backpressure. Resident containers/transport + frames must not impose artificial total file/edit/Workspace/Commit/flow/time + caps. Physical capacity, platform/format limits and explicit admission remain. + The one resident exception is a Workspace's open state, which grows with open + files and with the files the kernel currently references. +- Automatic bounded local reclamation follows last-owner release, including + during live activity and idle periods. A Commit's own reclamation (its + captured generation's retired rows) follows its answer (owner approval, + 2026-11-07). Terminal unmount drains requests, closes the Workspace's overlay connection and deletes its file. Neither deletes global history. -- **History is not a checkpoint of everything.** Immutable objects support - reuse and distribution; authority, reference closure, mutable history, - durability and safe GC still need exact contracts. Filesystem history is not - a process, memory, socket or stdout/stderr checkpoint. - -## Where calls run - -Per-call routes are in -[Where each call runs](docs/architecture/control/README.md#where-each-call-runs), -machines in [Platforms](docs/architecture/control/machines/platforms.md) and -workers in [Workers and reach](docs/architecture/control/machines/workers-and-reach.md). - -- **Store calls never run in the daemon.** Every daemon opens the global Store - directly and reads, saves and publishes in-process (owner, 2026-10-07; - revised 2026-10-18). A sqlite Store's calls run on the Store's kernel: in - the caller on a local `native` machine, else in that machine's worker. - PostgreSQL Store calls and bucket (s3) I/O run in the caller's process on - any OS (owner, 2026-11-02, #533 G2). Workspace calls (mount, Exec, Commit, - status, unmount) go to the daemon. Project Init into an existing Store is a - live add (2.0, #533). -- **One kernel, local filesystem.** A sqlite Store's openers share one kernel - and a local ext4, xfs or btrfs (Linux) or apfs or hfs (macOS) filesystem - (`kernel_guard`, #427); another kernel reaches it only through that - machine's worker. Anywhere else sqlite Store calls fail `Unsupported`, with - no fallback. -- **Routes come from the home's records** (machines, owner 2026-11-02, #533 - G2; `native` on macOS, 2026-11-01, #527 D5; `native` on Windows through a - WSL2 worker, 2026-10-22). Both Store kinds work on every machine (owner, - 2026-11-03, #535 D9): a sqlite Store is on exactly one machine, and a - PostgreSQL Store is usable from the machines it is attached to - (`store.attach`, each checked from that machine; supersedes #535 G3) where - the support table (`layerfs-platform` `support.rs`, `control/SUPPORT.json`) - allows, and refused elsewhere with the table's reason. -- **Workers** listen on loopback only, behind a secret; one attempt per call, - no retry and no replay; an uncertain outcome is reported, never resent. -- **Secrets:** a PostgreSQL password is resolved only in the caller and travels - only in one worker request and the Sandbox's private daemon setup file. -- **Plugin-neutral** (owner, 2026-10-24): LayerFS reads `LAYERFS_HOME` and the - trace settings only; no adapter name, integration check or code that names a - plugin. An integration maps its own prefixed names (such as `DSH_*`). -- **Removed stays removed** (owner, 2026-11-08): Firecracker, Microsandbox and - SSH machines are gone with their code; no setting or flag brings one back - without a new owner decision and source change. -- **One database per Store, one writer, short holds** (owner, 2026-10-18). A - Store's committed state lives in one database (a SQLite file, or one - PostgreSQL database, #506): no sharding, second database, concurrent-writer - engine or objects outside it. Writes go one at a time through SQLite's - single-writer rule, the Linux directory lock across writer processes, the - in-process admission queue and the publication lane (head check, merge, - publish as one step). These locks stay; they replace `Busy` failures under - the no-retry rule. Heavy work happens before the lane, only the atomic - publish holds it, and reads never take it. Concurrent Commits are expected - to be uncommon: optimize the uncontended Commit and keep contention correct - and bounded. Work inside the lane that grows with tree size, history or - Branch count is a defect. See the - [Store concurrency model](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/83-store-concurrency-model.md). -- Design references: the - [integration contract](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/303/06-cluster-one-integration.md), - [direct Store access](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/352/DIRECT_STORE.md), - [#533](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/533/README.md), - [#535](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/535/README.md) +- Target (owner direction 2026-10-07, Store calls revised 2026-10-18): every + daemon opens the global Store directly from a volume the daemons share and + reads, saves and publishes in-process. Project Init adds a project to an + existing Store as a live add (2.0, #533). **sqlite Store calls run in a + process on the Store's kernel; postgresql Store calls, and bucket (s3) I/O, + run in the caller's process on any OS (owner, 2026-11-02, #533 G2).** A + sqlite Store's openers share one kernel and a local filesystem holding the + Store (ext4, xfs, btrfs on Linux; apfs, hfs on macOS); a process on another + kernel reaches it only through that machine's worker. Anywhere else sqlite + Store calls fail `Unsupported`, with no fallback; the daemon does not serve + Store calls. Workspace calls (mount, Exec, Commit, + status, unmount) go to the daemon. Do not revive `layerfs-server` or build a + host/server adapter in the data path; the host-mediated SDK runtime is + retired, not extended. See the + [integration contract](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/303/06-cluster-one-integration.md) + and [direct Store access](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/352/DIRECT_STORE.md). + **Exception (owner, 2026-10-20, revised the same day): dsh Live on macOS + only.** The dsh-layerfs plugin may give macOS processes a ported SDK with + the SDK's API whose calls run in one long-lived worker per LayerFS box: a + Linux container running the real SDK, which opens each project's Store + directly and keeps it and its daemon connections open while the project is + in use (closed after a quiet minute). The worker listens on loopback only, + behind a secret. The product SDK and daemon do not change. (Widened to any + integration on 2026-10-24; see below.) +- **Additional exception (owner, 2026-10-22): dsh Live on Windows.** The + Windows dsh-layerfs adapter may forward its SDK binding calls to one + authenticated loopback worker per LayerFS home in a selected WSL2 Linux + distribution. The worker opens the shared Store directly on that + distribution's local ext4/xfs/btrfs filesystem and uses its Docker Engine + for Sandbox daemons. Windows owns the UI, bindings and traces; Linux owns + native handles and project discovery. This is independent of the macOS + adapter and changes neither the product SDK nor the daemon. A lost reply + is never replayed, and an uncertain Init retains its Linux home and volume. +- **Additional exception (owner, 2026-11-01, #527 D5; renamed 2026-11-02, + #533 G2): the `native` machine on macOS.** For a Store on the `native` + machine on macOS, the library runs in the caller's own process and opens the project's Store directly on a local apfs or hfs + directory (`kernel_guard` admits those two there, #427). Its Sandboxes are + daemon processes serving loopback NFS to this Mac's own client, one user, + no container (see the agent guide's native Sandboxes). Every opener is a + process on this one Mac, so the one-kernel, local-filesystem rule holds; + network and other filesystems are still refused. The Docker, worker and + `vm:` routes on macOS are unchanged. +- **Plugin-neutral LayerFS (owner, 2026-10-24; revised 2026-11-02, #533 + G2).** LayerFS is agnostic to dsh and every other plugin. The `layerfs` + package routes by the home's records: a call's route comes from its + Store's and machine's records, so one process uses Stores on several + machines at once, and the worker routes above are open to any caller of + the package. LayerFS reads `LAYERFS_HOME` and the trace settings only. + There is no adapter name, no integration check and no code that names a + plugin. An integration that can set only its own prefixed names, such as + dsh's `DSH_*`, maps them itself. The worker's rules above (loopback, + secret, one attempt, no replay) are unchanged. +- **Machines (owner 2026-11-02, #533 G2; replaces the 2026-10-31 VM + placement, V1).** A VM machine holds many Stores, with one worker per + machine. Machines are created, removed and started by the `machine` calls, + and the routes come from the home's records (`machine//record.json`, + `store//record.json`). `native` and `docker` are built in; VM kinds + (`apple-container`, `lima`) are created by name. Each VM + machine's data disk is mounted at + `/layerfs/data` and holds `store//store.sqlite` per Store. Its + worker runs as root behind its own secret, reached through one loopback + port on the host, with one attempt per call and no replay, the same rules + as the macOS and Windows routes. Guests never read the home: the caller + sends each worker call the settings it needs and writes every record. + **Every machine uses both Store kinds (owner, 2026-11-03, #535 D9, G1, + G3):** a machine holds any number of sqlite Stores, and every PostgreSQL + Store is usable from it. A sqlite Store is on exactly one machine; a + PostgreSQL Store's calls still run in the caller's process, and it is + usable from every machine where the support table (`layerfs-platform` + `support.rs`, `control/SUPPORT.json`) marks the platform supported; + elsewhere it is refused with the table's reason. The password is resolved only in the + caller and travels only in one worker request and the Sandbox's private + daemon setup file. This supersedes #533 O3/G13 ("VMs and ssh later"). + See [#533](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/533/README.md), [#535](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/535/README.md) and the [proposal](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/proposals/host-platform-connections.md). - -## Read first - -| Task | Read | -| --- | --- | -| Any implementation | the two cluster-one handbooks and the [architecture guide](docs/architecture/README.md): the layer page and every topic the change touches | -| Cluster two | also [Workspace](docs/architecture/engine/workspace/README.md), [Overlay](docs/architecture/engine/overlay/README.md), [Commit](docs/architecture/engine/commit/README.md), [Kernel routes](docs/architecture/engine/kernel-routes/README.md), [Daemon](docs/architecture/engine/daemon/README.md), [Bridge](docs/architecture/engine/bridge/README.md) | -| Documentation or a release | [documentation policy](docs/general/documentation-policy.md), [release policy](docs/general/release-policy.md) | -| A pull request or issue | [code review and triage policy](docs/policy/code-review.md) | -| A measurement | [measurement workflow](docs/general/agent-measurement-policy.md), [benchmark rules](docs/general/benchmark_rules.md), [report template](docs/general/benchmark_agent_report.md), the owning harness contract | - -## Measurements - -The [measurement workflow](docs/general/agent-measurement-policy.md) owns these -rules under the same numbers; moving them out of this file does not weaken -them. In short: - -1. **A warm cache never credits a measured phase.** Each phase pays for its - own work from an equally declared and enforced cache state; unknown or - mismatched cache state is INCOMPLETE/INELIGIBLE, not PASS. -2. **Reuse setup, never measurement.** Use `--setup clone` for - post-initialization cases where supported; a clone is an independent - writable copy, not a cold claim. Never reuse a mutated sample. -3. **Run once, pinned and bounded.** One sample per case and arm; fresh - append-only outputs, keeping failed, ineligible and unrun selections; pinned - identities; worktree-local locks and targets; a complete performance - command within 15 s (declared exceptions up to 25 s, independent proofs - under 10 s), with no timeout, workload or cache change to turn a miss into - PASS. Export `LAYERFS_CONSTRUCTION_WORKERS=1`: one construction producer per - Commit, capture or snapshot (Namespace Init keeps its parallel profile). A - Commit's one producer runs on its own thread and hands objects through one - bounded queue to the calling thread, which keeps the one Save and publishes - (owner approval, 2026-11-07; [Commit construction](docs/architecture/engine/commit/construction.md)). - Measure speed and storage; reject about 50% speed loss for about 5% storage. - -The workflow's §5 keeps the cache-credit failure these rules came from. +- **Removed platforms (owner, 2026-11-08; supersedes the layerfs-project#7 + restriction of 2026-11-06): Firecracker, Microsandbox and SSH machines are + removed.** Their code is gone and no platform is disabled with its code + kept. An old record of one is shown as unsupported with the reason + (``platform `` was removed in LayerFS 2.0``), every other call on it + is refused, and `machine remove` only forgets it. No runtime setting or + feature flag brings one back; that requires a later explicit owner + decision and source change. Historical evidence is unchanged. Native, + Docker, Lima and Apple Container retain their existing support rules. See + the [restriction contract](docs/architecture/control/machines/platforms.md#removed-platforms-platform_policyrs). +- **One global database, one writer, short holds (owner decision 2026-10-18).** + All committed state lives in one SQLite Store file; no sharding, second + database, concurrent-writer engine or objects outside it. Writes go one at a + time: SQLite's single-writer rule, the Linux directory lock across writer + processes, the in-process admission queue and the publication lane (head + check, merge, publish as one step). These locks stay; they replace `Busy` + failures under the no-retry rule. Performance comes from short holds: heavy + work happens before the lane, only the atomic publish step holds it, reads + never take it. Agents' tool calls are separated by model latency, so + concurrent Commits are expected to be uncommon; optimize the uncontended + Commit and keep contention correct and bounded. Work that grows with tree + size, history or Branch count inside the lane is a defect. See the + [Store concurrency model](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/83-store-concurrency-model.md) + and the [global SQLite Store](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/84-global-sqlite-store.md) + design reference. + Immutable objects support reuse/distribution; authority, reference closure, + mutable history, durability and safe GC still require exact contracts. + Filesystem history is not a process, memory, socket or stdout/stderr + checkpoint. + +## Read the current contract for the task + +Before implementation or integration, read both root handbooks and the +[architecture guide](docs/architecture/README.md): the layer page and every +topic the change touches (for cluster two: [Workspace](docs/architecture/engine/workspace/README.md), +[Overlay](docs/architecture/engine/overlay/README.md), [Commit](docs/architecture/engine/commit/README.md), +[Kernel routes](docs/architecture/engine/kernel-routes/README.md), [Daemon](docs/architecture/engine/daemon/README.md) +and [Bridge](docs/architecture/engine/bridge/README.md)). The former #303 contracts and the +[optimization investigation](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/303/fuse-optimization-investigation.md) +are archived research; they do not silently select new algorithms, profiles or capabilities. + +Completed/superseded Stage 0–6 prompts are historical references, not default +routing. Preserve canonical compatibility and evidence, but do not resume old +loops, reopen closed rows or run old campaigns unless the current task selects +that work. Do not choose authority by issue number alone. + +For documentation/release work, read the +[documentation policy](docs/general/documentation-policy.md) and +[release policy](docs/general/release-policy.md). +For opening, reviewing or triaging a pull request or issue, read the +[code review and triage policy](docs/policy/code-review.md). +For measurements, read the [agent measurement workflow](docs/general/agent-measurement-policy.md), +[benchmark rules](docs/general/benchmark_rules.md), +[report template](docs/general/benchmark_agent_report.md) and owning harness/family contract. +Benchmark suites, harnesses and released data live in the +[layerfs-benchmark](https://github.com/Ephemeral-AI-Lab/layerfs-benchmark) repository, moved out of +`benchmark/` on 2026-10-23. Benchmark routing is its +[fs-bench-pro/AGENTS.md](https://github.com/Ephemeral-AI-Lab/layerfs-benchmark/blob/main/fs-bench-pro/AGENTS.md). +**fs-bench-pro is frozen and left broken (owner, 2026-10-18).** Do not edit, +build, run or repair it, and leave it out of workspace builds and checks. Its +call sites broken by #352 F-020 (`ProjectApi` fork and commits, +`ForkRequest`, a `SandboxCreate` without `connect_host`; listed in the F-020 +fix record) stay broken; a product change never adapts it. Only an +explicit owner instruction reopens it. +The root `benchmark/` harness that built and measured the retired reference was +removed from HEAD; read it at tag `reference-v0.1.6-final`. Released +fs-bench-pro results (v0.1.4–v0.1.6) are in layerfs-benchmark's +[data/fs-bench-pro](https://github.com/Ephemeral-AI-Lab/layerfs-benchmark/blob/main/data/fs-bench-pro/README.md). +Older documents here still link to `benchmark/` paths; read those files in +layerfs-benchmark at the same path without the `benchmark/` prefix. + +## 1. A warm cache must never credit a measured phase + +Each measured phase pays for its own work from an equally declared/enforced +cache state. No setup/earlier-phase warmth, pre-touching, own-write cache credit, +lifetime-peak substitution or file-sized backing/page-cache growth disguised by a +bounded heap. Unknown or mismatched cache state is INCOMPLETE/INELIGIBLE, not PASS. +Preserve failures and historical verdicts. Full rules and the retained example: +[measurement workflow §1](docs/general/agent-measurement-policy.md#1-a-warm-cache-must-never-credit-a-measured-phase). + +## 2. Reuse setup (`--setup clone`); never reuse measurement + +Reuse closed prepared inputs, sealed builds/images and qualifying proof receipts +through the declared mechanisms. Use `--setup clone` for post-initialization +cases where supported; fresh is for initialization/fresh-output cases. +A clone is an independent writable byte copy, not a cold claim. Never reuse a +mutated sample or remove timed product work through setup. Record all reuse and +enforce residency for a cold claim. See +[measurement workflow §2](docs/general/agent-measurement-policy.md#2-reuse-setup---setup-clone-never-reuse-measurement). + +## 3. Running a measurement + +The [measurement workflow](docs/general/agent-measurement-policy.md#3-running-a-measurement) +owns detailed procedures, scoped SDK rules and budget exceptions. Its requirements +remain binding; moving them out of this file does not weaken them. + +1. One sample per case/arm, no best-of or repeated unchanged treatment. + Diagnose retained receipts or labeled count-driven causes. The separately + scoped regression-screen exception is not treatment resampling. +2. Fresh append-only outputs; retain failed, ineligible and unrun selections. +3. Pin source/product/compilation/dependency/binary/image/harness/workload/cache + identities. No dirty seal compared as a sealed arm. +4. Exploratory timing is diagnostic; verification is separate. At final identity, + run covering proof once. Reuse unaffected earlier-family evidence; rerun only + the necessary affected checkpoint, without relabeling old receipts. +5. Use worktree-local measurement locks and Cargo targets; no same-worktree + overlap. Declare cross-worktree interference; never interrupt another owner. +6. Read the report template before each invocation; record exact metrics, limits, + arithmetic, commands and all outcomes in the campaign-owned ledger/receipts. +7. Default complete performance command ≤15 s; declared exceptions up to 25 s; + default independent proof <10 s. Frozen family/profile limits retain their + declared scope. No timeout/workload/cache changes to turn a miss into PASS. +8. Commit/capture/snapshot have one construction producer in normal wiring and + each measured operation; export `LAYERFS_CONSTRUCTION_WORKERS=1`. + Namespace Init alone retains its supported parallel construction profile. + A Commit's one producer runs on its own thread and hands objects through + one bounded queue to the calling thread, which keeps the one Save and + publishes (owner approval, 2026-11-07); see + [Commit construction](docs/architecture/engine/commit/construction.md). + No other helper/lane to pass a gate; FUSE dispatch is a separate concern. +9. Measure both speed and storage. Reject about 50% speed loss for about 5% + storage benefit; report raw deltas and preserve frozen-gate failures. ## 4. Code, build and docs -- **One attempted operation.** No automatic retry, busy handler, - refresh/reprepare or failed-operation replay; readiness waits are not - retries. Preserve exact failure, uncertainty and custody: no guessed resend, - deletion or success. -- **No test command runs longer than 2 minutes.** Build first (`--no-run`) so - compilation is not mistaken for a slow test, then run each selection under - an explicit wall timeout of at most 120 s, per package when the suite cannot - fit. Never leave a test running, background it to wait it out or wrap it in - a repeat loop. Reaching the ceiling is a FAILED hang: diagnose it from source - and bounded output before any rerun, and report it. Write tests that cannot - wait forever (bounded waits; a thread's exit must not depend on another - thread finishing without panicking). The §3 budgets stay stricter. -- **CI reports; it never gates** (owner, 2026-10-23). `.github/workflows/ci.yml` - runs fmt, guards, Clippy and the fast tests after a push or pull request, - `nightly.yml` the slow and FUSE mount tests, `format.yml` the Ruff check; - [tools/ci_tests.py](tools/ci_tests.py) picks the lanes. Neither `ci.yml` nor - `format.yml` runs for a pull request into `layerfs_v2_0_0` (owner, - 2026-11-07). Nothing runs before a push and nothing is required: do not wait - for CI or treat a green run as verification. Verify the changed scope with - the [product checks](engine/AGENTS.md#checks-and-completion) and report - checks and gaps. `tools/preflight.sh` stays retired; restore no equivalent - wrapper. An empty guard scan proves no implementation. -- **Refactors delete tests rather than nurse them** (owner, 2026-10-18). A test +- [The parallel worktree launcher](docs/general/parallel-worktrees.md) and + [the cleanup script](docs/general/cleanup.md) are optional local tools. Disk + thresholds, check cadence and standing cleanup authorization depend on the + machine; they belong in the user's own agent instructions, not in this file. +- Independent E2E runs own their writable Stores, SDK homes, containers and + traces. Preserve a test's intentional multi-daemon shared Store. +- Preserve unrelated work. Build the current core implementation, not legacy + aliases, source includes or error-driven substitutes. Keep platform cfgs. +- No new dependency if an existing crate supplies the capability. Never patch, + fork, vendor, replace or edit third-party code/registry packages. The explicit + exception is the owner's selected [fuser Git fork](engine/AGENTS.md#fuser-git-source): + use its pinned revision, not a local vendor, patch or registry substitute. + Use locked builds and report an incompatible required dependency with evidence. +- One attempted operation; no automatic retry/busy handler, refresh/reprepare + or failed-operation replay. Readiness waits are not retries. Preserve exact + failure/uncertainty and custody; no guessed resend, deletion or success. +- **CI reports; it never gates (owner, 2026-10-23).** GitHub Actions runs a + fast lane after each push or pull request (`.github/workflows/ci.yml`: fmt, + guards, Clippy and the fast tests) and a nightly lane (`nightly.yml`: slow and + FUSE mount tests); `format.yml` checks the Ruff formatting below. + Neither `ci.yml` nor `format.yml` runs for a pull request into + `layerfs_v2_0_0` (owner, 2026-11-07). + [tools/ci_tests.py](tools/ci_tests.py) decides which tests go in + which lane. None is a required check, and nothing runs before a push. Do + not wait for CI, and do not treat a green run as + verification of your change: verify the changed scope yourself with + [product checks](engine/AGENTS.md#checks-and-completion) and report checks/gaps. + There is still no pre-push gate: `tools/preflight.sh` stays retired; do not + restore it or an equivalent wrapper. An empty guard scan proves no + implementation. +- **Migrations and refactors delete tests rather than nurse them (owner, + 2026-10-18).** When code is moved, replaced or refactored, a test that is tricky to adapt (it drives a removed route, internal type or retired model, - or needs scaffolding the new design lacks) is deleted, not patched; keep or - move a test only when it checks surviving behaviour cheaply. Where a deleted - test guarded a fix or contract that still holds, cover it on the new code. - List deleted tests in the commit message and the pull request or issue. - Never weaken a surviving assertion to make it pass. -- **Dependencies.** No new dependency when an existing crate covers the need. - Never patch, fork, vendor, replace or edit third-party code or registry - packages; the one exception is the owner's - [fuser Git fork](engine/AGENTS.md#fuser-git-source) at its pinned revision. - Use locked builds; report an incompatible required dependency with evidence. -- **Build the current product.** Preserve unrelated work. Build the current - implementation, not legacy aliases, source includes or error-driven - substitutes; keep platform cfgs. Keep the - [repository ARM64 build inputs](.cargo/config.toml): explicit RUSTFLAGS - repeat the profile. -- **Claim only real guarantees.** Persistence profiles are scoped to their - Store. No `fsync`/`fdatasync`/`sync_data`/`sync_all` on disposable Workspace - backing and no space reserved ahead of use on it. Memory hints are not - evidence of resident bounds. -- **Independent E2E runs** own their writable Stores, SDK homes, containers and - traces; keep a test's intentional multi-daemon shared Store. -- **Source shape.** New core, engine, control and API production files obey - the 999-line and 200-line `lib.rs`/`mod.rs` ceilings in the - [engine guide](engine/AGENTS.md#production-file-size-and-responsibility). -- **Docs.** Update affected architecture and API documentation in the same - commit, as the [architecture doc update policy](docs/policy/architecture-doc-update.md) - sets out; keep proposal, research, implemented and measured claims distinct. - Format changed Python tools with Ruff (`ruff.toml` names the version, scope - and command). -- **Local tools are optional.** The - [parallel worktree launcher](docs/general/parallel-worktrees.md) and - [cleanup script](docs/general/cleanup.md) are local tools; disk thresholds, - check cadence and cleanup authorization belong in the user's own agent - instructions, not here. + or needs scaffolding the new design does not have) is deleted, not patched. + Keep or move a test only when it still checks surviving behaviour cheaply; + where a deleted test guarded a fix or contract that still holds, cover that + behaviour with a test on the new code. List deleted tests in the commit + message and the fix record (the pull request or issue; `core/docs/issues/` + no longer holds records). Never weaken a surviving test's assertion to + make it pass. +- **No test command runs longer than 2 minutes.** Give every test invocation an + explicit wall timeout of at most 120 s and stop it when it expires; never + leave one running, background it to wait it out, or wrap it in a repeat loop. + Build first (`--no-run`) so compilation is not mistaken for a slow test, and + select by package/test when the whole suite cannot fit. A test that reaches + the ceiling has FAILED as a hang: diagnose it from source and bounded output + before any rerun, and report it. Write tests so they cannot wait forever: + bounded waits, and a spawned thread's exit must not depend on another thread + finishing without panicking. This is a ceiling, not a target; the scoped + measurement and proof budgets in §3 stay stricter. +- Preserve [repository ARM64 build inputs](.cargo/config.toml); explicit + RUSTFLAGS must repeat the profile. Details are in the core guide. +- Persistence profiles are scoped to the store. Claim only their guarantees; + no `fsync`/`fdatasync`/`sync_data`/`sync_all` on disposable Workspace backing, + and no space is reserved ahead of use on it. + Memory hints are hints, not evidence of resident bounds. +- New core/engine/API production files obey the 999-line ceiling and 200-line + declaration/delegation ceiling for `lib.rs`/`mod.rs`. Product-only source, + external tests and guard coverage are detailed in the layer guides. +- Format changed Python tools with Ruff; `ruff.toml` names the pinned version, + scope and command. `format.yml` reports formatting drift after a push; + nothing runs before one. +- Update affected source architecture/API documentation with implementation + changes, in the same commit, as the + [architecture doc update policy](docs/policy/architecture-doc-update.md) + sets out. Keep proposal, research, implemented and measured claims distinct. ### Production LOC comparison for every commit -Every commit records **production lines of code before, after and the signed -delta**. It is a source-size comparison, not a performance claim. - -- **What counts:** nonblank, non-comment lines of first-party product - implementation, including shipped runtime SQL and other shipped - implementation outside Rust `src/`; imports, declarations and forwarding - count, and a line with code and a comment counts once. Tests (including - inline test modules and test-only branches), fixtures, mocks, examples, - benchmark harnesses, tools, docs, manifests, lockfiles, third-party code and - generated artifacts do not. Never use raw line totals or Git statistics. -- **What is compared:** the first parent against the committed tree (an empty - tree for an initial commit; the first parent for a merge, declared). Prepare - it from the parent and the final staged tree before committing, confirm the - commit matches, and exclude unstaged and untracked work. Recompute after an - amendment, rebase or any change to the staged source. -- **Same method both sides:** the same counter and version, scope, exclusions - and handling of inline test code; record the command. Review classification - when files move or new product paths appear; never drop code silently. -- **Honest migration:** while old and new implementations coexist, report each - total and the combined one; include application-adapter code. Label - relocation, duplication and retirement; a scope change or the reference's - deletion is not an algorithmic simplification. Report the reference as - `reference absent (retired R9c; last counted 65,417 at tree 498dd1917812)`, - never as a bare 0. -- **Report:** in the commit message and handoff, the headline - `Production LOC: -> (delta )`, then one - line for each of `core`, `engine`, `control` and `api` in that order, even - when unchanged, so they sum to the headline; after a changed scope, each - crate or package whose count moved with its delta. Then the reference line - and the method. Scopes and crates are the ones - `tools/production_loc.py --detail` prints. A move between scopes or crates - is labelled relocation. Several commits give one block each, plus one for - the range. A test or docs-only commit still reports the unchanged total and - delta 0. - - ```text - Production LOC: 163,645 -> 163,685 (delta +40) - core 39,787 -> 39,795 (+8) layerfs-history +8 - engine 74,997 -> 75,029 (+32) layerfs-persistence +7, layerfs-store-service +25 - control 42,777 -> 42,777 (0) - api 6,084 -> 6,084 (0) - reference absent (retired R9c; last counted 65,417 at tree 498dd1917812) - Method: tools/production_loc.py --detail on parent 99ffd262a3 and the staged tree - ``` - -Growth is allowed when the product change justifies it. Never remove required -validation, compress code into dense lines or move implementation out of scope -to improve the number, and never estimate it. +Every Git commit must record **production source lines of code before, after, +and the signed delta**. Test, documentation and tooling changes do not contribute +to this number. This is a source-size comparison, not a performance claim. + +- **Count production code only.** Count nonblank, non-comment source lines in + first-party LayerFS product implementation, including required runtime SQL or + other shipped implementation outside Rust src/ directories. Imports, declarations + and forwarding code count. Exclude tests (including legacy inline test modules + and test-only branches), fixtures, mocks, examples, benchmark harnesses, development + tools, docs, manifests/lockfiles, third-party code and generated build artifacts. + A line with both code and a comment counts once. Do not substitute raw file-line + totals or Git insertion/deletion statistics for production LOC. +- **Compare the exact commit snapshots.** Before is the commit's first parent; + after is the committed tree. Prepare the comparison from the parent and final + staged tree before committing, then confirm the resulting commit matches it. + Exclude unstaged/untracked work. Use an empty tree for an initial commit. For + merges declare the first-parent comparison; recompute after amendments/rebases + or any change to the staged source. +- **Keep counting reproducible.** Use the same counter/version, source scope, + exclusions and handling of inline test code for both snapshots. Record the + command/method with the comparison. Review source classification when files + move or new product paths appear; do not silently drop code from the count. + A counter that includes legacy inline tests does not satisfy this rule. +- **Report migration honestly.** While old and replacement implementations + coexist, report their production totals separately as well as the combined + total. Include application-adapter production code when introduced. Label + relocation, duplication and legacy retirement; do not call a scope change or + deletion of the reference an algorithmic simplification. Since R9c the + reference scope is absent: report it as `reference absent (retired R9c; last + counted 65,417 at tree 498dd1917812)`, never as a bare 0. +- **Put the result in the commit message and handoff.** Use + `Production LOC: -> (delta )`, with scope + and counting method, plus migration subtotals when applicable. For multiple + commits, give a comparison for each. A test/docs-only commit still reports + the unchanged production total and delta 0; it does not report a fictitious + zero-sized product or add test/documentation LOC to the headline. + +LOC growth is allowed when justified by the product change; this rule does not +require every commit to shrink. Never remove required validation, compress code +into dense lines, or move implementation outside the declared scope to improve +the number. Complete the comparison before committing; do not invent estimates. ### File and folder layout listings -A plan, handoff, review or reply that shows a source layout uses one vertical -listing per crate in a fenced `diff` block: +When a plan, handoff, review or reply shows a source layout, current or +planned, use one vertical listing per crate in a fenced `diff` block, so the +renderer highlights what changes: ```diff construction/ [793] @@ -402,20 +460,60 @@ listing per crate in a fenced `diff` block: + records.rs — ``` -- Column 1 is the marker: `+` a path that does not exist yet, `-` one this - change removes, a space one that stays (unchanged, edited or moved in; say - where from). A listing of the current tree has a space on every line. -- One path per line, two spaces per directory level; directories end in `/` - and list their files before their subdirectories; no joined siblings or - brace groups. -- Each existing file shows its production LOC from - `tools/production_loc.py --files` at a stated commit, right-aligned; each - directory shows the bracketed sum of the files listed beneath it, removed - files included (say so when only part is shown). A file that does not exist - shows `—` with `new` or `cond.` (added only if existing code cannot cover - the need); never an estimate. -- Annotations are short, in parentheses, after the number. State the commit, - counter and included crates above the listing. - -A listing is a snapshot: it does not make a planned name a requirement, and -the source and `Cargo.toml` still decide what is built. +- **Column 1 is the change marker, then one space, then the path.** `+` marks + a file or directory that does not exist yet (green); `-` marks one this + change removes (red); a space marks one that stays, whether unchanged, + edited in place or moved in (say where from in its annotation). A listing of + the current tree alone has a space on every line. +- **One path per line.** Indent two spaces per directory level. Directories end + in `/`; list a directory's own files before its subdirectories. Do not join + siblings on one line or use brace groups such as `{mod,cursor}.rs`. +- **Every existing file carries its production LOC**, right-aligned in one + column, from the pinned `tools/production_loc.py --files` at a stated commit. + This is the `prod` column, never physical lines or Git statistics. +- **Every directory carries its subtotal in brackets**, summing all listed + production files beneath it as they are at the stated commit, removed files + included. If only part of a directory is shown, say so in its annotation; + the bracketed number then covers only the files shown. +- **A file that does not exist yet shows `—`**, followed by `new` (planned) or + `cond.` (added only if existing code cannot cover the need). Never print an + estimated LOC for it. +- **Annotations are short, in parentheses, after the number**: what changes in + this step, or why the file is listed. Keep ownership rationale in prose. +- **State the scope above the listing**: the commit, the counter, and which + crates or directories are included. Tests live outside `src/` and are not + counted; name their directory in prose if it matters. + +A listing is a snapshot. It does not make a planned name a scaffolding +requirement, and an existing source file or `Cargo.toml` still decides +what is built. + +## 5. Why these rules exist (worked example) + +The historical cache-credit failure and its immutable ledger remain linked in +[the measurement workflow](docs/general/agent-measurement-policy.md#5-why-these-rules-exist-worked-example). +Historical explanation is retained outside default implementation routing. + + +All product layers follow the detailed +[production source rules](engine/AGENTS.md#production-file-size-and-responsibility) +and [checks and completion rules](engine/AGENTS.md#checks-and-completion). +The root manifest builds four core crates, ten engine crates, eight control +crates under `control/crates` (the entity crates `layerfs-home`, +`layerfs-machine`, `layerfs-store` and `layerfs-sandbox` with its +`workspace/` module; the drivers `layerfs-platform`, `layerfs-bucket` and +`layerfs-sandbox-backend`; and `layerfs-control`) and four API crates under +`api/crates`: the call surface `layerfs-api`, the `layerfs` command +`layerfs-cli`, the Node binding `layerfs-node` and the PyO3 module +`layerfs-python`. `api/` also holds the published JavaScript and Python +packages. The control layer (`control/crates`, renamed from `connect/` by the +owner 2026-11-06, layerfs-project#101 D-6; #535 D4) holds the management code +and the crates that reach a machine or the network; see the +[control guide](control/AGENTS.md). The dependency +direction is `core < engine < control < api`, including dev dependencies and +test includes; nothing in engine depends on control. Workstream names do not change this layering. +After rebasing across #463, update each selected worktree's own target registration and prepare again +for the root `target/` path; never migrate another worktree's resource record. +After rebasing across #686 (fuser from its Git fork), or any later fuser `rev` change, run +`tools/worktree.py prepare` again for each platform before offline builds; a Cargo +home prepared earlier has no checkout of that revision and fails `--offline`. diff --git a/api/AGENTS.md b/api/AGENTS.md index 98364bacbb..195063987f 100644 --- a/api/AGENTS.md +++ b/api/AGENTS.md @@ -21,9 +21,7 @@ public surface: `node/layerfs.node` by `scripts/build-native.mjs`; the package loads that file or the one `LAYERFS_NATIVE` names, nothing else. - `crates/layerfs-python`: the PyO3 module `layerfs._native`. -- `node/`: the SDK for Node, `@layerfs/node`. -- `cli/`: the `layerfs` command's package, `@layerfs/cli`. -- `mcp/`: the MCP server over stdio, `@layerfs/mcp` (private). +- `node/`: the one published JavaScript package, `layerfs`. - `python/`: the Python package `layerfs`. The packages are thin shells over the table: each function is one library @@ -43,12 +41,12 @@ ordinary daemon binary before running tests that launch it. `LAYERFS_TEST_DAEMON` may select an explicit binary; otherwise the tests locate the sibling Cargo output. The owner keeps `layerfs-node` out of CI. -API code follows the engine guide's +All API product code follows the [production size and responsibility rules](../engine/AGENTS.md#production-file-size-and-responsibility) -and [checks](../engine/AGENTS.md#checks-and-completion), and the root guide's -[rules you must not break](../AGENTS.md#rules-you-must-not-break). Preserve the -existing unsafe-code boundaries; do not add test hooks or inline tests to -production src. +and [checks and completion rules](../engine/AGENTS.md#checks-and-completion), +including bounded test commands, build-before-run, locked dependencies, ARM64 +flags, per-commit LOC and the Disposable-only profile. Preserve the existing +unsafe-code boundaries; do not add test hooks or inline tests to production src. Do not add a call, argument or result name in JavaScript: change the call's entry in `crates/layerfs-api/src//.rs` and the package follows. `layerfs-api/src` holds only `lib.rs`, the call machinery `calls/` (table @@ -56,4 +54,5 @@ shape, checks, scope, keys, dispatcher, `Context`; layerfs-project#107) and one folder per family: its `mod.rs` (the family's calls in help order) and one file per call (its table entry and the function that runs it: one call of a control crate's per-op function, never an op name as a string). A -test pins this shape (`tests/calls_table.rs`). +test pins this shape (`tests/calls_table.rs`). CI reports and does not gate a +push. diff --git a/api/crates/layerfs-api/src/branch/publish.rs b/api/crates/layerfs-api/src/branch/publish.rs index 521fc4bad4..97faa7cfaf 100644 --- a/api/crates/layerfs-api/src/branch/publish.rs +++ b/api/crates/layerfs-api/src/branch/publish.rs @@ -1,5 +1,5 @@ //! `branch.publish`: a Branch's head or Commit as main's next Layer. -use crate::calls::table::Kind::{Bool, Name, Project}; +use crate::calls::table::Kind::{Name, Project}; use crate::calls::table::{a, call, opt, Call}; use crate::calls::Context; use layerfs_bridge::control::SdkValue; @@ -15,12 +15,8 @@ pub(crate) const CALL: Call = call( a("branch", Name), opt("commit", Name), ], - &[ - opt("expected_base", Name), - opt("expected_head", Name), - opt("force", Bool), - ], - "a Branch's head or Commit as main's next Layer, refusing a stale base", + &[], + "a Branch's head or Commit as main's next Layer", run, ); @@ -31,9 +27,6 @@ fn run(context: &mut Context<'_>, args: &SdkValue) -> Outcome { project: text("project").unwrap_or_default(), branch: text("branch").unwrap_or_default(), commit: text("commit"), - expected_base: text("expected_base"), - expected_head: text("expected_head"), - force: args.get("force").map(|_| args.get_bool("force")), }; let (home, events) = context.parts(); publish(home, events, &request) diff --git a/api/crates/layerfs-api/tests/calls_table.rs b/api/crates/layerfs-api/tests/calls_table.rs index bdebadb78d..39cdc1df3e 100644 --- a/api/crates/layerfs-api/tests/calls_table.rs +++ b/api/crates/layerfs-api/tests/calls_table.rs @@ -139,97 +139,6 @@ fn branch_family() { assert!(!reached.starts_with("InvalidInput"), "{reached}"); } -#[test] -fn branch_publish_declares_its_expected_base_options() { - // The shell, Node and Python front ends build their arguments from this - // table, so these options (and their shapes) are the CLI's and SDK's. - let call = layerfs_api::calls() - .into_iter() - .find(|call| call.family == "branch" && call.call == "publish") - .expect("branch.publish is in the table"); - let declared: Vec<(&str, Kind, bool)> = call - .options - .iter() - .map(|option| (option.name, option.kind, option.optional)) - .collect(); - assert_eq!( - declared, - vec![ - ("expected_base", Kind::Name, true), - ("expected_head", Kind::Name, true), - ("force", Kind::Bool, true), - ] - ); - assert_eq!( - call.help, - "a Branch's head or Commit as main's next Layer, refusing a stale base" - ); -} - -#[test] -fn branch_publish_expectations_are_plumbed() { - for (name, value, why) in [ - ( - "expected_base", - SdkValue::Int(1), - "`expected_base` must be text", - ), - ( - "expected_base", - SdkValue::Bool(true), - "`expected_base` must be text", - ), - ( - "expected_head", - SdkValue::Bool(true), - "`expected_head` must be text", - ), - ( - "expected_head", - SdkValue::Int(1), - "`expected_head` must be text", - ), - ("force", t("yes"), "`force` must be true or false"), - ("force", SdkValue::Int(1), "`force` must be true or false"), - ] { - let refused = code( - "branch.publish", - map(&[ - ("project", t("shop/api")), - ("branch", t("work")), - (name, value), - ]), - ); - assert!(refused.contains(why), "{name}: {refused}"); - } - // A misspelled option is not read as an argument of the call. - let typo = code( - "branch.publish", - map(&[ - ("project", t("shop/api")), - ("branch", t("work")), - ("expectedBase", t("00")), - ]), - ); - assert!( - typo.starts_with("InvalidInput: branch.publish: unknown argument `expectedBase`"), - "{typo}" - ); - // A well-shaped expectation reaches the call instead of being refused by - // argument checking. - let reached = code( - "branch.publish", - map(&[ - ("project", t("shop/api")), - ("branch", t("work")), - ("expected_base", t("00")), - ("expected_head", t("00")), - ("force", SdkValue::Bool(true)), - ]), - ); - assert!(!reached.starts_with("InvalidInput"), "{reached}"); -} - #[test] fn commit_family() { assert!(code( diff --git a/api/crates/layerfs-cli/tests/publish_options.rs b/api/crates/layerfs-cli/tests/publish_options.rs deleted file mode 100644 index ea824cf513..0000000000 --- a/api/crates/layerfs-cli/tests/publish_options.rs +++ /dev/null @@ -1,105 +0,0 @@ -//! `branch.publish`'s expected-base options are generated from the call -//! table: the shell accepts them, its usage line names them, and a malformed -//! value is a usage error naming the flag (#691). -use layerfs_api::Call; -use layerfs_bridge::control::SdkValue; - -fn publish() -> &'static Call { - layerfs_api::calls() - .into_iter() - .find(|call| call.family == "branch" && call.call == "publish") - .expect("branch.publish is in the table") -} - -fn parse(words: &[&str]) -> Result { - let words: Vec = words.iter().map(|word| (*word).to_owned()).collect(); - layerfs_cli::parse::parse(publish(), &words, &mut std::io::empty()) - .map_err(|why| why.0.to_owned()) -} - -/// The usage error `words` are refused with (`Parsed` has no `Debug`, so -/// `unwrap_err` cannot be used). -fn usage_error(words: &[&str]) -> String { - match parse(words) { - Err(why) => why, - Ok(_) => panic!("{words:?} must be a usage error"), - } -} - -const BASE: &str = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"; -const HEAD: &str = "abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789"; - -#[test] -fn the_shell_reads_expected_base_expected_head_and_force() { - let parsed = parse(&[ - "shop/api", - "work", - "--expected-base", - BASE, - "--expected-head", - HEAD, - "--force", - ]) - .expect("the options parse"); - assert_eq!(parsed.args.get_text("expected_base"), Some(BASE)); - assert_eq!(parsed.args.get_text("expected_head"), Some(HEAD)); - assert!( - matches!(parsed.args.get("force"), Some(SdkValue::Bool(true))), - "{:?}", - parsed.args - ); - assert_eq!( - layerfs_cli::help::usage(publish()), - "layerfs branch publish [] [--expected-base ] [--expected-head ] [--force]" - ); -} - -#[test] -fn the_options_take_inline_values_and_a_boolean_force() { - // `--flag=value` is the same option as `--flag value`, and a boolean - // `--force` carries no value of its own. - let inline = parse(&[ - "shop/api", - "work", - &format!("--expected-base={BASE}"), - &format!("--expected-head={HEAD}"), - "--force=false", - ]) - .expect("the inline options parse"); - assert_eq!(inline.args.get_text("expected_base"), Some(BASE)); - assert_eq!(inline.args.get_text("expected_head"), Some(HEAD)); - assert!( - matches!(inline.args.get("force"), Some(SdkValue::Bool(false))), - "{:?}", - inline.args - ); - - // Options may come before the positionals: `--force` consumes no word. - let ordered = parse(&["--force", "shop/api", "work"]).expect("the flag parses"); - assert!( - matches!(ordered.args.get("force"), Some(SdkValue::Bool(true))), - "{:?}", - ordered.args - ); -} - -#[test] -fn a_malformed_expectation_is_a_usage_error() { - assert_eq!( - usage_error(&["shop/api", "work", "--expected-base"]), - "--expected-base needs a value" - ); - assert_eq!( - usage_error(&["shop/api", "work", "--force=yes"]), - "branch.publish: `force` is true or false" - ); - let unknown = usage_error(&["shop/api", "work", &format!("--expected-bass={BASE}")]); - assert!( - unknown.starts_with("branch.publish: unknown option --expected-bass\naccepted options: "), - "{unknown}" - ); - // The refusal lists the flags the call accepts, including the new ones. - for flag in ["--expected-base", "--expected-head", "--force"] { - assert!(unknown.contains(flag), "{flag}: {unknown}"); - } -} diff --git a/api/node/src/index.d.ts b/api/node/src/index.d.ts index a37703ec6d..9fcdf3a9f4 100644 --- a/api/node/src/index.d.ts +++ b/api/node/src/index.d.ts @@ -131,13 +131,6 @@ export interface BranchMergeOptions extends CallOptions { units?: Value; } -/** Options of `branch.publish`; an unknown option is refused. */ -export interface BranchPublishOptions extends CallOptions { - expected_base?: string; - expected_head?: string; - force?: boolean; -} - /** Options of `branch.diff`; an unknown option is refused. */ export interface BranchDiffOptions extends CallOptions { paths_only?: boolean; @@ -327,11 +320,6 @@ export interface MachineStartOptions extends CallOptions { on_progress?: ProgressHandler; } -/** Options of `machine.stop`; an unknown option is refused. */ -export interface MachineStopOptions extends CallOptions { - force?: boolean; -} - /** Options of `machine.create`; an unknown option is refused. */ export interface MachineCreateOptions extends CallOptions { cpus?: WholeNumber; @@ -435,9 +423,9 @@ export interface BranchFamily { fork(project: string, name: string, from_branch: string, from_commit?: string | null, options?: CallOptions): Promise; /** merge two Branches into a new one */ merge(project: string, name: string, a: string, b: string, options?: BranchMergeOptions): Promise; - /** a Branch's head or Commit as main's next Layer, refusing a stale base */ - publish(project: string, branch: string, options: BranchPublishOptions): Promise; - publish(project: string, branch: string, commit?: string | null, options?: BranchPublishOptions): Promise; + /** a Branch's head or Commit as main's next Layer */ + publish(project: string, branch: string, options: CallOptions): Promise; + publish(project: string, branch: string, commit?: string | null, options?: CallOptions): Promise; /** files that differ between two Branches' heads */ diff(project: string, from_branch: string, to_branch: string, options?: BranchDiffOptions): Promise; } @@ -537,7 +525,7 @@ export interface MachineFamily { /** start a stopped machine and its worker; a running one answers at once */ start(name: string, options?: MachineStartOptions): Promise; /** stop a machine's worker and VM; its disks stay */ - stop(name: string, options?: MachineStopOptions): Promise; + stop(name: string, options?: CallOptions): Promise; /** create an enabled VM machine (apple-container, lima) */ create(name: string, platform: string, options?: MachineCreateOptions): Promise; /** delete a VM machine and its disk; one of a removed platform is only forgotten */ diff --git a/api/node/test/package.mjs b/api/node/test/package.mjs index d9a91b46f0..b94739a0d6 100644 --- a/api/node/test/package.mjs +++ b/api/node/test/package.mjs @@ -42,14 +42,6 @@ test("index.d.ts is exactly what the call table generates", () => { test("every call, argument and option of the table is in the typings", () => { const types = readFileSync(here("../src/index.d.ts"), "utf8"); const pascal = (word) => word.replace(/(^|_)([a-z])/g, (_, __, letter) => letter.toUpperCase()); - // the annotation each table kind is written as; a call's own options - // interface must name its options with exactly these (#691: `branch.publish` - // takes `expected_base`, `expected_head` and `force`). - const ARGUMENT = { - project: "string", name: "string", text: "string", int: "WholeNumber", bool: "boolean", - size: "Size", content: "Content", json: "Value", image: "string", - }; - const FUNCTION = { on_progress: "ProgressHandler", stdout: "OutputHandler", stderr: "OutputHandler" }; for (const spec of table) { const block = types.split(`export interface ${pascal(spec.family)}Family {\n`)[1].split("\n}\n")[0]; // the overloads that take the options right after the required positionals @@ -59,22 +51,6 @@ test("every call, argument and option of the table is in the typings", () => { assert.ok(lines.length > 0 && lines.every((text) => text.endsWith("): Promise;")), `${spec.family}.${spec.call}`); assert.ok(block.includes(` /** ${spec.help} */\n${lines[0]}`), `${spec.family}.${spec.call} help`); for (const arg of spec.args) assert.ok(line.includes(`${arg.name}${arg.optional ? "?" : ""}: `), `${spec.family}.${spec.call}(${arg.name})`); - if (spec.options.length === 0) { - assert.ok(line.endsWith("options?: CallOptions): Promise;"), `${spec.family}.${spec.call}: CallOptions`); - continue; - } - const name = `${pascal(spec.family)}${pascal(spec.call)}Options`; - assert.ok(line.endsWith(`options?: ${name}): Promise;`), `${spec.family}.${spec.call}: options?.${name}`); - const declaration = `export interface ${name} extends CallOptions {\n`; - const at = types.indexOf(declaration); - assert.ok(at >= 0, `${spec.family}.${spec.call}: no ${name} interface`); - const declared = types.slice(at + declaration.length).split("\n}\n")[0]; - const wanted = spec.options.map((opt) => { - const type = opt.kind === "function" ? FUNCTION[opt.name] : ARGUMENT[opt.kind]; - assert.ok(type, `${spec.family}.${spec.call}: no typings for ${opt.kind} option ${opt.name}`); - return ` ${opt.name}?: ${type};`; - }); - assert.equal(declared, wanted.join("\n"), `${spec.family}.${spec.call}: ${name} options`); for (const opt of spec.options) assert.ok(types.includes(` ${opt.name}?: `), `${spec.family}.${spec.call} ${opt.name}`); } }); diff --git a/api/python/src/layerfs/_client.pyi b/api/python/src/layerfs/_client.pyi index 6f24e42735..21fb65aa65 100644 --- a/api/python/src/layerfs/_client.pyi +++ b/api/python/src/layerfs/_client.pyi @@ -101,18 +101,8 @@ class BranchFamily: log_tag: str | None = None, ) -> Result: """merge two Branches into a new one""" - def publish( - self, - project: str, - branch: str, - commit: str | None = None, - *, - expected_base: str | None = None, - expected_head: str | None = None, - force: bool | None = None, - log_tag: str | None = None, - ) -> Result: - """a Branch's head or Commit as main's next Layer, refusing a stale base""" + def publish(self, project: str, branch: str, commit: str | None = None, *, log_tag: str | None = None) -> Result: + """a Branch's head or Commit as main's next Layer""" def diff( self, project: str, @@ -455,7 +445,7 @@ class MachineFamily: log_tag: str | None = None, ) -> Result: """start a stopped machine and its worker; a running one answers at once""" - def stop(self, name: str, *, force: bool | None = None, log_tag: str | None = None) -> Result: + def stop(self, name: str, *, log_tag: str | None = None) -> Result: """stop a machine's worker and VM; its disks stay""" def create( self, diff --git a/api/python/src/layerfs/aio.pyi b/api/python/src/layerfs/aio.pyi index f2ae365f06..2fc6d18ff7 100644 --- a/api/python/src/layerfs/aio.pyi +++ b/api/python/src/layerfs/aio.pyi @@ -105,12 +105,9 @@ class AsyncBranchFamily: branch: str, commit: str | None = None, *, - expected_base: str | None = None, - expected_head: str | None = None, - force: bool | None = None, log_tag: str | None = None, ) -> Result: - """a Branch's head or Commit as main's next Layer, refusing a stale base""" + """a Branch's head or Commit as main's next Layer""" async def diff( self, project: str, @@ -460,7 +457,7 @@ class AsyncMachineFamily: log_tag: str | None = None, ) -> Result: """start a stopped machine and its worker; a running one answers at once""" - async def stop(self, name: str, *, force: bool | None = None, log_tag: str | None = None) -> Result: + async def stop(self, name: str, *, log_tag: str | None = None) -> Result: """stop a machine's worker and VM; its disks stay""" async def create( self, diff --git a/api/python/test/test_calls.py b/api/python/test/test_calls.py index 03245d70b8..adeabfb50e 100644 --- a/api/python/test/test_calls.py +++ b/api/python/test/test_calls.py @@ -343,22 +343,6 @@ def test_the_table(self): self.assertEqual(len(layerfs.TABLE), 73) for spec in layerfs.TABLE: self.assertTrue(callable(getattr(getattr(layerfs, spec["family"]), spec["call"]))) - # the options the Publish change added are in the table the Python - # front end is built from, so `publish(expected_base=...)` is a keyword - # option rather than an unknown one (#691) - publish = next( - spec - for spec in layerfs.TABLE - if (spec["family"], spec["call"]) == ("branch", "publish") - ) - self.assertEqual( - [(opt["name"], opt["kind"], opt["optional"]) for opt in publish["options"]], - [ - ("expected_base", "name", True), - ("expected_head", "name", True), - ("force", "bool", True), - ], - ) with self.assertRaises(layerfs.LayerFSError) as caught: layerfs.client(home=temp("bogus")).store.get("shop", bogus=1) self.assertEqual(caught.exception.code, "InvalidInput") diff --git a/control/AGENTS.md b/control/AGENTS.md index 02325fd53a..cfb0293736 100644 --- a/control/AGENTS.md +++ b/control/AGENTS.md @@ -88,19 +88,26 @@ from it and a test compares them. ## Rules that stay -- The routes, worker and Store-kind rules are the root guide's - [Where calls run](../AGENTS.md#where-calls-run); control code implements - them in `place.rs` and `support.rs`. -- A PostgreSQL password is never written into a record, log, trace or - `Debug` output. +- sqlite Store calls run in a process on the Store's kernel (in process on + `native`, else that machine's worker); PostgreSQL Store calls and bucket + I/O run in the caller's process (#533 G2). A sqlite Store is on exactly one + machine; a PostgreSQL Store's Sandboxes run on the machines it is attached to + (`store.attach`, each checked from that machine; supersedes #535 G3). +- Workers listen on loopback behind a secret; one attempt per call, no + retry, no replay; an uncertain outcome is reported, never resent. +- A PostgreSQL password is resolved only in the caller, travels only in one + worker request and the Sandbox's private daemon setup file, and is never + written into a record, log, trace or `Debug` output. - Stored and displayed values do not follow internal renames: the `host_process` backend is stored and shown as `"native"` (#535 G6). -Control follows the engine guide's +All control product code follows the [production size and responsibility rules](../engine/AGENTS.md#production-file-size-and-responsibility) -and [checks](../engine/AGENTS.md#checks-and-completion), and the root guide's -[rules you must not break](../AGENTS.md#rules-you-must-not-break). The unsafe -allowance in control stays the one file +and [checks and completion rules](../engine/AGENTS.md#checks-and-completion): +production files at most 999 lines, `lib.rs`/`mod.rs` at most 200 lines of +declarations and re-exports, no tests or fixtures in `src/`, bounded test +commands, locked dependencies, per-commit production LOC and the +Disposable-only profile. The unsafe allowance in control stays the one file the guard names: the Linux `local/os/linux/seccomp.rs` of `layerfs-sandbox-backend`. The macOS process facts (`proc_pidinfo`) moved with `LocalProcess` to the bridge's audited `process/darwin.rs` (#112). diff --git a/control/crates/layerfs-store/src/calls/branch.rs b/control/crates/layerfs-store/src/calls/branch.rs index 023d276bb2..20dc55f4e9 100644 --- a/control/crates/layerfs-store/src/calls/branch.rs +++ b/control/crates/layerfs-store/src/calls/branch.rs @@ -106,8 +106,7 @@ pub struct BranchMerge { pub units: Option, } -/// `branch.publish(project, branch, commit, { expected_base, expected_head, -/// force })`. +/// `branch.publish(project, branch, commit)`. #[derive(Clone, Debug, Default)] pub struct BranchPublish { /// The project. @@ -116,12 +115,6 @@ pub struct BranchPublish { pub branch: String, /// The Commit, else its head. pub commit: Option, - /// The base Layer the caller expects the Branch and main to be at. - pub expected_base: Option, - /// The Branch head Commit the caller expects. - pub expected_head: Option, - /// Publish over a moved main, as before. - pub force: Option, } /// `branch.diff(project, from_branch, to_branch, { paths_only, page_size, diff --git a/control/crates/layerfs-store/src/calls/op.rs b/control/crates/layerfs-store/src/calls/op.rs index e8bdd8f653..26308d38cf 100644 --- a/control/crates/layerfs-store/src/calls/op.rs +++ b/control/crates/layerfs-store/src/calls/op.rs @@ -296,10 +296,7 @@ impl StoreOp<'_> { .json("units", r.units.as_ref()), Self::Publish(r) => args .text("branch", &r.branch) - .opt("commit", r.commit.as_deref()) - .opt("expectedBase", r.expected_base.as_deref()) - .opt("expectedHead", r.expected_head.as_deref()) - .boolean("force", r.force), + .opt("commit", r.commit.as_deref()), Self::DiffBranches(r) => args .text("from", &r.from_branch) .text("to", &r.to_branch) diff --git a/control/crates/layerfs-store/tests/store_ops.rs b/control/crates/layerfs-store/tests/store_ops.rs index 3ecd237aec..93f6c28e27 100644 --- a/control/crates/layerfs-store/tests/store_ops.rs +++ b/control/crates/layerfs-store/tests/store_ops.rs @@ -2,7 +2,7 @@ //! the same names and order the call's args map gave before, an absent //! option not sent. use layerfs_bridge::control::SdkValue; -use layerfs_store::calls::branch::{BranchDiff, BranchFork, BranchMerge, BranchPublish}; +use layerfs_store::calls::branch::{BranchDiff, BranchFork, BranchMerge}; use layerfs_store::calls::file::{FileApply, FileDiff, FileRead, FileWrite, Side}; use layerfs_store::calls::{Change, Content, DiffOptions, Page, StoreOp}; @@ -265,37 +265,3 @@ fn saved_query_selectors_and_numbered_pages_are_forwarded_without_resolving_head ] ); } - -#[test] -fn a_publish_sends_its_expected_base_head_and_force() { - // The API layer's snake_case options reach the Store as the camelCase - // names its method reads (#691); an absent expectation is not sent. - let publish = BranchPublish { - branch: t("work"), - commit: Some(t("cc")), - expected_base: Some(t("aa")), - expected_head: Some(t("bb")), - force: Some(false), - ..BranchPublish::default() - }; - let op = StoreOp::Publish(&publish); - assert_eq!(op.method(), "branch.publish"); - assert_eq!( - op.engine_args(), - vec![ - (t("branch"), SdkValue::text("work")), - (t("commit"), SdkValue::text("cc")), - (t("expectedBase"), SdkValue::text("aa")), - (t("expectedHead"), SdkValue::text("bb")), - (t("force"), SdkValue::Bool(false)), - ] - ); - let plain = BranchPublish { - branch: t("work"), - ..BranchPublish::default() - }; - assert_eq!( - StoreOp::Publish(&plain).engine_args(), - vec![(t("branch"), SdkValue::text("work"))] - ); -} diff --git a/core/AGENTS.md b/core/AGENTS.md index 30fd045587..fe4cd89699 100644 --- a/core/AGENTS.md +++ b/core/AGENTS.md @@ -42,10 +42,16 @@ formats are introduced. No third-party source or dependency changes are implied. ## Checks and completion -Core follows the engine guide's +All product layers follow the [production size and responsibility rules](../engine/AGENTS.md#production-file-size-and-responsibility) -and [checks](../engine/AGENTS.md#checks-and-completion), and the root guide's -[rules you must not break](../AGENTS.md#rules-you-must-not-break). Build and -test through the selected worktree's registered private resources from the -root manifest. For docs-only changes verify local links/anchors, status, -claims and whitespace; Rust/runtime checks do not apply. +and [verification rules](../engine/AGENTS.md#checks-and-completion). Build and +test through the selected worktree's registered private resources from the root +manifest. Build tests first with `--no-run`, then execute each selection under an +explicit wall timeout of at most 120 seconds. Preserve repository ARM64 flags, +locked dependencies, exact failure custody, the Disposable-only execution policy +and per-commit production LOC comparisons. Migrations retain cheap tests of +surviving behavior and delete tests that require retired scaffolding, with the +root guide's required behavior coverage and deletion record. + +For docs-only changes verify local links/anchors, status, claims and whitespace; +Rust/runtime checks do not apply. CI reports after a push and never gates it. diff --git a/core/crates/layerfs-history/src/contract/catalog.rs b/core/crates/layerfs-history/src/contract/catalog.rs index 9a39d81c1f..431c5563a7 100644 --- a/core/crates/layerfs-history/src/contract/catalog.rs +++ b/core/crates/layerfs-history/src/contract/catalog.rs @@ -21,8 +21,8 @@ use crate::identity::{BranchId, CatalogId, CommitId, HistoryName, LayerId, Layer use crate::records::{ AddLayerOutcome, AddLayerRequest, BranchMergeRequest, BranchRecord, BranchSnapshot, CommitHistoryRequest, CommitOutcome, CommitRecord, CommitRequest, ForkRequest, - LayerHistoryRequest, LayerRecord, LayerStackRecord, Page, PageResult, Reservation, - ReserveRequest, StackInitialization, + LayerHistoryRequest, LayerRecord, LayerStackRecord, Page, PageResult, PathRemovalOutcome, + PathRemovalRecord, PathRemovalRequest, Reservation, ReserveRequest, StackInitialization, }; /// Checked inputs that establish one catalog binding. #[derive(Clone, Eq, PartialEq)] @@ -195,17 +195,6 @@ pub trait HistoryCatalog: Send + Sync { )) } - /// The Layer `commit` of `branch` was published as, if it was: the same - /// record `add_layer` answers `UpToDate` from. One bounded read. - fn published_layer( - &self, - branch: BranchId, - commit: CommitId, - ) -> HistoryResult> { - let _ = (branch, commit); - Err(crate::error::HistoryError::Unsupported("published Layer")) - } - /// Reads maintained Store totals; no full object/history scan. fn statistics(&self) -> HistoryResult { Err(crate::HistoryError::Unsupported("Store statistics")) @@ -287,6 +276,121 @@ pub trait HistoryCatalog: Send + Sync { /// Consumes one checked half-open inode range for a scope. fn reserve_inodes(&self, request: &ReserveRequest) -> HistoryResult; + + /// Records a `branch.remove` marker. The default refuses. + /// + /// # Errors + /// + /// The default returns [`crate::error::HistoryError::Unsupported`] + /// because it keeps no removal markers; an implementation returns its own + /// [`crate::error::HistoryError`] when the marker cannot be recorded. + fn mark_branch_removed(&self, _branch: BranchId, _removed_ms: i64) -> HistoryResult<()> { + Err(crate::error::HistoryError::Unsupported( + "branch removal marker", + )) + } + + /// Records a `project.delete` marker. The default refuses. + /// + /// # Errors + /// + /// The default returns [`crate::error::HistoryError::Unsupported`] + /// because it keeps no deletion markers; an implementation returns its own + /// [`crate::error::HistoryError`] when the marker cannot be recorded. + fn mark_stack_deleted(&self, _stack: LayerStackId, _deleted_ms: i64) -> HistoryResult<()> { + Err(crate::error::HistoryError::Unsupported( + "project deletion marker", + )) + } + + /// The deleted_ms a project.delete marker recorded, when the stack is + /// retired. A same-home retry reads it by the stack id its durable intent + /// recorded before the marker; a live or absent stack has none. The + /// default refuses. + fn stack_deleted_ms(&self, _stack: LayerStackId) -> HistoryResult> { + Err(crate::error::HistoryError::Unsupported( + "project deletion marker read", + )) + } + + /// Records a `project.remove_file` marker. The default refuses. + /// + /// # Errors + /// + /// The default returns [`crate::error::HistoryError::Unsupported`] + /// because it keeps no path removal markers; an implementation returns its own + /// [`crate::error::HistoryError`] when the marker cannot be recorded. + fn mark_path_removed( + &self, + _stack: LayerStackId, + _path: &[u8], + _frontier_serial: u64, + _removed_ms: i64, + ) -> HistoryResult<()> { + Err(crate::error::HistoryError::Unsupported( + "path removal marker", + )) + } + + /// Applies one `project.remove_file` marker in one write transaction: + /// reads the Store high-water as the new frontier, returns `UpToDate` + /// when the path's marker already covers every project node, otherwise + /// advances the marker and the project epoch. The default refuses. + fn remove_path(&self, _request: &PathRemovalRequest) -> HistoryResult { + Err(crate::error::HistoryError::Unsupported("path removal")) + } + + /// Every `path_removed` marker of one Stack. The default refuses. + fn path_removals(&self, _stack: LayerStackId) -> HistoryResult> { + Err(crate::error::HistoryError::Unsupported( + "path removal markers", + )) + } + + /// The `history_serial` a Commit or Layer records, if it exists. The + /// default refuses. + fn node_history_serial(&self, _node: NodeId) -> HistoryResult> { + Err(crate::error::HistoryError::Unsupported( + "node history position", + )) + } + + /// The retained history roots of one Stack. The default refuses. + /// + /// # Errors + /// + /// The default returns [`crate::error::HistoryError::Unsupported`] + /// because it retains no roots; an implementation returns its own + /// [`crate::error::HistoryError`] when the roots cannot be read. + fn retained_roots(&self, _stack: LayerStackId) -> HistoryResult> { + Err(crate::error::HistoryError::Unsupported("retained roots")) + } + + /// The retained history roots of one Stack, for a caller that resolves + /// them separately. The default refuses. + /// + /// # Errors + /// + /// The default returns [`crate::error::HistoryError::Unsupported`] + /// because it retains no roots; an implementation returns its own + /// [`crate::error::HistoryError`] when the roots cannot be read. + fn get_retained_roots(&self, _stack: LayerStackId) -> HistoryResult> { + Err(crate::error::HistoryError::Unsupported("retained roots")) + } + + /// Removes the unreachable history rows of one Stack, or reports how many + /// a dry run would remove. The default refuses. + /// + /// # Errors + /// + /// The default returns [`crate::error::HistoryError::Unsupported`] + /// because it removes nothing; an implementation returns its own + /// [`crate::error::HistoryError`] when the rows cannot be removed. + fn remove_unreachable(&self, _stack: LayerStackId, _dry_run: bool) -> HistoryResult { + Err(crate::error::HistoryError::Unsupported( + "unreachable history removal", + )) + } } impl std::fmt::Debug for HistoryCatalogConfig { diff --git a/core/crates/layerfs-history/src/contract/error.rs b/core/crates/layerfs-history/src/contract/error.rs index 3cede4537b..e4b686b3d0 100644 --- a/core/crates/layerfs-history/src/contract/error.rs +++ b/core/crates/layerfs-history/src/contract/error.rs @@ -25,6 +25,99 @@ pub enum Missing { Catalog, } +/// Which kind of record holds a name. +/// +/// A later release may add a holder kind, so the enum is non-exhaustive: a +/// caller matches it with a wildcard arm and never constructs it. +#[non_exhaustive] +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum HolderKind { + /// A Branch. + Branch, + /// A project (LayerStack). + Project, +} + +/// State of the record that holds a name. +/// +/// A later release may add a state, so the enum is non-exhaustive: a +/// caller matches it with a wildcard arm and never constructs it. +#[non_exhaustive] +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum HolderState { + /// The record is live. + Live, + /// The record is retired by `branch.remove`. + Retired, + /// The project is deleted. + Deleted, +} + +/// The record that holds a name. +/// +/// A later release may add a field, so the struct is non-exhaustive: +/// a caller reads its fields, and construction belongs to this crate. +#[non_exhaustive] +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct NameHolder { + /// Record kind. + pub kind: HolderKind, + /// Name held. + pub name: String, + /// Holder state. + pub state: HolderState, +} + +/// Where a name is held. +/// +/// A later release may add a source, so the enum is non-exhaustive: a +/// caller matches it with a wildcard arm and never constructs it. +#[non_exhaustive] +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum NameSource { + /// The Store's own records. + Store, + /// A home project record. + Home, +} + +/// Context for a `NameTaken` refusal. +/// +/// A later release may add a field, so the struct is non-exhaustive: +/// a caller reads its fields, and construction belongs to this crate. +#[non_exhaustive] +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct NameTakenState { + /// The holder. + pub holder: NameHolder, + /// Where it is held. + pub source: NameSource, +} + +/// One hidden path. +/// +/// A later release may add a field, so the struct is non-exhaustive: +/// a caller reads its fields, and construction belongs to this crate. +#[non_exhaustive] +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct RemovedPath { + /// The request path. + pub path: String, + /// The governing frontier serial. + pub frontier_serial: u64, +} + +/// Context for a `Removed` refusal. +/// +/// A later release may add a field, so the struct is non-exhaustive: +/// a caller reads its fields, and construction belongs to this crate. +#[non_exhaustive] +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct RemovedState { + /// The hidden paths. + pub paths: Vec, +} + /// Expected versus observed Branch publication state. /// /// Both sides are reported, so a caller can distinguish a moved head from a @@ -50,6 +143,14 @@ pub enum HistoryError { Missing(Missing), /// A required capability is not provided by this build or provider. Unsupported(&'static str), + /// A name is held by a live, retired or deleted record. + NameTaken(Box), + /// A path is hidden by `remove_file`. + Removed(Box), + /// A required marker or fence row is missing or contradicts the request. + Conflict(&'static str), + /// The Store was replaced or moved under this handle. + StaleStore, /// Immediate metadata admission was refused; the caller may retry later. Busy, /// The content authority this catalog is bound to is not available. @@ -89,11 +190,13 @@ pub enum HistoryError { impl HistoryError { /// True when the attempted mutation's persistence outcome is unknown. + #[must_use] pub fn unknown(&self) -> bool { matches!(self.cause(), Self::UnknownOutcome) } /// The typed failure itself; kept for callers that match through it. + #[must_use] pub fn cause(&self) -> &Self { self } @@ -105,6 +208,10 @@ impl std::fmt::Display for HistoryError { Self::InvalidInput(what) => write!(formatter, "invalid input: {what}"), Self::Missing(what) => write!(formatter, "missing {what:?}"), Self::Unsupported(what) => write!(formatter, "unsupported: {what}"), + Self::NameTaken(state) => write!(formatter, "name taken: {:?}", state.holder), + Self::Removed(state) => write!(formatter, "removed: {} paths", state.paths.len()), + Self::Conflict(what) => write!(formatter, "conflict: {what}"), + Self::StaleStore => formatter.write_str("stale Store"), Self::Busy => formatter.write_str("busy"), Self::OwnershipUnavailable => formatter.write_str("ownership unavailable"), Self::Capacity(what) => write!(formatter, "capacity: {what}"), @@ -128,3 +235,56 @@ impl std::error::Error for HistoryError {} /// Result of one history operation. pub type HistoryResult = Result; + +impl NameHolder { + /// Builds a holder for a typed `NameTaken` context. + pub fn new(kind: HolderKind, name: impl Into, state: HolderState) -> Self { + Self { + kind, + name: name.into(), + state, + } + } +} + +impl NameTakenState { + /// Builds the context of a `NameTaken` refusal. + pub fn new(holder: NameHolder, source: NameSource) -> Self { + Self { holder, source } + } +} + +impl RemovedPath { + /// Builds one hidden path for a `Removed` refusal. + pub fn new(path: impl Into, frontier_serial: u64) -> Self { + Self { + path: path.into(), + frontier_serial, + } + } +} + +impl RemovedState { + /// Builds the context of a `Removed` refusal. + pub fn new(paths: Vec) -> Self { + Self { paths } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn constructors_build_the_typed_context() { + let holder = NameHolder::new(HolderKind::Branch, "main", HolderState::Retired); + let taken = NameTakenState::new(holder, NameSource::Store); + assert_eq!(taken.holder.name, "main"); + assert_eq!(taken.holder.kind, HolderKind::Branch); + assert_eq!(taken.source, NameSource::Store); + + let removed = RemovedState::new(vec![RemovedPath::new("a.txt", 7)]); + assert_eq!(removed.paths[0].path, "a.txt"); + assert_eq!(removed.paths[0].frontier_serial, 7); + } +} diff --git a/core/crates/layerfs-history/src/contract/mod.rs b/core/crates/layerfs-history/src/contract/mod.rs index 37313bcbc2..520aebbe72 100644 --- a/core/crates/layerfs-history/src/contract/mod.rs +++ b/core/crates/layerfs-history/src/contract/mod.rs @@ -2,5 +2,6 @@ pub mod catalog; pub mod error; pub mod identity; +pub mod path; pub mod query; pub mod records; diff --git a/core/crates/layerfs-history/src/contract/path.rs b/core/crates/layerfs-history/src/contract/path.rs new file mode 100644 index 0000000000..dc751b0403 --- /dev/null +++ b/core/crates/layerfs-history/src/contract/path.rs @@ -0,0 +1,154 @@ +//! The `project.remove_file` path grammar and its keys. +//! +//! The call accepts an absolute POSIX path and stores it as the normalized +//! relative path (`/a` and `a` are the same). A path is refused when it is +//! empty, `/`, `//a`, has an empty, `.` or `..` component, a trailing `/`, a +//! `\` or NUL byte, is longer than 4096 bytes, or has more than 256 +//! components. The stored key is `BLAKE3(normalized path)`; the trace key is +//! keyed by the Store's `trace_salt` over the stack and the normalized path. + +use crate::identity::LayerStackId; + +/// The most bytes one normalized path may have. +pub const MAXIMUM_PATH_BYTES: usize = 4096; +/// The most components one normalized path may have. +pub const MAXIMUM_PATH_COMPONENTS: usize = 256; + +/// A checked, normalized relative path. +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct NormalizedPath { + relative: String, +} + +impl NormalizedPath { + /// Checks `text` and returns its normalized relative path. + /// + /// One leading `/` is stripped; everything after it is the relative path. + /// The returned value never starts or ends with `/` and never holds an + /// empty, `.` or `..` component. + pub fn new(text: &str) -> Result { + if text.is_empty() { + return Err("path is empty"); + } + if text.as_bytes().contains(&0) { + return Err("path contains a NUL byte"); + } + if text.contains('\\') { + return Err("path contains a backslash"); + } + let relative = text.strip_prefix('/').unwrap_or(text); + if relative.is_empty() { + return Err("path is the root"); + } + if relative.starts_with('/') { + return Err("path has an empty component"); + } + let mut components = 0usize; + for component in relative.split('/') { + if component.is_empty() { + return Err("path has an empty component"); + } + if component == "." || component == ".." { + return Err("path has a dot component"); + } + components += 1; + } + if components > MAXIMUM_PATH_COMPONENTS { + return Err("path has too many components"); + } + if relative.len() > MAXIMUM_PATH_BYTES { + return Err("path is too long"); + } + Ok(Self { + relative: relative.to_owned(), + }) + } + + /// The normalized relative path, without a leading `/`. + pub fn as_str(&self) -> &str { + &self.relative + } + + /// The normalized path bytes as stored in the marker. + pub fn as_bytes(&self) -> &[u8] { + self.relative.as_bytes() + } + + /// `BLAKE3(normalized path)`, the marker key. + pub fn key(&self) -> [u8; 32] { + path_key(self.as_bytes()) + } +} + +/// `BLAKE3(path)` over already normalized path bytes, the marker key. +pub fn path_key(path: &[u8]) -> [u8; 32] { + *blake3::hash(path).as_bytes() +} + +/// True when `path` is `marker` or lies below it at a `/` boundary +/// (`p == m OR p begins m ++ b"/"`), compared over bytes, never `||`/`LIKE`. +pub fn governs(marker: &[u8], path: &[u8]) -> bool { + path == marker + || (path.len() > marker.len() && path.starts_with(marker) && path[marker.len()] == b'/') +} + +/// The key a trace hashes for one stack and path: keyed by the Store's +/// 32-byte `trace_salt` over the stack identity and the normalized path. +pub fn trace_key(salt: &[u8; 32], stack: LayerStackId, path: &[u8]) -> [u8; 32] { + let mut hasher = blake3::Hasher::new_keyed(salt); + hasher.update(stack.as_slice()); + hasher.update(path); + *hasher.finalize().as_bytes() +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn accepts_one_leading_slash_and_relative_paths() { + assert_eq!(NormalizedPath::new("/a").unwrap().as_str(), "a"); + assert_eq!(NormalizedPath::new("a").unwrap().as_str(), "a"); + assert_eq!(NormalizedPath::new("/a/b").unwrap().as_str(), "a/b"); + assert_eq!(NormalizedPath::new("a-b_c.d").unwrap().as_str(), "a-b_c.d"); + } + + #[test] + fn refuses_the_bad_grammar() { + for text in [ + "", "/", "//a", "a//b", "a/", "./a", "a/./b", ".", "..", "/./a", "a\\b", "a\0b", + ] { + assert!(NormalizedPath::new(text).is_err(), "accepted {text:?}"); + } + } + + #[test] + fn refuses_long_paths_and_many_components() { + assert!(NormalizedPath::new(&"a".repeat(MAXIMUM_PATH_BYTES)).is_ok()); + assert!(NormalizedPath::new(&"a".repeat(MAXIMUM_PATH_BYTES + 1)).is_err()); + let many = vec!["a"; MAXIMUM_PATH_COMPONENTS].join("/"); + assert!(NormalizedPath::new(&many).is_ok()); + let more = vec!["a"; MAXIMUM_PATH_COMPONENTS + 1].join("/"); + assert!(NormalizedPath::new(&more).is_err()); + } + + #[test] + fn subtree_matches_only_at_a_slash_boundary() { + assert!(governs(b"a", b"a")); + assert!(governs(b"a", b"a/b")); + assert!(!governs(b"a", b"ab")); + assert!(!governs(b"ab", b"a")); + assert!(governs(b"a/b", b"a/b/c")); + assert!(!governs(b"a/b", b"a/bc")); + } + + #[test] + fn keys_are_stable_and_salted_traces_differ() { + let path = NormalizedPath::new("/a/b").unwrap(); + assert_eq!(path.key(), path_key(b"a/b")); + let stack = LayerStackId::from_authority([7; 16]); + let first = trace_key(&[1; 32], stack, path.as_bytes()); + let second = trace_key(&[2; 32], stack, path.as_bytes()); + assert_ne!(first, second); + } +} diff --git a/core/crates/layerfs-history/src/contract/records.rs b/core/crates/layerfs-history/src/contract/records.rs index 5027cc8d47..394ba3de9f 100644 --- a/core/crates/layerfs-history/src/contract/records.rs +++ b/core/crates/layerfs-history/src/contract/records.rs @@ -463,11 +463,10 @@ pub enum CommitOutcome { #[derive(Clone, Debug, Eq, PartialEq)] pub enum AddLayerOutcome { /// A new immutable Layer was inserted on the stack head and the stack - /// advanced. `add_layer` itself is overwrite-only (the SDK's - /// `branch.publish` refuses a stale base before calling it, unless - /// `force`): the Layer's `overwrote_count` counts the stack's Layers - /// since the Branch's base that the published Commit does not contain; - /// they stay in history and `HistoryCatalog::overwritten` lists them. + /// advanced. Publication is overwrite-only: the Layer's + /// `overwrote_count` counts the stack's Layers since the Branch's base + /// that the published Commit does not contain; they stay in history and + /// `HistoryCatalog::overwritten` lists them. Added { /// The new Layer. layer: LayerRecord, @@ -637,3 +636,50 @@ pub struct InitStatistics { /// Init's wall time in milliseconds, up to the first Branch. pub millis: u64, } + +/// One `path_removed` marker as stored for a project. +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct PathRemovalRecord { + /// The normalized relative path the marker names. + pub path: Vec, + /// The Store high-water when the marker last advanced. + pub frontier_serial: u64, + /// When the marker last advanced, in milliseconds since the epoch. + pub removed_ms: i64, +} + +/// Whether a `project.remove_file` call advanced the marker or found it current. +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum PathRemovalKind { + /// The marker advanced to a new frontier. + Removed, + /// No project node is above the marker's frontier; the stored values are returned. + UpToDate, +} + +/// One `project.remove_file` request: the project, the normalized path and the +/// time the caller observed. +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct PathRemovalRequest { + /// Project (LayerStack) the path is removed in. + pub stack: LayerStackId, + /// The normalized relative path bytes; `BLAKE3` keys the marker. + pub path: Vec, + /// When the call was made, in milliseconds since the epoch. + pub removed_ms: i64, +} + +/// One `project.remove_file` result: the stored marker values. +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct PathRemovalOutcome { + /// Whether the call advanced the marker or found it current. + pub kind: PathRemovalKind, + /// Project (LayerStack) the marker belongs to. + pub stack: LayerStackId, + /// The stored normalized relative path. + pub path: Vec, + /// The stored frontier serial. + pub frontier_serial: u64, + /// The stored `removed_ms`. + pub removed_ms: i64, +} diff --git a/core/crates/layerfs-history/src/lib.rs b/core/crates/layerfs-history/src/lib.rs index 0134a0e65d..6f01165217 100644 --- a/core/crates/layerfs-history/src/lib.rs +++ b/core/crates/layerfs-history/src/lib.rs @@ -21,6 +21,7 @@ mod contract; pub use contract::catalog; pub use contract::error; pub use contract::identity; +pub use contract::path; pub use contract::query; pub use contract::records; @@ -34,6 +35,7 @@ pub use records::{ BranchSnapshot, CommitHistoryRequest, CommitMeta, CommitOutcome, CommitRecord, CommitRequest, CommitStatistics, CommitVia, FileDigest, ForkRequest, ForkSource, InitStatistics, LayerHistoryRequest, LayerRecord, LayerStackRecord, MergeCommit, MergeHead, MergeSide, Page, - PageResult, RecordKind, Reservation, ReserveRequest, StackInitialization, StoreStatistics, + PageResult, PathRemovalKind, PathRemovalOutcome, PathRemovalRecord, PathRemovalRequest, + RecordKind, Reservation, ReserveRequest, StackInitialization, StoreStatistics, FILE_DIGEST_WINDOW, MAXIMUM_MESSAGE_BYTES, MAXIMUM_PAGE_RECORDS, }; diff --git a/core/crates/layerfs-history/tests/contract_constructors.rs b/core/crates/layerfs-history/tests/contract_constructors.rs new file mode 100644 index 0000000000..94625590a7 --- /dev/null +++ b/core/crates/layerfs-history/tests/contract_constructors.rs @@ -0,0 +1,70 @@ +//! A provider outside this crate builds each refusal context through the +//! public constructors, and readers still reach the public fields. + +use layerfs_history::error::{ + HolderKind, HolderState, NameHolder, NameSource, NameTakenState, RemovedPath, RemovedState, +}; +use layerfs_history::HistoryError; + +#[test] +fn name_taken_context_is_constructed_and_read() { + let holder = NameHolder::new(HolderKind::Branch, "feature", HolderState::Retired); + assert_eq!(holder.kind, HolderKind::Branch); + assert_eq!(holder.name, "feature"); + assert_eq!(holder.state, HolderState::Retired); + + let state = NameTakenState::new(holder, NameSource::Home); + assert_eq!(state.source, NameSource::Home); + assert_eq!(state.holder.name, "feature"); + + let error = HistoryError::NameTaken(Box::new(state)); + match error.cause() { + HistoryError::NameTaken(state) => { + assert_eq!(state.holder.kind, HolderKind::Branch); + assert_eq!(state.holder.state, HolderState::Retired); + assert_eq!(state.source, NameSource::Home); + } + other => panic!("expected NameTaken, got {other:?}"), + } +} + +#[test] +fn removed_context_is_constructed_and_read() { + let path = RemovedPath::new("/gone.txt", 42); + assert_eq!(path.path, "/gone.txt"); + assert_eq!(path.frontier_serial, 42); + + let state = RemovedState::new(vec![path, RemovedPath::new("/other.txt", 7)]); + assert_eq!(state.paths.len(), 2); + assert_eq!(state.paths[1].path, "/other.txt"); + + let error = HistoryError::Removed(Box::new(state)); + match error.cause() { + HistoryError::Removed(state) => { + assert_eq!(state.paths[0].path, "/gone.txt"); + assert_eq!(state.paths[0].frontier_serial, 42); + assert_eq!(state.paths[1].frontier_serial, 7); + } + other => panic!("expected Removed, got {other:?}"), + } +} + +#[test] +fn every_holder_kind_and_state_builds() { + for kind in [HolderKind::Branch, HolderKind::Project] { + for state in [ + HolderState::Live, + HolderState::Retired, + HolderState::Deleted, + ] { + let holder = NameHolder::new(kind, "name", state); + assert_eq!(holder.kind, kind); + assert_eq!(holder.state, state); + } + } + for source in [NameSource::Store, NameSource::Home] { + let holder = NameHolder::new(HolderKind::Project, "project", HolderState::Live); + let state = NameTakenState::new(holder, source); + assert_eq!(state.source, source); + } +} diff --git a/core/docs/issues/project-41/LayerFS-GC-spec.md b/core/docs/issues/project-41/LayerFS-GC-spec.md index e54ced2101..3cd2a9a942 100644 --- a/core/docs/issues/project-41/LayerFS-GC-spec.md +++ b/core/docs/issues/project-41/LayerFS-GC-spec.md @@ -1,26 +1,36 @@ # LayerFS garbage collection: soft removal and Store compaction -> **Status:** Proposal; target LayerFS 2.0.0; not a released contract. +> **Status:** **Frozen for layerfs-project#41 (LayerFS 2.0.0) on 2026-11-15**, approved by the owner on layerfs-project#42. An amendment needs a new #42 decision entry. +> +> **Baseline.** Commit `ed14ed53b` (LayerFS project main layer +> `32770a67…`) is the frozen **2.0 GC implementation baseline**. Implementation +> starts here; residual review items are tracked as implementation/acceptance +> gaps on layerfs-project#41, not as contract changes. > **Authority.** This is the only GC design document in the tree. It replaces the > six-file set merged by #658 (removed by #671; last text in the #677 branch). -> The contract it states is the frozen set of layerfs-project#42 (items 1–13, -> 2026-11-13) as amended by #42 items 14–26 (2026-11-14) and 27–44 -> (2026-11-15, implementation choices). Where this file and an older document -> disagree, this file wins; a change to a contract sentence needs a new +> The contract it states is the frozen set of layerfs-project#42: S1–S11 +> and items 1–42 (2026-11-13 to 2026-11-15). Items 43–44 are recorded on +> layerfs-project#42 and are not contract text. Where this file and an older +> document disagree, this file wins; a change to a contract sentence needs a new > decision entry on #42 first. > +> **Sign-off.** S1–S11, items 1–42 and the amendments below were approved by +> the owner on layerfs-project#42; this file is the frozen text. The decision +> table indexes every decision that carries contract text here; the complete +> log is layerfs-project#42. +> > **Scope.** layerfs-project#41 and its sub-issues #42–#52. The durability > profile (#155 §7) is outside this contract (#42 item 26). GC is developed > and measured under the Disposable profile (#42 item 42). > > **Source.** Code references are to `layerfs_v2_0_0` at `e8fa0a7e00`; those -> added by items 27–44 were checked at `8cfb6e2734`. +> added by items 27–42 were checked at `8cfb6e2734`. ## 1. Requirements (owner policy, #41) -- Four operations, on SQLite and PostgreSQL: `branch.remove(name)`, - `project.delete(name)`, `project.remove_file(project, path)` and +- Four operations, on SQLite and PostgreSQL: `branch.remove(project, branch)`, + `project.delete(project)`, `project.remove_file(project, path)` and `store.compact(name)`. Each takes only a resource identifier: no confirm, force, dry-run, stop-Sandboxes, Store, URL or volume option. - The three removals are logical. They change visibility and retention and @@ -44,15 +54,21 @@ |---|---|---| | S1 | Reachability is computed per Store, over every project in it | §6 | | S2 | Exclusion reuses the Store lock, lease and op record; PostgreSQL adds advisory locks (#42 items 20, 33, 34) | §5 | +| S3 | Superseded by S5/§7–§8: the rebuild is its own module, not the transfer `Stage`; no transfer-intent sidecar (the rebuild artifacts of §7/§9 are separate) | §7, §8 | | S4 | No object age; collection depends only on reachability | §6 | | S5 | The unit that frees bytes is the pack. A rebuild copies only retained rows into a fresh Store; a published `pack` row is never updated or deleted in place | §7, §8 | | S6 | A SQLite Store compacts on the machine where it lives (`place::plan`, `control/crates/layerfs-store/src/place.rs:40-49`) | §7 | +| S7 | Report the three byte definitions, peak temporary space and residue; estimates carry `estimate: true`; a capped report names its cap | §11 | +| S8 | One fence: `(dev, ino)` per transaction on SQLite, the incarnation join on PostgreSQL; both `StaleStore` | §10 | +| S9 | Install is binary; `live_heads_digest` matches before and after | §7, §8 | | S10 | Logical removal never rewrites history or changes an ID | §4 | -| S11 | Compaction is exclusive over the whole Store; reads are unavailable during it | §5 | +| S11 | Compaction is exclusive against mounts and long-lived users; on PostgreSQL one-shot reads/writes may run until the swap, and a write that spans it makes the swap abandon; SQLite excludes all readers | §5 | +| 6 | Marker and position tables are added to 2.0 Stores at open by an additive step; 1.0 Stores are converted offline | §4.1 | | 7 | Removal markers live in marker tables; `branch` and `layer_stack` stay append-only | §4 | -| 14 | Compaction collects history rows no retained root reaches; Commits of a removed Branch that nothing retains become `NotFound` | §6.1 | +| 12a | PostgreSQL advisory key is `(kind, id)` over the Store, project and Branch scopes | §5.2 | +| 14 | Compaction collects history rows no retained root reaches; Commits of a removed Branch that nothing retains become `NoSuchCommit` | §6.1 | | 15 | A name held by a removed Branch or deleted project returns `NameTaken` until compaction removes the row; then it is free. `NameRetired` is not added | §4.4 | -| 16 | `remove_file` hides a path across the project's history at once and lets compaction reclaim what only that path reached; reads through it return `Removed` (details: item 35) | §4.3, §6.2 | +| 16 | `remove_file` hides a path across the project's history at once; 2.0 is visibility-only and frees no bytes for it (details: item 35) | §4.3, §6.2 | | 17 | `store.compact` has two paths, `none` and `rebuild`; there is no light path. C3 has no caller in 2.0 | §3 | | 18 | A leftover is judged by the Store generation; no `OrphanArtifact`, no leftover cap | §9 | | 19 | API codes follow #41 (`MountsActive`, `Busy`); one `StaleStore` for both backends | §10 | @@ -60,6 +76,7 @@ | 21 | PostgreSQL compaction builds in batches, then swaps two tables in one short transaction | §8 | | 22 | SQLite compaction is its own module: the walk is fused with the copy, the visited set is a temp-file table, verification is cheap by default | §6.3, §7 | | 23 | Removed: the `pinned` set, the per-transaction SQLite generation check, the capacity pre-check, the extra report fields | §6, §7, §11 | +| 26 | The durability profile is outside this contract | header | | 27 | One visit level: anything reached by a logical or base edge is kept with all its references | §6.2 | | 28 | `merged_from_id` and `displaced_id` are history edges | §6.1 | | 29 | A retired Branch is collected as a unit; the history closure is a fixpoint | §6.1 | @@ -68,13 +85,14 @@ | 32 | `project.remove` is retired; compaction reuses its removal statements | §4, §7, §8 | | 33 | Admission at project and Branch scope (project op, intent record before bind, PostgreSQL keys at three scopes) | §5 | | 34 | PostgreSQL presence locks only on long-lived sessions; SDK calls keep working through transaction poolers | §5.2 | -| 35 | `remove_file` marker with a frontier; whiteout at mount; object-ID reads not path-aware; purge best effort | §4.3 | +| 35 | `remove_file` marker with a Store history-serial frontier; every tree-building operation materializes the whiteout, not only mounts; a republished Layer keeps its source Commit's serial; object-ID reads not path-aware; 2.0 frees no bytes (visibility only) | §4.3 | | 36 | No format change for the PostgreSQL mark until #52 measures it | §6.2 | | 37 | All `inode_value_block` rows are kept in 2.0 | §6.4 | | 38 | The rebuild clears `binding_hint_block` | §6.4 | | 39 | All four calls are admin-profile and refused under a folder scope | §10 | | 40 | Compaction writes its phase into the op; `store.status` shows it | §11 | | 41 | A started rebuild completes | §3 | +| 42 | GC is developed and measured under the Disposable profile | header | Out of this release: object age (Q3), repacking partly dead packs (reported, not performed), in-place deletion of packs on either backend, a read-available @@ -87,11 +105,17 @@ compaction window (A'), pruning (#46). | `none` | Bytes in wholly dead packs are below **10%** of the Store's bytes | Mark only; nothing is written; the report gives the reclaimable bytes | unchanged | | `rebuild` | 10% or more | SQLite §7, PostgreSQL §8 | `G → G+1` | -The 10% value is a starting point, set after the first measurement (#52). +The numerator is `COALESCE(SUM(pack.length), 0)` over wholly dead packs and +the denominator `COALESCE(SUM(pack.length), 0)` over all packs, on both +backends; the test is `rebuild iff total > 0 AND dead * 10 >= total` (no +truncating integer division). The 10% value is frozen at 10%; a change needs a +#42 entry. #52 measures it. A Store with no packs, or packs summing to 0, is +`none` with `deadBytesBound = 0` and `zeroReason: "empty-store"`. Below it a rebuild would copy at least nine live bytes for each byte freed. The choice is made on the bound known after the metadata walk; a rebuild that has started completes even if the refined share falls below 10%, and the -report gives both numbers (#42 item 41). +report gives both numbers as `deadBytesBound` and `deadBytesRefined` (§11; +#42 item 41). There is no path that deletes rows in place: it would free no payload, touch most index pages of the hash-keyed tables and need its own deletion contract. @@ -99,39 +123,117 @@ most index pages of the hash-keyed tables and need its own deletion contract. All three removals write one marker row in one short write transaction. No pack, object, Commit or Layer row changes, and no ID changes. They do not -bump the Store generation. +bump the Store generation, but they do bump the project's `epoch` in +`project_epoch` (§5), which a concurrent writer checks before it commits. | Call | Marker table | Effect | |---|---|---| -| `branch.remove(name)` | `branch_retired(branch_id, removed_ms)` | Hidden from listings and name resolution; fork, merge, publish and mount refused; `main` → `ReservedName` | -| `project.delete(name)` | `layer_stack_retired(layer_stack_id, deleted_ms)` | Project hidden; its home entry is removed; its data stays until compaction; the Store stays addressable | -| `project.remove_file(project, path)` | `path_removed(layer_stack_id, path, frontier, removed_ms)` | §4.3 | +| `branch.remove(project, branch)` | `branch_retired(branch_id BLOB NOT NULL PRIMARY KEY REFERENCES branch(branch_id) ON DELETE CASCADE, removed_ms INTEGER NOT NULL) STRICT` | Hidden from listings and name resolution; fork, merge, publish and mount refused; `main` → `ReservedName` | +| `project.delete(project)` | `layer_stack_retired(layer_stack_id, deleted_ms)` | Project hidden; its home entry is removed; its data stays until compaction; the Store stays addressable | +| `project.remove_file(project, path)` | `path_removed(layer_stack_id BLOB NOT NULL, path_key BLOB/bytea NOT NULL, path BLOB/bytea NOT NULL, frontier_serial INTEGER NOT NULL, removed_ms INTEGER NOT NULL) STRICT`, `PRIMARY KEY(layer_stack_id, path_key)`, `REFERENCES layer_stack ON DELETE CASCADE`; `path_key = BLAKE3(normalized path)` (a 4096-byte `bytea` key cannot fit a PG btree), the stored `path` is compared on collision; `frontier_serial` is the marker frontier (§4.3) | §4.3 | `project.delete` is the only project removal: the hard-deleting -`project.remove` is retired, and compaction reuses its `removal_*` statements -(`engine/crates/layerfs-persistence/src/history/removal.rs`) to drop a deleted -project's history (#42 item 32). +`project.remove` is removed from the call table (an old client that sends it +gets `InvalidInput` "no call named `project.remove`", never a silent hard +delete), and its module is deleted; compaction reuses its `removal_*` +statements (`engine/crates/layerfs-persistence/src/history/removal.rs`) to +drop a deleted project's history (#42 item 32). Tests that call +`project.remove` migrate to `project.delete`. Retirement also covers the +worker op/route for `project.remove`, the remote-SQLite route for +`project.delete`, the legacy own-project identity save (project name == Store +name) before the home entry is removed, the call-count and MCP-profile +generators, and the PostgreSQL path. The worker (the Store's home) is named +`project.delete`; the frame is `{store, stack_id, deleted_ms?}` (the caller +writes `deleted_ms` to the sidecar before the call and resends it on +retries). The worker saves the legacy own-project identity, writes the +marker, removes its own entry and replies +`{kind: "Removed", project, stack_id, deleted_ms}`; the caller removes only +its own entry and echoes `home_entry_removed`. A lost reply is `Unknown`, +keeps the sidecar, and the retry resolves retired/live/absent; the old +`project.remove` worker arm is deleted. + +`project.delete` commits the Store marker first and removes the caller's home +entry after it. The marker is the commit point. If the home removal fails the +call returns success with `home_entry_removed: false` and the stored +`deleted_ms`; a retry from the same home is idempotent: it may read the +retired stack allow-listed, reuses `deleted_ms`, finishes the entry removal +and returns `home_entry_removed: true`. The durable delete intent +`{stack_id, deleted_ms, attempt_ms}` lives outside the removed entry (or the +entry is **not** renamed; one sidecar file at +`/store//delete/.json` with +`{store, project, stack_id, deleted_ms, attempt_ms, state}` is written +(temp + fsync + rename + dir fsync) after the identity save and before the +marker, with `pending|marked|done` updated atomically and unlinked after the +entry removal; the retry +re-reads the Store by the recorded stack id and compares it with the home +record, discarding a mismatch without touching a re-created project's entry; +a retired row reuses `deleted_ms` and finishes the entry; a live row with no +marker redoes the marker with the same `deleted_ms` and does **not** remove +the entry; an absent stack row finishes the entry. It never resolves a name. If the marker fails nothing is +removed. The legacy own-project identity save (project name == Store name) +runs before the marker transaction, is idempotent, and a failure refuses with +nothing removed. A different home's stale entry is not rewritten; the +`project.list`/`project.get` intersection in §4.2 hides it. ### 4.1 Representation -The marker tables are added at open by an additive step under the current -schema identity (#42 item 6). On SQLite this is one more `add_*` step, as -`add_file_digests` was; the existing steps, the table list and +The marker and intent tables are added at open by an additive step under +the current schema identity (#42 item 6). On SQLite this is one more `add_*` +step, as `add_file_digests` was; the existing steps, the table list and `tools/store_blocks.py` are updated with it. PostgreSQL has no such step today (its open check requires the exact table set and recorded definition, `backend/postgresql/schema.rs:158-252`), so new Stores get the tables in `schema.sql` and an additive step at open creates them and rewrites the recorded definition for older Stores (#42 item 31). 1.0 Stores are converted -offline. `branch` and `layer_stack` keep their full unique name indexes. +offline. The same step adds the position state: `history_meta.next_history_serial` +(the next serial to assign; never lowered), a `history_serial` column on +`commit` and `layer` (a Layer that republishes a Commit carries that +Commit's serial; every other Commit and Layer takes the next value in its +creation transaction), `project_epoch(layer_stack_id PRIMARY KEY, epoch)` +(one row created with the project, bumped in the same transaction as every +marker write, copied by a rebuild, collected with the project), and two +admission tables: `gc_intent(...)` (§5.1) and `gc_op(...)`. The same step +adds `store_policy.trace_salt` last, written as 32 random bytes by the +Store-creation INSERT (`randomblob(32)`/OS RNG) and backfilled for existing +rows; `store_policy` copies use an explicit column list. `gc_intent` is +the shared Store-side admission table (created empty, never copied); `gc_op` +is PostgreSQL's single compaction row (created by PG's additive step, +untouched by the swap, cleared on settle) with columns `generation`, +`phase`, `scratch_schema`, `mark_serial`, `mark_pack_id`, `mark_digest`, +`mark_totals`, `mark_stamp` (`MAX(content_signature.stamp)`), `started_ms`, +`dead_canonical`, `dead_stored`, `dead_metadata`, +`dead_logical`, `totals_applied`, `artifact`, `pid`, `backend_start`, +`usename`, `incarnation`. The `mark_*` columns are the first-statement tuple; the raw +head rows are hashed by the client. A GC-only `gc_dead(kind, id)` table (same additive step, never +copied, truncated on settle) carries the full dead set; a single `gc_op` row +cannot. SQLite keeps its phase in the +home/sidecar op and never writes the sealed DB, relying on the deterministic +artifact names for cross-machine recovery. The backfill runs on **every** +Store the step upgrades, including an existing 2.0 Store: Commits first by +`(created_ms, stack, id)`; then per Stack the genesis Layer first, a +republishing Layer takes its source Commit's serial, the rest by +`(source Commit created_ms, stack, id)`; `project_epoch(layer_stack_id, 0)` +for every existing stack; and +`next_history_serial = MAX(existing next_history_serial, +COALESCE(MAX(history_serial), 0) + 1)`; a Store with no stack/history keeps +its existing value. `branch` and `layer_stack` keep +their full unique name indexes. ### 4.2 Visibility filter No views: views count as schema on both backends. Every statement that lists or resolves a Branch or project carries a `NOT EXISTS` filter against the marker tables, and a guard test fails on any statement that reads `branch` or -`layer_stack` without it unless the statement is on an allow-list (the name -check and the fork/merge identity checks read retired rows on purpose). A -statement that bypasses the filter would resurrect a removed Branch. The +`layer_stack` without it unless the statement is on an allow-list: the +name-collision check (it must read the retired holder to return `NameTaken` +with its state), the fork/merge identity checks, compaction's mark/rebuild +statements, the `project.delete` retry, and two named by-id statements: +`branch_branch_by_id_any` (render/identity for `layer.list`/`layer.get`/ +`saved`/operator inspect) and `layerstack_stack_by_id_retired` (the retry's +live/absent read by the caller's recorded stack id). The ordinary +`branch_branch_by_id`/`layerstack_stack_by_id` stay filtered. Every +other statement that reads `branch` or `layer_stack` carries the filter; a new +allow-list entry needs a comment and a test. A statement that bypasses the filter would resurrect a removed Branch. The minimum is the Branch and project listings and name lookups (`branch_snapshot`, `branch_snapshot_named`, `layerstack_stack_by_name`), fork, merge, publish and mount resolution, and the project listing @@ -139,36 +241,134 @@ fork, merge, publish and mount resolution, and the project listing `identity::list_projects` and, since #678, the record rebuild in `backup.restore` (`control/crates/layerfs-store/src/backup/restore.rs`), so an unfiltered listing would write records for deleted projects back into a -home. +home. `project.list` and `project.get` intersect the home entries with this +filtered Store listing before paging: a stale entry (a different home, or a +failed home-entry removal) is absent from `project.list` and answers +`NoSuchProject` from `project.get`. The join is on the stack id recorded in +the home record: `manifest.stack` for a local SQLite record, +`record.stack` for PG/worker records; a missing stack is filled from the +filtered listing by name. The all-Stores listing omits an unreadable Store; +`project.list(store)` propagates `NoSuchStore`/`Busy`/`Unknown`. a Store that cannot be listed is omitted by `project.list` +and returns its own `NoSuchStore`/`Busy` from `project.get`, never the +unfiltered record. A hidden project is `NoSuchProject` for +the four calls, except `project.delete`'s same-home retry, which resolves the +caller's record by stack id and allow-lists its retired stack. ### 4.3 `remove_file` -- The marker names a path in one project and a **frontier**: the stack's - history serial when the marker was written. It applies to the Commits and - Layers at or below the frontier, the same way before and after any - compaction (#42 item 35): - - a read that resolves through the path returns `Removed`; - - a listing omits the entry and says that entries were removed (shape owned - by #47/#51); - - a mount of a Branch whose head is at or below the frontier applies a - whiteout for the path, so the next Commit records a real deletion. -- Commits after the frontier never inherit the path, and creating a file at - the same path later is an ordinary write that stays visible. -- Reads by object ID are not path-aware: they return the bytes until - compaction collects them, then `NotFound`; never different bytes. -- History rows and IDs are unchanged. Because the answer is `Removed` or - `NotFound`, not different bytes, an old ID never silently resolves to - changed contents. -- The call reports no freed bytes; compaction reports what it reclaimed (§6.2). -- The path is scoped to one project. The same bytes reached from another - project, another path, or as a delta base of a live object stay. -- Physical removal is best effort, and the documentation says so: a pack - shared with live objects is not repacked, and below the threshold - compaction writes nothing. A guaranteed purge (a targeted repack) is a - follow-up, not part of 2.0.0. -- Traces record a hash of the path, not the path. -- If #47 cannot deliver the path-aware walk in 2.0.0, the call ships as - visibility only and its documentation says plainly that it frees nothing. +- The marker names a path in one project and a **frontier**: the current + Store high-water, `next_history_serial - 1`, read under the Store write + lane in the same transaction as the marker; a node created concurrently is + ordered by its serial against that value, never by a later read. The first project's genesis Layer takes serial 1, so its first frontier + is 1; a Store with no Stack has none and `remove_file` is `NoSuchProject`. Every + Commit gets the next Store serial; a Layer that republishes a Commit + carries that Commit's serial, so `branch.publish` does not move a tree's + position, and every other Layer gets the next serial. A Commit or Layer is + *at or below* the frontier when its serial is at most the frontier; the + serial also covers Commits on unmerged Branches (#42 item 35). It applies + to the Commits and Layers at or below the frontier, before and after any + compaction: + - a read whose resolved tree actually contains the path (or a hidden + directory ancestor for an existing descendant) returns `Removed`; an + at-or-below node without the entry is `NotFound`/absent, and a + never-existing path hides nothing; + - a listing omits the entry and reports one top-level `removed_count` over + the queried subtree (present at 0, filtered before paging, no page slot or + admission cost); + - a mount of a Branch whose effective tree (its head Commit, else its base + Layer) is at or below the frontier applies a whiteout for the path, so + the next Commit records a real deletion. +- **Every operation that builds a new tree from a node at or below the + frontier materializes the whiteout**: a workspace Commit, a + non-fast-forward `branch.merge`, every one-shot tree creator (`file.write`, + `edit`, `apply`, `rm`, `mkdir`, `mv`, `cp`), and any server-side + construction. This tree/whiteout walk is mandatory in 2.0 and + distinct from the byte-freeing GC walk (§6.2). A fast-forward merge creates + no tree and is covered by the node it points at. `branch.publish` of a + Commit at or below the frontier republishes that tree as a Layer at the + same position: the marker hides the path and a mount of the Branch that + carries it whiteouts, so publish needs no whiteout of its own. A merge that + cannot materialize the whiteout is refused `Removed` with + `detail {paths: [...]}`. +- A path explicitly re-created after the frontier is not a whiteout: the + marker hides only content inherited from nodes at or below the frontier. + In a merge of a head at or below the frontier with a head whose path was + re-created after it, the re-creation wins; the whiteout removes the + inherited side and suppresses any conflict from that side. An ordinary edit + conflict is reported only when both sides are post-frontier (the marker does + not apply) or on other paths. +- One marker per project and path: a repeat `remove_file` writes the current + Store high-water (`next_history_serial - 1`) as the new frontier, so + everything at or below is hidden again. It is `UpToDate` iff no project + node has a serial greater than **this path's** marker frontier (a missing + marker is -1); a commit in another project changes + `next_history_serial - 1` but neither rewrites the marker nor advances its + frontier. A path that never existed is accepted and + hides nothing. The newest frontier subsumes the older ones. +- Every marker write that advances the frontier bumps the project's `epoch` + in the same transaction; an `UpToDate` reply does not. +- Commits after the frontier never inherit the path; creating a file at the + same path later is an ordinary visible write. +- A path is **hidden at node N** only when the entry exists at N and the + marker's frontier (the maximum over the path and its `/`-boundary + ancestors) is at least N's `history_serial`; a marked-but-absent entry is + absent/`NotFound`, and a marker on a never-existing path hides nothing. +- Reads by object ID are not path-aware: bytes while retained, then + `NotFound` once compaction collects it, never different bytes. +- The result is `{kind: "Removed", project, path, frontier_serial, + removed_ms}` or `{kind: "UpToDate", project, path, frontier_serial, + removed_ms}` (the stored values). Every refusal carries one detail shape: + `{paths: [{path, frontier_serial}]}` (one element for read/stat; the + governing maximum frontier and the request path). `file.read`/`file.stat` + refuse `Removed` with it. `file.list`/`file.glob` omit hidden entries and + report one top-level `removed_count` over the queried subtree (a hidden + target path is `Removed`; a marked-but-absent path is not counted). + `file.diff` has three states per side (visible/hidden/absent); hidden + sides never refuse: left visible + right hidden → `changeKind` `Removed`; + left hidden + right visible → `Added`; both hidden → `Removed` with both + side objects; a hidden side object is + `{removed: true, path, frontier_serial}`. A hidden side beats + `allow_missing`; only a genuinely absent side is `NoSuchPath` (with + `allow_missing: false`). `branch.diff`/`commit.diff` apply the same + per-side rule to their historical trees, so a removed path is absent, not a + spurious change. `file.revision` refuses `Removed` + when the anchor is hidden; otherwise it walks and **stops** at the first + hidden at-or-below version (never skips past it), returning only visible + versions. `file.blame` stops at the hidden version and attributes the + remaining lines at the tip. `file.grep`/`file.blame` omit hidden + hits. `project.export` omits it; a cp-from source is `Removed`; `file.edit` + is `Removed`; `file.mv` from a hidden path is `NotFound` and to a hidden + path is a visible re-creation; `mkdir`/`cp`/`apply` follow the same + construction rule; `file.rm` of a hidden path is `NotFound`; a + post-frontier write is the visible re-creation. +- The call accepts an absolute POSIX path and converts it to the Store's + normalized relative components (strip one leading `/`; `/a` and `a` are the + same). It refuses `InvalidInput` for empty, `/`, `//a`, `a//b`, `a/`, `./a`, + `a/./b`, `.`/`..` components, `\`, NUL, >4096 bytes or >256 components. A + directory argument hides its subtree only at a `/` boundary + (`p = m OR p begins m ++ b'/'`, computed as bytes in Rust, never `||` or + `LIKE` on a BLOB). A child of a removed **file** is `NotFound`; a read of + an existing descendant under a hidden directory is `Removed` with the + governing maximum frontier over the path and its ancestors (using + `history_serial`); an absent descendant is `NotFound`. An intermediate symlink component is `InvalidInput` (not followed); + a symlink leaf is hidden as an entry. Other hardlink names stay visible and + nothing is freed. +- History rows and IDs are unchanged. The trace key is BLAKE3(per-Store salt, + layer_stack_id, normalized relative path); the salt is a 32-byte + `store_policy.trace_salt` (random per Store, created by the additive step, + backfilled for converted Stores, copied by a rebuild). Every path argument + (`path`, `left_path`, `right_path`, `from_path`, `to_path`) is redacted + per call: the log records the normalized length as ``, never the + text; the daemon (native/worker) or the PG server-side call writes the keyed + hash, and the path argument names (`path`, `left_path`, `right_path`, + `from_path`, `to_path`, `path_prefix`, `pattern`) are in the per-call + redaction set. + Refusal details/messages are not traced; the `Removed` detail echoes the + request path, and raw paths never enter traces. +- The call frees no bytes in 2.0 (visibility only, §6.2); the path is scoped + to one project, so bytes reached elsewhere or as a live delta base stay. +- Physical removal, a path-aware GC walk and a guaranteed purge are follow-ups + that amend this contract before they ship. ### 4.4 Names @@ -177,12 +377,47 @@ home. name is never reused while its old rows exist. So: - A create or fork whose name a live row holds returns `NameTaken` (existing: - `engine/crates/layerfs-store-service/src/history/branch.rs:232,397`). + `engine/crates/layerfs-store-service/src/history/branch.rs:232,397`) with + `detail {holder: {kind: "branch"|"project", name, state: "live"}, + source: "store"}`. The detail rides + `HistoryError::NameTaken(Box)` (Holder/Source defined in + `layerfs-history`, boxed like `HeadMoved`); `lookup.rs` has an explicit arm + that projects `{holder, source}`. `source` is `store` only; a home-record + collision is a service `refuse("NameTaken", ..., {holder, source:"home"})`. + The `branch.rs:230-234,395-399` Null pre-checks are deleted, so every site + reaches the typed error. The branch query joins `branch_retired`; the + project query joins `layer_stack_retired`. - A name held by a removed Branch or deleted project also returns - `NameTaken`, with the holder's state in the detail. + `NameTaken`, with `detail {holder: {kind: "branch"|"project", name, + state: "live"|"retired"|"deleted"}, source: "store"}`; `source: "home"` is + used only for a project's home record. The `BranchId::named` identity check + and the name-collision statements both return this detail; the + `_ => "Failed"` arm is not allowed to swallow them. +- `branch_retired(branch_id BLOB NOT NULL PRIMARY KEY REFERENCES + branch(branch_id) ON DELETE CASCADE, removed_ms INTEGER NOT NULL) STRICT` + (same for `layer_stack_retired`), so a second remove cannot add a second + marker and a + collected base row takes its marker with it, on both backends. The write is + `INSERT ... ON CONFLICT(branch_id) DO NOTHING` with an affected-row check; + only an insert bumps `epoch`. A concurrent remove is `Busy` (phase + `"retire"`, no op), a repeat is `NoSuchBranch`; `project.delete`'s repeat is + the idempotent success in §4. +- `branch.fork` disambiguates by syntax: `main`, a valid HistoryName (a + Branch or `NoSuchBranch`, checked before `args::node`), or 66-hex (a Commit + or a Layer; a missing Layer is `NotFound`, a collected Commit + `NoSuchCommit`). A valid name is never retried as a node id; a Commit fork + keeps the retired row/name through `forked_from_id` and `commit.branch_id`, + and a Layer fork through `forked_from_id` and `layer.source_branch_id`. + A mount of a retired Branch is `NoSuchBranch`. For `merge`, a retired + **target** name is `NameTaken` while a retired source is `NoSuchBranch`; for + `publish`, a retired Branch is `NoSuchBranch` and a collected node is + `NoSuchCommit`. - Compaction removes a retired `branch` or `layer_stack` row once no retained row references it (`layer.source_branch_id`, `commit.branch_id` and the other history edges, `engine/crates/layerfs-persistence/sql/sqlite/history.sql:62-65`). + A Branch published by a retained Layer is pinned by that foreign key, and a + Branch named by a retained `commit.branch_id` is kept by the history + closure; only a Branch with neither reference is freed. The name is then free, and no retained row carries the old ID. The comment on `BranchId::named` ("never reused") is updated by #44 to say "never reused while its rows exist". @@ -193,54 +428,105 @@ name is never reused while its old rows exist. So: | Operation | Refused while | Code | |---|---|---| -| `branch.remove` | a mount or Sandbox of that Branch exists | `MountsActive` | -| `project.delete`, `project.remove_file` | any mount of the project (any Branch), or another user of the project | `MountsActive`, `Busy` | -| `store.compact` | any mount, client or operation on the Store, or unfinished Init/merge rows (#155 D-6) | `MountsActive`, `Busy` | - -Mounts take a Branch; there is no mount of a historical Commit. A Sandbox on a -machine that cannot be reached is `Busy`: its inactivity cannot be proven. - -The check and the exclusion are one atomic step: a new conflicting user is -refused from the check until the operation completes. +| `branch.remove` | a mount or Sandbox of that Branch, an unfinished Init/merge row names it, or another GC op at Branch scope | `MountsActive`, `Busy` | +| `project.delete`, `project.remove_file` | any mount of the project, another user of the project, an unfinished Init/merge row names it, or another GC op at project scope | `MountsActive`, `Busy` | +| `store.compact` | any mount, long-lived client or operation on the Store, or unfinished Init/merge rows | `MountsActive`, `Busy` | + +Mounts take a Branch; there is no mount of a historical Commit. All four +calls resolve their resource through the filtered statements (§4.2); a hidden +Branch is `NoSuchBranch`, a hidden project `NoSuchProject`, and a collected +Commit `NoSuchCommit` (the latter only on merge/publish), never a second +marker. The one exception is `project.delete`'s same-home retry, which +resolves the caller's recorded stack id under §4.2. + +`branch.merge` and `branch.publish` take no mount and are not in the table. A +non-fast-forward merge materializes the `remove_file` whiteout in its result +(§4.3) or is refused `Removed`; a publish of a Commit at or below the frontier +yields a Layer at the same position, which the marker hides. + +Every operation that creates a Commit or Layer, or writes under a project, +reads `project_epoch.epoch` at start and re-checks it before commit: +- SQLite: inside the one `BEGIN IMMEDIATE` transaction that also writes the + Commit/Layer, so a removal cannot interleave. +- PostgreSQL: the writer's last statement before commit is + `SELECT epoch FROM project_epoch WHERE layer_stack_id = $1 FOR SHARE` + (it blocks the marker's non-key `UPDATE`); a plain unlocked read is not + enough under READ COMMITTED. The marker transaction itself runs + `UPDATE project_epoch SET epoch = epoch + 1`, then reads the frontier + `next_history_serial - 1`, then inserts the marker. A missing epoch row + (post-cleanup project delete) is `Conflict`, never "unchanged". ### 5.1 SQLite -- **Store scope:** the existing Store lock, lease and op record - (`control/crates/layerfs-store/src/lease.rs`, `record.rs`, - `control/crates/layerfs-home/src/op.rs`). A live lease, reported by the - lease code as `InUse`, is returned as `Busy` at the API. -- **Project scope:** a project-level `op` on the project record, with the - same begin/settle pattern as the Store op - (`control/crates/layerfs-store/src/sqlite/family.rs:161-200`); Sandbox - create and mount check it (#42 item 33). -- **Branch scope:** a mount writes its Workspace intent record under the home - lock *before* bind, and removes it if bind fails. A removal checks the - records and writes its marker under the same lock, so a mount and a removal - can never both succeed; a bind that starts after the marker is refused by - the filtered statements (§4.2). +Admission is one Store-DB table and two `BEGIN IMMEDIATE` transactions +(SQLite 3.53.2 `lang_transaction`; a locked writer is `Busy`). + +`gc_intent(op_id PRIMARY KEY, kind, scope_kind, scope_id, workspace, sandbox, +machine, project, branch, owner_runtime, owner_boot, state, uncertain, phase, +generation, artifact, high_water, evidence, updated_ms)`, `STRICT`, with +`UNIQUE(scope_kind, scope_id, owner_boot)`; `kind` is `mount`, `init`, +`merge`, `removal` or `compaction`, `state` is `live`, `uncertain` or `done`, +`scope_kind` is `store`/`project`/`branch`, and the ids are typed per kind +(17-byte project/Branch, 33-byte commit/layer, 32-byte catalog id). The +migration step that adds it is the same additive step as §4.1. + +- A **mount** transaction checks that no operation op is running at every + scope it names (Store, project, Branch), inserts its intent and commits. A + **removal/compaction** transaction scans the intents in its scope, writes + its marker or op row, and commits. A live mount → `MountsActive`; a live + operation → `Busy`; a provably dead, uncertain op → `Unresolved`; a + provably dead, certain op is adopted or ignored per its `kind`. +- Compaction writes its `gc_intent` operation row **before** its first + checkpoint; the row lives in `.compact` during build/verify, moves + to the live Store after install, and is cleared at settle. `none` writes + and clears the same row (its only write); "nothing is written" in §3 means + no pack/object/history/generation change. +- Intent lifetime: the row is deleted in the `workspace.unmount` + transaction, on `sandbox.close`, daemon exit, machine stop, and in the + mount's error path if bind fails after the intent commit. A failed delete + is reported; a provably dead owner's row is ignored, never trusted. +- `machine_stopped` is provider evidence written into the intent/op; a live + but unreachable Sandbox is `Busy`; a removed machine record alone is not + proof. +- **Project scope:** `project.delete`/`remove_file` take a project op and + scan the intents naming the project. A mount is excluded by its own + intent, so several mounts of one project proceed. `project.delete` holds + its op through the marker transaction only; the retired-row entry-removal + retry needs no op, the live-row retry takes one and re-scans. +- **Branch scope:** the intent carries the Branch; a removal checks the + intents and writes its marker in the same transaction, so a mount and a + removal can never both succeed; a bind after the marker is refused by the + filtered statements (§4.2). +- `Unresolved` carries `{entity: {kind, name}, phase, op}`; a non-owning call + observes it, the owning call adopts/clears it. A `kind`-by-owner-state + matrix governs {mount, removal, compaction} × {live, unreachable, + dead-certain, dead-uncertain}. ### 5.2 PostgreSQL Leases are files in one home and cannot see another machine. So (#42 items 33, 34): -- **Presence.** Advisory keys at three scopes, `key(scope) = - blake3("layerfs/advisory/v1\0" ‖ catalog_id ‖ scope)[..8]` with scope the - Store, a project or a Branch (#42 item 12a, extended). Long-lived sessions - hold them shared on their own connection: the daemon for each mount (Store, - project and Branch keys), and compaction's own connection. A removal tries - the exclusive project or Branch key; compaction tries the exclusive Store - key. Failure is `Busy` or `MountsActive`, naming the holders (`pid`, - `client_addr`, `backend_start`). This makes an idle remote daemon with a - mount visible. -- **One-shot SDK calls** hold no presence lock. They rely on the fence below, - and on the swap's `ACCESS EXCLUSIVE NOWAIT` (§8): a transaction still open - at swap time makes the swap abandon. +- **Presence.** Advisory keys at three scopes, `key(kind, id) = + blake3("layerfs/advisory/v1\0" ‖ catalog_id ‖ kind ‖ id)[..8]` with kind 1 + the Store (32-byte `catalog_id`), 2 a project (17) and 3 a Branch (17); the + 8-byte truncation can cause a false `Busy` or a misclassified holder, never + a missed exclusion. A mount holds shared kind 1/2/3 on one dedicated + session for the daemon's life, refcounted per `(session, key)` and acquired + all-or-none with `pg_try_advisory_lock[_shared]`; a failed acquisition + unlocks what it took. Compaction holds only the exclusive kind 1 key, taken + before Phase M and held through Phase C/settle on that same pinned session. +- **Removals** take transaction-scoped locks in the marker transaction: + `branch.remove` = shared kind 1 + shared kind 2 + exclusive kind 3; + `project.delete`/`remove_file` = shared kind 1 + exclusive kind 2; order + Store → project → Branch. A failed shared attempt is `MountsActive`, a + failed exclusive attempt `Busy`; `pg_locks.mode` names the holder. +- **One-shot SDK calls** hold no presence lock. They rely on the + `project_epoch` `FOR SHARE` check (§5), the fence below, and the swap's + `ACCESS EXCLUSIVE NOWAIT` (§8). The four GC calls are not one-shot. - **Fence.** The generation check rides in the statements a client already sends: in the `BEGIN` batch of a transaction, and as a joined column of - `history_meta.catalog_incarnation` in the reads that run without a - transaction (`Mode::Immutable`/`Statement` skip `BEGIN`: - `engine/crates/layerfs-persistence/src/backend/postgresql/session/transaction.rs:263-268`). + `history_meta.catalog_incarnation` in reads without a transaction. No extra round trip. A mismatch is `StaleStore`. - **Connections.** Mounts and compaction need a direct or session-pooled connection; transaction pooling cannot hold a session lock and is @@ -248,7 +534,9 @@ Leases are files in one home and cannot see another machine. So (#42 items poolers. - **Dead clients.** A client whose TCP connection died holds its lock until the server notices. The guide documents `tcp_keepalives_idle`, - `client_connection_check_interval` and `pg_terminate_backend`. + `client_connection_check_interval` and `pg_terminate_backend`; a GC call's + lost reply is resolved on a second connection before the exclusive key is + taken. ## 6. Mark @@ -259,21 +547,32 @@ operations; a missing locator or corrupt dependency aborts with no writes. A recursive query from the retained roots over the history edges: -- Branch (not retired) → `head_commit_id`, `base_layer_id`, `forked_from_id` +- Branch → `head_commit_id`, `base_layer_id`, `forked_from_id` (followed for + every retained Branch; the **roots** are every non-retired Stack and every + Branch with no own or stack marker) - Stack (not retired) → `head_layer_id`, published Layers - Commit → `parent_commit_id`, `second_parent_id`, `base_layer_id`, - `merged_from_id`, `displaced_id` -- Layer → `parent_layer_id`, `source_commit_id` + `merged_from_id`, `displaced_id`, `branch_id` +- Layer → `parent_layer_id`, `source_commit_id`, `source_branch_id` `merged_from_id` and `displaced_id` are followed because doctor requires their targets (`diagnostic/history.rs:40,146`; #42 item 28). A retired Branch is collected as a unit: if a retained row references it (`commit.branch_id`, -`layer.source_branch_id`), its row stays and so does its head chain, because -`branch.head_commit_id` is a foreign key; the closure is the fixpoint of this -rule (#42 item 29). Every history edge is stack-local, so a live stack keeps -all its Layers and only Commits and retired Branch rows can go; a deleted -project goes whole. A fork keeps its source's history up to the fork point. -History rows outside the closure are collected by the rebuild (§7, §8); this +`layer.source_branch_id`), its row stays and so do `head_commit_id`, +`base_layer_id` and `forked_from_id`, and the head commit enters the Commit +edge set above; the closure is the fixpoint of these rules (#42 item 29). A +Layer's `source_branch_id` is a foreign key, so a published Branch and its +head chain stay retained while the Layer lives; that Branch keeps its name +(`NameTaken`) and compaction does not free it. "Free after compaction" (§4.4) +applies only to a Branch that nothing retains: no retained +`layer.source_branch_id` and no retained `commit.branch_id` (a fork's +`forked_from_id` is such a Commit). Every history edge is +stack-local, so a live stack keeps all its Layers and only Commits and +retired Branch rows can go; a deleted project goes whole. A fork keeps its source's history up to the fork point. +`store.history.branches` validates every retained `commit.branch_id`: it must +name a Branch in the same stack (SQLite cannot add the FK, so this is a doctor +edge). History rows outside the closure are collected by the rebuild (§7, §8); +this is required because the full doctor qualifies every Commit and Layer root it finds (`engine/crates/layerfs-persistence/src/diagnostic/history.rs:155-198`). @@ -285,31 +584,53 @@ finds (`engine/crates/layerfs-persistence/src/diagnostic/history.rs:155-198`). with `ChainBases::base_of` (`core/crates/layerfs-storage/src/encoding/delta/read.rs`). Do not reuse `qualify`: it skips file mappings, symlink targets and attribute trees and ignores delta bases. -- One visit level (#42 item 27): an object reached by a logical edge or a - base edge is kept with all of its references, because the full doctor - checks the references of every stored object - (`engine/crates/layerfs-persistence/src/diagnostic/scan.rs:129-224`). - Objects kept only because they are bases are reported in `keptWaste`. -- `remove_file`: when descending a root at or below a marker's frontier, the - walk does not enter the removed path. An object stays live if any other - path, project or delta chain reaches it. -- On PostgreSQL the base edges live inside pack groups, so a remote server - sends about the live payload to the client during the mark. No format - change is made for this until #52 measures it; the fallback is a nullable - `object_location.base_id` filled at publish (#42 item 36). +- Transitive closure (#42 item 27): each object is admitted once; admitting + it enqueues its logical references (`references()`) and its delta base + (`base_of`), and the queue runs to empty. An object reached only as a base + still keeps all of its references, because the full doctor checks the + references of every stored object (`diagnostic/scan.rs:129-224`). Objects + kept only because they are bases are reported in `keptWaste`. +- `remove_file` is **visibility-only in 2.0.0**: the marker hides the path in + reads, listings, mounts, merges and publishes, and compaction reclaims + nothing that only the removed path reached. A retained root's inode page + must stay (qualification refuses a missing inode, + `filesystem/qualify/walk.rs:183`), and rewriting the tree would change the + root and the Commit/Layer id (S10). A path-aware walk that frees bytes, and + per-entry page pruning, are a follow-up; `keptWaste` has no page component + in 2.0. +- On PostgreSQL the base edges live inside pack groups, so the mark (§8 + Phase M) reads that payload. A remote caller transfers it; `markBytes` is + the distinct live-pack body bytes read once, and #52 measures them. The + table copy moves no pack bytes. 2.0 keeps the format unchanged; adopting a + nullable `object_location.base_id` is a separate #52 decision with a schema + version and a defined NULL/backfilled distinction, not a runtime switch + (#42 item 36). - Window by window, sort lookups by object ID before locating them. - Whole files and chunks have no logical references; only their delta base is needed. Reading it still loads and authenticates the whole group - (`delta/read.rs:561-571`), so the payload part of the walk runs inside the - copy (§7 step 4), not as a separate pass. + (`delta/read.rs:561-571`); on SQLite the payload part of the walk runs + inside the copy (§7 step 4), and the PostgreSQL mark reads it in Phase M. ### 6.3 Visited set -A scratch table `visited(object_id PRIMARY KEY, pack_id) WITHOUT ROWID` -in a temporary file owned by the compaction connection (`temp_store=FILE`): -bounded memory, no second persistent database. Session connections keep -`temp_store=2` (`engine/crates/layerfs-persistence/src/backend/sqlite/session/profile.rs:88`). -On PostgreSQL, a temporary table. +A scratch table `visited(object_id PRIMARY KEY, pack_id) WITHOUT ROWID` in a +named scratch database beside the Store (`.compact-visited`), attached +to the compaction connection: it is in the op/artifact set and in +`peakTempBytes`. Its own journal (`.compact-visited-journal`) and any +multi-database super-journal (`-mj*`) are in the artifact set too; or the +visited DB runs `journal_mode=OFF` and the copy is not atomic with it. Session +connections keep `temp_store=2` +(`engine/crates/layerfs-persistence/src/backend/sqlite/session/profile.rs:88`); +both paths spill the closure to `.compact-visited` and choose the +path from the refined live set; the `none` path deletes the scratch before +replying, so it leaves no residue and never answers `Capacity` for a cap. On PostgreSQL it is a temporary table. + +`kept_node(kind, id)` is the history side of the same walk: `kind` is +`commit`, `layer`, `branch` or `stack`, and it holds the §6.1 closure. A +rebuild copies `layer_lineage.layer_id ∈ K_layer`, `commit_lineage.commit_id +∈ K_commit`, `absorbed.run ∈ R`, `reach.key ∈ K_stack ∪ R` (R := runs of kept +`commit_lineage` rows), and the history/catalog rows by `kept_node`; it +deletes the complement, and a deleted project goes whole. ### 6.4 Decision @@ -317,69 +638,153 @@ On PostgreSQL, a temporary table. - `none` or `rebuild` from §3, using the bytes of packs that hold no live object after the metadata walk (an upper bound that §7 step 4 refines). - Dead rows: `object_location`, `content_signature`, `file_digest` (keep a - row only if its root is retained, #155 R-4), history rows outside §6.1, and - retired `branch`/`layer_stack` rows (§4.4). + row only if its root is in `visited`; a digest or signature root is not a + mark root, so an uncommitted saved file's row is dead), history rows + outside §6.1, retired `branch`/`layer_stack` rows (§4.4), a `branch_retired` row whose + branch row is collected, a `layer_stack_retired` row whose stack is + collected, and `path_removed`/`project_epoch` rows of a deleted project (a + project goes whole). A live project's markers stay: `path_removed` is one + per path, and a serial allocated later never falls below a frontier. - Kept in 2.0: every `inode_value_block` row; the ordinals of pooled leaves are private to core, and the blocks are small. The report gives their bytes (#42 item 37). - Cleared: `binding_hint_block`. The hints are advisory and not per project, and they hold the file names of deleted projects and removed paths (#42 - item 38). + item 38). Clearing all of them costs only advisory lookup speed, never + correctness; a later Save repopulates hints for live paths. A per-project + clear is a follow-up, not 2.0.0. + +`live_heads_digest` (both backends) is BLAKE3 over +`"layerfs/live-heads/v1\0"`, then the non-retired stacks in id order as +`(stack_id, head_layer_id)`, then the Branches with no own or stack marker in +id order as +`(branch_id, head_commit_id or 33 zero bytes, base_layer_id)`, each +fixed-width. Commit ids are 33 bytes (tag `0x12` + 32), Layer ids 33 (tag +`0x32`), Branch and stack ids 17; a golden vector pins the bytes. A retired +row is absent, not zeroed; timestamps are not hashed, so SQLite and +PostgreSQL agree. ## 7. SQLite rebuild Runs as one operation on the Store's machine (S6). -1. Checkpoint the WAL and confirm no `-wal` remains (an immutable attach - ignores the WAL). -2. Create `.compact` beside the Store with a raw connection: +1. Take the persistence writer guard and hold it through step 7's rename. + Checkpoint the source with `PRAGMA wal_checkpoint(TRUNCATE)` and require + `busy == 0` (never unlink a WAL file); checked-close the last source + session and check for `-wal`/`-shm` while the guard is held; pass the + guard into the seal (no second acquire); use + `seal_holding() -> (size, mtime, db-header change counter, Writer)` + captured after the sidecar check and before release. The current + `Session::seal` releases the lane, so a returned Writer is required; there + is no re-acquire alternative. A busy checkpoint is + `Busy`. Record the sealed `(size, mtime, db-header change counter)`. +1.5 After the seal, `ATTACH` the old file `immutable=1` (it stays + byte-identical), create `.compact-visited`, run the complete + §6.1+§6.2 closure into it, and choose `none`/`rebuild` from the refined + live set. On `none`, delete the scratch and stop: no `.compact`, no + `.compact-visited`, no residue. On `rebuild`, keep the scratch and reuse + `visited` in step 4; step 2 creates the new file only on this path. +2. If the path is `rebuild`, create `.compact` beside the Store with a + raw connection: `page_size=4096`, the source's `auto_vacuum`, `schema::scripts`, - `application_id`, `user_version`. Give this connection a large - `cache_size`, and the attached old file a large `mmap_size`; the session - defaults are a 2 MiB cache and no mmap (`session/profile.rs:86-87`). -3. `ATTACH` the old file `immutable=1`; it stays byte-identical. -4. One rollback-journal transaction, foreign keys on, in this order: + `application_id`, `user_version`. The new file uses the source's + `(layout, acquisition)` derived from its `user_version`: `Monolithic` has + `pack.body` and no `pack_unit`; `GroupRows` has `pack_unit`; + `GroupRowsIndexed` also has its index. `schema_source`/`schema_definition` + must be those of that pair, or step 6's `validate` refuses. Give this connection a + large `cache_size`, and the attached old file a large `mmap_size`; the + session defaults are a 2 MiB cache and no mmap (`session/profile.rs:86-87`). +3. The old file is already attached `immutable=1` in step 1.5; it stays + byte-identical. +4. `PRAGMA foreign_keys=ON` outside the transaction (inside it the pragma + is a no-op), then one rollback-journal transaction, in this order: - Metadata walk (§6.1–§6.2 logical part) into `visited`. - - Payload sweep: live packs in descending `pack_id`. Copy each live pack - once, add the delta bases of its live records to `visited`. A base found - in a pack already passed goes on a late list and is copied at the end. - A base is not guaranteed to sit in a lower `pack_id` (first-wins - locators can move forward, - `core/crates/layerfs-storage/src/store/source.rs:9-13`), so the late - list is an indexed table, not an exception path. + - Payload sweep: a worklist of packs to copy, run to empty. Start with the + packs the mark marked live; copy each pack once. For every object + inserted into `visited` (a base, its references and their bases), run the + same `admit` fixpoint, insert each newly discovered object if absent and + enqueue **every** newly visited object's locating pack (passed or not); + do not follow records whose `object_id` is not in `visited`. A pack + reached only by a base is copied whole; `pack_id` is preserved and new `unit_id`s are + assigned; the dead set and `deadBytesRefined` are recomputed from the + final copied set. A base is not guaranteed to sit in a lower `pack_id` + (first-wins locators can move forward, + `core/crates/layerfs-storage/src/store/source.rs:9-13`), so the worklist + is an indexed table, not an exception path. - `pack_unit` for the copied packs, as a complete unit set (`engine/crates/layerfs-persistence/src/backend/sqlite/objects/units_read.rs:55-75`). + A `Monolithic` Store copies `pack` with its body and skips `pack_unit` + (the Monolithic decoder is used); `GroupRows` copies both; + `GroupRowsIndexed` copies both and the schema creates its unit index + (the index is not "copied"). `sqlite_sequence` is copied only when the + source has it; `store_totals` is UPDATEd as a whole row (keeps `init_*`) + and `store_policy` is `INSERT ... SELECT * FROM old.store_policy` + (allocators and salt verbatim); transient `init_*`/acquisition working + rows + are **not** copied, and a live Init op is admitted as `Busy`. - `object_location` by an ordered join with `visited`. - - `content_signature` entries that name a copied object; `file_digest`, - `inode_value_block`, history and catalog tables per §6.4, with a deleted - project's rows left out by the `removal_*` statements' selection; - `sqlite_sequence`, `store_policy` and `scope_allocator` copied; - `binding_hint_block` left empty; `history_meta` with - `catalog_incarnation = G + 1`. - - `store_totals`: `logical` is rebuilt by the history triggers and must - equal the old value minus the collected history rows' `logical`; - `canonical`, `stored` and `metadata` are summed. -5. Switch the new file to WAL and run the sidecar checks - (`engine/crates/layerfs-persistence/src/backend/sqlite/snapshot.rs:101-140`). -6. Verify: a real Session open (`validate`), `live_heads_digest` before and - after (a new function beside `heads_digest`, - `engine/crates/layerfs-persistence/src/backend/sqlite/heads.rs:83`, over - non-retired Branches and stacks only; #42 item 30), row counts against the mark, - `foreign_key_check`, the catalog allocator and totals checks. No sha256 of - either file. `quick_check` and the full doctor stay separate commands. -7. Install: re-read `G`; check `(catalog_id, incarnation, dev, ino)` and that - no sidecars exist; `sync_all` the new file; `rename` it over the Store; - fsync the directory. The old inode is freed at once. -8. A definite failure of compaction's own write (disk full, write refused) - aborts the transaction and deletes `.compact` in the same call, - returning `Capacity` or the failure (#155 D-22). + - `content_signature` entries that name a copied object; `file_digest` + rows whose root is in `visited`; all `inode_value_block` rows; history, + lineage (`layer_lineage`, `commit_lineage`) and catalog tables copied by + `kept_node`; `absorbed` by `run`; `reach` by its kind-mapped key; with a + deleted project's rows left out by the `kept_node`/retired predicate + (`removal_*` runs only on the writable new Store in Phase C); + `sqlite_sequence`, `store_policy`, `scope_allocator`, the marker tables + (`branch_retired`, `layer_stack_retired`, `path_removed`) and + `project_epoch` copied for rows the closure keeps; + `commit.history_serial`/`layer.history_serial` copied with their rows; + `history_meta.next_history_serial` copied unchanged; `binding_hint_block` + left empty; `history_meta` with `catalog_incarnation = G + 1`. + - `store_totals`: after **all** history copies, one UPDATE sets `logical` + to the triggers' value (or `old - dead_logical`), `canonical`/`stored`/ + `metadata` to the new sums, and `init_*` to the old row; assert + `old.logical == SUM(all history.logical)` before the copy. A copy before + the history insert would make the AFTER INSERT triggers add the + collected rows again (`old + collected`). +5. Switch the new file to WAL with a raw connection (`temp_store=MEMORY`, + `busy_timeout=0`, the source's `synchronous`); on macOS call + `disable_persistent_wal` before the checkpoint + (`snapshot.rs:116`, `seal.rs:31-32`); checkpoint with + `wal_checkpoint(TRUNCATE) == 0`; close it. No Session is opened. +6. Verify with an `immutable=1` open (`READ_ONLY|URI`, + `file:...?immutable=1`, percent-encoded path bytes so `?`/`#`/`%` do not + misdirect the URI, no profile apply/check, no lane) and run `validate` + there before the rename: + `schema::check`, + `foreign_key_check`, row counts against the mark, both `live_heads_digest` + values and the catalog allocator/totals checks. No sha256 of either file. + Close the reader; `quick_check` and the full doctor stay separate + commands. +7. Install: stat both paths for `-wal`/`-shm`/`-journal` and the + super-journal (`-mj*`); any sidecar is `Conflict` with no writes. Re-read + `G` through a fresh immutable handle and re-stat last; check + `(catalog_id, incarnation, dev, ino)` and the sealed `(size, mtime, + db-header change counter)`; a change is `Conflict`. `chmod 0600`; + `sync_all`; `rename`; fsync the directory; release the guard. A + transaction that began before install is caught at its next transaction + by the `(dev, ino)` fence, not at commit (documented limit). +8. A definite failure before the rename (disk full, write refused, sidecar, + checkpoint, validate, digest/count/FK, chmod/sync) aborts or rolls back and + deletes the artifact set `.compact{,-journal,-wal,-shm,-visited, + -visited-journal,-mj*}`; the old Store is unchanged apart from step 1's checkpoint. A + failure of the rename/fsync itself is installed-but-uncertain: keep the new + Store, settle the op `unknown` and report it; never delete the artifact as + if nothing happened. `SQLITE_FULL`/`BackendError::Capacity` is + `HistoryError::Capacity`, or the real `Refused { status: "Capacity" }` + path; `ENOSPC`/`EDQUOT`/`EFBIG` (`StorageFull`/`QuotaExceeded`/ + `FileTooLarge`) from chmod/sync/rename also map to `Capacity`, and + `SQLITE_FULL`/`SQLITE_TOOBIG`/`SQLITE_NOMEM` are Capacity on SQLite; no + `_` arm may turn it into another `Refused` status. `CapacityExceeded` is a + codec/canonical bound and is documented as `InvalidInput`/`ProviderFailure`, + not disk exhaustion. The existing transfer pipeline is not reused for this: its `Install` refuses an existing Store file and its `Retire` deletes one (`control/crates/layerfs-store/src/sqlite/transfer/steps.rs:262-288`). Cost: read about the live bytes plus the metadata, write about the live -bytes; peak disk = old Store + live bytes + the visited table. +bytes; peak disk = old Store + the new Store + the rollback journal/WAL + the +visited scratch file. ## 8. PostgreSQL rebuild @@ -387,66 +792,163 @@ Only `object_location.pack_id` references `pack`, so the payload lives in two tables. `content_signature` (at most 8192 rows, no foreign key: `sql/postgresql/schema.sql:45-50`) is cleaned in place. -**Phase A — build (exclusive advisory lock held, no table lock).** +**Phase M — mark (before Phase A).** Take the session-level exclusive Store +key before this phase and hold it through Phase C. The mark's **first +statement** reads `(catalog_incarnation, live_heads_digest, +history_meta.next_history_serial, next_pack_id, next_ordinal, +store_totals canonical/stored/metadata/logical)` and records that tuple; +any later publication makes Phase B `Conflict`, so a commit that lands during +the walk cannot be recorded while its objects are absent from `visited`. (Or +run the whole mark in one REPEATABLE READ transaction and record the tuple +inside it; state the vacuum cost.) One session, kept through Phase B, builds +`visited(object_id PRIMARY KEY, pack_id)` as a temporary table: start from the +retained roots, read pack groups in batches, follow +`FinalizedObject::admit(..).references()` and `ChainBases::base_of` as a +worklist to a fixpoint (each object once, bases of late objects included), and +record `live_heads_digest` and the live pack/object sets. After the walk, +write `gc_op` and `gc_dead` in batches (phase `mark`); a crash leaves both, +and the next `store.compact` adopts the op, truncates `gc_dead` and re-marks. +A `none` decision writes `gc_op` phase `mark`, truncates `gc_dead`, removes +the named artifact and clears `gc_op` before replying. +`gc_dead` carries `mark_stamp` for signature slots. The payload read is +reported as `markBytes`; the Phase A table copy itself moves no pack bytes. + +**Phase A — build (session-level exclusive Store advisory key, no table lock).** 1. Load the live object and pack IDs into temporary tables; `ANALYZE` them. -2. Create `pack` and `object_location` in a scratch schema with the Store's - column definitions and `STORAGE EXTERNAL` on `pack.body`, without the - trigger and the foreign key. -3. Copy live rows with `INSERT … SELECT` and a semi-join (`object_location` - has no unique key, `schema.sql:29-38`, so a plain join can duplicate rows), - in batches of about 1 GB by `pack_id`, each batch its own transaction. No - pack bytes cross the network. Batches may pause for standby lag. +2. Create `pack` and `object_location` in a scratch schema with + `CREATE TABLE ... (LIKE store.pack INCLUDING STORAGE, COMPRESSION, + CONSTRAINTS)` and the same for `object_location` (PK explicit; do not copy + `object_location_prefix`), plus the old `TABLESPACE` and `WITH + (reloptions)` read from `pg_class`. `pack.body` stays STORAGE EXTERNAL and + `object_location.object_id` stays STORAGE PLAIN; the trigger and foreign + key are added later. +3. Copy live rows with `INSERT … SELECT` and an `EXISTS` semi-join + (`object_location` has no unique key, `schema.sql:29-38`, so a plain join + can duplicate rows): `pack` rows with at least one visited object and + `object_location` rows whose `object_id` is visited, in batches of about + 1 GB by `pack_id`, each batch its own transaction. A retried batch first + deletes its own `pack_id` range from the scratch tables (idempotent), then + inserts; assert `count(*) = count(DISTINCT object_id) = |visited|`. No pack + bytes cross the network. Batches may pause for standby lag. 4. Build `object_location_prefix` and the foreign key, then `ANALYZE`. **Phase B — swap (one short transaction).** -5. `LOCK TABLE` the Store's tables `ACCESS EXCLUSIVE NOWAIT`. -6. Re-check `catalog_incarnation = G`, `live_heads_digest` (PostgreSQL gains - one; it has no heads digest today), and that `store_policy.next_pack_id` - and the ordinal allocator have not moved. Any change abandons the round - with no writes (`Conflict`). -7. Delete dead rows in place: `content_signature`, `file_digest`, history - rows (a deleted project's through the `removal_*` statements, which already - run on PostgreSQL) and retired name rows; clear `binding_hint_block`; - `inode_value_block` is kept (§6.4). `logical` in `store_totals` is reduced - by the collected history rows, as `removal_totals` does today - (`engine/crates/layerfs-persistence/src/history/removal.rs`); `canonical`, - `stored` and `metadata` are recomputed. -8. Copy the old tables' owner and ACL onto the new ones. -9. `DROP TABLE` the old `object_location` and `pack`; `ALTER TABLE … SET SCHEMA` +5. `LOCK TABLE` every base table of the Store schema (from `pg_class`, + including `store_lane`, `store_totals`, `history_meta`, + `content_signature`, the history tables and `gc_intent`/`gc_op`) in one + `ACCESS EXCLUSIVE NOWAIT`, with `store.*`/`scratch.*` qualified names. A + `55P03` abort is `Busy` with holder detail (not `Conflict`); `ROLLBACK` + first, then read holders and drop the scratch schema in a new + transaction. +6. Re-check `catalog_incarnation = G`, `live_heads_digest`, + `store_policy.next_pack_id`/the ordinal allocator, + `history_meta.next_history_serial` and `store_totals` + (`canonical`/`stored`/`metadata`/`logical`) against the values the mark + recorded. A + publication that reused a pre-reserved pack/ordinal may move none of the + first three but always bumps `store_totals`; any change abandons the round + with no writes (`Conflict`). No delta copy is attempted, so the new pack + stays in the old Store. +7. Copy the old tables' owner and ACL onto the new ones. +8. `DROP TABLE` the old `object_location` and `pack`; `ALTER TABLE … SET SCHEMA` the new ones into the Store's schema; create the `pack_immutable` trigger there, bound to the Store schema's `pack_immutable()` so doctor sees `EXECUTE FUNCTION pack_immutable()` (`diagnostic/postgresql_contract_expected.rs:213`). -10. Verify inside the transaction: row counts against the mark, - `live_heads_digest` unchanged, and the doctor contract query (tables, - constraints, indexes, trigger) clean. A failure rolls back. -11. `catalog_incarnation = G + 1`; `SET LOCAL synchronous_commit` to the - Store's setting or `local`, whichever is stronger; `COMMIT`. Then drop - the empty scratch schema. - -- **Crash:** a crash in phase A leaves only the scratch schema, which the op - record names and the next attempt removes. Phase B is atomic. -- **Lost `COMMIT` reply:** first prove the old backend has ended (the - exclusive advisory lock is free, or its recorded `pg_backend_pid()` is - gone), then read the incarnation. A read, not a replay. -- **Lock time:** `ACCESS EXCLUSIVE` is held for phase B only, seconds rather - than the length of the copy. No long transaction holds back vacuum for - other Stores in the same database. +9. Verify inside the transaction: row counts against the mark; + `SUM(new) = mark_totals - dead_*` with `logical` unchanged; the + `pg_attribute`/`attstorage`/`reloptions`/`tablespace` fingerprint recorded + in `gc_op` at mark equals the scratch tables' fingerprint; persist the + `dead_*` totals **before** the swap COMMIT. The settle transaction then + subtracts once and asserts the four `store_totals` sums. + `live_heads_digest` unchanged, and the bounded checks on the moved tables + (the `pack_immutable` trigger/function and `pg_attribute`/storage). Run only + the C/I contract entries for `pack`/`object_location` on the scratch tables + in Phase A (the trigger/function entries need the Store-schema function and + stay in the swap's bounded check), with `search_path=scratch`. A failure + rolls back. +10. `catalog_incarnation = G + 1`; the effective `synchronous_commit` is the + stronger of the Store's and the server's in the order + `off < local < remote_write < on < remote_apply`, set with `SET LOCAL` + around the swap and never lowered. It waits iff the level is at least + `remote_write` **and** `current_setting('synchronous_standby_names')` is + non-empty; then require the named FIRST/ANY standbys to appear in + `pg_stat_replication` with `state='streaming'` and the matching + `sync_state` (`sync`/`quorum`). Check before Phase M and again inside the + swap before COMMIT; a missing standby abandons with no writes (`Busy`, + standby list). A standby that dies after the check blocks the COMMIT; + `SET LOCAL` cannot bound that wait. + `DROP` the empty scratch schema before `COMMIT` (it is transactional). + +**Phase C — cleanup (batched, after the swap).** + +11. Delete only ids in the mark's **persisted dead set** in `gc_op` + (membership, not a predicate: exact `content_signature` slots, + `file_digest` roots, history/lineage ids, a deleted project's + `path_removed`/`project_epoch` rows, and marker/base row pairs). Per-table + watermarks (`commit`/`layer` serial, signature stamp) are only guards; a + row created after the mark is never a member, so a Commit/Layer/save + created during cleanup survives. Assert each batch's own `pack_id` range + and, after the last batch, `count(*) = count(DISTINCT object_id) = + |visited|`. Delete child-first (a project's closure + in one transaction; a marker with its base row) so the deferred FKs hold. + Clear `binding_hint_block` wholesale (it is not id-addressable, §6.4). + The swap already dropped dead packs and locations, so persist + `dead_canonical` (dropped `object_location.canonical_length`), + `dead_stored`/`dead_metadata` (dropped pack `length`; `stored` is all + domains, `metadata` only domain 0) and `dead_logical` in `gc_op` + **before** the swap COMMIT; `content_signature` deletes are by + `(slot, stamp)` with `stamp <= mark_stamp`; reduce + `store_totals` by those totals in the **same transaction that settles the + op**, or gate on `totals_applied`; never re-sum. Assert the new tables' + sums equal old totals minus dead totals. The op phase is `cleanup`; a crash resumes the + persisted set — there is no "re-derive from the current Store" path. The + Store is usable at `G + 1` throughout; the full doctor runs after cleanup. + +- **Scratch schema:** one deterministic, length-safe name per Store derived + from `catalog_id` (`lfs_gc_`, truncated to 63 bytes), recorded in the + Store-side `gc_op` row (§4.1) before `CREATE SCHEMA`; a crash names it from + the Store, not only from the caller's home. +- **Crash:** a crash in Phase M/A leaves only the scratch schema, which the + Store's op names and the next attempt removes. Phase B is atomic; a crash + after COMMIT resumes Phase C. +- **Lost `COMMIT` reply:** record `pid`, `backend_start`, `usename` and the + incarnation in `gc_op` in a transaction **before** the swap. Recovery on a + second connection (before taking the exclusive key) is one statement: + `SELECT pg_terminate_backend(a.pid, ) FROM pg_stat_activity a WHERE + a.pid = $1 AND a.backend_start = $2 AND a.usename = $3 AND + a.datname = current_database() AND a.pid <> pg_backend_pid()`, checking the + boolean and polling until the pair is gone, then reading the incarnation. + Terminating another role's backend needs `pg_signal_backend` or membership; + the precheck reports `Unsupported` when neither is available. +- **Lock time:** `ACCESS EXCLUSIVE` is held for the swap only, seconds rather + than the length of the copy or the cleanup. No long transaction holds back + vacuum for other Stores in the same database. - **Space:** old tables + new heap/TOAST + indexes + up to the live bytes of WAL. The guide states the WAL headroom requirement; a full `pg_wal` stops the whole server. -- **Privileges:** ownership of the Store's tables and `CREATE` on the schema, - otherwise `Unsupported`. +- **Privileges:** `CREATE` on the database (for the scratch schema), `TEMP` + (the visited tables), `USAGE` and `CREATE` on the Store schema, ownership + of the Store's tables, and `pg_read_all_stats` for `client_addr`/ + `backend_start` (which may be NULL); check all before Phase M, otherwise + `Unsupported`. ## 9. Failure and leftovers -- The op record names `G` and the staged artifact (`.compact` or the - scratch schema). +- The op record names `G` and the staged artifact. The SQLite artifact is + the set `.compact{,-journal,-wal,-shm,-visited,-visited-journal,-mj*}`; all of it is removed for the + reserved name, and the op is settled only after removal. The PostgreSQL + artifact is the scratch schema, whose deterministic name is recorded in the + Store's op row before it is created, so a crash on another machine can still + name and remove it. - The next `store.compact` judges it: the Store at `G` means not installed, so the named artifact is removed; `G+1` means installed, so the outcome is - recorded. A machine that cannot be reached reports `unknown`, never clean. + recorded. An op left uncertain by a dead owner is adopted by that + `store.compact` (its artifact and high-water are in `gc_op`); only a live + owner blocks it. A machine that cannot be reached reports `unknown`, never + clean. - A file or schema at the staged name that no record names is refused with the existing `Exists` and is not deleted. - Install is the only irreversible step: the Store is wholly old or wholly new. @@ -458,46 +960,130 @@ tables. `content_signature` (at most 8192 rows, no foreign key: - **SQLite:** at the start of every transaction, compare the Store path's `(dev, ino)` with the one recorded at open; a mismatch is `StaleStore` before anything is read or written. This covers both a reader and a writer - still holding the old file (formerly C8 and C8b). + still holding the old file (formerly C8 and C8b). Compaction holds + `lane::writer_admission` from the seal through the rename, so an admitted + writer cannot be in flight at install; a connection that bypassed admission + is caught at its next transaction. Install also requires the sealed + `(size, mtime, db-header change counter)` unchanged; a change is `Conflict` + with no writes. - **PostgreSQL:** the generation check of §5.2. - **`lfq1` tokens** of an older generation already fail, because their authenticator is keyed by the incarnation. | Code | Status | |---|---| -| `MountsActive`, `Busy`, `NotFound`, `NoSuchStore`, `Unsupported`, `ReservedName`, `Conflict`, `NameTaken`, `Capacity`, `Exists` | existing | +| `MountsActive`, `Busy`, `Unresolved`, `NotFound`, `NoSuchBranch`, `NoSuchCommit`, `NoSuchProject`, `NoSuchStore`, `Unsupported`, `ReservedName`, `Conflict`, `NameTaken`, `Capacity`, `Exists`, `InvalidInput` | existing (`Unresolved` now used by the four calls) | | `StaleStore` | new; one `HistoryError` variant with an explicit mapping arm, because the `_ => "Failed"` wildcard (`engine/crates/layerfs-store-service/src/history/lookup.rs:54,64`) would hide it | -| `Removed` | new; reading a path removed by `remove_file` | +| `Removed` | new; reading a path removed by `remove_file`; an explicit `HistoryError::Removed` arm in `lookup.rs` before the `_ => "Failed"` wildcard; detail `{paths: [{path, frontier_serial}]}` | -- A project create whose name is held returns `NameTaken` (today it returns - `Exists`), the same as a Branch create or fork. -- New `HistoryCatalog` methods get default `Unsupported` bodies so the eight - implementations need not change at once (C9). + +- `project.init` whose name is held returns `NameTaken` (today it returns + `Exists`), the same as a Branch create or fork; both the home record and the + Store name collision map to it. +- The new `HistoryCatalog` methods are enumerated in the #42 list + (`mark_*`, `retained_roots`, `get_retained_roots`, `remove_unreachable`); + both production providers implement them; the default body is `Unsupported`, + and `lookup.rs` maps `HistoryError::Unsupported` explicitly (C9). - All four calls belong to the MCP admin profile and are refused under a - folder scope (`api/crates/layerfs-api/src/calls/scope.rs`; #42 item 39). + folder scope (`api/crates/layerfs-api/src/calls/scope.rs:33`, `out of + scope: `; #42 item 39). A folder scope guards one folder and one + Branch; none of the four is a folder operation (`branch.remove` retires the + whole Branch, `project.*` the project, `store.compact` the Store), so they + are refused rather than reinterpreted at a wider scope. +- The four calls take `branch.remove(project, branch)`, + `project.delete(project)`, `project.remove_file(project, path)` and + `store.compact(name)`; no options. +- MCP: the admin classifier checks its admin list before the work families; + the four calls join it and `project.remove` leaves both the call table and + that list. `whyOf`, the profile prose and `--profile admin` usage name + the four calls; an old MCP `project_remove` is `InvalidParams "unknown + tool"`, a CLI usage error exits 2, and the library reports `InvalidInput`; + the precedence test covers `branch.remove` under `work`. The table becomes + 76 calls (38 work/38 admin); regenerate `gen-profiles`/`gen-coverage` and + the hard-coded 73/38/35 counts, and give `whyOf` one line per call. +- Results: `branch.remove` → `{kind: "Removed", project, branch, removed_ms}`; + `project.delete` → `{kind: "Removed", project, deleted_ms, + home_entry_removed}`; `project.remove_file` → §4.3's `Removed`/`UpToDate`; + `store.compact` → the §11 report. Success results and refusal details are + snake_case; refusal `detail` is snaked in the dispatcher (or built snake at + the refusal site), and a camelCase detail is a bug; the CLI prints the + rendered map; §11 names are logical. +- `MountsActive` carries `detail.holders` (mounts/Sandboxes); `Busy` carries + `{entity, phase, op}` plus the PostgreSQL holders (`pid`, `client_addr`, + `backend_start`). `InUse` stays for non-GC calls only. +- `Unresolved` carries `{entity: {kind, name}, phase, op}` and is observed by + a non-owning call and settled by the owning one: `store.compact` adopts a + dead compaction op; `project.delete`'s settlement reads the Store by the + recorded stack id (retired/absent finishes the entry, live redoes the + marker under the project op); `sandbox.close`/`machine.remove`/ + `store.remove` clear their own ops. `logs.reset` only truncates `calls.jsonl`; it cannot clear an op. + A live owner stays `Busy`; a provably dead, certain op is ignored. +- Capacity is the real `PersistenceError::Refused { status: "Capacity" }` + path (`StorageError::is_capacity_refusal`); add `StorageError::CapacityExceeded` + to that predicate, keep PG 53100/53200/53400/54000 (53300 + `too_many_connections` is transient and maps to `Busy`; 54001 is out), and + name the SQLite codes (`SQLITE_FULL`, `SQLITE_TOOBIG`, `SQLITE_NOMEM`). + `CapacityExceeded` (a codec/canonical bound) is documented as + `InvalidInput`/`ProviderFailure`, not disk exhaustion. The `_ => Refused` + arm gets an explicit Capacity arm. ## 11. Report -While compaction runs, it writes its phase (`mark`, `build`, `verify`, -`install`) into the op, and `store.status` shows it with the op's other fields +While compaction runs, it writes its phase into the op (`mark`, `build`, +`verify`, `install` on SQLite; `mark`, `build`, `swap`, `cleanup` on +PostgreSQL), and `store.status` shows it with the op's other fields (`control/crates/layerfs-home/src/op.rs:189-201`); there is no `PROGRESS` -option (#42 item 40). `store.compact` returns: +option (#42 item 40). On PostgreSQL `store.status` reads the Store-side +`gc_op`; on SQLite it reads the home op. The `op` projection is +`{id: op_id, name, phase, started: started_ms, uncertain}` from +`gc_op`/`gc_intent`, and `state` is `busy`/`unresolved` from it; a remote +SQLite Store's op is read from its home/sidecar via `step_on`. `store.compact` +returns: | Field | Meaning | |---|---| | `path` | `none` or `rebuild` | | `generation` | `{ before, after }` | -| `logicalDiscarded` | canonical bytes of collected objects | -| `backendReusable` | SQLite: the old file's free pages; PostgreSQL: 0 for rebuilt tables | +| `deadBytesBound` | bytes in wholly dead packs after the metadata walk; the value that chose the path | +| `deadBytesRefined` | the same bytes after the payload sweep (§7 step 4) | +| `logicalDiscarded` | the collected history rows' `logical` bytes (the `store_totals.logical` delta) | +| `backendReusable` | SQLite: rebuild = the new artifact's `freelist_count * page_size` before the rename (normally 0), none = the live Store's; PostgreSQL: 0 (no in-place reclaim) | | `physicalShrink` | SQLite: old file size minus new; PostgreSQL: `pg_total_relation_size` delta | -| `keptWaste` | bytes of dead objects in retained packs, bytes kept only as delta bases (with their references), and the kept `inode_value_block` bytes | - -A zero is always explained. Estimates are marked. +| `keptWaste` | object: `{deadInRetainedPacksBytes, deltaBases: {bytes, references: [id]}, inodeBlocksBytes}` | +| `peakTempBytes` | SQLite: the new Store + rollback journal/WAL + the `visited` scratch file and its journals/super-journal; PostgreSQL: the scratch schema + WAL | +| `residue` | object `{bytes, artifacts}`; a nonzero residue is explained | + +The result is `{kind: "Compacted", ...}`. `keptWaste` is +`{deadInRetainedPacksBytes, deltaBases: {bytes, references: [id]}, +inodeBlocksBytes}`; `residue` is `{bytes, artifacts: [string], reason?}`; +top-level `estimate: bool`; a capped field is `{value, cap}`; `zeroReason` +is per zero field with the closed vocabulary `empty-store`, `none-path`, +`no-history`, `no-waste`, `postgresql-in-place`; `markBytes` is a #52 metric, +not a result field. The SDK/CLI render every key +snake_case. On the `none` path `deadBytesRefined` equals `deadBytesBound` (no +payload sweep runs below the threshold), and the report says so. `physicalShrink` +reads the old size before the rename; `backendReusable` reads the SQLite +free-page count × page size before it; `logicalDiscarded` is the summed +`logical` of the collected history rows; PostgreSQL reports 0 for rebuilt tables and does +not count in-place free space. `physicalShrink` samples `pg_total_relation_size` +before and after the swap; on PostgreSQL `deadBytesRefined` equals +`deadBytesBound` (there is no payload sweep to refine it). A zero is always +explained; a capped field carries its cap (S7). ## 12. Acceptance (#52) +Each bullet is one case; its label is its case ID and must be unique. + - **Crash injection** (`kill -9`, per backend) at mark, build, verify, before and after install: wholly old or wholly new; the leftover is judged by §9. +- **Backend matrix:** every removal, admission, report and name item runs on + SQLite and PostgreSQL; a converted 1.0 Store opens with the additive step on + both. +- **Crash matrix:** named failpoints at every phase (SQLite: mark, build, + verify, install, WAL switch, artifact removal, op settle; PostgreSQL: mark, + build, swap, cleanup, scratch drop), plus lost COMMIT, power-loss + (fsync/rename) and unresolved op/lease takeover; assert §9 state, + generation, digest and closure. - **No live object lost:** property tests over random fork / commit / merge / publish / `branch.remove` / `project.delete` / `remove_file` histories; closure sets compared before and after. @@ -512,18 +1098,178 @@ A zero is always explained. Estimates are marked. transaction open at swap time makes the swap abandon. - **Admission races:** a mount and a removal of the same Branch started together never both succeed. -- **Names:** `NameTaken` after removal (Branch and project), a free name after - compaction collects the row, `ReservedName` for `main`. +- **Names:** `NameTaken` after removal (Branch and project) and on create + collision (Branch and project; project create was `Exists`), a free name + after a rebuild collects the row, `ReservedName` for `main`. - **`remove_file`:** `Removed` for Commits at or below the frontier, before and after compaction; the next Commit of a mounted older head records the - deletion; a file created later at the path is visible; bytes reclaimed only - when no other path, project or delta chain uses them. -- **Speed and space** (one sample per arm, declared caches, generated - fixtures): downtime per GB of live data and per GB freed, bytes written, - peak disk, `ACCESS EXCLUSIVE` duration on PostgreSQL. The rebuild's build - phase is compared with plain `VACUUM INTO` of the same Store; the mark is - reported separately. A storage option that costs about 50% speed for about - 5% storage is rejected. + deletion; a file created later at the path is visible; 2.0 frees no bytes + for the removed path (visibility only). +- **`remove_file` and derived nodes:** a non-fast-forward `branch.merge` of + two heads at or below the frontier keeps the path hidden (the whiteout is + materialized), and a file created after the frontier is visible; a + fast-forward merge is the control (no new tree); a `branch.publish` of a + Commit at or below the frontier yields a hidden Layer, a mount of it + whiteouts, and its next Commit records the deletion; a merge that cannot + apply the whiteout is refused `Removed`. +- **Removed path in a merge:** a non-fast-forward merge of two heads at or + below the frontier reports no conflict for the removed path and omits it; a + merge of an at-or-below head with a post-frontier re-creation keeps the + re-created file, and only an ordinary edit conflict on it is reported. +- **Unmerged Branches:** `remove_file` hides the path in a Commit on an + unmerged Branch created before the frontier; a Commit on that Branch after + the frontier does not inherit it. +- **Layer-headed mount:** a Branch whose head is empty and whose base Layer is + at or below the frontier whiteouts the path at mount. +- **Visibility only:** `remove_file` frees no bytes; an object reachable only + through the removed path is retained; the marker hides reads/listings and a + mount/merge/publish whiteouts the path. +- **Published Branch:** a Branch published by a retained Layer, or named by + a retained `commit.branch_id`, keeps its name after `branch.remove`; only an + unreferenced Branch's name is freed by a rebuild (not by a `none` pass). +- **Late base:** a delta base in an already-passed pack and a depth-2 chain + are copied by the worklist; `pack_id` is preserved. +- **Digest roots:** an uncommitted saved file's `file_digest` row is collected + unless another retained root reaches its content. +- **Path grammar:** `/a` and `a` are accepted; empty, `/`, `//a`, `a/`, + `a//b`, `./a`, `a/./b`, `.`/`..` components, `\`, NUL, >4096 bytes and >256 + components are refused `InvalidInput`; a directory hides its subtree; a + child of a removed non-directory is `NotFound`; an intermediate symlink is + `InvalidInput` and a leaf is hidden; a hardlink survives. +- **Repeated removal:** a second `remove_file` after a re-creation hides the + re-created file; a repeat with no newer project node is `UpToDate`, while a + commit in another project neither rewrites the marker nor advances it; a + never-existing path is accepted. +- **Marker epoch:** a one-shot write racing `branch.remove` or + `project.delete` gets `Conflict` and leaves no Commit; a merge racing + `remove_file` gets `Conflict`, and the retry applies the whiteout. +- **Concurrent merge and removal (property):** a merge and a `remove_file` + started together never leave the path visible in a derived tree: one side + wins, the other either sees the marker or gets `Conflict` and retries with + the whiteout. +- **Marker cleanup:** a rebuild of a Store with a deleted project leaves no + `path_removed` row for it and keeps a live project's markers; a `none` pass + writes nothing and retains them. +- **Position state:** a converted Store backfills serials; + `next_history_serial` never falls after compaction; `project_epoch` copies + and bumps only on a marker write. +- **Marker survival:** a retained `branch_retired`/`layer_stack_retired`/ + `path_removed` survives a rebuild; a collected row takes its marker with it. +- **project.delete retry:** the marker commits, the home removal fails → + hidden, entry present, `home_entry_removed: false`; a same-home retry + finishes and returns the stored `deleted_ms`, with no second marker; the + own-Init identity is saved before the marker and a save failure removes + nothing. +- **Home listing:** a second home's stale entry is absent from `project.list` + and `project.get` answers `NoSuchProject`; Store calls agree. +- **PG cleanup:** a Commit/save during Phase C survives the high-water bound; + a crash resumes from `gc_op`; a marker and its base row are deleted in one + batch (no resurrection/orphan), with a failpoint at the boundary. +- **Admission:** a mount from another home/machine is visible in the Store-side + intent; `machine_stopped` is ignored; a second mount of one project + proceeds; an uncertain dead op is `Unresolved` and the owning call clears it. +- **NameTaken:** `detail {holder: {kind, name, state: live|retired|deleted}, + source}`; a forced two-reference fixture (Layer FK and `commit.branch_id`) + keeps the name; one reference and neither are the controls. The error + matrix covers `MountsActive`, `Busy`, `NoSuchBranch`/`NoSuchCommit`/ + `NoSuchProject`, `NameTaken`, `ReservedName`, `Removed`, `InvalidInput`. +- **Trace salt:** the call log stores `` for the path; the trace key + is the keyed hash; `Removed` echoes the path; a rebuild copies the salt. +- **Admission (liveness):** a live-but-unreachable Sandbox is `Busy`; + `machine_stopped` is ignored; a `gc_intent` from another home is visible; + `Unresolved` is observed by a non-owning call and settled by the owner. +- **Checkpoint/install:** a live reader makes `wal_checkpoint(TRUNCATE)` + return `busy>0` → `Busy`, no artifact, no op; a sidecar or `-mj*` at install + is `Conflict` with no writes. +- **Bind failure:** a committed intent whose bind fails is deleted in the + error path; a failed delete is reported; no `MountsActive` remains. +- **PG refcount:** two mounts keep the Store/project keys after one unmount; + a failed all-or-none acquire unlocks the taken subset. +- **Worker/lost reply:** an unreachable worker op is `unknown, never clean`; + a lost reply keeps the sidecar and the retry resolves retired/live/absent. +- **Trace:** the six path names are redacted; refusal details are not traced; + the salt is backfilled for an upgraded Store. +- **Artifacts (cross-backend):** SQLite leaves/removes + `.compact{,-journal,-wal,-shm,-visited,-visited-journal,-mj*}`; + PostgreSQL's `lfs_gc_` is recorded before CREATE, removable from + another machine, and a stray schema is `Exists`. +- **Position state (2.0 backfill):** every upgraded Store (2.0 included) + gets one `project_epoch` row per stack; `gc_op` is cleared on settle and + not copied; `next_history_serial` never lowered. +- **Digest agreement:** SQLite and PostgreSQL report the same + `live_heads_digest` for the same fixture; a retired head changes it. +- **PostgreSQL mark:** the mark transfers only `markBytes`; the table copy + moves no bytes; a base object that only a dead pack holds is kept. +- **Reserved pack:** a Save that reserves a pack/ordinal before the mark and + publishes during Phase A (or Phase C) moves `store_totals`; the swap + `Conflict`s, the pack stays in the old Store, and the cleanup failpoint + keeps it; no dangling location. +- **PG cleanup (resume):** after the swap the Store is usable at `G + 1`; + Phase C resumes after a crash; totals/ACL/trigger/hints are asserted (no + `inode` on PG; that assertion belongs to the SQLite install). +- **PG privileges and reply:** missing `CREATE`/`TEMP`/`pg_read_all_stats` is + `Unsupported`; a lost `COMMIT` reply is resolved by proving the backend + ended, then reading the incarnation. +- **SQLite layouts:** schema 36 (`Monolithic`), 37 (`GroupRows`), 38 + (`GroupRowsIndexed`) and their 42/43/44 acquisition variants rebuild; + `Monolithic` copies `pack` (body) and no `pack_unit`; `GroupRows` copies + both; `GroupRowsIndexed` copies `pack_unit` and the schema creates its + index (not "copied"); `sqlite_sequence` is kept when present and + `init_*`/acquisition tables are empty. +- **Install fence:** a connection that bypassed the writer lane and opens a + transaction after install gets `StaleStore`; an in-flight transaction is + excluded by the lane (a documented limit); a sealed-state or sidecar change + makes install `Conflict` with no writes. +- **Artifacts (SQLite exact):** a crash leaves + `.compact{,-journal,-wal,-shm,-visited,-visited-journal,-mj*}`; the next + compact removes all of it; `none` leaves no `.compact`; a stray name with + no op is `Exists`. +- **Threshold and failures:** 9% wholly dead → `none`; 10% and 30% → + `rebuild`; an empty Store (0/0, COALESCEd sums) → `none` with a + `zeroReason`; a started rebuild completes when the refined share drops + below 10%; `Capacity` and a verify failure abort, delete the artifact set + and leave the old Store unchanged apart from step 1's checkpoint. +- **Mount/compact race:** a mount during compact is `Busy` with the compact + op; compaction returns `MountsActive` when a mount wins; a Commit during the + rebuild is not lost (PG: a write during Phase A makes the swap `Conflict`; + during Phase C it survives the high-water bound). +- **Profile and scope:** all four calls are refused under a folder scope + (`out of scope`) and allowed only in the MCP admin profile. +- **Progress:** while `store.compact` runs, `store.status` shows the op phase + (`mark`, `build`, `verify`, `install` on SQLite; `mark`, `build`, `swap`, + `cleanup` on PostgreSQL). +- **Report:** both paths; `kind:"Compacted"`, snake_case keys, every §11 + field with its unit and sampling point (`freelist_count * page_size`, old + size before rename, history `logical` delta, peak temp = new file + + journal/WAL + visited, `physicalShrink`, `keptWaste` components, + `residue {bytes,artifacts}`, `estimate`, capped `cap`, a `zeroReason`); + `markBytes` is a #52 metric (PG-only, not a result field); + `keptWaste` is `{deadInRetainedPacksBytes, deltaBases:{bytes,references}, + inodeBlocksBytes}`; bound≥refined; PG `backendReusable = 0` and PG/none + `deadBytesRefined == deadBytesBound`. +- **Read surface:** `file.read`/`stat`/`list`/`glob`/`diff`/`revision`/ + `export`/cp-from before and after compaction, with the §4.3 shapes; add + `edit`/`grep`/`blame`/`rm`/`mv`/`mkdir`/`cp`/`apply`, the + visible/hidden/absent diff table, `removed_count` (0 and >0), + `branch.diff`/`commit.diff`, and one whiteout case per one-shot tree + creator (`write`/`edit`/`apply`/`rm`/`mkdir`/`mv`/`cp`). +- **Restore and listing:** `backup.restore` before and after a removal; a + second home's stale entry; the §4.2 guard test and allow-list. +- **Error matrix:** exact code and message per case, including `StaleStore`, + `Removed`, `InvalidInput` for `project.remove`, `Exists`, `Unsupported` and + `Capacity`. +- **Digest golden vector:** a fixed-id fixture (out-of-order ids, a retired + Branch and stack, differing timestamps) hashes to one literal vector on both + backends; an absent head is 33 zero bytes, not 32, and widths are fixed at + 17/33. +- **Speed and space** (at least three runs per arm, median and range, + declared caches, generated fixtures): downtime per GB of live data and per + GB freed, bytes written, peak disk, `ACCESS EXCLUSIVE` duration on + PostgreSQL. Downtime per GB freed is absolute bytes when nothing is freed + (the `none` path and removal-only runs). The rebuild's build phase is + compared with plain `VACUUM INTO` of the same Store; the mark is reported + separately. A storage option that costs about 50% more time for about 5% + less space is rejected. ## 13. Estimates, not measurements @@ -532,3 +1278,18 @@ the review of 2026-11-14 estimated about 2–3 minutes of downtime for §7, and hours if the full doctor ran before and after. For PostgreSQL with the same Store, about 9 minutes in total with `ACCESS EXCLUSIVE` held for about two seconds. Nothing has been measured; #52 owns the numbers. + +#52 measures, per backend, these arms: `none` at 9% wholly dead; `rebuild` at +10% and 30%; a 50 GB total / 35 GB live (30% dead) Store, plus a 50 GB +live-store arm; and each removal type (branch, project, remove_file) as a +marker-only arm, plus removal-then-compact at a declared share. A +`remove_file`/`remove_file`-then-compact arm is 0-freed and reports absolute +bytes. At least three runs per arm, median and range; declare the hardware, +SQLite 3.53.2 with `page_size`, `journal_mode`, caches, `synchronous_commit` +and PostgreSQL version. Record the +§11 fields, `markBytes`, downtime per GB of live data and absolute bytes when +nothing is freed (0/0 or removal-only), bytes written, peak disk, +`ACCESS EXCLUSIVE` duration, WAL bytes and the `VACUUM INTO` baseline; use +`store.status` for the phase. A storage option that costs about 50% more time +for about 5% less space is rejected, naming the option and the measured +numbers. diff --git a/core/docs/issues/project-41/gc-acceptance-matrix.md b/core/docs/issues/project-41/gc-acceptance-matrix.md new file mode 100644 index 0000000000..70ad3d7a43 --- /dev/null +++ b/core/docs/issues/project-41/gc-acceptance-matrix.md @@ -0,0 +1,108 @@ +# GC acceptance matrix: #52 evidence plan for the §12 cases + +> Draft matrix for the frozen `LayerFS-GC-spec.md` §12 (the spec wins; this +> page adds no contract). Main layer `32d78561a`. AC-01..AC-57 are the §12 +> bullets one-to-one, in spec order; AC-58..AC-60 pin sub-steps of §7/§8 that +> §12 folds into broader cases. Each row's expected result restates spec text +> only; a mismatch is a spec bug, not a matrix change. +> +> Status vocabulary: `open` (no §12-conforming evidence recorded yet), +> `pass`, `fail`, `waived`. §13 says nothing has been measured: every E3 row +> starts `open`. Tiers are this page's evidence scale, not contract. + +## Evidence tiers (E0-E3) + +| Tier | Meaning | +|---|---| +| E0 | Spec reasoning only; no executable evidence. Does not accept a case. | +| E1 | One deterministic test on the named backend, fixed fixture, exit asserted. | +| E2 | Deterministic CI test on both backends (or the named backend's full named-phase/failpoint matrix), fixed fixture, state, refusal and digest asserted. | +| E3 | Measured or statistical: at least 3 runs with median and range (hardware, caches and versions declared), a property test over random histories, or a 50 GB scale arm. | + +## Matrix + +| Case ID | Contract (spec §) | Backend | Fixture / command | Expected result | Tier | Status | +|---|---|---|---|---|---|---| +| AC-01 | §12 Crash injection; §9 | both | Store with one wholly dead pack; `kill -9` the daemon at mark, build, verify, before and after install; next `store.compact` | Store is wholly old or wholly new; the leftover is judged by §9; generation and digest asserted | E2 | open | +| AC-02 | §12 Backend matrix; §4.1 | both | Run every removal, admission, report and name case on SQLite and PostgreSQL; open a converted 1.0 Store on both | Each case's §12 result holds on both backends; the additive step adds the §4.1 tables on both | E2 | open | +| AC-03 | §12 Crash matrix; §9 | both | Named failpoints: SQLite mark, build, verify, install, WAL switch, artifact removal, op settle; PG mark, build, swap, cleanup, scratch drop; plus lost COMMIT, power loss (fsync/rename) and unresolved op/lease takeover | §9 state, generation, digest and closure hold after resume; only a live owner blocks; an unreachable machine reports unknown, never clean | E2 | open | +| AC-04 | §12 No live object lost; §6.1-§6.2 | both | Property test over random fork, commit, merge, publish, `branch.remove`, `project.delete` and `remove_file` histories; compare closure sets before and after | No live object reaches a collected row; the retained closure is unchanged | E3 | open | +| AC-05 | §12 Independent walk; §9 | both | After compaction, walk the new Store with `diagnostic/scan.rs` and run the full doctor on the fixtures (#155 D-23) | Walk and doctor pass on the compacted Store; no dangling reference | E2 | open | +| AC-06 | §12 Regression; §6.4 | both | After compaction, Save bytes whose `file_digest` row named a collected root | The Save succeeds; no stale digest row blocks it | E2 | open | +| AC-07 | §12 Fences; §10 | both | A connection that bypassed admission; a second compaction; a remote PG daemon holding a mount; a one-shot SDK transaction open at swap time | Bypass gets `StaleStore`; second compaction gets `Busy`; the mount blocks compaction and the matching removal, naming itself; the open transaction makes the swap abandon | E2 | open | +| AC-08 | §12 Admission races; §5.1 | both | Start a mount and a removal of the same Branch together | Never both succeed; the loser gets `MountsActive` or `Busy` | E2 | open | +| AC-09 | §12 Names; §4.4 | both | `NameTaken` after removal (Branch and project); create collision (Branch and project; project create was `Exists`); free name after a rebuild collects the row; `ReservedName` for `main` | Exact codes per §4.4; only a rebuild frees an unreferenced name | E2 | open | +| AC-10 | §12 remove_file; §4.3 | both | `Removed` for Commits at or below the frontier, before and after compaction; the next Commit of a mounted older head records the deletion; a file created later at the path is visible | Codes and visibility per §4.3; 2.0 frees no bytes for the removed path | E2 | open | +| AC-11 | §12 remove_file and derived nodes; §4.3 | both | Non-FF `branch.merge` of two heads at or below the frontier; a post-frontier re-creation; a FF merge control; `branch.publish` of an at-or-below Commit; a merge that cannot apply the whiteout | Path stays hidden, whiteout materialized; post-frontier file visible; FF merge creates no tree; published Layer hidden, its mount whiteouts, next Commit records the deletion; inapplicable whiteout refused `Removed` | E2 | open | +| AC-12 | §12 Removed path in a merge; §4.3 | both | Non-FF merge of two at-or-below heads; merge of an at-or-below head with a post-frontier re-creation | No conflict for the removed path, it is omitted; the re-created file is kept and only an ordinary edit conflict on it is reported | E2 | open | +| AC-13 | §12 Unmerged Branches; §4.3 | both | `remove_file` then a Commit on an unmerged Branch created before the frontier; another Commit after the frontier | The earlier Commit's path is hidden, the later one does not inherit it | E2 | open | +| AC-14 | §12 Layer-headed mount; §4.3 | both | Branch whose head is empty and whose base Layer is at or below the frontier; mount it | The path is whiteouted at mount | E2 | open | +| AC-15 | §12 Visibility only; §6.2 | both | Object reachable only through the removed path; marker hiding reads/listings; mount, merge and publish whiteouts | No bytes freed; the object is retained; reads/listings hide the path; derived trees whiteout it | E2 | open | +| AC-16 | §12 Published Branch; §4.4 | both | Branch published by a retained Layer; Branch named by a retained `commit.branch_id`; `branch.remove` then rebuild | Name stays taken; only an unreferenced Branch's name is freed by a rebuild (not by a `none` pass) | E2 | open | +| AC-17 | §12 Late base; §7 step 4 | both | Delta base in an already-passed pack; a depth-2 chain | Worklist copies both; `pack_id` preserved | E2 | open | +| AC-18 | §12 Digest roots; §6.4 | both | Uncommitted saved file's `file_digest` row; another retained root reaching its content | Row collected unless the retained root reaches the content | E2 | open | +| AC-19 | §12 Path grammar; §4.3 | both | `/a` and `a`; empty, `/`, `//a`, `a/`, `a//b`, `./a`, `a/./b`, dot and dot-dot components, backslash, NUL, >4096 bytes, >256 components; directory subtree; child of a removed non-directory; symlink leaf and intermediate component; hardlink | Accepted paths hide correctly; bad forms are `InvalidInput`; directory hides its subtree; child of removed non-directory `NotFound`; intermediate symlink `InvalidInput`, leaf hidden; hardlink survives | E2 | open | +| AC-20 | §12 Repeated removal; §4.3 | both | Second `remove_file` after a re-creation; repeat with no newer project node; a commit in another project; never-existing path | Re-creation hidden; repeat `UpToDate`; other project neither rewrites nor advances the marker; never-existing path accepted, hides nothing | E2 | open | +| AC-21 | §12 Marker epoch; §5 | both | One-shot write racing `branch.remove` or `project.delete`; merge racing `remove_file` | `Conflict`, no Commit left; retry applies the whiteout | E2 | open | +| AC-22 | §12 Concurrent merge and removal (property); §4.3 | both | Property test: a merge and a `remove_file` started together | The path is never left visible in a derived tree; one side wins, the other sees the marker or gets `Conflict` and retries with the whiteout | E3 | open | +| AC-23 | §12 Marker cleanup; §6.4 | both | Rebuild of a Store with a deleted project and a live project with markers; a `none` pass | No `path_removed` row for the deleted project; live markers kept; `none` writes nothing and retains them | E2 | open | +| AC-24 | §12 Position state; §4.1 | both | Converted Store; `store.compact`; a marker write | Serials backfilled; `next_history_serial` never falls; `project_epoch` copied and bumped only on a marker write | E2 | open | +| AC-25 | §12 Marker survival; §4.4 | both | Retained `branch_retired`, `layer_stack_retired` and `path_removed`; a collected base row | Retained markers survive a rebuild; a collected row takes its marker with it | E2 | open | +| AC-26 | §12 project.delete retry; §4 | both | Marker commits and home removal fails; same-home retry; own-Init identity save failure | Hidden, entry present, `home_entry_removed: false`; retry finishes and returns the stored `deleted_ms`, no second marker; the identity save precedes the marker, a save failure removes nothing | E2 | open | +| AC-27 | §12 Home listing; §4.2 | both | Second home's stale entry; `project.list` and `project.get`; Store calls | Stale entry absent from `project.list`, `project.get` answers `NoSuchProject`; Store calls agree | E2 | open | +| AC-28 | §12 PG cleanup; §8 Phase C | PG | Commit/save during Phase C; crash mid-cleanup; failpoint at the marker/base-row batch boundary | The late write survives the high-water bound; cleanup resumes from `gc_op`; marker and base row deleted in one batch, no resurrection or orphan | E2 | open | +| AC-29 | §12 Admission; §5.1 | both | Mount from another home or machine; `machine_stopped` evidence; a second mount of one project; an uncertain dead op | The mount is visible in the Store-side intent; `machine_stopped` ignored; the second mount proceeds; `Unresolved` observed and settled by the owning call | E2 | open | +| AC-30 | §12 NameTaken; §4.4 | both | `NameTaken` detail for live, retired and deleted holders; forced two-reference fixture (Layer FK and `commit.branch_id`); one-reference and no-reference controls; error matrix | Detail carries holder kind, name and state; two references keep the name, the controls do not; matrix covers `MountsActive`, `Busy`, `NoSuchBranch`, `NoSuchCommit`, `NoSuchProject`, `NameTaken`, `ReservedName`, `Removed`, `InvalidInput` | E2 | open | +| AC-31 | §12 Trace salt; §4.3 | both | Call log with path arguments; `Removed` refusal; rebuild | Log stores ``, trace key is the keyed hash; `Removed` echoes the path; the salt is copied by the rebuild | E2 | open | +| AC-32 | §12 Admission (liveness); §5.1 | both | Live-but-unreachable Sandbox; `machine_stopped`; `gc_intent` from another home; non-owning call | Live-unreachable is `Busy`; `machine_stopped` ignored; the other home's intent visible; `Unresolved` observed by the non-owner and settled by the owner | E2 | open | +| AC-33 | §12 Checkpoint/install; §7 steps 1, 7 | SQLite | Live reader during `wal_checkpoint(TRUNCATE)`; a sidecar or `-mj*` present at install | `busy>0` makes the checkpoint `Busy` with no artifact and no op; a sidecar at install is `Conflict` with no writes | E2 | open | +| AC-34 | §12 Bind failure; §5.1 | both | Committed intent whose bind fails; failed intent delete | Intent deleted in the error path; the failed delete is reported; no `MountsActive` remains | E2 | open | +| AC-35 | §12 PG refcount; §5.2 | PG | Two mounts, one unmount; failed all-or-none acquire | Store and project advisory keys stay after one unmount; the failed acquire unlocks the taken subset | E2 | open | +| AC-36 | §12 Worker/lost reply; §4, §9 | both | Unreachable worker op; lost reply; retry | Op is `unknown`, never clean; the sidecar is kept; the retry resolves retired, live or absent | E2 | open | +| AC-37 | §12 Trace; §4.3 | both | Six path names; refusal details; upgraded Store | The six path names are redacted; refusal details are not traced; the salt is backfilled | E2 | open | +| AC-38 | §12 Artifacts (cross-backend); §9 | both | SQLite artifact set `.compact{,-journal,-wal,-shm,-visited,-visited-journal,-mj*}`; PG `lfs_gc_` schema; a stray schema | SQLite leaves and removes the full set; PG records the name before CREATE, a remote machine can remove it, a stray schema is `Exists` | E2 | open | +| AC-39 | §12 Position state (2.0 backfill); §4.1 | both | Every upgraded Store (2.0 included); `gc_op` settle; serial allocation | One `project_epoch` row per stack; `gc_op` cleared on settle and not copied; `next_history_serial` never lowered | E2 | open | +| AC-40 | §12 Digest agreement; §6.4 | both | Same fixture on SQLite and PostgreSQL; retired head | Same `live_heads_digest`; a retired head changes it | E2 | open | +| AC-41 | §12 PostgreSQL mark; §8 Phase M | PG | Mark transfer; table copy; a base object held only by a dead pack | Only `markBytes` crosses the wire; the copy moves no pack bytes; the base is kept | E2 | open | +| AC-42 | §12 Reserved pack; §8 Phase B | PG | Save reserves a pack/ordinal before the mark and publishes during Phase A (or C); cleanup failpoint | `store_totals` moves; the swap `Conflict`s; the pack stays in the old Store; the failpoint keeps it; no dangling location | E2 | open | +| AC-43 | §12 PG cleanup (resume); §8 Phase C | PG | After the swap at `G+1`; crash mid-cleanup; totals, ACL, trigger, hints | Store usable at `G+1`; cleanup resumes; totals/ACL/trigger/hints asserted; no `inode` on PG | E2 | open | +| AC-44 | §12 PG privileges and reply; §8 | PG | Missing `CREATE`, `TEMP` or `pg_read_all_stats`; lost `COMMIT` reply | `Unsupported`; the lost reply is resolved by proving the backend ended, then reading the incarnation | E2 | open | +| AC-45 | §12 SQLite layouts; §7 step 2 | SQLite | Schema 36 (`Monolithic`), 37 (`GroupRows`), 38 (`GroupRowsIndexed`) and their 42/43/44 acquisition variants rebuild | `Monolithic` copies `pack` body and no `pack_unit`; `GroupRows` copies both; `GroupRowsIndexed` copies `pack_unit` and the schema creates its index; `sqlite_sequence` kept when present; `init_*`/acquisition tables empty | E1 | open | +| AC-46 | §12 Install fence; §10 | SQLite | Connection that bypassed the writer lane opening a transaction after install; in-flight transaction; sealed-state or sidecar change at install | `StaleStore`; the in-flight transaction is excluded by the lane (documented limit); a sealed-state or sidecar change is `Conflict` with no writes | E2 | open | +| AC-47 | §12 Artifacts (SQLite exact); §9 | SQLite | Crash mid-compact; next compact; `none` pass; stray name with no op | Crash leaves the full `.compact` set; the next compact removes all of it; `none` leaves no `.compact`; a stray name is `Exists` | E2 | open | +| AC-48 | §12 Threshold and failures; §3 | both | 9%, 10% and 30% wholly dead; empty Store (0/0); started rebuild whose refined share falls below 10%; `Capacity`; verify failure | 9% is `none`, 10% and 30% are `rebuild`; empty Store is `none` with `zeroReason: empty-store`; the started rebuild completes; `Capacity` and verify failure abort, delete the artifact set and leave the old Store unchanged apart from step 1's checkpoint | E2 | open | +| AC-49 | §12 Mount/compact race; §5, §8 | both | Mount during compact; mount wins the race; Commit during the rebuild | `Busy` with the compact op; compaction returns `MountsActive` when the mount wins; the Commit is not lost (PG Phase A write makes the swap `Conflict`; Phase C survives the high-water bound) | E2 | open | +| AC-50 | §12 Profile and scope; §10 | both | All four calls under a folder scope; MCP admin profile | Refused `out of scope`; allowed only in the admin profile | E2 | open | +| AC-51 | §12 Progress; §11 | both | `store.compact` running; `store.status` | `store.status` shows the op phase: SQLite `mark`, `build`, `verify`, `install`; PG `mark`, `build`, `swap`, `cleanup` | E2 | open | +| AC-52 | §12 Report; §11 | both | `none` and `rebuild` paths; every §11 field | `kind: Compacted`, snake_case keys, every §11 field with its unit and sampling point (`freelist_count * page_size`, old size before rename, history `logical` delta, peak temp, `physicalShrink`, `keptWaste`, `residue`, `estimate`, capped `cap`, `zeroReason`); `markBytes` is a #52 metric, not a result field; bound at least refined; PG `backendReusable` = 0 and PG/none `deadBytesRefined` equals `deadBytesBound` | E2 | open | +| AC-53 | §12 Read surface; §4.3 | both | `file.read`/`stat`/`list`/`glob`/`diff`/`revision`/`export`/cp-from before and after compaction; `edit`/`grep`/`blame`/`rm`/`mv`/`mkdir`/`cp`/`apply`; `branch.diff`/`commit.diff` | §4.3 shapes; visible/hidden/absent diff table; `removed_count` 0 and >0; one whiteout case per one-shot tree creator | E2 | open | +| AC-54 | §12 Restore and listing; §4.2 | both | `backup.restore` before and after a removal; second home's stale entry | Restore succeeds; the stale entry stays out; the §4.2 guard test and allow-list hold | E2 | open | +| AC-55 | §12 Error matrix; §10 | both | Exact code and message per case, including `StaleStore`, `Removed`, `InvalidInput` for `project.remove`, `Exists`, `Unsupported`, `Capacity` | Exact code and message; snake_case details | E2 | open | +| AC-56 | §12 Digest golden vector; §6.4 | both | Fixed-id fixture (out-of-order ids, a retired Branch and stack, differing timestamps); absent head | One literal vector on both backends; absent head is 33 zero bytes, not 32; widths fixed at 17/33 | E3 | open | +| AC-57 | §12 Speed and space; §13 | both | At least 3 runs per arm, median and range, declared caches, generated fixtures; §13 arms: `none` at 9%, `rebuild` at 10% and 30%, 50 GB total/35 GB live, 50 GB live, each removal type, removal-then-compact, `remove_file` 0-freed | Downtime per GB live and per GB freed, bytes written, peak disk, PG `ACCESS EXCLUSIVE` duration; absolute bytes when nothing is freed; build phase vs plain `VACUUM INTO`; mark reported separately; a 50% time/5% space option rejected | E3 | open | +| AC-58 | §7 step 1.5 (refined decision) | SQLite | After the seal: `ATTACH` the old file `immutable=1`, create `.compact-visited`, run the complete §6.1+§6.2 closure, choose `none`/`rebuild` from the refined live set | `none` deletes the scratch and stops with no `.compact` and no residue; `rebuild` keeps the scratch, reuses `visited` in step 4; step 2 creates the new file only on `rebuild` | E2 | open | +| AC-59 | §7 step 1 (seal) | SQLite | `PRAGMA wal_checkpoint(TRUNCATE)`; checked-close the last session; sidecar check under the writer guard; `seal_holding()` | `busy == 0` required; WAL never unlinked; sealed `(size, mtime, db-header change counter)` recorded before release; the guard passes into the seal | E2 | open | +| AC-60 | §8 Phase B step 10 (standby) | PG | Effective `synchronous_commit` at least `remote_write` with `synchronous_standby_names` set; standby missing before Phase M; standby missing inside the swap; standby dies after the check | Wait only when required; `FIRST`/`ANY` standbys must be `state=streaming` with matching `sync_state`; a missing standby abandons before COMMIT with no writes (`Busy`, standby list); a standby dying after the check blocks COMMIT | E2 | open | + +## Coverage map (required area -> cases) + +- Removals: branch AC-09, AC-16; project AC-26, AC-27; file AC-10..AC-21, AC-23..AC-25, AC-54. +- Admission: AC-07, AC-08, AC-29, AC-32, AC-34, AC-35, AC-49; epoch AC-21, AC-24, AC-39. +- Mark and closure: AC-04, AC-05, AC-15, AC-17, AC-18, AC-25, AC-40, AC-41. +- SQLite compaction (step 1.5, seal, sidecars, install, artifacts): AC-33, AC-46, AC-47, AC-58, AC-59; crash coverage AC-01, AC-03, AC-48. +- PG compaction (mark, swap, cleanup, scratch, standby, lost reply): AC-28, AC-41, AC-42, AC-43, AC-44, AC-49, AC-60; scratch in AC-38. +- Report and `store.status`: AC-51, AC-52. +- API and error shapes: AC-09, AC-26, AC-30, AC-34, AC-36, AC-50, AC-55. +- Read surface: AC-53, AC-54. +- Crash failpoints: AC-01, AC-03, AC-28, AC-36, AC-46, AC-47. +- 9/10/30% threshold: AC-48; measured arms in AC-57. +- Golden digest: AC-40, AC-56. +- 50 GB capacity: AC-57 (§13 arms). +- Names: AC-09, AC-16, AC-30. +- Trace: AC-31, AC-37. + +## Notes + +- `markBytes` is measured in AC-41 and AC-57, never reported as a `store.compact` field (§11). +- A case is accepted at E2 only when the named backend's failpoints are exercised; E3 rows additionally need three runs or the property test, per §12 Speed and space. +- Evidence recording (run id, fixture hash, spec layer, artifact paths) goes beside the status once #52 runs each case. diff --git a/core/docs/issues/project-41/gc-api-contract.md b/core/docs/issues/project-41/gc-api-contract.md new file mode 100644 index 0000000000..1655c951ec --- /dev/null +++ b/core/docs/issues/project-41/gc-api-contract.md @@ -0,0 +1,120 @@ +# GC API contract (T11): four calls, results, refusals + +> Draft for T11 of layerfs-project#41, derived **only** from the frozen `LayerFS-GC-spec.md` (the spec wins; `§N` cites it). No behavior beyond the spec is asserted; unknowns are listed in §6. + +## 1. Calls and signatures + +| Call | Arguments | Scope | +|---|---|---| +| `branch.remove(project, branch)` | project, Branch | one Branch | +| `project.delete(project)` | project | one project | +| `project.remove_file(project, path)` | project, absolute POSIX path | one project + path | +| `store.compact(name)` | Store | one Store | + +Each takes only the resource identifier: no confirm, force, dry-run, stop-Sandboxes, Store, URL or volume option (§1). The +three removals are logical and reclaim nothing; `store.compact` is the only byte-reclaiming call (§1, §4). `project.remove` is +retired — an old client gets `InvalidInput` ("no call named `project.remove`"), never a silent hard delete (§4). +`branch.remove` of `main` is `ReservedName` (§4). `store.compact` picks `none`/`rebuild` by the 10% wholly-dead-pack rule; a +started rebuild completes even below 10% (§3, #42 item 41). + +## 2. Success shapes + +snake_case on every surface (§10); these fields are required: + +- `branch.remove` → `{kind: "Removed", project, branch, removed_ms}` (§10). +- `project.delete` → `{kind: "Removed", project, deleted_ms, home_entry_removed}`; the worker frame is `{store, stack_id, + deleted_ms?}`, its reply `{kind: "Removed", project, stack_id, deleted_ms}`, and the caller echoes `home_entry_removed` (§4, + §10). +- `project.remove_file` → `{kind: "Removed"|"UpToDate", project, path, frontier_serial, removed_ms}` — the stored marker + values. `UpToDate` iff no project node is above **this path's** marker frontier (missing = -1); a commit in another project + does not make it `UpToDate` (§4.3). +- `store.compact` → `{kind: "Compacted", ...}` with §11's report: `path` (`none`|`rebuild`), `generation {before, after}`, + `deadBytesBound`, `deadBytesRefined`, `logicalDiscarded`, `backendReusable`, `physicalShrink`, `keptWaste + {deadInRetainedPacksBytes, deltaBases {bytes, references}, inodeBlocksBytes}`, `peakTempBytes`, `residue {bytes, artifacts[, + reason]}` (§11). + +Binding rules: `none` leaves the generation unchanged, `rebuild` is `G → G+1` (§3, §11). `estimate: bool` is top-level; a +capped field is `{value, cap}`; every zero field names a `zeroReason` from `empty-store`, `none-path`, `no-history`, +`no-waste`, `postgresql-in-place` (§11, S7). On `none`, `deadBytesRefined == deadBytesBound`; on PostgreSQL they are always +equal (no payload sweep) (§11). `home_entry_removed: false` is still success — the marker is the commit point and a same-home +retry is idempotent (§4). A repeat `branch.remove` is `NoSuchBranch` (a concurrent one is `Busy`, phase `"retire"`); a repeat +`project.delete` is §4's idempotent success (§4.4). + +## 3. Refusals and detail shapes + +| Code | GC trigger | `detail` | +|---|---|---| +| `MountsActive` | a mount/Sandbox of the Branch or project is in scope (§5) | `holders` (mounts/Sandboxes) | +| `Busy` | another user of the project/Store, a live op, a busy lane, a concurrent remove (§5, §4.4) | `{entity, phase, op}` (+ PG `pid`, `client_addr`, `backend_start`) | +| `Unresolved` | a dead owner left an uncertain op; observed by a non-owning call (§5.1, §10) | `{entity: {kind, name}, phase, op}` | +| `Removed` | a read/stat/merge hits a `remove_file` marker (§4.3, §10) | `{paths: [{path, frontier_serial}]}` | +| `NameTaken` | a create/fork/merge target name still held; `project.init` (§4.4, §10) | `{holder: {kind: branch\|project, name, state: live\|retired\|deleted}, source: store\|home}` | +| `StaleStore` | SQLite `(dev, ino)` differs from open; PG generation/fence mismatch (§10, §5.2) | unspecified | +| `Capacity` | a real `PersistenceError::Refused {status: "Capacity"}` (§10) | unspecified | +| `Unsupported` | default catalog body; PG transaction pooling for mounts/compaction (§10, §5.2) | unspecified | +| `ReservedName` | `branch.remove` of `main` (§4) | — | +| `Conflict` | sealed tuple changed at install; missing `project_epoch` row after cleanup (§10, §5) | unspecified | +| `NotFound`, `NoSuchBranch`, `NoSuchCommit`, `NoSuchProject`, `NoSuchStore` | a hidden/collected resource, resolved through filtered statements (§4.2, §5) | — | +| `InvalidInput` | bad `remove_file` path, retired `project.remove`, symlink component (§4.3, §4) | — | +| `Exists` | a stray artifact at the staged compact name with no record (§9) | — | + +`InUse` stays for non-GC calls only (§10). **snake_case rule:** success results and refusal details are snake_case; the +dispatcher snakes the detail (or the site builds it snake); a camelCase detail is a bug; the CLI prints the rendered map; §11 +names are logical, not rendered (§10, §11). A busy checkpoint, a live owner and a reachable-but-unresponsive Sandbox stay +`Busy`; only a provably dead, uncertain op is `Unresolved` (§5.1, §7). + +## 4. MCP, folder scope, CLI/SDK, `store.status` + +- All four are **admin-profile** calls, refused under a folder scope with `out of scope: ` + (`api/crates/layerfs-api/src/calls/scope.rs:33`; #42 item 39): a folder scope guards one folder and one Branch, and none of + the four is a folder operation, so it is refused, never widened (§10). +- The admin classifier checks its admin list before the work families; the four join it and `project.remove` leaves both the + call table and that list. MCP names follow the existing `project_remove` mapping (`project.delete` → `project_delete`); an + old `project_remove` is `InvalidParams "unknown tool"`, a CLI usage error exits 2, the library reports `InvalidInput` (§10). +- The table becomes 76 (38 work / 38 admin): regenerate `gen-profiles`/`gen-coverage` and the hard-coded 73/38/35 counts; + `whyOf`, the profile prose and `--profile admin` usage name the four calls, one `whyOf` line each; a precedence test covers + `branch.remove` under `work` (§10). +- CLI/SDK surface: one command/one SDK method per call, ids only — `layerfs branch remove `, `layerfs + project delete `, `layerfs project remove-file `, `layerfs store compact ` (the runbook + already uses the last); results are the rendered snake_case map (§1, §10). +- `store.status` while compacting (§11): `op = {id, name, phase, started, uncertain}` from `gc_op` (PostgreSQL, Store-side) or + the home op (SQLite; a remote home via `step_on`); `state` is `busy`/`unresolved`; phases `mark`/`build`/`verify`/`install` + (SQLite) and `mark`/`build`/`swap`/`cleanup` (PostgreSQL); there is no `PROGRESS` option (#42 item 40). + +## 5. Implementation checklist (T11) + +1. Call table: add the four, delete `project.remove`; ids-only arguments (§1, §10). +2. `HistoryError`: add `Removed` and `StaleStore` with explicit arms in `lookup.rs` before `_ => "Failed"`; `NameTaken` rides + the boxed `NameTakenState` and `lookup.rs` projects `{holder, source}`; map `HistoryError::Unsupported` explicitly (§4.4, + §10). +3. `HistoryCatalog`: `mark_*`, `retained_roots`, `get_retained_roots`, `remove_unreachable`; both providers implement them, + default `Unsupported` (§10). +4. Dispatcher snakes every detail and result; builders carry §2's fields and §11's report with `estimate`, `{value, cap}`, + `zeroReason` (§10, §11). +5. `remove_file`: normalize the absolute POSIX path (strip one leading `/`); reject empty, `/`, `//a`, `a//b`, `a/`, `./a`, + `a/./b`, `.`/`..`, `\`, NUL, >4096 bytes, >256 components; subtree match only at a `/` boundary over bytes; frontier + `next_history_serial - 1` read under the Store write lane in the marker transaction; `epoch` bump only on advance; per-call + trace redaction with `trace_salt` (§4.3). +6. `project.delete`: worker frame, legacy identity save, sidecar `pending|marked|done`, retry order, `home_entry_removed` echo + (§4); a `project.init` name collision is `NameTaken` (§10). +7. Capacity: add `StorageError::CapacityExceeded` to the predicate, keep PG 53100/53200/53400/54000, map 53300 to `Busy`, drop + 54001, name `SQLITE_FULL`/`SQLITE_TOOBIG`/`SQLITE_NOMEM`, explicit arm before `_ => Refused` (§10). +8. MCP/profile generators, `whyOf`, counts, precedence test, folder-scope refusal (§4). +9. `store.status` op projection and phases; `gc_op` (PG) / home op (SQLite) (§11). + +## 6. Spec gaps and questions + +1. Only `Removed`, `MountsActive`, `Busy`, `Unresolved` and `NameTaken` have a stated `detail`; `StaleStore`, `Capacity`, + `Unsupported`, `Conflict` and `ReservedName` have none, and §10's "one detail shape" sentence covers only `Removed`. +2. `Unknown` is prose ("a lost reply is `Unknown`", §4; `project.list(store)` propagates `Busy`/`Unknown`, §4.2) but absent + from §10's code table and has no detail shape. +3. §11 names fields camelCase (`deadBytesBound`) while §10 requires snake_case rendering; the mapping is implicit. +4. §4.3 refuses a merge `Removed` with `detail {paths: [...]}`, while the canonical detail is `{paths: [{path, + frontier_serial}]}`; whether the merge echoes full elements is unstated. +5. Only `project.delete` has an idempotent retry contract (§4). For `branch.remove`/`remove_file` a lost reply "stays + uncertain" (§1) with no re-answer specified. +6. `project.delete` returns `Removed` with `home_entry_removed: false`, where another home's stale entry may persist (hidden + only by the §4.2 intersection); the result does not name that residue. + +Note: the brief's main ref `4e65ebba…` (40 hex) is not a LayerFS ref; `branch get main` reports Layer `3216923362f73a85…`, +root `27872fc7…`; T11 forked `main` as resolved. diff --git a/core/docs/issues/project-41/gc-rollout.md b/core/docs/issues/project-41/gc-rollout.md new file mode 100644 index 0000000000..5765c7f3c9 --- /dev/null +++ b/core/docs/issues/project-41/gc-rollout.md @@ -0,0 +1,119 @@ +# GC rollout, rollback and dry run (T1.6) + +> Plan for the frozen `LayerFS-GC-spec.md` (2.0.0; the spec wins, `§N` cites it). This page adds no contract and no option: the four calls take only a resource +> identifier, with no confirm, force or dry run (§1), and are admin-profile only, refused under a folder scope (§10). + +## 1. Rollout order + +| Step | Content | Contract | +|---|---|---| +| T1.x | Additive schema: marker/intent tables, position state, `gc_op`/`gc_dead`, `trace_salt` | §4.1 | +| T3/T4/T5 | Removals `branch.remove`, `project.delete`, `project.remove_file`: marker rows only, no bytes freed | §4 | +| T6 | Admission: `gc_intent`, `MountsActive`/`Busy`/`Unresolved`, epoch fence | §5 | +| T7 | Mark: reachability closure at `G`, `none`/`rebuild` from wholly dead packs | §3, §6 | +| T8 | SQLite rebuild: seal, `visited` scratch, verify, rename install | §7 | +| T9 | PostgreSQL rebuild: Phase M mark, Phase A build, Phase B swap, Phase C cleanup | §8 | +| T10/T11 | Report (§11 fields) and API surface | §9–§11 | +| T12 | Acceptance (§12 cases) and measurement (§13) | §12 | + +Order is forced: a rebuild copies T1.x's tables and cleans T3–T5's marker rows. T8/T9 are independent per backend; no step enables `store.compact` by itself. + +## 2. Expand and contract + +- **Expand (T1.x, T3–T5).** The schema step only adds tables and columns and one idempotent backfill under the current schema identity; no column changes type or meaning, + none is dropped, and `history_meta.next_history_serial` is never lowered (§4.1). Removals add marker rows and bump `project_epoch.epoch` (§4, §5). Both are additive: a + Store that never removed anything and never compacts is a plain 2.0 Store. +- **Old readers.** SQLite takes the step as one more `add_*`; PostgreSQL's open check requires the exact table set and recorded definition, so the step rewrites that + definition for older Stores (§4.1). An older PG binary that predates the step refuses the upgraded Store, and the spec defines no down-migration: **downgrade is not + supported**. Roll forward, or restore a pre-upgrade backup. +- **Contract.** Two API contractions are version cuts, not toggles: `project.remove` leaves the call table (an old client gets `InvalidInput`, never a silent hard delete) + and `project.init` returns `NameTaken` instead of `Exists` (§4, §10). Both land with T1.x and are asserted in the error matrix. +- **Flags.** None are added. `store.compact` is opt-in by the caller invoking it: no automatic collection, no online sweep (§1), and the spec defines no global + enable/disable switch. Do not invent one; the only "enable" is the release cut plus §7, and an owner may simply not call it. + +## 3. Dry run + +`store.compact` has no dry-run option (§1). Two non-mutating paths exist and neither is a new call: + +1. **Copy rehearsal.** `backup.create`, `backup.restore` into a scratch Store/home, run `store.compact` against the *scratch* name, compare. §9 names `backup.create` + before compaction as the recovery path for a mark bug; a copy is the closest honest rehearsal. +2. **`none` outcome.** Below 10% wholly dead packs the path is `none`: mark only, nothing written, generation unchanged, `deadBytesRefined == deadBytesBound` (§3, §11). + It cannot be requested, but on a Store expected to be `none` it is a real non-mutating run. + +Compare, before → after (both backends; §11): + +| Observation | `none` | `rebuild` | +|---|---|---| +| `store.status` | no op before/after; phase visible during | same; phases `mark`/`build`/`verify`/`install` (SQLite) or `mark`/`build`/`swap`/`cleanup` (PG) | +| `generation {before, after}` | equal, `G` | `G → G + 1` | +| `deadBytesBound` ≥ `deadBytesRefined` | equal | refined ≤ bound, from the payload sweep | +| `logicalDiscarded` | 0 with a `zeroReason` | collected history `logical` bytes | +| `keptWaste` | measured, unchanged by the rehearsal | `{deadInRetainedPacksBytes, deltaBases, inodeBlocksBytes}` | +| `store_totals` | unchanged | `logical` unchanged; `canonical`/`stored`/`metadata` = old − `dead_*` (§7 step 4, §8 step 9) | +| `residue` | empty | empty after settle; nonzero is explained | + +Also compare `physicalShrink` and `backendReusable` (§11). A rehearsal that disagrees with the live Store is a fixture difference, not a different path: both obey the +same §3 rule. + +## 4. Rollback + +- **Schema is additive.** The T1.x step is the only schema change and re-running it is a no-op; there is no down-migration. Do not un-create the tables by hand: without + `gc_intent`/`gc_op` the Store cannot admit or recover an operation, and PostgreSQL's open check (exact table set, recorded definition) refuses it (§4.1). +- **Markers are not undone by dropping tables.** Dropping `branch_retired`, `layer_stack_retired` or `path_removed` would resurrect names and paths whose history a + rebuild may already have collected (a deleted project's rows are dropped by the rebuild, §7/§8), and §1 promises no undo. The only supported re-creation is a Commit + **after** the frontier that writes the path again: a post-frontier re-creation is visible, and the marker hides only inherited content (§4.3). A Branch or project name + is re-created after a rebuild frees it; the history is gone (§4.2, §12 "Names"). +- **A failed compaction keeps `G`.** A definite failure before the rename aborts and deletes the artifact set + `.compact{,-journal,-wal,-shm,-visited,-visited-journal,-mj*}` (SQLite) or the recorded `lfs_gc_` scratch schema (PostgreSQL); the old Store is unchanged + apart from step 1's checkpoint (§7 step 8, §8, §9). Keep `G` and re-run later. +- **Installed-but-uncertain.** A failure of the rename/fsync itself, or a lost `COMMIT` reply, is not pre-install: keep the new Store at `G + 1`, settle the op `unknown` + and report it; never delete the artifact as if nothing happened (§7 step 8, §8). On PostgreSQL resolve a lost reply by proving the backend ended (`pid`, + `backend_start`, `usename`), then read the incarnation (§8). +- **Artifacts belong to the next `store.compact`.** Only a compaction that names them from its op removes them; a staged name no record names is `Exists` and is not + deleted (§9). Leave crash leftovers alone by hand. + +## 5. Verification gates + +| Phase | Tests | `store.status` / report | Doctor | +|---|---|---|---| +| T1.x | converted 1.0/2.0 Stores open; backfill (`history_serial`, `project_epoch`, `trace_salt`, never-lowered serial); `tools/store_blocks.py` | unchanged, no op row | full, before and after | +| T3–T5 | marker transaction, §4.2 visibility guard, admission races, whiteout on merge/mount/publish | no op left; `Removed`/`NoSuch*` shapes | full | +| T6 | intent lifetime, `machine_stopped`, `Unresolved` ownership, `holders` | `busy`/`unresolved` only while owned | full | +| T7 | closure vs reference walk, `live_heads_digest`, "no live object lost" (§12) | phase `mark`; path chosen | full on fixtures | +| T8 | §7 failpoints (mark/build/verify/install/WAL switch/artifact removal/op settle), exact artifact set, `StaleStore` | both paths' §11 fields; `physicalShrink`, `backendReusable` | `validate` before rename; full after | +| T9 | §8 failpoints (mark/build/swap/cleanup/scratch drop), `ACCESS EXCLUSIVE` duration, standby, reply loss, privileges | PG zeros (`backendReusable = 0`); resume from `gc_op` | full after cleanup | +| T10/T11 | §11 snake_case keys, `zeroReason` vocabulary, capped `cap`, scope refusals (§10) | phase and `op` projection | full | +| T12 | whole §12 matrix; independent `diagnostic/scan.rs` walk; 3 runs per arm (§13) | every field with unit and sampling point | full on fixtures | + +`quick_check` and the full doctor stay separate commands, not part of install (§7 step 6). A gate is green only on both backends, or on the rows §12 assigns. + +## 6. Failure handling + +- **Crash.** The Store is wholly old or wholly new (§9). The next `store.compact` judges the recorded `G`: at `G` the artifact is removed (not installed); at `G + 1` the + outcome is recorded and Phase C resumes. A PG crash in Phase M/A leaves only the scratch schema, which the op names; Phase B is atomic. +- **`Busy`.** A live mount, long-lived client or operation holds the scope. The call does not wait, retry or replay; prepare inactivity and call once later (§1, §5). +- **`Unresolved`.** A dead, uncertain owner's op. A non-owning call observes it; only the owning call (the next `store.compact`) adopts or clears it (§5, §10). An + unreachable machine reports `unknown`, never clean (§9). +- **Lost reply.** The outcome stays uncertain: read `store.status`, the op and the generation before deciding; never replay blind, and never delete a sidecar or an + artifact the op names. +- **Standby.** PostgreSQL waits only when `synchronous_commit` is at least `remote_write` and `synchronous_standby_names` is non-empty; the check runs before Phase M and + again inside the swap. A missing standby abandons with `Busy` (standby list); a standby dying after the check blocks the `COMMIT` unbounded (§8). SQLite has no such + gate. +- **`Capacity` / disk full.** Aborts, deletes the artifact, leaves the old Store unchanged apart from the checkpoint; `Capacity` is the real refusal path, not a codec + bound (§7). +- **`StaleStore`.** A connection that bypassed admission is caught at its next transaction; an in-flight transaction at install is a documented limit (§7, §10). + +## 7. Go/no-go: enabling `store.compact` on a shared Store + +1. `backup.create` succeeds and a `backup.restore` drill of this Store was done (§9). +2. `store.status` is clean: no `holders`, `state` not `busy`/`unresolved`, no live `gc_intent` row, no existing `.compact*` set or `lfs_gc_` schema. +3. All mounts, daemons and long-lived clients of the Store are gone, and no Init/merge row names it (§5); one owner knows nobody may start one during the window. +4. The backend's T8 or T9 gate and the §12 rows touching it are green; the full doctor is clean on a fixture of the same schema layout (§7 step 6, §12). +5. Disk headroom covers peak temp: old Store + new Store + journal/WAL + `visited` (SQLite); old tables + new heap/TOAST + indexes + live-bytes WAL and `pg_wal` headroom + (PG) (§7 cost, §8). +6. PostgreSQL only: `CREATE` on the database, `TEMP`, `USAGE`/`CREATE` on the Store schema, table ownership, `pg_read_all_stats`, else `Unsupported`; a direct or + session-pooled connection (transaction pooling is `Unsupported`); standby replication at the intended level (§5.2, §8). +7. The expected path is known: wholly dead bytes below 10% means `none` and no shrink — decide in advance whether the run is still wanted (§3). +8. The call is made under the MCP admin profile, not a folder scope (§10). +9. Rollback is understood: keep `G`, let only the next `store.compact` remove named artifacts, restore from backup if a mark bug is suspected (§9). +10. One attempt, no replay: a refusal from §6 is inspected before any second call (§1, §9). diff --git a/core/docs/issues/project-41/gc-runbook.md b/core/docs/issues/project-41/gc-runbook.md new file mode 100644 index 0000000000..b976397931 --- /dev/null +++ b/core/docs/issues/project-41/gc-runbook.md @@ -0,0 +1,114 @@ +# GC operator runbook: `store.compact` and the three removals + +> Operational page for the frozen `LayerFS-GC-spec.md` (the spec wins). Baseline +> `a6d84131b`; calls land in T3–T11. Admin-profile only, refused under a folder +> scope (§10). No confirm/force/dry-run; a lost reply stays uncertain, nothing retries itself (§1, §9). + +## 1. Triage: stuck compact or a phase in `store.status` + +`layerfs store status ` shows `holders`, `state` (`busy`/`unresolved`) and +`op = {id, name, phase, started, uncertain}` (§11). Phases — SQLite `mark`, `build`, +`verify`, `install`; PG `mark`, `build`, `swap`, `cleanup`. + +- **SQLite.** The op/phase is the *home op* (`store//record.json`, field `op`; + a remote home is read via `step_on`); the Store DB has the admission row: + `sqlite3 -readonly .sqlite "SELECT kind, scope_kind, state, uncertain, phase, generation, artifact FROM gc_intent WHERE state <> 'done';"` + Artifacts sit beside the Store file: `.compact{,-journal,-wal,-shm,-visited,-visited-journal,-mj*}`. + A direct reader may see `busy`/`locked` in the exclusive window; don't loop. Never unlink `-wal`/`-shm` (§7 step 1). +- **PostgreSQL.** `store.status` reads the Store-side `gc_op` (§11). On a direct or session-pooled `psql`: + + ```sql + SELECT generation, phase, scratch_schema, artifact, pid, backend_start, incarnation FROM gc_op; + SELECT (SELECT count(*) FROM gc_dead), (SELECT catalog_incarnation FROM history_meta); + SELECT kind, scope_kind, scope_id, state, uncertain, phase, artifact FROM gc_intent WHERE state <> 'done'; + SELECT nspname FROM pg_namespace WHERE nspname LIKE 'lfs_gc_%'; + ``` + + `gc_op` is the one compaction row; `gc_dead` is its persisted dead set (membership, + not a predicate); `gc_intent` is admission. `lfs_gc_` is recorded in `gc_op` + **before** `CREATE SCHEMA` (§8). Never INSERT/UPDATE/DELETE `gc_intent`, `gc_dead`, + marker tables or `project_epoch` by hand. + +## 2. `Unresolved`, live vs dead owner, lost reply + +`Unresolved` = a provably dead owner left an **uncertain** op; a non-owning call observes +it, the owning/next call settles it (§5.1, §10). A live owner is `Busy` — wait, don't kill it. + +1. Liveness comes from the recorded owner, not the clock: SQLite home op + `owner.pid`/`pid_start`/`pid_boot`; PG `gc_op.pid` + `backend_start` against `pg_stat_activity`. +2. Dead owner, uncertain op: run `layerfs store compact ` again; it adopts the dead + op with the artifact/high-water in the record (§9). `sandbox.close`/`machine.remove`/ + `store.remove` clear their own ops; `logs.reset` cannot (§10). A removed machine record alone is no proof (§5.1). +3. **PG lost `COMMIT` reply** — resolve on a *second* connection **before** the exclusive key + (§8). The swap recorded `pid`, `backend_start`, `usename` and the incarnation in + `gc_op` before the swap (`usename` is a `gc_op` column, §4.1). Prove the backend + ended, then read the generation: + + ```sql + SELECT pg_terminate_backend(a.pid, 30000) + FROM pg_stat_activity a + WHERE a.pid = AND a.backend_start = '' + AND a.usename = '' AND a.datname = current_database() + AND a.pid <> pg_backend_pid(); + ``` + + Check the boolean, poll until the `(pid, backend_start)` pair is gone, then read + `history_meta.catalog_incarnation`: `G` = not committed, `G+1` = installed. Needs PG 14+ + and `pg_signal_backend`/membership, else `Unsupported` (§8). Never terminate a live owner to clear `Busy`. + +## 3. Safe cleanup + +Default: **let the next `store.compact` judge it** (§9). The op names `G` and the artifact. +Store at `G` = not installed → next compact removes the named artifact set; `G+1` = +installed → it records the outcome. A dead owner's op is adopted; only a live owner blocks. + +- Delete by hand only when **no record names it**: no compaction `op`/`gc_op` row (PG: no + `gc_op.scratch_schema`/`artifact`), no live owner, no exclusive key. A stray at the staged + name with no record is refused `Exists` and not deleted by the code (§9); after the checks it is safe, otherwise leave it. +- Never remove `.compact-visited*` during a compact: it is the live visited set and part of `peakTempBytes` (§6.3, §11). +- **Never** delete a live Store's `gc_intent`/`gc_dead` by hand: admission and the Phase C + dead set are cleared only by the owning call on settle (§5.1, §8 Phase C). +- PG: drop `lfs_gc_` only when `gc_op` names no schema and no session holds the exclusive key; a named schema is the next compact's to remove. +- A definite pre-rename SQLite failure deletes its own artifact set; a rename/fsync failure is + **installed-but-uncertain** — keep the new Store, settle `unknown`, report it, never delete it (§7 step 8). + +## 4. Rollback and reversal (limits) + +- **Schema rollback is additive.** Marker/position/admission tables and the + `history_serial`/`trace_salt` columns are added at open; a downgrade leaves them (§4.1, + decision 6). No down migration; dropping them is not a rollback. The PG open check records + an exact table set (§4.1), so a pre-GC PG binary may refuse the added tables — forward-fix or restore instead. +- **Markers cannot be undone by dropping marker tables.** Dropping `branch_retired`/ + `layer_stack_retired`/`path_removed` resurrects retired rows without home entries, breaks + the "name never reused while rows exist" `BranchId::named` rule (§4.4), and cannot restore rows a rebuild already collected. +- **Supported reversal is re-creation after the frontier, never deleting markers** (§4.3): + a file is written/created at serial > `frontier_serial`; a Branch is created/forked once a + rebuild frees the name (new id/history); a project is `project.init`-ed once its retired stack row is collected (new empty project). +- **Limits.** Old ids/history/bytes never return; a Branch pinned by a retained Layer or + `commit.branch_id` keeps `NameTaken` (§4.4); `remove_file` reversal does not unhide + at-or-below versions; `rebuild` install is irreversible (§9). Content recovery is + `backup.create`/`backup.restore` from before compaction (SQLite; the server's backup for PG); + no undo copy is kept (§9, #155 D-24). +- `project.remove` is gone; an old client gets `InvalidInput` (§4, §10). + +## 5. Known gaps at this baseline + +- **PG existing-Store additive (T1.3b):** new PG Stores get the GC tables from `schema.sql`; + an existing 2.0 PG Store needs the additive step at open. Until then open/compact refuses; don't hand-create tables (§4.1). +- **Old-Store column ALTER (T1.2b):** new SQLite Stores get `commit.history_serial`, + `layer.history_serial`, `store_policy.trace_salt`; older Stores lack them until T1.2b's + ALTER + definition gate. GC refuses; don't hand-ALTER (§4.1). +- **`remove_file` is visibility-only in 2.0:** it hides reads/listings/mounts/merges/publishes + and frees no bytes; objects reachable only via the path are retained (§6.2, decisions 16/35). +- **No path-level physical reclaim:** path-aware walk and per-entry page pruning are follow-ups; `keptWaste` has no page component (§6.2). +- 1.0 Stores convert offline (§4.1); no online migration. + +## 6. First 10 minutes + +1. `layerfs store status `: read `state`, `holders`, `op.phase`, `op.uncertain`. Don't kill a live owner; `Busy` is expected. +2. `Unresolved`: prove the owner dead (home pid/start, or PG `pid`+`backend_start`), then + re-run `layerfs store compact ` to adopt. For a lost PG swap reply, use §2 on a second connection. +3. Before manual steps, back up (`backup.create` on SQLite; server backup on PG) and record generation `G`, op id, artifact name. +4. Compare artifact/schema with the op; leave anything a record names to the next compact (`G` removes, `G+1` records). +5. Never hand-edit `gc_intent`, `gc_dead`, markers or `project_epoch`; never unlink `-wal`/`-shm`; never drop a schema `gc_op` names. +6. To re-use a name after a removal, re-create after the frontier (or after a rebuild frees it); don't delete markers. Escalate if old content must be restored. diff --git a/core/docs/issues/project-41/gc-spec-gaps.md b/core/docs/issues/project-41/gc-spec-gaps.md new file mode 100644 index 0000000000..b4117e2ace --- /dev/null +++ b/core/docs/issues/project-41/gc-spec-gaps.md @@ -0,0 +1,96 @@ +# GC spec gaps: implementation decisions (T13b) + +> **Status:** implementation note for layerfs-project#41. `LayerFS-GC-spec.md` is **frozen** (2.0.0, +> owner-approved on #42): nothing here changes contract text. Where the spec is silent these are +> implementation decisions — the least-surprising option consistent with it. A "correction" entry is a +> proposed #42 amendment, not a change made here. + +**Baseline:** `main` at T13b; companion docs `gc-api-contract.md` (T11), `gc-acceptance-matrix.md` (T12), `gc-runbook.md`. + +## G1. Detail shapes missing for five codes (§10) +**Spec.** §10's table lists `StaleStore`, `Capacity`, `Unsupported`, `Conflict`, `ReservedName`, but states a +`detail` only for `MountsActive`, `Busy`, `Unresolved` (§10), `Removed` (§4.3, §10) and `NameTaken` (§4.4); +otherwise §10 requires only snake_case details. +**Decision.** All five carry an **empty detail** (`{}`). No field is invented: the spec names none, so no client may +depend on one, and `code` plus message are the surface. `ReservedName` (today only `branch.remove` of `main`, §4) +does not echo the Branch; `Conflict` (sealed tuple, §7/§10; missing `project_epoch`, §5) does not echo the failed +check; `Capacity`/`Unsupported` (§10) do not echo backend or catalog method; `StaleStore` (§10) does not echo +`(dev, ino)` or the generation. +**Impact.** T11: one empty-detail helper, explicit `HistoryError` arms before `_ => "Failed"`, never a `Failed` +mapping. T3/T4/T5: propagate `{}` unchanged, no `branch`/`project` field added at the site. T12: error-matrix rows +assert code, message and `detail == {}`, never an unnamed field. +**Correction.** None; additive fields later would need a #42 amendment. + +## G2. `Unknown` absent from §10's code table (§4, §4.2, §10) +**Spec.** §4: "a lost reply is `Unknown`, keeps the sidecar"; §4.2: `project.list(store)` propagates +`NoSuchStore`/`Busy`/`Unknown`. §10's table omits `Unknown`. +**Decision.** `Unknown` is a **service/control-plane outcome**, not a `HistoryError`/`lookup.rs` code: it exists for +the caller of a lost reply and for a Store listing that cannot be determined. It reuses the `Unresolved` detail +shape `{entity: {kind, name}, phase, op}` — same uncertainty, different observer (owning caller vs non-owning +call) — with `phase` and `op` `null` when no durable op is known. It joins neither `HistoryError` nor §10's table. +**Impact.** T4/T11: the `project.delete`/`store.compact` worker path keeps the sidecar and returns `Unknown`; T12 +"Worker/lost reply" asserts it and no cleanup; `project.list(store)` propagates it untouched. T3/T5 have no worker path. +**Correction.** Yes: a §10 row (or one §4 sentence) calling `Unknown` a service code with the `Unresolved` shape. + +## G3. §11 camelCase names vs §10 snake_case rendering (§10, §11) +**Spec.** §11 names `deadBytesBound`, `deadBytesRefined`, `logicalDiscarded`, `backendReusable`, `physicalShrink`, +`keptWaste.deadInRetainedPacksBytes`, `keptWaste.deltaBases`, `inodeBlocksBytes`, `peakTempBytes`, `zeroReason`; +§10 requires snake_case results/details and says "§11 names are logical, not rendered". +**Decision.** §11 names are Rust/logical only; every wire, CLI and SDK key is snake_case, mechanically: +`dead_bytes_bound`, `dead_bytes_refined`, `logical_discarded`, `backend_reusable`, `physical_shrink`, +`peak_temp_bytes`, `zero_reason`, `kept_waste {dead_in_retained_packs_bytes, delta_bases {bytes, references}, +inode_blocks_bytes}`. `kind`, `path`, `generation`, `before`, `after`, `residue`, `bytes`, `artifacts`, `reason`, +`value`, `cap`, `estimate`, `references` are single words. One serializer does it; no per-call map. +**Impact.** T10 owns the report struct (serde `rename_all = "snake_case"`); T11 renders verbatim; T12 asserts the +snake keys on both paths. T3/T4/T5 results are already snake_case (`removed_ms`, `deleted_ms`, `home_entry_removed`, +`frontier_serial`). +**Correction.** None needed; §11 already defers to §10. A one-line "rendered snake_case" note would remove the ambiguity. + +## G4. Merge `Removed` detail vs the canonical shape (§4.3, §10) +**Spec.** §4.3: "A merge that cannot materialize the whiteout is refused `Removed` with `detail {paths: [...]}`"; +the same section: "Every refusal carries one detail shape: `{paths: [{path, frontier_serial}]}`"; §10 repeats it. +**Decision.** One shape everywhere: `{paths: [{path, frontier_serial}]}`. The merge refusal lists every path it +cannot materialize as a full element (not a bare string), each with the governing maximum frontier over the path and +its `/`-boundary ancestors (§4.3); elements are sorted by normalized path bytes and deduplicated. read/stat emit one +element: the request path with its governing frontier. A directory element names the directory; descendants are +covered by the same frontier rule. +**Impact.** T5: the whiteout walk produces this list; T11: one detail serializer shared by read/stat/merge; T12: +"`remove_file` and derived nodes" and "Removed path in a merge" assert full elements and deterministic order. T3/T4 +never emit `Removed`. +**Correction.** No behavior change; §4.3's `[...]` is shorthand for §10's shape. A clarification amendment could say +"full §10 elements"; not made here. + +## G5. Idempotent retry only for `project.delete` (§1, §4, §4.4) +**Spec.** §4 gives `project.delete` a durable sidecar, marker-as-commit-point and an idempotent retry reusing +`deleted_ms`; §4.4 makes a repeat `branch.remove` `NoSuchBranch`; §1 says a lost reply stays uncertain and nothing +retries itself. +**Decision.** Keep the asymmetry. Only `project.delete` is retry-idempotent (sidecar `.../delete/.json`, +states `pending|marked|done`; the retry resolves retired/live/absent). `branch.remove` has no sidecar in 2.0: a lost +reply is `Unknown` (G2) and a repeat after success is `NoSuchBranch`; the API never turns that into `Removed`. +`remove_file` is re-callable but not replayable: a repeat is a new call that writes the current high-water as the +new frontier (§4.3), so it may hide newer content; it is `UpToDate` only when no project node is above this path's +marker frontier. No client, CLI, SDK or server auto-retries any of the three. +**Impact.** T3: repeat `NoSuchBranch`, no `deleted_ms` echo; T4: sole sidecar/retry path, plus `deleted_ms` reuse; +T5: repeat = new frontier/`UpToDate`, no stored reply; T11: no retry option/parameter, `Unknown` documented; T12: +retry case only for `project.delete`, the other repeats as above. +**Correction.** None; the asymmetry is intended (§4 is the single exception). + +## G6. `project.delete` `home_entry_removed:false` residue (§4, §4.2) +**Spec.** §4: the marker is the commit point; a home-removal failure returns success with `home_entry_removed:false` +and the stored `deleted_ms`; a different home's stale entry is not rewritten and is hidden by the §4.2 intersection. +**Decision.** Return exactly the §10 shape and add **no** residue field (that would be a contract change): +`{kind: "Removed", project, deleted_ms, home_entry_removed}`. The residue is another home's entry; it is named +nowhere in the result. The sidecar stays `marked` (durable evidence for a same-home retry); `project.list` omits the +entry and `project.get` answers `NoSuchProject`. A same-home retry finishes the entry removal and returns `true` with +the reused `deleted_ms`. Operator visibility is the sidecar and the call log, not the result. +**Impact.** T4: sidecar state machine and same-home retry; T11: no residue field; T12: "project.delete retry" and +"Home listing" cases (hidden, entry present, retry true). T3/T5 unaffected. +**Correction.** None; §4 already states this. A residue field in the result would be a #42 amendment. + +## Cross-cutting + +- Every refusal keeps one code and one stable message (§12 error matrix); these decisions add no code and no field. +- Details are built snake_case at the site or snaked once in the dispatcher (§10); an empty detail stays `{}` and is + never dropped or synthesized. +- These decisions bind T3, T4, T5, T11, T12 and the T10 report; the frozen spec remains the authority and wins on + disagreement. diff --git a/docs/architecture/api/sdk/branch.md b/docs/architecture/api/sdk/branch.md index e658f92160..480c6faae1 100644 --- a/docs/architecture/api/sdk/branch.md +++ b/docs/architecture/api/sdk/branch.md @@ -219,33 +219,19 @@ before the lane; only the final `merge_branch` publish holds it. ### branch.publish -Makes a Branch's head, or one of its Commits, `main`'s next Layer. By -default it refuses to drop work: if `main` gained a Layer since the Branch -forked, the publish is refused `HeadMoved` and nothing is written (#691). -Merge the Branch with `main` into a new Branch and publish that, or pass -`force` to overwrite `main` deliberately. +Makes a Branch's head, or one of its Commits, `main`'s next Layer. It +overwrites: Layers added to `main` since the Branch's base are reported, not +merged. ```text -JS await layerfs.branch.publish(project, branch, commit?, { expected_base, expected_head, force }) -Python layerfs.branch.publish(project, branch, commit=None, *, expected_base=None, expected_head=None, force=None) -CLI layerfs branch publish [] [--expected-base ] [--expected-head ] [--force] +JS await layerfs.branch.publish(project, branch, commit?) +Python layerfs.branch.publish(project, branch, commit=None) +CLI layerfs branch publish [] ``` `branch` must be a side Branch (`main` is `NoSuchBranch`); `commit` is a 66-hex Commit id (a Layer id is `InvalidInput`), default the head. -| Option | Kind | Rule | -| --- | --- | --- | -| `expected_base` | Layer id | `main`'s head must still be this Layer | -| `expected_head` | Commit id | the Branch's head must still be this Commit | -| `force` | bool | publish over Layers `main` gained since the fork; they are reported in `overwrote` | - -Without `force`, a publish goes ahead when `main` is still at the Branch's -fork point, when `main`'s head is this Branch's own earlier publication and -the Commit contains it (publishing again at milestones), or when `main` -already holds the Commit's tree (`NoChanges`). A Branch made by merging with -`main` has `main`'s head as its base, so it publishes as a fast-forward. - ```text Added: { kind: "Added", source: { branch, commit }, published_layer: { ...Layer, overwrote_count }, based_on, @@ -255,12 +241,9 @@ NoChanges: { kind: "NoChanges", head_layer } (its tree equals main's head) ``` The add is a compare-and-set on `main`'s head and the Branch's base: -`HeadMoved` when either moved, or when a check above refuses, never retried. -Its `detail` has `expectedBase`/`actualBase` (or `expectedHead`/`actualHead`). -Other errors: `NoCommit` (no saved Commit), `NotInHistory` (the Commit is not -on the Branch). After `Unknown`, publishing the same Commit again answers -`UpToDate` if it landed, even when `main` has moved on since or -`expected_base` names the head read before the first attempt. +`HeadMoved` when either moved, never retried. Other errors: `NoCommit` (no +saved Commit), `NotInHistory` (the Commit is not on the Branch). After +`Unknown`, publishing the same Commit again answers `UpToDate` if it landed. ### branch.diff diff --git a/docs/architecture/core/history/README.md b/docs/architecture/core/history/README.md index 36e9b5680b..f6f589e55a 100644 --- a/docs/architecture/core/history/README.md +++ b/docs/architecture/core/history/README.md @@ -75,7 +75,7 @@ or any process, memory, socket or stdout/stderr state. | Create a project | `initialize_layerstack` | [`layerstack.rs`](../../../../engine/crates/layerfs-persistence/src/history/layerstack.rs) | | Fork | `fork(ForkRequest)`: from `Layer(id)` (no head) or `Commit { branch, commit }`, which must be in that Branch's ancestry, else `NotInHistory` | [`branch.rs`](../../../../engine/crates/layerfs-persistence/src/history/branch.rs) | | Commit | `commit_branch(CommitRequest)` against the expected head and base | [`commit.rs`](../../../../engine/crates/layerfs-persistence/src/history/commit.rs) | -| Publish | `add_layer`: the Commit's root becomes the stack head (overwrite-only at this level; the SDK's `branch.publish` refuses a stale base first unless `force`); Layers it does not contain are counted in `overwrote_count` and paged by `overwritten` | [`layerstack.rs`](../../../../engine/crates/layerfs-persistence/src/history/layerstack.rs) | +| Publish | `add_layer`: the Commit's root becomes the stack head (overwrite-only); Layers it does not contain are counted in `overwrote_count` and paged by `overwritten` | [`layerstack.rs`](../../../../engine/crates/layerfs-persistence/src/history/layerstack.rs) | | Merge | `merge_branch`: a new Branch whose first state is a node (fast-forward) or a two-parent Commit | [`merge.rs`](../../../../engine/crates/layerfs-persistence/src/history/merge.rs) | | Remove | `remove_layerstack`: the one removal; packs, objects, file digests and the allocation scope stay | [`removal.rs`](../../../../engine/crates/layerfs-persistence/src/history/removal.rs) | @@ -112,7 +112,7 @@ database. `HistoryCatalogConfig` holds `binding_key` (1–128 bytes, hashed into | Kind | Methods | | --- | --- | -| Point reads | `layer_stack[_named]`, `branch[_named]`, `branch_snapshot`, `commit`, `layer`, `published_layer` (the Layer a Branch's Commit was published as) | +| Point reads | `layer_stack[_named]`, `branch[_named]`, `branch_snapshot`, `commit`, `layer` | | Paged reads | `layer_stacks`, `branches`, `commit_history`, `layer_history`, `overwritten` | | Ancestry | `merge_base`, `is_ancestor` | | Writes | `initialize_layerstack`, `fork`, `commit_branch`, `add_layer`, `merge_branch`, `remove_layerstack`, `reserve_inodes` | diff --git a/docs/architecture/engine/store-calls/README.md b/docs/architecture/engine/store-calls/README.md index ee2ca382f4..d7eecb0ed3 100644 --- a/docs/architecture/engine/store-calls/README.md +++ b/docs/architecture/engine/store-calls/README.md @@ -147,7 +147,7 @@ The Commit itself is the Workspace Commit; see [Commit](../commit/README.md). | --- | --- | --- | | `branch.fork` | One Branch row through the catalog; no tree copied, no object read. Source is `main`'s head Layer, a Branch, or a Commit/Layer id. `ReservedName`, `NameTaken`. | [`history/branch.rs`](../../../../engine/crates/layerfs-store-service/src/history/branch.rs) | | `branch.merge` | Creates a new Branch `name`; neither source changes. An ancestor side fast-forwards with no Save. Otherwise merge base (`NoMergeBase`), optional merge units, `merge_roots` with `/.lww` rules from `branch`, Save finish and bindings, all before the lane; then one `merge_branch` publish. A conflict drops the Save and answers `Conflict`. | same, [`history/units.rs`](../../../../engine/crates/layerfs-store-service/src/history/units.rs) | -| `branch.publish` | `add_layer` with expected stack head and Branch base: a compare-and-set, no retry. Before it, a publish that would skip a Layer `main` gained since the Branch's base is `HeadMoved` unless `force` (#691); `expectedBase`/`expectedHead` pin `main`'s head and the Branch head. A refused Commit already published answers `UpToDate` (one `published_layer` read, on the refusal path only). `Added` (first 50 overwritten Layers reported), `UpToDate`, `NoChanges`. | [`history/branch.rs`](../../../../engine/crates/layerfs-store-service/src/history/branch.rs) | +| `branch.publish` | `add_layer` with expected stack head and Branch base: a compare-and-set, no retry. `Added` (first 50 overwritten Layers reported), `UpToDate`, `NoChanges`; moved heads are `HeadMoved`. Publish overwrites; skipped Layers are reported, not merged. | [`history/branch.rs`](../../../../engine/crates/layerfs-store-service/src/history/branch.rs) | | `branch.list_commits`, `project.list_branches` | Newest first / byte-name order; catalog pages of `MAXIMUM_PAGE_RECORDS` (128). | same | | `branch.commits`, `branch.get_commit` | Saved history from a Branch-qualified pin (`resolve_context`): `main` follows publication Layers, a Branch its Commit parents; a Commit outside the Branch history is `NoSuchCommit`. | [`history/saved.rs`](../../../../engine/crates/layerfs-store-service/src/history/saved.rs), [`methods/branch/commits.rs`](../../../../engine/crates/layerfs-store-service/src/methods/branch/commits.rs) | diff --git a/docs/architecture/flows/merge.md b/docs/architecture/flows/merge.md index 2c7b7630ac..742ac5de29 100644 --- a/docs/architecture/flows/merge.md +++ b/docs/architecture/flows/merge.md @@ -60,7 +60,7 @@ never written. | 4 | `branch.merge` checks its arguments first, then fast-forwards when one side is an ancestor of the other. | [SDK: `branch.merge`](../api/sdk/branch.md#branchmerge), [Store calls: Branch history](../engine/store-calls/README.md#branch-history) | | 5 | Otherwise it finds the merge base, decides merge units, and builds the merged tree with `/.lww` rules in one Save. | [Compare and merge: merge](../core/compare-and-merge/merge.md), [Compare and merge](../core/compare-and-merge/README.md) | | 6 | A conflict drops the Save and creates nothing; a clean merge writes the new Branch at a merge Commit in one transaction that checks both sides are unchanged. | [Store: history calls](../engine/store/README.md#history-calls), [History: operations](../core/history/README.md#operations) | -| 7 | `branch.publish` adds the Branch head (or a named Commit) as a new `main` Layer by compare-and-set on the stack head and Branch base. A publish that would skip a Layer `main` gained since the Branch's base is refused `HeadMoved`; a Branch merged with `main` has `main`'s head as its base and publishes as a fast-forward. Only `force` overwrites, and its skipped Layers are counted and listed, not merged. | [SDK: `branch.publish`](../api/sdk/branch.md#branchpublish), [Store calls: Branch history](../engine/store-calls/README.md#branch-history) | +| 7 | `branch.publish` adds the Branch head (or a named Commit) as a new `main` Layer by compare-and-set on the stack head and Branch base. Publication is overwrite-only: skipped Layers are counted and listed, not merged. | [SDK: `branch.publish`](../api/sdk/branch.md#branchpublish), [Store calls: Branch history](../engine/store-calls/README.md#branch-history) | ## Related diff --git a/docs/architecture/glossary.md b/docs/architecture/glossary.md index f3a52b14a4..8654b30dc1 100644 --- a/docs/architecture/glossary.md +++ b/docs/architecture/glossary.md @@ -38,7 +38,7 @@ LayerFS (backend, install, scope), the row gives both. | Install | (1) Commit install: advancing a Workspace's base to the published root while rows written after the capture stay. (2) Store install: the one-time stream of a sealed Store file to a daemon. | [Commit: what it does](engine/commit/README.md#what-it-does), [Project Init: seal and install](engine/project-init/README.md#seal-and-install) | | Intent | The `op` field a multi-step operation writes into its entity's record before its first effect. A live owner makes other operations `Busy`; a gone owner's uncertain intent makes them `Unresolved`. | [Home: the `op` intent](control/home/README.md#the-op-intent) | | Kernel route | How the kernel reaches a Workspace: a FUSE session (Linux) or a loopback NFSv3 export (macOS). | [Kernel routes](engine/kernel-routes/README.md) | -| Layer | A published root on a project's main line. `branch.publish` adds one; it refuses to skip a Layer `main` gained since the Branch forked unless `force` overwrites. | [History: records](core/history/README.md#records) | +| Layer | A published root on a project's main line. `branch.publish` adds one; publication is overwrite-only. | [History: records](core/history/README.md#records) | | LayerStack | One project's linear timeline of Layers, with its Branches and Commits and one allocation scope. | [History: records](core/history/README.md#records) | | Lease | The file `store//open/` a process holds while it uses a Store, taken under the Store lock. Moves, removes and disconnects refuse `InUse` while a live lease exists. | [Home: locks and leases](control/home/README.md#locks-and-leases) | | Machine | A kernel that Sandboxes and sqlite Stores live on: the built-in `native` and `docker`, or created `lima` and `apple-container` machines. Every machine except `native` on macOS and Linux runs its sqlite Store calls in one worker. | [Machines](control/machines/README.md) | diff --git a/docs/general/parallel-worktrees.md b/docs/general/parallel-worktrees.md index 9d5947c619..f2fabb296a 100644 --- a/docs/general/parallel-worktrees.md +++ b/docs/general/parallel-worktrees.md @@ -209,16 +209,3 @@ record, so its target path must be updated first. Never scan or migrate another worktree's records. Keep an active cache warm and release the old cache through the cleanup procedure at handoff; retain records, sealed bundles and failed-run evidence. - -## After a fuser revision change - -After rebasing across #686 (fuser from its Git fork), or any later fuser `rev` -change, run `prepare` again for each platform before offline builds. A Cargo -home prepared earlier has no checkout of that revision and fails `--offline`. - -## Clippy on the Linux build image - -A Linux build image may lack the Clippy component (the stock -`rust:1.85.1-bookworm` image does), and `cargo` then refuses with -`'cargo-clippy' is not installed`. Run Clippy through `--platform macos`, and -keep Linux for the tests that need a Linux Store or mount. diff --git a/docs/research/README.md b/docs/research/README.md index cccc97bd39..afca002944 100644 --- a/docs/research/README.md +++ b/docs/research/README.md @@ -16,7 +16,6 @@ here is supporting context. - [AI-native filesystem vision](vision/README.md) - [Historical V2 replacement work](history/v2-replacement/README.md) - [Historical V3 engineering work](history/v3/README.md) -- [Agent guide history](history/agents-guides/README.md): status text moved out of the agent guides (2026-11-16) Historical paths are retained for design context and source-seal interpretation; they do not define the current product or roadmap. diff --git a/docs/research/history/agents-guides/README.md b/docs/research/history/agents-guides/README.md deleted file mode 100644 index f17807bee6..0000000000 --- a/docs/research/history/agents-guides/README.md +++ /dev/null @@ -1,303 +0,0 @@ -# Agent guide history - -> **Status:** Archived; retained for historical evidence only. - -Text moved word for word out of the agent guides when they were reorganized -on 2026-11-16, so the guides keep instructions only. It records how the -engine reached its current shape; it is not a current rule or a product -contract. Current behaviour is in the -[architecture guide](../../../architecture/README.md), and the rules are in -the [root](../../../../AGENTS.md) and [engine](../../../../engine/AGENTS.md) -guides. Only relative links were rewritten for the new location. - -A second pass on 2026-11-16 condensed the guides' wording without moving more -text here; the guides as they stood before it are at commit -[`84743424f1`](https://github.com/Ephemeral-AI-Lab/layerfs/tree/84743424f1/AGENTS.md), -and before any reorganization at -[`99ffd262a3`](https://github.com/Ephemeral-AI-Lab/layerfs/tree/99ffd262a3/AGENTS.md). - -## Engine scope and ownership: implementation status (S6 to R9c) - -From `engine/AGENTS.md`, "Scope and ownership", as of `layerfs_v2_0_0` at -`02188f0593`. - -- Active cluster-one members are content, storage, history, persistence, project - Init and telemetry. The initial cluster-two SQLite engine is `layerfs-overlay`; - its active membership does not complete Workspace/FUSE/daemon integration. - [Cargo.toml](../../../../Cargo.toml) is the membership authority. - Workspace now exposes immutable base binding/read, effective source-qualified - lookup/stat/paged name merge, prepared actor install and atomic ordinary - namespace operations, bounded writes/truncate/regrow and composed payload - reads. Independent file/captured/operation custody and bounded live reclamation - are implemented through S6, including indexed automatic reclamation (its - physical reservation and headroom were removed by #321: no space is reserved - ahead of use, and a full device rolls the write back as no space); - native/runtime/integrated exits remain - unfinished. See [S6 audit](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/S6-EXIT-AUDIT.md). Its relocated - predecessor `layerfs-workspace-legacy` was removed by R7-retire. - Since #321 the daemon holds one overlay file and connection per mounted - Workspace, each behind its own lock, with the calling thread running each - short job; the fair SQL owner thread, its queue, credits and shared memory - budget are removed. See the [terminology guide](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/64-overlay-terminology.md) - and the [#321 record](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/321/CHECKLIST.md). - Daemon also exposes a direct global Store adapter with a - mount-time reader per Workspace, exact quarantine and read-only bind snapshots, - one immutable cache shared across Workspaces, bounded root - binding and operation-owned exact failures. Concrete opening stays outside - the provider-independent adapter; see the - [direct Store boundary](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/61-direct-store-adapter.md). - The host-mediated SDK/upstream transport is retired; see - [the retirement record](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/PRE-S8-F12-20261007.md). - Workspace calls return typed values directly and count SQL/payload/allocation - work under the same lock. Exact attempted failures retain their cause and - backing; before-effect refusals retain their original input. Overlay/Daemon - also expose bounded indexed operation records in the same database, with guarded atomic changes and automatic last-owner - cleanup. Content now exposes a backed file editor over these records and - Workspace supplies an explicit operation/file adapter with first-original - failure custody. Run-aware localized edits reuse the frozen sparse scanner - and bounded positive immutable zero evidence. Streamed directory inputs share - the canonical filesystem algorithm; backed serial state supplies new-parent - membership, rebuilt roots and initial counts/final row streams. On the backed - route validation classifies in 64-row windows and keeps its topology evidence - in indexed records; resident routes keep their resource-derived limits. - Captured local inode/run points retain exact reader root/floor and bounded - forward metadata progress. Captured regular-file normalization now derives - authenticated base facts, coalesces final changes into indexed records and - consumes them once through the retained FileView/fallible edit source. Backed - reference reduction/release uses fixed rows, sealed membership passes and - guarded FIFO/cursor transitions. Captured namespace normalization is the - `CapturedNamespace` producer - ([captured namespace construction](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/78-captured-namespace-construction.md)); - the product Commit constructor calls it - ([product Commit](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/79-product-commit.md)). The Store - half of Commit is implemented with exact original failure/unknown custody; - see [Store Commit](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/65-store-commit-composition.md). - The external E01 example records original startup work, diagnostics and Stop - without creating a Workspace route. It supplies diagnostic receipt consistency, - not E1/E2 performance admission. Its relocated predecessor was removed by - R7-retire; standalone native executable/Exec integration is unfinished. - Pre-S8 direct-operation accounting now has actual registered macOS/Linux engine - and Store-half Commit runs, independent family/count validation, and supported - phase observations. Numerical latency/residency and exact continuous peaks - remain unqualified; see the [accounting scope](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/PRE-S8-F14-20261007.md). - SDK creates/imports the initial Store, publishes the first Branch and consumes - seal, returning a closed file plus explicit provider/locator/profile metadata. - It retains no Store handle or data service. Native install now streams the - sealed file once, opens it in the daemon and returns both actual SQLite versions; - see [F5 handoff](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/PRE-S8-F5-20261007.md). Authenticated mount - binding, Store-half Commit, status, terminal close, fork and history now compose - the direct ports; see [native control](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/68-native-workspace-control.md). - The prior client/runtime, daemon upstream, Bridge logical data framing and - excluded API-core are retired. Historical receipts keep their original topology - and verdicts. R7-retire (2026-10-10) - removed all seven excluded directories as retirement: `layerfs-server`, - `layerfs-fuse-legacy`, `layerfs-sandbox-legacy`, `layerfs-sdk-legacy`, - `layerfs-daemon-legacy`, `layerfs-bridge-legacy` and - `layerfs-workspace-legacy`. The root workspace lists the active - members across core, engine and api; the boundary guard refuses the return of a removed directory or - package name, and each source is recoverable from the commit named in the - [R7-retire record](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/R7-RETIRE-COMPLETION-20261010.md). - Pinned read-only SDK views (303/08 O-10) are deferred, not dropped: no - implementation is in the tree, and a kept feature needs a new design on the - SQLite overlay. Do not restore the predecessor code to supply them. - Project native import preserves opaque symlinks, complete membership and hard - links, with indexed acquisition state. The accepted host WAL baseline retains - its recorded speed FAIL; no new timing is implied by SDK composition. - Bridge retains pinned authenticated KK channels, bounded native records and - independent fences and bounded native install/control. FUSE/Exec qualification - and complete kernel mount readiness remain S8. - The replacement Sandbox is an active member with ordinary Engine execution, - owned lifecycle and authenticated SDK/daemon startup. Optional privileged - command cancellation/client provenance is outside the current owner scope. - The replacement Fuse is now active with a fixed shared dispatcher, actual - deferred Owner/Store ports, a read-path callback adapter with complete opcode - accounting, direct mount/session ownership and connection drain. - Its predecessor `layerfs-fuse-legacy` was removed by R7-retire; the last - commit holding that source is `15e33e22c`. The daemon - composes Attach/Locate/Ready and reversible normal unmount; see - [native request service](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/75-native-request-service.md) and - [native mount session](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/76-native-mount-session.md). - Ordinary native mutation is served through the existing Workspace plan and - one publishing Overlay job, with no notification and no writeback cache; see - [native mutation and kernel coherence](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/77-native-mutation-coherence.md). - A control Commit runs the captured namespace producer through the unchanged - driver, and takes the Workspace's overlay lock for each of its jobs; the mounted - proofs and their limits are in the - [R5 record](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/R5-COMPLETION-20261009.md). Forced unmount of one - Workspace is implemented as `ForceUnmount`: a before-effect guard, one abort - write on the connection's own control, the terminal fence, the local drain, - one plain detach, then Revoke and Close; see - [forced teardown](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/80-forced-teardown.md). It is refused where - no abort control is bound, which includes the current Sandbox topology, and a - stop after an effect is `Retained` with no way back. No daemon-wide graceful - drain exists and none is planned. Source acquisitions have their own owner - slots, and a failed cold read ends its own request with `EIO` while the - mount keeps serving; the concurrency, teardown and failure-scope proofs and - their limits are in the [R6 record](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/R6-COMPLETION-20261009.md). - A create that leaves fewer unconsumed inode serials than the daemon's - configured low-water makes one early reservation; every deployment and - harness sets 0 today, which makes none. A new Mount is refused `Capacity` at - `mount:debt` while maintenance is stopped. Both, the proofs added around - them and the rows still open are in the - [R8b record](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/R8B-COMPLETION-20261010.md). R8b closed not - qualified and R9b without removal. R8c then closed qualified on function, - counts and resources at the owner-rescoped scope, with the deployed - low-water at 256 ([R8c record](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/R8C-COMPLETION-20261010.md)), - and R9c retired the root reference - ([R9c record](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/R9C-COMPLETION-20261010.md)). - Add members only with real product boundaries and implementation. - -## Overlay schema history (schemas 17 to 37) - -From `engine/AGENTS.md`, "Architecture and public contracts". - -Overlay schema19 uses `directory_entry` and `operation_record` terminology, -including owned/indexed variants, throughout SQL and the active public ports. -Through schema 33 Workspaces shared physical tables partitioned by namespace; -since #321 (schema 34) every mounted Workspace has its own file with the same -tables, copied from a closed empty template at mount and deleted at terminal -unmount. The naming -change authorizes no physical table redesign; generic temporary buffers and -historical receipts retain their distinct terms. See the -[terminology guide](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/64-overlay-terminology.md). -The #320 product sweep (2026-10-14, schema 37) removes accounting triggers, -Workspace/allocator SQL rows and generic caller-minted leases. Workspace -lifecycle/frontier/dirty/base-reader counters are one journaled runtime value, -restored only after known whole-job rollback; unknown outcomes retain custody. -Commit runs/readers/base pins are journaled owner maps; captured generation -holds use a count index. Daemon-bound routing is checked once per call; Overlay -accepts no Route and stores no namespace. Cleanup debt remains SQL-backed after -the [memory-placement review](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/320/CLEANUP-MEMORY-DESIGN-20261014.md). -Resources computes row/length totals on demand and -retains exact owner counts from memory. See the current -[runtime state](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/64-overlay-terminology.md#runtime-workspace-state-and-remembered-sql-custody). - -Schema17 adds native mount/source/lookup/read associations, atomic Workspace -read observations and bounded revoked-lookup retirement. Schema18 adds native -file associations over the existing OpenFile owner and atomic open decisions; see -[native read custody](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/73-native-read-custody.md). This component -does not establish Ready, kernel service or full drain. Schema19 adds native -directory ownership, exact name-boundary cookies, retained parent updates and -bounded live cleanup; see [directory custody](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/74-native-directory-custody.md). - -## Source ownership review, 2026-10-08 (R2 to R5) - -From `engine/AGENTS.md`, "Filesystem access and execution ownership". - -Reviewed target source ownership2026-10-08: the replacement `layerfs-fuse` owns -its complete native connection/request service, including bounded dispatch, -parking/resumption, kernel handlers/replies and coherence. Daemon assembles one -shared Fuse service with its existing SQL/Store services, retaining registry, -overall Ready/unmount and Commit composition. Dependency is daemon -> fuse -> -workspace, never Fuse -> daemon. Request dispatch/read ports, mount/Ready and -per-Workspace normal drain are implemented ([R2 record](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/R2-COMPLETION-20261008.md)), -as is ordinary native mutation ([R3 record](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/R3-COMPLETION-20261008.md)); -as is the captured namespace producer with its bounded validation -([R4 record](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/R4-COMPLETION-20261008.md)), which the Commit -driver reaches through the product constructor -([product Commit](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/79-product-commit.md)). See the -[reviewed ownership](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/R2-R5-SOURCE-OWNERSHIP-REVIEW-20261008.md). -Moving blocking OwnerClient calls behind a port does not establish deferred service. - -## Root guide: Store routes as first written - -From `AGENTS.md`, "LayerFS mental model", as of `layerfs_v2_0_0` at -`7a6da3e4f8`. The current rules are in the root guide's "Where calls run". - -- Target (owner direction 2026-10-07, Store calls revised 2026-10-18): every - daemon opens the global Store directly from a volume the daemons share and - reads, saves and publishes in-process. Project Init adds a project to an - existing Store as a live add (2.0, #533). **sqlite Store calls run in a - process on the Store's kernel; postgresql Store calls, and bucket (s3) I/O, - run in the caller's process on any OS (owner, 2026-11-02, #533 G2).** A - sqlite Store's openers share one kernel and a local filesystem holding the - Store (ext4, xfs, btrfs on Linux; apfs, hfs on macOS); a process on another - kernel reaches it only through that machine's worker. Anywhere else sqlite - Store calls fail `Unsupported`, with no fallback; the daemon does not serve - Store calls. Workspace calls (mount, Exec, Commit, - status, unmount) go to the daemon. Do not revive `layerfs-server` or build a - host/server adapter in the data path; the host-mediated SDK runtime is - retired, not extended. See the - [integration contract](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/303/06-cluster-one-integration.md) - and [direct Store access](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/352/DIRECT_STORE.md). - **Exception (owner, 2026-10-20, revised the same day): dsh Live on macOS - only.** The dsh-layerfs plugin may give macOS processes a ported SDK with - the SDK's API whose calls run in one long-lived worker per LayerFS box: a - Linux container running the real SDK, which opens each project's Store - directly and keeps it and its daemon connections open while the project is - in use (closed after a quiet minute). The worker listens on loopback only, - behind a secret. The product SDK and daemon do not change. (Widened to any - integration on 2026-10-24; see below.) -- **Additional exception (owner, 2026-10-22): dsh Live on Windows.** The - Windows dsh-layerfs adapter may forward its SDK binding calls to one - authenticated loopback worker per LayerFS home in a selected WSL2 Linux - distribution. The worker opens the shared Store directly on that - distribution's local ext4/xfs/btrfs filesystem and uses its Docker Engine - for Sandbox daemons. Windows owns the UI, bindings and traces; Linux owns - native handles and project discovery. This is independent of the macOS - adapter and changes neither the product SDK nor the daemon. A lost reply - is never replayed, and an uncertain Init retains its Linux home and volume. -- **Additional exception (owner, 2026-11-01, #527 D5; renamed 2026-11-02, - #533 G2): the `native` machine on macOS.** For a Store on the `native` - machine on macOS, the library runs in the caller's own process and opens the project's Store directly on a local apfs or hfs - directory (`kernel_guard` admits those two there, #427). Its Sandboxes are - daemon processes serving loopback NFS to this Mac's own client, one user, - no container (see the agent guide's native Sandboxes). Every opener is a - process on this one Mac, so the one-kernel, local-filesystem rule holds; - network and other filesystems are still refused. The Docker, worker and - `vm:` routes on macOS are unchanged. -- **Plugin-neutral LayerFS (owner, 2026-10-24; revised 2026-11-02, #533 - G2).** LayerFS is agnostic to dsh and every other plugin. The `layerfs` - package routes by the home's records: a call's route comes from its - Store's and machine's records, so one process uses Stores on several - machines at once, and the worker routes above are open to any caller of - the package. LayerFS reads `LAYERFS_HOME` and the trace settings only. - There is no adapter name, no integration check and no code that names a - plugin. An integration that can set only its own prefixed names, such as - dsh's `DSH_*`, maps them itself. The worker's rules above (loopback, - secret, one attempt, no replay) are unchanged. -- **Machines (owner 2026-11-02, #533 G2; replaces the 2026-10-31 VM - placement, V1).** A VM machine holds many Stores, with one worker per - machine. Machines are created, removed and started by the `machine` calls, - and the routes come from the home's records (`machine//record.json`, - `store//record.json`). `native` and `docker` are built in; VM kinds - (`apple-container`, `lima`) are created by name. Each VM - machine's data disk is mounted at - `/layerfs/data` and holds `store//store.sqlite` per Store. Its - worker runs as root behind its own secret, reached through one loopback - port on the host, with one attempt per call and no replay, the same rules - as the macOS and Windows routes. Guests never read the home: the caller - sends each worker call the settings it needs and writes every record. - **Every machine uses both Store kinds (owner, 2026-11-03, #535 D9, G1, - G3):** a machine holds any number of sqlite Stores, and every PostgreSQL - Store is usable from it. A sqlite Store is on exactly one machine; a - PostgreSQL Store's calls still run in the caller's process, and it is - usable from the machines it is attached to (`store.attach`, each checked - from that machine; supersedes #535 G3), where the support table - (`layerfs-platform` `support.rs`, `control/SUPPORT.json`) marks the - platform supported; elsewhere it is refused with the table's reason. The password is resolved only in the - caller and travels only in one worker request and the Sandbox's private - daemon setup file. This supersedes #533 O3/G13 ("VMs and ssh later"). - See [#533](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/533/README.md), [#535](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/535/README.md) - and the [proposal](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/proposals/host-platform-connections.md). -- **Removed platforms (owner, 2026-11-08; supersedes the layerfs-project#7 - restriction of 2026-11-06): Firecracker, Microsandbox and SSH machines are - removed.** Their code is gone and no platform is disabled with its code - kept. An old record of one is shown as unsupported with the reason - (``platform `` was removed in LayerFS 2.0``), every other call on it - is refused, and `machine remove` only forgets it. No runtime setting or - feature flag brings one back; that requires a later explicit owner - decision and source change. Historical evidence is unchanged. Native, - Docker, Lima and Apple Container retain their existing support rules. See - the [restriction contract](../../../../docs/architecture/control/machines/platforms.md#removed-platforms-platform_policyrs). - -## Root guide: rebase notes - -From the end of `AGENTS.md`; the steps now live in the -[parallel worktree guide](../../../general/parallel-worktrees.md). - -After rebasing across #463, update each selected worktree's own target registration and prepare again -for the root `target/` path; never migrate another worktree's resource record. -After rebasing across #686 (fuser from its Git fork), or any later fuser `rev` change, run -`tools/worktree.py prepare` again for each platform before offline builds; a Cargo -home prepared earlier has no checkout of that revision and fails `--offline`. diff --git a/docs/versioned/2.0.0/calls.md b/docs/versioned/2.0.0/calls.md index 8989091833..9825815e42 100644 --- a/docs/versioned/2.0.0/calls.md +++ b/docs/versioned/2.0.0/calls.md @@ -138,18 +138,14 @@ folder is in place. Events arrive at most every 200 ms. | `get` | `project`, `branch` | — | one Branch and its head Commit | | `fork` | `project`, `name`, `from_branch`, `[from_commit]` | — | a new Branch from another Branch's head or Commit | | `merge` | `project`, `name`, `a`, `b` | `message`, `units` (JSON) | merges two Branches into a new one | -| `publish` | `project`, `branch`, `[commit]` | `expected_base`, `expected_head`, `force` | a Branch's head or Commit as `main`'s next Layer, refusing a stale base | +| `publish` | `project`, `branch`, `[commit]` | — | a Branch's head or Commit as `main`'s next Layer | | `diff` | `project`, `from_branch`, `to_branch` | `paths_only`, `page_size`, `page_pos`, `cursor` | files that differ between two Branches' heads | Run: `fork` answers `{kind: "Forked", branch, base, head}`; `publish` answers `{kind: "Added", source, published_layer, based_on, overwrote_count, overwrote, overwrote_has_more}`; `merge` answers `{kind: "Merged", fast_forward, branch, head, root, sources, merge_base, conflicts, lww, -units, …}`. `publish` is refused `HeadMoved` when `main` gained a Layer since -the Branch forked (merge with `main` into a new Branch and publish that), or -when `main` or the Branch head is not the `expected_base` or `expected_head` -given; `force` overwrites instead. A Commit already published answers -`UpToDate`, and a tree `main` already holds answers `NoChanges`. +units, …}`. ### `commit` (3) diff --git a/engine/AGENTS.md b/engine/AGENTS.md index 577f58ab6f..790bea2898 100644 --- a/engine/AGENTS.md +++ b/engine/AGENTS.md @@ -2,136 +2,297 @@ > **Status:** Current general guide. -Read [repository instructions](../AGENTS.md) first. This guide adds what is -specific to `engine/`: who owns what, the boundaries between crates, and how -to check a change. Current behaviour is in the -[architecture guide](../docs/architecture/README.md); how the engine got here -is in the [archived guide history](../docs/research/history/agents-guides/README.md). +Read [repository instructions](../AGENTS.md) first. This file adds implementation +boundaries for the engine layer under `engine/`; it does not repeat the +product mental model or campaign-specific benchmark procedures. ## Scope and ownership -| Crate | Owns | Architecture | -| --- | --- | --- | -| `layerfs-bridge` | the process facts the home and the Sandbox driver share (`process`), wire and native records, native install and control channels | [Bridge](../docs/architecture/engine/bridge/README.md) | -| `layerfs-daemon` | mounted Workspaces, their overlay connections and locks, mount/Ready/unmount, Commit composition, the direct Store adapter | [Daemon](../docs/architecture/engine/daemon/README.md) | -| `layerfs-fuse`, `layerfs-nfs`, `layerfs-mount-ports` | the kernel routes: FUSE and NFS as peers over the shared mount ports (#535 D3) | [Kernel routes](../docs/architecture/engine/kernel-routes/README.md) | -| `layerfs-overlay` | each Workspace's own overlay SQLite file | [Overlay](../docs/architecture/engine/overlay/README.md) | -| `layerfs-persistence` | global Store writes, SQLite and PostgreSQL (#535 D2) | [Store](../docs/architecture/engine/store/README.md) | -| `layerfs-project` | native import; Project Store Init and install (`layerfs-project::store`) | [Project Init](../docs/architecture/engine/project-init/README.md) | -| `layerfs-store-service` | the SDK Store calls (`src/methods/`) over `read`, `write` and `history` | [Store calls](../docs/architecture/engine/store-calls/README.md) | -| `layerfs-workspace` | the Workspace view and captured construction (`CapturedNamespace`) | [Workspace](../docs/architecture/engine/workspace/README.md), [Commit](../docs/architecture/engine/commit/README.md) | - -- **No engine crate depends on an API or control crate.** Core owns content, - history, storage and telemetry. Since #535 (owner, 2026-11-03, D1, D4) - Sandbox and network code live in the [control layer](../control/AGENTS.md) - (renamed from connect by layerfs-project#101, D-6): the Sandbox runtime - (`layerfs-sandbox-backend`), the machine layer (`layerfs-platform`) and - remote storage (`layerfs-bucket`, with `layerfs-store-service`'s `remote/`). -- **Store service methods** (#536 H3): `src/methods/` holds the method table - (`methods::call`), its argument reader and one file per `file.*`/`branch.*` - call, and is the facade over `read`, `write` and `history`, which know no - SDK method names. Their tests live in store-service's `tests/` - (`concurrent_commits`, `lane_merge` and `merge_report`, which need a daemon, - in `layerfs-daemon`'s). `layerfs-api` keeps its public re-exports of - Project Store Init and install, and its SDK-facing daemon tests include - daemon and Sandbox fixtures from this layer. -- **Retired code stays retired:** the seven `*-legacy` directories and - `layerfs-server` (R7-retire), the v0.1.6 reference and `core/reference-tests` - (R9c; see the [root guide](../AGENTS.md#retired-and-frozen)). The boundary - guard refuses their return. Do not restore predecessor code to supply a - feature: pinned read-only SDK views (303/08 O-10) are deferred, not dropped, - and need a new design on the SQLite overlay. Add a crate only with a real - product boundary and implementation. -- **Facts that shape every change:** each mounted Workspace has its own overlay - file and connection behind its own lock, and the calling thread runs each - short job; there is no SQL owner thread, queue, credit scheme or shared - memory budget (#321). There is no daemon-wide graceful drain, and none is - planned. Privileged command cancellation and client provenance are outside - the owner's scope. -- **Where to start:** cluster-one work with the - [handbook](../docs/general/cluster_one_handbook.md) and the - [CAS/CDC/delta guide](../docs/general/cas_cdc_deltaencoding_handbook.md); - cluster-two work with the engine topics it touches, the - [flows](../docs/architecture/flows/README.md) and +- Engine owns Bridge, daemon, FUSE, NFS, Overlay, Persistence, Project, + Persistence, Project, Store service and Workspace. Core owns content, + history, storage and telemetry. The library `layerfs-api` and its front ends + (`layerfs-cli`, `layerfs-node`, `layerfs-python`) are API crates under + `api/crates`; see the [API guide](../api/AGENTS.md). No engine crate may + depend on them. Since #535 (owner, 2026-11-03, D1, D4) Sandbox and network + code leave the engine for the connect layer, renamed the + [control layer](../control/AGENTS.md) by layerfs-project#101 (D-6): + the Sandbox runtime (now `layerfs-sandbox-backend`), the machine layer (now + `layerfs-platform`) and remote storage (now `layerfs-bucket`, with + `layerfs-store-service`'s `remote/`) moved out in #535 phase 1. The engine keeps + persistence (SQLite and PostgreSQL, D2), the daemon, the overlay, + Workspaces and the kernel routes (FUSE and NFS as peers over + `layerfs-mount-ports`, D3). No engine crate may depend on a control crate. +- `layerfs-store-service` owns the SDK Store calls in `src/methods/` (#536 + H3): the method table (`methods::call`), its argument reader and one file + per `file.*`/`branch.*` call. `read`, `write` and `history` know no SDK + method names; `methods/` is the facade over them. Their tests live in + store-service's `tests/` (`concurrent_commits`, `lane_merge` and + `merge_report`, which need a daemon, in `layerfs-daemon`'s). +- Project Store Init and install belong to `layerfs-project::store`; `layerfs-api` + retains its public re-exports. Its SDK-facing daemon integration tests + include daemon and Sandbox test fixtures from this layer. + The v0.1.6 reference under + root `crates/` and the archived `core/reference-tests` were retired by R9c + (2026-10-10) and are absent at HEAD; recover them at + local tag `reference-v0.1.6-final` (`d296981ac`, `crates` tree `498dd1917812`). + Neither was a dependency, source include or fallback; do not restore them. +- Active cluster-one members are content, storage, history, persistence, project + Init and telemetry. The initial cluster-two SQLite engine is `layerfs-overlay`; + its active membership does not complete Workspace/FUSE/daemon integration. + [Cargo.toml](../Cargo.toml) is the membership authority. + Workspace now exposes immutable base binding/read, effective source-qualified + lookup/stat/paged name merge, prepared actor install and atomic ordinary + namespace operations, bounded writes/truncate/regrow and composed payload + reads. Independent file/captured/operation custody and bounded live reclamation + are implemented through S6, including indexed automatic reclamation (its + physical reservation and headroom were removed by #321: no space is reserved + ahead of use, and a full device rolls the write back as no space); + native/runtime/integrated exits remain + unfinished. See [S6 audit](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/S6-EXIT-AUDIT.md). Its relocated + predecessor `layerfs-workspace-legacy` was removed by R7-retire. + Since #321 the daemon holds one overlay file and connection per mounted + Workspace, each behind its own lock, with the calling thread running each + short job; the fair SQL owner thread, its queue, credits and shared memory + budget are removed. See the [terminology guide](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/64-overlay-terminology.md) + and the [#321 record](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/321/CHECKLIST.md). + Daemon also exposes a direct global Store adapter with a + mount-time reader per Workspace, exact quarantine and read-only bind snapshots, + one immutable cache shared across Workspaces, bounded root + binding and operation-owned exact failures. Concrete opening stays outside + the provider-independent adapter; see the + [direct Store boundary](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/61-direct-store-adapter.md). + The host-mediated SDK/upstream transport is retired; see + [the retirement record](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/PRE-S8-F12-20261007.md). + Workspace calls return typed values directly and count SQL/payload/allocation + work under the same lock. Exact attempted failures retain their cause and + backing; before-effect refusals retain their original input. Overlay/Daemon + also expose bounded indexed operation records in the same database, with guarded atomic changes and automatic last-owner + cleanup. Content now exposes a backed file editor over these records and + Workspace supplies an explicit operation/file adapter with first-original + failure custody. Run-aware localized edits reuse the frozen sparse scanner + and bounded positive immutable zero evidence. Streamed directory inputs share + the canonical filesystem algorithm; backed serial state supplies new-parent + membership, rebuilt roots and initial counts/final row streams. On the backed + route validation classifies in 64-row windows and keeps its topology evidence + in indexed records; resident routes keep their resource-derived limits. + Captured local inode/run points retain exact reader root/floor and bounded + forward metadata progress. Captured regular-file normalization now derives + authenticated base facts, coalesces final changes into indexed records and + consumes them once through the retained FileView/fallible edit source. Backed + reference reduction/release uses fixed rows, sealed membership passes and + guarded FIFO/cursor transitions. Captured namespace normalization is the + `CapturedNamespace` producer + ([captured namespace construction](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/78-captured-namespace-construction.md)); + the product Commit constructor calls it + ([product Commit](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/79-product-commit.md)). The Store + half of Commit is implemented with exact original failure/unknown custody; + see [Store Commit](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/65-store-commit-composition.md). + The external E01 example records original startup work, diagnostics and Stop + without creating a Workspace route. It supplies diagnostic receipt consistency, + not E1/E2 performance admission. Its relocated predecessor was removed by + R7-retire; standalone native executable/Exec integration is unfinished. + Pre-S8 direct-operation accounting now has actual registered macOS/Linux engine + and Store-half Commit runs, independent family/count validation, and supported + phase observations. Numerical latency/residency and exact continuous peaks + remain unqualified; see the [accounting scope](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/PRE-S8-F14-20261007.md). + SDK creates/imports the initial Store, publishes the first Branch and consumes + seal, returning a closed file plus explicit provider/locator/profile metadata. + It retains no Store handle or data service. Native install now streams the + sealed file once, opens it in the daemon and returns both actual SQLite versions; + see [F5 handoff](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/PRE-S8-F5-20261007.md). Authenticated mount + binding, Store-half Commit, status, terminal close, fork and history now compose + the direct ports; see [native control](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/68-native-workspace-control.md). + The prior client/runtime, daemon upstream, Bridge logical data framing and + excluded API-core are retired. Historical receipts keep their original topology + and verdicts. R7-retire (2026-10-10) + removed all seven excluded directories as retirement: `layerfs-server`, + `layerfs-fuse-legacy`, `layerfs-sandbox-legacy`, `layerfs-sdk-legacy`, + `layerfs-daemon-legacy`, `layerfs-bridge-legacy` and + `layerfs-workspace-legacy`. The root workspace lists the active + members across core, engine and api; the boundary guard refuses the return of a removed directory or + package name, and each source is recoverable from the commit named in the + [R7-retire record](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/R7-RETIRE-COMPLETION-20261010.md). + Pinned read-only SDK views (303/08 O-10) are deferred, not dropped: no + implementation is in the tree, and a kept feature needs a new design on the + SQLite overlay. Do not restore the predecessor code to supply them. + Project native import preserves opaque symlinks, complete membership and hard + links, with indexed acquisition state. The accepted host WAL baseline retains + its recorded speed FAIL; no new timing is implied by SDK composition. + Bridge retains pinned authenticated KK channels, bounded native records and + independent fences and bounded native install/control. FUSE/Exec qualification + and complete kernel mount readiness remain S8. + The replacement Sandbox is an active member with ordinary Engine execution, + owned lifecycle and authenticated SDK/daemon startup. Optional privileged + command cancellation/client provenance is outside the current owner scope. + The replacement Fuse is now active with a fixed shared dispatcher, actual + deferred Owner/Store ports, a read-path callback adapter with complete opcode + accounting, direct mount/session ownership and connection drain. + Its predecessor `layerfs-fuse-legacy` was removed by R7-retire; the last + commit holding that source is `15e33e22c`. The daemon + composes Attach/Locate/Ready and reversible normal unmount; see + [native request service](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/75-native-request-service.md) and + [native mount session](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/76-native-mount-session.md). + Ordinary native mutation is served through the existing Workspace plan and + one publishing Overlay job, with no notification and no writeback cache; see + [native mutation and kernel coherence](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/77-native-mutation-coherence.md). + A control Commit runs the captured namespace producer through the unchanged + driver, and takes the Workspace's overlay lock for each of its jobs; the mounted + proofs and their limits are in the + [R5 record](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/R5-COMPLETION-20261009.md). Forced unmount of one + Workspace is implemented as `ForceUnmount`: a before-effect guard, one abort + write on the connection's own control, the terminal fence, the local drain, + one plain detach, then Revoke and Close; see + [forced teardown](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/80-forced-teardown.md). It is refused where + no abort control is bound, which includes the current Sandbox topology, and a + stop after an effect is `Retained` with no way back. No daemon-wide graceful + drain exists and none is planned. Source acquisitions have their own owner + slots, and a failed cold read ends its own request with `EIO` while the + mount keeps serving; the concurrency, teardown and failure-scope proofs and + their limits are in the [R6 record](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/R6-COMPLETION-20261009.md). + A create that leaves fewer unconsumed inode serials than the daemon's + configured low-water makes one early reservation; every deployment and + harness sets 0 today, which makes none. A new Mount is refused `Capacity` at + `mount:debt` while maintenance is stopped. Both, the proofs added around + them and the rows still open are in the + [R8b record](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/R8B-COMPLETION-20261010.md). R8b closed not + qualified and R9b without removal. R8c then closed qualified on function, + counts and resources at the owner-rescoped scope, with the deployed + low-water at 256 ([R8c record](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/R8C-COMPLETION-20261010.md)), + and R9c retired the root reference + ([R9c record](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/R9C-COMPLETION-20261010.md)). + Add members only with real product boundaries and implementation. +- Cluster-one work starts with the [handbook](../docs/general/cluster_one_handbook.md) and + [CAS/CDC/delta guide](../docs/general/cas_cdc_deltaencoding_handbook.md). Use public contracts, + source pins and limitations; preserve canonical compatibility. +- Cluster-two work starts with the [architecture guide](../docs/architecture/README.md): + the engine topics it touches, the [flows](../docs/architecture/flows/README.md) and [Where each call runs](../docs/architecture/control/README.md#where-each-call-runs). - Required corrections are real scope; research candidates are not - implemented features or excuses for workload caps. Migration, source moves - and retirement are explicit changes, never side effects of a component. + Required corrections are real engineering scope; research candidates are not + implemented features or excuses for accumulated workload caps. +- Closed Stage 0–6 handoffs and prior server/backend plans are historical. + Consult them for a selected compatibility/reference question; do not restart + their loops or inherit their retired topology. Do not revive `layerfs-server`. +- Keep unrelated work intact. Migration, source moves and retirement are explicit + changes, not side effects of implementing a component. ## Architecture and public contracts -- **Overlay naming:** `directory_entry` and `operation_record` (with their - owned and indexed variants) throughout SQL and the public ports. Since #321 - (schema 34) each mounted Workspace has its own overlay file with the same - tables, copied from a closed empty template at mount and deleted at terminal - unmount; the current schema is on the [Overlay](../docs/architecture/engine/overlay/README.md) - page. A naming change authorizes no physical table redesign. -- **Contracts:** public content, storage, history and persistence contracts - govern integrated callers. Stable capture and input, Store-derived policy, - backpressure, Save completion and atomic history publication are separate - obligations. Branch Commit overwrites the head and keeps its captured parent - ([owner supersession](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/BRANCH-OVERWRITE-DECISION-20261007.md)). -- **Project composition:** native import consumes SQLite-backed acquisition - through its backend-neutral port. Init into an existing Store is a live add - through the Store's own writer (2.0, #533); the Init that creates, seals and - installs a Store file stays for Store creation and install. Project's - owner-approved first-party edges (Bridge provisioning types, Persistence) - stay bounded by the guard's allowlist; Project uses no third-party database - driver directly. -- **Source pins:** `docs/architecture/` describes product source, not release - qualification. Record source pins and advance them explicitly; never - silently re-date evidence. +Overlay schema19 uses `directory_entry` and `operation_record` terminology, +including owned/indexed variants, throughout SQL and the active public ports. +Through schema 33 Workspaces shared physical tables partitioned by namespace; +since #321 (schema 34) every mounted Workspace has its own file with the same +tables, copied from a closed empty template at mount and deleted at terminal +unmount. The naming +change authorizes no physical table redesign; generic temporary buffers and +historical receipts retain their distinct terms. See the +[terminology guide](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/64-overlay-terminology.md). +The #320 product sweep (2026-10-14, schema 37) removes accounting triggers, +Workspace/allocator SQL rows and generic caller-minted leases. Workspace +lifecycle/frontier/dirty/base-reader counters are one journaled runtime value, +restored only after known whole-job rollback; unknown outcomes retain custody. +Commit runs/readers/base pins are journaled owner maps; captured generation +holds use a count index. Daemon-bound routing is checked once per call; Overlay +accepts no Route and stores no namespace. Cleanup debt remains SQL-backed after +the [memory-placement review](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/320/CLEANUP-MEMORY-DESIGN-20261014.md). +Resources computes row/length totals on demand and +retains exact owner counts from memory. See the current +[runtime state](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/64-overlay-terminology.md#runtime-workspace-state-and-remembered-sql-custody). + +Schema17 adds native mount/source/lookup/read associations, atomic Workspace +read observations and bounded revoked-lookup retirement. Schema18 adds native +file associations over the existing OpenFile owner and atomic open decisions; see +[native read custody](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/73-native-read-custody.md). This component +does not establish Ready, kernel service or full drain. Schema19 adds native +directory ownership, exact name-boundary cookies, retained parent updates and +bounded live cleanup; see [directory custody](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/74-native-directory-custody.md). + +`docs/architecture/` describes product source, not release qualification. +A source change to boundaries, canonical/physical format, algorithm or named +bound updates the affected architecture/API document in the same change. +Record source pins and advance them explicitly; never silently re-date evidence. +Keep implementation description separate from target proposal and research. + +Public content/storage/history/persistence contracts govern integrated callers. +Stable capture/input, Store-derived policy, backpressure, Save completion and +atomic history publication are separate obligations. Branch Commit overwrites the +head and keeps its captured parent; see the [owner supersession](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/BRANCH-OVERWRITE-DECISION-20261007.md). Immutable objects do +not replace authority, reference closure, mutable history or safe GC. + +Project's native import consumes SQLite-backed acquisition through its +backend-neutral port. Its Store composition lives in `layerfs-project::store`: +Init into an existing Store is a live add through the Store's own writer +(2.0, #533); the older Init that creates, seals and installs a Store file stays +for the Store's own creation and install. The +owner-approved first-party edges include Bridge provisioning types and +Persistence; the exact allowlist still bounds those edges, and Project still +cannot use a third-party database driver directly. Canonical construction stays +in Content, encoding/packing in Storage and database writes in Persistence. +The port and its provider remain the existing indexed acquisition algorithm. +See the [S7/S9 remaining plan](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/IMPLEMENTATION-PLAN-S7-S9-20261006.md). ### Filesystem access and execution ownership -Owner clarification 2026-10-08; implementation and proof requirements, not a -claim that S8 or S10 is qualified. - -- **FUSE serves any permitted process that can see the mount.** Request - admission and Workspace mutation or capture never require an Exec identity, - registered command or LayerFS-launched parent; external executors, - interactive shells and Sandbox execution get the same permissions, - semantics and ownership. -- **Commands belong to their launcher.** Sandbox setup or the external - executor establishes command identity, mount visibility and protection of - Store, Overlay and daemon credentials; execution owns standard I/O, exit - status and caller-requested cancellation. `workspace.exec` checks the - mounted directory and runs the command through the Sandbox runtime - ([where each call runs](../docs/architecture/control/README.md#where-each-call-runs)). - The daemon has no exec supervisor, launcher mode, per-Exec cgroups, command - registration or Exec stream/status/cancellation protocol. The SDK is the - call table in [layerfs-api](../api/crates/layerfs-api/src) - ([the calls](../docs/versioned/2.0.0/calls.md)). -- **FUSE owns lifetimes:** exact request, open, lookup, capture and Commit - lifetimes and complete filesystem and daemon-work drain. Zero registered - commands or shell exit never proves released descriptors, mappings, kernel - references or daemon jobs. Forced teardown kills no caller-owned process. - Normal unmount uses a reversible kernel busy probe while service stays - usable; force refuses active namespace control producers before any effect. -- **Commit** consumes the published frontier through the shared Content, - Storage and Persistence pipeline whatever the launch route. stdout/stderr are - not filesystem state unless written to a file in the Workspace. The old - managed-daemon Exec selections are withdrawn; keep their IDs, pins, receipts - and verdicts. -- **Proofs:** S8 verification includes a process launched outside the Exec API, - with no Exec registration, reading and mutating the mount and holding a - filesystem reference during an unmount attempt; S10 proves such changes - survive Commit and a fresh mount. Use ordinary public filesystem access. -- **FUSE source ownership** (reviewed 2026-10-08): `layerfs-fuse` owns its - native connection and request service (bounded dispatch, parking and - resumption, kernel handlers and replies, coherence); the daemon assembles one - shared FUSE service with its SQL and Store services and keeps the registry, - Ready/unmount and Commit composition. Dependency is daemon → fuse → - workspace, never fuse → daemon. Moving blocking OwnerClient calls behind a - port does not establish deferred service. +Owner clarification 2026-10-08: these are implementation and proof requirements, +not a claim that S8 or S10 is qualified. FUSE serves any permitted process that +can see the mount. Request admission and Workspace mutation/capture must not +require an Exec identity, registered command or LayerFS-launched parent. +Apply the same permissions, filesystem semantics and ownership to external +executors, interactive shells and ordinary Sandbox execution. + +Sandbox setup or the external executor establishes command identity, mount +visibility and protection of Store, Overlay and daemon credentials. Ordinary +runtime execution owns standard I/O, exit status and caller-requested cancellation. +SDK public organization is the eleven operations documented in +[the implemented SDK surface](../docs/general/sdk.md), with operation files under +[layerfs-api/src](../api/crates/layerfs-api/src). Project offers init/fork/ +branches/commits; Sandbox offers create/close; Workspace offers mount/exec/ +commit/status/unmount. Shared control/recovery/diagnostic plumbing lives in +base/ and the hidden internal module. Optional WorkspaceApi.exec selects the +mounted directory and delegates the existing Sandbox launcher; +the daemon has no exec/supervisor, launcher mode, per-Exec cgroups, command +registration or custom Exec stream/status/cancellation protocol. + +FUSE owns exact request/open/lookup/capture/Commit lifetimes and complete +filesystem/daemon-work drain. Zero registered commands or shell exit is never +evidence of released descriptors, mappings, kernel references or daemon jobs. +Forced filesystem teardown does not implicitly kill caller-owned processes. +Normal unmount uses a reversible kernel busy probe while ordinary service stays +usable. Force refuses active namespace control producers before any effect. + +Commit consumes the Workspace's published frontier through the shared Content, +Storage and Persistence pipeline regardless of launch route. stdout/stderr are +not filesystem state unless the command writes them to a file in the Workspace. +The old proposed managed-daemon Exec selections are withdrawn prospectively; +retain their original IDs, source pins, receipts and verdicts. Ordinary runtime +stream/result correctness and filesystem-lifetime proofs keep their real owners. + +S8 verification must include a process launched outside the Exec API, with no +Exec registration, reading and mutating the mount and retaining a filesystem +reference during an unmount attempt. S10 must prove that such changes survive +Commit and a fresh mount. Use ordinary public filesystem access in these proofs. + +Reviewed target source ownership2026-10-08: the replacement `layerfs-fuse` owns +its complete native connection/request service, including bounded dispatch, +parking/resumption, kernel handlers/replies and coherence. Daemon assembles one +shared Fuse service with its existing SQL/Store services, retaining registry, +overall Ready/unmount and Commit composition. Dependency is daemon -> fuse -> +workspace, never Fuse -> daemon. Request dispatch/read ports, mount/Ready and +per-Workspace normal drain are implemented ([R2 record](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/R2-COMPLETION-20261008.md)), +as is ordinary native mutation ([R3 record](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/R3-COMPLETION-20261008.md)); +as is the captured namespace producer with its bounded validation +([R4 record](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/R4-COMPLETION-20261008.md)), which the Commit +driver reaches through the product constructor +([product Commit](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/79-product-commit.md)). See the +[reviewed ownership](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/R2-R5-SOURCE-OWNERSHIP-REVIEW-20261008.md). +Moving blocking OwnerClient calls behind a port does not establish deferred service. ### Shared construction and completion boundaries -Namespace Init and Workspace Commit share Content, Storage and Persistence -through their public ports, with distinct input adapters and publication -lifecycles (names and paths rechecked 2026-11-16): +Source checked at `5747f0119`, guidance updated 2026-10-08. Namespace Init and +daemon construction share Content, Storage and Persistence through their public +ports. Their input adapters and publication lifecycles have distinct owners: -| Input/operation | Route | +| Input/operation | Existing route | | --- | --- | | Native Init file | Project import calls Content `construct_stream`; bounded output batches feed `Save::accept` | | New captured file | Workspace `CapturedFileEdits` calls Content `construct_runs` over data/zero windows | @@ -141,142 +302,192 @@ lifecycles (names and paths rechecked 2026-11-16): | Captured symlink and metadata | The same producer calls Content `emit_symlink` for a fresh target and patches or builds the portable attribute tree; a stored symlink keeps its base content root | | Object output | Content emits `FinalizedObject`; Storage `SaveSink` implements `FinalizedConsumer` by forwarding to `Save::accept` | -- **One encoder.** Canonical construction stays in Content, encoding and packing - in Storage, Store writes in Persistence; Overlay owns mutable payload, - metadata, operation records and construction scratch. Add no second encoder, - and keep SQLite types and paths out of Content and the daemon's - provider-independent `store/` adapter. Never materialize a native tree or - invoke Project Init to implement a Workspace Commit. -- **Use the real policy.** Use the Store-derived construction policy and - bounded capacities on both routes. An incremental file root need not equal a - fresh reconstruction: retained chunk boundaries can differ. -- **The product Commit:** `BoundWorkspace::commit_captured` runs - [`CapturedNamespace`](crates/layerfs-workspace/src/construction/driver.rs) - inside the Commit driver. It acquires the captured reader and one operation - owner, hands the driver the original cause on failure, and releases both - only once the outcome is known, keeping them with the failure when it is - not ([product Commit](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/79-product-commit.md)). -- **Completion boundaries:** `Save::finish()` completes object storage while the - Store stays open; Init then initializes history, and Commit calls - `commit_branch` against its expected head before installing the base - locally. Only the Init that creates a Store file calls - `SqliteHandles::seal()`, which consumes the sole session, checkpoints, closes - and hands off that file; its macOS unused-extent cleanup runs only there and - is no Content/pack format change or live-daemon maintenance. A live daemon - keeps the shared Store open across Saves and Commits. +Use the actual Store-derived construction policy and bounded capacities on +both routes. Fresh construction and incremental edits use appropriate public +entrypoints and common canonical formats; do not assume an incremental file root +must equal a fresh full reconstruction, since retained chunk boundaries can differ. +Overlay owns mutable payload, metadata, operation records and construction scratch +under Workspace prefixes. Content owns canonical algorithms; Storage owns +deduplication, encoding and packs; Persistence owns global Store writes. Keep +SQLite types and paths out of Content and the daemon's provider-independent +`store/` adapter. Do not materialize a native tree or invoke Project Init to +implement a Workspace Commit. Native Init acquisition backing remains a separate +host provisioning facility. + +Current source has captured-file normalization and the Store half of Commit. +`BoundWorkspace::commit` captures once, begins a Save, invokes a caller-supplied +Content constructor, finishes the Save, publishes through History and installs +the known root locally. Full captured namespace assembly, including names, +links, metadata and changed file roots, is +[`CapturedNamespace`](crates/layerfs-workspace/src/construction/driver.rs). +The product constructor `BoundWorkspace::commit_captured` runs it inside that +driver: it acquires the captured reader and one operation owner, hands the +driver the original cause on failure, and releases both owners only after the +driver's outcome is known, keeping them with the failure when it is not +([product Commit](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/79-product-commit.md)). A control Commit +uses it; `Service::execute` still takes a caller's constructor for driver tests. +See [captured-file construction](crates/layerfs-workspace/src/construction/captured/owner.rs), +[the shared Save sink](../core/crates/layerfs-storage/src/save/operation.rs) and +[Store Commit composition](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/65-store-commit-composition.md). + +Keep completion boundaries explicit: `Save::finish()` completes object storage; +Init then initializes history, while Commit calls `commit_branch` against its +expected head before local base installation. SDK Init subsequently calls `SqliteHandles::seal()` to +checkpoint, close and hand off one Store file. The macOS allocation correction +runs only at this sole-owner Store seal; it is not a Content/pack format change +or live-daemon maintenance. Linux daemons keep their shared Store open. The +[seal proof](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/SEAL-ALLOCATION-STRIDE1-20261007.md) retains its exact +platform and benchmark scope; it does not qualify daemon allocation or S10. ## Optimization and performance debugging -Before changing a performance-sensitive path or diagnosing a performance -finding, read the [optimization guide](../docs/general/optimization-guide.md); -its FUSE, SQLite and complexity rules are mandatory. Reject quadratic or worse -scaling; use SQLite-backed mutable indexes and state, not custom duplicate -trees or graphs; account for request, SQL, I/O, copy, residency and cleanup -work. A SQLite claim needs both EXPLAIN and correlated runtime database -profiling. The [optimization handbook](../docs/general/optimization-handbook.md) -is the working procedure. A docs-only change needs no campaign. +Before changing a performance-sensitive path or diagnosing a performance finding, +read the [optimization guide](../docs/general/optimization-guide.md). Its FUSE, +SQLite and complexity rules are mandatory: reject quadratic or worse scaling, +use SQLite-backed mutable indexes/state rather than custom duplicate trees/graphs, +and account for request/SQL/IO/copy/residency/cleanup work. Preserve cluster one's +canonical formats and public APIs; use count-driven evidence and the existing +measurement policy. SQLite debugging/optimization claims require both database +EXPLAIN and correlated runtime database profiling; overall wall time or a plan +alone is insufficient. An unrelated/docs-only change does not require a campaign. +The [optimization handbook](../docs/general/optimization-handbook.md) is the +working procedure on top of those rules: the counters that already exist, the +diagnostic groups, the scaling checks and the current candidate list. ## Product source and external verification -- **`src/` holds production code only** in core, engine, control and api, - including included or generated implementation; shipped runtime SQL and - other shipped formats follow the same rules. -- **Forbidden in product source:** inline tests or benches, test-only cfg or - features, fixtures, mocks, fake clocks, fault injection, test or benchmark - drivers, test-only public APIs, benchmark-selected algorithms and executable - rustdoc test blocks. Platform cfgs, real features, production validation and - actual telemetry are legitimate. -- **Where things go:** public-API tests and their fixtures in package `tests/`, - examples in `examples/`, benchmarks in `benches/`, product-wide harnesses in - [layerfs-benchmark](https://github.com/Ephemeral-AI-Lab/layerfs-benchmark), - development checks in `tools/`. Scaffold no empty placeholder directories. -- **Tests use the real library.** Do not include or recompile private source, - redirect target paths into test directories, import helpers into production - or expose an API only for a test. Test-only dependencies are - dev-dependencies; keep component restrictions such as std-only telemetry. -- **Deterministic observations.** Prefer public behaviour; telemetry tests use - completed reports and structural ordering, with no product fake clock or - narrow sleep-based thresholds. Verify disabled behaviour through its real - allocation and clock path. +The Rust `src/` directories in core, engine and api contain production code +exclusively, including +included/generated implementation. Runtime SQL and other shipped implementation +have the same ownership rules. + +Forbidden in product source: inline tests/benches, test-only cfg/features, +fixtures, mocks, fake clocks, fault injection, test/benchmark drivers, test-only +public APIs, benchmark-selected algorithms and executable rustdoc test blocks. +Platform cfgs, real features, production validation/invariants and actual telemetry +remain legitimate. Put API documentation with the API; examples outside src. + +| Content | Location | +| --- | --- | +| Product implementation | Core/engine crate `src/`, API Rust/JavaScript `src/` and required runtime source | +| Public-API tests and their helpers/fixtures | Package `tests/` | +| Runnable examples | Package `examples/` | +| Package benchmarks | Package `benches/` | +| Product-wide harness/diagnostics | the [layerfs-benchmark](https://github.com/Ephemeral-AI-Lab/layerfs-benchmark) repository; inspect the actual harness manifest/entrypoint | +| Development checks | `tools/` | + +Tests exercise the production library that normal consumers use. Do not +include/recompile private source, redirect target paths into test directories, +import helpers into production or expose an API only to enable a test. +Test-only dependencies belong in dev-dependencies; retain component-specific +restrictions, including std-only telemetry. + +Prefer deterministic public behavior and external observations. Telemetry tests +can use completed reports and structural ordering; do not add a product fake-clock +interface or narrow sleep-based wall thresholds. Verify disabled behavior through +its actual allocation/clock path. Scaffold no empty placeholder directories. ## Production file size and responsibility -- **999 physical lines** at most per new production file, comments and blanks - included; this covers shipped runtime SQL and other formats outside Rust - `src/` (extend guard coverage for a new format). Moving code does not evade - it, and no root `crates/` returns for product work. -- **200 physical lines** at most per product `lib.rs`/`mod.rs`: attributes, - declarations, imports and re-exports, API docs and thin forwarding only. - Definitions, impls, behaviour-encoding state, branching, loops, conversions, - validation, formatting and I/O go in focused implementation files. -- **Split by responsibility** (lookup, transactions, reconstruction, placement, - ownership) before the ceiling, not into numbered files or a renamed god - module. Use ordinary functions and concrete types: no factory or interface - per algorithm, no service locator. Keep real process and provider - boundaries independently usable. Do not minify, expand macros or add - includes to evade these rules. -- Physical lines differ from [production LOC](../AGENTS.md#production-loc-comparison-for-every-commit), - which every commit reports. +All new replacement/application-adapter production files have at most **999 +physical lines**, including comments/blanks. This includes required runtime SQL +or other shipped formats outside Rust src. Extend guard coverage for new formats; +moving implementation does not evade the rule. The legacy root source is +retired; do not recreate a root `crates/` for product work. + +Every product `lib.rs`/`mod.rs` has at most **200 physical lines**. These files +contain attributes, declarations, imports/reexports, API docs and thin forwarding. +Definitions, impls, state/constants encoding behavior, branching, loops, conversions, +validation, formatting and I/O belong in focused named implementation files. + +Split by responsibility before the ceiling: lookup, transactions, reconstruction, +placement and ownership, not numbered files or a renamed god module. Use ordinary +functions/concrete types; do not add a factory/interface per algorithm or a +service locator. Keep actual process/provider boundaries independently usable. +Do not minify, expand macros or add includes to evade size/ownership rules. + +Physical line ceilings differ from production LOC. Follow the root +[per-commit LOC policy](../AGENTS.md#production-loc-comparison-for-every-commit) +using exact parent/staged/committed trees and migration subtotals. Docs, tests, +fixtures, examples and tools do not contribute to production LOC. ## Failure, persistence and dependencies -- **One attempt** ([root rule](../AGENTS.md#4-code-build-and-docs)), including no - error-driven algorithm, backend or reference substitution and no guessed - recovery. Readiness waits and explicit cancellation have their own state - before an attempt and never permit replay after a failed or unknown - publication. Preserve definite-failure atomicity, original errors and exact - uncertain custody: no resend, rollback, deletion or success claim from an - unfenced read. An unsupported required platform or capability fails - explicitly, never as a silent no-op or fallback. -- **SQLite Stores are Disposable/WAL/OFF only**, on macOS and Linux - ([profile](../AGENTS.md#active-persistence-profile--owner-direction-2026-10-07)); - select it explicitly, never by default argument. Disposable claims - process-crash survival only. Open verifies WAL and never converts an existing - Store; select profile and layout before open. WAL stays on throughout Init - and Commit (owner supersession 2026-10-07): no private MEMORY import or - journal promotion. Checkpointing is described on the - [SQLite page](../docs/architecture/engine/store/sqlite.md#checkpoints) - (owner approval 2026-11-07; Durable keeps `wal_autocheckpoint=1000`); a - non-WAL journal was measured and refused - because it broke concurrent sessions and other processes. The measured host - Init regression is retained as FAIL and accepted as the baseline for that - scope. A PostgreSQL Store has no profile; its write setting is - `synchronous_commit` ([PostgreSQL](../docs/architecture/engine/store/postgresql.md)). -- **Seal** consumes sole ownership, checkpoints, closes and verifies one file. +- One attempted operation; no automatic retry, busy handler, refresh/reprepare, + error-driven algorithm/backend/reference substitution or guessed recovery. + Readiness waits and explicit cancellation have their own state before an + attempt; they do not permit replay after failed/unknown publication. +- Preserve definite-failure atomicity, original errors and exact uncertain + custody. No resend, rollback/delete or success claim based on an unfenced read. +- Global Persistence uses **Disposable/WAL/OFF only**, on macOS and Linux, + under the [owner's active-profile rule](../AGENTS.md#active-persistence-profile--owner-direction-2026-10-07). + Durable execution is disabled indefinitely until explicit owner reauthorization; + retained Durable source may compile but no test, diagnostic, measurement or + application invocation may execute it. Select Disposable explicitly rather than + relying on a default argument. Disposable claims process-crash survival only. + Open verifies WAL and never converts an + existing Store. Select profile/layout before open; Postgres stays unavailable. + Owner supersession2026-10-07 keeps WAL throughout Init and Commit; no private + MEMORY import or journal promotion remains. Owner approval, 2026-11-07: + Disposable runs `wal_autocheckpoint=0`; each writable Disposable session's + own checkpoint thread copies a log past 4 MiB with + `wal_checkpoint(PASSIVE)` on a second SQLite connection, off the commit + path. Durable keeps `wal_autocheckpoint=1000`. A non-WAL journal was + measured and refused: it broke concurrent sessions and other processes. The measured host Init regression + is retained as FAIL and accepted as the new baseline for that exact scope. + Host seal consumes sole ownership, checkpoints, closes and verifies one file. Its macOS completion releases unused extents beyond logical EOF through the - existing safe nix API, keeping the file's bytes and identity; it restores no - per-pack preallocation or allocation owner and never runs on a live shared - daemon Store ([seal boundary](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/60-shared-store-foundation.md)). -- **Writers:** several processes open the same Store directly, with no host in - the data path ([routes](../AGENTS.md#where-calls-run)). One Commit runs at a - time per Workspace; Commits on different Workspaces may run concurrently - (owner, 2026-10-12). Bounded before-effect admission queues SQL work within - a Session and coordinates short writer jobs on the Store's parent-directory - inode, on Linux and macOS (#647), so separate writers queue instead of - meeting SQLite's Busy; only an uncoordinated raw SQLite process still meets - the exact Busy refusal. Store files in one directory share this lane. It - gates no whole Save or Commit and replays no failed SQL attempt. Never open - and close a separate main, WAL or SHM descriptor for advisory ownership: - closing a main-file descriptor can cancel SQLite's own POSIX locks. Keep the - admission, lock and publication lane, and shorten what runs inside the lane - ([one writer, short holds](../AGENTS.md#where-calls-run)). Do not extend the - host-mediated SDK runtime, Bridge data codec or daemon upstream. -- **The daemon overlay is a separate profile:** one MEMORY/OFF/EXCLUSIVE - database file per mounted Workspace, opened by one connection behind that - Workspace's lock, sharing no connection, page cache or memory budget. It is - not a distributed Store; reader-pool alternatives need their own proofs. -- **Durability claims:** WAL and sync are permitted for a declared global Store - profile. Add no `fsync`/`fdatasync`/`sync_data`/`sync_all` on disposable - Workspace backing; claim only actual completion and durability contracts. -- **Dependencies:** no third-party patch, fork, vendor, registry edit or - replacement dependency except the [fuser Git fork](#fuser-git-source); use - existing capabilities and locked builds, and report an incompatible required - dependency with source evidence. + existing safe nix API, preserving the original file's bytes and identity. + This narrow seal-only correction does not restore per-pack preallocation or + the allocation owner, and is never invoked on a live shared daemon Store; see + the [seal boundary](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/60-shared-store-foundation.md). + See the [foundation checkpoint](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/PRE-S8-F1-F4-20261007.md). +- Target (owner direction 2026-10-07): the same Store opened directly by every + Linux daemon from a shared volume, several writer processes, no host in the + data path, using only Disposable/WAL/OFF. A contended + write still has one SQL attempt and exact failure/unknown custody. Owner + clarification2026-10-12 requires one ongoing Commit per Workspace and allows + concurrent Commits on different Workspaces. Bounded before-effect admission + now queues SQL work within a Session and coordinates short Linux writer jobs + on the existing local Store parent-directory inode; multiple Store files in + one directory share this coarse short-transaction lane. Do not open/close a + separate main/WAL/SHM descriptor for advisory ownership: closing a main-file + descriptor can cancel SQLite's own process POSIX locks. This admission does + not gate whole Saves or Commits or replay + a failed SQL attempt. macOS holds the same parent-directory lane as Linux, + over the same short writer jobs and its writable opens (#647), so separate + macOS writers queue instead of meeting SQLite's Busy; only a writer that + bypasses the lane (an uncoordinated raw SQLite process) retains the exact + Busy refusal. Follow the + [integration contract](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/303/06-cluster-one-integration.md) and its + [deepest-file plan](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/SERVERLESS-STORE-PLAN-20261007.md); do not + extend the host-mediated SDK runtime, Bridge data codec or daemon upstream. +- Store concurrency (owner decision 2026-10-18): one global database and one + writer at a time are the design, not a limitation to engineer around. Keep + the existing admission, directory lock and publication lane; shorten what + runs inside the lane (only head check, merge and publish) instead of adding + writers or databases. Judge speed on uncontended calls; contention must stay + correct, bounded and exact. See the + [Store concurrency model](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/architecture/83-store-concurrency-model.md). +- The daemon overlay is a separate owner/profile from global persistence: + one local database file per mounted Workspace, opened MEMORY/OFF/EXCLUSIVE by + one connection behind that Workspace's lock. Workspaces share no overlay + connection, page cache or memory budget. It is not an implemented distributed Store or a universal + global-Store rule; reader-pool alternatives need their own proofs. +- WAL/sync are permitted for a declared global Store profile. Do not add + fsync/fdatasync/sync_data/sync_all on disposable Workspace backing. Memory + hints do not prove bounds. Claim only actual completion/durability contracts. +- No third-party patch, fork, vendor, registry edit or replacement dependency, + except the owner's pinned [fuser Git fork](#fuser-git-source) below. + Use existing capabilities and locked builds; report an incompatible required + dependency with source evidence. Unsupported required platforms/capabilities + fail explicitly, not via a silent no-op or fallback. ## Checks and completion -For product, API, manifest or runtime-source changes, run from the repository -root with the root manifest: +Run commands from the repository root with the root workspace manifest. For product, +API, manifest or runtime-source changes, cover tests/examples, warning-denying +Clippy and formatting, plus the boundary guard and its self-tests: ```sh cargo +1.85.1 test --locked --all-targets --no-run @@ -287,77 +498,109 @@ python3 -B tools/check_product_boundary.py python3 -B -m unittest discover -s tools -p 'test_*.py' ``` -- **With the [worktree launcher](../docs/general/parallel-worktrees.md)**, run - the same checks per package through `tools/worktree.py cargo` (it refuses - workspace aggregates): tests on Linux where they need a Linux Store or - mount, Clippy on macOS when the Linux build image has none. -- **A bug fix** adds a test that fails on the parent commit and passes on the - fix; run it once against the parent's source and report the result. -- **Run once, at final identity:** the required covering checks; diagnose a - failure from its output and source before a justified repair and rerun. Do - not repeat unchanged passing checks or run an unrelated legacy sweep. For - tooling or harness changes, run the owning scoped tests. For docs-only - changes, check links, anchors, status, claims and whitespace, and say why - Rust checks do not apply. -- **The guard is not a proof.** It checks source text, line caps and common - prohibited constructs (including markers in comments and strings), not Rust - semantics or runtime. Review macros, helpers, dependency and target - declarations and generated inputs; extend scanners for new shipped formats. - A package left out of the workspace is not exercised by passing tests. -- **Test lanes:** [tools/ci_tests.py](../tools/ci_tests.py). Update its `SLOW` - list when a test target is renamed, moved or measured at 5 s or more; - Linux-only daemon targets and the named API mount targets go to the nightly - mount lane. Do not disable discovery. Report exact checks, failures and - gaps; a scan or proposal alone establishes no implementation. -- **Build flags are part of the check.** aarch64 binaries keep the repository - ARMv8 AEAD inputs (`--cfg aes_armv8 --cfg polyval_armv8 --cfg - chacha20_force_neon -C target-feature=+aes,+sha2`). The - [root Cargo config](../.cargo/config.toml) is found from the working - directory, whatever manifest is selected; builds outside the repository or - with explicit RUSTFLAGS repeat the profile. Record flags and config in build - identities; keep the transport's required-profile refusal. A docs build - proves no binary profile. -- **Measurements** also follow the - [measurement workflow](../docs/general/agent-measurement-policy.md) and the - [owning harness guide](https://github.com/Ephemeral-AI-Lab/layerfs-benchmark/blob/main/fs-bench-pro/AGENTS.md); - frozen identities, cache states and budgets are evidence contracts, not - product limits. +Every test invocation obeys the repository's +[2-minute test ceiling](../AGENTS.md#4-code-build-and-docs): run it under an +explicit timeout of at most 120 s, per package when the whole suite cannot fit, +and treat reaching the ceiling as a hang to diagnose, not a run to wait for. + +Migrations and refactors follow the root +[delete-rather-than-nurse rule](../AGENTS.md#4-code-build-and-docs): delete a +test that is tricky to adapt, keep or move only cheap tests of surviving +behaviour, and re-cover any still-valid fix or contract on the new code. +fs-bench-pro (in layerfs-benchmark) is frozen and left broken (owner, 2026-10-18): +do not build, run, check or repair it. + +For docs-only changes, check links/anchors, status/claim accuracy and whitespace; +report why Rust/runtime checks are inapplicable. For tooling/harness changes run +the owning scoped tests. At final implementation identity run required covering +checks once; diagnose a failure from its output/source before a justified repair +and rerun. Do not repeat unchanged passing checks or run an unrelated legacy sweep. + +The guard checks source text, line caps and common prohibited constructs, +including markers embedded in source comments/strings. It is not a Rust semantic +or runtime proof. Review ordinary macros, helpers, dependency/target declarations +and generated inputs; extend scanners for new shipped formats. An excluded +FUSE/daemon package is not exercised by passing active-workspace tests. + +CI reports after a push and never gates it (see the +[root rule](../AGENTS.md#4-code-build-and-docs)); test lane selection is +[tools/ci_tests.py](../tools/ci_tests.py). When a test target is renamed, moved or +measured at 5 s or more, update its `SLOW` list; Linux-only daemon targets and the named API mount targets go to +the nightly mount lane. There is no pre-push gate; +`tools/preflight.sh` is retired. Do not disable discovery or recreate an +equivalent wrapper. Report exact checks, failures and gaps. A scan or proposal +alone does not establish implementation. + +### Build flags are part of the check + +aarch64 binaries retain the repository ARMv8 AEAD build inputs: +`--cfg aes_armv8 --cfg polyval_armv8 --cfg chacha20_force_neon +-C target-feature=+aes,+sha2`. +The [root Cargo config](../.cargo/config.toml) is discovered from the working +directory, independently of the selected manifest. Builds outside the repository or +with explicit RUSTFLAGS must repeat that profile. Record flags/config in build +identities; preserve the transport's required-profile refusal when integrating +it. Documentation builds do not prove the binary profile. + +Measurement work additionally follows the +[agent measurement workflow](../docs/general/agent-measurement-policy.md) and +[owning core harness guide](https://github.com/Ephemeral-AI-Lab/layerfs-benchmark/blob/main/fs-bench-pro/AGENTS.md). Frozen identities, +cache states and budgets are evidence contracts, not product runtime limits. ### Fuser Git source -Owner direction 2026-11-15 (superseding the crates.io-only and local-patch -rules): use [Ephemeral-AI-Lab/fuser](https://github.com/Ephemeral-AI-Lab/fuser) -directly as a Git dependency at revision -`7393bca050fcbd0ed2a03f7e79bec0f8bd9755c4` (package 0.18.0), kept reachable by -the fork's tag `v0.18.0-layerfs.1`. - -- The fork owns its timestamp correction, observable receive-loop lifecycle and - negotiated receive buffers, maintained in that repository with its tests, - never as a second copy or local patch here. Its rulesets forbid deleting or - force-pushing `fuser-0.18.0` and moving or deleting `v*-layerfs.*` tags; tag - every new pinned revision the same way. -- Keep the full `rev` in `layerfs-fuse/Cargo.toml` and the matching Git source - and commit in the root lockfile. No floating branch or tag, registry - fallback, `[patch]`, `[replace]` or local vendored fuser; fail explicitly when - the source is unavailable. The boundary guard refuses any other source, an - override and a lock entry that differs from the manifest; it does not check - the fork's code. -- A revision change updates the manifest, lockfile and - [FUSE architecture page](../docs/architecture/engine/kernel-routes/fuse.md#fuser-git-dependency) - together, needs owner approval ([code review policy](../docs/policy/code-review.md)) - and a fresh `tools/worktree.py prepare` in each worktree. -- Earlier pins, decisions, Docker acceptance and the Linux fractional - signed-minimum FAIL stay in - [the historical records](https://github.com/Ephemeral-AI-Lab/layerfs/tree/8cfb6e273470d0bb2a4ec5a9256fcf4972441eb5/engine/patches/fuser-0.18.0); - switching the dependency does not relabel that evidence. - -### Audited unsafe files - -Each crate below uses `deny(unsafe_code)` with exactly one allowed module; the -guard and its tests enforce the path, and no other FFI or dependency change is -authorized. - -| Crate | Allowed module | Why | -| --- | --- | --- | -| `layerfs-persistence` | `src/backend/sqlite/lifecycle/file_control.rs`, macOS only | sets and reads back `SQLITE_FCNTL_PERSIST_WAL` under sole seal ownership and on a live-Store snapshot's own new destination connection (#506); owner authorization 2026-10-07, "Approve the narrowly audited seal wrapper" ([decision](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/SEAL-PERSIST-WAL-DECISION-20261007.md)); no manual sidecar deletion | -| `layerfs-bridge` | `src/process/darwin.rs`, macOS only | `proc_pidinfo`, `proc_listallpids` and the `sysctl` boot id, each with its SAFETY comment; moved below the drivers by layerfs-project#112 (finding R10, accepted 2026-11-06, #101 M-3) | +Owner direction 2026-11-15 supersedes the earlier crates.io-only and local-patch +rules: use [Ephemeral-AI-Lab/fuser](https://github.com/Ephemeral-AI-Lab/fuser) +directly as a Git dependency. The selected revision is +`7393bca050fcbd0ed2a03f7e79bec0f8bd9755c4` (package version 0.18.0), kept +reachable by the fork's tag `v0.18.0-layerfs.1`. Repository rulesets on the fork +forbid deleting or force-pushing `fuser-0.18.0` and moving or deleting +`v*-layerfs.*` tags; tag every new pinned revision the same way. +The fork owns its timestamp correction, observable receive-loop lifecycle and +negotiated receive buffers. These changes are independent of LayerFS; maintain +them in that repository, not through a second copy or local patch here. + +Keep the full `rev` in `layerfs-fuse/Cargo.toml` and the matching Git source +and commit in the root lockfile. Do not use a floating branch/tag, a registry +fallback, `[patch]`, `[replace]` or a local vendored fuser. A revision change +updates the manifest, lockfile and the +[FUSE architecture page](../docs/architecture/engine/kernel-routes/fuse.md#fuser-git-dependency) +together, needs owner approval ([code review policy](../docs/policy/code-review.md)), +and needs a fresh `tools/worktree.py prepare` in each worktree before its offline +builds. Use locked builds; fail explicitly when the selected source is unavailable. + +The fork's own repository maintains its code and tests; LayerFS uses ordinary +locked dependency builds and integration tests. The boundary guard +(`tools/check_product_boundary.py`) refuses a fuser source other than the fork at +a full `rev`, a `[patch]`/`[replace]`/`paths` override and a lock entry that +differs from the manifest; it does not check the fork's code. The earlier source pins, owner decisions, +Docker acceptance and Linux fractional signed-minimum FAIL remain unchanged in +[the historical records](https://github.com/Ephemeral-AI-Lab/layerfs/tree/8cfb6e273470d0bb2a4ec5a9256fcf4972441eb5/engine/patches/fuser-0.18.0). +Switching the dependency does not relabel that evidence or extend its scope. + +### Authorized Persistence seal file control + +Owner authorization 2026-10-07: "Approve the narrowly audited seal wrapper". +The system macOS SQLite enables persistent WAL and otherwise leaves empty +sidecars after checked close. Persistence uses `deny(unsafe_code)` with one +allowed module, `src/backend/sqlite/lifecycle/file_control.rs`, compiled on macOS +only. It sets and reads back `SQLITE_FCNTL_PERSIST_WAL` under sole seal +ownership, and on a live-Store snapshot's own new destination connection (#506). +No other Persistence module may contain unsafe Rust; the guard and its tests +enforce the exact path. This does not authorize third-party patches, dependency +changes, manual sidecar deletion or other FFI. See the +[decision and diagnostic](https://github.com/Ephemeral-AI-Lab/layerfs/blob/af2ad92bce866b4e3e11365f016b12806828a53b/core/docs/issues/307/SEAL-PERSIST-WAL-DECISION-20261007.md). + +### Bridge process identity file + +The bridge holds the process facts the home and the Sandbox driver share +(`layerfs_bridge::process`), moved below the drivers by layerfs-project#112 +under review finding R10, which the owner accepted on 2026-11-06 (#101 M-3; +compute-plane-2.0 rule 5 and the §6.6 row: "move the process facts, with the +one audited macOS unsafe file and its guard entry, below the drivers"). The +bridge uses `deny(unsafe_code)` with one allowed module, +`src/process/darwin.rs`, compiled on macOS only: `proc_pidinfo`, +`proc_listallpids` and the `sysctl` boot id, each with its SAFETY comment, as +they were in the driver. No other bridge module may contain unsafe Rust; the +guard and its tests enforce the exact path. This authorizes no other FFI or +dependency change. diff --git a/engine/crates/layerfs-daemon/src/operator/workspace.rs b/engine/crates/layerfs-daemon/src/operator/workspace.rs index ad7218a6c2..19d5cb8a64 100644 --- a/engine/crates/layerfs-daemon/src/operator/workspace.rs +++ b/engine/crates/layerfs-daemon/src/operator/workspace.rs @@ -155,15 +155,27 @@ fn scan( ) -> Result<()> { let binding = workspace.snapshot().map_err(args::failed)?; let root = binding.effective_root; + let node = binding.branch.head_commit.map_or( + layerfs_history::NodeId::Layer(binding.branch.base_layer), + layerfs_history::NodeId::Commit, + ); + // The base node `history_serial`, as every `Snap` carries it for the + // `remove_file` visibility rule; 0 is the same unknown fallback that + // `snapshot::resolve` takes. + let serial = match service + .store + .published(|history| history.node_history_serial(node)) + { + Ok(Some(serial)) => serial, + _ => 0, + }; let snap = Snap { kind: Kind::Commit, label: "Workspace base".into(), - node: binding.branch.head_commit.map_or( - layerfs_history::NodeId::Layer(binding.branch.base_layer), - layerfs_history::NodeId::Commit, - ), + node, root, tree: TreeTable::open(op.ports(), FilesystemRootId(root)).map_err(args::content)?, + serial, }; let pending = workspace .pending_conflicts() diff --git a/engine/crates/layerfs-persistence/sql/postgresql/gc.sql b/engine/crates/layerfs-persistence/sql/postgresql/gc.sql new file mode 100644 index 0000000000..713bb0c64e --- /dev/null +++ b/engine/crates/layerfs-persistence/sql/postgresql/gc.sql @@ -0,0 +1,73 @@ + +CREATE TABLE branch_retired ( + branch_id BYTEA PRIMARY KEY CHECK (length(branch_id) = 17), + removed_ms BIGINT NOT NULL CHECK (removed_ms >= 0), + FOREIGN KEY (branch_id) REFERENCES branch(branch_id) ON DELETE CASCADE +); +CREATE TABLE layer_stack_retired ( + layer_stack_id BYTEA PRIMARY KEY CHECK (length(layer_stack_id) = 17), + deleted_ms BIGINT NOT NULL CHECK (deleted_ms >= 0), + FOREIGN KEY (layer_stack_id) REFERENCES layer_stack(layer_stack_id) ON DELETE CASCADE +); +CREATE TABLE path_removed ( + layer_stack_id BYTEA NOT NULL CHECK (length(layer_stack_id) = 17), + path_key BYTEA NOT NULL CHECK (length(path_key) = 32), + path BYTEA NOT NULL CHECK (length(path) BETWEEN 1 AND 4096), + frontier_serial BIGINT NOT NULL CHECK (frontier_serial >= 0), + removed_ms BIGINT NOT NULL CHECK (removed_ms >= 0), + PRIMARY KEY (layer_stack_id, path_key), + FOREIGN KEY (layer_stack_id) REFERENCES layer_stack(layer_stack_id) ON DELETE CASCADE +); +CREATE TABLE project_epoch ( + layer_stack_id BYTEA PRIMARY KEY CHECK (length(layer_stack_id) = 17), + epoch BIGINT NOT NULL DEFAULT 0 CHECK (epoch >= 0), + FOREIGN KEY (layer_stack_id) REFERENCES layer_stack(layer_stack_id) ON DELETE CASCADE +); +CREATE TABLE gc_intent ( + op_id BYTEA PRIMARY KEY CHECK (length(op_id) = 16), + kind TEXT NOT NULL CHECK (kind IN ('mount','init','merge','removal','compaction')), + scope_kind TEXT NOT NULL CHECK (scope_kind IN ('store','project','branch')), + scope_id BYTEA, + workspace TEXT, + sandbox TEXT, + machine TEXT, + project TEXT, + branch TEXT, + owner_runtime TEXT, + owner_boot TEXT, + state TEXT NOT NULL CHECK (state IN ('live','uncertain','done')), + uncertain BOOLEAN NOT NULL, + phase TEXT, + generation BIGINT, + artifact TEXT, + high_water BIGINT, + evidence TEXT, + updated_ms BIGINT NOT NULL, + UNIQUE (scope_kind, scope_id, owner_boot) +); +CREATE TABLE gc_op ( + id BIGINT PRIMARY KEY CHECK (id = 1), + generation BIGINT NOT NULL, + phase TEXT NOT NULL, + scratch_schema TEXT, + mark_serial BIGINT, + mark_pack_id BIGINT, + mark_digest BYTEA, + mark_totals BYTEA, + mark_stamp BIGINT, + started_ms BIGINT NOT NULL, + dead_canonical BIGINT NOT NULL DEFAULT 0, + dead_stored BIGINT NOT NULL DEFAULT 0, + dead_metadata BIGINT NOT NULL DEFAULT 0, + dead_logical BIGINT NOT NULL DEFAULT 0, + totals_applied BOOLEAN NOT NULL DEFAULT FALSE, + artifact TEXT, + pid BIGINT, + backend_start TEXT, + incarnation BIGINT +); +CREATE TABLE gc_dead ( + kind TEXT NOT NULL, + id BYTEA NOT NULL, + PRIMARY KEY (kind, id) +); diff --git a/engine/crates/layerfs-persistence/sql/postgresql/queries/history/layerstack_stack_by_name.sql b/engine/crates/layerfs-persistence/sql/postgresql/queries/history/layerstack_stack_by_name.sql index cc5cfea883..263f881223 100644 --- a/engine/crates/layerfs-persistence/sql/postgresql/queries/history/layerstack_stack_by_name.sql +++ b/engine/crates/layerfs-persistence/sql/postgresql/queries/history/layerstack_stack_by_name.sql @@ -1 +1 @@ -SELECT layer_stack_id, name, scope_id, profile_id, head_layer_id FROM {s}.layer_stack WHERE name = $1; +SELECT layer_stack_id, name, scope_id, profile_id, head_layer_id FROM {s}.layer_stack WHERE name = $1 AND NOT EXISTS (SELECT 1 FROM {s}.layer_stack_retired r WHERE r.layer_stack_id = {s}.layer_stack.layer_stack_id); diff --git a/engine/crates/layerfs-persistence/sql/postgresql/schema.sql b/engine/crates/layerfs-persistence/sql/postgresql/schema.sql index 5b15c1bb69..8687c0371c 100644 --- a/engine/crates/layerfs-persistence/sql/postgresql/schema.sql +++ b/engine/crates/layerfs-persistence/sql/postgresql/schema.sql @@ -70,6 +70,7 @@ CREATE TABLE history_meta ( binding_key bytea NOT NULL CHECK (octet_length(binding_key) BETWEEN 1 AND 128), identity_format bigint NOT NULL CHECK (identity_format = 3), schema_version bigint NOT NULL CHECK (schema_version = 2), + next_history_serial bigint NOT NULL DEFAULT 1, schema_source text NOT NULL, schema_definition text NOT NULL ); diff --git a/engine/crates/layerfs-persistence/sql/sqlite/gc.sql b/engine/crates/layerfs-persistence/sql/sqlite/gc.sql new file mode 100644 index 0000000000..10c62a803c --- /dev/null +++ b/engine/crates/layerfs-persistence/sql/sqlite/gc.sql @@ -0,0 +1,82 @@ +CREATE TABLE branch_retired ( + branch_id BLOB PRIMARY KEY + CHECK (length(branch_id) = 17 AND substr(branch_id, 1, 1) = X'11'), + removed_ms INTEGER NOT NULL CHECK (removed_ms >= 0), + FOREIGN KEY (branch_id) REFERENCES branch(branch_id) ON DELETE CASCADE +) STRICT, WITHOUT ROWID; + +CREATE TABLE layer_stack_retired ( + layer_stack_id BLOB PRIMARY KEY + CHECK (length(layer_stack_id) = 17 AND substr(layer_stack_id, 1, 1) = X'31'), + deleted_ms INTEGER NOT NULL CHECK (deleted_ms >= 0), + FOREIGN KEY (layer_stack_id) REFERENCES layer_stack(layer_stack_id) ON DELETE CASCADE +) STRICT, WITHOUT ROWID; + +CREATE TABLE path_removed ( + layer_stack_id BLOB NOT NULL + CHECK (length(layer_stack_id) = 17 AND substr(layer_stack_id, 1, 1) = X'31'), + path_key BLOB NOT NULL CHECK (length(path_key) = 32), + path BLOB NOT NULL CHECK (length(path) BETWEEN 1 AND 4096), + frontier_serial INTEGER NOT NULL CHECK (frontier_serial >= 0), + removed_ms INTEGER NOT NULL CHECK (removed_ms >= 0), + PRIMARY KEY (layer_stack_id, path_key), + FOREIGN KEY (layer_stack_id) REFERENCES layer_stack(layer_stack_id) ON DELETE CASCADE +) STRICT, WITHOUT ROWID; + +CREATE TABLE project_epoch ( + layer_stack_id BLOB PRIMARY KEY + CHECK (length(layer_stack_id) = 17 AND substr(layer_stack_id, 1, 1) = X'31'), + epoch INTEGER NOT NULL DEFAULT 0 CHECK (epoch >= 0), + FOREIGN KEY (layer_stack_id) REFERENCES layer_stack(layer_stack_id) ON DELETE CASCADE +) STRICT, WITHOUT ROWID; + +CREATE TABLE gc_intent ( + op_id BLOB PRIMARY KEY CHECK (length(op_id) = 16), + kind TEXT NOT NULL CHECK (kind IN ('mount','init','merge','removal','compaction')), + scope_kind TEXT NOT NULL CHECK (scope_kind IN ('store','project','branch')), + scope_id BLOB, + workspace TEXT, + sandbox TEXT, + machine TEXT, + project TEXT, + branch TEXT, + owner_runtime TEXT, + owner_boot TEXT, + state TEXT NOT NULL CHECK (state IN ('live','uncertain','done')), + uncertain INTEGER NOT NULL CHECK (uncertain IN (0,1)), + phase TEXT, + generation INTEGER, + artifact TEXT, + high_water INTEGER, + evidence TEXT, + updated_ms INTEGER NOT NULL, + UNIQUE (scope_kind, scope_id, owner_boot) +) STRICT; + +CREATE TABLE gc_op ( + id INTEGER PRIMARY KEY CHECK (id = 1), + generation INTEGER NOT NULL, + phase TEXT NOT NULL, + scratch_schema TEXT, + mark_serial INTEGER, + mark_pack_id INTEGER, + mark_digest BLOB, + mark_totals BLOB, + mark_stamp INTEGER, + started_ms INTEGER NOT NULL, + dead_canonical INTEGER NOT NULL DEFAULT 0, + dead_stored INTEGER NOT NULL DEFAULT 0, + dead_metadata INTEGER NOT NULL DEFAULT 0, + dead_logical INTEGER NOT NULL DEFAULT 0, + totals_applied INTEGER NOT NULL DEFAULT 0, + artifact TEXT, + pid INTEGER, + backend_start TEXT, + incarnation INTEGER +) STRICT; + +CREATE TABLE gc_dead ( + kind TEXT NOT NULL, + id BLOB NOT NULL, + PRIMARY KEY (kind, id) +) STRICT, WITHOUT ROWID; diff --git a/engine/crates/layerfs-persistence/sql/sqlite/history.sql b/engine/crates/layerfs-persistence/sql/sqlite/history.sql index 6a587b78cc..7c3ec88f1c 100644 --- a/engine/crates/layerfs-persistence/sql/sqlite/history.sql +++ b/engine/crates/layerfs-persistence/sql/sqlite/history.sql @@ -6,6 +6,7 @@ CREATE TABLE history_meta ( binding_key BLOB NOT NULL CHECK (length(binding_key) BETWEEN 1 AND 128), identity_format INTEGER NOT NULL CHECK (identity_format = 3), schema_version INTEGER NOT NULL CHECK(schema_version=1), + next_history_serial INTEGER NOT NULL DEFAULT 1, schema_source TEXT NOT NULL, schema_definition TEXT NOT NULL ) STRICT; @@ -50,6 +51,7 @@ CREATE TABLE layer ( source_commit_id IS NULL OR (length(source_commit_id) = 33 AND substr(source_commit_id, 1, 1) = X'12') ), + history_serial INTEGER NOT NULL DEFAULT 0, CHECK ( (parent_layer_id IS NULL AND source_branch_id IS NULL AND source_commit_id IS NULL) OR @@ -102,6 +104,7 @@ CREATE TABLE "commit" ( new_metadata INTEGER NOT NULL DEFAULT 0 CHECK(new_metadata>=0), lww BLOB NOT NULL, displaced_id BLOB CHECK(displaced_id IS NULL OR length(displaced_id)=33), + history_serial INTEGER NOT NULL DEFAULT 0, FOREIGN KEY (layer_stack_id, base_layer_id) REFERENCES layer(layer_stack_id, layer_id) DEFERRABLE INITIALLY DEFERRED, FOREIGN KEY (layer_stack_id, parent_commit_id) diff --git a/engine/crates/layerfs-persistence/sql/sqlite/metadata.sql b/engine/crates/layerfs-persistence/sql/sqlite/metadata.sql index 64671b6172..93c874272f 100644 --- a/engine/crates/layerfs-persistence/sql/sqlite/metadata.sql +++ b/engine/crates/layerfs-persistence/sql/sqlite/metadata.sql @@ -6,7 +6,8 @@ CREATE TABLE store_policy ( chunk_delta_max_depth INTEGER NOT NULL CHECK(chunk_delta_max_depth BETWEEN 0 AND 50), metadata_delta_max_depth INTEGER NOT NULL CHECK(metadata_delta_max_depth BETWEEN 0 AND 50), next_pack_id INTEGER NOT NULL DEFAULT 1 CHECK(next_pack_id>0), - next_ordinal INTEGER NOT NULL DEFAULT 1 CHECK(next_ordinal BETWEEN 1 AND 4294967296) + next_ordinal INTEGER NOT NULL DEFAULT 1 CHECK(next_ordinal BETWEEN 1 AND 4294967296), + trace_salt BLOB NOT NULL DEFAULT (X'0000000000000000000000000000000000000000000000000000000000000000') CHECK(length(trace_salt)=32) ) STRICT; CREATE TABLE object_location ( object_id BLOB PRIMARY KEY CHECK(length(object_id)=32), diff --git a/engine/crates/layerfs-persistence/sql/sqlite/queries/history/commit_insert_commit.sql b/engine/crates/layerfs-persistence/sql/sqlite/queries/history/commit_insert_commit.sql index 48d2df4768..9af2640f05 100644 --- a/engine/crates/layerfs-persistence/sql/sqlite/queries/history/commit_insert_commit.sql +++ b/engine/crates/layerfs-persistence/sql/sqlite/queries/history/commit_insert_commit.sql @@ -1 +1 @@ -INSERT INTO "commit" (commit_id, layer_stack_id, root_id, parent_commit_id, base_layer_id, second_parent_id, branch_id, via, message, merged_from_id, created_ms, logical, new_canonical, new_stored, new_metadata, lww, displaced_id) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17) ON CONFLICT (commit_id) DO NOTHING RETURNING 1; +INSERT INTO "commit" (commit_id, layer_stack_id, root_id, parent_commit_id, base_layer_id, second_parent_id, branch_id, via, message, merged_from_id, created_ms, logical, new_canonical, new_stored, new_metadata, lww, displaced_id, history_serial) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18) ON CONFLICT (commit_id) DO NOTHING RETURNING 1; diff --git a/engine/crates/layerfs-persistence/sql/sqlite/queries/history/layerstack_bump_project_epoch.sql b/engine/crates/layerfs-persistence/sql/sqlite/queries/history/layerstack_bump_project_epoch.sql new file mode 100644 index 0000000000..a975e5236a --- /dev/null +++ b/engine/crates/layerfs-persistence/sql/sqlite/queries/history/layerstack_bump_project_epoch.sql @@ -0,0 +1 @@ +INSERT INTO project_epoch (layer_stack_id, epoch) VALUES ($1, 1) ON CONFLICT (layer_stack_id) DO UPDATE SET epoch = epoch + 1 RETURNING 1; diff --git a/engine/crates/layerfs-persistence/sql/sqlite/queries/history/layerstack_deleted_ms.sql b/engine/crates/layerfs-persistence/sql/sqlite/queries/history/layerstack_deleted_ms.sql new file mode 100644 index 0000000000..860a613de8 --- /dev/null +++ b/engine/crates/layerfs-persistence/sql/sqlite/queries/history/layerstack_deleted_ms.sql @@ -0,0 +1 @@ +SELECT deleted_ms FROM layer_stack_retired WHERE layer_stack_id = $1; diff --git a/engine/crates/layerfs-persistence/sql/sqlite/queries/history/layerstack_insert_layer.sql b/engine/crates/layerfs-persistence/sql/sqlite/queries/history/layerstack_insert_layer.sql index c483c0caf3..7d908b00d7 100644 --- a/engine/crates/layerfs-persistence/sql/sqlite/queries/history/layerstack_insert_layer.sql +++ b/engine/crates/layerfs-persistence/sql/sqlite/queries/history/layerstack_insert_layer.sql @@ -1 +1 @@ -INSERT INTO layer (layer_id,layer_stack_id,parent_layer_id,root_id,source_branch_id,source_commit_id,logical, overwrote) VALUES ($1,$2,$3,$4,$5,$6,$7, $8) RETURNING 1; +INSERT INTO layer (layer_id,layer_stack_id,parent_layer_id,root_id,source_branch_id,source_commit_id,logical, overwrote, history_serial) VALUES ($1,$2,$3,$4,$5,$6,$7, $8, $9) RETURNING 1; diff --git a/engine/crates/layerfs-persistence/sql/sqlite/queries/history/layerstack_insert_project_epoch.sql b/engine/crates/layerfs-persistence/sql/sqlite/queries/history/layerstack_insert_project_epoch.sql new file mode 100644 index 0000000000..9aa115b2dd --- /dev/null +++ b/engine/crates/layerfs-persistence/sql/sqlite/queries/history/layerstack_insert_project_epoch.sql @@ -0,0 +1 @@ +INSERT INTO project_epoch (layer_stack_id, epoch) VALUES ($1, 0) RETURNING 1; diff --git a/engine/crates/layerfs-persistence/sql/sqlite/queries/history/layerstack_insert_retired.sql b/engine/crates/layerfs-persistence/sql/sqlite/queries/history/layerstack_insert_retired.sql new file mode 100644 index 0000000000..2bf5257bbe --- /dev/null +++ b/engine/crates/layerfs-persistence/sql/sqlite/queries/history/layerstack_insert_retired.sql @@ -0,0 +1 @@ +INSERT INTO layer_stack_retired (layer_stack_id, deleted_ms) VALUES ($1, $2) ON CONFLICT (layer_stack_id) DO NOTHING RETURNING 1; diff --git a/engine/crates/layerfs-persistence/sql/sqlite/queries/history/layerstack_stack_by_id.sql b/engine/crates/layerfs-persistence/sql/sqlite/queries/history/layerstack_stack_by_id.sql index bda0abe24a..688aa805c1 100644 --- a/engine/crates/layerfs-persistence/sql/sqlite/queries/history/layerstack_stack_by_id.sql +++ b/engine/crates/layerfs-persistence/sql/sqlite/queries/history/layerstack_stack_by_id.sql @@ -1 +1 @@ -SELECT layer_stack_id, name, scope_id, profile_id, head_layer_id FROM layer_stack WHERE layer_stack_id = $1; +SELECT layer_stack_id, name, scope_id, profile_id, head_layer_id FROM layer_stack WHERE layer_stack_id = $1 AND NOT EXISTS (SELECT 1 FROM layer_stack_retired r WHERE r.layer_stack_id = layer_stack.layer_stack_id); diff --git a/engine/crates/layerfs-persistence/sql/sqlite/queries/history/layerstack_stack_by_name.sql b/engine/crates/layerfs-persistence/sql/sqlite/queries/history/layerstack_stack_by_name.sql index 0d92910086..f4e015667c 100644 --- a/engine/crates/layerfs-persistence/sql/sqlite/queries/history/layerstack_stack_by_name.sql +++ b/engine/crates/layerfs-persistence/sql/sqlite/queries/history/layerstack_stack_by_name.sql @@ -1 +1 @@ -SELECT layer_stack_id, name, scope_id, profile_id, head_layer_id FROM layer_stack WHERE name = $1; +SELECT layer_stack_id, name, scope_id, profile_id, head_layer_id FROM layer_stack WHERE name = $1 AND NOT EXISTS (SELECT 1 FROM layer_stack_retired r WHERE r.layer_stack_id = layer_stack.layer_stack_id); diff --git a/engine/crates/layerfs-persistence/sql/sqlite/queries/history/layerstack_stack_first_page.sql b/engine/crates/layerfs-persistence/sql/sqlite/queries/history/layerstack_stack_first_page.sql index 6e4c239eec..1e9afc88ef 100644 --- a/engine/crates/layerfs-persistence/sql/sqlite/queries/history/layerstack_stack_first_page.sql +++ b/engine/crates/layerfs-persistence/sql/sqlite/queries/history/layerstack_stack_first_page.sql @@ -1 +1 @@ -SELECT layer_stack_id, name, scope_id, profile_id, head_layer_id FROM layer_stack ORDER BY name LIMIT $1; +SELECT layer_stack_id, name, scope_id, profile_id, head_layer_id FROM layer_stack WHERE NOT EXISTS (SELECT 1 FROM layer_stack_retired r WHERE r.layer_stack_id = layer_stack.layer_stack_id) ORDER BY name LIMIT $1; diff --git a/engine/crates/layerfs-persistence/sql/sqlite/queries/history/layerstack_stack_next_page.sql b/engine/crates/layerfs-persistence/sql/sqlite/queries/history/layerstack_stack_next_page.sql index 1bcd9ffd33..f8d2ed28f7 100644 --- a/engine/crates/layerfs-persistence/sql/sqlite/queries/history/layerstack_stack_next_page.sql +++ b/engine/crates/layerfs-persistence/sql/sqlite/queries/history/layerstack_stack_next_page.sql @@ -1 +1 @@ -SELECT layer_stack_id, name, scope_id, profile_id, head_layer_id FROM layer_stack WHERE name > $1 ORDER BY name LIMIT $2; +SELECT layer_stack_id, name, scope_id, profile_id, head_layer_id FROM layer_stack WHERE name > $1 AND NOT EXISTS (SELECT 1 FROM layer_stack_retired r WHERE r.layer_stack_id = layer_stack.layer_stack_id) ORDER BY name LIMIT $2; diff --git a/engine/crates/layerfs-persistence/sql/sqlite/queries/history/paths_advance_epoch.sql b/engine/crates/layerfs-persistence/sql/sqlite/queries/history/paths_advance_epoch.sql new file mode 100644 index 0000000000..15fa42d851 --- /dev/null +++ b/engine/crates/layerfs-persistence/sql/sqlite/queries/history/paths_advance_epoch.sql @@ -0,0 +1 @@ +UPDATE project_epoch SET epoch=epoch+1 WHERE layer_stack_id=$1 RETURNING 1; diff --git a/engine/crates/layerfs-persistence/sql/sqlite/queries/history/paths_marker.sql b/engine/crates/layerfs-persistence/sql/sqlite/queries/history/paths_marker.sql new file mode 100644 index 0000000000..049788375c --- /dev/null +++ b/engine/crates/layerfs-persistence/sql/sqlite/queries/history/paths_marker.sql @@ -0,0 +1 @@ +SELECT path,frontier_serial,removed_ms FROM path_removed WHERE layer_stack_id=$1 AND path_key=$2; diff --git a/engine/crates/layerfs-persistence/sql/sqlite/queries/history/paths_markers.sql b/engine/crates/layerfs-persistence/sql/sqlite/queries/history/paths_markers.sql new file mode 100644 index 0000000000..d3bd1e7c8a --- /dev/null +++ b/engine/crates/layerfs-persistence/sql/sqlite/queries/history/paths_markers.sql @@ -0,0 +1 @@ +SELECT path,frontier_serial,removed_ms FROM path_removed WHERE layer_stack_id=$1 ORDER BY path; diff --git a/engine/crates/layerfs-persistence/sql/sqlite/queries/history/paths_upsert.sql b/engine/crates/layerfs-persistence/sql/sqlite/queries/history/paths_upsert.sql new file mode 100644 index 0000000000..f9171232e1 --- /dev/null +++ b/engine/crates/layerfs-persistence/sql/sqlite/queries/history/paths_upsert.sql @@ -0,0 +1 @@ +INSERT INTO path_removed(layer_stack_id,path_key,path,frontier_serial,removed_ms) VALUES($1,$2,$3,$4,$5) ON CONFLICT(layer_stack_id,path_key) DO UPDATE SET path=excluded.path, frontier_serial=excluded.frontier_serial, removed_ms=excluded.removed_ms; diff --git a/engine/crates/layerfs-persistence/sql/sqlite/queries/history/position_high_water.sql b/engine/crates/layerfs-persistence/sql/sqlite/queries/history/position_high_water.sql new file mode 100644 index 0000000000..c76bf13a1e --- /dev/null +++ b/engine/crates/layerfs-persistence/sql/sqlite/queries/history/position_high_water.sql @@ -0,0 +1 @@ +SELECT next_history_serial - 1 FROM history_meta WHERE id=1; diff --git a/engine/crates/layerfs-persistence/sql/sqlite/queries/history/position_next_serial.sql b/engine/crates/layerfs-persistence/sql/sqlite/queries/history/position_next_serial.sql new file mode 100644 index 0000000000..a7f0c03ba9 --- /dev/null +++ b/engine/crates/layerfs-persistence/sql/sqlite/queries/history/position_next_serial.sql @@ -0,0 +1 @@ +UPDATE history_meta SET next_history_serial=next_history_serial+1 WHERE id=1 RETURNING next_history_serial-1; diff --git a/engine/crates/layerfs-persistence/sql/sqlite/queries/history/position_node_serial.sql b/engine/crates/layerfs-persistence/sql/sqlite/queries/history/position_node_serial.sql new file mode 100644 index 0000000000..5a3ff1e15c --- /dev/null +++ b/engine/crates/layerfs-persistence/sql/sqlite/queries/history/position_node_serial.sql @@ -0,0 +1 @@ +SELECT history_serial FROM "commit" WHERE commit_id=$1 UNION ALL SELECT history_serial FROM layer WHERE layer_id=$1 LIMIT 1; diff --git a/engine/crates/layerfs-persistence/sql/sqlite/queries/history/position_stack_high_water.sql b/engine/crates/layerfs-persistence/sql/sqlite/queries/history/position_stack_high_water.sql new file mode 100644 index 0000000000..fab8fefa98 --- /dev/null +++ b/engine/crates/layerfs-persistence/sql/sqlite/queries/history/position_stack_high_water.sql @@ -0,0 +1 @@ +SELECT COALESCE(MAX(history_serial),0) FROM (SELECT history_serial FROM "commit" WHERE layer_stack_id=$1 UNION ALL SELECT history_serial FROM layer WHERE layer_stack_id=$1) AS serials; diff --git a/engine/crates/layerfs-persistence/src/backend/postgresql/schema.rs b/engine/crates/layerfs-persistence/src/backend/postgresql/schema.rs index 329a1c5568..fa3115e6ba 100644 --- a/engine/crates/layerfs-persistence/src/backend/postgresql/schema.rs +++ b/engine/crates/layerfs-persistence/src/backend/postgresql/schema.rs @@ -9,10 +9,15 @@ //! tables are absent: group-row layouts are SQLite physical choices. A //! schema made by [`create`] also has Init's acquisition tables (#533 step //! 10b, `acquisition.sql`), recorded in its schema source; one an earlier -//! release imported from SQLite has none and refuses acquisition. A merge's +//! release imported from SQLite has none and refuses acquisition. A Store +//! written before the GC tables gains them at its first writable open +//! ([`add_gc`]); any other schema mismatch is still refused. A merge's //! records live in the writer connection's temporary tables //! (`merge_records.rs`), not here. -use super::session::{connection::Session, transaction::Transaction}; +use super::session::{ + connection::Session, + transaction::{Mode, Transaction}, +}; use crate::backend::records::{BackendError, Param}; use crate::PostgresCheck; use layerfs_history::{CatalogId, HistoryCatalogConfig}; @@ -22,7 +27,18 @@ use postgres::SimpleQueryRow; #[path = "schema_diagnostic.rs"] mod diagnostic; -pub(crate) const SCHEMA: &str = include_str!("../../../sql/postgresql/schema.sql"); +pub(crate) const SCHEMA: &str = concat!( + include_str!("../../../sql/postgresql/schema.sql"), + include_str!("../../../sql/postgresql/gc.sql") +); +/// The Store tables a schema had before the GC tables ([`GC`]) were appended. +const PRE_GC_SCHEMA: &str = include_str!("../../../sql/postgresql/schema.sql"); +/// The `history_meta` position column as `schema.sql` writes it; it sits on +/// its own line, so removing the line leaves the text an older Store recorded. +const NEXT_SERIAL: &str = " next_history_serial bigint NOT NULL DEFAULT 1,\n"; +/// The GC marker, position and admission tables; a Store written before them +/// gains them at open ([`add_gc`]). +const GC: &str = include_str!("../../../sql/postgresql/gc.sql"); /// Init's acquisition tables, created after [`SCHEMA`] in a new schema. const ACQUISITION: &str = include_str!("../../../sql/postgresql/acquisition.sql"); /// The acquisition tables, in name order. @@ -32,8 +48,37 @@ const ACQUISITION_TABLES: [&str; 3] = ["init_entry", "init_host_file", "init_ope /// (#91) and object locations in narrow rows under an 8-byte-prefix index /// (#90); a version 1 schema is refused at open, never converted. pub(crate) const SCHEMA_VERSION: i64 = 2; -/// Every table of one Store schema. -pub(crate) const TABLES: [&str; 19] = [ +/// Every table of one Store schema, in name order. +pub(crate) const TABLES: [&str; 26] = [ + "absorbed", + "binding_hint_block", + "branch", + "branch_retired", + "commit", + "commit_lineage", + "content_signature", + "file_digest", + "gc_dead", + "gc_intent", + "gc_op", + "history_meta", + "inode_value_block", + "layer", + "layer_lineage", + "layer_stack", + "layer_stack_retired", + "object_location", + "pack", + "path_removed", + "project_epoch", + "reach", + "scope_allocator", + "store_lane", + "store_policy", + "store_totals", +]; +/// The tables a Store schema had before the GC tables were appended. +const PRE_GC_TABLES: [&str; 19] = [ "absorbed", "binding_hint_block", "branch", @@ -77,6 +122,24 @@ fn source(acquisition: bool) -> String { } } +/// The recorded schema source of a Store written before the GC tables. +fn pre_gc_source(acquisition: bool) -> String { + match acquisition { + true => format!("{PRE_GC_SCHEMA}{ACQUISITION}"), + false => PRE_GC_SCHEMA.to_owned(), + } +} + +/// The current source before [`NEXT_SERIAL`] was written. +fn source_without_next_serial(acquisition: bool) -> String { + source(acquisition).replacen(NEXT_SERIAL, "", 1) +} + +/// The pre-GC source before [`NEXT_SERIAL`] was written. +fn pre_gc_source_without_next_serial(acquisition: bool) -> String { + pre_gc_source(acquisition).replacen(NEXT_SERIAL, "", 1) +} + /// Creates the new schema `tx.schema` with the Store and acquisition tables, /// its policy row and its catalog row, inside the caller's transaction. A /// schema of that name that already exists is refused before anything is @@ -126,6 +189,140 @@ pub(crate) fn create( Ok(()) } +/// Adds the GC marker, position and admission tables to a Store written +/// before them, then records the current schema: an additive migration under +/// the same conditions as the SQLite Store's `add_gc`. A schema that already +/// has `gc_intent` is left alone; a recorded source, table set or definition +/// that is not the pre-GC schema is refused, never migrated. +pub(crate) fn add_gc(session: &Session) -> Result<(), PersistenceError> { + // A Store that already has the GC tables is left alone: one statement, + // no transaction and no lane wait. + let present = session.run_as::<_, BackendError>(Mode::Statement, |tx| { + let sql = format!( + "SELECT 1 FROM information_schema.tables WHERE table_schema='{}' AND table_name='gc_intent' AND table_type='BASE TABLE'", + tx.schema + ); + Ok(!tx.query(&sql, vec![])?.is_empty()) + })?; + if present { + return Ok(()); + } + session.run_as(Mode::Write, |tx| -> Result<(), PersistenceError> { + // Re-check under the lane: another process may have migrated first. + let present = tx.query( + "SELECT 1 FROM information_schema.tables WHERE table_schema=current_schema() AND table_name='gc_intent' AND table_type='BASE TABLE'", + vec![], + )?; + if !present.is_empty() { + return Ok(()); + } + let rows = tx.query( + "SELECT schema_source,schema_definition FROM history_meta WHERE id=1", + vec![], + )?; + let [row] = rows.as_slice() else { + return Err(BackendError::Integrity.into()); + }; + let recorded_source: String = row.get(0)?; + let acquisition = if recorded_source == pre_gc_source(false) + || recorded_source == pre_gc_source_without_next_serial(false) + { + false + } else if recorded_source == pre_gc_source(true) + || recorded_source == pre_gc_source_without_next_serial(true) + { + true + } else { + return Err(BackendError::Integrity.into()); + }; + let tables = tx.query(TABLE_NAMES, vec![])?; + let actual: Vec = tables.iter().map(|r| r.get(0)).collect::>()?; + let mut expected = PRE_GC_TABLES.to_vec(); + if acquisition { + expected.extend(ACQUISITION_TABLES); + } + expected.sort_unstable(); + if !actual.iter().map(String::as_str).eq(expected) { + return Err(BackendError::Integrity.into()); + } + if row.get::(1)? != definition(tx)? { + return Err(BackendError::Integrity.into()); + } + tx.batch(GC)?; + let definition = definition(tx)?; + tx.query( + "UPDATE history_meta SET schema_source=$1,schema_definition=$2 WHERE id=1", + vec![Param::Text(source(acquisition)), Param::Text(definition)], + )?; + Ok(()) + }) +} + +/// Adds the position column `history_meta.next_history_serial` (T1.4) to a +/// Store written before it, then records the current schema: an additive +/// migration under the same conditions as [`add_gc`]. A schema that already +/// has the column is left alone; a recorded source or definition that is not +/// one of the earlier schemas is refused, never migrated. PostgreSQL has no +/// `commit`/`layer` serial columns yet, so only the column and one +/// `project_epoch` row per project are recorded here; the serial backfill is +/// SQLite's. +pub(crate) fn add_position_state(session: &Session) -> Result<(), PersistenceError> { + // A Store that already has the column is left alone: one statement, no + // transaction and no lane wait. + let present = session.run_as::<_, BackendError>(Mode::Statement, |tx| { + let sql = format!( + "SELECT 1 FROM information_schema.columns WHERE table_schema='{}' AND table_name='history_meta' AND column_name='next_history_serial'", + tx.schema + ); + Ok(!tx.query(&sql, vec![])?.is_empty()) + })?; + if present { + return Ok(()); + } + session.run_as(Mode::Write, |tx| -> Result<(), PersistenceError> { + // Re-check under the lane: another process may have migrated first. + let present = tx.query( + "SELECT 1 FROM information_schema.columns WHERE table_schema=current_schema() AND table_name='history_meta' AND column_name='next_history_serial'", + vec![], + )?; + if !present.is_empty() { + return Ok(()); + } + let rows = tx.query( + "SELECT schema_source,schema_definition FROM history_meta WHERE id=1", + vec![], + )?; + let [row] = rows.as_slice() else { + return Err(BackendError::Integrity.into()); + }; + let recorded_source: String = row.get(0)?; + let acquisition = if recorded_source == source(false) + || recorded_source == source_without_next_serial(false) + { + false + } else if recorded_source == source(true) + || recorded_source == source_without_next_serial(true) + { + true + } else { + return Err(BackendError::Integrity.into()); + }; + if row.get::(1)? != definition(tx)? { + return Err(BackendError::Integrity.into()); + } + tx.batch("ALTER TABLE history_meta ADD COLUMN next_history_serial bigint NOT NULL DEFAULT 1;")?; + tx.batch( + "INSERT INTO project_epoch(layer_stack_id, epoch) SELECT s.layer_stack_id, 0 FROM layer_stack s WHERE NOT EXISTS (SELECT 1 FROM project_epoch e WHERE e.layer_stack_id = s.layer_stack_id);", + )?; + let definition = definition(tx)?; + tx.query( + "UPDATE history_meta SET schema_source=$1,schema_definition=$2 WHERE id=1", + vec![Param::Text(source(acquisition)), Param::Text(definition)], + )?; + Ok(()) + }) +} + /// What an open's check read: the catalog's incarnation, whether the schema /// has the acquisition tables, the recorded binding, the frozen policy and /// the server's version. diff --git a/engine/crates/layerfs-persistence/src/backend/postgresql/schema_diagnostic.rs b/engine/crates/layerfs-persistence/src/backend/postgresql/schema_diagnostic.rs index eee4764b7f..dc4eb5ea1f 100644 --- a/engine/crates/layerfs-persistence/src/backend/postgresql/schema_diagnostic.rs +++ b/engine/crates/layerfs-persistence/src/backend/postgresql/schema_diagnostic.rs @@ -12,6 +12,8 @@ branch name text NO branch base_layer_id bytea NO branch forked_from_id bytea YES branch head_commit_id bytea YES +branch_retired branch_id bytea NO +branch_retired removed_ms bigint NO commit commit_id bytea NO commit layer_stack_id bytea NO commit root_id bytea NO @@ -41,12 +43,53 @@ content_signature signature bytea NO file_digest digest bytea NO file_digest length bigint NO file_digest root bytea NO +gc_dead kind text NO +gc_dead id bytea NO +gc_intent op_id bytea NO +gc_intent kind text NO +gc_intent scope_kind text NO +gc_intent scope_id bytea YES +gc_intent workspace text YES +gc_intent sandbox text YES +gc_intent machine text YES +gc_intent project text YES +gc_intent branch text YES +gc_intent owner_runtime text YES +gc_intent owner_boot text YES +gc_intent state text NO +gc_intent uncertain boolean NO +gc_intent phase text YES +gc_intent generation bigint YES +gc_intent artifact text YES +gc_intent high_water bigint YES +gc_intent evidence text YES +gc_intent updated_ms bigint NO +gc_op id bigint NO +gc_op generation bigint NO +gc_op phase text NO +gc_op scratch_schema text YES +gc_op mark_serial bigint YES +gc_op mark_pack_id bigint YES +gc_op mark_digest bytea YES +gc_op mark_totals bytea YES +gc_op mark_stamp bigint YES +gc_op started_ms bigint NO +gc_op dead_canonical bigint NO +gc_op dead_stored bigint NO +gc_op dead_metadata bigint NO +gc_op dead_logical bigint NO +gc_op totals_applied boolean NO +gc_op artifact text YES +gc_op pid bigint YES +gc_op backend_start text YES +gc_op incarnation bigint YES history_meta id bigint NO history_meta catalog_id bytea NO history_meta catalog_incarnation bigint NO history_meta binding_key bytea NO history_meta identity_format bigint NO history_meta schema_version bigint NO +history_meta next_history_serial bigint NO history_meta schema_source text NO history_meta schema_definition text NO init_entry operation_id bigint NO @@ -100,6 +143,8 @@ layer_stack name text NO layer_stack scope_id bytea NO layer_stack profile_id bytea NO layer_stack head_layer_id bytea NO +layer_stack_retired layer_stack_id bytea NO +layer_stack_retired deleted_ms bigint NO object_location pack_id bigint NO object_location canonical_length integer NO object_location role smallint NO @@ -111,6 +156,13 @@ pack domain bigint NO pack digest bytea NO pack length bigint NO pack body bytea NO +path_removed layer_stack_id bytea NO +path_removed path_key bytea NO +path_removed path bytea NO +path_removed frontier_serial bigint NO +path_removed removed_ms bigint NO +project_epoch layer_stack_id bytea NO +project_epoch epoch bigint NO reach key bytea NO reach x bigint NO reach height bigint NO diff --git a/engine/crates/layerfs-persistence/src/backend/sqlite/schema.rs b/engine/crates/layerfs-persistence/src/backend/sqlite/schema.rs index 5d87c51a72..188dcb1a9b 100644 --- a/engine/crates/layerfs-persistence/src/backend/sqlite/schema.rs +++ b/engine/crates/layerfs-persistence/src/backend/sqlite/schema.rs @@ -15,6 +15,9 @@ pub(crate) const LINEAGE: &str = include_str!("../../../sql/sqlite/lineage.sql") pub(crate) const SCRATCH: &str = include_str!("../../../sql/sqlite/merge_scratch.sql"); /// Whole-content file digests (#364); a Store written before them gains them at open. pub(crate) const DIGESTS: &str = include_str!("../../../sql/sqlite/file_digest.sql"); +/// GC markers, positions and admission records; a Store written before them +/// gains them at open. +pub(crate) const GC: &str = include_str!("../../../sql/sqlite/gc.sql"); const UNIT_INDEX: &str = include_str!("../../../sql/sqlite/objects_units_index.sql"); const UNITS: &str = include_str!("../../../sql/sqlite/objects_units.sql"); const ACQUISITION: &str = include_str!("../../../sql/sqlite/acquisition/schema.sql"); @@ -34,6 +37,7 @@ pub(super) fn scripts((layout, acquisition): Selection) -> Vec { LINEAGE.to_owned(), SCRATCH.to_owned(), DIGESTS.to_owned(), + GC.to_owned(), ]; if acquisition == crate::SqliteAcquisitionSchema::Tables { scripts.push(ACQUISITION.to_owned()); @@ -60,6 +64,156 @@ pub(crate) fn definition(c: &Connection, w: &RefCell) -> Result &'static str { + let start = HISTORY + .find(" next_history_serial INTEGER NOT NULL DEFAULT 1") + .expect("history schema holds the next serial column"); + let end = start + HISTORY[start..].find('\n').expect("column line ends") + 1; + &HISTORY[start..end] +} + +/// The `commit` and `layer` position column as `history.sql` writes it; it +/// appears once in each of the two tables. +fn history_serial_column() -> &'static str { + let start = HISTORY + .find(" history_serial INTEGER NOT NULL DEFAULT 0") + .expect("history schema holds the position column"); + let end = start + HISTORY[start..].find('\n').expect("column line ends") + 1; + &HISTORY[start..end] +} + +/// The `store_policy` position salt column as `metadata.sql` writes it. +fn trace_salt_column() -> &'static str { + let start = METADATA + .find(" trace_salt BLOB NOT NULL DEFAULT") + .expect("metadata schema holds the salt column"); + let end = start + METADATA[start..].find('\n').expect("column line ends") + 1; + &METADATA[start..end] +} + +/// `source` as a Store written before the position columns recorded it: the +/// two `history_serial` columns, `store_policy.trace_salt` and +/// `history_meta.next_history_serial` removed. The table migrations compare +/// against this so a Store from before those columns still matches its +/// recorded source. +fn without_position_columns(source: String) -> String { + source + .replacen(history_serial_column(), "", 2) + .replacen(trace_salt_column(), "", 1) + .replacen(next_history_serial_column(), "", 1) +} + +/// The ALTERs and salt backfill an old Store needs; existing rows keep serial +/// 0 and the Store keeps one random salt. +fn add_position_columns(tx: &Transaction<'_>) -> Result<(), BackendError> { + tx.query( + "ALTER TABLE \"commit\" ADD COLUMN history_serial INTEGER NOT NULL DEFAULT 0", + vec![], + )?; + tx.query( + "ALTER TABLE layer ADD COLUMN history_serial INTEGER NOT NULL DEFAULT 0", + vec![], + )?; + tx.query( + "ALTER TABLE store_policy ADD COLUMN trace_salt BLOB NOT NULL DEFAULT (X'0000000000000000000000000000000000000000000000000000000000000000') CHECK(length(trace_salt)=32)", + vec![], + )?; + tx.query( + "UPDATE store_policy SET trace_salt=randomblob(32) WHERE id=1", + vec![], + )?; + Ok(()) +} + +/// The serials and epochs a Store written before T1.4 needs: Commits take the +/// next Store serials in `(created_ms, commit_id)` order, then each project's +/// genesis Layer takes the next one in stack-id order; every other Layer +/// republishes its source Commit and carries that Commit's serial. Existing +/// serials are kept, `project_epoch` gets one zero row per project and +/// `next_history_serial` never falls below `MAX(serial) + 1`. +fn backfill_position_state(tx: &Transaction<'_>) -> Result<(), BackendError> { + let next: i64 = tx + .query( + "SELECT next_history_serial FROM history_meta WHERE id=1", + vec![], + )? + .first() + .ok_or(BackendError::Integrity)? + .get(0)?; + let commit_max: i64 = tx + .query( + "SELECT COALESCE(MAX(history_serial),0) FROM \"commit\"", + vec![], + )? + .first() + .ok_or(BackendError::Integrity)? + .get(0)?; + let layer_max: i64 = tx + .query("SELECT COALESCE(MAX(history_serial),0) FROM layer", vec![])? + .first() + .ok_or(BackendError::Integrity)? + .get(0)?; + let base = next.max(commit_max.max(layer_max) + 1); + tx.query( + &format!( + "WITH ordered(commit_id,serial) AS ( \ + SELECT commit_id,{base}+ROW_NUMBER() OVER (ORDER BY created_ms,commit_id)-1 \ + FROM \"commit\" WHERE history_serial=0) \ + UPDATE \"commit\" SET history_serial=( \ + SELECT ordered.serial FROM ordered \ + WHERE ordered.commit_id=\"commit\".commit_id) \ + WHERE history_serial=0" + ), + vec![], + )?; + let after_commits: i64 = tx + .query( + "SELECT COALESCE(MAX(history_serial),0)+1 FROM \"commit\"", + vec![], + )? + .first() + .ok_or(BackendError::Integrity)? + .get(0)?; + tx.query( + &format!( + "WITH ordered(layer_id,serial) AS ( \ + SELECT layer_id,{after_commits}+ROW_NUMBER() OVER ( \ + ORDER BY layer_stack_id,layer_id)-1 \ + FROM layer WHERE history_serial=0 AND parent_layer_id IS NULL) \ + UPDATE layer SET history_serial=( \ + SELECT ordered.serial FROM ordered \ + WHERE ordered.layer_id=layer.layer_id) \ + WHERE history_serial=0 AND parent_layer_id IS NULL" + ), + vec![], + )?; + tx.query( + "UPDATE layer SET history_serial=( \ + SELECT \"commit\".history_serial FROM \"commit\" \ + WHERE \"commit\".layer_stack_id=layer.layer_stack_id \ + AND \"commit\".commit_id=layer.source_commit_id) \ + WHERE history_serial=0 AND source_commit_id IS NOT NULL", + vec![], + )?; + tx.query( + "INSERT INTO project_epoch(layer_stack_id,epoch) \ + SELECT s.layer_stack_id,0 FROM layer_stack s \ + WHERE NOT EXISTS ( \ + SELECT 1 FROM project_epoch e WHERE e.layer_stack_id=s.layer_stack_id)", + vec![], + )?; + tx.query( + "UPDATE history_meta SET next_history_serial=MAX(next_history_serial,( \ + SELECT COALESCE(MAX(history_serial),0)+1 FROM ( \ + SELECT history_serial FROM \"commit\" \ + UNION ALL SELECT history_serial FROM layer))) WHERE id=1", + vec![], + )?; + Ok(()) +} pub(crate) fn check( c: &Connection, w: &RefCell, @@ -91,6 +245,13 @@ pub(crate) fn check( "commit_lineage", "content_signature", "file_digest", + "gc_dead", + "gc_intent", + "gc_op", + "project_epoch", + "branch_retired", + "layer_stack_retired", + "path_removed", "history_meta", "inode_value_block", "layer", @@ -137,13 +298,54 @@ impl super::session::connection::Session { for script in scripts(selection) { tx.bootstrap(&script)?; } tx.query(&format!("PRAGMA application_id={APPLICATION_ID}"),vec![])?; tx.query(&format!("PRAGMA user_version={version}"),vec![])?; - tx.query(&format!("INSERT INTO store_policy(id,schema_version,format_profile,small_file_threshold_bytes,whole_file_delta_max_depth,chunk_delta_max_depth,metadata_delta_max_depth) VALUES(1,{version},?1,?2,?3,?4,?5)"),vec![Param::I64(i64::from(policy.format_profile())),Param::I64(policy.small_file_threshold_bytes() as i64),Param::I64(i64::from(policy.whole_file_delta_max_depth())),Param::I64(i64::from(policy.chunk_delta_max_depth())),Param::I64(i64::from(policy.metadata_delta_max_depth()))])?; + tx.query(&format!("INSERT INTO store_policy(id,schema_version,format_profile,small_file_threshold_bytes,whole_file_delta_max_depth,chunk_delta_max_depth,metadata_delta_max_depth,trace_salt) VALUES(1,{version},?1,?2,?3,?4,?5,randomblob(32))"),vec![Param::I64(i64::from(policy.format_profile())),Param::I64(policy.small_file_threshold_bytes() as i64),Param::I64(i64::from(policy.whole_file_delta_max_depth())),Param::I64(i64::from(policy.chunk_delta_max_depth())),Param::I64(i64::from(policy.metadata_delta_max_depth()))])?; let definition=definition(tx.connection,tx.work)?; tx.query(&format!("INSERT INTO history_meta(id,catalog_id,catalog_incarnation,binding_key,identity_format,schema_version,schema_source,schema_definition) VALUES(1,?1,?2,?3,3,{version},?4,?5)"),vec![Param::Bytes(catalog_id.as_slice().to_vec()),Param::I64(history.incarnation as i64),Param::Bytes(history.binding_key.clone()),Param::Text(source(selection)),Param::Text(definition)])?; Ok(()) })?; Ok(()) } + /// Adds the three position columns T1.2a introduced to a Store written + /// before them: `commit.history_serial`, `layer.history_serial` (default 0) + /// and `store_policy.trace_salt` (32 random bytes). It is an additive + /// migration (REBUILD.md kind 2) under the same recorded-schema gate as + /// [`Self::add_binding_hints`]; because it runs before the GC tables are + /// added, the recorded source it matches is the live one with `GC` stripped. + pub(crate) fn add_gc_columns(&self) -> Result<(), BackendError> { + self.run::<_, BackendError>(true, |tx| { + let present = tx.query( + "SELECT 1 FROM pragma_table_info('commit') WHERE name='history_serial'", + vec![], + )?; + if !present.is_empty() { + return Ok(()); + } + let rows = tx.query( + "SELECT schema_source,schema_definition FROM history_meta WHERE id=1", + vec![], + )?; + let Some(row) = rows.first() else { + return Ok(()); + }; + let current = without_position_columns(source((self.layout, self.acquisition))) + .replacen(GC, "", 1); + if row.get::(0)? != current + || row.get::(1)? != definition(tx.connection, tx.work)? + { + return Ok(()); + } + add_position_columns(tx)?; + let definition = definition(tx.connection, tx.work)?; + tx.query( + "UPDATE history_meta SET schema_source=?1,schema_definition=?2 WHERE id=1", + vec![ + Param::Text(source((self.layout, self.acquisition)).replacen(GC, "", 1)), + Param::Text(definition), + ], + )?; + Ok(()) + }) + } /// Adds the binding hint table (#352 `49499abac`, blocks since #91) to a Store written /// before it existed, then records the current schema: an additive /// migration (REBUILD.md kind 2). Only a Store whose recorded schema is @@ -169,10 +371,11 @@ impl super::session::connection::Session { }; // The schema this migration records is the one before the // lineage and merge tables; later migrations add those. - let current = source((self.layout, self.acquisition)) + let current = without_position_columns(source((self.layout, self.acquisition))) .replacen(DIGESTS, "", 1) .replacen(SCRATCH, "", 1) - .replacen(LINEAGE, "", 1); + .replacen(LINEAGE, "", 1) + .replacen(GC, "", 1); let table = HISTORY .find("CREATE TABLE binding_hint_block (") .and_then(|start| Some((start, start + HISTORY[start..].find(";\n")? + 2))) @@ -216,9 +419,10 @@ impl super::session::connection::Session { let Some(row) = rows.first() else { return Ok(()); }; - let current = source((self.layout, self.acquisition)) + let current = without_position_columns(source((self.layout, self.acquisition))) .replacen(DIGESTS, "", 1) - .replacen(SCRATCH, "", 1); + .replacen(SCRATCH, "", 1) + .replacen(GC, "", 1); if row.get::(0)? != current.replacen(LINEAGE, "", 1) || row.get::(1)? != definition(tx.connection, tx.work)? { @@ -253,7 +457,9 @@ impl super::session::connection::Session { let Some(row) = rows.first() else { return Ok(()); }; - let current = source((self.layout, self.acquisition)).replacen(DIGESTS, "", 1); + let current = without_position_columns(source((self.layout, self.acquisition))) + .replacen(DIGESTS, "", 1) + .replacen(GC, "", 1); if row.get::(0)? != current.replacen(SCRATCH, "", 1) || row.get::(1)? != definition(tx.connection, tx.work)? { @@ -288,7 +494,8 @@ impl super::session::connection::Session { let Some(row) = rows.first() else { return Ok(()); }; - let current = source((self.layout, self.acquisition)); + let current = without_position_columns(source((self.layout, self.acquisition))) + .replacen(GC, "", 1); if row.get::(0)? != current.replacen(DIGESTS, "", 1) || row.get::(1)? != definition(tx.connection, tx.work)? { @@ -303,6 +510,84 @@ impl super::session::connection::Session { Ok(()) }) } + /// Adds the GC marker, position and admission tables to a Store written + /// before them, then records the current schema: an additive migration + /// under the same conditions as [`Self::add_binding_hints`]. + pub(crate) fn add_gc(&self) -> Result<(), BackendError> { + self.run::<_, BackendError>(true, |tx| { + let present = tx.query( + "SELECT 1 FROM sqlite_schema WHERE type='table' AND name='gc_intent'", + vec![], + )?; + if !present.is_empty() { + return Ok(()); + } + let rows = tx.query( + "SELECT schema_source,schema_definition FROM history_meta WHERE id=1", + vec![], + )?; + let Some(row) = rows.first() else { + return Ok(()); + }; + let current = source((self.layout, self.acquisition)); + if row.get::(0)? != current.replacen(GC, "", 1) + || row.get::(1)? != definition(tx.connection, tx.work)? + { + return Ok(()); + } + tx.bootstrap(GC)?; + let definition = definition(tx.connection, tx.work)?; + tx.query( + "UPDATE history_meta SET schema_source=?1,schema_definition=?2 WHERE id=1", + vec![Param::Text(current), Param::Text(definition)], + )?; + Ok(()) + }) + } + /// Adds the position state T1.4 introduced (`history_meta.next_history_serial`, + /// appended NOT NULL DEFAULT 1) and backfills the `commit`/`layer` serials + /// and `project_epoch` rows in the same write transaction: an additive + /// migration (REBUILD.md kind 2) under the same recorded-schema gate as + /// [`Self::add_gc_columns`]. `recorded == current` also matches a Store an + /// earlier step of this same open just brought forward, because those steps + /// record the current source before this column is appended. + pub(crate) fn add_position_state(&self) -> Result<(), BackendError> { + self.run::<_, BackendError>(true, |tx| { + let present = tx.query( + "SELECT 1 FROM pragma_table_info('history_meta') WHERE name='next_history_serial'", + vec![], + )?; + if !present.is_empty() { + return Ok(()); + } + let rows = tx.query( + "SELECT schema_source,schema_definition FROM history_meta WHERE id=1", + vec![], + )?; + let Some(row) = rows.first() else { + return Ok(()); + }; + let current = source((self.layout, self.acquisition)); + let before = current.replacen(next_history_serial_column(), "", 1); + let recorded = row.get::(0)?; + if (recorded != before && recorded != current) + || row.get::(1)? != definition(tx.connection, tx.work)? + { + return Ok(()); + } + tx.query( + "ALTER TABLE history_meta ADD COLUMN next_history_serial INTEGER NOT NULL DEFAULT 1", + vec![], + )?; + backfill_position_state(tx)?; + let definition = definition(tx.connection, tx.work)?; + tx.query( + "UPDATE history_meta SET schema_source=?1,schema_definition=?2 WHERE id=1", + vec![Param::Text(current), Param::Text(definition)], + )?; + Ok(()) + }) + } pub(crate) fn validate( &self, catalog_id: layerfs_history::CatalogId, diff --git a/engine/crates/layerfs-persistence/src/backend/sqlite/session/query.rs b/engine/crates/layerfs-persistence/src/backend/sqlite/session/query.rs index 5207567e69..345873410b 100644 --- a/engine/crates/layerfs-persistence/src/backend/sqlite/session/query.rs +++ b/engine/crates/layerfs-persistence/src/backend/sqlite/session/query.rs @@ -102,6 +102,18 @@ pub(crate) fn history(key: &str) -> Result<&'static str, BackendError> { "layerstack_stack_name_taken" => Ok(include_str!( "../../../../sql/sqlite/queries/history/layerstack_stack_name_taken.sql" )), + "layerstack_insert_project_epoch" => Ok(include_str!( + "../../../../sql/sqlite/queries/history/layerstack_insert_project_epoch.sql" + )), + "layerstack_bump_project_epoch" => Ok(include_str!( + "../../../../sql/sqlite/queries/history/layerstack_bump_project_epoch.sql" + )), + "layerstack_insert_retired" => Ok(include_str!( + "../../../../sql/sqlite/queries/history/layerstack_insert_retired.sql" + )), + "layerstack_deleted_ms" => Ok(include_str!( + "../../../../sql/sqlite/queries/history/layerstack_deleted_ms.sql" + )), "layerstack_stack_next_page" => Ok(include_str!( "../../../../sql/sqlite/queries/history/layerstack_stack_next_page.sql" )), @@ -171,6 +183,30 @@ pub(crate) fn history(key: &str) -> Result<&'static str, BackendError> { "removal_stack" => Ok(include_str!( "../../../../sql/sqlite/queries/history/removal_stack.sql" )), + "position_high_water" => Ok(include_str!( + "../../../../sql/sqlite/queries/history/position_high_water.sql" + )), + "position_stack_high_water" => Ok(include_str!( + "../../../../sql/sqlite/queries/history/position_stack_high_water.sql" + )), + "position_next_serial" => Ok(include_str!( + "../../../../sql/sqlite/queries/history/position_next_serial.sql" + )), + "position_node_serial" => Ok(include_str!( + "../../../../sql/sqlite/queries/history/position_node_serial.sql" + )), + "paths_marker" => Ok(include_str!( + "../../../../sql/sqlite/queries/history/paths_marker.sql" + )), + "paths_markers" => Ok(include_str!( + "../../../../sql/sqlite/queries/history/paths_markers.sql" + )), + "paths_upsert" => Ok(include_str!( + "../../../../sql/sqlite/queries/history/paths_upsert.sql" + )), + "paths_advance_epoch" => Ok(include_str!( + "../../../../sql/sqlite/queries/history/paths_advance_epoch.sql" + )), _ => Err(BackendError::Integrity), } } diff --git a/engine/crates/layerfs-persistence/src/history/catalog.rs b/engine/crates/layerfs-persistence/src/history/catalog.rs index bccd1c05d8..498ac93606 100644 --- a/engine/crates/layerfs-persistence/src/history/catalog.rs +++ b/engine/crates/layerfs-persistence/src/history/catalog.rs @@ -136,14 +136,6 @@ impl HistoryCatalog for HistoryProvider { self.read(|tx| super::lineage::overwritten(tx, layer, position, usize::from(limit))) } - fn published_layer( - &self, - branch: BranchId, - commit: CommitId, - ) -> HistoryResult> { - self.read_statement(|tx| layerstack::published_source(tx, branch, commit)) - } - fn layer_history( &self, request: &LayerHistoryRequest, @@ -182,6 +174,14 @@ impl HistoryCatalog for HistoryProvider { self.write(|tx| super::removal::remove_layerstack(tx, stack)) } + fn mark_stack_deleted(&self, stack: LayerStackId, deleted_ms: i64) -> HistoryResult<()> { + self.write(|tx| layerstack::mark_stack_deleted(tx, stack, deleted_ms)) + } + + fn stack_deleted_ms(&self, stack: LayerStackId) -> HistoryResult> { + self.read(|tx| layerstack::stack_deleted_ms(tx, stack)) + } + fn publish_lane(&self) -> HistoryResult { match &self.engine { crate::backend::Engine::Sqlite(session) => { @@ -209,4 +209,26 @@ impl HistoryCatalog for HistoryProvider { fn reserve_inodes(&self, request: &ReserveRequest) -> HistoryResult { self.write(|tx| allocation::reserve_inodes(tx, self.catalog_id, request)) } + + fn mark_path_removed( + &self, + stack: LayerStackId, + path: &[u8], + frontier_serial: u64, + removed_ms: i64, + ) -> HistoryResult<()> { + self.write(|tx| super::paths::mark(tx, stack, path, frontier_serial, removed_ms)) + } + + fn remove_path(&self, request: &PathRemovalRequest) -> HistoryResult { + self.write(|tx| super::paths::remove_path(tx, request)) + } + + fn path_removals(&self, stack: LayerStackId) -> HistoryResult> { + self.read(|tx| super::paths::markers(tx, stack)) + } + + fn node_history_serial(&self, node: NodeId) -> HistoryResult> { + self.read_statement(|tx| super::position::node_serial(tx, node)) + } } diff --git a/engine/crates/layerfs-persistence/src/history/commit.rs b/engine/crates/layerfs-persistence/src/history/commit.rs index 6c2cda501a..fc7120f3bc 100644 --- a/engine/crates/layerfs-persistence/src/history/commit.rs +++ b/engine/crates/layerfs-persistence/src/history/commit.rs @@ -206,6 +206,8 @@ pub(crate) fn new_commit( second_parent, meta, }; + // Every new Commit takes the next Store serial in this transaction. + let serial = super::position::next_serial(tx)?; // The insert names a new identity; an existing one is then verified. let inserted = tx .execute( @@ -232,6 +234,7 @@ pub(crate) fn new_commit( .map_err(|_| HistoryError::Capacity("metadata bytes"))?, super::facts::encode(&record.meta.lww, &record.meta.units)?, record.meta.displaced.map(|id| id.to_bytes().to_vec()), + i64::try_from(serial).map_err(|_| HistoryError::Capacity("history serials"))?, ], ) .map_err(sql)?; diff --git a/engine/crates/layerfs-persistence/src/history/layerstack.rs b/engine/crates/layerfs-persistence/src/history/layerstack.rs index c7465ab32e..661f911ac5 100644 --- a/engine/crates/layerfs-persistence/src/history/layerstack.rs +++ b/engine/crates/layerfs-persistence/src/history/layerstack.rs @@ -47,6 +47,14 @@ const STACK_HEAD: &str = "layerstack_stack_head"; const LAYER_BY_SOURCE: &str = "layerstack_layer_by_source"; +const INSERT_PROJECT_EPOCH: &str = "layerstack_insert_project_epoch"; + +const BUMP_PROJECT_EPOCH: &str = "layerstack_bump_project_epoch"; + +const INSERT_RETIRED: &str = "layerstack_insert_retired"; + +const DELETED_MS: &str = "layerstack_deleted_ms"; + /// Creates the genesis Layer and the stack in one transaction. pub(crate) fn initialize_layerstack( tx: &Transaction<'_>, @@ -59,6 +67,7 @@ pub(crate) fn initialize_layerstack( return Err(HistoryError::Integrity("LayerStack name")); } let genesis = LayerId::derive(request.stack, None, request.genesis_root); + let serial = super::position::next_serial(tx)?; tx.execute( INSERT_STACK, params![ @@ -70,6 +79,8 @@ pub(crate) fn initialize_layerstack( ], ) .map_err(sql)?; + tx.execute(INSERT_PROJECT_EPOCH, [request.stack.as_slice()]) + .map_err(sql)?; tx.execute( INSERT_LAYER, params![ @@ -81,6 +92,7 @@ pub(crate) fn initialize_layerstack( Option::>::None, i64::try_from(request.logical).map_err(|_| HistoryError::Capacity("logical bytes"))?, Vec::::new(), + i64::try_from(serial).map_err(|_| HistoryError::Capacity("history serials"))?, ], ) .map_err(sql)?; @@ -104,6 +116,41 @@ pub(crate) fn layer_stack_named( one(tx, STACK_BY_NAME, params![name.as_str()], stack_row) } +/// Records a project.delete marker in one write transaction: the project's +/// epoch is bumped with the marker, and the marker is the commit point. A +/// hidden or absent stack, and a repeat, are Missing(LayerStack); a repeat +/// writes nothing, so it cannot bump the epoch twice. +pub(crate) fn mark_stack_deleted( + tx: &Transaction<'_>, + stack: LayerStackId, + deleted_ms: i64, +) -> HistoryResult<()> { + if layer_stack(tx, stack)?.is_none() { + return Err(HistoryError::Missing(Missing::LayerStack)); + } + tx.execute(BUMP_PROJECT_EPOCH, [stack.as_slice()]) + .map_err(sql)?; + let marked = tx + .execute(INSERT_RETIRED, params![stack.as_slice(), deleted_ms]) + .map_err(sql)?; + if marked == 0 { + return Err(HistoryError::Missing(Missing::LayerStack)); + } + Ok(()) +} + +/// The deleted_ms a project.delete marker recorded. The same-home retry +/// reads it by the stack id its intent recorded before the marker; a live or +/// absent stack has none. +pub(crate) fn stack_deleted_ms( + tx: &Transaction<'_>, + stack: LayerStackId, +) -> HistoryResult> { + one(tx, DELETED_MS, [stack.as_slice()], |row| { + cell::(row, 0) + }) +} + /// Reads stacks in name order, one bounded page at a time. pub(crate) fn layer_stacks( tx: &Transaction<'_>, @@ -315,9 +362,8 @@ pub(crate) fn add_layer( actual: stack.head_layer, }); } - // Overwrite-only here: the new Layer sits on the stack head, whatever - // main gained since the Branch's base. `branch.publish` refuses that - // case before this call unless the caller passed `force` (#691). + // Overwrite-only: the new Layer sits on the stack head, whatever main + // gained since the Branch's base. if commit.root == layer_root(tx, stack.head_layer)? { return Ok(AddLayerOutcome::NoChanges { head: stack.head_layer, @@ -338,6 +384,9 @@ pub(crate) fn add_layer( source_commit: Some(request.commit), logical: commit.meta.statistics.logical, }; + // A Layer that republishes a Commit carries that Commit's serial. + let serial = + super::position::node_serial(tx, layerfs_history::NodeId::Commit(commit.id))?.unwrap_or(0); if !verify_layer(tx, &layer)? { tx.execute( INSERT_LAYER, @@ -352,6 +401,7 @@ pub(crate) fn add_layer( // Overwritten Layers are addressed by height (`overwritten`); // only Layers published before #353 list them here. Vec::::new(), + i64::try_from(serial).map_err(|_| HistoryError::Capacity("history serials"))?, ], ) .map_err(sql)?; @@ -381,7 +431,7 @@ pub(crate) fn add_layer( Ok(AddLayerOutcome::Added { layer }) } -pub(crate) fn published_source( +fn published_source( tx: &Transaction<'_>, branch: layerfs_history::identity::BranchId, commit: layerfs_history::identity::CommitId, diff --git a/engine/crates/layerfs-persistence/src/history/mod.rs b/engine/crates/layerfs-persistence/src/history/mod.rs index 966992d594..865402ca69 100644 --- a/engine/crates/layerfs-persistence/src/history/mod.rs +++ b/engine/crates/layerfs-persistence/src/history/mod.rs @@ -11,6 +11,8 @@ mod layerstack; mod lineage; mod lineage_backfill; mod merge; +mod paths; +mod position; mod provider; mod removal; mod rows; diff --git a/engine/crates/layerfs-persistence/src/history/paths.rs b/engine/crates/layerfs-persistence/src/history/paths.rs new file mode 100644 index 0000000000..6ab1ff46c7 --- /dev/null +++ b/engine/crates/layerfs-persistence/src/history/paths.rs @@ -0,0 +1,112 @@ +//! The `project.remove_file` marker: one `path_removed` row per project and +//! path key, advanced in one write transaction with the project's epoch. + +use super::position; +use super::rows::{cell, many, one, unsigned}; +use super::transaction::Transaction; +use super::{layerstack, params}; +use crate::backend::records::Record as Row; +use layerfs_history::error::{HistoryError, HistoryResult, Missing}; +use layerfs_history::identity::LayerStackId; +use layerfs_history::path::path_key; +use layerfs_history::records::{ + PathRemovalKind, PathRemovalOutcome, PathRemovalRecord, PathRemovalRequest, +}; + +const MARKER: &str = "paths_marker"; + +const MARKERS: &str = "paths_markers"; + +const UPSERT: &str = "paths_upsert"; + +const ADVANCE_EPOCH: &str = "paths_advance_epoch"; + +/// One marker row decoded from the Store. +fn marker(row: &Row) -> HistoryResult { + Ok(PathRemovalRecord { + path: cell(row, 0)?, + frontier_serial: unsigned(row, 1)?, + removed_ms: cell::(row, 2)?, + }) +} + +/// Inserts or advances one marker, without the UpToDate decision. +pub(crate) fn mark( + tx: &Transaction<'_>, + stack: LayerStackId, + path: &[u8], + frontier_serial: u64, + removed_ms: i64, +) -> HistoryResult<()> { + if path.is_empty() { + return Err(HistoryError::InvalidInput("removed path")); + } + let key = path_key(path); + let frontier = + i64::try_from(frontier_serial).map_err(|_| HistoryError::Capacity("path frontier"))?; + tx.execute( + UPSERT, + params![stack.as_slice(), key.as_slice(), path, frontier, removed_ms], + )?; + Ok(()) +} + +/// Applies one `project.remove_file` marker, or reports it current. +pub(crate) fn remove_path( + tx: &Transaction<'_>, + request: &PathRemovalRequest, +) -> HistoryResult { + if layerstack::layer_stack(tx, request.stack)?.is_none() { + return Err(HistoryError::Missing(Missing::LayerStack)); + } + let key = path_key(&request.path); + let stored = one( + tx, + MARKER, + [request.stack.as_slice(), key.as_slice()], + marker, + )?; + let stack_high = position::stack_high_water(tx, request.stack)?; + if let Some(stored) = &stored { + if stored.path != request.path { + return Err(HistoryError::Integrity("path marker key")); + } + if stack_high <= stored.frontier_serial { + return Ok(PathRemovalOutcome { + kind: PathRemovalKind::UpToDate, + stack: request.stack, + path: stored.path.clone(), + frontier_serial: stored.frontier_serial, + removed_ms: stored.removed_ms, + }); + } + } + // The new frontier is the Store high-water, read in this transaction. + let frontier = position::high_water(tx)?; + mark( + tx, + request.stack, + &request.path, + frontier, + request.removed_ms, + )?; + let advanced = tx.execute(ADVANCE_EPOCH, [request.stack.as_slice()])?; + if advanced != 1 { + return Err(HistoryError::Conflict("project epoch")); + } + Ok(PathRemovalOutcome { + kind: PathRemovalKind::Removed, + stack: request.stack, + path: request.path.clone(), + frontier_serial: frontier, + removed_ms: request.removed_ms, + }) +} + +/// Every marker of one Stack. +pub(crate) fn markers( + tx: &Transaction<'_>, + stack: LayerStackId, +) -> HistoryResult> { + many(tx, MARKERS, [stack.as_slice()], marker) +} diff --git a/engine/crates/layerfs-persistence/src/history/position.rs b/engine/crates/layerfs-persistence/src/history/position.rs new file mode 100644 index 0000000000..2f78dad153 --- /dev/null +++ b/engine/crates/layerfs-persistence/src/history/position.rs @@ -0,0 +1,47 @@ +//! The Store history position: the `history_meta.next_history_serial` counter +//! a `remove_file` frontier names, and the serial each Commit and Layer +//! records. +//! +//! Every new Commit takes `next_history_serial` and advances the counter in +//! the same write transaction; a Layer that republishes a Commit carries that +//! Commit's serial. The high-water a marker stores is `next_history_serial - +//! 1`, read in the marker transaction, so a node created concurrently is +//! ordered by its serial and never by a later read. + +use super::params; +use super::rows::{one, unsigned}; +use super::transaction::Transaction; +use layerfs_history::error::{HistoryError, HistoryResult}; +use layerfs_history::identity::{LayerStackId, NodeId}; + +const HIGH_WATER: &str = "position_high_water"; + +const NEXT_SERIAL: &str = "position_next_serial"; + +const STACK_HIGH_WATER: &str = "position_stack_high_water"; + +const NODE_SERIAL: &str = "position_node_serial"; + +/// The Store high-water: `next_history_serial - 1`. +pub(crate) fn high_water(tx: &Transaction<'_>) -> HistoryResult { + one(tx, HIGH_WATER, params![], |row| unsigned(row, 0))? + .ok_or(HistoryError::Integrity("history position")) +} + +/// Takes the next Store serial, advancing `next_history_serial` in this +/// transaction. +pub(crate) fn next_serial(tx: &Transaction<'_>) -> HistoryResult { + one(tx, NEXT_SERIAL, params![], |row| unsigned(row, 0))? + .ok_or(HistoryError::Integrity("history position")) +} + +/// The largest `history_serial` in one Stack, for the `UpToDate` test. +pub(crate) fn stack_high_water(tx: &Transaction<'_>, stack: LayerStackId) -> HistoryResult { + one(tx, STACK_HIGH_WATER, [stack.as_slice()], |row| unsigned(row, 0))? + .ok_or(HistoryError::Integrity("history position")) +} + +/// The `history_serial` one Commit or Layer records, when it exists. +pub(crate) fn node_serial(tx: &Transaction<'_>, node: NodeId) -> HistoryResult> { + one(tx, NODE_SERIAL, [node.as_slice()], |row| unsigned(row, 0)) +} diff --git a/engine/crates/layerfs-persistence/src/store/open.rs b/engine/crates/layerfs-persistence/src/store/open.rs index 1acc83aad3..213707bcc7 100644 --- a/engine/crates/layerfs-persistence/src/store/open.rs +++ b/engine/crates/layerfs-persistence/src/store/open.rs @@ -95,10 +95,20 @@ impl SqliteHandles { config.sqlite_acquisition, )?); if writable { + // A Store at the released 2.0 table set recorded its schema + // without the position columns; it gains them before any table + // migration runs. + session.add_gc_columns()?; session.add_binding_hints()?; session.add_lineage(crate::history::backfill_lineage)?; session.add_merge_scratch()?; session.add_file_digests()?; + // A Store an earlier migration just brought forward recorded its + // source without the position columns, so it gains them here, + // before `add_gc` records the full schema. + session.add_gc_columns()?; + session.add_gc()?; + session.add_position_state()?; } let handles = Self::validated(session, directory, binding, cursor_key)?; if writable { diff --git a/engine/crates/layerfs-persistence/src/store/postgresql.rs b/engine/crates/layerfs-persistence/src/store/postgresql.rs index beb4c18c25..85741559e2 100644 --- a/engine/crates/layerfs-persistence/src/store/postgresql.rs +++ b/engine/crates/layerfs-persistence/src/store/postgresql.rs @@ -1,7 +1,9 @@ //! A PostgreSQL Store (#506): explicit creation and open. //! //! The same shared history and Metadata semantics as the SQLite Store run -//! over the PostgreSQL engine. Opening never creates or migrates a schema; +//! over the PostgreSQL engine. Opening never creates a schema; a writable +//! open adds the GC tables to a Store written before them (additive, as on +//! SQLite), and every other mismatch is refused. //! [`PostgresHandles::create`] makes a new empty one with Init's acquisition //! tables (#533 step 10b). Projects reach it only by Init; nothing converts //! a SQLite Store (step 10c). @@ -177,6 +179,10 @@ impl PostgresHandles { } CatalogId::derive(binding).map_err(|_| PersistenceError::Malformed)?; let session = Arc::new(Session::connect(selection, writable)?); + if writable { + schema::add_gc(&session)?; + schema::add_position_state(&session)?; + } Self::validated(session, selection.check, Some(binding), cursor_key) .map(|(_, handles)| handles) } diff --git a/engine/crates/layerfs-persistence/tests/gc_backfill.rs b/engine/crates/layerfs-persistence/tests/gc_backfill.rs new file mode 100644 index 0000000000..aa65bd1518 --- /dev/null +++ b/engine/crates/layerfs-persistence/tests/gc_backfill.rs @@ -0,0 +1,492 @@ +//! T1.4: a Store written before the position state opens writable and gains +//! `history_meta.next_history_serial`, the backfilled `commit`/`layer` +//! serials and one `project_epoch` row per project. A Store that already has +//! the column is left alone, and a serial already recorded is never lowered. +#![cfg(any(target_os = "macos", target_os = "linux"))] +mod support; + +use layerfs_history::{ + AddLayerOutcome, AddLayerRequest, BranchId, CommitMeta, CommitOutcome, CommitRecord, + CommitRequest, CommitVia, ForkRequest, ForkSource, HistoryCatalog, HistoryCatalogConfig, + LayerRecord, LayerStackId, StackInitialization, +}; +use layerfs_persistence::{PersistenceConfig, SqliteHandles, SqlitePersistenceProfile}; +use layerfs_storage::StoragePolicy; +use support::{branch_id, name, root, stack, Temp, BINDING}; + +/// The `history_meta` position column as `history.sql` writes it. +const NEXT_SERIAL: &str = " next_history_serial INTEGER NOT NULL DEFAULT 1,\n"; + +fn config(path: &std::path::Path) -> PersistenceConfig { + PersistenceConfig::sqlite(path).with_sqlite_profile(SqlitePersistenceProfile::Disposable) +} + +fn create(path: &std::path::Path) -> SqliteHandles { + SqliteHandles::create( + config(path), + StoragePolicy::frozen_default(), + &HistoryCatalogConfig { + binding_key: BINDING.to_vec(), + cursor_key: [71; 32], + incarnation: 7, + }, + ) + .expect("Store creation") +} + +fn writable( + path: &std::path::Path, +) -> Result { + SqliteHandles::open_writable(config(path), BINDING, [71; 32]) +} + +fn definition(sql: &rusqlite::Connection) -> String { + let mut statement = sql + .prepare("SELECT type,name,sql FROM sqlite_schema WHERE substr(name,1,7)!='sqlite_' ORDER BY type,name") + .unwrap(); + let rows = statement + .query_map([], |r| Ok([r.get::<_, String>(0)?, r.get(1)?, r.get(2)?])) + .unwrap(); + rows.map(|r| r.unwrap().map(|v| v + "\n").concat()) + .collect() +} + +/// Rewrites a fresh Store into the format written before T1.4: no +/// `history_meta.next_history_serial`, every Commit and Layer at serial 0 and +/// no `project_epoch` row, recorded as that revision wrote it. Returns the +/// fresh source string it replaced so a test can assert the migrated Store +/// records the current schema again. +fn strip_position_state(path: &std::path::Path) -> String { + let sql = rusqlite::Connection::open(path).unwrap(); + let fresh: String = sql + .query_row("SELECT schema_source FROM history_meta", [], |r| r.get(0)) + .unwrap(); + sql.execute_batch( + "ALTER TABLE history_meta DROP COLUMN next_history_serial; \ + UPDATE \"commit\" SET history_serial=0; \ + UPDATE layer SET history_serial=0; \ + DELETE FROM project_epoch;", + ) + .unwrap(); + let old = fresh.replacen(NEXT_SERIAL, "", 1); + assert_ne!(old, fresh, "a fresh Store holds the position column"); + sql.execute( + "UPDATE history_meta SET schema_source=?1,schema_definition=?2", + rusqlite::params![old, definition(&sql)], + ) + .unwrap(); + fresh +} + +fn initialize(handles: &SqliteHandles, tag: u8) -> layerfs_history::LayerStackRecord { + handles + .history + .initialize_layerstack(&StackInitialization { + logical: 10, + stack: stack(tag), + name: name(&format!("project-{tag}")), + scope: root(200), + profile: root(201), + genesis_root: root(1), + }) + .unwrap() +} + +fn fork( + handles: &SqliteHandles, + stack: LayerStackId, + branch: BranchId, + tag: u8, + source: ForkSource, +) { + handles + .history + .fork(&ForkRequest { + stack, + branch, + name: name(&format!("branch-{tag}")), + source, + }) + .unwrap(); +} + +fn commit(handles: &SqliteHandles, branch: BranchId, tag: u8) -> CommitRecord { + let snapshot = handles.history.branch(branch).unwrap().unwrap(); + let expected_root = match snapshot.head_commit { + Some(head) => handles.history.commit(head).unwrap().unwrap().root, + None => { + handles + .history + .layer(snapshot.base_layer) + .unwrap() + .unwrap() + .root + } + }; + match handles + .history + .commit_branch(&CommitRequest { + branch, + expected_head: snapshot.head_commit, + expected_base: snapshot.base_layer, + expected_root, + candidate_root: root(tag), + profile: root(201), + scope: root(200), + second_parent: None, + meta: CommitMeta::new(CommitVia::Files), + force: false, + }) + .unwrap() + { + CommitOutcome::Committed { record, .. } => *record, + other => panic!("commit: {other:?}"), + } +} + +fn publish(handles: &SqliteHandles, stack: LayerStackId, branch: BranchId) -> LayerRecord { + let branch_record = handles.history.branch(branch).unwrap().unwrap(); + let stack_record = handles.history.layer_stack(stack).unwrap().unwrap(); + match handles + .history + .add_layer(&AddLayerRequest { + stack, + branch, + commit: branch_record.head_commit.expect("a committed Branch"), + expected_stack_head: stack_record.head_layer, + expected_branch_base: branch_record.base_layer, + }) + .unwrap() + { + AddLayerOutcome::Added { layer } => layer, + other => panic!("publish: {other:?}"), + } +} + +fn column_default(sql: &rusqlite::Connection, table: &str, column: &str) -> (i64, Option) { + sql.query_row( + "SELECT \"notnull\",dflt_value FROM pragma_table_info(?1) WHERE name=?2", + rusqlite::params![table, column], + |r| Ok((r.get(0)?, r.get(1)?)), + ) + .unwrap() +} + +fn next_serial(sql: &rusqlite::Connection) -> i64 { + sql.query_row("SELECT next_history_serial FROM history_meta", [], |r| { + r.get(0) + }) + .unwrap() +} + +fn epochs(sql: &rusqlite::Connection) -> Vec<(Vec, i64)> { + let mut statement = sql + .prepare("SELECT layer_stack_id,epoch FROM project_epoch ORDER BY layer_stack_id") + .unwrap(); + statement + .query_map([], |r| Ok((r.get(0)?, r.get(1)?))) + .unwrap() + .map(|r| r.unwrap()) + .collect() +} + +#[test] +fn an_empty_store_keeps_its_next_serial_and_gains_no_epoch() { + let t = Temp::new("gc-backfill-empty"); + let path = t.join("db"); + drop(create(&path)); + let fresh = strip_position_state(&path); + + writable(&path).expect("a pre-T1.4 Store opens writable"); + drop(writable(&path).expect("an upgraded Store reopens")); + + let sql = rusqlite::Connection::open(&path).unwrap(); + assert_eq!( + column_default(&sql, "history_meta", "next_history_serial"), + (1, Some("1".to_owned())), + "the column is appended NOT NULL DEFAULT 1" + ); + assert_eq!(next_serial(&sql), 1, "no history keeps the serial at 1"); + assert!( + epochs(&sql).is_empty(), + "a Store with no project has no epoch" + ); + let (source, recorded): (String, String) = sql + .query_row( + "SELECT schema_source,schema_definition FROM history_meta", + [], + |r| Ok((r.get(0)?, r.get(1)?)), + ) + .unwrap(); + assert_eq!(source, fresh, "the Store records the current schema"); + assert_eq!( + recorded, + definition(&sql), + "the recorded definition is the live one" + ); + drop(sql); + support::reopen(&path); +} + +#[test] +fn commits_layers_and_epochs_are_backfilled_in_deterministic_order() { + let t = Temp::new("gc-backfill-graph"); + let path = t.join("db"); + let fresh; + let (stack_one, stack_two); + { + let handles = create(&path); + let one = initialize(&handles, 1); + let two = initialize(&handles, 2); + let a = branch_id(1); + fork(&handles, one.id, a, 1, ForkSource::Layer(one.head_layer)); + commit(&handles, a, 10); + publish(&handles, one.id, a); + commit(&handles, a, 11); + publish(&handles, one.id, a); + + let b = branch_id(2); + fork(&handles, two.id, b, 2, ForkSource::Layer(two.head_layer)); + commit(&handles, b, 20); + publish(&handles, two.id, b); + stack_one = one.id; + stack_two = two.id; + drop(handles); + fresh = strip_position_state(&path); + } + + writable(&path).expect("a pre-T1.4 Store opens writable"); + let sql = rusqlite::Connection::open(&path).unwrap(); + + // Commits take consecutive Store serials in (created_ms, commit_id) order. + let mut commits: Vec<(Vec, i64, i64)> = { + let mut statement = sql + .prepare("SELECT commit_id,created_ms,history_serial FROM \"commit\"") + .unwrap(); + let rows = statement + .query_map([], |r| Ok((r.get(0)?, r.get(1)?, r.get(2)?))) + .unwrap(); + rows.map(|r| r.unwrap()).collect() + }; + commits.sort_by(|a, b| (a.1, &a.0).cmp(&(b.1, &b.0))); + let ordered: Vec = commits.iter().map(|c| c.2).collect(); + assert_eq!( + ordered, + (1..=ordered.len() as i64).collect::>(), + "serials are dense and follow the deterministic order" + ); + + // Per project the genesis Layer takes the next serial, in stack-id order. + let genesis_one: i64 = sql + .query_row( + "SELECT history_serial FROM layer WHERE parent_layer_id IS NULL AND layer_stack_id=?1", + rusqlite::params![stack_one.as_slice()], + |r| r.get(0), + ) + .unwrap(); + let genesis_two: i64 = sql + .query_row( + "SELECT history_serial FROM layer WHERE parent_layer_id IS NULL AND layer_stack_id=?1", + rusqlite::params![stack_two.as_slice()], + |r| r.get(0), + ) + .unwrap(); + assert_eq!(genesis_one.min(genesis_two), commits.len() as i64 + 1); + assert_eq!(genesis_one.max(genesis_two), commits.len() as i64 + 2); + + // Every republished tree carries the serial of the Commit it publishes. + for (commit_id, _, serial) in &commits { + let layer: i64 = sql + .query_row( + "SELECT history_serial FROM layer WHERE source_commit_id=?1", + rusqlite::params![commit_id], + |r| r.get(0), + ) + .unwrap(); + assert_eq!(layer, *serial, "a Layer carries its source Commit serial"); + } + assert_eq!( + next_serial(&sql), + commits.len() as i64 + 3, + "the next serial follows the highest assigned one" + ); + + // One epoch row per project, both zero. + let rows = epochs(&sql); + assert_eq!(rows.len(), 2); + let mut expected = vec![stack_one, stack_two]; + expected.sort_by(|a, b| a.as_slice().cmp(b.as_slice())); + for (row, expected) in rows.iter().zip(expected.iter()) { + assert_eq!(row.0, expected.as_slice()); + assert_eq!(row.1, 0); + } + + let before = next_serial(&sql); + drop(sql); + drop(writable(&path).expect("reopening changes nothing")); + let sql = rusqlite::Connection::open(&path).unwrap(); + assert_eq!(next_serial(&sql), before, "reopening keeps the serials"); + assert_eq!(epochs(&sql).len(), 2); + let (source, recorded): (String, String) = sql + .query_row( + "SELECT schema_source,schema_definition FROM history_meta", + [], + |r| Ok((r.get(0)?, r.get(1)?)), + ) + .unwrap(); + assert_eq!(source, fresh); + assert_eq!(recorded, definition(&sql)); +} + +#[test] +fn republishing_layers_share_their_source_commit_serial() { + let t = Temp::new("gc-backfill-republish"); + let path = t.join("db"); + let (shared_commit, first_layer, second_layer); + { + let handles = create(&path); + let one = initialize(&handles, 1); + let a = branch_id(1); + fork(&handles, one.id, a, 1, ForkSource::Layer(one.head_layer)); + commit(&handles, a, 10); + let first = publish(&handles, one.id, a); + // Advance the stack so publishing the older Commit again is not the + // no-change shortcut. + commit(&handles, a, 11); + let advanced = publish(&handles, one.id, a); + + let b = branch_id(2); + fork( + &handles, + one.id, + b, + 2, + ForkSource::Commit { + branch: a, + commit: first.source_commit.unwrap(), + }, + ); + // Re-publish the Commit that already has a Layer on branch `a`. + let second = publish(&handles, one.id, b); + assert_ne!(first.id, second.id); + assert_ne!(advanced.id, second.id); + assert_eq!(first.source_commit, second.source_commit); + shared_commit = first.source_commit.unwrap(); + first_layer = first.id; + second_layer = second.id; + drop(handles); + strip_position_state(&path); + } + + writable(&path).expect("a pre-T1.4 Store opens writable"); + let sql = rusqlite::Connection::open(&path).unwrap(); + let serial: i64 = sql + .query_row( + "SELECT history_serial FROM \"commit\" WHERE commit_id=?1", + rusqlite::params![shared_commit.as_slice()], + |r| r.get(0), + ) + .unwrap(); + for layer in [first_layer, second_layer] { + let layer_serial: i64 = sql + .query_row( + "SELECT history_serial FROM layer WHERE layer_id=?1", + rusqlite::params![layer.as_slice()], + |r| r.get(0), + ) + .unwrap(); + assert_eq!( + layer_serial, serial, + "a republishing Layer carries its source Commit serial" + ); + } + assert_eq!(epochs(&sql).len(), 1); + drop(sql); +} + +#[test] +fn existing_serials_are_kept_and_new_ones_start_above_the_maximum() { + let t = Temp::new("gc-backfill-monotonic"); + let path = t.join("db"); + let (kept_commit, other_commit); + { + let handles = create(&path); + let one = initialize(&handles, 1); + let a = branch_id(1); + fork(&handles, one.id, a, 1, ForkSource::Layer(one.head_layer)); + let kept = commit(&handles, a, 10); + publish(&handles, one.id, a); + let other = commit(&handles, a, 11); + publish(&handles, one.id, a); + kept_commit = kept.id; + other_commit = other.id; + drop(handles); + strip_position_state(&path); + } + + // A Store brought part-way forward may already hold serials; they are kept + // and the backfill starts above the highest one. + { + let sql = rusqlite::Connection::open(&path).unwrap(); + sql.execute( + "UPDATE \"commit\" SET history_serial=100 WHERE commit_id=?1", + rusqlite::params![kept_commit.as_slice()], + ) + .unwrap(); + // A Layer that already holds a serial is not rewritten either. + sql.execute( + "UPDATE layer SET history_serial=7 WHERE source_commit_id=?1", + rusqlite::params![kept_commit.as_slice()], + ) + .unwrap(); + drop(sql); + } + + writable(&path).expect("a part-way Store opens writable"); + let sql = rusqlite::Connection::open(&path).unwrap(); + let kept: i64 = sql + .query_row( + "SELECT history_serial FROM \"commit\" WHERE commit_id=?1", + rusqlite::params![kept_commit.as_slice()], + |r| r.get(0), + ) + .unwrap(); + assert_eq!(kept, 100, "an assigned Commit serial is never rewritten"); + let assigned: i64 = sql + .query_row( + "SELECT history_serial FROM \"commit\" WHERE commit_id=?1", + rusqlite::params![other_commit.as_slice()], + |r| r.get(0), + ) + .unwrap(); + assert_eq!(assigned, 101, "new serials start above the maximum"); + let existing_layer: i64 = sql + .query_row( + "SELECT history_serial FROM layer WHERE source_commit_id=?1", + rusqlite::params![kept_commit.as_slice()], + |r| r.get(0), + ) + .unwrap(); + assert_eq!( + existing_layer, 7, + "an assigned Layer serial is never rewritten" + ); + let genesis: i64 = sql + .query_row( + "SELECT history_serial FROM layer WHERE parent_layer_id IS NULL", + [], + |r| r.get(0), + ) + .unwrap(); + assert_eq!(genesis, 102, "a genesis Layer follows the commits"); + assert_eq!(next_serial(&sql), 103); + + // A high-water written later is never lowered by another open. + sql.execute("UPDATE history_meta SET next_history_serial=500", []) + .unwrap(); + drop(sql); + drop(writable(&path).expect("a forward Store reopens")); + let sql = rusqlite::Connection::open(&path).unwrap(); + assert_eq!(next_serial(&sql), 500); + drop(sql); +} diff --git a/engine/crates/layerfs-persistence/tests/old_format_columns.rs b/engine/crates/layerfs-persistence/tests/old_format_columns.rs new file mode 100644 index 0000000000..ff4da595c7 --- /dev/null +++ b/engine/crates/layerfs-persistence/tests/old_format_columns.rs @@ -0,0 +1,154 @@ +//! A Store written before `commit`/`layer.history_serial`, +//! `store_policy.trace_salt` and `history_meta.next_history_serial` existed +//! opens writable and gains them (T1.2b, T1.4): the additive steps match the +//! recorded schema source and definition, give existing rows serial 0, the +//! Store a random salt and the next serial 1, and record the current schema. +mod support; +use layerfs_persistence::{PersistenceConfig, SqliteHandles, SqlitePersistenceProfile}; +use support::*; + +const GC: &str = include_str!("../sql/sqlite/gc.sql"); +/// The `commit` and `layer` column as `history.sql` writes it; it appears +/// once in each of the two tables. +const HISTORY_SERIAL: &str = " history_serial INTEGER NOT NULL DEFAULT 0,\n"; +/// The `store_policy` column as `metadata.sql` writes it. +const TRACE_SALT: &str = " trace_salt BLOB NOT NULL DEFAULT (X'0000000000000000000000000000000000000000000000000000000000000000') CHECK(length(trace_salt)=32)\n"; +/// The `history_meta` next serial column as `history.sql` writes it (T1.4). +const NEXT_SERIAL: &str = " next_history_serial INTEGER NOT NULL DEFAULT 1,\n"; + +fn definition(sql: &rusqlite::Connection) -> String { + let mut statement = sql + .prepare("SELECT type,name,sql FROM sqlite_schema WHERE substr(name,1,7)!='sqlite_' ORDER BY type,name") + .unwrap(); + let rows = statement + .query_map([], |r| Ok([r.get::<_, String>(0)?, r.get(1)?, r.get(2)?])) + .unwrap(); + rows.map(|r| r.unwrap().map(|v| v + "\n").concat()) + .collect() +} + +/// Rewrites a fresh Store into the format written before the GC schema and +/// before the position columns existed: the seven GC tables and the three +/// columns are gone, and the recorded schema is what that revision wrote. +fn make_old(path: &std::path::Path) { + let sql = rusqlite::Connection::open(path).unwrap(); + sql.execute_batch( + "ALTER TABLE \"commit\" DROP COLUMN history_serial; \ + ALTER TABLE layer DROP COLUMN history_serial; \ + ALTER TABLE store_policy DROP COLUMN trace_salt; \ + ALTER TABLE history_meta DROP COLUMN next_history_serial; \ + DROP TABLE branch_retired; DROP TABLE layer_stack_retired; DROP TABLE path_removed; \ + DROP TABLE project_epoch; DROP TABLE gc_intent; DROP TABLE gc_op; DROP TABLE gc_dead;", + ) + .unwrap(); + let source: String = sql + .query_row("SELECT schema_source FROM history_meta", [], |r| r.get(0)) + .unwrap(); + assert_eq!(source.matches(HISTORY_SERIAL).count(), 2); + assert_eq!(source.matches(TRACE_SALT).count(), 1); + assert_eq!(source.matches(NEXT_SERIAL).count(), 1); + let old = source + .replacen(HISTORY_SERIAL, "", 2) + .replacen(TRACE_SALT, "", 1) + .replacen(NEXT_SERIAL, "", 1) + .replacen(GC, "", 1); + assert_ne!(old, source, "the current schema holds the columns and GC"); + sql.execute( + "UPDATE history_meta SET schema_source=?1, schema_definition=?2", + rusqlite::params![old, definition(&sql)], + ) + .unwrap(); +} + +fn writable( + path: &std::path::Path, +) -> Result { + SqliteHandles::open_writable( + PersistenceConfig::sqlite(path).with_sqlite_profile(SqlitePersistenceProfile::Disposable), + BINDING, + [71; 32], + ) +} + +#[test] +fn an_old_format_store_gains_the_position_columns_and_a_salt() { + let t = Temp::new("old-format-columns"); + let path = t.join("db"); + let fresh_source: String = { + drop(create(&path)); + let sql = rusqlite::Connection::open(&path).unwrap(); + sql.query_row("SELECT schema_source FROM history_meta", [], |r| r.get(0)) + .unwrap() + }; + make_old(&path); + // Read-only authority cannot alter, so it still refuses. + assert!(sqlite::open_read_only(&path, BINDING, [71; 32]).is_err()); + + let handles = writable(&path).expect("an old-format Store opens writable"); + drop(handles); + + let sql = rusqlite::Connection::open(&path).unwrap(); + for (table, column) in [ + ("commit", "history_serial"), + ("layer", "history_serial"), + ("store_policy", "trace_salt"), + ] { + let present: i64 = sql + .query_row( + "SELECT count(*) FROM pragma_table_info(?1) WHERE name=?2", + rusqlite::params![table, column], + |r| r.get(0), + ) + .unwrap(); + assert_eq!(present, 1, "{table}.{column} exists after the migration"); + } + for table in ["commit", "layer"] { + let default: Option = sql + .query_row( + "SELECT dflt_value FROM pragma_table_info(?1) WHERE name='history_serial'", + rusqlite::params![table], + |r| r.get(0), + ) + .unwrap(); + assert_eq!( + default.as_deref(), + Some("0"), + "{table}.history_serial defaults to 0" + ); + } + let (length, salt): (i64, String) = sql + .query_row( + "SELECT length(trace_salt),hex(trace_salt) FROM store_policy WHERE id=1", + [], + |r| Ok((r.get(0).unwrap(), r.get(1).unwrap())), + ) + .unwrap(); + assert_eq!(length, 32, "the salt is 32 bytes"); + assert_ne!( + salt, + "00".repeat(32), + "the salt is random, not the zero default" + ); + let (source, recorded): (String, String) = sql + .query_row( + "SELECT schema_source,schema_definition FROM history_meta", + [], + |r| Ok((r.get(0).unwrap(), r.get(1).unwrap())), + ) + .unwrap(); + assert_eq!( + source, fresh_source, + "the migrated Store records the current schema" + ); + // SQLite rewrites the CREATE TABLE text on ADD COLUMN, so a migrated + // table's definition differs from a fresh Store's in whitespace only; + // `validate` requires the recorded definition to be the live one. + assert_eq!( + recorded, + definition(&sql), + "the recorded definition is the live definition" + ); + drop(sql); + reopen(&path); + writable(&path).expect("a migrated Store reopens"); +} diff --git a/engine/crates/layerfs-persistence/tests/old_format_open.rs b/engine/crates/layerfs-persistence/tests/old_format_open.rs index 7bcf056a27..d5fba96983 100644 --- a/engine/crates/layerfs-persistence/tests/old_format_open.rs +++ b/engine/crates/layerfs-persistence/tests/old_format_open.rs @@ -10,6 +10,14 @@ const HISTORY: &str = include_str!("../sql/sqlite/history.sql"); const LINEAGE: &str = include_str!("../sql/sqlite/lineage.sql"); const SCRATCH: &str = include_str!("../sql/sqlite/merge_scratch.sql"); const DIGESTS: &str = include_str!("../sql/sqlite/file_digest.sql"); +const GC: &str = include_str!("../sql/sqlite/gc.sql"); + +/// The three position columns T1.2a added, as the schemas write them; the old +/// format this fixture builds lacks them. +const HISTORY_SERIAL: &str = " history_serial INTEGER NOT NULL DEFAULT 0,\n"; +const TRACE_SALT: &str = " trace_salt BLOB NOT NULL DEFAULT (X'0000000000000000000000000000000000000000000000000000000000000000') CHECK(length(trace_salt)=32)\n"; +/// The `history_meta` next serial column as `history.sql` writes it (T1.4). +const NEXT_SERIAL: &str = " next_history_serial INTEGER NOT NULL DEFAULT 1,\n"; /// The table's definition as the history schema writes it, with its trailing /// blank line: the text an old Store's recorded schema source lacks. @@ -32,24 +40,34 @@ fn definition(sql: &rusqlite::Connection) -> String { /// Rewrites a fresh Store into the format the daemons before `49499abac` /// wrote: no `binding_hint` table (nor the later #353 lineage and #355 merge -/// tables), and the +/// tables), no position columns (including T1.4's next serial), and the /// schema source and definition they recorded. fn make_old(path: &std::path::Path) { let sql = rusqlite::Connection::open(path).unwrap(); sql.execute_batch( - "DROP TABLE binding_hint_block; DROP TABLE layer_lineage; DROP TABLE commit_lineage; \ + "ALTER TABLE \"commit\" DROP COLUMN history_serial; \ + ALTER TABLE layer DROP COLUMN history_serial; \ + ALTER TABLE store_policy DROP COLUMN trace_salt; \ + ALTER TABLE history_meta DROP COLUMN next_history_serial; \ + DROP TABLE binding_hint_block; DROP TABLE layer_lineage; DROP TABLE commit_lineage; \ DROP TABLE absorbed; DROP TABLE reach; DROP TABLE merge_operation; \ - DROP TABLE merge_record; DROP TABLE file_digest;", + DROP TABLE merge_record; DROP TABLE file_digest; \ + DROP TABLE branch_retired; DROP TABLE layer_stack_retired; DROP TABLE path_removed; \ + DROP TABLE project_epoch; DROP TABLE gc_intent; DROP TABLE gc_op; DROP TABLE gc_dead;", ) .unwrap(); let source: String = sql .query_row("SELECT schema_source FROM history_meta", [], |r| r.get(0)) .unwrap(); let old = source + .replacen(HISTORY_SERIAL, "", 2) + .replacen(TRACE_SALT, "", 1) + .replacen(NEXT_SERIAL, "", 1) .replacen(&hint_table(), "", 1) .replacen(LINEAGE, "", 1) .replacen(SCRATCH, "", 1) - .replacen(DIGESTS, "", 1); + .replacen(DIGESTS, "", 1) + .replacen(GC, "", 1); assert_ne!(old, source, "the history schema holds the table"); sql.execute( "UPDATE history_meta SET schema_source=?1, schema_definition=?2", diff --git a/engine/crates/layerfs-persistence/tests/path_removal.rs b/engine/crates/layerfs-persistence/tests/path_removal.rs new file mode 100644 index 0000000000..6074771ff2 --- /dev/null +++ b/engine/crates/layerfs-persistence/tests/path_removal.rs @@ -0,0 +1,172 @@ +//! `project.remove_file`: one marker, its Store-high-water frontier, the +//! `UpToDate` rule, the `/`-boundary subtree, and the per-project scope. +mod support; + +use layerfs_history::error::Missing; +use layerfs_history::path::NormalizedPath; +use layerfs_history::records::{ + CommitMeta, CommitOutcome, CommitRecord, CommitRequest, CommitVia, ForkRequest, ForkSource, + PathRemovalKind, PathRemovalRequest, StackInitialization, +}; +use layerfs_history::{BranchId, HistoryCatalog, HistoryError, LayerStackId, NodeId}; +use support::{branch_id, name, root, stack, Temp}; + +const REMOVED_AT: i64 = 1_700_000_000_000; + +fn project(history: &dyn HistoryCatalog, n: u8, text: &str) -> (LayerStackId, BranchId) { + let record = history + .initialize_layerstack(&StackInitialization { + logical: 0, + stack: stack(n), + name: name(text), + scope: root(200), + profile: root(201), + genesis_root: root(n), + }) + .unwrap(); + let branch = branch_id(n); + history + .fork(&ForkRequest { + stack: stack(n), + branch, + name: name("work"), + source: ForkSource::Layer(record.head_layer), + }) + .unwrap(); + (stack(n), branch) +} + +fn commit(history: &dyn HistoryCatalog, branch: BranchId, tag: u8) -> CommitRecord { + let record = history.branch(branch).unwrap().unwrap(); + let current = match record.head_commit { + Some(head) => history.commit(head).unwrap().unwrap().root, + None => history.layer(record.base_layer).unwrap().unwrap().root, + }; + let outcome = history + .commit_branch(&CommitRequest { + branch, + expected_head: record.head_commit, + expected_base: record.base_layer, + expected_root: current, + candidate_root: root(tag), + profile: root(201), + scope: root(200), + second_parent: None, + meta: CommitMeta::new(CommitVia::Files), + force: false, + }) + .unwrap(); + let CommitOutcome::Committed { record, .. } = outcome else { + panic!("{outcome:?}"); + }; + *record +} + +fn remove( + history: &dyn HistoryCatalog, + stack: LayerStackId, + path: &str, + at: i64, +) -> layerfs_history::PathRemovalOutcome { + history + .remove_path(&PathRemovalRequest { + stack, + path: NormalizedPath::new(path).unwrap().as_bytes().to_vec(), + removed_ms: at, + }) + .unwrap() +} + +#[test] +fn marker_frontier_and_up_to_date_follow_the_store_position() { + let temp = Temp::new("path-removal"); + let history = support::create(&temp.join("store")); + let (shop, branch) = project(&history, 1, "shop"); + let genesis = history.layer_stack(shop).unwrap().unwrap().head_layer; + assert_eq!( + history.node_history_serial(NodeId::Layer(genesis)).unwrap(), + Some(1) + ); + let first = commit(&history, branch, 5); + assert_eq!( + history + .node_history_serial(NodeId::Commit(first.id)) + .unwrap(), + Some(2) + ); + let _second = commit(&history, branch, 6); + + let removed = remove(&history, shop, "/a", REMOVED_AT); + assert_eq!(removed.kind, PathRemovalKind::Removed); + assert_eq!(removed.stack, shop); + assert_eq!(removed.path, b"a"); + assert_eq!( + removed.frontier_serial, 3, + "frontier is next_history_serial - 1" + ); + assert_eq!(removed.removed_ms, REMOVED_AT); + + // No project node is above the marker frontier: the repeat is UpToDate + // and keeps the stored values. + let repeat = remove(&history, shop, "/a", REMOVED_AT + 1); + assert_eq!(repeat.kind, PathRemovalKind::UpToDate); + assert_eq!(repeat.frontier_serial, 3); + assert_eq!(repeat.removed_ms, REMOVED_AT); + + // A commit in another project raises the Store high-water but leaves + // this path UpToDate: it is not this project's node. + let (_, api) = project(&history, 2, "api"); + let _api_commit = commit(&history, api, 7); + let still = remove(&history, shop, "/a", REMOVED_AT + 2); + assert_eq!(still.kind, PathRemovalKind::UpToDate); + assert_eq!(still.frontier_serial, 3); + + // A new commit in this project advances the path. + let _third = commit(&history, branch, 8); + let advanced = remove(&history, shop, "/a", REMOVED_AT + 3); + assert_eq!(advanced.kind, PathRemovalKind::Removed); + assert_eq!(advanced.frontier_serial, 6); + + // One marker per (stack, path key); `a` and `ab` are different paths. + let ab = remove(&history, shop, "/ab", REMOVED_AT + 4); + assert_eq!(ab.kind, PathRemovalKind::Removed); + assert_eq!(ab.frontier_serial, 6); + let markers = history.path_removals(shop).unwrap(); + let paths: Vec<_> = markers.iter().map(|marker| marker.path.clone()).collect(); + assert_eq!(paths, vec![b"a".to_vec(), b"ab".to_vec()]); + assert_eq!(markers[0].frontier_serial, 6); + assert_eq!(markers[1].frontier_serial, 6); + // `ab` is not governed by the `a` marker. + assert!(!layerfs_history::path::governs(b"a", b"ab")); +} + +#[test] +fn a_raw_marker_frontier_also_yields_up_to_date() { + let temp = Temp::new("path-removal-raw"); + let history = support::create(&temp.join("store")); + let (shop, _branch) = project(&history, 1, "shop"); + history + .mark_path_removed(shop, b"old", 42, REMOVED_AT) + .unwrap(); + let stored = history.path_removals(shop).unwrap(); + assert_eq!(stored.len(), 1); + assert_eq!(stored[0].frontier_serial, 42); + let current = remove(&history, shop, "old", REMOVED_AT + 1); + assert_eq!(current.kind, PathRemovalKind::UpToDate); + assert_eq!(current.frontier_serial, 42); + assert_eq!(current.removed_ms, REMOVED_AT); +} + +#[test] +fn an_unknown_project_is_missing() { + let temp = Temp::new("path-removal-missing"); + let history = support::create(&temp.join("store")); + assert_eq!( + history.remove_path(&PathRemovalRequest { + stack: stack(9), + path: b"a".to_vec(), + removed_ms: REMOVED_AT, + }), + Err(HistoryError::Missing(Missing::LayerStack)) + ); +} diff --git a/engine/crates/layerfs-persistence/tests/project_delete.rs b/engine/crates/layerfs-persistence/tests/project_delete.rs new file mode 100644 index 0000000000..bdb25f3be1 --- /dev/null +++ b/engine/crates/layerfs-persistence/tests/project_delete.rs @@ -0,0 +1,207 @@ +//! project.delete: the layer_stack_retired marker, the visibility filter every +//! project lookup carries, and the retry classification the Store side needs. +mod support; + +use layerfs_history::error::Missing; +use layerfs_history::{ + HistoryCatalog, HistoryError, LayerStackId, LayerStackRecord, Page, StackInitialization, +}; +use support::{name, root, stack, Temp}; + +fn project(history: &dyn HistoryCatalog, n: u8, text: &str) -> LayerStackRecord { + history + .initialize_layerstack(&StackInitialization { + logical: 10, + stack: stack(n), + name: name(text), + scope: root(200), + profile: root(201), + genesis_root: root(1), + }) + .unwrap() +} + +fn projects(history: &dyn HistoryCatalog) -> Vec { + let mut out = Vec::new(); + let mut page = Page { + cursor: None, + limit: 1, + }; + loop { + let found = history.layer_stacks(&page).unwrap(); + out.extend( + found + .records + .into_iter() + .map(|record| record.name.as_str().to_owned()), + ); + match found.continuation { + Some(cursor) => page.cursor = Some(cursor), + None => return out, + } + } +} + +fn epoch(path: &std::path::Path, id: LayerStackId) -> i64 { + let connection = rusqlite::Connection::open(path).unwrap(); + connection + .query_row( + "SELECT epoch FROM project_epoch WHERE layer_stack_id = ?1", + [id.as_slice()], + |row| row.get(0), + ) + .unwrap() +} + +#[test] +fn deleting_a_project_hides_it_and_marks_once() { + let temp = Temp::new("project-delete"); + let path = temp.join("store"); + let history = support::create(&path); + let shop = project(&history, 1, "shop"); + let api = project(&history, 2, "api"); + assert_eq!(projects(&history), ["api", "shop"]); + + history.mark_stack_deleted(shop.id, 1234).unwrap(); + assert_eq!(history.stack_deleted_ms(shop.id).unwrap(), Some(1234)); + + // Hidden from every project lookup, and only from the deleted one. + assert_eq!(history.layer_stack(shop.id).unwrap(), None); + assert_eq!(history.layer_stack_named(&name("shop")).unwrap(), None); + assert_eq!(projects(&history), ["api"]); + assert!(history.layer_stack(api.id).unwrap().is_some()); + assert!(history.layer_stack_named(&name("api")).unwrap().is_some()); + + // A repeat writes nothing: the marker and its deleted_ms stay, and an + // absent stack is the same typed absence. + assert_eq!( + history.mark_stack_deleted(shop.id, 999), + Err(HistoryError::Missing(Missing::LayerStack)) + ); + assert_eq!(history.stack_deleted_ms(shop.id).unwrap(), Some(1234)); + assert_eq!( + history.mark_stack_deleted(stack(9), 1), + Err(HistoryError::Missing(Missing::LayerStack)) + ); + drop(history); + + // The marker transaction bumped the project epoch exactly once. + assert_eq!(epoch(&path, shop.id), 1); + assert_eq!(epoch(&path, api.id), 0); +} + +#[test] +fn retry_classifies_retired_live_and_absent() { + let temp = Temp::new("project-delete-retry"); + let history = support::create(&temp.join("store")); + let shop = project(&history, 1, "shop"); + + // Live with no marker: the retry redoes the marker with its own deleted_ms. + assert_eq!(history.stack_deleted_ms(shop.id).unwrap(), None); + assert!(history.layer_stack(shop.id).unwrap().is_some()); + + // Retired: the retry reuses the recorded deleted_ms. + history.mark_stack_deleted(shop.id, 77).unwrap(); + assert_eq!(history.stack_deleted_ms(shop.id).unwrap(), Some(77)); + assert_eq!(history.layer_stack(shop.id).unwrap(), None); + + // Absent: neither a live row nor a marker, so the caller finishes its entry. + assert_eq!(history.stack_deleted_ms(stack(9)).unwrap(), None); + assert_eq!(history.layer_stack(stack(9)).unwrap(), None); +} + +#[test] +fn paging_skips_a_hidden_project() { + let temp = Temp::new("project-delete-paging"); + let history = support::create(&temp.join("store")); + for (n, text) in [(1, "a"), (2, "b"), (3, "c")] { + project(&history, n, text); + } + history.mark_stack_deleted(stack(2), 5).unwrap(); + // Two windows of one record skip the hidden middle project. + let mut page = Page { + cursor: None, + limit: 1, + }; + let mut names = Vec::new(); + loop { + let found = history.layer_stacks(&page).unwrap(); + names.extend( + found + .records + .into_iter() + .map(|record| record.name.as_str().to_owned()), + ); + match found.continuation { + Some(cursor) => page.cursor = Some(cursor), + None => break, + } + } + assert_eq!(names, ["a", "c"]); +} + +#[test] +fn every_layer_stack_lookup_carries_the_marker_filter() { + for (file, sql) in [ + ( + "layerstack_stack_by_id.sql", + include_str!("../sql/sqlite/queries/history/layerstack_stack_by_id.sql"), + ), + ( + "layerstack_stack_by_name.sql", + include_str!("../sql/sqlite/queries/history/layerstack_stack_by_name.sql"), + ), + ( + "layerstack_stack_first_page.sql", + include_str!("../sql/sqlite/queries/history/layerstack_stack_first_page.sql"), + ), + ( + "layerstack_stack_next_page.sql", + include_str!("../sql/sqlite/queries/history/layerstack_stack_next_page.sql"), + ), + ] { + assert!(sql.contains("layer_stack_retired"), "{file}: {sql}"); + assert!(sql.contains("NOT EXISTS"), "{file}: {sql}"); + } + // PostgreSQL overrides the by-name statement because it names the schema + // in one statement; it carries the same filter. + let postgresql = include_str!("../sql/postgresql/queries/history/layerstack_stack_by_name.sql"); + assert!(postgresql.contains("layer_stack_retired"), "{postgresql}"); + assert!(postgresql.contains("NOT EXISTS"), "{postgresql}"); + // The name-collision check is allow-listed: it must see a retired holder + // so project.init/fork can answer NameTaken with its state. + let taken = include_str!("../sql/sqlite/queries/history/layerstack_stack_name_taken.sql"); + assert!(!taken.contains("layer_stack_retired"), "{taken}"); +} + +#[test] +fn doctor_accepts_a_retired_stack() { + use layerfs_persistence::diagnostic::Outcome; + let fixture = support::doctor::Fixture::new(); + let shop = project(&fixture.handles.history, 1, "shop"); + fixture + .handles + .history + .mark_stack_deleted(shop.id, 1234) + .unwrap(); + let observations = fixture.audit(false); + assert!( + !observations + .iter() + .any(|observation| observation.outcome == Outcome::Fail), + "{observations:?}" + ); + // The marker table joins the audited schema and the history is still whole. + for id in [ + "store.schema", + "store.database.foreign_keys", + "store.history.summary", + ] { + assert!( + observations + .iter() + .any(|observation| observation.id == id && observation.outcome == Outcome::Pass), + "missing {id}: {observations:?}" + ); + } +} diff --git a/engine/crates/layerfs-persistence/tests/sqlite_publication.rs b/engine/crates/layerfs-persistence/tests/sqlite_publication.rs index faf287ca8d..3700b91c85 100644 --- a/engine/crates/layerfs-persistence/tests/sqlite_publication.rs +++ b/engine/crates/layerfs-persistence/tests/sqlite_publication.rs @@ -111,8 +111,9 @@ fn wal_full_settings_and_unavailable_engine_are_explicit() { .unwrap(); // #327 dropped workspace_stage and added store_totals; #352 added binding_hint; // #353 added layer_lineage, commit_lineage, absorbed and reach; #355 - // added merge_operation and merge_record; #364 added file_digest. - assert_eq!(names, 20); + // added merge_operation and merge_record; #364 added file_digest; + // the GC work added the markers, project_epoch, gc_intent, gc_op and gc_dead. + assert_eq!(names, 27); } #[test] fn final_body_multirow_locators_and_first_wins_have_one_atomic_ack() { diff --git a/engine/crates/layerfs-store-service/src/history/branch.rs b/engine/crates/layerfs-store-service/src/history/branch.rs index df583fd841..f2519b5131 100644 --- a/engine/crates/layerfs-store-service/src/history/branch.rs +++ b/engine/crates/layerfs-store-service/src/history/branch.rs @@ -4,8 +4,8 @@ //! `b`. When one side contains the other it is a fast-forward to the newer //! node; otherwise the sides are merged against their merge base in one Save, //! with `a`'s `/.lww`, and a clean result becomes one Commit with two parents. -//! A conflict creates nothing. Publishing is a compare-and-swap on main's -//! head by default, and reports the main Layers a forced publication skipped. +//! A conflict creates nothing. Publishing is overwrite-only and reports +//! the main Layers it skipped. use super::lookup::{ self as args, branch_node, commit_value, hex, history, invalid, layer_value, node_text, refuse, root, short, Answer, @@ -17,9 +17,9 @@ use layerfs_content::{ merge::{merge_roots, MergeInput, MergeUnit, RulesFrom, Units}, }; use layerfs_history::{ - AddLayerOutcome, AddLayerRequest, BranchId, BranchMergeRequest, CommitHistoryRequest, CommitId, - CommitMeta, CommitVia, ForkRequest, ForkSource, HistoryError, LayerHistoryRequest, LayerId, - LayerStackRecord, MergeCommit, MergeHead, MergeSide, NodeId, MAXIMUM_PAGE_RECORDS, + AddLayerOutcome, AddLayerRequest, BranchId, BranchMergeRequest, CommitHistoryRequest, + CommitMeta, CommitVia, ForkRequest, ForkSource, LayerHistoryRequest, LayerId, LayerStackRecord, + MergeCommit, MergeHead, MergeSide, NodeId, MAXIMUM_PAGE_RECORDS, }; use layerfs_workspace::InodeSerials; use std::sync::Arc; @@ -660,23 +660,12 @@ pub fn merge( } /// Publishes `commit` (the Branch's head when absent) of `branch` as main's -/// next Layer. -/// -/// By default the publication is refused `HeadMoved` when it would replace a -/// Layer main gained since the Branch forked; a Branch may publish again over -/// its own earlier Layers. `expected_base` and `expected_head` (the Branch's -/// head), when given, are strict compare-and-swap expectations, and `force` -/// deliberately keeps the older overwrite-only behavior. A Commit already -/// published answers `UpToDate` whichever check would refuse it, and a tree -/// main already holds answers `NoChanges`. Nothing is written on a refusal. +/// next Layer: overwrite-only. pub fn publish( store: &Arc, stack: &LayerStackRecord, branch: &layerfs_history::BranchRecord, commit: Option, - expected_base: Option, - expected_head: Option, - force: bool, ) -> Answer { let commit = match commit { Some(node) => match node { @@ -691,41 +680,6 @@ pub fn publish( ) })?, }; - if let Some(expected) = expected_head { - if branch.head_commit != Some(expected) { - return Err(moved_head(branch, expected, branch.head_commit)); - } - } - // A caller-named base is a strict compare-and-swap: main must still be - // the Layer the caller read. Without one, publishing is allowed while it - // cannot drop a Layer main gained since the Branch forked: main is still - // at the fork point, already holds this tree, or its head is this - // Branch's own earlier publication. - let refusal = match expected_base { - Some(expected) if stack.head_layer != expected => { - Some(moved_base(expected, stack.head_layer)) - } - _ if !force && !safe_to_publish(store, stack, branch, commit)? => { - Some(skipped(branch, branch.base_layer, stack.head_layer)) - } - _ => None, - }; - if let Some(refusal) = refusal { - // A Commit already on main answers `UpToDate`, as `add_layer` would: - // repeating a publication after main moved on is not a stale one. - return match store - .history() - .published_layer(branch.id, commit) - .map_err(history)? - { - Some(layer) => Ok(up_to_date(layer)), - None => Err(refusal), - }; - } - // The stack head read here is checked in `add_layer`'s write transaction, - // so a publication that lands between the check above and the write is - // not overwritten: it is refused instead. `force` asks for the previous - // overwrite-only behavior, still CAS'd against a concurrent publication. let outcome = store .history() .add_layer(&AddLayerRequest { @@ -735,15 +689,7 @@ pub fn publish( expected_stack_head: stack.head_layer, expected_branch_base: branch.base_layer, }) - .map_err(|error| match (error.cause(), expected_base) { - (HistoryError::StackMoved { actual, .. }, Some(expected)) => { - moved_base(expected, *actual) - } - (HistoryError::StackMoved { actual, .. }, None) if !force => { - skipped(branch, branch.base_layer, *actual) - } - _ => history(error), - })?; + .map_err(history)?; Ok(match outcome { AddLayerOutcome::Added { layer } => { // The overwritten Layers are read after the publication, one @@ -771,7 +717,10 @@ pub fn publish( ("overwroteHasMore", SdkValue::Bool(more)), ]) } - AddLayerOutcome::UpToDate { layer } => up_to_date(layer), + AddLayerOutcome::UpToDate { layer } => SdkValue::map([ + ("kind", SdkValue::text("UpToDate")), + ("layer", SdkValue::text(hex(layer.as_slice()))), + ]), AddLayerOutcome::NoChanges { head } => SdkValue::map([ ("kind", SdkValue::text("NoChanges")), ("headLayer", SdkValue::text(hex(head.as_slice()))), @@ -779,109 +728,6 @@ pub fn publish( }) } -/// The answer for a Commit already published as `layer`. -fn up_to_date(layer: LayerId) -> SdkValue { - SdkValue::map([ - ("kind", SdkValue::text("UpToDate")), - ("layer", SdkValue::text(hex(layer.as_slice()))), - ]) -} - -/// Whether publishing `commit` cannot drop a Layer `main` gained since the -/// Branch forked: main is at the fork point, main's tree is already the -/// Commit's (`NoChanges`), or its head is this Branch's own publication and -/// the Commit contains that publication's source. -fn safe_to_publish( - store: &Arc, - stack: &LayerStackRecord, - branch: &layerfs_history::BranchRecord, - commit: CommitId, -) -> Result { - if stack.head_layer == branch.base_layer { - return Ok(true); - } - let head = args::layer(store, stack.head_layer)?; - if head.root == root(store, NodeId::Commit(commit))? { - return Ok(true); - } - if head.stack != stack.id || head.source_branch != Some(branch.id) { - return Ok(false); - } - let Some(source) = head.source_commit else { - return Ok(false); - }; - if source == commit { - return Ok(true); - } - store - .history() - .is_ancestor(NodeId::Commit(source), NodeId::Commit(commit)) - .map_err(history) -} - -/// A refusal for a main that is not the Layer the caller expected. -fn moved_base(expected: LayerId, actual: LayerId) -> SdkRefusal { - refuse( - "HeadMoved", - format!( - "main is at {}, not the expected base {}; read main again and publish, or pass force to overwrite", - short(NodeId::Layer(actual)), - short(NodeId::Layer(expected)), - ), - SdkValue::map([ - ("expectedBase", SdkValue::text(hex(expected.as_slice()))), - ("actualBase", SdkValue::text(hex(actual.as_slice()))), - ]), - ) -} - -/// A refusal for a publication that would replace main's head with a tree -/// that does not contain it. -fn skipped(branch: &layerfs_history::BranchRecord, base: LayerId, head: LayerId) -> SdkRefusal { - refuse( - "HeadMoved", - format!( - "publishing {} would skip main's Layer {}: its fork point is {}; merge {} with main into a new Branch and publish that, or pass force to overwrite", - branch.name.as_str(), - short(NodeId::Layer(head)), - short(NodeId::Layer(base)), - branch.name.as_str(), - ), - SdkValue::map([ - ("expectedBase", SdkValue::text(hex(base.as_slice()))), - ("actualBase", SdkValue::text(hex(head.as_slice()))), - ]), - ) -} - -/// A refusal for a Branch head that is not the expected Commit. -fn moved_head( - branch: &layerfs_history::BranchRecord, - expected: CommitId, - actual: Option, -) -> SdkRefusal { - let (shown, detail) = match actual { - Some(actual) => ( - short(NodeId::Commit(actual)), - SdkValue::text(hex(actual.as_slice())), - ), - None => ("no Commit".to_owned(), SdkValue::Null), - }; - refuse( - "HeadMoved", - format!( - "{} head moved: expected {} but it is {}; read the Branch again and publish the Commit you expect", - branch.name.as_str(), - short(NodeId::Commit(expected)), - shown, - ), - SdkValue::map([ - ("expectedHead", SdkValue::text(hex(expected.as_slice()))), - ("actualHead", detail), - ]), - ) -} - /// Up to `limit` Layers `layer` overwrote after `position`, newest first, and /// whether more follow. pub(crate) fn overwritten( diff --git a/engine/crates/layerfs-store-service/src/history/deletion.rs b/engine/crates/layerfs-store-service/src/history/deletion.rs new file mode 100644 index 0000000000..488ce0493a --- /dev/null +++ b/engine/crates/layerfs-store-service/src/history/deletion.rs @@ -0,0 +1,89 @@ +//! The Store side of project.delete: the marker and the retry it makes safe. +//! +//! The marker is the commit point and is written before the caller removes its +//! home entry, so a lost reply is retried by the stack id the caller recorded +//! in its durable intent. A retry reads the marker first: a retired stack +//! reuses the stored deleted_ms, a live stack with no marker redoes the marker +//! with the same deleted_ms and keeps the entry, and an absent stack lets the +//! caller finish removing its entry. This module never resolves a name: the +//! caller has the stack id, and a hidden project has no name to resolve. +use crate::history::lookup; +use crate::store::Store; +use layerfs_bridge::control::SdkRefusal; +use layerfs_history::error::Missing; +use layerfs_history::{HistoryError, LayerStackId}; +use std::sync::Arc; + +/// What the Store side of project.delete found or did. +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum Deletion { + /// The marker was already recorded; its deleted_ms is reused. + Retired { + /// The deleted_ms the marker holds. + deleted_ms: i64, + }, + /// A live stack was marked with the caller's deleted_ms. + Marked { + /// The deleted_ms written. + deleted_ms: i64, + }, + /// Neither a live stack nor a marker: the caller removes its stale entry. + Absent, +} + +impl Deletion { + /// The deleted_ms a project.delete result reports: the marker's when the + /// stack was already retired, else the requested one. + pub fn deleted_ms(self, requested: i64) -> i64 { + match self { + Self::Retired { deleted_ms } | Self::Marked { deleted_ms } => deleted_ms, + Self::Absent => requested, + } + } +} + +/// Records the project.delete marker for stack, in one Store transaction, or +/// classifies why it cannot: retired (the marker is there), live (mark it now), +/// or absent (nothing is there to mark). A repeat is Retired, never a second +/// marker: the caller reuses the marker's deleted_ms. +pub fn delete_stack( + store: &Arc, + stack: LayerStackId, + deleted_ms: i64, +) -> Result { + let history = store.history(); + if let Some(recorded) = history.stack_deleted_ms(stack).map_err(lookup::history)? { + return Ok(Deletion::Retired { + deleted_ms: recorded, + }); + } + let live = history + .layer_stack(stack) + .map_err(lookup::history)? + .is_some(); + if !live { + // A concurrent attempt may have marked it after the read above. + return Ok( + match history.stack_deleted_ms(stack).map_err(lookup::history)? { + Some(recorded) => Deletion::Retired { + deleted_ms: recorded, + }, + None => Deletion::Absent, + }, + ); + } + match history.mark_stack_deleted(stack, deleted_ms) { + Ok(()) => Ok(Deletion::Marked { deleted_ms }), + // Another attempt marked it between the read and the write; reuse its + // deleted_ms rather than fail a retry that lost only a reply. + Err(HistoryError::Missing(Missing::LayerStack)) => Ok( + match history.stack_deleted_ms(stack).map_err(lookup::history)? { + Some(recorded) => Deletion::Retired { + deleted_ms: recorded, + }, + None => Deletion::Absent, + }, + ), + Err(error) => Err(lookup::history(error)), + } +} diff --git a/engine/crates/layerfs-store-service/src/history/lookup.rs b/engine/crates/layerfs-store-service/src/history/lookup.rs index e755b66cdb..7f28a16058 100644 --- a/engine/crates/layerfs-store-service/src/history/lookup.rs +++ b/engine/crates/layerfs-store-service/src/history/lookup.rs @@ -8,8 +8,9 @@ use crate::store::{PortError, Store}; use layerfs_bridge::control::{SdkRefusal, SdkValue}; use layerfs_content::{ContentError, ObjectId}; use layerfs_history::{ - error::Missing, BranchRecord, CommitId, CommitRecord, HistoryError, HistoryName, LayerId, - LayerRecord, LayerStackRecord, NodeId, + error::{HolderKind, HolderState, Missing, NameSource, NameTakenState, RemovedState}, + BranchRecord, CommitId, CommitRecord, HistoryError, HistoryName, LayerId, LayerRecord, + LayerStackRecord, NodeId, }; use std::sync::Arc; @@ -50,10 +51,83 @@ pub fn history(error: HistoryError) -> SdkRefusal { HistoryError::NotInHistory(_) => "NotInHistory", HistoryError::Busy => "Busy", HistoryError::Capacity(_) => "Capacity", + HistoryError::NameTaken(_) => "NameTaken", + HistoryError::Removed(_) => "Removed", + HistoryError::Conflict(_) => "Conflict", + HistoryError::StaleStore => "StaleStore", + HistoryError::Unsupported(_) => "Unsupported", HistoryError::UnknownOutcome => "Unknown", _ => "Failed", }; - refuse(code, message, SdkValue::Null) + let detail = match error.cause() { + HistoryError::NameTaken(state) => name_taken_detail(state), + HistoryError::Removed(state) => removed_detail(state), + _ => SdkValue::Null, + }; + refuse(code, message, detail) +} + +fn name_taken_detail(state: &NameTakenState) -> SdkValue { + SdkValue::map([ + ( + "holder", + SdkValue::map([ + ( + "kind", + SdkValue::text(match state.holder.kind { + HolderKind::Branch => "branch", + HolderKind::Project => "project", + // A newer history crate may add a holder kind; keep + // the old daemon readable instead of failing. + _ => "unknown", + }), + ), + ("name", SdkValue::text(&state.holder.name)), + ( + "state", + SdkValue::text(match state.holder.state { + HolderState::Live => "live", + HolderState::Retired => "retired", + HolderState::Deleted => "deleted", + // A newer history crate may add a holder state; keep + // the old daemon readable instead of failing. + _ => "unknown", + }), + ), + ]), + ), + ( + "source", + SdkValue::text(match state.source { + NameSource::Store => "store", + NameSource::Home => "home", + // A newer history crate may add a source; keep the old daemon + // readable instead of failing. + _ => "unknown", + }), + ), + ]) +} + +fn removed_detail(state: &RemovedState) -> SdkValue { + SdkValue::map([( + "paths", + SdkValue::List( + state + .paths + .iter() + .map(|path| { + SdkValue::map([ + ("path", SdkValue::text(&path.path)), + ( + "frontier_serial", + SdkValue::Int(path.frontier_serial as i64), + ), + ]) + }) + .collect(), + ), + )]) } /// A content failure as a refusal. diff --git a/engine/crates/layerfs-store-service/src/history/mod.rs b/engine/crates/layerfs-store-service/src/history/mod.rs index dbc1980206..a439163e4f 100644 --- a/engine/crates/layerfs-store-service/src/history/mod.rs +++ b/engine/crates/layerfs-store-service/src/history/mod.rs @@ -1,6 +1,7 @@ //! Branch history over the global Store: typed fork, merge, publish and //! listings, and the lookups and refusals the Store calls share. pub mod branch; +pub mod deletion; pub mod lookup; pub mod pages; pub mod saved; diff --git a/engine/crates/layerfs-store-service/src/methods/args.rs b/engine/crates/layerfs-store-service/src/methods/args.rs index c6ee3bbc8d..ce5e687878 100644 --- a/engine/crates/layerfs-store-service/src/methods/args.rs +++ b/engine/crates/layerfs-store-service/src/methods/args.rs @@ -30,6 +30,14 @@ pub fn snapshot( Ok((reader, snap)) } +/// The project, its reader and the snapshot a read names, in that order. +pub fn snapshot_full( + service: &StoreService, + args: &SdkValue, +) -> Result<(LayerStackRecord, Arc, Snap), SdkRefusal> { + crate::operator::args::snapshot(&service.store, args) +} + /// The required absolute path argument `name`, checked. pub fn path(args: &SdkValue, name: &str) -> Result { files::path(text(args, name)?) diff --git a/engine/crates/layerfs-store-service/src/methods/branch/publish.rs b/engine/crates/layerfs-store-service/src/methods/branch/publish.rs index ff48c073d7..6d34d45d4c 100644 --- a/engine/crates/layerfs-store-service/src/methods/branch/publish.rs +++ b/engine/crates/layerfs-store-service/src/methods/branch/publish.rs @@ -5,44 +5,11 @@ use crate::{ StoreService, }; use layerfs_bridge::control::SdkValue; -use layerfs_history::NodeId; -/// `publish({ branch, commit?, expectedBase?, expectedHead?, force? })`: one new -/// main Layer holding the Branch head, refused `HeadMoved` when main moved -/// since the Branch forked unless `force` asks for the overwrite. +/// `publish({ branch, commit? })`: one new main Layer holding the Branch head, reporting the Layers it overwrote. pub fn publish(service: &StoreService, args: &SdkValue) -> Answer { let stack = project(service, args)?; let named = lookup::branch(&service.store, &stack, text(args, "branch")?)?; let commit = args.get_text("commit").map(lookup::node).transpose()?; - let expected_base = match args.get_text("expectedBase") { - None => None, - Some(base) => match lookup::node(base)? { - NodeId::Layer(layer) => Some(layer), - NodeId::Commit(_) => { - return Err(lookup::invalid( - "expected_base must name a Layer, not a Commit", - )) - } - }, - }; - let expected_head = match args.get_text("expectedHead") { - None => None, - Some(head) => match lookup::node(head)? { - NodeId::Commit(commit) => Some(commit), - NodeId::Layer(_) => { - return Err(lookup::invalid( - "expected_head must name a Commit, not a Layer", - )) - } - }, - }; - branch::publish( - &service.store, - &stack, - &named, - commit, - expected_base, - expected_head, - args.get_bool("force"), - ) + branch::publish(&service.store, &stack, &named, commit) } diff --git a/engine/crates/layerfs-store-service/src/methods/dispatch.rs b/engine/crates/layerfs-store-service/src/methods/dispatch.rs index d71c543d94..73e1cf034a 100644 --- a/engine/crates/layerfs-store-service/src/methods/dispatch.rs +++ b/engine/crates/layerfs-store-service/src/methods/dispatch.rs @@ -22,6 +22,7 @@ pub fn call(service: &StoreService, call: &SdkCall) -> Answer { "store.stats" => store::stats(service), "project.stats" => project::stats(service, args), "project.list_branches" => project::list_branches(service, args), + "project.remove_file" => project::remove_file(service, args), "branch.fork" => branch::fork(service, args), "branch.list_commits" => branch::list_commits(service, args), "branch.commits" => branch::commits(service, args), diff --git a/engine/crates/layerfs-store-service/src/methods/file/glob.rs b/engine/crates/layerfs-store-service/src/methods/file/glob.rs index c4d978cd46..c7b21f7548 100644 --- a/engine/crates/layerfs-store-service/src/methods/file/glob.rs +++ b/engine/crates/layerfs-store-service/src/methods/file/glob.rs @@ -1,16 +1,22 @@ //! `fs.glob`. -use crate::methods::args::{optional_path, paging, snapshot, text, Answer}; +use crate::methods::args::{optional_path, paging, snapshot_full, text, Answer}; use crate::{ - read::{paged_reads, query, reads}, + read::{paged_reads, query, reads, snapshot, visibility}, StoreService, }; use layerfs_bridge::control::SdkValue; /// `glob(pattern, { branch, commit?, path?, pageSize?, pagePos? })`: paths matching a `/.lww`-syntax pattern. pub fn glob(service: &StoreService, args: &SdkValue) -> Answer { - let (reader, snap) = snapshot(service, args)?; + let (stack, reader, snap) = snapshot_full(service, args)?; let pattern = text(args, "pattern")?; let path = optional_path(args)?; + let seen = visibility::Visibility::load(&service.store, &stack, &snap); + if snapshot::entry(&reader, &snap.tree, &path)?.is_some() { + if let Some(refusal) = visibility::refuse(&path, &seen) { + return Err(refusal); + } + } if args.get("page").is_some() { return paged_reads::pin( &service.store, @@ -22,13 +28,15 @@ pub fn glob(service: &StoreService, args: &SdkValue) -> Answer { pattern, &path, query::result_page(args)?, - ), + ) + .map(|value| visibility::filter(value, &seen)), ); } let (limit, pos) = paging(args)?; paged_reads::pin( &service.store, &snap, - reads::glob(&reader, &snap, pattern, &path, limit, pos), + reads::glob(&reader, &snap, pattern, &path, limit, pos) + .map(|value| visibility::filter(value, &seen)), ) } diff --git a/engine/crates/layerfs-store-service/src/methods/file/list.rs b/engine/crates/layerfs-store-service/src/methods/file/list.rs index f3cedc8e50..e983ab9080 100644 --- a/engine/crates/layerfs-store-service/src/methods/file/list.rs +++ b/engine/crates/layerfs-store-service/src/methods/file/list.rs @@ -1,15 +1,21 @@ //! `fs.list`. -use crate::methods::args::{optional_path, paging, snapshot, Answer}; +use crate::methods::args::{optional_path, paging, snapshot_full, Answer}; use crate::{ - read::{paged_reads, query, reads}, + read::{paged_reads, query, reads, snapshot, visibility}, StoreService, }; use layerfs_bridge::control::SdkValue; /// `list(path?, { branch, commit?, pageSize?, pagePos? })`: one window of a directory. pub fn list(service: &StoreService, args: &SdkValue) -> Answer { - let (reader, snap) = snapshot(service, args)?; + let (stack, reader, snap) = snapshot_full(service, args)?; let path = optional_path(args)?; + let seen = visibility::Visibility::load(&service.store, &stack, &snap); + if snapshot::entry(&reader, &snap.tree, &path)?.is_some() { + if let Some(refusal) = visibility::refuse(&path, &seen) { + return Err(refusal); + } + } let answer = if args.get("page").is_some() { paged_reads::list( &service.store, @@ -20,6 +26,7 @@ pub fn list(service: &StoreService, args: &SdkValue) -> Answer { ) } else { reads::list(&reader, &snap, &path, paging(args)) - }; + } + .map(|value| visibility::filter(value, &seen)); paged_reads::pin(&service.store, &snap, answer) } diff --git a/engine/crates/layerfs-store-service/src/methods/file/read.rs b/engine/crates/layerfs-store-service/src/methods/file/read.rs index eb7f29dcd2..41532e9989 100644 --- a/engine/crates/layerfs-store-service/src/methods/file/read.rs +++ b/engine/crates/layerfs-store-service/src/methods/file/read.rs @@ -1,15 +1,21 @@ //! `fs.read`. -use crate::methods::args::{line, path, snapshot, Answer}; +use crate::methods::args::{line, path, snapshot_full, Answer}; use crate::{ - read::{paged_reads, reads}, + read::{paged_reads, reads, snapshot, visibility}, StoreService, }; use layerfs_bridge::control::SdkValue; /// `read(path, { branch, commit?, startLine?, endLine? })`: one window of a file at a Branch, Commit or Layer. pub fn read(service: &StoreService, args: &SdkValue) -> Answer { - let (reader, snap) = snapshot(service, args)?; + let (stack, reader, snap) = snapshot_full(service, args)?; let path = path(args, "path")?; + let seen = visibility::Visibility::load(&service.store, &stack, &snap); + if snapshot::entry(&reader, &snap.tree, &path)?.is_some() { + if let Some(refusal) = visibility::refuse(&path, &seen) { + return Err(refusal); + } + } let answer = reads::read( &reader, &snap, diff --git a/engine/crates/layerfs-store-service/src/methods/file/stat.rs b/engine/crates/layerfs-store-service/src/methods/file/stat.rs index be76fc4e46..6fb48d0332 100644 --- a/engine/crates/layerfs-store-service/src/methods/file/stat.rs +++ b/engine/crates/layerfs-store-service/src/methods/file/stat.rs @@ -1,17 +1,20 @@ //! `fs.stat`. -use crate::methods::args::{path, snapshot, Answer}; +use crate::methods::args::{path, snapshot_full, Answer}; use crate::{ - read::{paged_reads, reads}, + read::{paged_reads, reads, snapshot, visibility}, StoreService, }; use layerfs_bridge::control::SdkValue; /// `stat(path, { branch, commit? })`: type, inode, size and mode of one path. pub fn stat(service: &StoreService, args: &SdkValue) -> Answer { - let (reader, snap) = snapshot(service, args)?; - paged_reads::pin( - &service.store, - &snap, - reads::stat(&reader, &snap, &path(args, "path")?), - ) + let (stack, reader, snap) = snapshot_full(service, args)?; + let path = path(args, "path")?; + let seen = visibility::Visibility::load(&service.store, &stack, &snap); + if snapshot::entry(&reader, &snap.tree, &path)?.is_some() { + if let Some(refusal) = visibility::refuse(&path, &seen) { + return Err(refusal); + } + } + paged_reads::pin(&service.store, &snap, reads::stat(&reader, &snap, &path)) } diff --git a/engine/crates/layerfs-store-service/src/methods/project.rs b/engine/crates/layerfs-store-service/src/methods/project.rs index 98af26cd67..b42521d620 100644 --- a/engine/crates/layerfs-store-service/src/methods/project.rs +++ b/engine/crates/layerfs-store-service/src/methods/project.rs @@ -1,7 +1,11 @@ -//! `project.list_branches`. -use crate::methods::args::{invalid, paging, project, Answer}; -use crate::{history::branch, StoreService}; +//! `project.list_branches`, `project.stats` and `project.remove_file`. +use crate::methods::args::{invalid, paging, project, text, Answer}; +use crate::{ + history::{branch, lookup}, + StoreService, +}; use layerfs_bridge::control::SdkValue; +use layerfs_history::{PathRemovalKind, PathRemovalRequest}; /// `list_branches({ name?, pageSize?, pagePos? })`: every Branch with its id, base and head, and main, in name order. pub fn list_branches(service: &StoreService, args: &SdkValue) -> Answer { @@ -78,3 +82,45 @@ pub fn stats(service: &StoreService, args: &SdkValue) -> Answer { crate::methods::args::text(args, "project")?, ) } + +/// `project.remove_file({ project, path })`: hide one path from every node at +/// or below the frontier. Visibility only; no byte is freed in 2.0. +pub fn remove_file(service: &StoreService, args: &SdkValue) -> Answer { + let stack = project(service, args)?; + let requested = text(args, "path")?; + let normalized = layerfs_history::path::NormalizedPath::new(requested) + .map_err(|why| invalid(format!("path: {why}")))?; + let removed_ms = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map_or(0, |elapsed| { + i64::try_from(elapsed.as_millis()).unwrap_or(i64::MAX) + }); + let outcome = service + .store + .history() + .remove_path(&PathRemovalRequest { + stack: stack.id, + path: normalized.as_bytes().to_vec(), + removed_ms, + }) + .map_err(lookup::history)?; + Ok(SdkValue::map([ + ( + "kind", + SdkValue::text(match outcome.kind { + PathRemovalKind::Removed => "Removed", + PathRemovalKind::UpToDate => "UpToDate", + }), + ), + ("project", SdkValue::text(text(args, "project")?)), + ( + "path", + SdkValue::text(String::from_utf8_lossy(&outcome.path).as_ref()), + ), + ( + "frontier_serial", + SdkValue::Int(outcome.frontier_serial as i64), + ), + ("removed_ms", SdkValue::Int(outcome.removed_ms)), + ])) +} diff --git a/engine/crates/layerfs-store-service/src/read/mod.rs b/engine/crates/layerfs-store-service/src/read/mod.rs index 221476651a..3885404fe2 100644 --- a/engine/crates/layerfs-store-service/src/read/mod.rs +++ b/engine/crates/layerfs-store-service/src/read/mod.rs @@ -22,5 +22,6 @@ pub(crate) mod revision_pages; pub mod search; pub mod snapshot; pub mod tree_stats; +pub mod visibility; pub mod walk; pub mod window; diff --git a/engine/crates/layerfs-store-service/src/read/snapshot.rs b/engine/crates/layerfs-store-service/src/read/snapshot.rs index 53e4a0e2e0..5c7a3ece9d 100644 --- a/engine/crates/layerfs-store-service/src/read/snapshot.rs +++ b/engine/crates/layerfs-store-service/src/read/snapshot.rs @@ -35,6 +35,8 @@ pub struct Snap { pub node: NodeId, pub root: ObjectId, pub tree: TreeTable, + /// The node's `history_serial`, for the `remove_file` visibility rule. + pub serial: u64, } /// The stack's object reader. @@ -89,12 +91,17 @@ pub fn resolve( } let root = args::root(store, node)?; let tree = TreeTable::open(reader, FilesystemRootId(root)).map_err(args::content)?; + let serial = match store.published(|history| history.node_history_serial(node)) { + Ok(Some(serial)) => serial, + _ => 0, + }; Ok(Snap { kind, label, node, root, tree, + serial, }) } diff --git a/engine/crates/layerfs-store-service/src/read/visibility.rs b/engine/crates/layerfs-store-service/src/read/visibility.rs new file mode 100644 index 0000000000..756d417b2f --- /dev/null +++ b/engine/crates/layerfs-store-service/src/read/visibility.rs @@ -0,0 +1,98 @@ +//! The `project.remove_file` visibility filter over one resolved snapshot. +//! +//! A path is hidden at node `N` when its entry exists at `N` and the marker +//! frontier, the maximum over the path and its `/`-boundary ancestors, is at +//! least `N`'s `history_serial` (`§4.3`). A marked-but-absent path hides +//! nothing, so callers check existence first. Reads and stats refuse with one +//! `Removed` detail; listings and globs omit hidden entries and report one +//! top-level `removed_count`. + +use crate::read::snapshot::Snap; +use crate::store::Store; +use layerfs_bridge::control::{SdkRefusal, SdkValue}; +use layerfs_history::error::{RemovedPath, RemovedState}; +use layerfs_history::{HistoryError, LayerStackRecord, PathRemovalRecord}; +use std::sync::Arc; + +/// The markers and node position one snapshot's reads filter against. +pub struct Visibility { + serial: u64, + markers: Vec, +} + +impl Visibility { + /// Loads the snapshot's serial and its project's markers. A provider that + /// does not implement the marker reads (a test double) sees none. + pub fn load(store: &Arc, stack: &LayerStackRecord, snap: &Snap) -> Self { + let serial = snap.serial; + let markers = match store.published(|history| history.path_removals(stack.id)) { + Ok(markers) => markers, + Err(_) => Vec::new(), + }; + Self { serial, markers } + } + + /// The governing frontier for a path: the maximum marker frontier over the + /// path and its `/`-boundary ancestors, if any marker applies. + fn frontier(&self, path: &str) -> Option { + let relative = path.strip_prefix('/').unwrap_or(path); + if relative.is_empty() { + return None; + } + self.markers + .iter() + .filter(|marker| layerfs_history::path::governs(&marker.path, relative.as_bytes())) + .map(|marker| marker.frontier_serial) + .max() + } + + /// The governing frontier when the entry at `path` is hidden at this node. + pub fn hidden(&self, path: &str) -> Option { + let frontier = self.frontier(path)?; + (frontier >= self.serial).then_some(frontier) + } + + /// The refusal for one hidden read or stat, `{paths: [{path, frontier_serial}]}`. + pub fn refusal(path: &str, frontier_serial: u64) -> SdkRefusal { + crate::history::lookup::history(HistoryError::Removed(Box::new(RemovedState::new(vec![ + RemovedPath::new(path, frontier_serial), + ])))) + } +} + +/// The `Removed` refusal a hidden read or stat reports. +pub fn refuse(path: &str, visibility: &Visibility) -> Option { + visibility + .hidden(path) + .map(|frontier| Visibility::refusal(path, frontier)) +} + +/// Removes hidden entries from a list or glob answer and adds `removed_count`. +/// Entries themselves exist, so only marked-and-present paths are counted. +pub fn filter(mut value: SdkValue, visibility: &Visibility) -> SdkValue { + let mut removed = 0i64; + if let SdkValue::Map(fields) = &mut value { + if let Some((_, SdkValue::List(items))) = fields.iter_mut().find(|(k, _)| k == "items") { + items.retain(|item| { + let path = match item { + SdkValue::Text(path) => path.clone(), + other => other + .get_text("path") + .map(str::to_owned) + .unwrap_or_default(), + }; + if path.is_empty() { + return true; + } + if visibility.hidden(&path).is_some() { + removed += 1; + false + } else { + true + } + }); + } + fields.push(("removed_count".into(), SdkValue::Int(removed))); + } + value +} diff --git a/engine/crates/layerfs-store-service/tests/golden/sdk_methods.txt b/engine/crates/layerfs-store-service/tests/golden/sdk_methods.txt index d5f472bdac..0fb8409998 100644 --- a/engine/crates/layerfs-store-service/tests/golden/sdk_methods.txt +++ b/engine/crates/layerfs-store-service/tests/golden/sdk_methods.txt @@ -34,16 +34,16 @@ fs.read -> refused NoSuchPath "/nope does not exist at feat@" {path: "/nope fs.read -> refused NoSuchBranch "Branch ghost does not exist" null fs.read -> refused InvalidInput "missing text argument `path`" null fs.read -> refused Binary "text contains NUL or invalid UTF-8" null -fs.list -> ok {kind: "ok", snapshot: "feat@", root: "", path: "/", items: [{name: "a.txt", path: "/a.txt", type: "file", inode: 2}, {name: "applied-dir", path: "/applied-dir", type: "dir", inode: 201}, {name: "applied.txt", path: "/applied.txt", type: "file", inode: 200}, {name: "blob.bin", path: "/blob.bin", type: "file", inode: 3}, {name: "dir", path: "/dir", type: "dir", inode: 4}, {name: "made", path: "/made", type: "dir", inode: 72}, {name: "moved.txt", path: "/moved.txt", type: "file", inode: 137}, {name: "new.txt", path: "/new.txt", type: "file", inode: 8}], hasMore: false, nextPagePos: null, at: {id: "", kind: "Commit", branch: "feat", root: ""}} -fs.list -> ok {kind: "ok", snapshot: "feat@", root: "", path: "/dir", items: [{name: "b.txt", path: "/dir/b.txt", type: "file", inode: 5}], hasMore: true, nextPagePos: 1, at: {id: "", kind: "Commit", branch: "feat", root: ""}} +fs.list -> ok {kind: "ok", snapshot: "feat@", root: "", path: "/", items: [{name: "a.txt", path: "/a.txt", type: "file", inode: 2}, {name: "applied-dir", path: "/applied-dir", type: "dir", inode: 201}, {name: "applied.txt", path: "/applied.txt", type: "file", inode: 200}, {name: "blob.bin", path: "/blob.bin", type: "file", inode: 3}, {name: "dir", path: "/dir", type: "dir", inode: 4}, {name: "made", path: "/made", type: "dir", inode: 72}, {name: "moved.txt", path: "/moved.txt", type: "file", inode: 137}, {name: "new.txt", path: "/new.txt", type: "file", inode: 8}], hasMore: false, nextPagePos: null, removed_count: 0, at: {id: "", kind: "Commit", branch: "feat", root: ""}} +fs.list -> ok {kind: "ok", snapshot: "feat@", root: "", path: "/dir", items: [{name: "b.txt", path: "/dir/b.txt", type: "file", inode: 5}], hasMore: true, nextPagePos: 1, removed_count: 0, at: {id: "", kind: "Commit", branch: "feat", root: ""}} fs.list -> refused NotDirectory "expected a directory" null fs.stat -> ok {kind: "ok", snapshot: "main@", root: "", path: "/a.txt", type: "file", inode: 2, size: 14, mode: 420, at: {id: "", kind: "Layer", branch: "main", root: ""}} fs.stat -> ok {kind: "ok", snapshot: "feat@", root: "", path: "/dir", type: "dir", inode: 4, size: 0, mode: 493, at: {id: "", kind: "Commit", branch: "feat", root: ""}} fs.grep -> ok {kind: "ok", snapshot: "feat@", root: "", path: "/", items: [], skipped: 1, hasMore: false, nextPagePos: null, at: {id: "", kind: "Commit", branch: "feat", root: ""}} fs.grep -> ok {kind: "ok", snapshot: "feat@", root: "", path: "/dir", items: [{path: "/dir/b.txt", line: 1, text: "bee\n", lineTruncated: false}], skipped: 0, hasMore: false, nextPagePos: null, at: {id: "", kind: "Commit", branch: "feat", root: ""}} fs.grep -> ok {kind: "ok", snapshot: "feat@", root: "", path: "/", items: [{path: "/dir/sub/c.rs", line: 1, text: "fn main() {}\n", lineTruncated: false}], skipped: 1, hasMore: false, nextPagePos: null, at: {id: "", kind: "Commit", branch: "feat", root: ""}} -fs.glob -> ok {kind: "ok", snapshot: "feat@", root: "", path: "/", items: ["/a.txt", "/applied.txt", "/dir/b.txt", "/moved.txt", "/new.txt"], hasMore: false, nextPagePos: null, at: {id: "", kind: "Commit", branch: "feat", root: ""}} -fs.glob -> ok {kind: "ok", snapshot: "feat@", root: "", path: "/", items: ["/dir/b.txt", "/dir/sub", "/dir/sub/c.rs"], hasMore: false, nextPagePos: null, at: {id: "", kind: "Commit", branch: "feat", root: ""}} +fs.glob -> ok {kind: "ok", snapshot: "feat@", root: "", path: "/", items: ["/a.txt", "/applied.txt", "/dir/b.txt", "/moved.txt", "/new.txt"], hasMore: false, nextPagePos: null, removed_count: 0, at: {id: "", kind: "Commit", branch: "feat", root: ""}} +fs.glob -> ok {kind: "ok", snapshot: "feat@", root: "", path: "/", items: ["/dir/b.txt", "/dir/sub", "/dir/sub/c.rs"], hasMore: false, nextPagePos: null, removed_count: 0, at: {id: "", kind: "Commit", branch: "feat", root: ""}} fs.blame -> ok {kind: "ok", snapshot: "feat@", root: "", path: "/a.txt", at: {id: "", kind: "Commit", branch: "feat", root: ""}, coverage: {status: "complete", comparison: "exact", reasons: []}, lines: [{n: 1, owner: {ref: {id: "", kind: "Layer", branch: "main", root: ""}, message: null}, text: "one", lineTruncated: false, clipped: false, commit: "", message: null}, {n: 2, owner: {ref: {id: "", kind: "Commit", branch: "feat", root: ""}, message: "Apply filesystem changes"}, text: "TWO", lineTruncated: false, clipped: false, commit: "", message: "Apply filesystem changes"}, {n: 3, owner: {ref: {id: "", kind: "Layer", branch: "main", root: ""}, message: null}, text: "three", lineTruncated: false, clipped: false, commit: "", message: null}], origins: [{ref: {id: "", kind: "Commit", branch: "feat", root: ""}, message: "Apply filesystem changes"}, {ref: {id: "", kind: "Layer", branch: "main", root: ""}, message: null}], versions: 8, comparisonCoarse: false, nextLine: null} fs.blame -> ok {kind: "ok", snapshot: "feat@", root: "", path: "/a.txt", at: {id: "", kind: "Commit", branch: "feat", root: ""}, coverage: {status: "complete", comparison: "exact", reasons: []}, lines: [{n: 2, owner: {ref: {id: "", kind: "Commit", branch: "feat", root: ""}, message: "Apply filesystem changes"}, text: "TWO", lineTruncated: false, clipped: false, commit: "", message: "Apply filesystem changes"}, {n: 3, owner: {ref: {id: "", kind: "Layer", branch: "main", root: ""}, message: null}, text: "three", lineTruncated: false, clipped: false, commit: "", message: null}], origins: [{ref: {id: "", kind: "Commit", branch: "feat", root: ""}, message: "Apply filesystem changes"}, {ref: {id: "", kind: "Layer", branch: "main", root: ""}, message: null}], versions: 8, comparisonCoarse: false, nextLine: null} fs.diff_files -> ok {kind: "Text", left: {path: "/a.txt", snapshot: "main@", inode: 2}, right: {path: "/a.txt", snapshot: "feat@", inode: 2}, contentEqual: false, sameInode: true, hunksOmitted: false, hunks: [{op: " ", text: "one"}, {op: "-", text: "two"}, {op: "+", text: "TWO"}, {op: " ", text: "three"}, {op: " ", text: ""}], leftRef: {id: "", kind: "Layer", branch: "main", root: ""}, rightRef: {id: "", kind: "Commit", branch: "feat", root: ""}, before: {type: "file", inode: 2, size: 14, mode: 420}, after: {type: "file", inode: 2, size: 14, mode: 420}, changeKind: "Modified", changes: {content: true, metadata: true, identity: false, attribution: null}, contentCoverage: {status: "complete", comparison: "exact", reasons: []}, attributionCoverage: {status: "not_requested", comparison: null, reasons: []}, omissionReason: null, rows: [], page: 1, pageSize: 50, hasMore: false} @@ -67,7 +67,7 @@ fs.write -> ok {kind: "Committed", branch: "left", head: "", commit: {id: " branch.fork -> ok {kind: "Forked", branch: "right", base: "", head: null} fs.write -> ok {kind: "Committed", branch: "right", head: "", commit: {id: "", branch: "right", parent: null, secondParent: null, base: "", root: "", via: "fs", message: "Apply filesystem changes", mergedFrom: null, time: , logical: 42, newCanonical: 958, newStored: , newMetadata: , displaced: null, lww: [], lwwTruncated: false}} branch.merge -> ok {kind: "Merged", fastForward: false, branch: "both", head: "", root: "", sources: [{ref: "left", node: ""}, {ref: "right", node: ""}], mergeBase: "", conflicts: [], lww: [], conflictsTruncated: false, lwwTruncated: false, units: [], unitsTruncated: false} -fs.list -> ok {kind: "ok", snapshot: "both@", root: "", path: "/", items: [{name: "a.txt", path: "/a.txt", type: "file", inode: 2}, {name: "blob.bin", path: "/blob.bin", type: "file", inode: 3}, {name: "dir", path: "/dir", type: "dir", inode: 4}, {name: "left.txt", path: "/left.txt", type: "file", inode: 328}, {name: "right.txt", path: "/right.txt", type: "file", inode: 392}], hasMore: false, nextPagePos: null, at: {id: "", kind: "Commit", branch: "both", root: ""}} +fs.list -> ok {kind: "ok", snapshot: "both@", root: "", path: "/", items: [{name: "a.txt", path: "/a.txt", type: "file", inode: 2}, {name: "blob.bin", path: "/blob.bin", type: "file", inode: 3}, {name: "dir", path: "/dir", type: "dir", inode: 4}, {name: "left.txt", path: "/left.txt", type: "file", inode: 328}, {name: "right.txt", path: "/right.txt", type: "file", inode: 392}], hasMore: false, nextPagePos: null, removed_count: 0, at: {id: "", kind: "Commit", branch: "both", root: ""}} branch.merge -> refused NameTaken "Branch both already exists" null branch.merge -> refused InvalidInput "merge takes two different sources" null branch.merge -> refused ReservedName "\"main\" names the project's main line" null diff --git a/engine/crates/layerfs-store-service/tests/history_ancestry_oracle.rs b/engine/crates/layerfs-store-service/tests/history_ancestry_oracle.rs index ef5136a30d..79d92d759f 100644 --- a/engine/crates/layerfs-store-service/tests/history_ancestry_oracle.rs +++ b/engine/crates/layerfs-store-service/tests/history_ancestry_oracle.rs @@ -226,12 +226,7 @@ fn random_history(seed: u64) { continue; } let branch = pool[rng.below(pool.len())].clone(); - // Any Branch may publish next: the oracle explores the overwrite - // topology, which is explicit now (#691). - fixture.call( - "branch.publish", - vec![("branch", text(&branch)), ("force", SdkValue::Bool(true))], - ); + fixture.call("branch.publish", vec![("branch", text(&branch))]); publishes += 1; } } diff --git a/engine/crates/layerfs-store-service/tests/project_delete.rs b/engine/crates/layerfs-store-service/tests/project_delete.rs new file mode 100644 index 0000000000..1662ca7dc1 --- /dev/null +++ b/engine/crates/layerfs-store-service/tests/project_delete.rs @@ -0,0 +1,81 @@ +//! project.delete on a Store: the marker hides the project from name +//! resolution, and the retry classifies retired/live/absent by stack id. +#[path = "support/installed_store.rs"] +#[allow(dead_code)] +mod installed; +#[path = "support/sdk.rs"] +#[allow(dead_code)] +mod support; +use layerfs_history::{HistoryName, LayerStackId, LayerStackRecord}; +use layerfs_store_service::history::deletion::{delete_stack, Deletion}; +use support::Fixture; + +fn fixture() -> Fixture { + Fixture::new("project-delete", |source| { + std::fs::write( + source.join("a"), + "base +", + ) + .unwrap(); + 1 + }) +} + +fn installed(f: &Fixture) -> LayerStackRecord { + f.service + .store + .history() + .layer_stack_named(&HistoryName::new("installed").unwrap()) + .unwrap() + .unwrap() +} + +#[test] +fn delete_marks_once_and_hides_the_project() { + let f = fixture(); + let shop = installed(&f); + + // A live stack with no marker is marked with the call's deleted_ms. + assert_eq!( + delete_stack(&f.service.store, shop.id, 1234).unwrap(), + Deletion::Marked { deleted_ms: 1234 } + ); + assert_eq!( + f.service.store.history().stack_deleted_ms(shop.id).unwrap(), + Some(1234) + ); + + // A hidden project resolves to NoSuchProject on every Store call. + for method in ["project.list_branches", "project.stats"] { + let refused = f.try_call(method, vec![]).unwrap_err(); + assert_eq!(refused.code, "NoSuchProject", "{method}: {refused:?}"); + } + assert!(f + .service + .store + .history() + .layer_stacks(&layerfs_history::Page { + cursor: None, + limit: 10, + }) + .unwrap() + .records + .is_empty()); + + // A retry by the same stack id reuses the marker's deleted_ms. + assert_eq!( + delete_stack(&f.service.store, shop.id, 999).unwrap(), + Deletion::Retired { deleted_ms: 1234 } + ); + assert_eq!(Deletion::Retired { deleted_ms: 1234 }.deleted_ms(999), 1234); + + // An absent stack lets the caller finish its stale entry. + let absent = LayerStackId::from_authority([9; 16]); + assert_eq!( + delete_stack(&f.service.store, absent, 7).unwrap(), + Deletion::Absent + ); + assert_eq!(Deletion::Absent.deleted_ms(7), 7); + f.cleanup(); +} diff --git a/engine/crates/layerfs-store-service/tests/publish_expected_base.rs b/engine/crates/layerfs-store-service/tests/publish_expected_base.rs deleted file mode 100644 index 25409eec44..0000000000 --- a/engine/crates/layerfs-store-service/tests/publish_expected_base.rs +++ /dev/null @@ -1,544 +0,0 @@ -//! `branch.publish` refuses a publication that would drop a Layer main gained -//! since the Branch forked (#691): main must still be the fork point, or its -//! head must be this Branch's own earlier publication. The caller may pin the -//! base and the head it read. A refused publish writes nothing, leaves main -//! and the Branch alone, and `force` keeps the old overwrite-only behavior. A -//! merged Branch still fast-forwards, and a Branch may publish again over its -//! own Layer. A Commit already published stays `UpToDate` and a tree main -//! already holds stays `NoChanges` after main moved. -#[path = "support/installed_store.rs"] -#[allow(dead_code)] -mod installed; -#[path = "support/sdk.rs"] -#[allow(dead_code)] -mod support; - -use layerfs_bridge::control::SdkValue; -use support::{paths, write, Fixture}; - -/// One Branch's field from `project.list_branches`. -fn branch_field(fixture: &Fixture, name: &str, field: &str) -> Option { - let page = fixture.call( - "project.list_branches", - vec![("name", SdkValue::text(name))], - ); - support::items(&page) - .first() - .and_then(|item| item.get_text(field)) - .map(str::to_owned) -} - -/// main's current head Layer. -fn main_head(fixture: &Fixture) -> String { - branch_field(fixture, "main", "base").expect("main is listed with its head Layer") -} - -/// The paths in main's root. -fn main_paths(fixture: &Fixture) -> Vec { - let page = fixture.call( - "fs.list", - vec![ - ("branch", SdkValue::text("main")), - ("path", SdkValue::text("/")), - ], - ); - paths(&page).into_iter().map(str::to_owned).collect() -} - -/// One field of a publish answer's `publishedLayer`. -fn published_field(value: &SdkValue, field: &str) -> String { - value - .get("publishedLayer") - .and_then(|layer| layer.get_text(field)) - .unwrap_or_else(|| panic!("no publishedLayer.{field} in {value:?}")) - .to_owned() -} - -/// A Store with one seed file, and two Branches forked from the same main. -fn two_branches(label: &str) -> Fixture { - let fixture = Fixture::new(label, |source| { - std::fs::write(source.join("seed.txt"), "seed\n").unwrap(); - 1 - }); - fixture.fork("feat-a"); - fixture.fork("feat-b"); - fixture.apply("feat-a", vec![write("/a.txt", "from A\n")]); - fixture.apply("feat-b", vec![write("/b.txt", "from B\n")]); - fixture -} - -#[test] -fn a_stale_base_publish_is_refused_and_main_is_unchanged() { - let fixture = two_branches("publish-stale-base"); - let a = fixture.call("branch.publish", vec![("branch", SdkValue::text("feat-a"))]); - assert_eq!(a.get_text("kind"), Some("Added")); - let a_layer = published_field(&a, "id"); - let main_before = main_head(&fixture); - assert_eq!(main_before, a_layer, "main is now A's Layer"); - let b_base = branch_field(&fixture, "feat-b", "base").expect("feat-b's base"); - let b_head = branch_field(&fixture, "feat-b", "head").expect("feat-b's head"); - - let refusal = fixture - .try_call("branch.publish", vec![("branch", SdkValue::text("feat-b"))]) - .expect_err("a Branch forked before main moved is refused"); - assert_eq!(refusal.code, "HeadMoved"); - assert!( - refusal - .message - .contains("merge feat-b with main into a new Branch"), - "the refusal says how to recover: {}", - refusal.message - ); - assert!( - refusal.message.contains("force"), - "the refusal names the escape: {}", - refusal.message - ); - assert_eq!( - refusal.detail.get_text("expectedBase"), - Some(b_base.as_str()) - ); - assert_eq!( - refusal.detail.get_text("actualBase"), - Some(a_layer.as_str()) - ); - - // main did not move, and never received B's tree. - assert_eq!(main_head(&fixture), main_before); - let names = main_paths(&fixture); - assert!(names.contains(&"/a.txt".to_owned()), "{names:?}"); - assert!(!names.iter().any(|path| path == "/b.txt"), "{names:?}"); - // The Branch itself did not move. - assert_eq!( - branch_field(&fixture, "feat-b", "base").as_deref(), - Some(b_base.as_str()) - ); - assert_eq!( - branch_field(&fixture, "feat-b", "head").as_deref(), - Some(b_head.as_str()) - ); - fixture.cleanup(); -} - -#[test] -fn matching_expectations_publish_and_wrong_ones_are_refused() { - let fixture = two_branches("publish-expected-base"); - let a = fixture.call("branch.publish", vec![("branch", SdkValue::text("feat-a"))]); - let a_layer = published_field(&a, "id"); - let b_head = branch_field(&fixture, "feat-b", "head").expect("feat-b's head"); - - // C forks after A published, so its base is main's current head. - fixture.fork("feat-c"); - fixture.apply("feat-c", vec![write("/c.txt", "from C\n")]); - let c_base = branch_field(&fixture, "feat-c", "base").expect("feat-c's base"); - let c_head = branch_field(&fixture, "feat-c", "head").expect("feat-c's head"); - assert_eq!(c_base, a_layer); - - let published = fixture.call( - "branch.publish", - vec![ - ("branch", SdkValue::text("feat-c")), - ("expectedBase", SdkValue::text(c_base.as_str())), - ("expectedHead", SdkValue::text(c_head.as_str())), - ], - ); - assert_eq!(published.get_text("kind"), Some("Added")); - assert_eq!(published.get_text("basedOn"), Some(c_base.as_str())); - let c_layer = published_field(&published, "id"); - - // D forks after C published; an expected base from before C is stale. - fixture.fork("feat-d"); - fixture.apply("feat-d", vec![write("/d.txt", "from D\n")]); - let d_base = branch_field(&fixture, "feat-d", "base").expect("feat-d's base"); - let d_head = branch_field(&fixture, "feat-d", "head").expect("feat-d's head"); - assert_eq!(d_base, c_layer); - - let moved_base = fixture - .try_call( - "branch.publish", - vec![ - ("branch", SdkValue::text("feat-d")), - ("expectedBase", SdkValue::text(a_layer.as_str())), - ], - ) - .expect_err("a stale expected base is refused"); - assert_eq!(moved_base.code, "HeadMoved"); - assert_eq!( - moved_base.detail.get_text("expectedBase"), - Some(a_layer.as_str()) - ); - assert_eq!( - moved_base.detail.get_text("actualBase"), - Some(d_base.as_str()) - ); - - let moved_head = fixture - .try_call( - "branch.publish", - vec![ - ("branch", SdkValue::text("feat-d")), - ("expectedBase", SdkValue::text(d_base.as_str())), - ("expectedHead", SdkValue::text(b_head.as_str())), - ], - ) - .expect_err("a wrong expected head is refused"); - assert_eq!(moved_head.code, "HeadMoved"); - assert_eq!( - moved_head.detail.get_text("expectedHead"), - Some(b_head.as_str()) - ); - assert_eq!( - moved_head.detail.get_text("actualHead"), - Some(d_head.as_str()) - ); - - // Neither refusal published anything: main is still C's Layer. - assert_eq!(main_head(&fixture), c_layer); - assert_eq!( - branch_field(&fixture, "feat-d", "head").as_deref(), - Some(d_head.as_str()) - ); - - // The exact base and head the caller read publish as a fast-forward. - let d = fixture.call( - "branch.publish", - vec![ - ("branch", SdkValue::text("feat-d")), - ("expectedBase", SdkValue::text(d_base.as_str())), - ("expectedHead", SdkValue::text(d_head.as_str())), - ], - ); - assert_eq!(d.get_text("kind"), Some("Added")); - assert_eq!(d.get_int("overwroteCount"), Some(0)); - fixture.cleanup(); -} - -#[test] -fn force_keeps_the_overwrite_behavior() { - let fixture = two_branches("publish-force"); - let a = fixture.call("branch.publish", vec![("branch", SdkValue::text("feat-a"))]); - let a_layer = published_field(&a, "id"); - - let forced = fixture.call( - "branch.publish", - vec![ - ("branch", SdkValue::text("feat-b")), - ("force", SdkValue::Bool(true)), - ], - ); - assert_eq!(forced.get_text("kind"), Some("Added")); - assert_eq!(forced.get_int("overwroteCount"), Some(1)); - let overwrote = match forced.get("overwrote") { - Some(SdkValue::List(items)) => items.clone(), - other => panic!("overwrote: {other:?}"), - }; - assert_eq!(overwrote.len(), 1); - assert_eq!(overwrote[0], SdkValue::text(a_layer.as_str())); - // The forced Layer is a child of the Layer it replaced but carries B's root. - assert_eq!(published_field(&forced, "parent"), a_layer); - let names = main_paths(&fixture); - assert!(names.contains(&"/b.txt".to_owned()), "{names:?}"); - assert!(!names.iter().any(|path| path == "/a.txt"), "{names:?}"); - - // A Commit that is already published is UpToDate before the stale check, - // as it was before the fix. - let again = fixture.call("branch.publish", vec![("branch", SdkValue::text("feat-b"))]); - assert_eq!(again.get_text("kind"), Some("UpToDate")); - fixture.cleanup(); -} - -#[test] -fn publishing_a_merged_branch_is_a_fast_forward() { - let fixture = two_branches("publish-merged"); - let a = fixture.call("branch.publish", vec![("branch", SdkValue::text("feat-a"))]); - let a_layer = published_field(&a, "id"); - let merged = fixture.call( - "branch.merge", - vec![ - ("branch", SdkValue::text("main")), - ("with", SdkValue::text("feat-b")), - ("name", SdkValue::text("merged")), - ], - ); - assert_eq!(merged.get_text("kind"), Some("Merged")); - // The merge base is main's Layer, so publishing it fast-forwards. - let merged_base = branch_field(&fixture, "merged", "base").expect("merged's base"); - assert_eq!(merged_base, a_layer); - let published = fixture.call( - "branch.publish", - vec![ - ("branch", SdkValue::text("merged")), - ("expectedBase", SdkValue::text(merged_base.as_str())), - ], - ); - assert_eq!(published.get_text("kind"), Some("Added")); - assert_eq!(published.get_int("overwroteCount"), Some(0)); - assert_eq!(published_field(&published, "parent"), a_layer); - let names = main_paths(&fixture); - assert!(names.contains(&"/a.txt".to_owned()), "{names:?}"); - assert!(names.contains(&"/b.txt".to_owned()), "{names:?}"); - fixture.cleanup(); -} - -#[test] -fn a_branch_publishes_again_over_its_own_layer() { - // Publishing at milestones is the normal flow: after a Branch's own Layer - // reaches main, its next Commit still contains that Layer, so nothing is - // skipped and the default publish stays allowed. - let fixture = Fixture::new("publish-milestones", |source| { - std::fs::write(source.join("seed.txt"), "seed\n").unwrap(); - 1 - }); - fixture.fork("work"); - fixture.apply("work", vec![write("/work.txt", "one\n")]); - let first = fixture.call("branch.publish", vec![("branch", SdkValue::text("work"))]); - assert_eq!(first.get_text("kind"), Some("Added")); - assert_eq!(first.get_int("overwroteCount"), Some(0)); - let first_layer = published_field(&first, "id"); - - fixture.apply("work", vec![write("/work.txt", "two\n")]); - let second = fixture.call("branch.publish", vec![("branch", SdkValue::text("work"))]); - assert_eq!(second.get_text("kind"), Some("Added")); - assert_eq!(second.get_int("overwroteCount"), Some(0)); - assert_eq!(published_field(&second, "parent"), first_layer); - let names = main_paths(&fixture); - assert!(names.contains(&"/work.txt".to_owned()), "{names:?}"); - fixture.cleanup(); -} - -#[test] -fn a_repeated_publish_after_main_moved_is_up_to_date() { - // A lost reply leaves the publication in place; asking again after main - // moved on reports that Layer, not a stale base. - let fixture = two_branches("publish-repeat-after-move"); - let a = fixture.call("branch.publish", vec![("branch", SdkValue::text("feat-a"))]); - let a_layer = published_field(&a, "id"); - let a_base = a.get_text("basedOn").expect("A's base").to_owned(); - fixture.fork("feat-c"); - fixture.apply("feat-c", vec![write("/c.txt", "from C\n")]); - let c = fixture.call("branch.publish", vec![("branch", SdkValue::text("feat-c"))]); - let c_layer = published_field(&c, "id"); - - let again = fixture.call("branch.publish", vec![("branch", SdkValue::text("feat-a"))]); - assert_eq!(again.get_text("kind"), Some("UpToDate")); - assert_eq!(again.get_text("layer"), Some(a_layer.as_str())); - // The base the caller read before its first attempt is stale now; the - // answer is still the Layer that attempt published. - let pinned = fixture.call( - "branch.publish", - vec![ - ("branch", SdkValue::text("feat-a")), - ("expectedBase", SdkValue::text(a_base.as_str())), - ], - ); - assert_eq!(pinned.get_text("kind"), Some("UpToDate")); - assert_eq!(pinned.get_text("layer"), Some(a_layer.as_str())); - assert_eq!(main_head(&fixture), c_layer, "nothing was published"); - fixture.cleanup(); -} - -#[test] -fn a_tree_main_already_holds_is_no_changes() { - // A second Branch at the same Commit as the first: once the first is - // published, the second drops nothing, so it answers NoChanges instead of - // a stale-base refusal. - let fixture = Fixture::new("publish-same-tree", |source| { - std::fs::write(source.join("seed.txt"), "seed\n").unwrap(); - 1 - }); - fixture.fork("same-a"); - fixture.apply("same-a", vec![write("/same.txt", "same\n")]); - let a_head = branch_field(&fixture, "same-a", "head").expect("same-a's head"); - fixture.call( - "branch.fork", - vec![ - ("name", SdkValue::text("same-b")), - ("from", SdkValue::text(a_head.as_str())), - ], - ); - let a = fixture.call("branch.publish", vec![("branch", SdkValue::text("same-a"))]); - let a_layer = published_field(&a, "id"); - let b = fixture.call("branch.publish", vec![("branch", SdkValue::text("same-b"))]); - assert_eq!(b.get_text("kind"), Some("NoChanges"), "{b:?}"); - assert_eq!(b.get_text("headLayer"), Some(a_layer.as_str())); - assert_eq!(main_head(&fixture), a_layer); - fixture.cleanup(); -} - -#[test] -fn expected_head_is_the_branch_head_even_with_a_named_commit() { - let fixture = two_branches("publish-expected-head-named"); - let b_head = branch_field(&fixture, "feat-b", "head").expect("feat-b's head"); - fixture.apply("feat-b", vec![write("/b2.txt", "B again\n")]); - let b_now = branch_field(&fixture, "feat-b", "head").expect("feat-b's new head"); - // The caller names the Commit it read and expects it to be the head; the - // refusal reports the Branch's real head, not the named Commit. - let refusal = fixture - .try_call( - "branch.publish", - vec![ - ("branch", SdkValue::text("feat-b")), - ("commit", SdkValue::text(b_head.as_str())), - ("expectedHead", SdkValue::text(b_head.as_str())), - ], - ) - .expect_err("the Branch head moved past the expected Commit"); - assert_eq!(refusal.code, "HeadMoved"); - assert_eq!( - refusal.detail.get_text("expectedHead"), - Some(b_head.as_str()) - ); - assert_eq!(refusal.detail.get_text("actualHead"), Some(b_now.as_str())); - fixture.cleanup(); -} - -/// main's published Layers, newest first. -fn main_layers(fixture: &Fixture) -> Vec { - let page = fixture.call( - "branch.list_commits", - vec![ - ("branch", SdkValue::text("main")), - ("pageSize", SdkValue::Int(100)), - ], - ); - support::items(&page) - .iter() - .map(|item| item.get_text("id").expect("a Layer id").to_owned()) - .collect() -} - -#[test] -fn two_threads_publishing_one_expected_base_race_cleanly() { - use std::sync::Barrier; - - // Each round forks two Branches from main, then both threads publish with - // main's head pinned as `expectedBase`. Whichever thread reaches the - // Store's write transaction first adds the only Layer; the other finds the - // stack head already moved, by the compare-and-swap in `add_layer` (or by - // the read above it) and is refused `HeadMoved`. A barrier starts the two - // calls together; no sleep orders them, and either thread may win. - const ROUNDS: usize = 20; - let fixture = Fixture::new("publish-two-threads", |source| { - std::fs::write(source.join("seed.txt"), "seed\n").unwrap(); - 1 - }); - for round in 0..ROUNDS { - let a = format!("race-a-{round}"); - let b = format!("race-b-{round}"); - let a_file = format!("/a-{round}.txt"); - let b_file = format!("/b-{round}.txt"); - let a_text = format!("A {round}\n"); - let b_text = format!("B {round}\n"); - let base = main_head(&fixture); - fixture.fork(&a); - fixture.fork(&b); - fixture.apply(&a, vec![write(&a_file, a_text.clone())]); - fixture.apply(&b, vec![write(&b_file, b_text.clone())]); - let a_head = branch_field(&fixture, &a, "head").expect("race-a's head"); - let b_head = branch_field(&fixture, &b, "head").expect("race-b's head"); - let layers_before = main_layers(&fixture); - - let barrier = Barrier::new(2); - let publish = |branch: &str, head: &str| { - fixture.try_call( - "branch.publish", - vec![ - ("branch", SdkValue::text(branch)), - ("expectedBase", SdkValue::text(base.as_str())), - ("expectedHead", SdkValue::text(head)), - ], - ) - }; - let (a_result, b_result) = std::thread::scope(|scope| { - let a_thread = scope.spawn(|| { - barrier.wait(); - publish(&a, &a_head) - }); - let b_thread = scope.spawn(|| { - barrier.wait(); - publish(&b, &b_head) - }); - (a_thread.join().unwrap(), b_thread.join().unwrap()) - }); - - // Exactly one publication wins; the other is refused before any write. - let a_won = a_result.is_ok(); - assert!( - a_won ^ b_result.is_ok(), - "round {round}: exactly one publish must win: a={a_result:?} b={b_result:?}" - ); - let (winner_answer, loser_refusal) = if a_won { - (a_result.as_ref().unwrap(), b_result.as_ref().unwrap_err()) - } else { - (b_result.as_ref().unwrap(), a_result.as_ref().unwrap_err()) - }; - let (winner_file, winner_text, loser_file, loser_head, loser_name) = if a_won { - (&a_file, &a_text, &b_file, &b_head, &b) - } else { - (&b_file, &b_text, &a_file, &a_head, &a) - }; - - assert_eq!( - winner_answer.get_text("kind"), - Some("Added"), - "round {round}" - ); - assert_eq!( - winner_answer.get_text("basedOn"), - Some(base.as_str()), - "round {round}" - ); - assert_eq!( - winner_answer.get_int("overwroteCount"), - Some(0), - "round {round}" - ); - let winner_layer = published_field(winner_answer, "id"); - - assert_eq!(loser_refusal.code, "HeadMoved", "round {round}"); - assert_eq!( - loser_refusal.detail.get_text("expectedBase"), - Some(base.as_str()), - "round {round}" - ); - assert_eq!( - loser_refusal.detail.get_text("actualBase"), - Some(winner_layer.as_str()), - "round {round}" - ); - - // No write from the refused thread: main moved once, to the winner's - // Layer, and has exactly one new Layer. - assert_eq!(main_head(&fixture), winner_layer, "round {round}"); - let layers_after = main_layers(&fixture); - assert_eq!(layers_after.len(), layers_before.len() + 1, "round {round}"); - assert_eq!(layers_after[0], winner_layer, "round {round}"); - assert_eq!( - &layers_after[1..], - &layers_before[..], - "round {round}: main gained exactly the winner's Layer" - ); - - // The winner's file is in main; the loser's is not, and its Branch - // did not move. - let read = fixture.call( - "fs.read", - vec![ - ("branch", SdkValue::text("main")), - ("path", SdkValue::text(winner_file.as_str())), - ], - ); - assert_eq!(read.get_text("text"), Some(winner_text.as_str())); - let names = main_paths(&fixture); - assert!( - !names.iter().any(|path| path == loser_file), - "round {round}: the refused Branch's file reached main: {names:?}" - ); - assert_eq!( - branch_field(&fixture, loser_name, "head").as_deref(), - Some(loser_head.as_str()), - "round {round}: the refused Branch did not move" - ); - } - fixture.cleanup(); -} diff --git a/engine/crates/layerfs-store-service/tests/scale_lineage_cap.rs b/engine/crates/layerfs-store-service/tests/scale_lineage_cap.rs index 3ddde057f1..82ca29b0d8 100644 --- a/engine/crates/layerfs-store-service/tests/scale_lineage_cap.rs +++ b/engine/crates/layerfs-store-service/tests/scale_lineage_cap.rs @@ -182,12 +182,7 @@ fn sdk_merge(fixture: &Fixture, branch: &str, with: &str, name: &str) -> Work { fn sdk_publish(fixture: &Fixture, branch: &str) -> (Work, SdkValue) { let before = fixture.work(); - // These publishes measure the deliberate-overwrite path (#691), so they - // ask for it: a default publish now refuses to skip main's Layers. - let published = fixture.call( - "branch.publish", - vec![("branch", text(branch)), ("force", SdkValue::Bool(true))], - ); + let published = fixture.call("branch.publish", vec![("branch", text(branch))]); assert_eq!(published.get_text("kind"), Some("Added"), "{published:?}"); (fixture.since(before), published) } diff --git a/engine/crates/layerfs-store-service/tests/sdk_path_removal.rs b/engine/crates/layerfs-store-service/tests/sdk_path_removal.rs new file mode 100644 index 0000000000..de94729385 --- /dev/null +++ b/engine/crates/layerfs-store-service/tests/sdk_path_removal.rs @@ -0,0 +1,135 @@ +//! `project.remove_file` over the public Store method table: one marker hides +//! a read and a stat with a `Removed` detail, a listing and a glob omit the +//! entry and count it, the subtree matches only at `/`, and a post-frontier +//! re-creation is visible again. The path grammar is refused as `InvalidInput`. +#[path = "support/installed_store.rs"] +#[allow(dead_code)] +mod installed; +#[path = "support/sdk.rs"] +#[allow(dead_code)] +mod support; + +use layerfs_bridge::control::SdkValue; +use std::fs; +use support::{paths, write, Fixture}; + +fn text(value: &str) -> SdkValue { + SdkValue::text(value) +} + +fn remove(fixture: &Fixture, path: &str) -> SdkValue { + fixture.call("project.remove_file", vec![("path", text(path))]) +} + +fn read(fixture: &Fixture, path: &str) -> Result { + fixture.try_call( + "fs.read", + vec![("branch", text("work")), ("path", text(path))], + ) +} + +#[test] +fn a_removed_path_is_hidden_in_reads_stat_list_and_glob_until_recreated() { + let fixture = Fixture::new("sdk-path-removal", |source| { + fs::write(source.join("a"), "a\n").unwrap(); + fs::write(source.join("ab"), "ab\n").unwrap(); + 2 + }); + fixture.fork("work"); + fixture.apply("work", vec![write("/x", "x\n")]); + + let removed = remove(&fixture, "/a"); + assert_eq!(removed.get_text("kind"), Some("Removed")); + assert_eq!(removed.get_text("project"), Some("installed")); + assert_eq!( + removed.get_text("path"), + Some("a"), + "the stored normalized path" + ); + let frontier = removed.get_int("frontier_serial").unwrap(); + assert!(frontier >= 2, "the Store position covers the commits"); + let removed_ms = removed.get_int("removed_ms").unwrap(); + + // Reads and stats refuse with the one detail shape. + let refusal = read(&fixture, "/a").expect_err("hidden read"); + assert_eq!(refusal.code, "Removed"); + let Some(SdkValue::List(hidden)) = refusal.detail.get("paths") else { + panic!("no paths detail: {:?}", refusal.detail); + }; + assert_eq!(hidden.len(), 1); + assert_eq!(hidden[0].get_text("path"), Some("/a")); + assert_eq!(hidden[0].get_int("frontier_serial"), Some(frontier)); + let stat = fixture + .try_call( + "fs.stat", + vec![("branch", text("work")), ("path", text("/a"))], + ) + .expect_err("hidden stat"); + assert_eq!(stat.code, "Removed"); + + // The subtree matches only at a `/` boundary. + let sibling = read(&fixture, "/ab").expect("ab stays visible"); + assert_eq!(sibling.get_text("text"), Some("ab\n")); + + // A listing omits the entry and counts it; a glob does the same. + let listed = fixture.call( + "fs.list", + vec![("branch", text("work")), ("path", text("/"))], + ); + let names = paths(&listed); + assert!(names.contains(&"/ab"), "{names:?}"); + assert!(names.contains(&"/x"), "{names:?}"); + assert!(!names.contains(&"/a"), "{names:?}"); + assert_eq!(listed.get_int("removed_count"), Some(1)); + let globbed = fixture.call( + "fs.glob", + vec![ + ("branch", text("work")), + ("pattern", text("/*")), + ("path", text("/")), + ], + ); + let matched: Vec<&str> = match globbed.get("items") { + Some(SdkValue::List(items)) => items + .iter() + .filter_map(|item| match item { + SdkValue::Text(text) => Some(text.as_str()), + _ => None, + }) + .collect(), + other => panic!("no glob items: {other:?}"), + }; + assert!(matched.contains(&"/ab"), "{matched:?}"); + assert!(!matched.contains(&"/a"), "{matched:?}"); + assert_eq!(globbed.get_int("removed_count"), Some(1)); + + // No newer project node: the repeat is UpToDate and keeps the stored values. + let repeat = remove(&fixture, "/a"); + assert_eq!(repeat.get_text("kind"), Some("UpToDate")); + assert_eq!(repeat.get_int("frontier_serial"), Some(frontier)); + assert_eq!(repeat.get_int("removed_ms"), Some(removed_ms)); + + // A post-frontier re-creation is an ordinary visible write. + fixture.apply("work", vec![write("/a", "again\n")]); + let again = read(&fixture, "/a").expect("the re-creation is visible"); + assert_eq!(again.get_text("text"), Some("again\n")); + fixture.cleanup(); +} + +#[test] +fn the_remove_file_path_grammar_is_refused() { + let fixture = Fixture::new("sdk-path-grammar", |source| { + fs::write(source.join("a"), "a\n").unwrap(); + 1 + }); + fixture.fork("work"); + for bad in [ + "", "/", "//a", "a//b", "a/", "./a", "a/./b", ".", "..", "a/b/", "a/../b", + ] { + let refusal = fixture + .try_call("project.remove_file", vec![("path", text(bad))]) + .unwrap_err(); + assert_eq!(refusal.code, "InvalidInput", "{bad:?} was accepted"); + } + fixture.cleanup(); +}