Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ The LayerFS engine: Workspaces, file I/O, metadata and Commit.
| --- | --- | --- | --- |
| [fio](fio/) | speed | buffered sequential and random file I/O | 2026-11-06 · `76a32a9a` (pgr2-six) · Diagnostic terminal: INCOMPLETE. 36 registered rows; execution {'PASS': 36}. Final infrastructure cleanup: INCOMPLETE · [report](result/2026-11-06/fio/76a32a9a10e7b7b1379699ece52d45e233da5a50/pgr2-six/report.md) |
| [fs-bench-cloudflare-computer](fs-bench-cloudflare-computer/) | speed | Cloudflare's fs-bench.sh on LayerFS | 2026-11-07 · `85a0865c` · 8 of 8 rows ran all twelve scenarios (8 PASS); final cleanup COMPLETE · [report](result/2026-11-07/fs-bench-cloudflare-computer/85a0865cde92f1d85a4e8bf402c597c935ff8db6/report.md) |
| [fs-bench-pro](fs-bench-pro/) | speed, correctness | cold project export from each machine into a host folder | 2026-11-08 · `97fac452` · All 8 export samples QUALIFIED: every machine exported both fixtures into a host folder with exact content, from a cold Store, within budget · [report](result/2026-11-08/fs-bench-pro/97fac452bd4c1f834bdc2df005cf01caf8b415d0/report.md) |
| [fs-bench-quick](fs-bench-quick/) | speed | CLI case times for eighteen small file workloads | 2026-11-06 · `76a32a9a` (pgr2-final) · Diagnostic terminal: INCOMPLETE. 162 registered rows; execution {'PASS': 162}. Final infrastructure cleanup: INCOMPLETE · [report](result/2026-11-06/fs-bench-quick/76a32a9a10e7b7b1379699ece52d45e233da5a50/pgr2-final/report.md) |
| [fs-storage-dsh](fs-storage-dsh/) | storage | Store bytes for a real repository history replayed through Workspace Commits | 2026-11-06 · `7691ac20` · 4 of 4 rows PASS (applied, committed, measured and verified at state 157); net Store bytes D10 47,927,296 B SQLite / 53,272,576 B PostgreSQL, D03 61,014,016 B / 68,419,584 B, against a Git pack + idx of 41,805,317 / 48,166,223 B; final cleanup COMPLETE · [report](result/2026-11-06/fs-storage-dsh/7691ac20b0a8c8b741dd2b701e7aa77b63aaa8a3/report.md) |
| [fs-storage-history](fs-storage-history/) | storage | Store bytes by history shape: long history, branches, forks and projects | 2026-11-06 · `7691ac20` · All 14 rows (7 cases × Docker × SQLite, PostgreSQL) PASS with every Branch head verified on final integration 7691ac20b0 (= 8b701f42f0 + #92 + #91 + #90, PRs pending owner merge); against the Phase 0 reference b4fa173e40, net Store bytes fell in every comparable row (SQLite −2.93 % … −17.93 %, PostgreSQL −18.03 % … −53.26 %), Σ new_metadata fell 19–41 % with payload unchanged, and PostgreSQL ÷ SQLite is now 1.07–1.10× in every case (Phase 0: 1.11–2.22×). Final cleanup COMPLETE · [report](result/2026-11-06/fs-storage-history/7691ac20b0a8c8b741dd2b701e7aa77b63aaa8a3/report.md) |
Expand Down
3 changes: 3 additions & 0 deletions fs-bench-pro/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
/results/
/environments.local.json
/tools/resident-linux-*
104 changes: 104 additions & 0 deletions fs-bench-pro/AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
# fs-bench-pro agent instructions

> Status: Current general guide.

These instructions apply to `fs-bench-pro/` and extend the repository-root
[CLI and cache rules](../AGENTS.md). This file is the suite's current contract;
[cases.py](src/cases.py) pins the cases and expected trees. Reading it does not
start a campaign or authorize repeating completed samples.

This is a new suite, created on 2026-11-08 at the owner's instruction. The
earlier fs-bench-pro in `.archive/` stays frozen: do not edit, build or run it.

## Families and cases

Each family is a case group in [cases.py](src/cases.py) with its own workload
version. Only `export` exists:

| Case | Work | Fixture |
| --- | --- | --- |
| E01 | `project export` of 10,000 small files, 50 symlinks, 132 folders (46,486,561 bytes) | `small` |
| E02 | `project export` of one 256 MiB file and a README (268,435,480 bytes) | `large` |

A case exports branch `main` of one project from a SQLite Store on the selected
machine (native, Docker, Lima, Apple Container) into a fresh folder on the
macOS host. Workload version `export-v1` pins the fixtures' generator and
manifests. Changing a fixture, the command or the verifier needs a new
workload version. Add a family by adding its cases, fixtures and verifier
here, not a second runner.

## Select the product

- Record the full source commit, branch and whether it is pushed; build the
CLI and the Linux bundle from that one snapshot and record their hashes.
- Use the absolute CLI path from the config file, never `layerfs` on `PATH`.
- Use a private `LAYERFS_HOME` and `XDG_CACHE_HOME`, and
`LAYERFS_CONSTRUCTION_WORKERS=1`. Store profile: Disposable only.

## Prepare (untimed, through the CLI)

```text
layerfs store create <store>
layerfs project init <store>/small <fixtures>/small
layerfs project init <store>/large <fixtures>/large
layerfs store push <store> <machine> # guests only
```

`run.py setup` does this for every machine. Init runs on native from the same
fixture bytes; a guest Store is then moved, not rebuilt. Prepared Stores are
reused across samples because export only reads them; the exported folder is
the sample's output and is removed after verification.

## Cold control

A cold claim needs enforcement and evidence on owned resources only. Never
purge the host's or Docker Desktop's shared cache.

- Guests: `tools/resident.c`, built static and copied to `/tmp/resident` in the
guest, runs `--evict` (fdatasync + `posix_fadvise(DONTNEED)`) on the Store's
own files, then counts their resident pages with mincore.
- Native: the home sits on an owned APFS disk image; detach and re-attach it,
then count the Store's resident pages with mincore.
- Any resident page or a failed control: INELIGIBLE. The control's cost is
overhead, recorded apart from case time.

Limit to state in every report: the guest control empties the guest's page
cache only. The host may still cache the VM's disk file, which belongs to the
shared VM and is not purged, so a guest's cold state is weaker than native's.

## Run one sample per command

```text
python3 src/run.py run --env environments.local.json --case <E01|E02> --machine <machine>
```

- One sample per case per machine; no warm-up, rerun or best-of. A failure
keeps its receipt and is reported.
- Order: E01 then E02; within a case native, Docker, Lima, Apple Container.
- Record the 1-minute load (the runner does) and any build or test another
session runs at the same time.
- The timed export has a 15 s budget, verification 10 s, and every command a
wall limit of at most 120 s.
- A labeled diagnostic (for example a warm export to separate cold read cost)
may follow, outside the receipts, and is reported as a diagnostic.

## Verdicts

QUALIFIED needs execution PASS (exit 0 and counts equal to the fixture),
verification PASS (manifest equal to the pinned hash), a shown cold Store and
the export within budget. Otherwise OVER_BUDGET, INELIGIBLE or FAIL.
Receipts record the harness version; a harness change never relabels an
earlier receipt.

## Report

Publish `result/<date>/fs-bench-pro/<full-layerfs-sha>/report.md` per the root
rules: bold outcome sentence, configuration and identities, one table per case
(native first) from `src/report.py`, observations, limits, failures and
cleanup. Then run `python3 shared/src/readme_index.py`. Receipts stay local.

## Cleanup

`run.py cleanup` removes the family's Stores. Remove machines with the CLI,
detach the disk image and delete fixtures and outputs you created; report
anything kept.
127 changes: 127 additions & 0 deletions fs-bench-pro/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
# fs-bench-pro — cold project export from each machine into a host folder
> group: filesystem · measures: speed, correctness

## Basic info

| | |
| --- | --- |
| Tool | In-house: Python standard library in [src/](src/), plus a small static C tool for cold-cache control in [tools/](tools/); every LayerFS step uses the public CLI |
| Product | LayerFS 2.0 with `project.export` ([layerfs-project#53](https://github.com/Ephemeral-AI-Lab/layerfs-project/issues/53)), pinned per campaign |
| Machines | macOS host: native, Docker, Lima, Apple Container |
| Store backends | SQLite (Disposable) |
| Families | `export` (E01, E02); more families are added as their own case groups |
| Status | Current general guide. New suite (2026-11-08); the earlier fs-bench-pro is archived in `.archive/` and stays frozen |
| Frozen rules | [AGENTS.md](AGENTS.md) (cases, cold control, verdicts, cleanup); root [CLI and cache rules](../AGENTS.md) |

Agents must read the root [CLI and cache rules](../AGENTS.md) and the
[suite instructions](AGENTS.md) before preparing or running selections.

The suite root holds documentation and the configuration example. Python
implementation lives in `src/`, focused checks in `tests/`, and the guest
cache tool's source in `tools/`. [cases.py](src/cases.py) pins the cases and
the expected trees.

## Overview

fs-bench-pro measures one product operation per family from a declared,
enforced cold state, with an independent content check. The `export` family
times `layerfs project export <store>/<project> <host folder> --progress`:
the Store lives on one machine and the exported tree must land, complete and
exact, in a fresh folder on the macOS host.

| Case | Work | Fixture |
| --- | --- | --- |
| E01 | Export 10,000 small files (46.5 MB), 50 symlinks, 132 folders | `small` |
| E02 | Export one 256 MiB file and a README | `large` |

Fixtures come from [fixture.py](src/fixture.py): deterministic, incompressible
and distinct per file, so chunk dedupe saves nothing. Their manifest hashes
(path, kind, mode, size, content hash or link target, mtime) are pinned in
[cases.py](src/cases.py).

One `run` is one sample of one case on one machine:

1. **Cold control (untimed).** Guests: `resident --evict` syncs and drops the
Store's own files from the guest page cache, then `resident` counts the
pages still resident (mincore). Native: the home lives on an owned APFS disk
image, which is detached and re-attached; mincore then counts the Store's
resident pages. Any resident page, or a failed control, makes the row
INELIGIBLE.
2. **Timed export.** One CLI process; each progress line is stamped as it
arrives. The export wall time is the case time.
3. **Verification (untimed).** The CLI's counts must match the fixture, and the
manifest of the exported folder must equal the pinned hash. No staging
folder may be left behind.
4. **Release.** The exported folder is removed; the receipt is written to
`results/<run-id>/receipt.json` (ignored, local).

Verdicts: **QUALIFIED** (execution and verification PASS, cold shown, export
within budget), **OVER_BUDGET**, **INELIGIBLE** (cold not shown) or **FAIL**.

## Setup

### Prerequisites

- A LayerFS CLI and Linux bundle built from one clean source snapshot, with
`project export` (`<cli> help project export`).
- Ready machines for the guests you select, created through the CLI
(`layerfs machine …`), each with its own SQLite Store.
- An owned APFS sparse disk image mounted where the private `LAYERFS_HOME`
lives, so native cold control never touches a shared volume:

```bash
hdiutil create -size 20g -type SPARSE -fs APFS -volname lfs-bench native.sparseimage
hdiutil attach -nobrowse -mountpoint vol native.sparseimage
```

- The guest tool, built once for the guests' architecture:

```bash
docker run --rm -v "$PWD/tools:/w" -w /w gcc gcc -O2 -static -o resident-linux-aarch64 resident.c
```

- The fixtures: `python3 src/fixture.py <fixtures>/small small` and
`python3 src/fixture.py <fixtures>/large large`; each prints its manifest,
which must equal the pinned one.

### Config file

Copy [environments.example.json](environments.example.json) to
`environments.local.json` (ignored) and fill in absolute paths, the guest exec
prefixes and the campaign's identities. `XDG_CACHE_HOME` points at a private
cache so the bundle never mixes with another session's.

### One selection

```bash
python3 src/run.py setup --env environments.local.json
python3 src/run.py run --env environments.local.json --case E01 --machine lima
python3 src/report.py <run-id>...
python3 src/run.py cleanup --env environments.local.json
```

`setup` creates one Store per machine with both fixtures, pushes guest Stores
and installs the guest tool. `cleanup` removes those Stores; machines, the disk
image and fixtures are released by the caller.

### Time limits

The timed export has the 15 s default budget; verification has 10 s. No case
has a declared exception. The cold control's cost (up to 12 s for a disk-image
detach) is enforcement overhead, recorded as `cold.control_s` and never case
time.

### Focused checks

```bash
python3 -m unittest discover -s tests -v
```

## Past runs

<!-- runs:begin -->
| Date | LayerFS commit | Outcome | Report |
| --- | --- | --- | --- |
| 2026-11-08 | `97fac452bd` | All 8 export samples QUALIFIED: every machine exported both fixtures into a host folder with exact content, from a cold Store, within budget | [report](../result/2026-11-08/fs-bench-pro/97fac452bd4c1f834bdc2df005cf01caf8b415d0/report.md) |
| 2026-11-08 | `4cd63f16dd` | Export works from all four machines into a host folder: 7 of 8 samples completed with exact content (6 QUALIFIED, native E01 OVER_BUDGET under the first harness rule); Apple Container E01 FAILED mid-stream | [report](../result/2026-11-08/fs-bench-pro/4cd63f16ddbea5791c88fe34be369de27bcfff81/report.md) |
<!-- runs:end -->
57 changes: 57 additions & 0 deletions fs-bench-pro/environments.example.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
{
"cli": "/ABSOLUTE/layerfs-checkout/target/release/layerfs",
"home": "/ABSOLUTE/bench-root/vol/home",
"xdg_cache": "/ABSOLUTE/bench-root/xdg-cache",
"fixtures": "/ABSOLUTE/bench-root/fixtures",
"out": "/ABSOLUTE/bench-root/out",
"guest_tool": "/ABSOLUTE/bench-root/tools/resident-linux-aarch64",
"native": {
"store": "s-native",
"image": "/ABSOLUTE/bench-root/native.sparseimage",
"mount": "/ABSOLUTE/bench-root/vol"
},
"machines": {
"docker": {
"machine": "docker",
"store": "s-docker",
"guest": [
"docker",
"exec",
"-i",
"lfs-<home-id>-docker"
]
},
"lima": {
"machine": "lm1",
"store": "s-lima",
"guest": [
"limactl",
"shell",
"lfs-<home-id>-lm1",
"sudo"
]
},
"apple-container": {
"machine": "ac1",
"store": "s-ac",
"guest": [
"container",
"exec",
"-i",
"lfs-<home-id>-ac1"
]
}
},
"identity": {
"source": "<fill per campaign>",
"cli_sha256": "<fill per campaign>",
"bundle_json_sha256": "<fill per campaign>",
"linux_layerfs_sha256": "<fill per campaign>",
"profile": "<fill per campaign>",
"host": "<fill per campaign>",
"docker": "<fill per campaign>",
"lima": "<fill per campaign>",
"apple_container": "<fill per campaign>",
"store": "<fill per campaign>"
}
}
39 changes: 39 additions & 0 deletions fs-bench-pro/src/cases.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
"""The export family: pinned cases, fixtures and expected results.

A case exports one committed project (`main`) from a SQLite Store on one
machine to a fresh folder on the macOS host with the public
`layerfs project export <store>/<project> <destination> --progress`.
"""

FAMILY = "export"
WORKLOAD_VERSION = "export-v1"
MACHINES = ("native", "docker", "lima", "apple-container")

# fixture kind -> expected tree, from fixture.py at WORKLOAD_VERSION
FIXTURES = {
"small": {
"manifest": "8638d2973cfce62d1169e771f9649a35c5d2d78f5f553695c6342b8544e50c1e",
"files": 10000,
"directories": 132,
"symlinks": 50,
"bytes": 46486561,
},
"large": {
"manifest": "a4735ecdaf8a95a18953f474c5ee616b47eca599769843ac56ccd148b8b5e205",
"files": 2,
"directories": 0,
"symlinks": 0,
"bytes": 268435480,
},
}

CASES = {
"E01": {"fixture": "small", "work": "export 10,000 small files (46.5 MB) with 50 symlinks and 132 folders"},
"E02": {"fixture": "large", "work": "export one 256 MiB file and a README"},
}

# complete command (cold control + timed export + verification + cleanup)
BUDGET_S = 15.0
# declared exceptions, each up to 25 s (none yet)
DECLARED_EXCEPTIONS: dict = {}
VERIFY_BUDGET_S = 10.0
Loading