Repository navigation
Description
Nine-family fast baseline — completed, 2026-09-05
The user's latest scope and simplification instructions supersede the expanded two-arm/581-performance/119-proof replay: one complete current-product performance sample per registered case, reusable preparation, and quick selected independent verification. No SDK full-byte option or dedicated full-byte test. Other families are deferred to #39.
- Unified host SDK/Workspace/embedded SQLite plus Linux daemon/real FUSE/Exec through the existing authenticated binding. Identical observed container limits: 2 CPUs, 2 GiB, no swap, 256 PIDs, no Docker data mounts; host CPU remains separately scoped and uncapped.
- All nine families passed: payload_create_read, dedup_workspace_reuse, dedup_cross_file, dedup_cdc_locality, edit_length_preserving, edit_length_changing, edit_canonical_chunk_count, init_namespace, store_footprint.
- 118/118 single performance samples PASS; exact registry coverage, declared timers, source/image provenance and cleanup checked. Maximum product time 4.817 seconds, below 15 seconds. Single samples establish a baseline; no new median/scaling-distribution qualification is claimed.
- 27/27 selected independent proofs PASS, 94.268 seconds combined proof wall, maximum 14.571 seconds. Separate preparation misses are preserved and excluded from that sum. Coverage includes all 14 SDK edit/outcome shapes at 1 MiB plus capped 500 MiB insert; all four namespace sizes; all three main storage controls; selected 500 MiB checks for each original family and the proof-only CDC boundaries.
- SDK proof uses prepared canonical roots/counts, lengths, inode/payload retention and at most 192 KiB replacement/boundary bytes through canonical and FUSE reads. Capped 500 MiB proof fell from 16.277 to 4.370 seconds.
- Namespace proof uses exact imported file/byte counts, persisted root equality and at most 11 deterministic FUSE file samples with metadata and prefix-byte checks. The 100,000-file proof passed in 5.615 seconds (reopen/sample check 219.860 ms), after the retained 45.050-second full-tree timeout.
- Storage proof keeps storage accounting/publication/reopen and independent canonical/FUSE edit-boundary checks; full-tree FUSE digest replay removed. Main proofs: 5.566, 7.287 and 2.663 seconds.
- Repeated native-input cache rehashes removed: owned caches validate prepared recipe/ownership; modifying a disposable native cache requires recreating it. SQLite master integrity and independent closed-copy checks remain. Receipts explicitly distinguish this policy and sampled coverage from exhaustive byte/namespace verification.
- Host/Linux builds and strict benchmark Clippy PASS; 22 shared tests PASS before final cache simplification, affected 11 runner tests PASS afterward. No owned sample containers or host sample directories remain.
Implementation is local commit fa33ece0e (not pushed/merged/released). Report: docs/roadmap/0.1/0.1.3/nine-family-fast-baseline.md.
Performance and unchanged SDK proof source: 72f408b467840389be75dc14bd21495a07f6c39b6f2164d0ad6c7aea985e11d9. Final simplified namespace/storage/cache verifier source: aa907534bb9f760048739de4205a29c0c7630ad7de6a7b3c21b5e50e52b66cb3. Product seal is identical: 2a0f830a18ed0b9df7197c2dd0d7df921215b8c78cf60fe07c658715de5efee5. The completed performance baseline was preserved under its producing identity rather than replayed or relabeled after verifier/setup-only changes.
Raw evidence and assessments: benchmark-results/nine-family-fast-baseline/{72f408b467840389,aa907534bb9f7600}/. Prior four-family campaigns, the stopped oversized campaign and all failed receipts remain historical evidence. This completion is for the explicitly simplified fast baseline and sampled proof scope.
Current user-authorized scaling gate — final 2026-09-05 amendment
The user explicitly chose to accept the frozen performance with a strictly < 1.25 normalized adjacent-tier MEDIAN time-growth / actual-work-growth gate and proceed to the host-owned SQLite migration. This supersedes the earlier <1.10 gate and interrupted1.20 proposal. Describe this as a25% scaling-overhead allowance, not proportional scaling.
All other requirements remain unchanged: prescribed114unique samples/seeds1–3, explicit actualwork and every adjacenttier, per-sample15s/runtime/resource limits, no deliberate slowing of smaller tiers, exact applicable semantics and bounded separateverification, authentic Docker/FUSE/no-data-mount setup for the frozen campaign, independent Workspaceclones/nativefresh outputs, cleanup and evidence-bound completion. Preserve rawreceipts and the historical <=1.00/<1.10 misses, individualseed ratios and variability. All30 frozenCandidate3 mediancomparisons clear <1.25; the earlier payload1.105934087 and cross-file mixed1.157331012 misses now clear only the newly amended gate.
Finish remaining frozen assessment/requiredchecks and truthful report updates, then proceed promptly in this same task/worktree with host SDK/coordinator+Workspace/Store/localSQLite and Linux daemon/FUSE/workload via existing authenticated transport. No newHTTP layer, system cache/Docker memory tuning, or separateagent/task/worktree. Host reusable preparation needs protected consistentSQLite masters/backups, independent writable samples and real cache reuse; benchmark topology/source/resource identities and throughput comparisons must be explicit. ExistingDocker-owned results are not host-Store qualification.
The earlier amendment and historical requirements below remain intact as prior records; this paragraph controls the current threshold and sequence.
Current user-authorized scaling gate — 2026-09-05 amendment
For #38 and all four children #41–#44, each required adjacent-tier normalized median time-growth / actual-work-growth ratio must be strictly < 1.10. This permits bounded scaling overhead; it is not a claim of no worse than proportional growth. The user explicitly authorized this amendment in the side conversation and notified the integration owner.
All other gates remain unchanged: original three seeds, 114 unique final candidate samples plus applicable invalidated controls, explicit work dimensions and whole-curve comparisons, every sample within the 15-second product cutoff and resource bounds, no deliberate slowing of smaller tiers, exact applicable semantics and bounded separate verification, Mac + Docker/Linux with real FUSE where applicable and no Docker data mounts, independent Workspace clones, native fresh output Stores, cleanup and evidence-bound issue completion. Individual seed ratios and variability remain visible. This amendment does not automatically close any family.
Older <=1.00 / no-worse-than-proportional clauses below retain the previous strict-gate plan and historical assessments; only their numerical terminal threshold is superseded. Preserve the old raw results and strict-gate failures, and report amended outcomes separately. Candidate2's payload 100→500 median ratio 1.057185870 and cross-file unique ratio 1.053614071 clear this amended median gate; cross-file unique seed3 around1.134 remains an individual variability flag. Complete family qualification remains pending.
Comparison policy amendment — 2026-09-05
The user's latest instruction permits practical historical comparisons and overrides earlier blanket requirements to recollect a matched baseline. Do not rerun or resurrect the old harness merely to obtain a comparison table. Reuse retained baseline receipts/results when the workload and measurement meaning are sufficiently similar; small documented harness differences are acceptable without a new baseline run.
- Classify comparisons as
matched,historical-approximate, ornot-comparable, and state the relevant difference in the table. Different source revisions are expected; preserve their provenance. Historical host/split-owner/APFS results may be used as approximate context where the selected product timer still measures comparable work, but must not be relabeled matched Docker/Linux evidence. - Judge similarity from actual operation, bytes/files/history depth, seed schedule, timing boundary, and resource scope. Do not introduce an arbitrary percentage tolerance or an extra certification campaign. If a difference could plausibly explain the claimed gain, qualify the conclusion rather than attributing it entirely to the product change.
- Materially compacted fixtures (
compact-v2/compact-v3versus original large backgrounds), changed measured operations, and incompatible timing/resource scopes are not direct speedup comparisons. Show their absolute results side by side with the difference; leave speedup blank. Do not pool changed low tiers with old high tiers as an unchanged scaling curve. - Collect a new baseline with the current harness only for a specific unresolved decision that historical evidence cannot answer, using the smallest necessary selection. A full baseline campaign is not a prerequisite. Retain the prescribed candidate coverage, correctness checks, runtime limits and scaling criteria; this exception concerns baseline reuse, not permission to mark untested cases passing.
- Derive tables on demand from compact logs; no benchmark rerun is needed to change presentation. See [v0.1.3] Optimize benchmark infrastructure: Docker/FUSE, reusable setup, three modes, compact logs #45 for field mapping. Preserve sample counts and seeds: N repetitions of one seed are not N different seeds. Verification comes from a separate identity-matched receipt, never inferred from performance PASS.
Infrastructure prerequisite and handoff
Execution starts after #45's benchmark-infrastructure migration is complete and its handoff is ready. The current infrastructure task is 01a06da4-c2ba-7013-92cb-392a348dfdfe (Implement LayerFS Phase 2.1). Do not start product optimization, builds or benchmark collection in parallel with that migration. This issue update defines the successor work; it does not launch a handoff agent.
Read #45 and docs/roadmap/0.1/0.1.3/benchmark-infrastructure-optimization-spec.md at the final handoff revision. Consume its actual migrated entrypoints, compact smoke results, slow-case inventory and unsupported/failure dispositions; do not assume a proposed path is already implemented. A small smoke PASS proves the migrated route works, not that larger tiers or this scaling target pass. Reuse valid smoke/proof evidence; do not rerun it merely to begin a new task. Slow or blocked product cases remain explicit diagnosis input, not a reason to repeat known-slow large runs or label them passing.
#40 is the product-mechanism prerequisite. Use its recorded Approach B implementation/handoff, with the namespace <=2.2-second and 10% CPU-reduction targets still explicitly missed. Do not silently turn that MISS into PASS or restart namespace tuning as a gate for this work. Audit actual adoption of its six mechanisms; the completion summary alone does not establish full caller transfer. The user's sequencing is infrastructure completion, then the #38 optimization handoff.
Two implementation tracks after #45
| Track | Child issues | Shared ownership and initial work | Final unique candidate samples |
|---|---|---|---|
| Native initialization | #42 cross-file CAS; #43 CDC locality | One owner for native discovery/task distribution/initial-tree construction and its admission adapter. Begin with #42 as the first controlled exercise, then reuse useful changes in #43. | 30 + 60 = 90 |
| Workspace creation and Commit | #41 payload creation; #44 Workspace unique-content reuse | One owner for shared capture/candidate construction/admission/continuation. Preserve one-large-file vs per-file-fsync differences. | 12 + 12 = 24 |
The full parent scope remains four families, ten affected curves and eleven historical adjacent-tier violations. Final coverage is 114 unique candidate samples, before invalidated regression controls. CAS has one shared tier-1 anchor, reused across its three curves. Final counts are not iteration-loop requirements or forecasts of easy fixes.
The two tracks may investigate and implement independently, but do not assign four competing owners to the same files. Common content/Store primitive changes are coordinated, integrated once and checked in both applicable tracks. A shared fix may satisfy a sibling without another patch; every original curve still needs its own qualification. #39's live bulk/delete/read/Git/history recovery is outside this execution scope, although reusable fixes may later help it.
Mechanism adoption before new implementation
Audit at 810bb3a589ac58d103483df34bb58ecfe0f0ddf4 compared with pre-#40 4c9b14a6 found:
| Mechanism | Native direct import | Ordinary modified Workspace Commit |
|---|---|---|
| Exact eight-entry metadata cache | Used, with shared helper extracted in #40 | Adopted in frontier/localized builders |
| Checked insertion | Shared helper on direct route; preserve fallback distinctions | Adopted; former separate membership-planning pass removed |
| Carried admission | Owned direct batches | Adopted after construction; object limit 127 -> 8191, byte cap remains <4 MiB |
| Owned producer slabs / overlapping admission | Existing direct pipeline | Not transferred: completed candidate, possible spill, later borrowed-byte copy/admission |
| New sorted affected-page updater | Not used by old initial-directory/inode builders | Adopted with bounded batches and fallbacks |
| Content builders / borrowed extents | Existing canonical builders | Existing builders/incremental paths; largely predate #40 |
| Producer execution / ownership | Existing bounded scoped workers, fixture-shape limits | Local ownership/staging adopted; no shared multi-file producer pool |
Reinspect the final handoff source before treating these findings as current. Do not reimplement cache/checked batching already integrated. The initializer's private direct admission requires an empty Store and cannot simply replace Workspace admission. Use existing owned-object and sorted-tree primitives through the smallest justified adapter; no universal engine or public bulk interface.
For each hypothesis: record predicted work-counter change; measure one representative case/seed or size pair; change one shared mechanism; perform the smallest relevant correctness check and separate bounded verification; compare complete operation wall, total CPU, resources and work counters. Retain misses, revise the hypothesis or reject the change. Neither more workers nor code reuse alone proves scaling improvement. Only expand to the final curve after the route stabilizes.
File layout, shared code and refactoring boundary
Keep the existing product crate ownership. This is a map of relevant existing files, not an instruction to modify them all:
crates/
layerfs-content/src/
filesystem/change.rs # shared exact metadata cache
filesystem/apply.rs # current initial-directory builder
file/rope/build.rs # existing content/CDC construction
tree/batch.rs # sorted directory/inode updates
tree/inode/table.rs # current initial inode-table builder
layerfs-layerstack-store/src/
objects.rs # deferred candidates, owned slabs, checked batches
layerstack.rs # native discovery and task scheduling
workspace.rs # Workspace candidate admission/publication
staging.rs # completed-root stage lifecycle
layerfs-workspace/src/
changes.rs # candidate selection/construction
capture.rs # existing single-file capture
file_io.rs # mutable file I/O
lifecycle.rs # Commit and continuing-view installation
layerfs-fuse/src/
proxy_client.rs / proxy_host.rs # separate live-operation transport
layerfs-sdk/src/client.rs # existing public entrypoints
Shared product code means PortableMetadataCache::get_or_build, existing rope/incremental builders, sorted tree helpers, and insert_checked_object_batch under their current owners. Workspace stage/publication is operation-specific. Native and Workspace object delivery are not yet identical. No new crate, universal engine, public bulk endpoint or generic construction-pool facade is prescribed.
The smallest potential Workspace ownership change is confined to Store objects.rs and workspace.rs: adapt existing test-only DeferredObjectStore::consume_prevalidated_pages to feed owned selected objects into admit_checked_objects, retaining checked insertion and carried bounds. It removes memory-candidate copies only; the spill path still copies and no construction/admission overlap is created. Use it only if copy counters justify the change.
A larger direct-delivery transfer may adapt InitializationObjectSlab, InitializationSlabWriter and the admission accumulator with a narrow nonempty-Store boundary. InitializationSegmentAdmission::new currently requires an empty Store; do not call it unchanged for Workspace. Preserve candidate read-your-writes, reachable-root validation, old roots, collision checks, bounded resources and staged failure semantics. Reuse existing capture instead of rebuilding already captured content; no shared multi-file worker pool currently exists to switch on.
Native work stays in direct_initialize_root_directories_inner and initial-tree callers: test task distribution separately from transfer of build_initial_directory / build_initial_inode_table_from_pairs to the existing sorted helpers. Preserve exact same-seed canonical output and needed fallbacks. Workspace sorted updates are already adopted; investigate repeated bounded batch work rather than reimplementing them.
One native owner handles #42/#43; one Workspace owner handles #41/#44. A single integration owner coordinates shared Store/content edits and serializes resource-sensitive work. Product algorithms must never move into benchmark family modules as shortcuts.
#45's separate benchmark target is shared/{runner.py,runtime.sh,preparation.py}, the sole verify-selected.py, and families/<canonical-id>/{mod.rs,setup.sh,perf.sh,verify.sh}. Family launchers bind IDs and forward arguments; setup, timing, Docker lifecycle, verification and cleanup stay shared. Consume the actual completed layout, not a second harness implementation.
Admission complexity and optimization claims
For current ordinary Workspace admission, let N be selected candidate objects, B their canonical bytes, D bytes of existing-object conflicts, and M preexisting Store objects. Under a logarithmic indexed-operation model:
T_admit = T_candidate_visit + O(B + D + N log(M + N + 1)) + transaction costs + physical I/O.
Admission has bounded additional buffers: <4 MiB canonical batch payload plus up to 8,191 object records, visitor buffers and bookkeeping; SQLite cache is additional. This is not a <4 MiB whole-process bound. Deferred candidates, spill/index/reachability structures, Workspace state, continuation and durable storage are separate costs. #40's 127 -> 8191 object-cap increase reduces applicable transaction overhead, not per-object indexed complexity. Candidate reachability has a documented quadratic ID-spill fallback at very large sets; verify activation before targeting it, rather than labeling the admission loop quadratic.
Every chosen refactor must name its expected effect: remove an extra copy, remove candidate write/readback, reduce repeated tree work, reduce transaction count, or improve overlap. Owned-page movement usually changes constants, not big-O. More workers can reduce wall while increasing CPU. Report relevant counters, total CPU, memory and temporary/durable I/O separately; no asymptotic or zero-copy claim from one faster sample.
Planning documents in docs/roadmap/0.1/0.1.3/: phase-2-shared-code-layout.md, phase-2.1-mechanism-adoption-audit.md, and workspace-admission-complexity.md. Current local copies are in /Users/yifanxu/Ephemeral-AI-Lab/layerfs/docs/roadmap/0.1/0.1.3/; these edits are local and not claimed committed/pushed. The key instructions are included here so this issue is self-contained. Audit source was 810bb3a58; recheck the final #45 handoff source and paths before editing.
API reduction counts and algorithm consolidation
The reviewed target is fewer independent implementations, not an immediate large reduction of public SDK methods. Inventory at product source 810bb3a58: 26 production Client methods plus two feature-gated verification helpers. Reinventory the final #45 handoff source before editing.
| Surface | Initial change | Later candidate |
|---|---|---|
| Public Rust SDK | 0 methods removed; 26 remain, with compatible forwarding wrappers and consistent validation/observation | One versioned Commit-variant reduction, 26 -> 25 only if no other API changes occur; migrate callers and preserve full publication/presentation status first |
| FUSE wire protocol | Preserve acknowledged readonly Pin and current compatibility | 1 identified obsolete request: PinRead, with no current production sender; pruning requires an explicit compatibility disposition and is not a guaranteed net request-count change |
| Internal Rust methods | No validated net deletion count | Count the reviewed implementation diff; a shared helper may remove duplicated bodies without reducing total methods |
| Algorithm area | Current implementations | Proposed target |
|---|---|---|
| Native scheduling/orchestration | 3 | 1 scheduling implementation, with explicit appropriate sinks |
| Admission accumulation/orchestration | 3 | 1 shared bounded accumulator; operation-specific publication stays explicit |
| Native regular-file construction bodies | 2 | 1 |
| Workspace inode-record construction bodies | 2 | 1 |
| Workspace candidate planners | 3 | Eventually 1, after all fallback domains and sparse/resource behavior transfer |
These are not additive method-removal counts, performance promises or mandatory deletion quotas. Compatibility wrappers may remain. Do not create a catch-all endpoint or policy-flag framework merely to reduce enum/method names. Do not remove a route until its real callers and supported states transfer; test-only differential oracles are not extra production routes.
For this parent, native consolidation belongs to #42/#43 and Workspace consolidation to #41/#44. One integration owner coordinates shared content/Store edits. Retain typed SDK operations and useful adapters. Canonical full Commit status, unified edit observation and rejection of ignored Query filters are small consistency fixes, not evidence that another Commit/edit algorithm must be built. Stage/head/ownership behavior must remain explicit.
Implementation proceeds after the completed #45 handoff. Apply only evidence-backed simplifications relevant to the active track; broader public API or wire pruning is a separately validated compatibility change, not a prerequisite that expands every family task. Preserve the 114 unique final candidate samples and original scaling/correctness/resource criteria. Metadata/copy/transaction/tree counters must substantiate any speed or complexity claim.
Detailed local plans: docs/roadmap/0.1/0.1.3/api-algorithm-simplification-audit.md, phase-2-shared-code-layout.md, and workspace-admission-complexity.md in /Users/yifanxu/Ephemeral-AI-Lab/layerfs. These documents are local planning edits, not claimed pushed commits. The counts and scope are included here so the issue is self-contained.
Successor handoff checklist
- Read [v0.1.3] Optimize benchmark infrastructure: Docker/FUSE, reusable setup, three modes, compact logs #45's terminal migration state and pin product, harness, runtime image and prepared-input identities; inspect its four relevant family smoke/slow/failure records.
- Establish one native owner ([v0.1.3][Phase 2] Fix cross-file CAS deduplication scaling #42/[v0.1.3][Phase 2] Fix CDC locality scaling #43), one Workspace owner ([v0.1.3][Phase 2] Fix payload creation scaling #41/[v0.1.3][Phase 2] Fix Workspace unique-content reuse scaling #44) and a single resource-sensitive run schedule/shared-file integration owner.
- Confirm actual [v0.1.3][Phase 2.1] Share construction and Workspace staging; reach 2.2s initialization with lower CPU #40 mechanism adoption and obtain the smallest necessary matched Docker-only baseline for each first hypothesis.
- Implement and verify one evidenced shared change at a time; reuse compatible preparation and exact proof identities.
- Integrate stable changes before final qualification; run all 114 unique prescribed candidate samples with original seeds and only required extra controls, retaining baseline evidence with its matched/approximate classification under the comparison policy amendment.
- Produce the parent before/after assessment from compact logs; leave any failed, timed-out, missing or inconsistent curve open/inconclusive. Infrastructure smoke completion never closes a product gate.
Migrated benchmark contract
- All fixture/reference preparation, sample copying, SDK/coordinator, Store, daemon and product work run inside Docker/Linux. Host orchestration and compact result collection only. No runtime data bind mounts, named/anonymous volumes, Docker socket, APFS clone or host-side measured SDK/Store. Real FUSE/Exec remains where the family operation requires it; pure imports do not gain a synthetic timed Workspace lifecycle.
- Use the migrated
families/<canonical-family-id>/{mod.rs,setup.sh,perf.sh,verify.sh}interfaces backed by shared infrastructure.setup.shis optional prewarming; perf acquires compatible preparation automatically. Historical wrappers are forwarding/archival compatibility only as recorded by [v0.1.3] Optimize benchmark infrastructure: Docker/FUSE, reusable setup, three modes, compact logs #45; do not build a parallel runner. - Default
--perf-fastmeans one complete selected workload sample, not smaller data or weaker checks.--perf-samples Nrepeats the same selected seed/input sequentially; sample ordinal is not seed, repetition or history depth.--verificationis an explicit separate selection through the single shared verifier. Do not use N to invent new seeds or replace the final prescribed seed schedule. - Import measurements reuse prepared source files but always initialize a fresh/absent output Store (
fresh-output); reject--setup clone. Post-initialization Workspace cases may use authenticated--setup cloneorfresh, with independent writable state and SQLite copy-up completed outside product timing. Record actual strategy and cache policy, not an unsupported cold-cache/reflink claim. - Reuse retained historical baselines under the comparison policy amendment above; no old-harness rerun or blanket baseline recollection is required. When a new baseline is necessary for a specific unresolved decision, run only the selected cases with the current harness and align platform, CPU, setup/cache/input and timing boundaries with the candidate.
- Retain
perf.jsonl(header, attempted samples, summary) orverification.json, plus an optional boundedfailure.log(default <=1 MiB). No automatic Markdown/HTML/CSV reports, per-sample evidence trees, retained Stores/fixtures/containers, or raw sampler streams. Summaries required for issue closure are derived from compact logs on demand. Prepared cache remains finite and benchmark-owned; reuse [v0.1.3] Optimize benchmark infrastructure: Docker/FUSE, reusable setup, three modes, compact logs #45's cleanup and retention rules. - Verification has a hard 59-second end-to-end deadline starting before setup/authentication and ending after cleanup/receipt publication. Reuse existing independent oracles and identity-matched proofs, explicitly report omissions, and return PASS/FAIL/TIMEOUT/INCOMPLETE. Unsupported long proofs are not shortened or relabeled passing. No unchanged automatic retries or withdrawn exhaustive Phase 1 replay.
- Investigation and independent source edits may run concurrently after handoff. Serialize resource-sensitive builds, preparation, verification and performance across both tracks, not merely within each output directory. Use one coordinator for measurement scheduling and shared-file integration; obey aggregate <=8 participating CPUs and stricter applicable family limits.
Problem and outcome
Several cases completed below Phase 1's 15-second product-time cutoff but became slower per unit of work at larger tiers. Passing that cutoff did not establish good scaling. Fix the shared causes so the affected curves grow no worse than proportionally with actual workload, while preserving all operations, correctness, resource bounds, and existing smaller-tier performance.
This issue owns previously active cases with observed scaling violations. Companion issue #39 owns the 15 runtime-suppressed cases. A shared implementation may address both; retain separate case-level evidence and closure criteria.
Measured inventory
These are observations from the retained Phase 1 selection, not results on an optimized candidate. Median ratios use unrounded product-time samples. Over linear = (larger median / smaller median) / (larger work / smaller work) - 1. Seeds over counts corresponding seed pairs exceeding proportional growth; it is descriptive, not a statistical confidence claim.
| Exact curve prefix | Tier transition | Actual work growth | Median seconds | Time growth | Over linear | Seeds over |
|---|---|---|---|---|---|---|
payload-create (MiB suffix m) |
100→500 | 5.000× | 0.475692→3.744340 | 7.871× | 57.4% | 3/3 |
dedup-cross-file-identical |
100→500 | 5.000× | 0.156959→0.808296 | 5.150× | 3.0% | 2/3 |
dedup-cross-file-mixed |
100→500 | 5.000× | 0.203634→1.056044 | 5.186× | 3.7% | 3/3 |
dedup-cross-file-unique |
100→500 | 5.000× | 0.176675→0.941368 | 5.328× | 6.6% | 3/3 |
dedup-cdc-common-body |
10→100 | 9.182× | 0.021993→0.268495 | 12.208× | 33.0% | 3/3 |
dedup-cdc-delete |
100→500 | 4.960× | 0.169896→0.910435 | 5.359× | 8.0% | 3/3 |
dedup-cdc-insert |
100→500 | 4.961× | 0.171250→0.915639 | 5.347× | 7.8% | 3/3 |
dedup-cdc-overwrite |
100→500 | 4.960× | 0.170187→0.895133 | 5.260× | 6.0% | 3/3 |
dedup-cdc-scattered |
1→10 | 5.500× | 0.005955→0.056221 | 9.441× | 71.7% | 3/3 |
dedup-cdc-scattered |
100→500 | 4.960× | 0.289743→1.681869 | 5.805× | 17.0% | 2/3 |
dedup-workspace-unique |
100→500 | 5.000× | 0.726023→3.847812 | 5.300× | 6.0% | 3/3 |
Four affected families, ten affected curves, eleven adjacent-tier violations. Include the whole existing four-tier curve in final scaling evaluation, not only the listed pair.
Additional variability finding: dedup-cdc-common-body 100→500 clears the median comparison but exceeds proportional growth by 8.7% using the mean, with two of three seed pairs over. Keep this visible alongside the primary inventory.
CDC imports one fixed 1 MiB reference plus N variants. Use actual logical input bytes, including the reference and insertion/deletion length differences: 1→10 variants is approximately 5.5× bytes, not 10×. Cross-file CAS uses N prepared 1 MiB files; Workspace reuse uses N new 1 MiB files on the unchanged base. CAS/CDC timings are initialization from prepared files, not SDK-edit timers. Do not compare unlike operations or infer asymptotic complexity from three samples.
Evidence and starting investigation
- Final Phase 1 results.
- Qualified selection:
benchmark-results/fs-bench-pro/phase1-v013/results/review.json, overlaid by exact case/seed fromqualification/readonly-pin-performance-checkpoint/incremental-performance-rows.json,qualification/rename-cache-performance-checkpoint/incremental-performance-rows.json, andqualification/cdc-delete-collision-final-checkpoint/incremental-performance-rows.json; exclude the persistent suppression inventory. - Affected baseline sources are
7948df2d, plus correcteddedup-cdc-delete-500seed 3 ate24a3b34. Preserve producing source, environment, input, and seed identities; compare against a controlled candidate and report source differences explicitly. - Payload creation 100→500: Exec medians 0.186139→1.428129 seconds (7.67×); Commit 0.275633→2.291853 (8.31×). Investigate both paths using existing phase/CPU/I/O/admission receipts. The baseline-derived 500 MiB proportional bound is 2.378459 seconds; observed median is 3.744340 seconds.
- Three seed growth factors for payload creation are 5.36×, 7.92×, and 14.05× for 5× input; this finding is not caused solely by one slow observation.
Acceptance
- Diagnose shared product causes from existing evidence before selecting implementation changes; reuse ongoing relevant optimization work rather than creating competing replacements.
- For each affected curve, record actual work denominators and all adjacent-tier ratios using the original three seeds and prescribed operations. Require normalized median time growth / actual work growth strictly <1.25 under the final user amendment; retain earlier gate assessments. Report individual samples, mean, min/max, and corresponding-seed ratios; unresolved inconsistent samples are explicitly inconclusive, not a blanket family pass. Do not introduce an unapproved tolerance or discard inconvenient samples.
- Do not obtain a passing ratio by slowing smaller cases. Preserve their baseline performance and show before/after absolute timings as well as normalized growth. Keep every candidate sample within the 15-second product-time cutoff.
- Preserve byte/namespace/Commit/CAS/CDC semantics, required failure propagation, resource bounds and cleanup. Per-file maximum remains 500 MiB and aggregate logical files remain under 1 GiB at every workload state.
- Publish one source-bound before/after table covering the affected curves and the specific related regression controls invalidated by the changes. No claims of sublinear asymptotic complexity or storage-savings qualification based only on latency.
Fast iteration
Follow the two-track and migrated-infrastructure contracts above. Start with a small explicit diagnostic selection; use 1/10-tier smoke evidence from #45 where identity-compatible, then a larger pair only when needed to explain scaling. Preserve actual denominators, original case definitions and public operations. Recollect only invalidated evidence and perform the scoped final comparison once. Required product work stays inside its timer; setup, independent verification and outer cleanup remain separately observable. This update schedules work after the infrastructure handoff; it does not launch execution now, authorize release publication or weaken correctness/scaling criteria.
Host-owned SQLite follow-up — completed in the same task/worktree
- Reuse existing authenticated ContainerBinding/ProxyHost with host SDK/coordinator, Workspace/capture/spool and embedded SQLite; container daemon/FUSE/workload; no new HTTP or data mounts.
- Implement explicit host topology and separate host binary/source versus Linux image identities, without relabeling frozen Docker evidence or changing system cache/memory settings.
- Use ignored project-local benchmark-results/host-store for compatible protected preparation, independent samples and compact results; preserve ordinary caller-selected Store paths and distinguish evictable cache from durable backups.
- Prove safe consistent SQLite snapshots, miss→hit preparation reuse across irrelevant execution changes, identical starting state, independent writable sample, unchanged master, bounded cleanup/eviction and native fresh-output initialization.
- Prove host publication/container-visible bytes and metadata, continuation/failure handling, then compare same optimized product/cases/seeds at100/500MiB with Exec/Commit/transport and distinct host/container resources. Similar throughput is a goal; <1.25 is not a topology-regression allowance.
Activity
yifanxuaaa commentedon Sep 4, 2026
Implementation started from 46de7d4 on local branch codex/issue38-four-families. Two implementation owners are coordinated by one measurement owner using the shared lock, Docker/Linux, two CPUs/j2, no Docker data mounts, clone Workspace setup and fresh native output Stores. Nothing is pushed or merged.
The first current-source payload100/seed1 run failed with MissingObject after completing all writes; cleanup passed. A focused regression reproduces the root cause: spilled candidate reads assumed physical insertion order while reachable selection uses graph order. The local fix uses the existing offset index. A separate capture predicate discarded valid multi-piece sequential capture; the local fix accepts exact contiguous spool pieces and retains invalidation/fallback. Native scheduling now splits the CAS/CDC flat directory across bounded existing workers while preserving inode preorder, verified against the legacy canonical objects.
Supplementary developer checks:44 Store checks passed,1 preexisting large-spill check ignored; capture reuse and backward-write fallback checks passed. Docker/FUSE verification and final qualification remain INCOMPLETE. Baseline CAS unique100/500 seed1 initialization:0.237528541/1.116269208 seconds; this is one diagnostic pair, not median qualification. Failed payload evidence is retained under target/issue38-evidence/baseline/payload100-s1/perf.jsonl; other compact diagnostics are adjacent.
Fixture reconciliation: payload-create compact IDs retain empty genesis and identical N MiB prefix writes, so the creation curve is comparable. Workspace unique reuse needs explicit fixed128-file-base low selections: dedup-workspace-unique-{1,10}-base128-v3, preserving the128MiB base, original additions/seeds/per-file fsync and unchanged100/500 cases. Compact low smoke cases remain separately registered and these fixed-base selections are excluded from smoke by actual size. This corrects experiment comparability; no workload failure is removed. Final required coverage remains114 unique samples, with all scaling/runtime/verification gates unchanged.
No child acceptance or terminal PASS is claimed. Current progress record: docs/roadmap/0.1/0.1.3/issue38-progress.md.
yifanxuaaa commentedon Sep 5, 2026
Frozen Docker-owned qualification completed under the user's final strictly<1.25 normalized-median gate. Earlier <=1.00/<1.10 misses remain in the report and raw receipts. This is a25% scaling-overhead allowance, not proportional-scaling or statistical-confidence evidence.
Local commit
0dc2565f0d3f2ff6271fc743bf4a904b7267b58f(not pushed/merged). Measured source sealeb9b79cd87e6d655e26cad37b69327c70379e831b4d61fb4dd49838795d6fd25, product20aee20d96b09e0cd1d934b751a1e0370523987adfabc220aa63f283bb7a7d9e, imagesha256:f66ce7083572d8c81d43948f5308cc4b96080f548e5a796b57af8d3b40c2c988. Full distributions, historical-approximate comparisons, phase/resources, all30ratios and receipt links:/Users/yifanxu/Ephemeral-AI-Lab/layerfs/docs/roadmap/0.1/0.1.3/issue38-candidate3-docker-results.md; compact evidence undertarget/issue38-evidence/candidate3/.All samples retain actual work and independent setup, remain below15product seconds with zero swap/OOM, and report cleanupPASS. Runtime guards enforce Mac orchestration+LinuxDocker, no Docker data mounts/volumes/socket, and real FUSE/Exec where applicable. Workspace samples use independently writable verified copies of closed masters; native imports reuse source fixtures and initialize fresh outputs. No sample containers remain; preparedcache8entries/1,282,396,850data bytes.
Required checks:289 native tests passed,0failed,1preexistingignored in Docker;2-test-job/2CPU warmphase96s passes the unchanged120s gate. Formatting and Clippy(warnings denied)pass. The prior1-job177s scheduling failure remains retained. Four final representative verifiers total44.786109666s, maximum16.128966208s. Their no-reuse current-state mode reads all selected current file bytes through canonical reads and FUSE, verifies namespace/metadata/aliases and applicable independent CAS/CDC transcripts; exhaustive typed-object/storage census remains explicitly omitted. No#39 recovery, #40 namespace performance success or release is claimed.
All four children are closed for their completed frozen scope: #41(12samples),#42(30),#43(60),#44(12), totaling114unique samples and30adjacent median comparisons. Parent remains active for the subsequent user-authorized host-owned SQLite migration in this same task/worktree, with unchanged optimized product, safe reusable snapshots/preparation and measured same-seed100/500MiB topology comparison. No new task/agent/worktree, HTTP layer or system cache/memory tuning.
yifanxuaaa commentedon Sep 5, 2026
Terminal qualification: frozen Docker and host follow-up PASS
Frozen Docker:114 samples and all30 strict<1.25 normalized median comparisons PASS. Host:114 distinct samples and all30 comparisons PASS; maximum ratio1.034081602 (Workspace unique100→500), maximum sample3.875961457s. All prescribed work, environment/setup/resource and cleanup checks passed. Host coverage is12 payload+30 CAS+60 CDC+12 Workspace unique, with original seeds1–3.
Four final500-tier host verifiers PASS: payload7.224484500s, CAS mixed13.656885417s, CDC insert17.583759959s, Workspace unique13.998164958s. Total52.463294833s, each<59s, unchanged45s work allowance, aggregate<600s. Focused changed-route smoke proofs are recorded separately.
Host-owned SQLite follow-up is complete in the same task/worktree, with local commit
95508a3d72c5e3f6ae7796938c773f52d5def45eafter frozen product commit0dc2565f0d3f2ff6271fc743bf4a904b7267b58f. No push, merge or release was performed.The host owns SDK/coordinator, Workspace capture/spool and embedded SQLite. Linux containers retain the authenticated daemon, workload and real FUSE. Existing ContainerBinding/ProxyHost is reused, with no new transport/HTTP/data mounts. Product seal remains
20aee20d96b09e0cd1d934b751a1e0370523987adfabc220aa63f283bb7a7d9e. Qualified host sourcef1eb7c6f3dd1ebd31f639e888953be523c842ad21c8d77d45b1d7c37cef42181; Linux image source remainseb9b79cd87e6d655e26cad37b69327c70379e831b4d61fb4dd49838795d6fd25. Final delivered host source69cf6ec212bed950f89b25ec547e6713f33e4dadc1fc19c68e9f70c534b5fa5fhas only the verifier lint simplification and final harness identity/output guards after qualification; performance operations/product code are unchanged, and a final small performance/verifier check passed. Old receipts retain their actual identities.Host resources are separate: container2 CPUs/2 GiB/no swap; host CPU is not container-capped and native initialization uses up to8 existing workers. Host system SQLite3.51.0 and host page-cache ownership differ from the Docker reference. These are matched product/case/seed/timer/work comparisons with a changed topology/resource scope, not pure-efficiency claims. Workspace reuse500 is5.12% slower on host; the <1.25 scaling gate is not a topology-regression allowance.
Protected closed SQLite masters and native sources live under ignored project-local
benchmark-results/host-store/. Independent writable byte copies, fsync, quick_check, matching starting hashes/distinct inodes, unchanged master checks, actual miss→hit across executor changes and bounded ownership-checked eviction passed. Live SQLite sidecars are rejected. Native imports always create fresh output Stores. Final inventory: zero owned containers, empty host samples,8 host cache entries/509,078,386 bytes;8 Docker prepared entries retained. Cache is evictable, not a durable backup; ordinary caller-selected product Store paths are outside cleanup.Final checks:20 shared tests, formatting, strict benchmark Clippy, host release build and host v5 migration/staging/publication failure-injection check PASS. Existing live SDK host proof passed post-attach failure/disconnect cleanup; payload verification proves same-session reads/writes after two full-status Commits, returned/published heads, bytes/mode, fresh projection and clean End. Frozen product Linux checks remain289 PASS/one preexisting ignored,96s warm suite, strict Clippy PASS.
Full per-seed distributions, all normalized ratios, phase/Exec/Commit/transport/copy/admission/host-vs-container resource tables and evidence links are committed locally in
docs/roadmap/0.1/0.1.3/issue38-host-store-results.md; frozen Docker evidence remains separately recorded inissue38-candidate3-docker-results.md. Compact host receipts are inbenchmark-results/host-store/results/. The initialNotReadyverifier failure is preserved; it was a harness owner-lifetime problem corrected by retaining one daemon binding across Store close/reopen. The independent verifiers check all selected current bytes/namespace/metadata/aliases and applicable CAS/CDC transcripts, explicitly not an exhaustive typed-object/storage census.All same-product topology comparisons