Skip to content

[v0.1.3][Phase 2] Fix worse-than-linear scaling in cases below the runtime cutoff #38

Closed
4 of 4 issues completed

Description

@yifanxuaaa

Nine-family fast baseline — completed, 2026-09-05

The user's latest scope and simplification instructions supersede the expanded two-arm/581-performance/119-proof replay: one complete current-product performance sample per registered case, reusable preparation, and quick selected independent verification. No SDK full-byte option or dedicated full-byte test. Other families are deferred to #39.

  • Unified host SDK/Workspace/embedded SQLite plus Linux daemon/real FUSE/Exec through the existing authenticated binding. Identical observed container limits: 2 CPUs, 2 GiB, no swap, 256 PIDs, no Docker data mounts; host CPU remains separately scoped and uncapped.
  • All nine families passed: payload_create_read, dedup_workspace_reuse, dedup_cross_file, dedup_cdc_locality, edit_length_preserving, edit_length_changing, edit_canonical_chunk_count, init_namespace, store_footprint.
  • 118/118 single performance samples PASS; exact registry coverage, declared timers, source/image provenance and cleanup checked. Maximum product time 4.817 seconds, below 15 seconds. Single samples establish a baseline; no new median/scaling-distribution qualification is claimed.
  • 27/27 selected independent proofs PASS, 94.268 seconds combined proof wall, maximum 14.571 seconds. Separate preparation misses are preserved and excluded from that sum. Coverage includes all 14 SDK edit/outcome shapes at 1 MiB plus capped 500 MiB insert; all four namespace sizes; all three main storage controls; selected 500 MiB checks for each original family and the proof-only CDC boundaries.
  • SDK proof uses prepared canonical roots/counts, lengths, inode/payload retention and at most 192 KiB replacement/boundary bytes through canonical and FUSE reads. Capped 500 MiB proof fell from 16.277 to 4.370 seconds.
  • Namespace proof uses exact imported file/byte counts, persisted root equality and at most 11 deterministic FUSE file samples with metadata and prefix-byte checks. The 100,000-file proof passed in 5.615 seconds (reopen/sample check 219.860 ms), after the retained 45.050-second full-tree timeout.
  • Storage proof keeps storage accounting/publication/reopen and independent canonical/FUSE edit-boundary checks; full-tree FUSE digest replay removed. Main proofs: 5.566, 7.287 and 2.663 seconds.
  • Repeated native-input cache rehashes removed: owned caches validate prepared recipe/ownership; modifying a disposable native cache requires recreating it. SQLite master integrity and independent closed-copy checks remain. Receipts explicitly distinguish this policy and sampled coverage from exhaustive byte/namespace verification.
  • Host/Linux builds and strict benchmark Clippy PASS; 22 shared tests PASS before final cache simplification, affected 11 runner tests PASS afterward. No owned sample containers or host sample directories remain.

Implementation is local commit fa33ece0e (not pushed/merged/released). Report: docs/roadmap/0.1/0.1.3/nine-family-fast-baseline.md.

Performance and unchanged SDK proof source: 72f408b467840389be75dc14bd21495a07f6c39b6f2164d0ad6c7aea985e11d9. Final simplified namespace/storage/cache verifier source: aa907534bb9f760048739de4205a29c0c7630ad7de6a7b3c21b5e50e52b66cb3. Product seal is identical: 2a0f830a18ed0b9df7197c2dd0d7df921215b8c78cf60fe07c658715de5efee5. The completed performance baseline was preserved under its producing identity rather than replayed or relabeled after verifier/setup-only changes.

Raw evidence and assessments: benchmark-results/nine-family-fast-baseline/{72f408b467840389,aa907534bb9f7600}/. Prior four-family campaigns, the stopped oversized campaign and all failed receipts remain historical evidence. This completion is for the explicitly simplified fast baseline and sampled proof scope.


Current user-authorized scaling gate — final 2026-09-05 amendment

The user explicitly chose to accept the frozen performance with a strictly < 1.25 normalized adjacent-tier MEDIAN time-growth / actual-work-growth gate and proceed to the host-owned SQLite migration. This supersedes the earlier <1.10 gate and interrupted1.20 proposal. Describe this as a25% scaling-overhead allowance, not proportional scaling.

All other requirements remain unchanged: prescribed114unique samples/seeds1–3, explicit actualwork and every adjacenttier, per-sample15s/runtime/resource limits, no deliberate slowing of smaller tiers, exact applicable semantics and bounded separateverification, authentic Docker/FUSE/no-data-mount setup for the frozen campaign, independent Workspaceclones/nativefresh outputs, cleanup and evidence-bound completion. Preserve rawreceipts and the historical <=1.00/<1.10 misses, individualseed ratios and variability. All30 frozenCandidate3 mediancomparisons clear <1.25; the earlier payload1.105934087 and cross-file mixed1.157331012 misses now clear only the newly amended gate.

Finish remaining frozen assessment/requiredchecks and truthful report updates, then proceed promptly in this same task/worktree with host SDK/coordinator+Workspace/Store/localSQLite and Linux daemon/FUSE/workload via existing authenticated transport. No newHTTP layer, system cache/Docker memory tuning, or separateagent/task/worktree. Host reusable preparation needs protected consistentSQLite masters/backups, independent writable samples and real cache reuse; benchmark topology/source/resource identities and throughput comparisons must be explicit. ExistingDocker-owned results are not host-Store qualification.

The earlier amendment and historical requirements below remain intact as prior records; this paragraph controls the current threshold and sequence.

Current user-authorized scaling gate — 2026-09-05 amendment

For #38 and all four children #41–#44, each required adjacent-tier normalized median time-growth / actual-work-growth ratio must be strictly < 1.10. This permits bounded scaling overhead; it is not a claim of no worse than proportional growth. The user explicitly authorized this amendment in the side conversation and notified the integration owner.

All other gates remain unchanged: original three seeds, 114 unique final candidate samples plus applicable invalidated controls, explicit work dimensions and whole-curve comparisons, every sample within the 15-second product cutoff and resource bounds, no deliberate slowing of smaller tiers, exact applicable semantics and bounded separate verification, Mac + Docker/Linux with real FUSE where applicable and no Docker data mounts, independent Workspace clones, native fresh output Stores, cleanup and evidence-bound issue completion. Individual seed ratios and variability remain visible. This amendment does not automatically close any family.

Older <=1.00 / no-worse-than-proportional clauses below retain the previous strict-gate plan and historical assessments; only their numerical terminal threshold is superseded. Preserve the old raw results and strict-gate failures, and report amended outcomes separately. Candidate2's payload 100→500 median ratio 1.057185870 and cross-file unique ratio 1.053614071 clear this amended median gate; cross-file unique seed3 around1.134 remains an individual variability flag. Complete family qualification remains pending.

Comparison policy amendment — 2026-09-05

The user's latest instruction permits practical historical comparisons and overrides earlier blanket requirements to recollect a matched baseline. Do not rerun or resurrect the old harness merely to obtain a comparison table. Reuse retained baseline receipts/results when the workload and measurement meaning are sufficiently similar; small documented harness differences are acceptable without a new baseline run.

  • Classify comparisons as matched, historical-approximate, or not-comparable, and state the relevant difference in the table. Different source revisions are expected; preserve their provenance. Historical host/split-owner/APFS results may be used as approximate context where the selected product timer still measures comparable work, but must not be relabeled matched Docker/Linux evidence.
  • Judge similarity from actual operation, bytes/files/history depth, seed schedule, timing boundary, and resource scope. Do not introduce an arbitrary percentage tolerance or an extra certification campaign. If a difference could plausibly explain the claimed gain, qualify the conclusion rather than attributing it entirely to the product change.
  • Materially compacted fixtures (compact-v2/compact-v3 versus original large backgrounds), changed measured operations, and incompatible timing/resource scopes are not direct speedup comparisons. Show their absolute results side by side with the difference; leave speedup blank. Do not pool changed low tiers with old high tiers as an unchanged scaling curve.
  • Collect a new baseline with the current harness only for a specific unresolved decision that historical evidence cannot answer, using the smallest necessary selection. A full baseline campaign is not a prerequisite. Retain the prescribed candidate coverage, correctness checks, runtime limits and scaling criteria; this exception concerns baseline reuse, not permission to mark untested cases passing.
  • Derive tables on demand from compact logs; no benchmark rerun is needed to change presentation. See [v0.1.3] Optimize benchmark infrastructure: Docker/FUSE, reusable setup, three modes, compact logs #45 for field mapping. Preserve sample counts and seeds: N repetitions of one seed are not N different seeds. Verification comes from a separate identity-matched receipt, never inferred from performance PASS.

Infrastructure prerequisite and handoff

Execution starts after #45's benchmark-infrastructure migration is complete and its handoff is ready. The current infrastructure task is 01a06da4-c2ba-7013-92cb-392a348dfdfe (Implement LayerFS Phase 2.1). Do not start product optimization, builds or benchmark collection in parallel with that migration. This issue update defines the successor work; it does not launch a handoff agent.

Read #45 and docs/roadmap/0.1/0.1.3/benchmark-infrastructure-optimization-spec.md at the final handoff revision. Consume its actual migrated entrypoints, compact smoke results, slow-case inventory and unsupported/failure dispositions; do not assume a proposed path is already implemented. A small smoke PASS proves the migrated route works, not that larger tiers or this scaling target pass. Reuse valid smoke/proof evidence; do not rerun it merely to begin a new task. Slow or blocked product cases remain explicit diagnosis input, not a reason to repeat known-slow large runs or label them passing.

#40 is the product-mechanism prerequisite. Use its recorded Approach B implementation/handoff, with the namespace <=2.2-second and 10% CPU-reduction targets still explicitly missed. Do not silently turn that MISS into PASS or restart namespace tuning as a gate for this work. Audit actual adoption of its six mechanisms; the completion summary alone does not establish full caller transfer. The user's sequencing is infrastructure completion, then the #38 optimization handoff.

Two implementation tracks after #45

Track Child issues Shared ownership and initial work Final unique candidate samples
Native initialization #42 cross-file CAS; #43 CDC locality One owner for native discovery/task distribution/initial-tree construction and its admission adapter. Begin with #42 as the first controlled exercise, then reuse useful changes in #43. 30 + 60 = 90
Workspace creation and Commit #41 payload creation; #44 Workspace unique-content reuse One owner for shared capture/candidate construction/admission/continuation. Preserve one-large-file vs per-file-fsync differences. 12 + 12 = 24

The full parent scope remains four families, ten affected curves and eleven historical adjacent-tier violations. Final coverage is 114 unique candidate samples, before invalidated regression controls. CAS has one shared tier-1 anchor, reused across its three curves. Final counts are not iteration-loop requirements or forecasts of easy fixes.

The two tracks may investigate and implement independently, but do not assign four competing owners to the same files. Common content/Store primitive changes are coordinated, integrated once and checked in both applicable tracks. A shared fix may satisfy a sibling without another patch; every original curve still needs its own qualification. #39's live bulk/delete/read/Git/history recovery is outside this execution scope, although reusable fixes may later help it.

Mechanism adoption before new implementation

Audit at 810bb3a589ac58d103483df34bb58ecfe0f0ddf4 compared with pre-#40 4c9b14a6 found:

Mechanism Native direct import Ordinary modified Workspace Commit
Exact eight-entry metadata cache Used, with shared helper extracted in #40 Adopted in frontier/localized builders
Checked insertion Shared helper on direct route; preserve fallback distinctions Adopted; former separate membership-planning pass removed
Carried admission Owned direct batches Adopted after construction; object limit 127 -> 8191, byte cap remains <4 MiB
Owned producer slabs / overlapping admission Existing direct pipeline Not transferred: completed candidate, possible spill, later borrowed-byte copy/admission
New sorted affected-page updater Not used by old initial-directory/inode builders Adopted with bounded batches and fallbacks
Content builders / borrowed extents Existing canonical builders Existing builders/incremental paths; largely predate #40
Producer execution / ownership Existing bounded scoped workers, fixture-shape limits Local ownership/staging adopted; no shared multi-file producer pool

Reinspect the final handoff source before treating these findings as current. Do not reimplement cache/checked batching already integrated. The initializer's private direct admission requires an empty Store and cannot simply replace Workspace admission. Use existing owned-object and sorted-tree primitives through the smallest justified adapter; no universal engine or public bulk interface.

For each hypothesis: record predicted work-counter change; measure one representative case/seed or size pair; change one shared mechanism; perform the smallest relevant correctness check and separate bounded verification; compare complete operation wall, total CPU, resources and work counters. Retain misses, revise the hypothesis or reject the change. Neither more workers nor code reuse alone proves scaling improvement. Only expand to the final curve after the route stabilizes.

File layout, shared code and refactoring boundary

Keep the existing product crate ownership. This is a map of relevant existing files, not an instruction to modify them all:

crates/
  layerfs-content/src/
    filesystem/change.rs       # shared exact metadata cache
    filesystem/apply.rs        # current initial-directory builder
    file/rope/build.rs          # existing content/CDC construction
    tree/batch.rs               # sorted directory/inode updates
    tree/inode/table.rs         # current initial inode-table builder
  layerfs-layerstack-store/src/
    objects.rs                 # deferred candidates, owned slabs, checked batches
    layerstack.rs              # native discovery and task scheduling
    workspace.rs               # Workspace candidate admission/publication
    staging.rs                 # completed-root stage lifecycle
  layerfs-workspace/src/
    changes.rs                 # candidate selection/construction
    capture.rs                 # existing single-file capture
    file_io.rs                 # mutable file I/O
    lifecycle.rs               # Commit and continuing-view installation
  layerfs-fuse/src/
    proxy_client.rs / proxy_host.rs  # separate live-operation transport
  layerfs-sdk/src/client.rs     # existing public entrypoints

Shared product code means PortableMetadataCache::get_or_build, existing rope/incremental builders, sorted tree helpers, and insert_checked_object_batch under their current owners. Workspace stage/publication is operation-specific. Native and Workspace object delivery are not yet identical. No new crate, universal engine, public bulk endpoint or generic construction-pool facade is prescribed.

The smallest potential Workspace ownership change is confined to Store objects.rs and workspace.rs: adapt existing test-only DeferredObjectStore::consume_prevalidated_pages to feed owned selected objects into admit_checked_objects, retaining checked insertion and carried bounds. It removes memory-candidate copies only; the spill path still copies and no construction/admission overlap is created. Use it only if copy counters justify the change.

A larger direct-delivery transfer may adapt InitializationObjectSlab, InitializationSlabWriter and the admission accumulator with a narrow nonempty-Store boundary. InitializationSegmentAdmission::new currently requires an empty Store; do not call it unchanged for Workspace. Preserve candidate read-your-writes, reachable-root validation, old roots, collision checks, bounded resources and staged failure semantics. Reuse existing capture instead of rebuilding already captured content; no shared multi-file worker pool currently exists to switch on.

Native work stays in direct_initialize_root_directories_inner and initial-tree callers: test task distribution separately from transfer of build_initial_directory / build_initial_inode_table_from_pairs to the existing sorted helpers. Preserve exact same-seed canonical output and needed fallbacks. Workspace sorted updates are already adopted; investigate repeated bounded batch work rather than reimplementing them.

One native owner handles #42/#43; one Workspace owner handles #41/#44. A single integration owner coordinates shared Store/content edits and serializes resource-sensitive work. Product algorithms must never move into benchmark family modules as shortcuts.

#45's separate benchmark target is shared/{runner.py,runtime.sh,preparation.py}, the sole verify-selected.py, and families/<canonical-id>/{mod.rs,setup.sh,perf.sh,verify.sh}. Family launchers bind IDs and forward arguments; setup, timing, Docker lifecycle, verification and cleanup stay shared. Consume the actual completed layout, not a second harness implementation.

Admission complexity and optimization claims

For current ordinary Workspace admission, let N be selected candidate objects, B their canonical bytes, D bytes of existing-object conflicts, and M preexisting Store objects. Under a logarithmic indexed-operation model:

T_admit = T_candidate_visit + O(B + D + N log(M + N + 1)) + transaction costs + physical I/O.

Admission has bounded additional buffers: <4 MiB canonical batch payload plus up to 8,191 object records, visitor buffers and bookkeeping; SQLite cache is additional. This is not a <4 MiB whole-process bound. Deferred candidates, spill/index/reachability structures, Workspace state, continuation and durable storage are separate costs. #40's 127 -> 8191 object-cap increase reduces applicable transaction overhead, not per-object indexed complexity. Candidate reachability has a documented quadratic ID-spill fallback at very large sets; verify activation before targeting it, rather than labeling the admission loop quadratic.

Every chosen refactor must name its expected effect: remove an extra copy, remove candidate write/readback, reduce repeated tree work, reduce transaction count, or improve overlap. Owned-page movement usually changes constants, not big-O. More workers can reduce wall while increasing CPU. Report relevant counters, total CPU, memory and temporary/durable I/O separately; no asymptotic or zero-copy claim from one faster sample.

Planning documents in docs/roadmap/0.1/0.1.3/: phase-2-shared-code-layout.md, phase-2.1-mechanism-adoption-audit.md, and workspace-admission-complexity.md. Current local copies are in /Users/yifanxu/Ephemeral-AI-Lab/layerfs/docs/roadmap/0.1/0.1.3/; these edits are local and not claimed committed/pushed. The key instructions are included here so this issue is self-contained. Audit source was 810bb3a58; recheck the final #45 handoff source and paths before editing.

API reduction counts and algorithm consolidation

The reviewed target is fewer independent implementations, not an immediate large reduction of public SDK methods. Inventory at product source 810bb3a58: 26 production Client methods plus two feature-gated verification helpers. Reinventory the final #45 handoff source before editing.

Surface Initial change Later candidate
Public Rust SDK 0 methods removed; 26 remain, with compatible forwarding wrappers and consistent validation/observation One versioned Commit-variant reduction, 26 -> 25 only if no other API changes occur; migrate callers and preserve full publication/presentation status first
FUSE wire protocol Preserve acknowledged readonly Pin and current compatibility 1 identified obsolete request: PinRead, with no current production sender; pruning requires an explicit compatibility disposition and is not a guaranteed net request-count change
Internal Rust methods No validated net deletion count Count the reviewed implementation diff; a shared helper may remove duplicated bodies without reducing total methods
Algorithm area Current implementations Proposed target
Native scheduling/orchestration 3 1 scheduling implementation, with explicit appropriate sinks
Admission accumulation/orchestration 3 1 shared bounded accumulator; operation-specific publication stays explicit
Native regular-file construction bodies 2 1
Workspace inode-record construction bodies 2 1
Workspace candidate planners 3 Eventually 1, after all fallback domains and sparse/resource behavior transfer

These are not additive method-removal counts, performance promises or mandatory deletion quotas. Compatibility wrappers may remain. Do not create a catch-all endpoint or policy-flag framework merely to reduce enum/method names. Do not remove a route until its real callers and supported states transfer; test-only differential oracles are not extra production routes.

For this parent, native consolidation belongs to #42/#43 and Workspace consolidation to #41/#44. One integration owner coordinates shared content/Store edits. Retain typed SDK operations and useful adapters. Canonical full Commit status, unified edit observation and rejection of ignored Query filters are small consistency fixes, not evidence that another Commit/edit algorithm must be built. Stage/head/ownership behavior must remain explicit.

Implementation proceeds after the completed #45 handoff. Apply only evidence-backed simplifications relevant to the active track; broader public API or wire pruning is a separately validated compatibility change, not a prerequisite that expands every family task. Preserve the 114 unique final candidate samples and original scaling/correctness/resource criteria. Metadata/copy/transaction/tree counters must substantiate any speed or complexity claim.

Detailed local plans: docs/roadmap/0.1/0.1.3/api-algorithm-simplification-audit.md, phase-2-shared-code-layout.md, and workspace-admission-complexity.md in /Users/yifanxu/Ephemeral-AI-Lab/layerfs. These documents are local planning edits, not claimed pushed commits. The counts and scope are included here so the issue is self-contained.

Successor handoff checklist

Migrated benchmark contract

  • All fixture/reference preparation, sample copying, SDK/coordinator, Store, daemon and product work run inside Docker/Linux. Host orchestration and compact result collection only. No runtime data bind mounts, named/anonymous volumes, Docker socket, APFS clone or host-side measured SDK/Store. Real FUSE/Exec remains where the family operation requires it; pure imports do not gain a synthetic timed Workspace lifecycle.
  • Use the migrated families/<canonical-family-id>/{mod.rs,setup.sh,perf.sh,verify.sh} interfaces backed by shared infrastructure. setup.sh is optional prewarming; perf acquires compatible preparation automatically. Historical wrappers are forwarding/archival compatibility only as recorded by [v0.1.3] Optimize benchmark infrastructure: Docker/FUSE, reusable setup, three modes, compact logs #45; do not build a parallel runner.
  • Default --perf-fast means one complete selected workload sample, not smaller data or weaker checks. --perf-samples N repeats the same selected seed/input sequentially; sample ordinal is not seed, repetition or history depth. --verification is an explicit separate selection through the single shared verifier. Do not use N to invent new seeds or replace the final prescribed seed schedule.
  • Import measurements reuse prepared source files but always initialize a fresh/absent output Store (fresh-output); reject --setup clone. Post-initialization Workspace cases may use authenticated --setup clone or fresh, with independent writable state and SQLite copy-up completed outside product timing. Record actual strategy and cache policy, not an unsupported cold-cache/reflink claim.
  • Reuse retained historical baselines under the comparison policy amendment above; no old-harness rerun or blanket baseline recollection is required. When a new baseline is necessary for a specific unresolved decision, run only the selected cases with the current harness and align platform, CPU, setup/cache/input and timing boundaries with the candidate.
  • Retain perf.jsonl (header, attempted samples, summary) or verification.json, plus an optional bounded failure.log (default <=1 MiB). No automatic Markdown/HTML/CSV reports, per-sample evidence trees, retained Stores/fixtures/containers, or raw sampler streams. Summaries required for issue closure are derived from compact logs on demand. Prepared cache remains finite and benchmark-owned; reuse [v0.1.3] Optimize benchmark infrastructure: Docker/FUSE, reusable setup, three modes, compact logs #45's cleanup and retention rules.
  • Verification has a hard 59-second end-to-end deadline starting before setup/authentication and ending after cleanup/receipt publication. Reuse existing independent oracles and identity-matched proofs, explicitly report omissions, and return PASS/FAIL/TIMEOUT/INCOMPLETE. Unsupported long proofs are not shortened or relabeled passing. No unchanged automatic retries or withdrawn exhaustive Phase 1 replay.
  • Investigation and independent source edits may run concurrently after handoff. Serialize resource-sensitive builds, preparation, verification and performance across both tracks, not merely within each output directory. Use one coordinator for measurement scheduling and shared-file integration; obey aggregate <=8 participating CPUs and stricter applicable family limits.

Problem and outcome

Several cases completed below Phase 1's 15-second product-time cutoff but became slower per unit of work at larger tiers. Passing that cutoff did not establish good scaling. Fix the shared causes so the affected curves grow no worse than proportionally with actual workload, while preserving all operations, correctness, resource bounds, and existing smaller-tier performance.

This issue owns previously active cases with observed scaling violations. Companion issue #39 owns the 15 runtime-suppressed cases. A shared implementation may address both; retain separate case-level evidence and closure criteria.

Measured inventory

These are observations from the retained Phase 1 selection, not results on an optimized candidate. Median ratios use unrounded product-time samples. Over linear = (larger median / smaller median) / (larger work / smaller work) - 1. Seeds over counts corresponding seed pairs exceeding proportional growth; it is descriptive, not a statistical confidence claim.

Exact curve prefix Tier transition Actual work growth Median seconds Time growth Over linear Seeds over
payload-create (MiB suffix m) 100→500 5.000× 0.475692→3.744340 7.871× 57.4% 3/3
dedup-cross-file-identical 100→500 5.000× 0.156959→0.808296 5.150× 3.0% 2/3
dedup-cross-file-mixed 100→500 5.000× 0.203634→1.056044 5.186× 3.7% 3/3
dedup-cross-file-unique 100→500 5.000× 0.176675→0.941368 5.328× 6.6% 3/3
dedup-cdc-common-body 10→100 9.182× 0.021993→0.268495 12.208× 33.0% 3/3
dedup-cdc-delete 100→500 4.960× 0.169896→0.910435 5.359× 8.0% 3/3
dedup-cdc-insert 100→500 4.961× 0.171250→0.915639 5.347× 7.8% 3/3
dedup-cdc-overwrite 100→500 4.960× 0.170187→0.895133 5.260× 6.0% 3/3
dedup-cdc-scattered 1→10 5.500× 0.005955→0.056221 9.441× 71.7% 3/3
dedup-cdc-scattered 100→500 4.960× 0.289743→1.681869 5.805× 17.0% 2/3
dedup-workspace-unique 100→500 5.000× 0.726023→3.847812 5.300× 6.0% 3/3

Four affected families, ten affected curves, eleven adjacent-tier violations. Include the whole existing four-tier curve in final scaling evaluation, not only the listed pair.

Additional variability finding: dedup-cdc-common-body 100→500 clears the median comparison but exceeds proportional growth by 8.7% using the mean, with two of three seed pairs over. Keep this visible alongside the primary inventory.

CDC imports one fixed 1 MiB reference plus N variants. Use actual logical input bytes, including the reference and insertion/deletion length differences: 1→10 variants is approximately 5.5× bytes, not 10×. Cross-file CAS uses N prepared 1 MiB files; Workspace reuse uses N new 1 MiB files on the unchanged base. CAS/CDC timings are initialization from prepared files, not SDK-edit timers. Do not compare unlike operations or infer asymptotic complexity from three samples.

Evidence and starting investigation

  • Final Phase 1 results.
  • Qualified selection: benchmark-results/fs-bench-pro/phase1-v013/results/review.json, overlaid by exact case/seed from qualification/readonly-pin-performance-checkpoint/incremental-performance-rows.json, qualification/rename-cache-performance-checkpoint/incremental-performance-rows.json, and qualification/cdc-delete-collision-final-checkpoint/incremental-performance-rows.json; exclude the persistent suppression inventory.
  • Affected baseline sources are 7948df2d, plus corrected dedup-cdc-delete-500 seed 3 at e24a3b34. Preserve producing source, environment, input, and seed identities; compare against a controlled candidate and report source differences explicitly.
  • Payload creation 100→500: Exec medians 0.186139→1.428129 seconds (7.67×); Commit 0.275633→2.291853 (8.31×). Investigate both paths using existing phase/CPU/I/O/admission receipts. The baseline-derived 500 MiB proportional bound is 2.378459 seconds; observed median is 3.744340 seconds.
  • Three seed growth factors for payload creation are 5.36×, 7.92×, and 14.05× for 5× input; this finding is not caused solely by one slow observation.

Acceptance

  • Diagnose shared product causes from existing evidence before selecting implementation changes; reuse ongoing relevant optimization work rather than creating competing replacements.
  • For each affected curve, record actual work denominators and all adjacent-tier ratios using the original three seeds and prescribed operations. Require normalized median time growth / actual work growth strictly <1.25 under the final user amendment; retain earlier gate assessments. Report individual samples, mean, min/max, and corresponding-seed ratios; unresolved inconsistent samples are explicitly inconclusive, not a blanket family pass. Do not introduce an unapproved tolerance or discard inconvenient samples.
  • Do not obtain a passing ratio by slowing smaller cases. Preserve their baseline performance and show before/after absolute timings as well as normalized growth. Keep every candidate sample within the 15-second product-time cutoff.
  • Preserve byte/namespace/Commit/CAS/CDC semantics, required failure propagation, resource bounds and cleanup. Per-file maximum remains 500 MiB and aggregate logical files remain under 1 GiB at every workload state.
  • Publish one source-bound before/after table covering the affected curves and the specific related regression controls invalidated by the changes. No claims of sublinear asymptotic complexity or storage-savings qualification based only on latency.

Fast iteration

Follow the two-track and migrated-infrastructure contracts above. Start with a small explicit diagnostic selection; use 1/10-tier smoke evidence from #45 where identity-compatible, then a larger pair only when needed to explain scaling. Preserve actual denominators, original case definitions and public operations. Recollect only invalidated evidence and perform the scoped final comparison once. Required product work stays inside its timer; setup, independent verification and outer cleanup remain separately observable. This update schedules work after the infrastructure handoff; it does not launch execution now, authorize release publication or weaken correctness/scaling criteria.

Host-owned SQLite follow-up — completed in the same task/worktree

  • Reuse existing authenticated ContainerBinding/ProxyHost with host SDK/coordinator, Workspace/capture/spool and embedded SQLite; container daemon/FUSE/workload; no new HTTP or data mounts.
  • Implement explicit host topology and separate host binary/source versus Linux image identities, without relabeling frozen Docker evidence or changing system cache/memory settings.
  • Use ignored project-local benchmark-results/host-store for compatible protected preparation, independent samples and compact results; preserve ordinary caller-selected Store paths and distinguish evictable cache from durable backups.
  • Prove safe consistent SQLite snapshots, miss→hit preparation reuse across irrelevant execution changes, identical starting state, independent writable sample, unchanged master, bounded cleanup/eviction and native fresh-output initialization.
  • Prove host publication/container-visible bytes and metadata, continuation/failure handling, then compare same optimized product/cases/seeds at100/500MiB with Exec/Commit/transport and distinct host/container resources. Similar throughput is a goal; <1.25 is not a topology-regression allowance.

Activity

  1. yifanxuaaa commented on Sep 4, 2026

    @yifanxuaaa
    ContributorAuthor

    Implementation started from 46de7d4 on local branch codex/issue38-four-families. Two implementation owners are coordinated by one measurement owner using the shared lock, Docker/Linux, two CPUs/j2, no Docker data mounts, clone Workspace setup and fresh native output Stores. Nothing is pushed or merged.

    The first current-source payload100/seed1 run failed with MissingObject after completing all writes; cleanup passed. A focused regression reproduces the root cause: spilled candidate reads assumed physical insertion order while reachable selection uses graph order. The local fix uses the existing offset index. A separate capture predicate discarded valid multi-piece sequential capture; the local fix accepts exact contiguous spool pieces and retains invalidation/fallback. Native scheduling now splits the CAS/CDC flat directory across bounded existing workers while preserving inode preorder, verified against the legacy canonical objects.

    Supplementary developer checks:44 Store checks passed,1 preexisting large-spill check ignored; capture reuse and backward-write fallback checks passed. Docker/FUSE verification and final qualification remain INCOMPLETE. Baseline CAS unique100/500 seed1 initialization:0.237528541/1.116269208 seconds; this is one diagnostic pair, not median qualification. Failed payload evidence is retained under target/issue38-evidence/baseline/payload100-s1/perf.jsonl; other compact diagnostics are adjacent.

    Fixture reconciliation: payload-create compact IDs retain empty genesis and identical N MiB prefix writes, so the creation curve is comparable. Workspace unique reuse needs explicit fixed128-file-base low selections: dedup-workspace-unique-{1,10}-base128-v3, preserving the128MiB base, original additions/seeds/per-file fsync and unchanged100/500 cases. Compact low smoke cases remain separately registered and these fixed-base selections are excluded from smoke by actual size. This corrects experiment comparability; no workload failure is removed. Final required coverage remains114 unique samples, with all scaling/runtime/verification gates unchanged.

    No child acceptance or terminal PASS is claimed. Current progress record: docs/roadmap/0.1/0.1.3/issue38-progress.md.

  2. yifanxuaaa commented on Sep 5, 2026

    @yifanxuaaa
    ContributorAuthor

    Frozen Docker-owned qualification completed under the user's final strictly<1.25 normalized-median gate. Earlier <=1.00/<1.10 misses remain in the report and raw receipts. This is a25% scaling-overhead allowance, not proportional-scaling or statistical-confidence evidence.

    Local commit 0dc2565f0d3f2ff6271fc743bf4a904b7267b58f (not pushed/merged). Measured source seal eb9b79cd87e6d655e26cad37b69327c70379e831b4d61fb4dd49838795d6fd25, product 20aee20d96b09e0cd1d934b751a1e0370523987adfabc220aa63f283bb7a7d9e, image sha256:f66ce7083572d8c81d43948f5308cc4b96080f548e5a796b57af8d3b40c2c988. Full distributions, historical-approximate comparisons, phase/resources, all30ratios and receipt links: /Users/yifanxu/Ephemeral-AI-Lab/layerfs/docs/roadmap/0.1/0.1.3/issue38-candidate3-docker-results.md; compact evidence under target/issue38-evidence/candidate3/.

    All samples retain actual work and independent setup, remain below15product seconds with zero swap/OOM, and report cleanupPASS. Runtime guards enforce Mac orchestration+LinuxDocker, no Docker data mounts/volumes/socket, and real FUSE/Exec where applicable. Workspace samples use independently writable verified copies of closed masters; native imports reuse source fixtures and initialize fresh outputs. No sample containers remain; preparedcache8entries/1,282,396,850data bytes.

    Required checks:289 native tests passed,0failed,1preexistingignored in Docker;2-test-job/2CPU warmphase96s passes the unchanged120s gate. Formatting and Clippy(warnings denied)pass. The prior1-job177s scheduling failure remains retained. Four final representative verifiers total44.786109666s, maximum16.128966208s. Their no-reuse current-state mode reads all selected current file bytes through canonical reads and FUSE, verifies namespace/metadata/aliases and applicable independent CAS/CDC transcripts; exhaustive typed-object/storage census remains explicitly omitted. No#39 recovery, #40 namespace performance success or release is claimed.

    All four children are closed for their completed frozen scope: #41(12samples),#42(30),#43(60),#44(12), totaling114unique samples and30adjacent median comparisons. Parent remains active for the subsequent user-authorized host-owned SQLite migration in this same task/worktree, with unchanged optimized product, safe reusable snapshots/preparation and measured same-seed100/500MiB topology comparison. No new task/agent/worktree, HTTP layer or system cache/memory tuning.

  3. yifanxuaaa commented on Sep 5, 2026

    @yifanxuaaa
    ContributorAuthor

    Terminal qualification: frozen Docker and host follow-up PASS

    Frozen Docker:114 samples and all30 strict<1.25 normalized median comparisons PASS. Host:114 distinct samples and all30 comparisons PASS; maximum ratio1.034081602 (Workspace unique100→500), maximum sample3.875961457s. All prescribed work, environment/setup/resource and cleanup checks passed. Host coverage is12 payload+30 CAS+60 CDC+12 Workspace unique, with original seeds1–3.

    Four final500-tier host verifiers PASS: payload7.224484500s, CAS mixed13.656885417s, CDC insert17.583759959s, Workspace unique13.998164958s. Total52.463294833s, each<59s, unchanged45s work allowance, aggregate<600s. Focused changed-route smoke proofs are recorded separately.

    Host-owned SQLite follow-up is complete in the same task/worktree, with local commit 95508a3d72c5e3f6ae7796938c773f52d5def45e after frozen product commit 0dc2565f0d3f2ff6271fc743bf4a904b7267b58f. No push, merge or release was performed.

    The host owns SDK/coordinator, Workspace capture/spool and embedded SQLite. Linux containers retain the authenticated daemon, workload and real FUSE. Existing ContainerBinding/ProxyHost is reused, with no new transport/HTTP/data mounts. Product seal remains 20aee20d96b09e0cd1d934b751a1e0370523987adfabc220aa63f283bb7a7d9e. Qualified host source f1eb7c6f3dd1ebd31f639e888953be523c842ad21c8d77d45b1d7c37cef42181; Linux image source remains eb9b79cd87e6d655e26cad37b69327c70379e831b4d61fb4dd49838795d6fd25. Final delivered host source 69cf6ec212bed950f89b25ec547e6713f33e4dadc1fc19c68e9f70c534b5fa5f has only the verifier lint simplification and final harness identity/output guards after qualification; performance operations/product code are unchanged, and a final small performance/verifier check passed. Old receipts retain their actual identities.

    Host resources are separate: container2 CPUs/2 GiB/no swap; host CPU is not container-capped and native initialization uses up to8 existing workers. Host system SQLite3.51.0 and host page-cache ownership differ from the Docker reference. These are matched product/case/seed/timer/work comparisons with a changed topology/resource scope, not pure-efficiency claims. Workspace reuse500 is5.12% slower on host; the <1.25 scaling gate is not a topology-regression allowance.

    Protected closed SQLite masters and native sources live under ignored project-local benchmark-results/host-store/. Independent writable byte copies, fsync, quick_check, matching starting hashes/distinct inodes, unchanged master checks, actual miss→hit across executor changes and bounded ownership-checked eviction passed. Live SQLite sidecars are rejected. Native imports always create fresh output Stores. Final inventory: zero owned containers, empty host samples,8 host cache entries/509,078,386 bytes;8 Docker prepared entries retained. Cache is evictable, not a durable backup; ordinary caller-selected product Store paths are outside cleanup.

    Final checks:20 shared tests, formatting, strict benchmark Clippy, host release build and host v5 migration/staging/publication failure-injection check PASS. Existing live SDK host proof passed post-attach failure/disconnect cleanup; payload verification proves same-session reads/writes after two full-status Commits, returned/published heads, bytes/mode, fresh projection and clean End. Frozen product Linux checks remain289 PASS/one preexisting ignored,96s warm suite, strict Clippy PASS.

    Full per-seed distributions, all normalized ratios, phase/Exec/Commit/transport/copy/admission/host-vs-container resource tables and evidence links are committed locally in docs/roadmap/0.1/0.1.3/issue38-host-store-results.md; frozen Docker evidence remains separately recorded in issue38-candidate3-docker-results.md. Compact host receipts are in benchmark-results/host-store/results/. The initial NotReady verifier failure is preserved; it was a harness owner-lifetime problem corrected by retaining one daemon binding across Store close/reopen. The independent verifiers check all selected current bytes/namespace/metadata/aliases and applicable CAS/CDC transcripts, explicitly not an exhaustive typed-object/storage census.

    All same-product topology comparisons

    Case Actual work bytes Host seeds1 / 2 / 3 s Docker median s Host median s Host/Docker
    payload-create-1m-compact-v2 1048576 0.025027875 / 0.024693665 / 0.025669625 0.015796376 0.025027875 1.584406
    payload-create-10m-compact-v2 10485760 0.062607749 / 0.057788541 / 0.062064000 0.054880708 0.062064000 1.130889
    payload-create-100m 104857600 0.529123250 / 0.408488916 / 0.396174500 0.428373834 0.408488916 0.953580
    payload-create-500m 524288000 2.402669499 / 1.881036958 / 1.899561791 2.368766125 1.899561791 0.801920
    dedup-cross-file-anchor-1 1048576 0.004671792 / 0.003829709 / 0.004263250 0.004265958 0.004263250 0.999365
    dedup-cross-file-unique-10 10485760 0.022828708 / 0.022332958 / 0.022747708 0.026099250 0.022747708 0.871585
    dedup-cross-file-unique-100 104857600 0.098148750 / 0.093801500 / 0.095276917 0.161730500 0.095276917 0.589109
    dedup-cross-file-unique-500 524288000 0.452314208 / 0.488961083 / 0.446352417 0.778405167 0.452314208 0.581078
    dedup-cross-file-identical-10 10485760 0.018363958 / 0.018145083 / 0.019082000 0.020164375 0.018363958 0.910713
    dedup-cross-file-identical-100 104857600 0.030205709 / 0.030316708 / 0.029878000 0.092081209 0.030205709 0.328033
    dedup-cross-file-identical-500 524288000 0.107891500 / 0.107734417 / 0.108330917 0.479259916 0.107891500 0.225121
    dedup-cross-file-mixed-10 10485760 0.023141333 / 0.024174542 / 0.023856416 0.026917083 0.023856416 0.886293
    dedup-cross-file-mixed-100 104857600 0.116676333 / 0.112735500 / 0.115391125 0.154316417 0.115391125 0.747757
    dedup-cross-file-mixed-500 524288000 0.536459750 / 0.553384000 / 0.544910292 0.892975875 0.544910292 0.610218
    dedup-cdc-overwrite-1 2097152 0.004060917 / 0.004324042 / 0.004098791 0.003939375 0.004098791 1.040467
    dedup-cdc-overwrite-10 11534336 0.018678250 / 0.018764333 / 0.018548208 0.020071250 0.018678250 0.930597
    dedup-cdc-overwrite-100 105906176 0.033183459 / 0.032378208 / 0.032318541 0.096835166 0.032378208 0.334364
    dedup-cdc-overwrite-500 525336576 0.123970000 / 0.122797792 / 0.123218500 0.488210250 0.123218500 0.252388
    dedup-cdc-insert-1 2101248 0.004123625 / 0.004260458 / 0.004345875 0.004345250 0.004260458 0.980486
    dedup-cdc-insert-10 11575296 0.018393458 / 0.019338625 / 0.019226250 0.021310917 0.019226250 0.902178
    dedup-cdc-insert-100 106315776 0.035708625 / 0.034826500 / 0.034841416 0.099640042 0.034841416 0.349673
    dedup-cdc-insert-500 527384576 0.153700750 / 0.134772125 / 0.134277208 0.512471958 0.134772125 0.262984
    dedup-cdc-delete-1 2093056 0.004046750 / 0.004079375 / 0.004003417 0.004234583 0.004046750 0.955643
    dedup-cdc-delete-10 11493376 0.019286208 / 0.019020583 / 0.018947791 0.020706333 0.019020583 0.918588
    dedup-cdc-delete-100 105496576 0.034486125 / 0.033338416 / 0.032912917 0.095872334 0.033338416 0.347738
    dedup-cdc-delete-500 523288576 0.132063958 / 0.126776875 / 0.131976542 0.485700708 0.131976542 0.271724
    dedup-cdc-common-body-1 2097152 0.004420958 / 0.004261917 / 0.004629417 0.004362959 0.004420958 1.013294
    dedup-cdc-common-body-10 11534336 0.021174500 / 0.021269666 / 0.021296333 0.023337375 0.021269666 0.911399
    dedup-cdc-common-body-100 105906176 0.060739041 / 0.058585041 / 0.060719208 0.123581792 0.060719208 0.491328
    dedup-cdc-common-body-500 525336576 0.247651584 / 0.243209792 / 0.234852958 0.595288625 0.243209792 0.408558
    dedup-cdc-scattered-1 2097152 0.004809583 / 0.005062542 / 0.004848750 0.006169500 0.004848750 0.785923
    dedup-cdc-scattered-10 11534336 0.024949292 / 0.023127958 / 0.025459167 0.026922958 0.024949292 0.926692
    dedup-cdc-scattered-100 105906176 0.100217250 / 0.099687208 / 0.097014042 0.152760500 0.099687208 0.652572
    dedup-cdc-scattered-500 525336576 0.437729708 / 0.457377208 / 0.434046083 0.829035084 0.437729708 0.527999
    dedup-workspace-unique-100 104857600 0.748318958 / 0.741998125 / 0.761773416 0.700712791 0.748318958 1.067940
    dedup-workspace-unique-500 524288000 3.875961457 / 3.776175416 / 3.869114334 3.680507918 3.869114334 1.051245
    dedup-workspace-unique-1-base128-v3 1048576 0.028680375 / 0.028497334 / 0.027837291 0.016188501 0.028497334 1.760344
    dedup-workspace-unique-10-base128-v3 10485760 0.100320542 / 0.098560499 / 0.103581042 0.083100749 0.100320542 1.207216
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    v0.1.3LayerFS v0.1.3 checkpoint and delivery

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions