Skip to content

[v0.1.3][Stage 1][03] Build and baseline directory construction and whole-workspace reads #25

Closed

Description

@yifanxuaaa

Latest Phase 1 scope amendment (2026-09-04)

The user authorized permanent Phase 1 scheduling suppressions under the committed 15-second policy. This supersedes the original full-inventory execution wording below. Keep every case implementation and historical outcome for Phase 2 testing/optimization; suppression is not deletion or a passing result.

The original 130 new timed definitions remain intact. The initial 14 suppressed scenario IDs remove 42 performance samples and their 42 associated verifiers from required Phase 1 scheduling, leaving 116 active new cases / 348 performance samples, plus 25 capped inherited samples. All 29 standalone proof recipes remain required. Any additional single performance sample exceeding 15 seconds of cumulative declared product time permanently suppresses that exact scenario across all seeds/repetitions and future Phase 1 sources. Preparation, standalone proofs and external cleanup are exempt. Still-active correctness, resource, custody and cleanup gates remain unchanged; #21 stays open.

Phase 1 completion amendment (2026-09-04)

The user requires functional failures to be repaired in Phase 1. The committed amendment supersedes earlier baseline-only closure and deferral wording below. Required functional, correctness, integrity, resource, safety-deadline and cleanup gates must pass before PHASE1_TERMINAL_PASS. Optional latency/storage optimization of correct operations remains Phase 2.

Preserve all original baseline failures and producing identities; record corrected candidate evidence separately. Fix shared causes and recollect only genuinely invalidated coverage. Do not shrink fixtures, omit cases, weaken oracles, raise budgets/timeouts or add benchmark-only bypasses. Keep affected family issues and #35 open until corrected execution and independent verification pass; #21 stays open for later phases/release.

Parent: #21
Depends on: #22 (shared infrastructure and committed specification).

Problem statement

LayerFS needs an authentic initial baseline for directory construction and whole-workspace reads before deciding product improvements.

Goal and family scope

Construct chains, scan all metadata, and stream all contents across the shared four-tier tree, including wide-directory pagination and the deep spine.

Inventory: 12 new timed cases, 36 prescribed initial performance sample slots, and 0 standalone proof recipes. The canonical planning file is docs/roadmap/0.1/0.1.3/directory-construction-traversal.md. The snapshot below makes the current proposal reviewable until its committed revision is published.

Stage 1 work

  • Freeze this family and its exact fixture/schedule/oracle identities under the shared contract; implement its complete membership in the existing harness with one canonical definition and thin adapter.
  • Qualify route, call counts, size/resource bounds, independent oracles and selected-case self-checks. Resolve harness defects before collecting baseline evidence.
  • Run one selected case/seed to establish a valid execution path, then collect every prescribed initial sample and separate proof variant on the pinned unoptimized product. Required extended cases are explicit but remain required.
  • Retain raw failures/timeouts as classified product findings. Keep setup, performance, verification, cleanup and external command wall distinct.
  • Produce a source-bound family report with all outcomes and measured questions for later optimization; do not tune product algorithms within this issue.

Acceptance criteria

  • Complete family implementation, unique IDs, exact counts, deterministic fixtures/schedules and independent oracle checks.
  • Committed specification and product/harness/build/runtime/input identities linked in the evidence manifest.
  • Every required case/subcase executed with a retained initial outcome; no missing mandatory counter, unimplemented member or silent skipped slot.
  • All prescribed valid sample repetitions retained, with separate independent verification of distinct input/schedule variants. Failed/invalid outcomes remain explicit and ineligible for performance claims.
  • Preparation reuse, per-sample isolation, byte limits and cleanup checked; actual first-use/cache-hit command wall reported.
  • Family baseline summary delivered to the consolidated report issue; known product failures remain failing release gates and are recorded for Stage 2.

Files to read

docs/roadmap/0.1/0.1.3/directory-construction-traversal.md, docs/roadmap/0.1/0.1.3/testing-rules.md, docs/general/benchmark_rules.md, and the existing source/test references in the specification below.

Stage 1 boundary

Build the benchmark and retain its initial outcomes. Product algorithm, storage-format and performance optimization are deferred to Stage 2. Improve benchmark preparation/reuse and correct harness defects now; do not silently change the measured product to make results pass. Pin the unoptimized product identity and distinguish harness changes from product changes.

A slow result, product failure or timeout is an initial finding, not permission to lower a gate, drop a case or start tuning. Keep outcome, harness validity, verification, resources, cleanup and coverage completeness separate. Failed/invalid cases cannot support performance claims. Unimplemented cases, missing mandatory observations and unexecuted slots remain incomplete.

Shared execution rules

  • Reuse fs-bench-pro, existing generators, runner helpers, custody, qualified pristine inputs, independent writable samples and reports. Add only needed family adapters.
  • Use 1/10/100/500 of the declared unit; no Cartesian size/count matrix. Each workload file is at most 500 MiB and aggregate logical content is strictly below 1 GiB at every intermediate state, including temporary files and Git internals. Physical Store/spool/cache budgets are separately reported.
  • Resolve the selected case/seed before preparation. Reuse compatible artifacts; prepare only required inputs. Measured creation/import/history must still execute all work.
  • Iterate one selected case and seed; ordinary warm-prepared 1–5 second command wall is a target to qualify, not a blanket guarantee. Required extended cases are explicitly selected and cannot be omitted from complete baseline coverage.
  • Separate performance and verification. Added benchmark hashes, full manifests, reopen and fault injection stay outside performance timing; intrinsic CAS/Git hashing remains measured.
  • Collect several metrics from one execution and regenerate reports from raw evidence without rerunning successful workloads. Retain all valid results and failures, with exact source, fixture, oracle, runtime and cache identities.
Planning specification snapshot — directory-construction-traversal.md

Source file: docs/roadmap/0.1/0.1.3/directory-construction-traversal.md. Local draft SHA-256: 16a0bf48287f0146ad2f2a725aefa53bb489071c558c3868d61a5af9db505a43. This snapshot is reviewable now; it is not a committed admission contract. The infrastructure issue must publish/freeze the source revision before implementation and collection.

Directory construction and whole-workspace traversal

Status: Current v0.1.3 planning specification; no release candidate or
measured result is implied.

Family ID: directory_construction_traversal. This v0.1.3 implementation
contract contains 12 new timed cases and no standalone proofs. Its family
adapters are not implemented yet.

Purpose and scope

Measure directory construction, complete metadata enumeration, and complete
content scans as distinct operations. A scan must visit the whole selected
workspace, including its wide directory and deep path witness; visiting a few
independent short chains does not establish whole-tree behavior.

Use the shared testing rules (docs/roadmap/0.1/0.1.3/testing-rules.md; planning file awaiting publication) for seeds, sample preparation,
source custody, exact manifests, byte accounting, timing, and resource gates.
All cases use public Workspace Create, managed execution through real FUSE,
one final Commit attempt, and clean End. Separate verification reconnects to
the Store and mounts a new Workspace.

Cases and load units

Expand N over exactly 1, 10, 100, 500; the three rows below define four
scenario IDs each.

Scenario IDs Primary load Measured operation Expected Commit
directory-construct-{N} N directory chains Create each missing chain with ordinary root-to-leaf mkdir calls Created
directory-metadata-scan-{N} N shared-profile shards Enumerate the complete workspace, lstat every entry, and count completed entries UpToDate
directory-content-scan-{N} N shared-profile shards Enumerate the complete workspace, open/read/close every regular file, and count completed bytes UpToDate

Construction uses a fixed 500-shard untouched shared workspace, with new chains
under new-directories/. Its 500-chain seed-bound schedule uses domain
directory-construction; each smaller case is an exact prefix. Chain depths
repeat by scheduled ordinal:

1, 4, 2, 8, 3, 10, 5, 7, 6, 9

Names, modes, and normalized mtimes derive from the sealed fixture schedule.
Report chain units and actual mkdir/affected-directory counts separately.
Creation does not rewrite or copy the background payload.

Both scan curves use the complete N-shard
shared workspace fixture (docs/roadmap/0.1/0.1.3/testing-rules.md#shared-workspace-fixture; planning file awaiting publication), including
its fixed wide-directory layout and 128-component spine. Each shard contains
200 files and exactly 1 MiB: tiers are 200/2,000/20,000/100,000 files and
1/10/100/500 MiB. Scan every directory component as well as every file; retain
actual entry count, fan-out, and maximum depth. Ordered path transcripts and
their digests are collected only in separate verification executions.

Traversal uses ordinary directory handles and consumes every pagination page
until EOF. Resolve and visit entries in bytewise name order. Performance mode
retains cheap counts; separate verification records the exact transcript and
rejects missing or repeated entries even when a total count matches.
Metadata scans do not open/read file payloads. Content scans use bounded buffers,
read each entire file through EOF, and count completed bytes. Only verification
mode computes per-file hashes and an ordered whole-tree content digest. Metadata
and payload scans remain distinct because metadata work can pass while payload
delivery fails or scales badly.

The largest scan or construction workspace contains 500 MiB of logical file
payload; added construction paths carry no payload. The shared profile's
largest file is 48 KiB. Fixed directory witnesses add no regular-file bytes.
No secondary materialized tree is created in the tested namespace.

Timing, metrics, and independent oracle

Measure the entire traversal or construction inside the workload timer.
Directory reads, per-entry metadata calls, and content reads are measured work.
Full hashes, transcript checks, and tree oracles never run in performance mode.
Keep Workspace Create, ordinary sync for construction,
Commit, visibility, End, and complete lifecycle walls separately attributable.
Independent full verification has its own wall, receipts, and deadline, and
never contributes to a performance distribution.

Record directories and regular files processed, resolved components, payload
bytes, fan-out/depth, FUSE lookup/getattr/mkdir/opendir/readdir/releasedir/read
counts, CPU, memory, swap/OOM, Store/object changes, and cleanup. Content scans
report logical payload throughput; metadata scans report entries per second.
Do not divide directory-only work by payload bytes to invent throughput.

The expected manifest is sealed independently from the product and timed
traversal. The separate verifier compares every path, type, size, mode, declared
mtime, content digest, and relevant link identity after fresh Store reconnect
and real-FUSE remount. Construction adds exactly the scheduled directories and
preserves all unrelated paths. Scans preserve the whole manifest, Branch head,
and canonical root, return UpToDate, and create no durable payload state.
Verify exact traversal membership, EOF behavior, full-content read transcripts,
transient byte bounds, and absence of leaked handles or runtime resources.

Execution and completion

Three fresh performance samples per case produce 36 timed executions;
every case receives separate independent verification. Reuse the shared shard
manifests and prepared input Stores through the existing custody/sample-clone
machinery; each timed execution still owns independent mutable runtime state.
The benchmark binary, workload helper, runner, and report machinery are reused.

Prospective selection is one case and one seed, followed by its focused
verification; the selector is not implemented yet. Start with the affected
directory-construct-1, directory-metadata-scan-1, or
directory-content-scan-1. The provisional ordinary selected-run target is
1–5 seconds. Large scans, whole families, and exhaustive verification use the
longer lane with baseline-derived budgets; the 100,000-file scan is never
advertised as a guaranteed few-second test.

Completion requires all 12 cases and seed samples, exact independent full-tree
proofs, all pages and file bytes visited, bounded resource and size evidence,
and clean teardown. Existing historical shell-directory controls keep their
own meanings and are not additional members of this family.

Activity

  1. self-assigned this
    on Sep 4, 2026
  2. yifanxuaaa commented on Sep 4, 2026

    @yifanxuaaa
    ContributorAuthor

    Performance-results checkpoint for #25: 33 independently validated active VM8 performance samples. The report validates 0 retained full proof(s) for this family at this checkpoint; further full checks are running. This comment does not claim global Phase1 terminal pass.

    Published report, exact rows, source identities and verification status, phase/counter evidence.

    VM8 performance source: 7948df2de269e5ffd47a232ffd8091ff83f8869f; fixed 8-CPU Docker VM, unchanged 2-CPU/2-GiB benchmark-container caps. No VM4/VM8 timing pooling. Every listed performance row passed the independent authenticity, custody, observation, resource and cleanup validation; workload admission still requires its separate full proof.

    Values below are medians across the prescribed three seeds. Product time is the declared cumulative public-operation time; preparation and complete sample wall are separate. Store growth is the observed physical Store-file delta, not a deduplication ratio or verified canonical byte count.

    Case n Product ms Preparation s Complete sample s Store delta MiB Full proofs in checkpoint
    directory-construct-1 3 145.402 4.445 4.954 0.000 0/3
    directory-construct-10 3 155.081 4.474 5.036 0.438 0/3
    directory-construct-100 3 1071.433 4.406 6.022 2.938 0/3
    directory-construct-500 3 5253.349 4.462 10.234 7.250 0/3
    directory-content-scan-1 3 228.851 2.454 3.052 0.000 0/3
    directory-content-scan-10 3 1320.354 2.445 4.253 0.000 0/3
    directory-content-scan-100 3 12578.807 2.798 15.869 0.000 0/3
    directory-metadata-scan-1 3 103.801 2.366 2.849 0.000 0/3
    directory-metadata-scan-10 3 215.148 2.382 2.963 0.000 0/3
    directory-metadata-scan-100 3 1650.183 3.533 5.599 0.000 0/3
    directory-metadata-scan-500 3 9039.163 4.724 14.236 0.000 0/3

    Explicit Phase1 suppressions (not passing coverage): directory-content-scan-500. Definitions, historical failures and incomplete attempts remain retained for Phase2.

    All prior actual failures, source incompatibilities, invalidations and preparation interruptions remain in the linked report and raw evidence. The fast development verifier contributes zero full-verification credit. Completed family tickets will close only after their remaining mandatory full checks pass; #21 remains open for later phases.

  3. yifanxuaaa commented on Sep 4, 2026

    @yifanxuaaa
    ContributorAuthor

    The user has amended Phase 1 acceptance: qualified fast verification is sufficient for routine benchmark verification. The published amendment supersedes earlier comments requiring every routine exhaustive proof before closure. The reference qualification contract preserves independent expected states, affected-content/metadata/alias/absence checks, authenticated references and deterministic witnesses.

    The posted performance results remain valid at their original source/environment identities. Completed full proofs remain accepted and are reused. At the scope checkpoint,48 routine slots already had full proofs; the305 unrun routine exhaustive counterparts are now deferred to Phase2 while their qualified fast checks are implemented and collected. Fast evidence keeps its actual assurance and explicit skipped coverage; it will not be labelled fully_verified.

    Required targeted reliability checks, the CDC boundary proof, exact expected-error oracles, resource and cleanup gates still apply. Observed failures still require focused repair confirmation. The already-qualified600-second proof is retained rather than repeated. This amendment does not manufacture missing evidence or itself establish terminal pass.

    Remaining routine families will close after their accepted fast/full coverage and applicable gates pass. #21 remains open for later phases and release.

  4. yifanxuaaa commented on Sep 4, 2026

    @yifanxuaaa
    ContributorAuthor

    Read-only pin repair performance results for #25 — 6 corrected qualified PASS; 1 corrected non-PASS outcome(s). The corrected passing samples have completed independent validation with zero issues or resource violations. It does not claim terminal pass.

    Corrected runtime source: 30d13deeec72b46ff7bc411f1ec08a46990541e1. Other rows retain their original qualified checkpoint at 7948df2de269e5ffd47a232ffd8091ff83f8869f with their original numbers and source. Exactly 45 historical slots across #23/#25/#29/#33 were affected. The correction produced 42 independently qualified PASS slots; directory-content-scan-100 seed 1 exceeded the cumulative 15-second limit and its seeds 2/3 were not run. That case is now suppressed for all Phase 1 seeds/sources: 15 disabled case IDs, 370 active performance slots (328 retained plus 42 corrected). No old/new timings are pooled.

    The changed product branch now acknowledges the existing read-only regular-file Pin before exposing the Open handle. Historical affected rows remain actual observations of their old source; they are superseded for the repaired candidate, not deleted or called failed. No speedup claim is made.

    Product ms, preparation s, complete sample s and physical Store-file delta MiB use the prior table definitions. Corrected medians use only the listed completed qualified PASS seeds; incomplete three-seed groups are explicitly partial. They are medians of independently qualified samples. The full-proof column is copied from the prior checkpoint and is not a refreshed current proof count. Current routine acceptance follows the authorized fast-verification amendment; a fast result remains fully_verified=false. Targeted reliability gates are unchanged.

    Case n Product ms Preparation s Complete sample s Store delta MiB Full proofs in prior checkpoint Source Disposition
    directory-construct-1 3 145.402 4.445 4.954 0.000 0/3 7948df2d Retained prior qualified checkpoint
    directory-construct-10 3 155.081 4.474 5.036 0.438 0/3 7948df2d Retained prior qualified checkpoint
    directory-construct-100 3 1071.433 4.406 6.022 2.938 0/3 7948df2d Retained prior qualified checkpoint
    directory-construct-500 3 5253.349 4.462 10.234 7.250 0/3 7948df2d Retained prior qualified checkpoint
    directory-content-scan-1 3 275.933 2.345 3.011 0.000 0/3 30d13dee Qualified PASS
    directory-content-scan-10 3 1803.143 2.415 4.683 0.000 0/3 30d13dee Qualified PASS
    directory-content-scan-100 0 admitted — — — — 0/3 30d13dee Suppressed policy: seed 1 FAIL at 15.000758416s product time; seeds 2/3 not run
    directory-metadata-scan-1 3 103.801 2.366 2.849 0.000 0/3 7948df2d Retained prior qualified checkpoint
    directory-metadata-scan-10 3 215.148 2.382 2.963 0.000 0/3 7948df2d Retained prior qualified checkpoint
    directory-metadata-scan-100 3 1650.183 3.533 5.599 0.000 0/3 7948df2d Retained prior qualified checkpoint
    directory-metadata-scan-500 3 9039.163 4.724 14.236 0.000 0/3 7948df2d Retained prior qualified checkpoint

    Historical affected rows (preserved exactly from the prior qualified table; excluded from corrected medians):

    Case n Product ms Preparation s Complete sample s Store delta MiB Full proofs in prior checkpoint
    directory-content-scan-1 3 228.851 2.454 3.052 0.000 0/3
    directory-content-scan-10 3 1320.354 2.445 4.253 0.000 0/3
    directory-content-scan-100 3 12578.807 2.798 15.869 0.000 0/3

    Exact corrected outcomes and historical supersession mapping:

    • directory-content-scan-1 seed 1: directory-content-scan-1-s1-performance-92c92e149728 (independently qualified PASS), replaces timing selection directory-content-scan-1-s1-performance-82027712abc8.
    • directory-content-scan-1 seed 2: directory-content-scan-1-s2-performance-d598baf149f1 (independently qualified PASS), replaces timing selection directory-content-scan-1-s2-performance-4d2bd9b22e70.
    • directory-content-scan-1 seed 3: directory-content-scan-1-s3-performance-683aa35574d7 (independently qualified PASS), replaces timing selection directory-content-scan-1-s3-performance-ef311ced5ce5.
    • directory-content-scan-10 seed 1: directory-content-scan-10-s1-performance-a4695b940d6d (independently qualified PASS), replaces timing selection directory-content-scan-10-s1-performance-782988abe1ea.
    • directory-content-scan-10 seed 2: directory-content-scan-10-s2-performance-6ea254169d5f (independently qualified PASS), replaces timing selection directory-content-scan-10-s2-performance-3e69aab4f67a.
    • directory-content-scan-10 seed 3: directory-content-scan-10-s3-performance-da5e887db0ea (independently qualified PASS), replaces timing selection directory-content-scan-10-s3-performance-b329714a68c7.
    • directory-content-scan-100 seed 1: directory-content-scan-100-s1-performance-3d28647a3175 (fail; awaiting independent validation), replaces timing selection directory-content-scan-100-s1-performance-07322589de78.
    • directory-content-scan-100 seed 2: not run under persistent Phase 1 time-budget suppression; historical selection directory-content-scan-100-s2-performance-38a989f175c1 remains preserved and contributes no corrected admission.
    • directory-content-scan-100 seed 3: not run under persistent Phase 1 time-budget suppression; historical selection directory-content-scan-100-s3-performance-e2a280392b74 remains preserved and contributes no corrected admission.

    All attempt paths are under benchmark-results/fs-bench-pro/phase1-v013/attempts/. Source, custody, observations, resources and cleanup were independently checked; matching routine correctness status is reported separately. Original failures and suppressions remain preserved; central #21 stays open.

    Published corrected timing checkpoint and per-case medians.

  5. yifanxuaaa commented on Sep 4, 2026

    @yifanxuaaa
    ContributorAuthor

    Phase 1 is closed under the user's explicit withdrawal of all remaining verification tests. This is a scope withdrawal, NOT a claim that the original verification terminal criteria passed.

    The assistant designed an excessively expensive verification suite and spent too long expanding it after the user requested a much faster finish. That design is withdrawn and must be reassessed in Phase 2. No more verification work is scheduled.

    Final performance results:370 independently qualified active observations, with actual source identities and unoptimized timings. All15 suppressed case definitions, original failures and incomplete attempts remain preserved. Completed29 targeted proofs,48 routine full proofs and qualified fast evidence remain historical evidence; unrun verification is withdrawn/deferred, never relabeled passing.

    Explicit withdrawal and accountability record. Separate Phase 2 backlog. Central#21 stays open for Phase 2/release. No release tag was created.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

new featureA new user-facing capabilityv0.1.3LayerFS v0.1.3 checkpoint and delivery

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions