Repository navigation
Description
Low-tier applicability finding — 2026-09-06
The current optimized product has now completed all four requested bulk tier1/tier10 samples. The observed Commit reductions extend to small create workloads and tier10 deletion. Tier1 deletion remains approximately7ms; its0.354667ms increase is too small to classify as a demonstrated regression from these individual samples.
One seed1 sample per unchanged -compact-v2 case. All values below are milliseconds, preserving nanosecond precision.
| Case | Affected files / MiB | Current Exec ms | Historical Commit ms | Current Commit ms | Current full lifecycle ms |
|---|---|---|---|---|---|
tiny-bulk-create-1-compact-v2 |
50 / 1 | 206.973125 | 45.502000 | 9.743625 | 230.565041 |
tiny-bulk-delete-1-compact-v2 |
50 / 1 | 96.636750 | 6.740833 | 7.095500 | 114.032375 |
tiny-bulk-create-10-compact-v2 |
500 / 10 | 550.552083 | 193.929583 | 40.627167 | 604.193624 |
tiny-bulk-delete-10-compact-v2 |
500 / 10 | 199.131625 | 23.885833 | 10.010417 | 220.390332 |
Interpretation and stopping rule
- Historical and current fixture compatibility dictionaries and selected operation/normalization counts match. Product/source/harness/binary/image differ; these are historical unpaired observations, not a controlled speedup experiment or statistical distribution.
- Tier1 delete changes from6.740833ms to7.095500ms (+0.354667ms). Its small50-file workload leaves less removable work and can be relatively sensitive to fixed coordination/transaction/publication costs. This is a plausible explanation, not a measured phase attribution. A single sample cannot distinguish ordinary timing variation from a small added overhead. Treat it as roughly unchanged for this practical review; preserve both exact results.
- Do not start a timing-only optimization or repeat campaign for tier1 delete. The measured reductions in create Commit and tier10 delete support the applicability of the shared changes; no claim that every phase improved. Higher create Exec timings and the small delete1 Commit increase are retained.
- These results still do not include the separately owned unfinished/unmerged [v0.1.3] Optimize live FUSE Exec in parallel with the Commit migration #48 Exec implementation. Continue next with [v0.1.3] Redesign FUSE Exec around one shared live Workspace core #49's scoped producer simplification as already agreed; combined-product acceptance remains separate.
Shared mechanisms and evidence
Create rows exercise shared spool extents, existing full-file canonical construction, immutable owned delivery through the bounded pipeline, checked admission and in-place checkpoint. They use1/11physical segments and have zero selected spill readback. Delete rows use the same Commit driver, ordered reference processing, checked structural admission and checkpoint; each produces7objects/12,502bytes with no file-payload queue or spool. Do not attribute a payload-spool gain to deletion.
The low-tier witness is unchanged at50files/1MiB (distinct from the high-tier mixed-v3 witness). All samples passed cleanup, prepared-master isolation and resource checks; OOM/swap0, physical spoolfiles/bytes0 after Commit. No new product edits, rebuild, high-tier reruns, low-tier targets or independent proofs were introduced.
Evidence commit 9600e7f90; executing source 459eefaa01db3fc08c7b40c930da5981d8b04bc1, source seal b6159377eadfc07806aaba102525a8b5d7b10c8ef2c0b48c834f3b2fb60731ca, product seal 4d1f3e9348a516fa4815defe8f6279c9ada45f20d76285b5d56e14a629bf74c8. Full identities, hashes, raw paths and historical comparison are in the integration worktree's benchmark-results/host-store/results/issue47-compact-low-tier-assessment.json and docs/roadmap/0.1/0.1.3/issue47-subsecond-workspace-results.md, attempts17–20.
All four bulk tiers now have current-product absolute observations. Independent final proofs and combined Exec integration remain pending; #47 and its parents stay open.
Current measured checkpoint and next step — 2026-09-06
Mixed-v3 migration and both requested tier pairs are complete. The separate #48 Exec implementation is not yet finished/integrated into this measured product. Proceed next to producer simplification #49; preserve the current Commit improvements and do not resume threshold-chasing Commit tuning. #47 remains open pending combined-product full-lifecycle acceptance and final proofs.
This current section supersedes historical pending-migration, tier500-deferral and earlier300ms exploration wording below. The accepted tier100 Commit objective is around400ms with approximately50ms tolerance. Historical amendments and raw receipts are retained, not rewritten as new results.
Exact measurements
One seed1 performance sample per revised case. All times below are milliseconds, retaining nanosecond precision in six decimal places. Complete time is the sum of Create session + Exec + Commit + visibility + End.
| Case | Session create ms | Exec ms | Commit ms | Visibility ms | End ms | Complete ms |
|---|---|---|---|---|---|---|
tiny-bulk-create-100-mixed-v3 |
8.932250 | 956.095375 | 370.395375 | 0.075542 | 6.622875 | 1342.121417 |
tiny-bulk-delete-100-mixed-v3 |
10.784792 | 280.367875 | 15.033084 | 0.068708 | 2.991291 | 309.245750 |
tiny-bulk-create-500-mixed-v3 |
10.247667 | 4307.807708 | 1862.498042 | 0.096875 | 19.426417 | 6200.076709 |
tiny-bulk-delete-500-mixed-v3 |
9.871667 | 874.646083 | 50.838875 | 0.070291 | 4.588333 | 940.015249 |
- Tier100 create: Commit370.395375ms meets the practical Commit objective; complete1,342.121417ms is a strict [v0.1.3] Redesign shared Workspace processing for subsecond bulk create/delete at tier 100 #47 TARGET_MISS,342.121417ms above1second. Exec956.095375ms remains part of that result. No missing Exec-track gain is assumed or subtracted.
- Tier100 delete: complete309.245750ms passes the strict subsecond performance target; Commit15.033084ms. Final independent proof remains NOT_RUN.
- Tier500 create/delete: complete6,200.076709ms /940.015249ms both pass the existing15second family performance threshold. There is no tier500 sub400ms Commit gate. These rows do not close the broader20-case family.
- The mixed distribution changes workload cardinality: tier100 has1,000affected files/100MiB, tier5005,000/500MiB, plus unchanged200-file/1MiB witness. New absolute results must not be advertised as a product speedup over the original20k/100k-file workloads.
Provenance, checks and diagnostic allowances
Migration spec 66983181a, implementation 0ac1ebcf4, tier100 results 9b68a715a; tier500 allowance spec 6fea9d2b5, harness change 459eefaa0, results 4b266eb3c. Current local integration worktree HEAD: 4b266eb3cedcd1f1a991a318656958fba9f669c7 in /Users/yifanxu/.codex/worktrees/7311/layerfs.
Product seal is unchanged across all four samples: 4d1f3e9348a516fa4815defe8f6279c9ada45f20d76285b5d56e14a629bf74c8. Each tier's create/delete pair has identical delivered source/product/image identity. Tier500 differs from tier100 only in recorded benchmark allowance/source wiring and workload, not an additional product optimization or adoption of unpublished Exec changes.
- Tier100: source
e80a564bcdbdaf38b3b8f18a8b8b4e50b4161a2dc3f63c50a4308c8132960417; imagesha256:9b4638f099f0d2e0ed8a739a533c2e4f964f53bc9eec6538181e2f08ba70ed50; host binarybf38adb473a5b87973a38a77267095af357032c28bf615797abc7841fbfe9480. - Tier500: source
b6159377eadfc07806aaba102525a8b5d7b10c8ef2c0b48c834f3b2fb60731ca; imagesha256:ff14c13a66f1224fe5879d682b0345015e52a626eab381a5f9f27995ce985002; host binaryba22f2936ef28e07a74fc6a5a6dc5abc822b36da59c52958e1c16ae4290e20cf.
Tier500 used the explicitly authorized600second product/630second outer/600second preparation diagnostic allowances; the default product watchdog remains120seconds. Thresholds, resource caps and45s-work/59s-hard independent proof limits did not change. Two watchdog checks and16 runner checks passed for the allowance change. Profile arithmetic/registration/custody checks were completed before mixed-v3 collection.
All four samples: cleanupPASS, prepared masters unchanged, no OOM/swap, host SQLite/no data mounts, existing2CPU/2GiB/no-swap/256PID container constraints, zero physical spool files/retained bytes after Commit. Host resources are separately reported. No independent final proofs have run and no issue is qualified by a partial phase alone.
Raw receipts under benchmark-results/host-store/results/:
issue47-mixed-v3-create100/perf.jsonl; SHA-25693a116a6281d2da42b7230cc9ca335d0d0df8034dbac8728a30f3b6f85ad23af; input94478fdc58ea9c3e677dc734ed375029e30b84958775a24a5b43ade4c3624027.issue47-mixed-v3-delete100/perf.jsonl; SHA-256af70bce5b3d2e4c83013ac8758c510e7c8973370ef67b3c5e48e55fbd547fe1b; input57afdc6a47c2bf3c2d694be5667c04a8dc60c74843993c3a7034350bcf2a151a.issue47-mixed-v3-create500/perf.jsonl; SHA-256fc19ed12c45fe6f1a6cb3f6bd0f07c519578aa5b918072782105117410411394; inputf9524311661812fd26332de47f379b053121602ab33a059043b4b1d5522acce3.issue47-mixed-v3-delete500/perf.jsonl; SHA-2561832a640eefa649fcb515b11d20771ce9954d00187a9532a9a7456f59d8b0965; input2a752bda5e8a7022f2f7e97a4a1bf4ffb12399a52d74df8187cf3a1b67a791a5.
Machine assessments: issue47-mixed-v3-definition/pair-assessment.json and issue47-mixed-v3-definition/tier500-pair-assessment.json. Full evidence: docs/roadmap/0.1/0.1.3/issue47-subsecond-workspace-results.md, attempts13–16. Raw receipts and precise identities are local to the integration worktree; no artifact upload is implied.
Next sequence
- Record this checkpoint and move next to [v0.1.3] Redesign FUSE Exec around one shared live Workspace core #49: simplify shared producer task execution/construction and caller-supplied resource configuration, preserving defaults initially and existing finality/recovery/resource guarantees.
- Continue coordinating with the separately owned [v0.1.3] Optimize live FUSE Exec in parallel with the Commit migration #48 Exec work. These measurements do not include that unfinished/unmerged implementation; do not infer combined speed from separate tasks.
- Integrate the delivered changes, then measure the actual combined product on the required revised tier100 cases. Keep final independent proofs at the agreed stage. Preserve exact misses; no automatic closure of [v0.1.3] Redesign shared Workspace processing for subsecond bulk create/delete at tier 100 #47/[v0.1.3][Phase 2] Optimize tiny_file_churn Exec and Commit as the first shared recovery step #46/[v0.1.3][Phase 2] Optimize and restore the 15 runtime-suppressed benchmark cases #39.
Approved mixed bulk workload v3 — 2026-09-06
This explicit user amendment supersedes earlier requirements to retain 20,000/100,000 affected files for the tier-100/500 bulk create/delete rows. Total affected payload remains unchanged; this is a versioned workload revision, not a product speedup.
| Tier | Affected files | Affected payload | Distribution |
|---|---|---|---|
| 100 | 1,000 | 100 MiB | 1 x 50 MiB; 800 x 4 KiB; 199 medium files totaling 46.875 MiB |
| 500 | 5,000 | 500 MiB | 3 x 100 MiB; 4,000 x 4 KiB; 997 medium files totaling 184.375 MiB |
Exact medium sizes: tier100 has194 x246,995B +5 x246,994B; tier500 has936 x193,913B +61 x193,912B. The separate untouched200-file/1MiB witness is unchanged. Populated totals including witness are1,200files/101MiB and5,200files/501MiB. Counts above exclude directories.
Replace the four active high-tier bulk IDs with tiny-bulk-create-100-mixed-v3, tiny-bulk-delete-100-mixed-v3, tiny-bulk-create-500-mixed-v3, and tiny-bulk-delete-500-mixed-v3. Family membership remains20; low-tier compact and individual create/stat/unlink definitions are unchanged. Preserve historical raw receipts and IDs; do not pool, relabel or compare across profiles as a product speedup.
Status: docs/issues approved; registry/generator/verifier migration is pending. Existing IDs still run the old workloads. Before new performance collection, migrate registration, bulk fixture/expected/apply, sampled proof (cover every large file with bounded ranges), input/cache identity and assessment together; commit the specification with the implementation. New target preparations must not reuse old target trees. Reuse existing bounded content generation, directory shapes and shared init/Commit pipeline; no product case/size policies or large in-memory writes.
#47 and #48 qualification now refers to the revised tier100 IDs once implemented, with each complete lifecycle strictly below1000ms. Approximate300ms Commit remains exploratory, not an extra phase gate or tier500 target. #46 broader20-case scope and15s classifier, host-SQLite/no-data-mount topology, serial iteration,120s/130s diagnostic allowances and final-stage45s/59s proofs remain unchanged. Tier500 remains later work. Do not close any issue based on historical-ID samples or claim the file-count reduction as an optimization gain.
Specification with deterministic path/size mapping and migration checks: docs/roadmap/0.1/0.1.3/tiny-file-churn-mixed-v3.md. Benchmark entrypoint note: benchmark/fs-bench-pro/families/tiny_file_churn/README.md.
Unified construction/persistence amendment — 2026-09-06
Initialization and ordinary Workspace Commit must share the actual bounded construction, final-object delivery and checked persistence machinery, with different source inputs and lifecycle boundaries. Create/delete/edit/rename stay in one Commit implementation. No benchmark-name, size or deletion-density engine selection.
Current implementation checkpoint: 8cbbb425b shares physical spool segments; the owner reports 51 Workspace tests, targeted lifetime/accounting checks, 12 SDK file-edit integration tests and host build passed. Its performance sample was pending at this review. Pre-shared-spool create Commit measured 3.562 s, including 2.079 s retirement; delete Commit measured 0.460 s. These are diagnostic observations, not final qualification.
Implementation order:
- Finish the already-started shared-spool measurement; reuse newer evidence if available.
- Define stable inode inputs and proven-final output ownership. Reuse existing PieceTree/rope/FileMutationBatch, exact metadata, ordered directory/inode builders and the already-shared checked consumer.
- Transfer finalized owned objects without a mandatory complete candidate payload spill/readback (latest create read back 113.4 MB). Preserve incremental reads and finality of provisional structural objects.
- Reuse bounded queue/backpressure and overlap construction/admission. Start with one producer to isolate handoff, then a small bounded worker set if justified. No full-workspace RAM buffer or new scheduler framework.
- Keep one ordered namespace/reference finalizer and existing publication/checkpoint recovery. Remove superseded ordinary wrappers/buffering only after their real callers and guarantees transfer.
Deduplication is already shared: conflict-aware insertion followed by bounded equality reads only for existing IDs. About 1% of create candidate bytes were reused. Staging is a root-reference row, not another payload upload; publication was 0.223 ms, with stage outside that timer. Neither is supported as the seconds-scale bottleneck. Do not drop collision checks, expected-head publication, retained-candidate recovery, or copy initialization's empty-Store/whole-Store failure cleanup.
Account all worker scratch, queued/partial batches, results, retained sources and physical segments. Producers must drain/join on failure. Host construction and SQLite are already colocated; Docker writes generally transfer during Exec. A changed drain must remain in full lifecycle timing.
Subsecond create Commit is a plausible objective and delete has a retained 460 ms sample; no new restrictive per-phase/100 ms gate is introduced. Existing full-lifecycle #47 acceptance, host-only topology, serial iteration, final-only bounded proofs and replan-on-no-go remain unchanged. Parent issues and sibling qualification remain open.
Detailed source map, diagram, failure contract, deletion ledger and experiments: docs/roadmap/0.1/0.1.3/issue47-unified-construction-pipeline.md (saved project documentation; implementation owner must bring this amendment into its worktree). Linked from the #46/#47 guides.
Execution order amendment — Commit first, then Exec
The latest user instruction prioritizes shared Commit optimization before Exec:
- Replace reconstructed ordinary rebase with candidate-bound in-place checkpointing, preserving exact returned state, live nodes/handles, dirty bookkeeping and publication/install recovery.
- Reduce remaining repeated Commit construction, candidate payload replay where safe, and deletion reference traversal using existing shared primitives.
- Record a measured Commit-phase checkpoint after the major work reductions and focused regressions; then optimize Exec metadata/namespace dependencies and shared data ownership. No new standalone Commit time gate or marginal-tuning campaign.
- Full targets remain both original bulk tier-100 lifecycles strictly below 1000 ms. Independent sampled proofs remain final-stage only after both performance targets pass, with 45-second work / 59-second hard limits.
The local issue47-subsecond-workspace-plan.md has been updated to this order. A Commit-only improvement does not close this child, and removing rebase reconstruction does not mean skipping checkpoint bookkeeping or silently removing active-execution guards.
Outcome and scope
Parent: #46; grandparent: #39. Optimize exactly tiny-bulk-create-100 and tiny-bulk-delete-100, each with the full original workload and complete declared product lifecycle strictly below 1,000 ms. These are the 20,000-file / 100 MiB bulk cases, not the sparse tiny-create-100 / tiny-unlink-100 cases.
This is a smaller qualification scope with permission for substantial shared architectural changes. It is not another marginal cache-tuning exercise. The implementation must improve ordinary LayerFS Exec/Commit for applicable siblings, maximally reusing existing code. No benchmark-name, file-size, or deletion-density policy selects a product engine.
The previous implementation task's existing work is checkpointed locally on main at 3faaf3839 (not pushed). It includes host tiny admission, longer diagnostic execution, bounded final sampling, closed-create and framing improvements, final new-inode reference accounting, directory caching/paging, and parent-resolution reuse. This is incomplete progress, not #46 completion.
Current evidence, not a speedup requirement
| Case / receipt | Product total | Exec | Commit | Main Commit cost |
|---|---|---|---|---|
create-100, issue46-parent-create100 |
22.142764 s | 15.086029 s | 6.988436 s | continuation 5.258958 s |
delete-100, issue46-pages-delete100 |
4.991831 s | 3.297524 s | 1.679852 s | namespace 1.671591 s |
Receipts are retained under benchmark-results/host-store/results/. The delete receipt predates the parent-cache product revision; these are latest available observations, not a same-source final pair. Reuse them for diagnosis; collect final evidence on the delivered product identity. Historical absolute timings are not optimization denominators.
The complete earlier create measurement exposes independent costs: metadata normalization 10.375 s, spool-write processing 1.887 s, and non-rebase Commit 1.636 s. These are implementation costs, not physical floors. Removing only rebase or only metadata cannot achieve subsecond completion.
Architectural direction
1. One coherent live state for Exec
Investigate authoritative execution of metadata and namespace operations in the existing FUSE-side live view, with bounded ordered handoff to host Workspace state. Reuse reservations, cached inode/binding data, callback draining, pause/invalidation, and authenticated batch acquisition. Avoid host enumeration/materialization followed by another host resolution of the same bindings for deletion.
Success must be decided at an owner with sufficient inode/lifetime, validation, ordering, and resource facts. Simply marking chmod/mtime asynchronous is not equivalent. Preserve the prescribed separate chmod and timestamp syscalls, synchronous errors, fsync/host-observer visibility, aliases, and open-unlinked behavior. Host SDK/Workspace persistence and SQLite remain authoritative; do not add a second Store or distributed-framework service.
2. Ordinary Commit checkpoints the existing Workspace
Remove reconstructed rebase for ordinary successful Commit. Candidate construction retains bounded installation facts keyed by stable NodeId and bound to the candidate root. After checked admission and conditional publication, advance exact returned root/head/base, attach final canonical identities, clear only published changes, and continue using existing paths/nodes/handles. Retain publication and unfinished installation facts through recovery so retry does not recommit.
Replace reread-based validation with construction/checkpoint consistency checks; do not delete integrity guarantees. Reconciliation can genuinely change the live state and must retain correct semantics. Two-store V2 history is not a reason to preserve ordinary reconstruction.
Same-session continuation is required. Commit while another managed execution remains active is not required by these two workloads and is a separate generation-boundary design obligation; do not silently delete the current Busy guard. If ownership changes require generations, preserve post-boundary writes and open handles without expanding this child into a full concurrent-execution feature campaign.
3. Remove per-logical-file host spool ownership where it limits the target
Use shared bounded segment extents for ordinary modified data instead of 20,000 host spool files/descriptors and retirements. Reuse PieceTree base/zero/extent composition and existing bounded storage primitives. Preserve sparse/overlap behavior, short-write rollback, durability, physical accounting, open-unlinked lifetime, and bounded reclamation. No tiny/large spool selector or in-memory-only benchmark exception.
4. Compile final state once
Reuse sorted affected-page directory/inode updates and checked owned admission. Retain the completed final-reference fix. For deletion, consume bounded directory pages and batched authenticated inode reads while preserving sequential reference effects; do not repeatedly seek one child from the root or add a density-based survivor engine. Native inode-pair serialization is not a general sorter. Investigate write-to-canonical reuse only when it removes duplicate work or produces measured critical-path overlap.
Execution and replanning
- Inspect current code and evidence first; do not reimplement checkpointed work.
- Start with a work/dependency ledger for these two tier-100 cases. Attribute real metadata requests, local callbacks, spool processing, construction, checkpoint installation/cleanup, and deletion release.
- If needed, time initialization of the exact prepared final tiny tree using the existing host initializer. It is a nonqualifying construction comparator, not a new registry case or a replacement for timed FUSE work. Native/control timings are not mathematical lower bounds.
- Measure required FUSE/dependency costs before committing to a broad ownership/spool refactor. Keep controls explicitly nonqualifying.
- Implement the smallest coherent architectural slice, with a predicted reduction in messages, file opens, canonical passes, page reads, or reconstruction. One selected performance sample/seed, serial; reuse safe preparation. Use focused semantic regressions during development.
- Replan on every no-go. Do not repeat unchanged failures, silently relax the target, or substitute partial-phase success. Continue through recoverable problems; report real external blockers precisely and never invent PASS.
- No tier 500, all-20 campaign, automatic multi-seed/median/scaling gates, or arbitrary per-phase budgets. Complete this two-case step before expanding. Once both practical gates and final correctness pass, stop marginal tuning.
Fixed environment and timing
macOS owns SDK, Workspace/spool storage and embedded SQLite. Docker Linux runs the daemon, workload helper and real FUSE only. Docker-owned SQLite is permanently prohibited. No data-sharing mounts/volumes or Docker socket. Container: 2 CPUs / 2 GiB / no swap / 256 PIDs; host CPU uncapped and separately observed. Preserve existing product and workload memory/storage limits.
pure_call_sum_ns includes Create, Exec, complete Commit/checkpoint, visibility and End. Required creation, bytes, metadata, traversal, sync, cleanup and publications remain inside their declared timing. Moving work to End or outside the product timer is not elimination.
Keep the 120-second diagnostic watchdog / 130-second outer allowance. The existing 15-second parent PASS classifier is not this child's acceptance: results from 1–15 seconds remain child TARGET_MISS. Use one explicit child assessment over complete source-bound receipts, not benchmark-specific product behavior.
Final acceptance
- Original
tiny-bulk-create-100, seed 1: completepure_call_sum_ns < 1,000,000,000. - Original
tiny-bulk-delete-100, seed 1: completepure_call_sum_ns < 1,000,000,000. - Both final observations use the delivered product identity and required host topology; full inputs/operations, resources, isolation and cleanup pass.
- Shared implementation removes demonstrated work and preserves incremental small-change behavior; existing primitives and applicable sibling callers are explicitly mapped. No per-family optimization integration or new policy engine.
- Focused regressions cover ownership/ordering, reference counts/aliases, open-unlinked state, write rollback, fsync, exact-snapshot continuation and publication/install failure recovery.
- Only after both performance targets pass, run selected independent sampled proofs serially: 45 seconds work / 59 seconds hard end-to-end each, bounded oracle generation and explicit coverage/omissions. No exhaustive proof campaign.
- Publish exact before/after attempts, counters, identities, final proof coverage, remaining limitations, and shared-code adoption. A feasibility result is not terminal PASS; no guaranteed subsecond claim before measurement.
No automatic closure of #46/#39 or siblings, no release, and no pushing/merging unrelated work. Detailed implementation plan will be recorded beside the #46 guide under docs/roadmap/0.1/0.1.3/.
Draft implementation plan
Read-only Exec, Commit/spool, and evidence reviews are consolidated in:
/Users/yifanxu/Ephemeral-AI-Lab/layerfs/docs/roadmap/0.1/0.1.3/issue47-subsecond-workspace-plan.md
The draft includes the proposed architecture, two diagrams, staged experiments, semantic failure contracts, and strict two-case acceptance. It is recorded in local documentation commit 8aec76f76; product progress is checkpoint 3faaf3839. Neither commit has been pushed. Final review found no material issues; no new architectural implementation or subsecond result is claimed.
Activity
yifanxuaaa commentedon Sep 5, 2026
Implementation checkpoint:
4b40af1c4, local and not pushed. Work continues in the isolated worktree. Both subsecond targets and final proofs remain outstanding.Issue 47 implementation results
Status: in progress; neither full subsecond target has passed. No independent benchmark proofs have run.
First implementation slice
Ordinary Commit now installs candidate-bound backing facts into existing Workspace nodes. It retains paths, NodeIds, pins and directory-parent associations. A bounded anonymous metadata journal survives publication/install failure. Canonical construction checks final record kind/content/length/metadata/reference facts before publication; installation checks the exact candidate root, mutation generation and live attributes. Reconciliation still refreshes from its exact published snapshot.
47 Workspace library tests passed with
cargo +1.85.1 test -p layerfs-workspace --lib --no-default-features -j2 -- --test-threads=1. This includes incremental file reuse, sparse/overlap/write rollback, alias/reference behavior, old-root use, exact returned snapshot, rejected publication, partial installation recovery, Busy guards and bounded policies. These are component regressions, not the required independent sampled benchmark proofs.Symbol and caller transfer ledger
Evidence handling
Retained create baseline:
issue46-parent-create100, total 22.142764 s, Exec 15.086029 s, Commit 6.988436 s, reconstructed continuation 5.258958 s. Retained delete observation:issue46-pages-delete100, total 4.991831 s, Exec 3.297524 s, Commit 1.679852 s. Delete predates parent-cache source; they are not a same-source final pair.New checkpoint measurements will be appended with source/product/image/harness identities. Historical raw receipts are immutable. The strict child assessment remains complete
pure_call_sum_ns < 1,000,000,000, separately from the parent's 15-second classifier.Attempt 1 — in-place checkpoint, create-100
Receipt:
benchmark-results/host-store/results/issue47-checkpoint-create100/perf.jsonl; child assessment beside it. Complete original workload, seed 1, 20,000 writes / 104,857,600 bytes, one publication. TARGET_MISS: 19,916,002,292 ns, strictly above the child limit. Create 9,009,625 ns; Exec 14,649,304,542 ns; Commit 5,225,464,125 ns; visibility 72,750 ns; End 32,151,250 ns.Commit content 1,184,619,625 ns; namespace (including checked final-record handoff) 782,133,209 ns; candidate finish 98,398,917 ns; admission 432,169,750 ns; checkpoint 2,723,398,708 ns, including 2,691,903,129 ns spool retirement. Only 5 Commit-scoped snapshot database calls versus the retained create receipt's 108,006. Candidate admission still reads 113,424,403 spilled bytes and delivers 82,669 selected objects in 28 transactions. This demonstrates removal of snapshot rediscovery, not completion of Commit optimization.
Producing identities:
d32305155dc66eb73607ac18b03c26efe396253f6c02842777824a69fa78e599c30ebcb2adcd6f4a04c68247e704191dabbdb64c45e9355db7f4544ac8f3b5ccsha256:7e15e4d30bcb5231a85bab4a75edf8f930505baffb36d6a044338527f6fc618ec3610164231c40bfa3e77c27e5c1526b5e9656c046c9e382d095b0d6c75d1abcf3cb375cbb649463e80f8f06e66c69e9e2f56c67e0417f3a8521c1d33f40397eSource was dirty on documentation checkpoint
8aec76f76when built; the following local implementation commit records that source. No push. Host macOS SQLite, real Linux FUSE, no mounts, 2 CPUs / 2 GiB / no swap / 256 PIDs were validated. Compatible protected create preparation was copied into isolated cache; cache hit, independent closed-byte-copy sample, master unchanged, cleanup PASS. No OOM or swap. Linux observed peak 41,447,424 bytes; host process observed peak 194,330,624 bytes. No independent proof ran.No-go revision: removal of path reconstruction alone is insufficient. Per-file unlink/close remains the dominant checkpoint cost; retaining final coalesced inode records will remove intermediate construction and the 0.782 s record reread validation introduced by this conservative first handoff. Next: bounded ordered final inode deltas and paged deletion accounting, then shared checked admission consumer. Minimal storage-lifetime work is now evidenced as necessary for the remaining 2.692 s retirement cost, but broad Exec ownership/segment redesign remains behind the substantive Commit checkpoint.
yifanxuaaa commentedon Sep 5, 2026
Major Commit handoff checkpoint recorded:
55c950979; follow-up unseen-alias correctness/reporting commit46463287c. Both local, not pushed, oncodex/issue47-commit-first.Second implementation slice — final inode stream and deletion pages
The ordinary localized builder and its selectors are now removed. All ordinary candidates use the existing frontier/content algorithms, including the retained 1 KiB supported domain. The existing dirty-node set now tracks metadata and directory/new-node changes as well as file edits, avoiding an all-materialized-node probe. Remaining spool accounting visits retained descriptors, not the complete live graph.
FrontierInodesnow coalesces final records over the immutable base. Its bounded map spills sorted fixed-width records; reads use pending → spilled (including tombstones) → authenticated base precedence. Existing spilled keys update in place. New-key merges replace ownership only after complete writes, and a regression injects a failed merge before retry. After accounting, each final record is encoded once and one ordered ID stream feeds the existing sorted inode builder. Its supported-domain resource fallback replays final records from the original root. Superseded ordinaryFrontierInodes::flushintermediate tree construction is deleted. The known large-change ceiling is repeated new-key spill merging; it is documented in code, not presented as a universal external sorter.Deletion release now consumes bounded directory pages and batches immutable inode-table/record reads, checking latest reference mutations again before using prefetched records. Lookup batch width separately reserves raw/decoded canonical-page scratch; small policies retain single-key canonical reads. Added a regression with moves, repeated aliases, unseen surviving aliases and several pages under a 16 KiB policy. Nested
deletion_cursor_nsanddeletion_records_nsdistinguish cursor work from reference/record work without double-counting total attribution.PortableMetadataCache::get_by_rootreuses existing exact constructed metadata during final-fact checking.consume_checked_owned_pagenow owns/sorts/authenticates bounded pages, commits checked insertion, and returns metrics to ordinary Workspace and native initialization. Deleted the duplicated native admission transaction wrapper and Workspace per-object outcome accumulation. Native empty-Store/cleanup/final-publication behavior and reconciliation planned admission stay with their real callers. This is code reuse, not removal of the remaining candidate payload spill/readback.Regression discovery and repair: a rejected write can establish editable backing without semantic mutation. Retained-descriptor accounting preserves its PieceTree charge across another file's Commit. A stronger pinned/unseen-alias test then failed with published
abcinstead of editedxbc; construction had skipped every pathless node. Canonical inodes with remaining references now participate even when their materialized path set is empty, preserving the same NodeId and publishing edited bytes to the unseen alias. The regression checks both canonical and live bytes.Validation so far: all 49 Workspace library tests passed before the additional paged-release test; that new test and the revised lookup-width test passed separately. All 20 object-store unit tests passed; explicit native multi-batch publication-failure cleanup and Workspace candidate/admission/publication failure retry integration tests passed. Builds/tests are serial. No independent benchmark proof ran. Next measurement is one original delete-100 sample, seed 1; its old protected preparation was evicted, so supported infrastructure will regenerate it once.
Attempt 2 — ordered/paged Commit, delete-100
Receipt:
benchmark-results/host-store/results/issue47-ordered-delete100/perf.jsonl; strict child assessment alongside. Child TARGET_MISS: 4,061,720,833 ns; the runner's parent 15-second PASS is not child PASS. Original 20,000 unlinks / 233 rmdirs, full traversal/metadata/sync, seed 1, one publication. Create 13,896,459 ns; Exec 3,580,485,958 ns; Commit 460,068,958 ns; visibility 89,250 ns; End 7,180,208 ns.Commit namespace 454,455,959 ns, including cursor 15,309,616 ns and record/reference work 347,007,136 ns. Checkpoint 723,583 ns. Admission is still exactly nine selected objects / 22,318 bytes, with no spill readback. Commit snapshot calls 1,385 versus the retained older delete observation's 27,546; observed Commit 0.460069 s versus 1.679852 s. The retained observation predates the parent-cache revision; this is not a controlled same-source final pair.
Producing identities:
18954962c98ff1fb959a626820f0ad0df575950cc4e4e97aa53e568531b0f12f35c9141e56b236f8af8377fba86d342ec31e83601d3b9fde758979ccb4645a6fsha256:b51dd5afb527c57f55c450f8f3c7ccd9c7dd5badcd842b986d7157c8c0011ec7c3610164231c40bfa3e77c27e5c1526b5e9656c046c9e382d095b0d6c75d1abcd4f9271acc1737f52491808ef1a53cdfdabef33bee5f0657e7b97320cf40660aImplementation commit
55c950979a81958dcd8d948dd5af6e3b177baf69, local and not pushed. The build sidecar reports dirty=true (retained source seal above is authoritative); raw identities are unchanged. Host topology/container bounds were validated, no data mounts, no OOM/swap. Supported infrastructure regenerated the missing deletion preparation, then used an independent closed-copy sample; master unchanged and cleanup PASS. Linux observed peak 9,101,312 bytes. No independent proof ran.This records a stable checkpoint for the major construction, checkpoint and deletion handoffs, not a standalone Commit acceptance gate. No-go revision: remaining delete time is now primarily Exec (3.580 s). Create's measured 2.692 s per-file retirement still requires the documented physical-lifetime dependency: bounded shared append segments referenced directly by existing PieceTree ranges, preserving rollback/read-plan/open-unlinked ownership and measured retirement. Carry forward to coherent Exec after recording this checkpoint; both full targets and final sampled proofs remain outstanding.
A follow-up regression extended unseen-alias coverage to unpinned edited files. It failed with published
abcinstead ofxbc:reclaimremoved the pathless dirty inode before Commit despite remaining unseen file links. Reclaim now retains unpublished non-directory inodes with surviving links; directory removal and clean canonical reclamation keep their existing behavior. Both pinned/unpinned forms and the focused construction regressions pass. This correctness repair follows the measured55c950979source and is not retroactively attributed to its receipt.Coordination boundary
A separately assigned Exec issue/task owns layerfs-fuse and agreed transport changes in another worktree. This task remains Commit and integration owner for Workspace, Store, content, shared spool lifetime and #47 guide/results. No uncommitted FUSE changes exist here. Cross-boundary projection/telemetry hooks require one named patch owner. All performance-sensitive builds/tests/samples use the same host measurement lock; no other task's caches, containers or artifacts are cleaned. Final #47 qualification must use the combined product.
yifanxuaaa commentedon Sep 5, 2026
Shared-spool implementation committed locally as 8cbbb42, not pushed. Full targets remain unmet.
Attempt 3 — ordered Commit, create-100
Original complete create-100 / seed 1 on
46463287c, before shared spools. Child TARGET_MISS:18940567833 ns. This directly measures the second Commit slice; gains are not inferred from delete. Receipt and strict assessment:benchmark-results/host-store/results/issue47-ordered-create100/.Commit detail:
Producing identities:
772a892b09a3d39e7aa82e5610da1bc4d3a82faf3c3beb96b07555cc1f3d967c641afd33103ef39c1453ed8321101799567068eb1fb967836eabedcb4e40872fsha256:ac58ab1abda379bab689f66919c7889e6cd2f9a6ed162f16fb3ceac368dd6f43c3610164231c40bfa3e77c27e5c1526b5e9656c046c9e382d095b0d6c75d1abc28fcd85776a5f85c1b4919cea46374dcb3bc0424c51f1cd41a37fbc7334c2acfComplete host topology and container bounds validated; protected preparation reused through independent closed-copy sample, master unchanged, cleanup PASS, no OOM/swap. No independent proof. The next change targets shared physical segment ownership; snapshot rediscovery is already removed and no further lookup-cache tuning is planned.
Shared physical spool ownership slice
Existing
Piece::Spoolranges now carry anArc<SpoolSegment>and absolute physical offset/length. The compact PieceTree case owns a one-word range handle instead of a private path/descriptor; the existing 100,000-file / 800,000-byte logical compact-piece charge regression remains valid. PieceTree split/replace and canonical FileMutationBatch/rope algorithms remain unchanged. Range binding metadata replaces per-logical-file physical ownership; it is bounded by the live range graph.SpoolSegmentuses a private create-new descriptor and removes its temporary name immediately. Workspace shares append capacity across files and checks physical high-water before writes. Nominal segment capacity is 1 MiB; a larger single write receives a segment large enough for that write within the unchanged payload allowance. There is one storage mechanism. New empty files, inline edits and logical zero ranges need no physical file.edited_nodestracks logical editable state; the physical registry tracks unique segment descriptors.Delayed
ReadPlanand rollback pieces retain segment ownership directly. Short append rollback truncates only the current unpublished physical tail. If tail cleanup itself fails, observed retained bytes are added to segment charge while logical file state remains unchanged. Admission checks the existing logical charge and retained physical segment charge; partially dead shared segments remain charged. Checkpoint drops published ranges and retires unused segments, including on a clean subsequent Commit after a held read finishes. Reconciliation transfers still-held segments and charge into the refreshed Workspace. Physical allocation/peak/error observations remain tied to actual segment lifetime.Retired ordinary production symbols/state: per-node
open_spools,FileData::Edited.spool,spool_file,create_spool,remove_spool_file,remove_spool_if_exists, and zero-basedcontiguous_spool_len. Their callers transfer to segment-owned ranges,edited_nodes, direct descriptor reads andretire_spool_segments. The generic Workspace runtimespooldirectory remains for owned candidate journals/runtime artifacts. Capture eligibility follows the unchanged logical sequential-write checks and accepts physical segment transitions.Validation: 51 Workspace library tests passed together after the main migration; targeted reconciliation-held-charge and failed-tail-cleanup regressions passed afterward. Existing tests still cover dense 100,000 compact ranges, sparse/overlap/zero data, edit limits, exact rollback, capture equivalence, canonical history, alias/reference behavior, partial-install recovery and repeated Commit. A new interleaved two-file regression holds a prepared read across overwrite, sparse changes, short append, unlink, rollover and two Commits, proving correct bytes and segment retirement. The first cleanup-failure test setup incorrectly placed the failed tail in an entirely unreferenced segment, which was correctly reclaimable; the test was corrected to retain a partial tail in the same live segment. No product weakening was made to satisfy it.
Normal benchmark observations now include passive maintained physical allocation, open physical file count and retained segment bytes before and after Commit. The directory walk is explicitly named-file-only; anonymous segment allocation is not inferred from that walk. These observations perform no oracle, namespace scan or independent proof. Performance remains unmeasured for this slice until the next source-bound receipt; no subsecond claim.
The full 12-test SDK file-edit integration suite also passed serially after the spool migration, including candidate/admission/publication failure retry, projection refresh/reopen, aliases, owner composition, stale heads and discard. No independent benchmark proof ran.
Attempt 4 — shared segments, create-100
Original complete create-100 / seed 1 on
8cbbb425b, child TARGET_MISS:16186989166 ns. Receipt and assessment:benchmark-results/host-store/results/issue47-segments-create100/.Measured Commit/storage detail:
Physical segment observations:
{"allocated_bytes": 104914944, "kind": "workspace-physical-spool", "method": "verification_workspace_state", "observation_count": 20202, "observation_errors": 0, "open_physical_files": 101, "peak_bytes": 104914944, "phase": "after-workload-before-commit", "precision": "mutation-event-aggregate-allocation", "retained_segment_bytes": 104857600, "scope": "passive maintained Workspace counters; no independent verification"}{"allocated_bytes": 0, "kind": "workspace-physical-spool", "method": "verification_workspace_state", "observation_count": 20202, "observation_errors": 0, "open_physical_files": 0, "peak_bytes": 104914944, "phase": "after-commit", "precision": "mutation-event-aggregate-allocation", "retained_segment_bytes": 0, "scope": "passive maintained Workspace counters; no independent verification"}Producing identities:
a1cff705acbc35adad92c3b7115ab42dde35a944e6a4396049e68b3e0377149fb3f0333e3ce623aa59c8ce1312d2cb4f52aa55ad45905438ceeb22a5dabac5cbsha256:445fb85c77c344069325af3debd132b8240dd158e37be2f9ad18de1eff200366c3610164231c40bfa3e77c27e5c1526b5e9656c046c9e382d095b0d6c75d1abcf9796fea5d4552c28a77f9499f783a70c1235e686c7e855f356634e23a6ecdcaHost topology/container bounds validated, protected preparation reused with independent closed-copy sample, master unchanged, cleanup PASS, no OOM/swap. No independent proof. No-go revision: physical per-file lifecycle is removed; next target is candidate payload retention/readback while Exec remains separately owned. Full subsecond acceptance remains unmet.
yifanxuaaa commentedon Sep 6, 2026
Closing at the maintainer's request to retire the older combined Exec/Commit optimization track following #49/#53. The final reported create-100 observation was 1.0377 s, accepted by the maintainer but not a mathematical subsecond PASS; retain the original targets and receipts. Further Exec investigation remains on #50, and #51 stays open.
feat(gc): schema, markers and logical removals (layerfs-project#42, #44…