Skip to content

[v0.1.3] Optimize live FUSE Exec in parallel with the Commit migration #48

Closed as not planned

Description

@yifanxuaaa

Current Exec research status — 2026-09-06

Keep this issue OPEN. Retain the measured, sampled-verified Exec research baseline. No further large, evidence-backed product optimization was identified in the latest investigation. Stop speculative metadata grant/page tuning; remaining work is clean delivery, interface/integration review, and validation of the combined product and its performance margin. This is not a closure or release-qualification claim.

Retained mixed-v3 observations

Same research product/source, approved tier-100 mixed-v3 recipes, seed 1, detailed diagnostics OFF; n=1 per operation (min=median=max at the values below).

Operation Exec Commit Complete lifecycle Independent sampled research proof
tiny-bulk-create-100-mixed-v3 479.150250 ms 471.756458 ms 968.009875 ms PASS
tiny-bulk-delete-100-mixed-v3 239.114375 ms 8.249417 ms 258.102210 ms PASS

Both observed Exec values satisfy the working 500 ms objective and both observed complete lifecycles are strictly below 1,000 ms. Create has only 20.849750 ms Exec margin / 31.990125 ms lifecycle margin. These single observations do not establish repeatability, robustness, or acceptance of the separately integrated Commit + Exec source. Do not sum phase results from different tasks or product identities.

The recipe affects 1,000 files / 100 MiB and retains the separate 200-file / 1 MiB witness. The reduced file count is a workload revision, not a product speedup over the historical 20,000-file profile. Specification 66983181a9039181779bf420f6541627afc07245 and benchmark migration 0ac1ebcf43b2cfcd266703b4bb40bfbea8be7e5a were available as committed source and the benchmark-only migration was applied to the isolated Exec tree. No other task's uncommitted product edits were adopted.

Measured product improvements on the unchanged historical recipe

For the same original 20,000-file / 100 MiB create workload, seed 1, host Store + real Linux FUSE, compare the initial source-sealed sample with the latest retained diagnostics-OFF reference. These are observed changes between individual research samples, not statistical estimates or comparisons against mixed-v3.

Metric Initial baseline Latest retained reference Observed reduction Ratio
Exec 16,396.634083 ms 6,400.973000 ms 9,995.661083 ms / 60.96% 2.56× faster
Metadata normalization, included in Exec 11,667.489046 ms 2,629.238876 ms 9,038.250170 ms / 77.47% 4.44× faster
Complete lifecycle 17,892.572331 ms 7,668.768584 ms 10,223.803747 ms / 57.14% 2.33× faster
Host CPU, whole product window 2,763.488042 ms 1,751.980625 ms 1,011.507417 ms / 36.60% Different scope from Exec-only CPU

Normalization is nested in Exec; do not add its saving to the Exec saving. Complete lifecycle and host CPU include other phases, so their differences are not independent Exec-only improvements. Reduction = (baseline − retained) / baseline; ratio = baseline / retained. Each producing sample has n=1, with min=median=max at its reported value.

The progression and concrete retained mechanisms were:

Research state Exec Normalization Change from immediately preceding listed experiment
Initial baseline 16,396.634083 ms 11,667.489046 ms —
Ordered metadata authority/replay and completion/observation handling 7,654.838042 ms 2,851.246626 ms Exec −8,741.796041 ms (53.31%)
NOFLUSH for newly created DIRECT_IO handles 6,684.170958 ms 2,838.483918 ms Exec −970.667084 ms (12.68%); redundant FLUSH callbacks 20,000 → 0, with release and explicit sync retained
Bounded write-buffer allocation 6,589.877750 ms 2,775.100835 ms Exec −94.293208 ms (1.41%); small one-sample delta, not a demonstrated repeatable benefit
Latest retained reference, including subsequent correctness fixes 6,400.973000 ms 2,629.238876 ms Used for the cumulative comparison above; later differences are not assigned to one edit

The retained reference accepts 40,701 metadata updates through 553 bounded replay pages plus 553 standalone grants, preserving ordered host application and required observation/completion drains. The metadata row is a combined implementation slice; it does not isolate batching from its accompanying correctness/completion changes. The NOFLUSH source also includes a lock-lifetime correction that was not exercised by this single-write workload.

An earlier checked sample recorded Exec 6,250.741334 ms, but the headline uses the latest retained 6,400.973000 ms reference, not the fastest historical observation. Later RPC-reduction experiments were rejected and are excluded from these retained improvements.

Producing source seals: initial 1f194de66eb186150174fc9e8e36240cc67a8816d14cc9305f86bc437a324a8f; latest old-profile reference aedb31e31ec017a51fc5d741776b357a5e7a2ec21ca54cdd437767d94f6f4fb6. Raw records: research/issue48-exec/real-fuse/baseline-create100/perf.jsonl and allowance-baseline-create100/perf.jsonl; all stages and identities are in the local comparison.json.

Revised-workload numbers and target margin

The retained mixed-v3 values above are absolute results for a new 1,000-file / 100 MiB recipe, not a percentage product gain from the historical recipe. The product seal is unchanged between the latest historical reference and this v3 baseline.

Operation Exec vs working 500 ms objective Complete lifecycle vs strict 1,000 ms threshold
Mixed-v3 create 479.150250 ms, 20.849750 ms margin (4.17% of threshold) 968.009875 ms, 31.990125 ms margin (3.20%)
Mixed-v3 delete 239.114375 ms, 260.885625 ms margin (52.18%) 258.102210 ms, 741.897790 ms margin (74.19%)

Both exact-source independent sampled research proofs PASS. There is no matched pre-optimization delete measurement on this task's retained v3 product series, so no delete speedup percentage is claimed. Single observations and sampled proofs do not establish reliable margins or integrated-product acceptance.

Validation and evidence limits

  • Exact baseline source has passing independent selected create/delete research proofs, completing in 2.831458 s / 2.568900 s with cleanup. Create samples seven files and 340,992 bytes per view, including beginning/middle/end ranges of the 50 MiB file and selected witness/small/medium data. Delete checks selected target absence and retained witness bytes.
  • Those proofs omit exhaustive namespace/full-file bytes/object census, alias coverage and failure injection. Separate focused checks cover metadata ordering, generation boundaries, alias/unlink observers, concurrent metadata/observations and deferred errors; do not silently attribute that coverage to the sampled proofs. Final proofs of the eventual combined integration source remain outstanding.
  • macOS owns Workspace, spool and SQLite; Docker remains Linux daemon/workload/FUSE on 6.12.76-linuxkit, 2 CPUs / 2 GiB / no swap / 256 PIDs, no data mounts. Both retained samples report unchanged protected masters, cleanup PASS, zero OOM/swap and zero benchmark verifier/reopen/injection in performance. Host resources are reported separately.
  • Research builds/checks/samples used the existing shared nonblocking measurement lock and task-local short bounds; selected proofs used their 45 s work / 59 s hard bound. This latest continuation added no kernel upgrade, permission bypass, TTL extension, thread sweep, payload-cache expansion or memory-filesystem control.

Investigation conclusions

  1. Creation GETATTR target identified on the historical recipe. Of 19,582 creation-interior GETATTRs, all returned directories and 19,351 (98.82%) matched the most recent successful create's parent inode. This strongly supports parent permission revalidation after namespace mutation; it is not a kernel call-stack trace.
  2. TTL expiry is not the v3 opportunity. One bounded v3 trace recorded 1,414 LOOKUPs: 1,138 names without prior replies and 276 prior-positive entries still within advertised validity. There were zero expired positive entries and zero tracker overflows. Linux 6.12.76 explicitly invalidates a directory dentry after a successful mode setter, consistent with the workload's chmod/mtime sequence. The trace records reply-call completion, not proof of kernel acceptance or all lookup flags. Matching kernel source.
  3. Data transfer/final drain does not justify a new runtime. Retained v3 create normalization is 113.211791 ms. Measured host write dispatch is 40.810295 ms, containing 37.694717 ms of spool writes; completion drains are below 1 ms. Nested timings must not be added as independent savings. Time outside callback bodies remains unattributed, not demonstrated avoidable waiting.
  4. Two RPC-reduction experiments were not promoted. Both passed focused checks and reduced requests, but neither demonstrated better elapsed time. No repeat search was run for a favorable result.
Experiment Workload Mechanism result Exec baseline → candidate Disposition
Return next allowance with metadata batch acknowledgement Historical 20,000-file create-100 Standalone grants 553 → 238 (−56.96%) 6,400.973000 → 6,488.188042 ms (+1.36% slower) Archived; reverted
Use existing synchronous setter to acknowledge prior asynchronous mutations and apply the next metadata update Mixed-v3 create-100 Grants/pages 157 each → 20 each (−87.26%) 479.150250 → 498.067708 ms (+3.95% slower) Archived; reverted

The latter candidate's complete lifecycle was 1,069.004749 ms versus 968.009875 ms baseline, missing the strict lifecycle threshold despite the parent's 15-second classifier reporting PASS. Single pairs do not establish repeatable regressions or prove that unrelated phase differences were caused by these changes. They provide no evidence to promote either candidate. The retained source was restored byte-for-byte to the baseline used for performance and sampled proofs; trace instrumentation is archived and normal diagnostics are OFF.

Producing identity and retained artifacts

These are local dirty research artifacts, not delivered clean integration commits:

  • Base revision: 8cbbb425b720d3c5f639c69375e85b197ca2515a, worktree 401f, branch codex/issue48-exec-research-401f.
  • Retained source seal: c09f058ee659e72428c6e9e07a67e20850db165d70dbc1c0115e9c7aa0d94d0b.
  • Product seal: 68cf387dba5749287c0e9c5f950cbe60128084ae153455fa5618edbf97a9859c.
  • Image: layerfs-bench-infra:c09f058ee659e724.
  • Source patch SHA-256: aa517dcb91a04020f6f15fe245591932d7b0c7a34f5a53444eb281a9e2dc0c7c.
  • Create raw receipt SHA-256: d7a88fbbca778ccfb30116819705a2252086c1aef0587cbad6144e7c5edfa0f9.
  • Delete raw receipt SHA-256: fa2f1eae061ce3600cff1dd6f94ec3f45bf5cb6d7e4ae6b94cde3da63172ffc1.

Local reports under research/issue48-exec/real-fuse/: mixed-v3-baseline.md, mixed-v3-baseline-pair.json, mixed-v3-baseline-proofs.json, mixed-v3-next-investigation.md, and setter-ack-comparison.json. Raw results/proofs retain separately named directories there. Source/evidence checkpoint: research/checkpoints/issue48-mixed-v3-baseline-20260906/. Rejected experiments are preserved in issue48-allowance-20260906/ and issue48-setter-ack-20260906/ under that checkpoints directory's parent. These files have not been published by this issue update.

Resume performance exploration only on new evidence: an integration regression, repeated target misses under an agreed measurement plan, or a trace identifying a specific removable cost. No broad transport/namespace/Session rewrite follows from the current evidence. Keep the delivery/integration acceptance items below open until actually completed.


Approved mixed bulk workload v3 — 2026-09-06

This explicit user amendment supersedes earlier requirements to retain 20,000/100,000 affected files for the tier-100/500 bulk create/delete rows. Total affected payload remains unchanged; this is a versioned workload revision, not a product speedup.

Tier Affected files Affected payload Distribution
100 1,000 100 MiB 1 x 50 MiB; 800 x 4 KiB; 199 medium files totaling 46.875 MiB
500 5,000 500 MiB 3 x 100 MiB; 4,000 x 4 KiB; 997 medium files totaling 184.375 MiB

Exact medium sizes: tier100 has194 x246,995B +5 x246,994B; tier500 has936 x193,913B +61 x193,912B. The separate untouched200-file/1MiB witness is unchanged. Populated totals including witness are1,200files/101MiB and5,200files/501MiB. Counts above exclude directories.

Replace the four active high-tier bulk IDs with tiny-bulk-create-100-mixed-v3, tiny-bulk-delete-100-mixed-v3, tiny-bulk-create-500-mixed-v3, and tiny-bulk-delete-500-mixed-v3. Family membership remains20; low-tier compact and individual create/stat/unlink definitions are unchanged. Preserve historical raw receipts and IDs; do not pool, relabel or compare across profiles as a product speedup.

Status: specification and registry/generator/verifier migration are implemented in committed source (66983181a / 0ac1ebcf4). The isolated Exec research tree has applied that benchmark migration and collected the separately identified v3 observations above. Preserve the migrated registration, bulk fixture/expected/apply, bounded all-large-file sampling, input/cache identity and assessment as one contract. New target preparations must not reuse old target trees. Reuse existing bounded content generation, directory shapes and shared init/Commit pipeline; no product case/size policies or large in-memory writes.

#47 and #48 qualification now refers to the revised tier100 IDs once implemented, with each complete lifecycle strictly below1000ms. Approximate300ms Commit remains exploratory, not an extra phase gate or tier500 target. #46 broader20-case scope and15s classifier, host-SQLite/no-data-mount topology, serial iteration,120s/130s diagnostic allowances and final-stage45s/59s proofs remain unchanged. Tier500 remains later work. Do not close any issue based on historical-ID samples or claim the file-count reduction as an optimization gain.

Specification with deterministic path/size mapping and migration checks: docs/roadmap/0.1/0.1.3/tiny-file-churn-mixed-v3.md. Benchmark entrypoint note: benchmark/fs-bench-pro/families/tiny_file_churn/README.md.


Objective

Parallel Exec implementation track for #47, under #46/#39. Optimize the approved tiny-bulk-create-100-mixed-v3 and tiny-bulk-delete-100-mixed-v3 live filesystem workloads through shared LayerFS FUSE/Exec code. Combine the result with the separate Commit track; the parent goal remains each complete lifecycle <1,000 ms.

Commit task: codex://threads/01a07283-4940-7ba1-9b4c-548ddc459e35, worktree /Users/yifanxu/.codex/worktrees/7311/layerfs. This issue gets its own Codex task and separate worktree. Parallel means independent implementation; performance-sensitive executions remain serialized.

Agreed ownership and integration

The Commit owner acknowledged that no uncommitted FUSE/daemon/execution/shared benchmark source changes overlap this track. Stable local Commit changes: 4b40af1c4 (in-place checkpoint) and 55c950979 (ordered inode changes, dirty frontier, paged deletion, shared checked admission), not pushed. The Commit task remains the integration owner and handles the shared spool lifetime work.

Owner Files/responsibilities
Exec task crates/layerfs-fuse/ implementation and tests; its own issue/plan/results
Commit/integration task crates/layerfs-workspace/ checkpoint, construction, namespace state, spool/extent/capture/lifetimes; layerfs-layerstack-store/; layerfs-content/; #47 plan/results
Explicit interface agreement before edits Workspace/projection hooks, daemon/execution transport, shared benchmark diagnostics/runner/schema/build files; one named task edits each shared file

Do not independently change another owner's files, copy implementations into a second engine, cherry-pick uncommitted work, or perform broad formatting across their scope. Exchange small interface requirements and clean commit hashes. The integration owner adopts the stable Exec commits, resolves any deliberate interface changes, then measures the combined product. Do not infer combined speed from separate branch measurements.

Use the same existing absolute measurement lock across worktrees; do not make a worktree-specific lock by changing TMPDIR. Coordinate direct cargo checks as well as runner builds/benchmarks. Leave another task's containers, caches, binaries and evidence untouched; use task-owned sample/output paths and source-sealed image tags.

Research and evidence

Read the latest local research (it may not yet be committed into a new worktree):
/Users/yifanxu/Ephemeral-AI-Lab/layerfs/docs/roadmap/0.1/0.1.3/issue47-exec-root-cause-research.md

Read the latest #47 plan and QUICKSTART, preserving the user policies below. Reinspect current source rather than rerunning the same general audit.

Retained main-checkout observations: create-100 Exec 15.086 s, including normalization 10.175 s and host spool open/write processing 1.930 s; delete-100 Exec 3.298 s. These are implementation costs, not physical floors or future speedup requirements. The newer Commit branch reports delete total 4.061721 s, Exec 3.580486 s, Commit 0.460069 s; preserve its producing identities. Use current same-source measurements for decisions.

Recommended Exec direction

  1. Coherent live metadata/namespace execution. Most normalization runs after files close and currently issues synchronous host chmod/mtime exchanges. Reuse pending state, node reservation, callback draining, pause/invalidation and acquired inode/binding data to execute locally only where the owner has authoritative lifetime, validation, ordering and resource facts. Hand bounded ordered changes to host observers/fsync/Commit. A stale Attr or fire-and-forget command is not a valid acknowledgement.
  2. Reuse acquired directory state. Service enumeration, lookup/getattr and ordered unlink/rmdir through the same coherent view; avoid host enumeration followed by another host resolution of every name. Preserve prescribed recursive POSIX operations, stable directory-cookie behavior, aliases, missing/nonempty/type errors and open-unlinked handles.
  3. Reduce callback amplification with correct coherence. Current mount has one FUSE worker and one-second TTL. Measure local versus remote calls and expiry/invalidation before changing kernel caching. No blind TTL increase, permission bypass, writeback switch or thread-count sweep.
  4. Owned transport and bounded buffering. Existing closed-create batching and vectored framing are already implemented. Remove actual remaining copies/drains only when measured. The Commit task owns per-file-spool replacement; coordinate the data interface rather than building another spool backend here.

Preserve synchronous error semantics: moving actual validated execution to an authorized owner is different from acknowledging an operation before the host might reject it. Shared validation and resource reservations may need hooks supplied by the Commit owner. Do not preserve current latency merely because it is current, but do not weaken behavior to obtain a passing timing.

Iteration and constraints

  • macOS SDK/Workspace backing/SQLite; Linux daemon/workload/real FUSE. Docker-owned SQLite permanently prohibited. No data mounts/volumes/socket. Linux 2 CPUs /2 GiB /no swap /256 PIDs; host CPU uncapped, resources reported separately.
  • Approved mixed-v3 tier-100 bulk cases, seed 1, required bytes/metadata/syscalls/sync/cleanup retained. Original 20,000-file receipts remain historical. This Exec research update does not authorize a tier-500 or all-20 campaign.
  • One selected complete performance sample per iteration; reusable protected preparation and independent writable samples. Diagnostic watchdog120s, outer130s. Retain complete TARGET_MISS outcomes.
  • Add only missing attribution: normalization sort/chmod/time splits; proxy local/remote counts; actual wait/host service; directory acquisition versus cached callbacks; batch fullness and barriers that actually wait. Kernel callbacks are not RPC counts; current aggregate syscall counter omits normalization calls.
  • Focused semantic regressions during implementation. No independent benchmark proof after each sample. Final selected proofs run through the integration owner after combined performance acceptance, with45s work /59s hard end-to-end and explicit sampled coverage.
  • Replan on no-go; preserve rejected evidence and proceed with the next meaningful shared mechanism. No alternative family/size/density optimization engines, unbounded caches, speculative frameworks, or arbitrary percent/throughput requirements. 500 MB/s is not a separate acceptance gate.

Delivery and acceptance

  • Deliver shared Exec improvements with reduced measured remote dependencies, repeated materialization, or other dominant work; preserve all relevant operation/lifetime/error/resource semantics.
  • Both revised tier-100 Exec phases meet the working 500 ms component objective on the delivered source, with explicit evidence limits. The local observations above meet it once per case; this is not sufficient for integration or parent PASS.
  • Existing semantics covered by focused tests: metadata validation, ownership/revocation, host-observer visibility, create/write/close/rename/unlink/recreate order, hardlinks/open-unlinked files, rmdir errors, fsync and capacity/failure behavior.
  • Provide clean local commits and an interface/adoption ledger to the integration owner. No unsolicited push/merge/release or parent closure.
  • Integration owner combines Commit + Exec commits and collects both complete lifecycle <1,000ms results on that combined identity, then bounded final proofs. Record whether this track is delivered, awaiting integration, or independently incomplete; do not fabricate combined PASS.

The final user outcome is both optimizations in one validated product, not two incompatible implementations. Avoid duplicate work with the Commit task; unresolved shared-file changes must have a named owner before implementation.

Activity

  1. yifanxuaaa commented on Sep 6, 2026

    @yifanxuaaa
    ContributorAuthor

    Closing at the maintainer's request to retire the separate pre-migration Exec optimization track following the shared-live-Workspace integration in #49/#53. Remaining Exec investigation is consolidated under existing #50. Preserve research measurements as source-specific historical evidence, not a claim that every candidate was integrated or qualified. #51 also remains open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    v0.1.3LayerFS v0.1.3 checkpoint and delivery

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions