Skip to content

[v0.1.3] Redesign FUSE Exec around one shared live Workspace core #49

Closed
2 of 2 issues completed
Closed

Description

@yifanxuaaa

Latest explicit user instruction — active, supersedes conflicting guidance below: completely withdraw ALL local fuser patches and return to an unmodified upstream dependency. Drop the buffer/thread changes and cache-retrieval additions; migrate LayerFS callers using supported upstream APIs, then remove the vendor directory/Cargo override and obsolete references. Do not relocate patches into a fork or custom protocol implementation. Preserve the shared Workspace architecture and required coherence/concurrency/error behavior. Take the fastest correct route to finish: no new frameworks, speculative extraction or optional optimization campaign; focused changed-seam checks, existing real create/delete qualification and final selected proofs. If a reproduced upstream limitation prevents correctness, report exact evidence promptly rather than retain patches or weaken behavior. Earlier permission to retain a minimal local extension is superseded. Existing targets and evidence requirements remain.

Architecture-first completion; later optimization tracked separately: the user is satisfied with current performance if the migration is completed correctly. Deferred child issue #50 records the read-only comparison with optimized #48 and the remaining Exec hypotheses. It must not expand or block the current migration. Finish architecture/caller adoption, coherence, recovery, resource bounds, cleanup and required qualification. Existing acceptance remains; no added performance gate. Do not reopen optional tuning because #50 exists.

Latest user target revision: preferred create-100 complete lifecycle is 0.9000 s, superseding the earlier 0.7000–0.8000 s ambition and indicative Exec phase budget below. Roughly 0.0500 s above target is acceptable; avoid prolonged minor tuning. Formal same-source create/delete complete lifecycles remain strictly <1.0000 s. The final optional optimization scope and mandatory correctness/final qualification obligations remain unchanged.

Final optional performance scope — 2026-09-06

User direction: this is the last optional optimization pass. Three read-only reviews support conditional targeted FOPEN_NOFLUSH for existing newly created DIRECT_IO handles, followed only if worthwhile by one combined APPEND_CLOSE_RESERVE rollover exchange. Reviewed implementation details and stop rules.

Finish the active concurrent mapping / SDK-edit / Commit coherence correction first. NOFLUSH must preserve the required error contract: current FLUSH can expose failed-owner state, and RELEASE cannot replace close-error delivery. Skip NOFLUSH if that boundary is required. Combined rollover must distinguish successful old append/tail-close acknowledgment from failure to reserve the next window; preserve exact ownership, no replay, bounded charges and existing sequential fallback. No two-window pipeline or new scheduler.

Latest development observation: create-100 Exec 0.5716 s, Commit 0.4046 s, complete 0.9946 s (1,000 files / 100 MiB mixed-v3); 1,000 FLUSH callbacks and 0.1759 s cumulative backing exchange wait. These are opportunities, not guaranteed savings. The earlier 0.9533 s total was a different source, not a repeat. Preferred 0.9000 s is the revised working target; formal same-source create/delete complete lifecycles remain strictly <1.0000 s. No added mandatory headroom gate.

Use changed-seam checks and one unchanged clone-backed create sample per substantive candidate, with existing host SQLite + Docker/FUSE setup and matching source/product receipts. Stop optional tuning after measured keep/reject decisions; roughly 0.0500 s outside the preferred band does not reopen scope. Required caller adoption, concurrency/coherence, recovery, resource bounds, cleanup, same-source delete control and final selected proofs still must finish. A formal MISS or correctness failure must be reported honestly, never labeled terminal PASS. No physical 100-workspace campaign.


Corrected filesystem contract: Commit succeeds with running commands

Multiple commands/processes may remain running against a Workspace while Commit succeeds, and continue afterward on the same mount, cwd and handles. Shell/Exec is a launcher and benchmark surface; command lifetime does not define filesystem quiescence. The earlier instruction to preserve the managed-Exec Busy policy is withdrawn.

Remove has_executions()-based Commit rejection and the command-exit/ordinary-writable-fd-close requirements in quiescence and ordinary owner operations. Preserve command accounting only for actual launcher/resource/teardown duties. Use a filesystem-operation cut: queue conflicting mutations, drain admitted operations, finish required backing acknowledgment, freeze known state, publish/checkpoint the same live nodes, and resume later mutations as new dirty work. A bounded initial mutation pause through publication/install is acceptable; zero-blocking concurrent-generation Commit is not required. Supported calls wait at the cut rather than failing just because Commit is active.

Do not kill processes, unmount, reconstruct the Workspace, or wait for whole commands to finish. Preserve rename/alias/partial-write atomicity, open-unlinked backing, exact publication/install recovery and actual failure/deadline/resource/head-conflict errors. Handle supported writable mappings and kernel dirty pages explicitly; do not remove guards without implementing their consistency responsibility or silently drop supported behavior. End/discard/owner-loss cleanup is distinct from Commit.

Add focused coverage with two live commands/handles across successful Commit, writes before/after the cut, direct non-SDK access, and continued cwd/mount/handle identity. A command can span multiple snapshots; Commit captures filesystem state, not the command's application-level transaction. Create-100 clone-based iteration and the inference-only 100-workspace guideline remain unchanged.


Fixed topology and clone-based fast iteration

Use the current implementation plan and QUICKSTART. Development performance and end-to-end verification must exercise macOS host SQLite/SDK coordinator/physical spool/canonical construction and Docker Linux daemon/workload/real FUSE. The new live operation core belongs with the Linux daemon by design; it does not move SQLite or benchmark coordination into Docker. The benchmark instructions now state this boundary explicitly.

Normal reset is --setup clone. Use families/tiny_file_churn/setup.sh to prepare or reuse the protected compatible host master, then perf.sh --case tiny-bulk-create-100-mixed-v3 --seed 1 --setup clone --perf-fast for one complete sample. Both accept explicit host-store topology and sealed image. Setup is preparation-only; performance acquires compatible preparation automatically, so do not redundantly run setup before every sample. The runner's clone method is a checked closed-quiescent byte copy into a new disposable sample Store, not APFS/reflink or copying a live SQLite database. The runner owns sample container/FUSE startup and cleanup.

Retain incremental Cargo/Docker caches. Build matching host/image artifacts after relevant product changes, preserve compatibility-based fixture reuse, and never bypass seal/master-isolation checks. Do not routinely use fresh preparation, reset Docker, prune images/caches, delete protected masters, or regenerate the namespace. The CLI option is fresh, not refresh; it is reserved for an explicit setup/invalidation investigation, not ordinary iteration. Input/schema incompatibility must acquire the appropriate fixture through the runner.

Cycle: one hypothesis → one focused code change/check → matching cached builds → one clone-backed create-100 sample → inspect complete phases/resources/cleanup → retain or revise. No broad family/seed multiplication. Independent sampled proofs remain final-only with exact performance identities and45-second work/59-second hard limits. Physical100-workspace qualification stays inference-only/deferred. This update launches no benchmark.


Current implementation direction and stronger working objective

Implement the shared FUSE rewrite directly on current main with the finalized Commit implementation. The user is not requesting migration/cherry-picking of the Exec patch from the research worktree; do not add that as a prerequisite or handoff task. Use the research as evidence for successful mechanisms, necessary safety checks and rejected hypotheses.

The preferred create-100 working ambition is approximately 0.7000–0.8000 s complete lifecycle, with an indicative 0.3000–0.4000 s Exec budget while preserving current Commit efficiency. These are planning objectives, not measured predictions or new hard phase gates. The earlier approximately 0.5000 s Exec aim is a reference checkpoint, not the ambition of the rewrite. The research result used older Commit processing (0.4718 s versus finalized reference 0.3587 s); its 0.9680 s lifecycle should not be treated as the desired endpoint. Do not add independent phase measurements and claim an achieved combined result.

The formal parent requirement remains strictly below 1.0000 s per tier100 create/delete lifecycle. A roughly 0.0500 s difference from the preferred band is not a reason for prolonged minor tuning when the structural outcome is sound. Preserve full accounting, correctness, existing lower-volume iteration and the inference-only 100-workspace guideline. This amendment supersedes any older instruction below to migrate the research Exec patch first.


Final implementation plan — create-100 first

Three read-only audits are consolidated in the implementation plan, published on main at e1621ab0e55ea326449e4262e332ee890d696dc0. It defines the final ownership/optimization mental model, before/after diagrams, exact C1–C8 file/type/method changes and deletions, confidence/exploration captions for every component, and P0–P6 execution order. Use this plan for implementation; its precise observer/resource/scope rules supersede broader older proposal wording below.

Primary case confirmed by the user: tiny-bulk-create-100-mixed-v3 creates 1,000 files totaling 100 MiB, including one 50 MiB file, plus the unchanged separate 200-file/1 MiB witness. No migration to 2,000 files is requested.

Sequence: minimum portable range/state seam using existing algorithms → small fuser/runtime entry → one complete local create/write/metadata/SDK/Commit/continue path → create-100 fast iteration → remaining supported callers and runtime consolidation → deletion/control/final selected proof evidence. Do not extract the entire system or implement a generic effect/scheduler framework before the first coherent slice. Do not ship split live authority.

The decisive boundaries are one live owner; bounded acquisition outside state locks with revalidation; existing host backing and canonical Commit pipeline; cheap owner metadata observations for current session/diff fields; exact checkpoint recovery. Keep one small receive loop per mount and reuse the decoder. Correct actual thread/buffer multipliers with a narrow maintained dependency/runtime change; a custom FUSE multiplexer is not required solely to preserve an arbitrary memory estimate.

Implementation iteration uses one substantive change, focused changed-seam checks, and one serial create-100 performance sample with exact source/image/recipe. Inspect full Exec/Commit/End and resource/cleanup costs. Parent tier100 lifecycle target remains strictly below 1.0000 s for create and delete; no tighter isolated phase target. Independent proofs stay at the final stable candidate, 45-second work/59-second hard per proof. No all-family/100-workspace campaign is introduced.

100 workspaces remains a guideline inferred from existing lower-volume evidence and source accounting, not a current physical verification gate. The resource review records known costs and provisional budgets. The 109-item map now traces every obligation to components and steps, without turning them into 109 per-iteration gates or marking them implemented. This planning update changes documentation only and runs no benchmark.


Current capacity guidance — 100 workspaces is a design reference

The user cannot physically exercise 100 workspaces now or in the near term. Keep 100 independent agent workspaces as a design guideline, and infer feasibility from existing lower-volume benchmark evidence plus source-based resource accounting. A physical 100-workspace run is not a current implementation, optimization or completion gate. This amendment supersedes older mandatory 100-mount-test language in this issue and pinned documents.

Record measured lower-volume results, code-derived bounds, workload assumptions and projected totals separately. Reuse existing receipts with their original source/product/topology identities. CPU estimates must include host and container scopes; memory estimates must separate shared baseline, true per-mount state, active buffers and retained dirty/captured data. Do not multiply whole-process peaks or assume linear throughput/tail latency. Keep nonlinear file-count, byte-volume, cold-state and contention effects explicit. Source-confirmed process/thread, receive-buffer, capture and descriptor problems remain design findings to address.

Physical 100-workspace capacity remains inferred/unverified. Future high-concurrency scenarios are retained as reference qualification only; do not launch or schedule them now or block agreed lower-volume work waiting for them. This does not weaken its existing correctness or performance obligations.


Outcome and issue scope

Parent: #47, under #46/#39. This issue now owns the FUSE Exec redesign. At the user's request it replaces #49's former shared-producer simplification scope, which is delivered and retained in the historical implementation/adoption ledger. Earlier comments remain historical evidence; they are not the current work order. Do not redo the completed Commit/producer refactor.

Build a substantially faster, simpler FUSE execution model using one authoritative live Workspace operation core on the execution side, bounded host backing, and the existing shared Commit construction/persistence pipeline. Benefits must come from shared operation semantics and reusable code, not benchmark/application/size/density-specific routing.

Required starting code: current merged main

  • Finalized Commit branch: codex/issue47-commit-first at d6cc03f500a9aaf3f629fd5a76e9fc26e95ae00f.
  • Published redesign starting commit on main: 27761a2f4494df7b9660c9a951bb402e764a2d16; branch codex/issue49-fuse-redesign-baseline retains this exact checkpoint.
  • Fetch remote main before starting implementation and record the actual starting SHA. Preserve this completed Commit ancestry; if main advances, review the intervening changes rather than silently using a stale checkout.
  • The integration is a fast-forward plus documentation and small CI corrections: Rust 1.96.0 formatting in two benchmark files, scoped annotations preserving existing producer/frontier interfaces and backing layout, and equivalent expression simplifications in producer/fsync code. Existing operation, ordering, error and timer contracts are preserved. Historical measurements retain their original source/product seals; these corrections are not a new performance qualification.
  • The separate [v0.1.3] Optimize live FUSE Exec in parallel with the Commit migration #48 research task codex://threads/01a072c8-2e80-7982-966b-df26da9fdeb5 supplies measured optimization lessons. Its unmerged research source is not the starting product and must not overwrite newer Workspace/Store code. Preserve the lessons and safety obligations in the new implementation; exclude rejected experiments. Research patch migration is not a prerequisite.

Normative design documents

  1. Rewrite specification: architecture, ownership, protocols, limits, deletion plan and experiments.
  2. Full redesign requirements checklist.
  3. All 109 requirements mapped to design sections and required tests.

These documents are specifications, not implementation PASS. All 109 requirements remain obligations. Follow the documents on the actual implementation branch; the links above pin the approved starting version. Any change to the design must update its checklist mapping and evidence.

Proposed shared flow

Shell / editor / installer / SDK live operation
                       |
             Thin platform adapter
                       |
          One shared live Workspace core
          bindings / attrs / handles / pieces
             |                       |
       authoritative local      missing immutable state
       validation + mutation    -> bounded acquisition
             |
       bounded owned bytes + resolved change facts
             |
       existing host spool / acknowledged recovery prefix
             |
       stable Commit cut, known changed inputs
             |
       existing shared producer / canonical builders
       checked Store admission / SQLite / publication
             |
       checkpoint the same live nodes and handles

One writable owner per Workspace lifetime; multiple processes and at least 100 simultaneously mounted independent live workspaces per daemon. Host SDK live reads/edits invoke the same owner. Immutable snapshot reads stay Store-local. Host retained recovery facts do not form a second mutable filesystem. The first version retains host spool placement and does not introduce another disk-backing engine.

Required implementation boundaries

  • Extract/reuse existing Workspace inode, binding, validation, PieceTree/COW and range-ownership algorithms; native-host and daemon callers must share real code.
  • Keep mount setup, credentials, request/reply conversion, cache notifications and platform capabilities in thin adapters. Linux first; no unused macFUSE/Windows scaffolding or claim of identical platform semantics.
  • Execute operations locally when the owner has sufficient authoritative state; acquire cold inode/directory/content state lazily in bounded pages. No whole-workspace materialization or mandatory loopback RPC for local-host placement.
  • Reserve capacity before mutation. Success means the owner actually validated/applied the operation; no optimistic success pending a remote semantic decision. Retain bytes/facts until acknowledged backing ownership exists.
  • Acknowledgments cover complete operation groups, never half a rename or metadata referencing unavailable bytes. Preserve exact partial writes, failed-transfer ownership and open-reader backing lifetime.
  • SDK observers/edits, kernel cache invalidation, external shell access and Commit share explicit visibility and exclusion rules. Do not infer absence of writers solely from managed Exec tracking.
  • At Commit, order filesystem mutations and drain admitted work for a consistent cut while command processes remain alive; no blanket command-level Busy gate. Reuse [v0.1.3] Redesign FUSE Exec around one shared live Workspace core #49's delivered producer/finality machinery, expected-head publication and exact checkpoint-install retry; preserve NodeIds, handles, aliases, old roots and sparse/zero ranges. No reconstructed Workspace or whole-namespace rediscovery.
  • No state locks during socket waits, queue-capacity waits, drains or joins. Use defined multi-object ordering and bounded fair scheduling; a stalled workspace must not monopolize unrelated mounts. Cancellation/release/drain has reserved capacity.
  • CPU, requests, live state, buffers, caches, retained backing and handles have numeric per-workspace and aggregate bounds. The spec requires 100 simultaneously mounted independent live workspaces with concurrent clients, sharing 2 execution workers, 128 aggregate ordinary request slots and a 256 MiB execution managed-memory envelope. Use 2 shared nonblocking transport workers per side, not a blocking transport thread per mount. Per-mount progress reserves are 256 KiB transfer and 64 KiB operation/result space, charged inside aggregate budgets; full per-workspace maxima are not preallocated. Validate compatibility and actual allocation accounting before adoption; managed bytes are not whole-process RSS.
  • Host SQLite only; no Docker SQLite, host-data mounts, new core privileges or mandatory kernel accelerators. Preserve container 2 CPUs / 2 GiB / no swap / 256 PIDs; report host resources separately.
  • Preserve applicable [v0.1.3] Optimize live FUSE Exec in parallel with the Commit migration #48 NOFLUSH, allocation, observation and error/lifetime improvements. No blind TTL/writeback changes. Fewer messages alone do not prove improvement.
  • Remove superseded proxy mutation paths, authority negotiations, redundant drains and duplicate host live mutation entrypoints after their real responsibilities/callers transfer. No permanent old/new policy engine.

Source-bound baseline and target

The finalized Commit campaign records 138/138 performance cases and 34/34 selected independent proofs PASS for its requested fast scope, with 136 new performance samples and 2 matching reused samples. Full case-specific tables, input/receipt hashes, resource measurements and proof omissions are in #46 results and #46 provenance, with the campaign ledger retained in the repository.

Mixed-v3 case Exec (s) Commit (s) Complete lifecycle (s)
Tier100 create 0.9172 0.3587 1.2914
Tier100 delete 0.2684 0.0148 0.2980
Tier500 create 4.1284 1.7790 5.9276
Tier500 delete 0.9103 0.0541 0.9771

Display seconds to four decimal places; pass/fail uses unrounded receipts. These are single observations with fixed profiles, not a statistical distribution. Source seal 4ce40e2c3c3573902e3820d8e16c5f1be793f42dbeb14e4bb72ed4274557fc1f; product 3fb2f18f1c636b8c3aa8b2a901bb3ca2a56cb075e674b6f520e8c685ca2b1053; product build commit 6de381837d1c22e5eb21dfaf446f85c9c30ea6ec.

Parent #47's strict tier100 create/delete complete-lifecycle target remains <1.0000 s each. Create is still a miss; its strict final proof pair remains deferred. The 15-second family PASS does not close that gap. No new tighter isolated Exec/Commit target is introduced. Preserve Commit gains and seek a substantive Exec/lifecycle improvement; stop minor tuning once the accepted outcome is achieved. Do not infer linear scaling between counts, bytes or tiers.

The #48 research checkpoint separately measured create Exec 0.4792 s / lifecycle 0.9680 s and delete Exec 0.2391 s / lifecycle 0.2581 s. Those are a different product, not phases to add to the finalized Commit numbers. Its grant/replay reduction experiment did not improve elapsed time. Reinspect its retained source/evidence and compare an actually integrated product before claiming combined gains.

Implementation and fast iteration

  1. Start from merged current main; inventory actual shared callers, acknowledgment/durability guarantees, cache behavior, wire bounds and simultaneous resource ownership. Preserve old source/evidence identities.
  2. Learn from retained [v0.1.3] Optimize live FUSE Exec in parallel with the Commit migration #48 mechanisms and evidence while implementing directly on current main; no research Exec patch migration is required. Freeze unchanged mixed-v3 recipe, resource settings and synchronization duties.
  3. Implement a vertical slice of locally owned metadata/binding operations through the shared core, with bounded acknowledged host backing and SDK visibility. Keep kernel flags/TTL unchanged initially.
  4. Run focused semantics and one selected serial real-FUSE performance sample. Attribute callback/queue/local work/host dependency/reply/application resumption, plus complete Exec/Commit/End and resource/cleanup costs. Reduced RPC counts or shifted work are not a win.
  5. Replan when a hypothesis fails; add neither a generic protocol framework nor benchmark-specific exceptions to rescue it. Transfer the remaining supported operations through the same core, and exercise two-workspace stall/full-queue/control progress early.
  6. Remove replaced paths after caller/semantic coverage. Final independent sampled proofs run after performance, within 45-second work / 59-second hard end-to-end limits. Preserve the shared measurement lock and other tasks' artifacts; no exhaustive proof campaign per iteration or favorable-repeat hunting.

Completion evidence

  • All checklist requirements have implemented behavior, source-bound evidence and explicit platform/support limits; unimplemented obligations are not checked off by prose.
  • One shared live operation core has real native-host/daemon callers; existing Commit/canonical mechanisms are reused and superseded paths are removed with a caller/deletion ledger.
  • Supported ordinary filesystem and SDK operations, aliases/open-unlinked handles, old snapshots, cache invalidation and stable Commit continuation pass focused checks.
  • Partial writes, quota/transfer/daemon/host failures, lost replies, publication/install retry, cancellation and teardown preserve their stated acknowledgment and recovery contracts.
  • Per-workspace/aggregate CPU and memory/backing limits have explicit ownership/enforcement. Review thread/buffer/descriptor accounting and existing lower-volume evidence; document inference and unknowns for 100 agent workspaces. Physical 100-mount testing is deferred and is not a current completion gate; admission uses actual deployment limits rather than an artificial 101st-mount requirement.
  • Matched integrated measurements demonstrate useful Exec and complete-lifecycle improvement without hidden resource, cleanup or overload regression; parent strict tier100 result is reported honestly.
  • Final selected bounded proofs and complete source/product/workload/receipt/resource identities are recorded. Replan unresolved no-go results rather than declaring terminal PASS from partial evidence.

Keep #47/#46/#39 open unless their own acceptance is independently satisfied. Rewriting this issue and publishing the starting code do not implement the FUSE redesign or authorize a parent closure.

Integration validation on the published starting revision

Linux CI PASS at 27761a2f4494df7b9660c9a951bb402e764a2d16: Rust 1.96.0 formatting, the complete native workspace test workflow on Rust 1.85.1, and full-workspace Clippy with warnings denied. Local full-workspace Clippy and the focused producer task-coverage/failure-drain regression also pass. Initial formatting/Clippy failures were corrected in the integration follow-ups; the final run is green. No new benchmark performance claim or FUSE redesign completion is implied.

Capacity requirement amendment: 100 workspaces

The user requires 100 simultaneously mounted live workspaces, superseding the original 8-mount design default. This is mounted capacity with concurrent clients; active CPU work is fairly scheduled within shared limits, not 100 dedicated worker pools. Memory remains bounded and backing is allocated on demand. The host retained-backing ceiling becomes 100 GiB aggregate with the existing 1 GiB per-workspace maximum; this is a limit, not an eager allocation or a claim that every host has that free disk capacity.

Retain up to one FUSE receive thread per mount initially, but use shared nonblocking data/control transport. Inventory all helper threads, stack footprints, sockets and child processes. Preserve the matched deployment's 256-PID limit: 100 mounts do not imply support for 100 arbitrary process trees within that limit. The design should support bounded progress at the 100-client reference scale. Existing lower-volume evidence can inform that projection, with actual scale and assumptions disclosed; it does not establish a measured 100-client result. Physical idle and saturated-load measurements at that scale are deferred, not required for current completion.

The 100-workspace specification amendment is published on main at ba57b4917aa0c160d7db65a37ccc7102a3d77fe5. The earlier code baseline and its CI evidence remain identified above; this documentation-only amendment does not change implementation capacity.

Activity

  1. yifanxuaaa commented on Sep 5, 2026

    @yifanxuaaa
    ContributorAuthor

    Implemented the reviewed shared-producer scope in the integration worktree /Users/yifanxu/.codex/worktrees/7311/layerfs.

    Local commits (not pushed):

    • cc6d3b0ef: reviewed plan adoption.
    • 2570c0d81: shared checked full-file completion facts and native regular-file consolidation.
    • da40c431c: shared task execution, bounded ordered Workspace results and caller adoption.
    • 6de381837: implementation/adoption/deletion/test ledger.

    run_finalized_output now owns claiming, cancellation, per-task execution, result sealing, writer completion, error/panic handling and drain/join. Native direct and private fallback callers and Workspace Commit/Preview use it. Workspace partitions eligible inodes once, retains generation/source ownership, writes bounded worker-local result journals, and restores order using fixed ordinal slots containing worker/journal offsets. The single-result extraction and duplicated whole-worker loops are removed. Empty file-task sets are valid.

    The existing rope builder now exposes completed root/length/counter facts while retaining its public wrapper. Native traversal and direct-file import share one regular-file body using already-acquired metadata; Workspace full rebuilds share checked completion while remaining private until success. Incremental/captured/no-op paths retain their semantics. Namespace/reference finalizers, native cleanup and Workspace stage/publication/install recovery remain caller-owned.

    Defaults remain native's existing up-to-eight/task-limited allowance and one production Commit worker. Aggregate journal/candidate/index allowances divide across workers; queue/slab limits remain. Workspace result descriptors are on disk, not an unbounded completion map. Multiworker spill-peak aggregation is a conservative bound, not an exact concurrent peak.

    Verification under the shared host lock:

    • Store suite: 51 passed, one existing large-spill test ignored; additional actual-spill direct/private completion and mid-read-failure check passed.
    • Driver: serial/four-worker coverage, reverse completion, zero tasks, producer/consumer/result-finish/writer-finish failures and injected panics all drain/join.
    • Final Workspace suite: 58 passed, including alias/capture-once, capture reuse despite deliberately corrupted backing, reverse result journals, incremental/sparse behavior, preview privacy, low budgets and existing recovery/lifetime tests.
    • File-edit integration: 12 passed. Reconciliation integration: 2 passed.
    • Targeted core/native canonical checks and final SDK compilation passed.

    Full ledger: docs/roadmap/0.1/0.1.3/issue49-shared-producers-plan.md, implementation/adoption section. Worktree is clean. No performance sample, worker sweep, benchmark-definition change, independent proof or unpublished #48 adoption was added. Earlier timings remain bound to their earlier source identities; code reuse is not a measured performance improvement. #47/#46/#39 and combined Exec/performance/proof obligations remain open. No issue is closed by this update.

  2. changed the title [-][v0.1.3] Simplify shared file producers with caller-supplied concurrency[/-] [+][v0.1.3] Redesign FUSE Exec around one shared live Workspace core[/+] on Sep 5, 2026
  3. yifanxuaaa commented on Sep 6, 2026

    @yifanxuaaa
    ContributorAuthor

    The restart has reached the first connected real-FUSE development sample. Work continues on codex/issue49-live-integration; this is not terminal PASS and is not adopted on main.

    Implemented startup into one live owner, host immutable/HostSpool backing, frozen inputs into the existing Commit builders, streamed installation into the same live nodes, and owner generation queries. The native/real-TCP focused test retains one open handle and NodeId across two Commits; existing exact checkpoint/reference regressions pass. Linux builds and the ordinary create-100 path run successfully.

    One unchanged tiny-bulk-create-100-mixed-v3, seed1, --setup clone sample per iteration:

    Source seal prefix Exec (s) Commit (s) Complete (s) Strict result
    818d350ef7b803e8 1.2028 0.8335 2.0520 TARGET_MISS
    1c67db448b4a851b 0.8117 0.3799 1.2087 TARGET_MISS
    efcc2fb86601678b 0.9150 0.3994 1.3334 TARGET_MISS
    bbe0fb1fec04b642 0.8857 0.3288 1.2475 TARGET_MISS

    The first sample created the compatible protected master in the new checkout; subsequent samples reused it. All used checked closed-quiescent clone reset, unchanged 1,000-file/100MiB workload and normalization/root sync, the required host Store/backing + Docker/FUSE topology and resource limits, and successful master-isolation/cleanup checks. No final independent proof, delete control or other-tier sweep has run. Raw receipts remain in the new checkout's benchmark-results/host-store/results/; full source/image identities and artifact paths are in the implementation ledger.

    Paging frozen/checkpoint records removed the large added Commit transfer cost. Reservation reuse has not demonstrated an elapsed gain; I am not crediting it with a speedup. Added actual attribution shows 0.4470s cumulative backing exchange wait, 0.0073s physical queue time, 0.0623s host handling, 0.0301s write dispatch and 0.0027s prepare/apply in the last sample. Exchange time contains the host spans; these are not additive phases. The next focused correction sends frame prefix and existing payload together via vectored I/O instead of separate TCP_NODELAY writes.

    Remaining required work includes queued callback/kernel-mapping coherence across running-command Commit, supported namespace and remote SDK/reconciliation transfers, daemon/helper consolidation, complete resource/recovery semantics, obsolete caller removal, same-source strict create/delete qualification and final bounded sampled proofs. The development slice is explicitly recorded as incomplete; no parent issue is closed.

  4. yifanxuaaa commented on Sep 6, 2026

    @yifanxuaaa
    ContributorAuthor

    Implemented the next live-owner integration on codex/issue49-live-integration through 51d82265a:

    • Shared link/unlink/rename, immutable directory acquisition, stable directory cookies, and open-directory pins now drive the connected FUSE path.
    • SDK range edits prepare/apply in the live owner; streamed batches retain the result rather than all overwritten input.
    • Metadata observation uses a separate bounded connection and avoids the host construction lock; owner head/generation are observed coherently.
    • The daemon mount route now owns FUSE directly on its shared runtime. Two mounted Workspaces run without helper processes; ending one preserves the other's running mappings. Lifecycle frames have independent bounded admission and event-driven shutdown.

    Two mounted cache-invalidation deadlocks were reproduced and retained in the ledger. Corrections admit fault READs during laundering and release post-edit fault replies onto the installed view before the second notification. Latest source-matched managed attachment/disconnect/remount cleanup, two-Workspace pwrite/mmap/SDK-edit continuation, and Linux component checks pass. General kernel internal-writeback drain coverage remains open.

    One unchanged create100 mixed-v3/seed1/clone sample: 953,317,710ns strict timing PASS (Exec571,754,250ns; Commit363,790,334ns; End7,047,042ns). Receipt run-0acf47533e78, source df9b9117da737a8dbc6548b86952a0566cfde1c927d55c0243599d25249222a4, image sha256:43e9b3ef2685c08b02922686b2b3022e8b7a9978050594901da6744700a9cdec. Master unchanged and cleanup PASS; container2CPUs/2GiB/no swap/256PIDs/no data mounts.

    Development result, not terminal PASS. Native/direct placement, reconciliation, full state/descriptor accounting, kernel drain audit, exact failure/retry behavior and obsolete-path retirement remain. Same-source delete control and final bounded independent proofs are still pending. Continuing implementation; no parent closure or final qualification claim.

  5. yifanxuaaa commented on Sep 6, 2026

    @yifanxuaaa
    ContributorAuthor

    Upstream-only checkpoint on codex/issue49-live-integration: effee4e9b withdraws the entire fuser fork; b3c5e75e3 adds failure diagnostics. Official crates.io fuser 0.18.0 is restored (verified checksum b82b6597d216503555ead6b358f341ef748869bf5c6fbae6a0cb9dd231baecfd). Vendor tree, root patch/exclude, all retrieve/probe extensions and vendor-only Docker checks are removed. Shared LiveOwner/native/direct/daemon integration remains. No replacement fork or custom protocol.

    Direct/TCP backing/checkpoint tests pass. Linux component checks pass: five daemon tests, native host-fuse compilation, owned reply completion, two live-runtime tests. Existing obsolete native-helper warnings remain.

    Correctness remains unresolved. On upstream source fe636187dda7364064954f9125e54deea20800a8600af51c3f19ee8bbcd4d483, ordinary_writes_queue_during_mapped_commit failed in 2.20 s after successful SDK edits and concurrent Commit: the held MAP_SHARED client failed p[0]=='A' && p[4095]=='B' && p[777]=='S'. Cleanup passed. Receipt: benchmark-results/issue49/live-cut-1788693308810961000.

    The supported-API attempt uses inval_inode + syncfs, admits cached write-through callbacks during invalidation, then invalidates after releasing the SDK cut so pending faults can finish. Successful invalidation does not establish an empty cache. This is evidence against the current synchronization attempt, not proof that every upstream solution is impossible.

    The existing check now prints mapped/pread values at failure and attempts a post-failure Commit snapshot. Three bounded diagnostic attempts all passed (2.33/2.34/2.32 s) without a product behavior change, so failing byte values and stale-cache versus persisted-lost-update classification remain unavailable. Receipts: live-cut-1788693593455030000, live-cut-1788693612716076000, live-cut-1788693632415837000. No favorable pass replaces the failed run. Controller ordering review found the shared lifecycle mutex held and the go marker issued only after SDK edit and concurrent Commit returned.

    No final PASS, no upstream performance claim, and no final create/delete or independent proof runs. Deferred optimization #50 remains deferred. Full chronology and evidence identifiers are in docs/roadmap/0.1/0.1.3/issue49-live-workspace-ledger.md.

  6. yifanxuaaa commented on Sep 6, 2026

    @yifanxuaaa
    ContributorAuthor

    Follow-up correction: e6d66d676 fixes the observed SDK/mapping lost-update path using unmodified upstream fuser.

    The prior stop was premature. A temporary 10 ms widening of the invalidation-to-cut window produced owner traces showing a full-page WRITE with byte 777 = 0 immediately after the SDK apply for each mapped file (live-cut-1788693963864072000). The post-edit invalidation could launder the old dirty cached page back over the SDK replacement. This identifies a LayerFS synchronization defect, not an upstream impossibility.

    The fix sends the affected edited bytes through public Notifier::store before existing invalidation. It preserves unrelated ranges, streams changed data in at most 1 MiB buffers, reserves buffer memory before applying the edit, and fails the owner if cache synchronization fails. It also fixes the diagnostic marker check to match exact output lines rather than the substring after inside an error message. Temporary delay and tracing are removed.

    Validation:

    • Widened-race test with the correction: PASS, 2.58 s (live-cut-1788694261492203000).
    • Final implementation ordinary pwrite + mappings + SDK/concurrent Commit: PASS, 2.31 s (live-cut-1788694499557925000).
    • Final implementation mapping-only version: PASS, 2.32 s (live-cut-1788694512237047000).
    • Direct/TCP host backing/checkpoint tests: 2 PASS.
    • Linux native host-FUSE compile, 5 daemon tests, owned reply test, 2 runtime checks: PASS (cache-store-linux-check-1788694550757377000). Existing obsolete native-helper warnings remain.
    • All owned test containers cleaned up.

    Tested source af3a4790d8d11007f1df8bee2e0f29b1747e75044a3b40eaace02b2d0f9366e6; image layerfs-bench-infra:af3a4790d8d11007. The commit adds documentation to the tested implementation. The fork remains completely removed. These focused correction results do not close the broader remaining #49 transfer/resource/recovery work or final create/delete qualification.

  7. yifanxuaaa commented on Sep 6, 2026

    @yifanxuaaa
    ContributorAuthor

    Published and pushed 3f884ccad84e5d2745225e10509f0c2497cde5b2 on codex/issue49-live-integration. Committed full report.

    Closing only #49, as explicitly requested by the user, after this full tiny-churn benchmark and selected verification run. The 15-second family result and the separate strict subsecond miss are reported separately below; this closure does not assert completion of all broader unverified redesign obligations. #46 and other issues remain unchanged.

    Issue 49 final requested tiny-churn benchmark

    20/20 performance cases PASS and 7/7 selected independent proofs PASS for the requested benchmark scope. Collected on 2026-09-06. One seed-1 observation per registered case: 19 new samples and the exact matching create-100 result reused unchanged. The implementation and harness were held fixed throughout collection.

    At the user's explicit request, this report is the closure handoff for #49 only. It does not close #46, #47, #50, #51 or other issues. The separate strict tier100 create target still misses; this report does not assert that every broader redesign/checklist, recovery or scale obligation has been independently qualified. The implementation history and remaining limits are retained in the live Workspace ledger.

    Per-case timing

    Seconds to four decimal places; status uses unrounded nanoseconds. Total is the declared complete-lifecycle pure_call_sum_ns, including Create + Exec + Commit + visibility + End. Each row is one observation, not a median distribution or percentile estimate. The family threshold is 15 seconds.

    Case Create Exec Commit Visibility End Total Family
    tiny-create-1-compact-v2 0.0108 0.0068 0.0034 0.0001 0.0027 0.0238 PASS
    tiny-create-10-compact-v2 0.0089 0.0179 0.0050 0.0001 0.0025 0.0343 PASS
    tiny-create-100 0.0104 0.0646 0.0594 0.0001 0.0034 0.1378 PASS
    tiny-create-500 0.0078 0.3030 0.1050 0.0001 0.0040 0.4199 PASS
    tiny-stat-1-compact-v2 0.0075 0.0053 0.0015 0.0001 0.0027 0.0171 PASS
    tiny-stat-10-compact-v2 0.0079 0.0124 0.0015 0.0001 0.0021 0.0240 PASS
    tiny-stat-100 0.0077 0.0646 0.0016 0.0001 0.0021 0.0761 PASS
    tiny-stat-500 0.0088 0.2613 0.0015 0.0001 0.0027 0.2743 PASS
    tiny-unlink-1-compact-v2 0.0084 0.0069 0.0032 0.0001 0.0030 0.0215 PASS
    tiny-unlink-10-compact-v2 0.0077 0.0182 0.0043 0.0001 0.0029 0.0332 PASS
    tiny-unlink-100 0.0108 0.0785 0.0474 0.0001 0.0034 0.1402 PASS
    tiny-unlink-500 0.0096 0.3205 0.0816 0.0001 0.0034 0.4152 PASS
    tiny-bulk-create-1-compact-v2 0.0094 0.0685 0.0122 0.0001 0.0030 0.0931 PASS
    tiny-bulk-create-10-compact-v2 0.0090 0.2482 0.0511 0.0001 0.0035 0.3119 PASS
    tiny-bulk-create-100-mixed-v3 0.0106 0.6402 0.4133 0.0001 0.0055 1.0697 PASS
    tiny-bulk-create-500-mixed-v3 0.0070 3.0903 1.9054 0.0004 0.0139 5.0170 PASS
    tiny-bulk-delete-1-compact-v2 0.0099 0.1296 0.0080 0.0001 0.0026 0.1501 PASS
    tiny-bulk-delete-10-compact-v2 0.0086 0.2115 0.0104 0.0001 0.0033 0.2338 PASS
    tiny-bulk-delete-100-mixed-v3 0.0091 0.2988 0.0147 0.0001 0.0028 0.3253 PASS
    tiny-bulk-delete-500-mixed-v3 0.0076 0.9825 0.0506 0.0001 0.0028 1.0436 PASS

    Low tiers retain compact-v2 definitions; high-tier bulk cases use mixed-v3. Bulk tier100 creates/deletes 1,000 files /100 MiB and tier500 5,000 files /500 MiB, with the separate 200-file /1 MiB witness. Individual-operation cases retain their registered large base fixtures. These are distinct recipes, not a uniform scaling series.

    Strict tier100 assessment: create 1.069666876 s = TARGET_MISS by 0.069666876 s; delete 0.325349709 s = PASS. The family PASS does not satisfy the separate subsecond pair. The tier100 strict-gate proof pair remains deferred; no favorable repeat was taken.

    Per-case resource observations

    Host RSS is the host process peak reported after product work; container memory is the sample-container lifetime peak, including setup/file-cache effects. CPU scopes are host before-to-after-product process CPU and the separate Linux container command window. These are not exact per-phase memory peaks. Spool is separately accounted physical host backing; retained backing returns to zero after Commit in these samples.

    Case Host CPU s Container CPU s Host peak RSS MiB Container lifetime peak MiB Physical spool peak MiB
    tiny-create-1-compact-v2 0.0275 0.0206 9.53 5.65 0.00
    tiny-create-10-compact-v2 0.0329 0.0253 10.42 5.89 0.02
    tiny-create-100 0.0880 0.0458 53.25 5.41 0.16
    tiny-create-500 0.1582 0.1574 71.44 6.86 0.79
    tiny-stat-1-compact-v2 0.0267 0.0203 8.88 5.90 0.00
    tiny-stat-10-compact-v2 0.0290 0.0223 11.34 5.41 0.00
    tiny-stat-100 0.0535 0.0300 31.19 5.38 0.00
    tiny-stat-500 0.1478 0.0703 40.02 4.92 0.00
    tiny-unlink-1-compact-v2 0.0280 0.0207 9.41 4.93 0.00
    tiny-unlink-10-compact-v2 0.0337 0.0233 12.44 7.65 0.00
    tiny-unlink-100 0.1078 0.0400 52.69 5.16 0.00
    tiny-unlink-500 0.2387 0.0991 62.39 5.65 0.00
    tiny-bulk-create-1-compact-v2 0.0369 0.0591 14.30 6.80 1.00
    tiny-bulk-create-10-compact-v2 0.0905 0.1581 35.38 8.75 10.01
    tiny-bulk-create-100-mixed-v3 0.5004 0.3462 74.59 16.15 100.15
    tiny-bulk-create-500-mixed-v3 2.3839 1.5771 95.22 22.91 500.23
    tiny-bulk-delete-1-compact-v2 0.0710 0.0609 10.52 5.39 0.00
    tiny-bulk-delete-10-compact-v2 0.0896 0.0954 23.03 5.43 0.00
    tiny-bulk-delete-100-mixed-v3 0.1136 0.1374 37.77 5.14 0.00
    tiny-bulk-delete-500-mixed-v3 0.3174 0.4365 53.61 6.40 0.00

    All sample containers were validated at 2 CPUs /2 GiB /no swap /256 PIDs, with no host data mounts. macOS owns SDK/coordinator, physical spool and SQLite; Linux owns daemon/FUSE/workload and live operation state. Host CPU is uncapped. All master-isolation and cleanup checks pass; exact campaign-owned container and host-sample inventories are empty. Protected preparations remain available.

    Selected independent proofs

    Case Result Proof wall s
    tiny-create-1-compact-v2 PASS 1.5870
    tiny-stat-1-compact-v2 PASS 1.6090
    tiny-unlink-1-compact-v2 PASS 1.6561
    tiny-bulk-create-1-compact-v2 PASS 1.8890
    tiny-bulk-delete-1-compact-v2 PASS 1.7230
    tiny-bulk-create-500-mixed-v3 PASS 6.8302
    tiny-bulk-delete-500-mixed-v3 PASS 5.1849

    Total proof wall 20.4793 s; maximum 6.8302 s. Each proof retains its 45-second work /59-second hard deadline; separate preparation precedes the proof. All seven bind the corresponding performance source/product/input/image identities.

    Coverage: individual create/stat/unlink, compact bulk create/delete, and both mixed-v3 tier500 bulk cases. The tier500 create proof samples beginning/middle/end 64 KiB ranges of every large file, selected small/medium paths and witness; delete verifies selected target absence and witness retention. Exact paths/ranges are retained in the accompanying JSON. Omissions include unselected paths/bytes, exhaustive namespace/object census, aliases, failure injection, physical 100-workspace qualification and other platforms. Existing focused mapping/SDK/Commit tests remain separate evidence; these seven proofs do not replace the entire redesign checklist.

    Provenance and retained evidence

    • Implementation commit: e6d66d676b398e9ba941caba3cdcd34e74e8e608; official unmodified fuser 0.18.0.
    • Source seal: af3a4790d8d11007f1df8bee2e0f29b1747e75044a3b40eaace02b2d0f9366e6.
    • Product seal: 22e4ad3e355eabef447b2aabf568990e114a95a33d76707d5a876c98daebf60d.
    • Workload SHA-256: 2ec809798656deef9ac05903c9e3bd77921f6ef9c6c87f24060812a81b93bf76.
    • Image: layerfs-bench-infra:af3a4790d8d11007 / sha256:5054ec704fec09e95c1db8b898484f665ad9ab8444df51cc541400e9a99e6c03.
    • Host binary SHA-256: 2e4d2ba7894ead636b4e8e258b8d5448eaa4a02c59fdbd64ea3daa51ceae7f7f.
    • The sealed build identity retains its original dirty-source flag; source/product content seals, not that flag alone, bind the tested implementation. Later commits contain reports only.
    • Raw campaign: benchmark-results/host-store/campaigns/issue49-tiny-final-1788695216299617000.
    • Reused current-source create100 receipt: benchmark-results/host-store/results/run-48e8fff4646c/perf.jsonl.
    • Machine-readable per-case timings, resources, input identities, raw SHA-256 hashes and sampled proof paths.
    • Raw receipts, commands, preparation logs and assessments remain append-only in the local campaign directory. The JSON contains exact raw-file identities; raw receipts are not rewritten into a new source or pooled with historical products.
  8. added 3 commits that reference this issue on Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    v0.1.3LayerFS v0.1.3 checkpoint and delivery

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions