Skip to content

[Follow-up to #49] Recover remaining Exec overhead after shared-live-owner migration #50

Closed

Description

@yifanxuaaa

Purpose and scheduling

Follow-up to #49. Deferred until the FUSE architecture migration and its required qualification are complete. This issue must not expand or block the current migration. The user is satisfied with the current performance if the architecture is migrated correctly. Keep the current 0.9000 s working lifecycle target / approximately 0.0500 s tolerance and formal same-source create/delete <1.0000 s gate separate from this later investigation; this issue introduces no new mandatory timing gate.

Investigate why the new live-owner Exec path has not yet beaten the optimized #48 research path. Reuse the new shared architecture and existing optimizations; do not restore split mutable authorities or introduce benchmark-specific routes. Read-only review on 2026-09-06 identified concrete differences, but did not run experiments or establish a causal attribution of every millisecond.

Evidence: comparable recipe, different implementations

All listed create observations use tiny-bulk-create-100-mixed-v3, seed 1: 1,000 files / 100 MiB, including one 50 MiB file, 800 × 4 KiB files and 199 medium files; separate 200-file / 1 MiB witness. Workload source and fixture agree. Host macOS owns SQLite/spool/coordination; Docker Linux runs workload/daemon/FUSE with 2 CPUs / 2 GiB / no swap / 256 PIDs and no data mounts.

Product observation Exec Commit Complete lifecycle
Adopted pre-rewrite product 0.9172 s 0.3587 s 1.2914 s
Optimized old #48 research 0.4792 s 0.4718 s 0.9680 s
Rewrite early integrated sample 0.5718 s 0.3638 s 0.9533 s
Rewrite resource-correction sample 0.5716 s 0.4046 s 0.9946 s

#48 did not contain the finalized Commit optimization. Do not credit the rewrite with inherited Commit gains. Replacing #48's Commit time arithmetically with the separately measured 0.3587 s yields about 0.8549 s, but that is a counterfactual planning estimate, never a measured combined-product result. Neither the old optimized combined product nor the final qualified rewrite has been measured in a controlled pair here. Each listed result is n=1 on a different development source; do not claim statistical regression or causal precision.

Raw old-to-new differences locate the investigation:

  • Exec: +0.092441459 s; ordinary workload wall: +0.093173167 s.
  • Command-window container CPU: 0.305569000 → 0.314112000 s (+0.008543000 s); this scope is not isolated FUSE callback CPU or host Exec CPU.
  • Metadata normalization: 0.113211791 → 0.098395458 s (improved).
  • Workload root sync: 0.000769958 → 0.008480041 s (worse).
  • Removing the two disjoint workload subspans above leaves approximately +0.100279417 s in the remaining workload path. This residual is not a proven removable wait budget.
  • FLUSH callbacks: 0 → 1,000. Both recorded 1,414 LOOKUP, 1,139 GETATTR, 2,416 SETATTR, 1,099 WRITE and 1,000 CREATE/RELEASE callbacks. Callback inflation is localized to FLUSH, not extra lookups/setters/writes.
  • Host write dispatch / nested physical spool service: 0.0408 / 0.0377 s → 0.0360 / 0.0328 s. Host spool service did not get slower in these observations.

Confirmed mechanism differences and hypotheses

1. Backing acknowledgment moved serial waits onto the write path — first investigation

Old ProxyClient::raw_send_at sends writes and records pending requests without waiting for each response; host proxy_host::serve handles those writes without individual replies, with errors/acknowledgment at subsequent boundaries. The old design is already batched/local in important places, not the original per-operation synchronous baseline.

The rewrite's LiveOwner::write_owned / flush_append and serial BackingConnection::call wait at backing-window rollover:

Old optimized path:
  local write / queue send -> continue callbacks
      host applies ordered writes -> later acknowledgment/error boundary

Current rewrite rollover:
  APPEND -> wait -> CANCEL unused tail, if any -> wait
         -> RESERVE next window -> wait -> continue write

Bounded first candidate, if not already landed in #49:
  APPEND_CLOSE_RESERVE -> one reply
      old prefix/tail-close ACK + separately tagged next reservation outcome

The rewrite recorded 273 backing calls and 0.175924333 s cumulative exchange wait. This is an envelope, not all avoidable time, and cannot be subtracted wholesale from Exec. Local mutable authority does not automatically make backing persistence asynchronous.

First check the final #49 outcome; do not redo accepted/rejected work without new evidence. Reuse existing HostSpool append/reservation/tail-close code. Keep one bounded payload window; no new scheduler, transport connection, background worker framework or unbounded queue. Validate fields before mutation. Old append success plus next-reservation ENOSPC must acknowledge the successful old write separately and must not cause replay. Preserve uncertain-response ownership, error propagation, lifetimes and accounting. If future overlap is justified by measurement, explicitly define bounded ownership/error boundaries before changing completion; do not blindly copy old fire-and-forget semantics.

2. Missing targeted NOFLUSH — conditional small candidate

Old #48 uses NOFLUSH for newly created DIRECT_IO handles; the measured rewrite does not. Current FLUSH includes admission and callback_gate, which can return failed/closing-owner errors. It is not universally behaviorless.

Check whether #49 already accepted or rejected this candidate. Preserve required close-error semantics, RELEASE, explicit FSYNC, Commit/End, mappings and ordinary-open behavior. If close is a required deferred-error boundary, keep FLUSH. Do not add a new error protocol merely to suppress callbacks. Count reduction alone is not an elapsed-time improvement.

3. Copy placement and ownership work — lower priority

The old large-write path reuses header/payload vectored writing; smaller closed-create frames still copy. Both receipts count 100 MiB request copies, while payload-frame copy bytes increase from 50 MiB to 100 MiB. The rewrite eagerly owns callback bytes in submit_write, then copies into the append frame in write_owned.

However, old host decode also copied 50 MiB and new host decode reports zero: aggregate explicit copy counters remain 200 MiB in both observations. This shifts 50 MiB of copying onto the container; it does not establish more aggregate copying. Recorded encoding time increases only about 0.0034 s, so copying is not an evidenced explanation of the whole Exec gap. Explore reuse of charged owned buffers or the existing vectored transport pattern only after identifying material cost. Preserve cancellation-safe ownership and bounded capacity; don't replace required ownership with borrowed callback memory.

Current scheduler.submit polls ready futures inline and spawns only on Pending. Do not assume a thread/task hop per callback. Additional admission, gates, cache ordering and resource bookkeeping exist, but neither lock contention nor scheduler time is established as the root cause by these receipts. Instrument the actual critical path before redesigning either.

Concrete code locations inspected (moving working source)

Old crates/layerfs-fuse/src/: proxy_client.rs::raw_send_at (~221), buffer_write (~250), send_buffer_locked (~317), synchronize_locked (~419), unpin (~830); protocol.rs::Request::no_reply (~53) and write_request_measured (~168); proxy_host.rs::serve (~275); filesystem.rs::create (~577).

Rewrite crates/layerfs-fuse/src/: live_owner.rs::write_owned (~285), flush_append (~420), submit_write (~842), admit_callback (~752), ordered (~235); live_transport.rs::BackingConnection::call (~364) and serve (~231); filesystem.rs::flush (~802), create (~1343); live_runtime.rs::Scheduler::submit (~86). Host side: crates/layerfs-workspace/src/live_backing.rs::BackingOwner::request (~231). Exact line numbers can move; resolve methods on the final source before editing.

Read-only inspection saw old HEAD 8cbbb425b720d3c5f639c69375e85b197ca2515a with research edits and rewrite HEAD 88e4d9a88db3cea4d62c45d5e9c5704b9493174d with active edits. These are not substitutes for the measured source seals below.

Later implementation sequence and completion

  1. Start from the completed, qualified [v0.1.3] Redesign FUSE Exec around one shared live Workspace core #49 source and record which reviewed candidates already landed or were rejected. Use its exact source/product-bound baseline; old [v0.1.3] Optimize live FUSE Exec in parallel with the Commit migration #48 remains a reference unless a clean combined-source control is deliberately justified.
  2. Add minimal counters/spans for actual rollover opcode counts, disjoint request waiting versus host service, sync/freeze costs, callback admission/dispatch and copies. Reuse diagnostics; no permanent tracing framework. Preserve timing nesting and instrumentation scope.
  3. Pick one demonstrated cost, make the smallest generic change in the shared path, run focused changed-seam checks, build matching cached host/image artifacts and run one unchanged create-100 sample. Do not repeat for favorable timing.
  4. Judge complete lifecycle as well as Exec; no shifting work into Commit/End or increasing unbounded CPU/memory. A material recovery of part of the observed approximately 0.0900 s Exec gap would be useful; matching the historical approximately 0.4800 s Exec is a reference, not a hard gate or guarantee. Do not reopen the old 0.7000–0.8000 s target. Differences around 0.0500 s do not justify prolonged minor tuning.
  5. Finish with a measured keep/reject decision and an honest causal explanation. If no safe material gain exists, preserve the correct implementation and document that result; no automatic further rewrite. Any retained code change needs same-source delete control and final selected proofs as appropriate.

Use benchmark/fs-bench-pro/QUICKSTART.md, existing family setup/perf/verify scripts, --setup clone, compatible protected preparation and shared serial measurement lock. Host SQLite only, no data mounts, no container coordinator, no native-directory shortcut. Independent proofs remain final-only, 45 s work / 59 s hard per proof; preparation/builds separate. No physical 100-workspace or broad family campaign. Preserve CAS/CDC/COW/rope/extent builders, same live nodes/handles across Commit, concurrent command behavior, failure recovery and bounded resources. Benefits must apply generically to ordinary shell/SDK operations and #39 sibling families without workload-specific policy.

Reproducible evidence locations

  • Old worktree /Users/yifanxu/.codex/worktrees/401f/layerfs, research/issue48-exec/real-fuse/mixed-v3-baseline.md and mixed-v3-baseline-create100/perf.jsonl.
  • Old source c09f058ee659e72428c6e9e07a67e20850db165d70dbc1c0115e9c7aa0d94d0b; product 68cf387dba5749287c0e9c5f950cbe60128084ae153455fa5618edbf97a9859c.
  • Rewrite worktree /Users/yifanxu/.codex/worktrees/1b4b/layerfs, benchmark-results/host-store/results/run-7cccdb7e6378/perf.jsonl.
  • Rewrite source bf446dbec87009dce0aa73297b055108831af68f7d28c102a42781399311cef5; product fa8357152fb3adf06b6434774dbef7b749747d9faff9257daf9e4228e03437bd.
  • Earlier rewrite receipt run-0acf47533e78/perf.jsonl. Current working files are moving and do not automatically equal those measured source seals.
  • Existing bounded plan: docs/roadmap/0.1/0.1.3/issue49-final-optimization-pass.md. This issue is the deferred continuation, not an instruction to reopen the current final pass.

Activity

  1. added 8 commits that reference this issue on Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    new featureA new user-facing capability

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions