Repository navigation
Description
Purpose and scheduling
Follow-up to #49. Deferred until the FUSE architecture migration and its required qualification are complete. This issue must not expand or block the current migration. The user is satisfied with the current performance if the architecture is migrated correctly. Keep the current 0.9000 s working lifecycle target / approximately 0.0500 s tolerance and formal same-source create/delete <1.0000 s gate separate from this later investigation; this issue introduces no new mandatory timing gate.
Investigate why the new live-owner Exec path has not yet beaten the optimized #48 research path. Reuse the new shared architecture and existing optimizations; do not restore split mutable authorities or introduce benchmark-specific routes. Read-only review on 2026-09-06 identified concrete differences, but did not run experiments or establish a causal attribution of every millisecond.
Evidence: comparable recipe, different implementations
All listed create observations use tiny-bulk-create-100-mixed-v3, seed 1: 1,000 files / 100 MiB, including one 50 MiB file, 800 × 4 KiB files and 199 medium files; separate 200-file / 1 MiB witness. Workload source and fixture agree. Host macOS owns SQLite/spool/coordination; Docker Linux runs workload/daemon/FUSE with 2 CPUs / 2 GiB / no swap / 256 PIDs and no data mounts.
| Product observation | Exec | Commit | Complete lifecycle |
|---|---|---|---|
| Adopted pre-rewrite product | 0.9172 s | 0.3587 s | 1.2914 s |
| Optimized old #48 research | 0.4792 s | 0.4718 s | 0.9680 s |
| Rewrite early integrated sample | 0.5718 s | 0.3638 s | 0.9533 s |
| Rewrite resource-correction sample | 0.5716 s | 0.4046 s | 0.9946 s |
#48 did not contain the finalized Commit optimization. Do not credit the rewrite with inherited Commit gains. Replacing #48's Commit time arithmetically with the separately measured 0.3587 s yields about 0.8549 s, but that is a counterfactual planning estimate, never a measured combined-product result. Neither the old optimized combined product nor the final qualified rewrite has been measured in a controlled pair here. Each listed result is n=1 on a different development source; do not claim statistical regression or causal precision.
Raw old-to-new differences locate the investigation:
- Exec: +0.092441459 s; ordinary workload wall: +0.093173167 s.
- Command-window container CPU: 0.305569000 → 0.314112000 s (+0.008543000 s); this scope is not isolated FUSE callback CPU or host Exec CPU.
- Metadata normalization: 0.113211791 → 0.098395458 s (improved).
- Workload root sync: 0.000769958 → 0.008480041 s (worse).
- Removing the two disjoint workload subspans above leaves approximately +0.100279417 s in the remaining workload path. This residual is not a proven removable wait budget.
- FLUSH callbacks: 0 → 1,000. Both recorded 1,414 LOOKUP, 1,139 GETATTR, 2,416 SETATTR, 1,099 WRITE and 1,000 CREATE/RELEASE callbacks. Callback inflation is localized to FLUSH, not extra lookups/setters/writes.
- Host write dispatch / nested physical spool service: 0.0408 / 0.0377 s → 0.0360 / 0.0328 s. Host spool service did not get slower in these observations.
Confirmed mechanism differences and hypotheses
1. Backing acknowledgment moved serial waits onto the write path — first investigation
Old ProxyClient::raw_send_at sends writes and records pending requests without waiting for each response; host proxy_host::serve handles those writes without individual replies, with errors/acknowledgment at subsequent boundaries. The old design is already batched/local in important places, not the original per-operation synchronous baseline.
The rewrite's LiveOwner::write_owned / flush_append and serial BackingConnection::call wait at backing-window rollover:
Old optimized path:
local write / queue send -> continue callbacks
host applies ordered writes -> later acknowledgment/error boundary
Current rewrite rollover:
APPEND -> wait -> CANCEL unused tail, if any -> wait
-> RESERVE next window -> wait -> continue write
Bounded first candidate, if not already landed in #49:
APPEND_CLOSE_RESERVE -> one reply
old prefix/tail-close ACK + separately tagged next reservation outcome
The rewrite recorded 273 backing calls and 0.175924333 s cumulative exchange wait. This is an envelope, not all avoidable time, and cannot be subtracted wholesale from Exec. Local mutable authority does not automatically make backing persistence asynchronous.
First check the final #49 outcome; do not redo accepted/rejected work without new evidence. Reuse existing HostSpool append/reservation/tail-close code. Keep one bounded payload window; no new scheduler, transport connection, background worker framework or unbounded queue. Validate fields before mutation. Old append success plus next-reservation ENOSPC must acknowledge the successful old write separately and must not cause replay. Preserve uncertain-response ownership, error propagation, lifetimes and accounting. If future overlap is justified by measurement, explicitly define bounded ownership/error boundaries before changing completion; do not blindly copy old fire-and-forget semantics.
2. Missing targeted NOFLUSH — conditional small candidate
Old #48 uses NOFLUSH for newly created DIRECT_IO handles; the measured rewrite does not. Current FLUSH includes admission and callback_gate, which can return failed/closing-owner errors. It is not universally behaviorless.
Check whether #49 already accepted or rejected this candidate. Preserve required close-error semantics, RELEASE, explicit FSYNC, Commit/End, mappings and ordinary-open behavior. If close is a required deferred-error boundary, keep FLUSH. Do not add a new error protocol merely to suppress callbacks. Count reduction alone is not an elapsed-time improvement.
3. Copy placement and ownership work — lower priority
The old large-write path reuses header/payload vectored writing; smaller closed-create frames still copy. Both receipts count 100 MiB request copies, while payload-frame copy bytes increase from 50 MiB to 100 MiB. The rewrite eagerly owns callback bytes in submit_write, then copies into the append frame in write_owned.
However, old host decode also copied 50 MiB and new host decode reports zero: aggregate explicit copy counters remain 200 MiB in both observations. This shifts 50 MiB of copying onto the container; it does not establish more aggregate copying. Recorded encoding time increases only about 0.0034 s, so copying is not an evidenced explanation of the whole Exec gap. Explore reuse of charged owned buffers or the existing vectored transport pattern only after identifying material cost. Preserve cancellation-safe ownership and bounded capacity; don't replace required ownership with borrowed callback memory.
Current scheduler.submit polls ready futures inline and spawns only on Pending. Do not assume a thread/task hop per callback. Additional admission, gates, cache ordering and resource bookkeeping exist, but neither lock contention nor scheduler time is established as the root cause by these receipts. Instrument the actual critical path before redesigning either.
Concrete code locations inspected (moving working source)
Old crates/layerfs-fuse/src/: proxy_client.rs::raw_send_at (~221), buffer_write (~250), send_buffer_locked (~317), synchronize_locked (~419), unpin (~830); protocol.rs::Request::no_reply (~53) and write_request_measured (~168); proxy_host.rs::serve (~275); filesystem.rs::create (~577).
Rewrite crates/layerfs-fuse/src/: live_owner.rs::write_owned (~285), flush_append (~420), submit_write (~842), admit_callback (~752), ordered (~235); live_transport.rs::BackingConnection::call (~364) and serve (~231); filesystem.rs::flush (~802), create (~1343); live_runtime.rs::Scheduler::submit (~86). Host side: crates/layerfs-workspace/src/live_backing.rs::BackingOwner::request (~231). Exact line numbers can move; resolve methods on the final source before editing.
Read-only inspection saw old HEAD 8cbbb425b720d3c5f639c69375e85b197ca2515a with research edits and rewrite HEAD 88e4d9a88db3cea4d62c45d5e9c5704b9493174d with active edits. These are not substitutes for the measured source seals below.
Later implementation sequence and completion
- Start from the completed, qualified [v0.1.3] Redesign FUSE Exec around one shared live Workspace core #49 source and record which reviewed candidates already landed or were rejected. Use its exact source/product-bound baseline; old [v0.1.3] Optimize live FUSE Exec in parallel with the Commit migration #48 remains a reference unless a clean combined-source control is deliberately justified.
- Add minimal counters/spans for actual rollover opcode counts, disjoint request waiting versus host service, sync/freeze costs, callback admission/dispatch and copies. Reuse diagnostics; no permanent tracing framework. Preserve timing nesting and instrumentation scope.
- Pick one demonstrated cost, make the smallest generic change in the shared path, run focused changed-seam checks, build matching cached host/image artifacts and run one unchanged create-100 sample. Do not repeat for favorable timing.
- Judge complete lifecycle as well as Exec; no shifting work into Commit/End or increasing unbounded CPU/memory. A material recovery of part of the observed approximately 0.0900 s Exec gap would be useful; matching the historical approximately 0.4800 s Exec is a reference, not a hard gate or guarantee. Do not reopen the old 0.7000–0.8000 s target. Differences around 0.0500 s do not justify prolonged minor tuning.
- Finish with a measured keep/reject decision and an honest causal explanation. If no safe material gain exists, preserve the correct implementation and document that result; no automatic further rewrite. Any retained code change needs same-source delete control and final selected proofs as appropriate.
Use benchmark/fs-bench-pro/QUICKSTART.md, existing family setup/perf/verify scripts, --setup clone, compatible protected preparation and shared serial measurement lock. Host SQLite only, no data mounts, no container coordinator, no native-directory shortcut. Independent proofs remain final-only, 45 s work / 59 s hard per proof; preparation/builds separate. No physical 100-workspace or broad family campaign. Preserve CAS/CDC/COW/rope/extent builders, same live nodes/handles across Commit, concurrent command behavior, failure recovery and bounded resources. Benefits must apply generically to ordinary shell/SDK operations and #39 sibling families without workload-specific policy.
Reproducible evidence locations
- Old worktree
/Users/yifanxu/.codex/worktrees/401f/layerfs,research/issue48-exec/real-fuse/mixed-v3-baseline.mdandmixed-v3-baseline-create100/perf.jsonl. - Old source
c09f058ee659e72428c6e9e07a67e20850db165d70dbc1c0115e9c7aa0d94d0b; product68cf387dba5749287c0e9c5f950cbe60128084ae153455fa5618edbf97a9859c. - Rewrite worktree
/Users/yifanxu/.codex/worktrees/1b4b/layerfs,benchmark-results/host-store/results/run-7cccdb7e6378/perf.jsonl. - Rewrite source
bf446dbec87009dce0aa73297b055108831af68f7d28c102a42781399311cef5; productfa8357152fb3adf06b6434774dbef7b749747d9faff9257daf9e4228e03437bd. - Earlier rewrite receipt
run-0acf47533e78/perf.jsonl. Current working files are moving and do not automatically equal those measured source seals. - Existing bounded plan:
docs/roadmap/0.1/0.1.3/issue49-final-optimization-pass.md. This issue is the deferred continuation, not an instruction to reopen the current final pass.
Activity
docs(gc): add decision ownership table; point README at the upstream PR
docs(gc): record where R1, Q1 and C3 ask for sign-off
docs(gc): drop the two redundant summaries, trim the checklist
docs(gc): reduce the checklist to acceptance criteria and dependencies
docs(gc): embed the six GC-41 diagrams
docs(gc): merge the Q6/§5.1 entries in the checklist
feat(gc): report, API/MCP, acceptance and docs (#50-#52)
feat(gc): report, API/MCP, acceptance and docs (#50-#52)
feat(gc): report, API/MCP, acceptance and docs (#50-#52)
feat(gc): SQLite and PostgreSQL compaction (layerfs-project#49, #50)
fix(#50): PG status/sync/physical_after round-4 fixes
merge fixr4-50: fix(#50): PG status/sync/physical_after round-4 fixes
fix(#50): round-5 — legacy standby list, reader holder, Phase M statu…
fix(#50): round-6 — standby tokens parse without spacing, and a prova…