Repository navigation
Description
Comparison policy amendment — 2026-09-05
The user's latest instruction permits practical historical comparisons and overrides earlier blanket requirements to recollect a matched baseline. Do not rerun or resurrect the old harness merely to obtain a comparison table. Reuse retained baseline receipts/results when the workload and measurement meaning are sufficiently similar; small documented harness differences are acceptable without a new baseline run.
- Classify comparisons as
matched,historical-approximate, ornot-comparable, and state the relevant difference in the table. Different source revisions are expected; preserve their provenance. Historical host/split-owner/APFS results may be used as approximate context where the selected product timer still measures comparable work, but must not be relabeled matched Docker/Linux evidence. - Judge similarity from actual operation, bytes/files/history depth, seed schedule, timing boundary, and resource scope. Do not introduce an arbitrary percentage tolerance or an extra certification campaign. If a difference could plausibly explain the claimed gain, qualify the conclusion rather than attributing it entirely to the product change.
- Materially compacted fixtures (
compact-v2/compact-v3versus original large backgrounds), changed measured operations, and incompatible timing/resource scopes are not direct speedup comparisons. Show their absolute results side by side with the difference; leave speedup blank. Do not pool changed low tiers with old high tiers as an unchanged scaling curve. - Collect a new baseline with the current harness only for a specific unresolved decision that historical evidence cannot answer, using the smallest necessary selection. A full baseline campaign is not a prerequisite. Retain the prescribed candidate coverage, correctness checks, runtime limits and scaling criteria; this exception concerns baseline reuse, not permission to mark untested cases passing.
- Derive tables on demand from compact logs; no benchmark rerun is needed to change presentation. See [v0.1.3] Optimize benchmark infrastructure: Docker/FUSE, reusable setup, three modes, compact logs #45 for field mapping. Preserve sample counts and seeds: N repetitions of one seed are not N different seeds. Verification comes from a separate identity-matched receipt, never inferred from performance PASS.
Phase 2.1 prerequisite
Blocked by #40: shared construction/admission, the three-column Workspace staging workflow, and namespace initialization at <=2.2 seconds with lower matched CPU under an aggregate eight-core ceiling. Apply its staging, no-conflict-resolution, limited-recovery, and bounded-qualification scope to this work. Reuse its implementation; do not create a competing engine.
This issue retains recovery of all fifteen suppressed cases, including live create/delete/read/Git/history costs that shared initialization or Commit improvements cannot remove alone.
Parent: #21. Phase 2 performance work following the completed Phase 1 baseline (#35).
Problem and outcome
Fifteen exact case/subset combinations remain suppressed because they exceeded the Phase 1 15-second cumulative product-time budget. Their benchmark implementations and historical evidence exist, but suppression is not PASS. Diagnose and optimize these operations, then recover the original coverage in an explicit Phase 2 lane with correct outcomes and measured timings.
This issue owns runtime-suppressed cases. Companion scaling issue #38 owns previously active cases that completed under the cutoff but scaled worse than proportionally. Shared fixes should be reused across both issues.
Complete inventory
The following completed historical timings are clean-source 6c54f8d7 starting observations, not current VM8/candidate baselines. Values are three-seed medians except namespace500 (one completed sample). The last row is a later incomplete timeout at source 30d13dee, not a completed timing or valid speedup denominator.
| Exact scenario | Workload | Historical product seconds |
|---|---|---|
dedup-history-unrelated-500 |
500 full-tree rewrite/Commit cycles | 390.17 |
workspace-dense-rewrite-500 |
100,000 files / 500 MiB rewritten | 305.17 |
tiny-bulk-create-500 |
100,000 files / 500 MiB created | 175.04 |
dedup-history-unrelated-100 |
100 full-tree rewrite/Commit cycles | 72.86 |
directory-content-scan-500 |
100,000 files / 500 MiB read | 65.64 |
workspace-dense-rewrite-100 |
20,000 files / 100 MiB rewritten | 54.53 |
namespace-subtree-relocate-delete-500 |
Original tier-500 subtree mutation | 53.29; seed 1 only |
tiny-bulk-delete-500 |
100,000 files / 500 MiB deleted | 47.17 |
tiny-bulk-create-100 |
20,000 files / 100 MiB created | 30.09 |
git-tool-500 |
Original Git workflow, 500 scheduled changes | 26.53 |
git-tool-1 |
Original Git workflow, one scheduled change | 23.19 |
git-tool-10 |
Original Git workflow, 10 scheduled changes | 23.97 |
git-tool-100 |
Original Git workflow, 100 scheduled changes | 23.80 |
dedup-history-distributed-500 |
500 distributed SDK edit/Commit cycles | 17.34 |
directory-content-scan-100 |
20,000 files / 100 MiB read | Timeout at 15.000758416; incomplete seed 1 |
Six families are affected: tiny-file churn, directory construction/traversal, Git workflow, subtree mutation, Workspace change locality, and single-Branch history. Keep exact IDs; do not suppress or claim failure for unaffected smaller tiers.
Evidence and investigation
- Final Phase 1 report and suppression inventory.
- Phase 2 backlog, evidence anchors and phase breakdowns. Its top-level withdrawal notice supersedes historical pending-verification instructions.
- Persistent ledger:
benchmark-results/fs-bench-pro/phase1-v013/phase1-runtime-suppressions.json; source-bound attempts and slot history remain immutable. - Existing bulk create/delete investigation notes: approximately 2.7-second v0.1.1 initialization is a useful architectural reference, not the same public operation as live bulk creation. The 3–5-second bulk-create aspiration remains a stretch investigation, not proof or a universal target.
Prioritize reusable shared causes: many-file write/capture/Commit costs, structural namespace mutation, full-tree read transport, repeated history Commit work, and Git's fixed workflow cost. Use existing phase receipts to identify the actual bottleneck; do not prescribe an architecture solely from aggregate timings. Coordinate with ongoing bulk-create optimization work and preserve its edits.
API reduction counts and algorithm consolidation
The reviewed target is fewer independent implementations, not an immediate large reduction of public SDK methods. Inventory at product source 810bb3a58: 26 production Client methods plus two feature-gated verification helpers. Reinventory the final #45 handoff source before editing.
| Surface | Initial change | Later candidate |
|---|---|---|
| Public Rust SDK | 0 methods removed; 26 remain, with compatible forwarding wrappers and consistent validation/observation | One versioned Commit-variant reduction, 26 -> 25 only if no other API changes occur; migrate callers and preserve full publication/presentation status first |
| FUSE wire protocol | Preserve acknowledged readonly Pin and current compatibility | 1 identified obsolete request: PinRead, with no current production sender; pruning requires an explicit compatibility disposition and is not a guaranteed net request-count change |
| Internal Rust methods | No validated net deletion count | Count the reviewed implementation diff; a shared helper may remove duplicated bodies without reducing total methods |
| Algorithm area | Current implementations | Proposed target |
|---|---|---|
| Native scheduling/orchestration | 3 | 1 scheduling implementation, with explicit appropriate sinks |
| Admission accumulation/orchestration | 3 | 1 shared bounded accumulator; operation-specific publication stays explicit |
| Native regular-file construction bodies | 2 | 1 |
| Workspace inode-record construction bodies | 2 | 1 |
| Workspace candidate planners | 3 | Eventually 1, after all fallback domains and sparse/resource behavior transfer |
These are not additive method-removal counts, performance promises or mandatory deletion quotas. Compatibility wrappers may remain. Do not create a catch-all endpoint or policy-flag framework merely to reduce enum/method names. Do not remove a route until its real callers and supported states transfer; test-only differential oracles are not extra production routes.
Applicability to later #39 work: shared admission and Workspace inode-record construction can transfer from #38 without a second engine; live create/open/closed lifecycle and one-kernel-callback attribute handling are separate Exec simplification candidates. UnlinkBatch already uses the same Workspace unlink algorithm. Fence/fsync, pin/read, reserved/acknowledged creation and open/closed lifetimes remain distinct where semantics require it.
Use the completed #45 Docker-only infrastructure handoff before new product runs. #38 simplification does not launch this issue, remove any of its fifteen suppressed definitions or certify their recovery. Preserve all historical Commit counts, alias/open-unlinked behavior, error propagation and per-case runtime/scaling criteria. A no-sender PinRead cleanup must never reintroduce asynchronous readonly-pin acknowledgement. Existing repaired correctness failures are regression constraints, not proof of new failures.
Detailed local plans: docs/roadmap/0.1/0.1.3/api-algorithm-simplification-audit.md, phase-2-shared-code-layout.md, and workspace-admission-complexity.md in /Users/yifanxu/Ephemeral-AI-Lab/layerfs. These documents are local planning edits, not claimed pushed commits. The counts and scope are included here so the issue is self-contained.
Acceptance
- Retain all fifteen definitions and historical outcomes; record separate Phase 2 status and keep Phase 1 suppressions immutable. Confirm exact source/environment/input identities and freeze the candidate comparison before collection.
- Restore the complete prescribed workload and three seeds per case in the Phase 2 lane. Initial recovery target: every complete sample ≤15 seconds cumulative declared product time. Report unresolved cases as open, not passed because disabled. Do not repeatedly rerun known slow full cases merely to reconfirm the baseline.
- Evaluate each restored curve against its comparable smaller tiers using actual bytes/files/operations/cycles. Also require no worse-than-proportional median scaling; passing 15 seconds alone cannot establish a full performance pass. Report individual sample ratios and unresolved variability explicitly. Do not slow smaller tiers to manufacture a passing ratio.
- Preserve all file operations, Git commands, historical Commit counts, public routes, CAS/CDC sharing semantics and failure/cleanup behavior. Fix actual functional failures rather than relabeling them as latency problems. Deleting live paths must not reclaim chunks still referenced by immutable history.
- Keep each file ≤500 MiB and aggregate logical files <1 GiB at every workload state, with existing resource bounds. Do not reduce file count, shorten history, omit intermediate publications, raise the budget to turn failure into PASS, or move product work into preparation.
- Publish exact before/after outcomes, every measured sample and distribution, product phases, preparation/sample wall, and CPU/memory/storage observations. Never compare an incomplete timeout with successful completion as a measured speedup.
- Run only the focused correctness/regression checks required by the actual change, outside product timing, and a final affected-path check. Timing completion alone does not certify release or the withdrawn verification suite.
Fast iteration
Reuse the existing fs-bench-pro family files, shared 1/10/100/500 fixtures, SDK/FUSE runtime, collectors, prepared-input cache and qualified evidence. Diagnose with the smallest representative case, implement a shared fix once, and recollect only invalidated measurements. Avoid repeat runs of green cases until final scoped evidence collection; do not restart the withdrawn exhaustive Phase 1 verification campaign. Preparing inputs should be reused and timed separately. This issue records the Phase 2 backlog and does not launch execution or authorize release publication.
Activity
yifanxuaaa commentedon Sep 6, 2026
Issue #54 remaining-family collection is closed. Leftover failed/slow rows from that campaign (not the original 15 suppressed cases) are tracked in #61. This issue still owns the Phase 2 15-case restoration; do not treat #61 as closing it.
yifanxuaaa commentedon Sep 6, 2026
Do not restore old
dedup-history-unrelated-{100,500}(200 files × 100/500 unique full-tree rewrites) as Phase 2 product work.Load reduction is a versioned workload on #65:
dedup-history-unrelated-{100,500}-mixed-v2, 10 files / 1 MiB, still 100/500 Commits, 20× fewer writes. Old 90.1 s / 300 s receipts stay historical. #39 remains product optimization of suppressed cases on their current active IDs, not a silent shrink of the 200×500 recipe.yifanxuaaa commentedon Sep 6, 2026
Reopened because Phase 2 restore/optimize of suppressed cases is still product work. #65 / PR #66 (
2b12825d8) landed mixed-v2 for unrelated-100/500 and does not close this issue. Do not restore the old 200×500 unrelated recipe; current active IDs arededup-history-unrelated-{100,500}-mixed-v2.yifanxuaaa commentedon Sep 6, 2026
Closing at the maintainer's request to retire this remaining v0.1.3 qualification scope. The one-sample collection and versioned workload replacements are recorded in #54/#62/#65; they do not establish the original three-seed/scaling qualification or turn historical suppressed/failed workloads into PASS. Only #50 and #51 remain active for v0.1.3.