Skip to content

[v0.1.3][Phase 2] Optimize and restore the 15 runtime-suppressed benchmark cases #39

Closed as not planned
1 of 1 issue completed
Closed as not planned

Description

@yifanxuaaa

Comparison policy amendment — 2026-09-05

The user's latest instruction permits practical historical comparisons and overrides earlier blanket requirements to recollect a matched baseline. Do not rerun or resurrect the old harness merely to obtain a comparison table. Reuse retained baseline receipts/results when the workload and measurement meaning are sufficiently similar; small documented harness differences are acceptable without a new baseline run.

  • Classify comparisons as matched, historical-approximate, or not-comparable, and state the relevant difference in the table. Different source revisions are expected; preserve their provenance. Historical host/split-owner/APFS results may be used as approximate context where the selected product timer still measures comparable work, but must not be relabeled matched Docker/Linux evidence.
  • Judge similarity from actual operation, bytes/files/history depth, seed schedule, timing boundary, and resource scope. Do not introduce an arbitrary percentage tolerance or an extra certification campaign. If a difference could plausibly explain the claimed gain, qualify the conclusion rather than attributing it entirely to the product change.
  • Materially compacted fixtures (compact-v2/compact-v3 versus original large backgrounds), changed measured operations, and incompatible timing/resource scopes are not direct speedup comparisons. Show their absolute results side by side with the difference; leave speedup blank. Do not pool changed low tiers with old high tiers as an unchanged scaling curve.
  • Collect a new baseline with the current harness only for a specific unresolved decision that historical evidence cannot answer, using the smallest necessary selection. A full baseline campaign is not a prerequisite. Retain the prescribed candidate coverage, correctness checks, runtime limits and scaling criteria; this exception concerns baseline reuse, not permission to mark untested cases passing.
  • Derive tables on demand from compact logs; no benchmark rerun is needed to change presentation. See [v0.1.3] Optimize benchmark infrastructure: Docker/FUSE, reusable setup, three modes, compact logs #45 for field mapping. Preserve sample counts and seeds: N repetitions of one seed are not N different seeds. Verification comes from a separate identity-matched receipt, never inferred from performance PASS.

Phase 2.1 prerequisite

Blocked by #40: shared construction/admission, the three-column Workspace staging workflow, and namespace initialization at <=2.2 seconds with lower matched CPU under an aggregate eight-core ceiling. Apply its staging, no-conflict-resolution, limited-recovery, and bounded-qualification scope to this work. Reuse its implementation; do not create a competing engine.

This issue retains recovery of all fifteen suppressed cases, including live create/delete/read/Git/history costs that shared initialization or Commit improvements cannot remove alone.

Parent: #21. Phase 2 performance work following the completed Phase 1 baseline (#35).

Problem and outcome

Fifteen exact case/subset combinations remain suppressed because they exceeded the Phase 1 15-second cumulative product-time budget. Their benchmark implementations and historical evidence exist, but suppression is not PASS. Diagnose and optimize these operations, then recover the original coverage in an explicit Phase 2 lane with correct outcomes and measured timings.

This issue owns runtime-suppressed cases. Companion scaling issue #38 owns previously active cases that completed under the cutoff but scaled worse than proportionally. Shared fixes should be reused across both issues.

Complete inventory

The following completed historical timings are clean-source 6c54f8d7 starting observations, not current VM8/candidate baselines. Values are three-seed medians except namespace500 (one completed sample). The last row is a later incomplete timeout at source 30d13dee, not a completed timing or valid speedup denominator.

Exact scenario Workload Historical product seconds
dedup-history-unrelated-500 500 full-tree rewrite/Commit cycles 390.17
workspace-dense-rewrite-500 100,000 files / 500 MiB rewritten 305.17
tiny-bulk-create-500 100,000 files / 500 MiB created 175.04
dedup-history-unrelated-100 100 full-tree rewrite/Commit cycles 72.86
directory-content-scan-500 100,000 files / 500 MiB read 65.64
workspace-dense-rewrite-100 20,000 files / 100 MiB rewritten 54.53
namespace-subtree-relocate-delete-500 Original tier-500 subtree mutation 53.29; seed 1 only
tiny-bulk-delete-500 100,000 files / 500 MiB deleted 47.17
tiny-bulk-create-100 20,000 files / 100 MiB created 30.09
git-tool-500 Original Git workflow, 500 scheduled changes 26.53
git-tool-1 Original Git workflow, one scheduled change 23.19
git-tool-10 Original Git workflow, 10 scheduled changes 23.97
git-tool-100 Original Git workflow, 100 scheduled changes 23.80
dedup-history-distributed-500 500 distributed SDK edit/Commit cycles 17.34
directory-content-scan-100 20,000 files / 100 MiB read Timeout at 15.000758416; incomplete seed 1

Six families are affected: tiny-file churn, directory construction/traversal, Git workflow, subtree mutation, Workspace change locality, and single-Branch history. Keep exact IDs; do not suppress or claim failure for unaffected smaller tiers.

Evidence and investigation

Prioritize reusable shared causes: many-file write/capture/Commit costs, structural namespace mutation, full-tree read transport, repeated history Commit work, and Git's fixed workflow cost. Use existing phase receipts to identify the actual bottleneck; do not prescribe an architecture solely from aggregate timings. Coordinate with ongoing bulk-create optimization work and preserve its edits.

API reduction counts and algorithm consolidation

The reviewed target is fewer independent implementations, not an immediate large reduction of public SDK methods. Inventory at product source 810bb3a58: 26 production Client methods plus two feature-gated verification helpers. Reinventory the final #45 handoff source before editing.

Surface Initial change Later candidate
Public Rust SDK 0 methods removed; 26 remain, with compatible forwarding wrappers and consistent validation/observation One versioned Commit-variant reduction, 26 -> 25 only if no other API changes occur; migrate callers and preserve full publication/presentation status first
FUSE wire protocol Preserve acknowledged readonly Pin and current compatibility 1 identified obsolete request: PinRead, with no current production sender; pruning requires an explicit compatibility disposition and is not a guaranteed net request-count change
Internal Rust methods No validated net deletion count Count the reviewed implementation diff; a shared helper may remove duplicated bodies without reducing total methods
Algorithm area Current implementations Proposed target
Native scheduling/orchestration 3 1 scheduling implementation, with explicit appropriate sinks
Admission accumulation/orchestration 3 1 shared bounded accumulator; operation-specific publication stays explicit
Native regular-file construction bodies 2 1
Workspace inode-record construction bodies 2 1
Workspace candidate planners 3 Eventually 1, after all fallback domains and sparse/resource behavior transfer

These are not additive method-removal counts, performance promises or mandatory deletion quotas. Compatibility wrappers may remain. Do not create a catch-all endpoint or policy-flag framework merely to reduce enum/method names. Do not remove a route until its real callers and supported states transfer; test-only differential oracles are not extra production routes.

Applicability to later #39 work: shared admission and Workspace inode-record construction can transfer from #38 without a second engine; live create/open/closed lifecycle and one-kernel-callback attribute handling are separate Exec simplification candidates. UnlinkBatch already uses the same Workspace unlink algorithm. Fence/fsync, pin/read, reserved/acknowledged creation and open/closed lifetimes remain distinct where semantics require it.

Use the completed #45 Docker-only infrastructure handoff before new product runs. #38 simplification does not launch this issue, remove any of its fifteen suppressed definitions or certify their recovery. Preserve all historical Commit counts, alias/open-unlinked behavior, error propagation and per-case runtime/scaling criteria. A no-sender PinRead cleanup must never reintroduce asynchronous readonly-pin acknowledgement. Existing repaired correctness failures are regression constraints, not proof of new failures.

Detailed local plans: docs/roadmap/0.1/0.1.3/api-algorithm-simplification-audit.md, phase-2-shared-code-layout.md, and workspace-admission-complexity.md in /Users/yifanxu/Ephemeral-AI-Lab/layerfs. These documents are local planning edits, not claimed pushed commits. The counts and scope are included here so the issue is self-contained.

Acceptance

  • Retain all fifteen definitions and historical outcomes; record separate Phase 2 status and keep Phase 1 suppressions immutable. Confirm exact source/environment/input identities and freeze the candidate comparison before collection.
  • Restore the complete prescribed workload and three seeds per case in the Phase 2 lane. Initial recovery target: every complete sample ≤15 seconds cumulative declared product time. Report unresolved cases as open, not passed because disabled. Do not repeatedly rerun known slow full cases merely to reconfirm the baseline.
  • Evaluate each restored curve against its comparable smaller tiers using actual bytes/files/operations/cycles. Also require no worse-than-proportional median scaling; passing 15 seconds alone cannot establish a full performance pass. Report individual sample ratios and unresolved variability explicitly. Do not slow smaller tiers to manufacture a passing ratio.
  • Preserve all file operations, Git commands, historical Commit counts, public routes, CAS/CDC sharing semantics and failure/cleanup behavior. Fix actual functional failures rather than relabeling them as latency problems. Deleting live paths must not reclaim chunks still referenced by immutable history.
  • Keep each file ≤500 MiB and aggregate logical files <1 GiB at every workload state, with existing resource bounds. Do not reduce file count, shorten history, omit intermediate publications, raise the budget to turn failure into PASS, or move product work into preparation.
  • Publish exact before/after outcomes, every measured sample and distribution, product phases, preparation/sample wall, and CPU/memory/storage observations. Never compare an incomplete timeout with successful completion as a measured speedup.
  • Run only the focused correctness/regression checks required by the actual change, outside product timing, and a final affected-path check. Timing completion alone does not certify release or the withdrawn verification suite.

Fast iteration

Reuse the existing fs-bench-pro family files, shared 1/10/100/500 fixtures, SDK/FUSE runtime, collectors, prepared-input cache and qualified evidence. Diagnose with the smallest representative case, implement a shared fix once, and recollect only invalidated measurements. Avoid repeat runs of green cases until final scoped evidence collection; do not restart the withdrawn exhaustive Phase 1 verification campaign. Preparing inputs should be reused and timed separately. This issue records the Phase 2 backlog and does not launch execution or authorize release publication.

Activity

  1. yifanxuaaa commented on Sep 6, 2026

    @yifanxuaaa
    ContributorAuthor

    Issue #54 remaining-family collection is closed. Leftover failed/slow rows from that campaign (not the original 15 suppressed cases) are tracked in #61. This issue still owns the Phase 2 15-case restoration; do not treat #61 as closing it.

  2. yifanxuaaa commented on Sep 6, 2026

    @yifanxuaaa
    ContributorAuthor

    Do not restore old dedup-history-unrelated-{100,500} (200 files × 100/500 unique full-tree rewrites) as Phase 2 product work.

    Load reduction is a versioned workload on #65: dedup-history-unrelated-{100,500}-mixed-v2, 10 files / 1 MiB, still 100/500 Commits, 20× fewer writes. Old 90.1 s / 300 s receipts stay historical. #39 remains product optimization of suppressed cases on their current active IDs, not a silent shrink of the 200×500 recipe.

  3. yifanxuaaa commented on Sep 6, 2026

    @yifanxuaaa
    ContributorAuthor

    Reopened because Phase 2 restore/optimize of suppressed cases is still product work. #65 / PR #66 (2b12825d8) landed mixed-v2 for unrelated-100/500 and does not close this issue. Do not restore the old 200×500 unrelated recipe; current active IDs are dedup-history-unrelated-{100,500}-mixed-v2.

  4. yifanxuaaa commented on Sep 6, 2026

    @yifanxuaaa
    ContributorAuthor

    Closing at the maintainer's request to retire this remaining v0.1.3 qualification scope. The one-sample collection and versioned workload replacements are recorded in #54/#62/#65; they do not establish the original three-seed/scaling qualification or turn historical suppressed/failed workloads into PASS. Only #50 and #51 remain active for v0.1.3.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    v0.1.3LayerFS v0.1.3 checkpoint and delivery

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions