Skip to content

[v0.1.3][Stage 1][09] Build and baseline cdc locality and resynchronization #31

Closed

Description

@yifanxuaaa

Phase 1 completion amendment (2026-09-04)

The user requires functional failures to be repaired in Phase 1. The committed amendment supersedes earlier baseline-only closure and deferral wording below. Required functional, correctness, integrity, resource, safety-deadline and cleanup gates must pass before PHASE1_TERMINAL_PASS. Optional latency/storage optimization of correct operations remains Phase 2.

Preserve all original baseline failures and producing identities; record corrected candidate evidence separately. Fix shared causes and recollect only genuinely invalidated coverage. Do not shrink fixtures, omit cases, weaken oracles, raise budgets/timeouts or add benchmark-only bypasses. Keep affected family issues and #35 open until corrected execution and independent verification pass; #21 stays open for later phases/release.

Parent: #21
Depends on: #22 (shared infrastructure and committed specification).

Problem statement

LayerFS needs an authentic initial baseline for cdc locality and resynchronization before deciding product improvements.

Goal and family scope

Measure overwrite, insertion, deletion, common-body and scattered-change variants; run the separate CDC boundary proof.

Inventory: 20 new timed cases, 60 prescribed initial performance sample slots, and 1 standalone proof recipes. The canonical planning file is docs/roadmap/0.1/0.1.3/dedup-cdc-locality.md. The snapshot below makes the current proposal reviewable until its committed revision is published.

Stage 1 work

  • Freeze this family and its exact fixture/schedule/oracle identities under the shared contract; implement its complete membership in the existing harness with one canonical definition and thin adapter.
  • Qualify route, call counts, size/resource bounds, independent oracles and selected-case self-checks. Resolve harness defects before collecting baseline evidence.
  • Run one selected case/seed to establish a valid execution path, then collect every prescribed initial sample and separate proof variant on the pinned unoptimized product. Required extended cases are explicit but remain required.
  • Retain raw failures/timeouts as classified product findings. Keep setup, performance, verification, cleanup and external command wall distinct.
  • Produce a source-bound family report with all outcomes and measured questions for later optimization; do not tune product algorithms within this issue.

Acceptance criteria

  • Complete family implementation, unique IDs, exact counts, deterministic fixtures/schedules and independent oracle checks.
  • Committed specification and product/harness/build/runtime/input identities linked in the evidence manifest.
  • Every required case/subcase executed with a retained initial outcome; no missing mandatory counter, unimplemented member or silent skipped slot.
  • All prescribed valid sample repetitions retained, with separate independent verification of distinct input/schedule variants. Failed/invalid outcomes remain explicit and ineligible for performance claims.
  • Preparation reuse, per-sample isolation, byte limits and cleanup checked; actual first-use/cache-hit command wall reported.
  • Family baseline summary delivered to the consolidated report issue; known product failures remain failing release gates and are recorded for Stage 2.

Files to read

docs/roadmap/0.1/0.1.3/dedup-cdc-locality.md, docs/roadmap/0.1/0.1.3/testing-rules.md, docs/general/benchmark_rules.md, and the existing source/test references in the specification below.

Stage 1 boundary

Build the benchmark and retain its initial outcomes. Product algorithm, storage-format and performance optimization are deferred to Stage 2. Improve benchmark preparation/reuse and correct harness defects now; do not silently change the measured product to make results pass. Pin the unoptimized product identity and distinguish harness changes from product changes.

A slow result, product failure or timeout is an initial finding, not permission to lower a gate, drop a case or start tuning. Keep outcome, harness validity, verification, resources, cleanup and coverage completeness separate. Failed/invalid cases cannot support performance claims. Unimplemented cases, missing mandatory observations and unexecuted slots remain incomplete.

Shared execution rules

  • Reuse fs-bench-pro, existing generators, runner helpers, custody, qualified pristine inputs, independent writable samples and reports. Add only needed family adapters.
  • Use 1/10/100/500 of the declared unit; no Cartesian size/count matrix. Each workload file is at most 500 MiB and aggregate logical content is strictly below 1 GiB at every intermediate state, including temporary files and Git internals. Physical Store/spool/cache budgets are separately reported.
  • Resolve the selected case/seed before preparation. Reuse compatible artifacts; prepare only required inputs. Measured creation/import/history must still execute all work.
  • Iterate one selected case and seed; ordinary warm-prepared 1–5 second command wall is a target to qualify, not a blanket guarantee. Required extended cases are explicitly selected and cannot be omitted from complete baseline coverage.
  • Separate performance and verification. Added benchmark hashes, full manifests, reopen and fault injection stay outside performance timing; intrinsic CAS/Git hashing remains measured.
  • Collect several metrics from one execution and regenerate reports from raw evidence without rerunning successful workloads. Retain all valid results and failures, with exact source, fixture, oracle, runtime and cache identities.
Planning specification snapshot — dedup-cdc-locality.md

Source file: docs/roadmap/0.1/0.1.3/dedup-cdc-locality.md. Local draft SHA-256: 8a53de6bf394e9c4d29b9bd0932504a2400f3a8a7fe11bb653554f21ebc38f48. This snapshot is reviewable now; it is not a committed admission contract. The infrastructure issue must publish/freeze the source revision before implementation and collection.

CDC locality across independently written files

Status: Draft family dedup_cdc_locality: 20 new timed cases and one
proof-only case. No implementation or evidence is admitted by this document.

Question and shared rules

How much exact payload survives localized changes, shifted content, a shared
middle, and scattered damage when each file is independently imported?
Measure real CDC work and stored chunks rather than infer reuse from textual
similarity. This full-import family does not measure SDK partial-edit borrowing.

The shared testing rules (docs/roadmap/0.1/0.1.3/testing-rules.md; planning file awaiting publication) own common lifecycle, preparation,
sampling, caps, and timing. Reuse the fresh-import operation, independent file
materialization, complete payload accounting, and Store/resource receipts from
cross-file exact deduplication (docs/roadmap/0.1/0.1.3/dedup-cross-file.md; planning file awaiting publication). Exact-copy and unique
controls belong to that family; do not register duplicate controls here.

Exact membership

Every timed case imports one 1 MiB reference plus N = 1, 10, 100, 500
independently written variants. N counts variants, so the N=1 case already
compares two files. Each row below expands to four IDs with suffixes
-1, -10, -100, and -500.

Scenario prefix Variant of reference B File bytes
dedup-cdc-overwrite Replace 64 interior bytes with a nonzero deterministic XOR mask 1,048,576
dedup-cdc-insert Insert 4,096 deterministic bytes at an interior offset 1,052,672
dedup-cdc-delete Remove 4,096 bytes at an interior offset 1,044,480
dedup-cdc-common-body Unique 128 KiB prefix, B's middle 768 KiB, unique 128 KiB suffix 1,048,576
dedup-cdc-scattered Change one byte in every 4 KiB block, using distinct nonzero masks 1,048,576

There are 20 timed IDs and 60 timed samples, plus exactly
dedup-cdc-boundaries-proof. The reference is fixed inside each profile and
seed; lower tiers use exact prefixes of its sealed 500-variant schedule.
Profiles use fresh Stores and separate domain labels. Reject duplicate variant
digests and unintended equality with B during fixture qualification.

Fixture and execution

Reuse the existing bounded namespace stream and materialization machinery.
Seed framing must unambiguously bind shared seed, family, profile, reference,
variant ordinal, and byte/offset role. Interior offsets are at least 64 KiB
from each end and are chosen before collection; never select offsets using a
candidate's observed CDC behavior. Base and unchanged bytes are regenerated
for each output file; no copying, reflinking, hard-linking, sparse files, or
precomputed roots manufacture sharing.

The largest cohort is insertion at N=500:
1,048,576 + 500 × 1,052,672 = 527,384,576 bytes (502.953125 MiB).
Its largest file is 1 MiB + 4 KiB. All other cohorts are smaller. Stream
fixture generation and verification; do not materialize another complete
cohort or accumulate output copies inside the logical workload. Shared caps
apply at every intermediate state, not just after import.

Performance calls Client::initialize_layerstack from the independently
materialized directory into a fresh output Store, scanning every logical byte.
No edit follows merely to obtain a Commit. Time import acknowledgement and
declared product phases; collect exact canonical/FUSE reopen proof separately.
Reuse namespace runner, source sealing, cached pristine inputs, and result
machinery without copying a prepared Store over the measured import.

Exact locality oracle

Seal every reference/variant byte digest and its independent ordered FastCDC
transcript before candidate work. Read each completed file from its root and
compare every actual ObjectId, logical boundary, and decoded payload length.
Matching bytes are necessary but cannot replace chunk evidence.

For each variant report:

  • exact reference/variant shared chunk IDs and logical coverage;
  • candidate, inserted, reused, and unique file-payload bytes;
  • common-prefix and common-suffix chunk counts/bytes;
  • for insertion/deletion, base and variant logical positions where the common
    suffix resumes, and distance from the respective edit boundary; and
  • not-found when no shared suffix exists, never a misleading zero distance.

Report per-variant and whole-cohort equations separately: repeated chunks among
variants can improve whole-cohort reuse even when reference coverage is worse.
The common-body case must establish sharing inside the middle, not merely
reuse of prefixes accidentally identical between variants. Scattered edits are
the negative control for fuzzy similarity; use the exact qualified transcript
rather than claim "no sharing" solely from the mask schedule.

No historical 5 MB-file 85%/95% thresholds carry over. Freeze percentage and
resynchronization gates from fixture qualification/untouched baseline before
candidate optimization. Retain a reference limitation if a seeded input does
not resynchronize within the hoped-for window; do not search for a friendlier
seed after a candidate miss. The frozen CDC profile itself is unchanged.

Boundary proof

dedup-cdc-boundaries-proof executes once and includes all three seed cohorts,
separately attributed. For lengths 0, 1, 8,191, 8,192, 16,384, 32,768,
32,769 bytes
, independently materialize an exact pair; for nonempty lengths
also materialize a one-byte mutation at length / 2.

Require exact bytes, matching transcripts for each pair, all chunk lengths
nonzero and at most 32,768, total lengths equal file lengths, no chunks for the
empty input, and at least two chunks for 32,769 bytes. One-byte mutations of
inputs no longer than 8,192 bytes cannot retain that input's sole payload
chunk. Other sharing follows the sealed oracle rather than a guessed bound.
Fresh canonical and FUSE reopen must preserve every cohort.

Reuse the existing CDC unit checks for fragmented reads and frozen profile
boundaries. Extend their evidence receipt into this public import/reopen proof;
do not create a new timed family or one latency row per input fragmentation.
The proof has correctness/resource results and its own wall, no latency
distribution or extra timed membership.

Evidence and completion

Retain full regular-file payload transcripts, canonical namespace/metadata
bytes, logical scan versus unique insertion throughput, actual SQLite
length/allocation/live-page deltas, CPU/memory/I/O/spool peaks, transactions,
and cleanup under the shared rules. Neither monitor saved_fraction nor
physical_bytes replaces these receipts. No verification, digest construction,
failure injection, or FUSE readback runs inside the import timer.

One fresh performance sample per seed gives 60 rows. Use a small selected case
for a 1–5 second development aspiration after cached preparation; qualify all
four tiers with honest complete import work and frozen family budgets. No new
runner framework, Store schema, CDC profile, or object identity is authorized.

Source references

Activity

  1. self-assigned this
    on Sep 4, 2026
  2. yifanxuaaa commented on Sep 4, 2026

    @yifanxuaaa
    ContributorAuthor

    Performance-results checkpoint for #31: 60 independently validated active VM8 performance samples. The report validates 0 retained full proof(s) for this family at this checkpoint; further full checks are running. This comment does not claim global Phase1 terminal pass.

    Published report, exact rows, source identities and verification status, phase/counter evidence.

    VM8 performance source: 7948df2de269e5ffd47a232ffd8091ff83f8869f; fixed 8-CPU Docker VM, unchanged 2-CPU/2-GiB benchmark-container caps. No VM4/VM8 timing pooling. Every listed performance row passed the independent authenticity, custody, observation, resource and cleanup validation; workload admission still requires its separate full proof.

    Values below are medians across the prescribed three seeds. Product time is the declared cumulative public-operation time; preparation and complete sample wall are separate. Store growth is the observed physical Store-file delta, not a deduplication ratio or verified canonical byte count.

    Case n Product ms Preparation s Complete sample s Store delta MiB Full proofs in checkpoint
    dedup-cdc-common-body-1 3 4.953 3.860 4.211 1.562 0/3
    dedup-cdc-common-body-10 3 21.993 3.958 4.321 4.562 0/3
    dedup-cdc-common-body-100 3 268.495 4.941 5.642 33.812 0/3
    dedup-cdc-common-body-500 3 1300.897 11.517 13.817 161.688 0/3
    dedup-cdc-delete-1 3 5.027 3.836 4.193 1.250 0/3
    dedup-cdc-delete-10 3 19.997 3.941 4.310 1.688 0/3
    dedup-cdc-delete-100 3 169.896 4.095 4.643 4.750 0/3
    dedup-cdc-delete-500 3 910.513 8.867 10.264 15.938 0/3
    dedup-cdc-insert-1 3 5.150 3.781 4.109 1.250 0/3
    dedup-cdc-insert-10 3 20.371 2.361 2.710 1.812 0/3
    dedup-cdc-insert-100 3 171.250 5.871 6.429 5.562 0/3
    dedup-cdc-insert-500 3 915.639 11.594 12.969 18.500 0/3
    dedup-cdc-overwrite-1 3 4.834 5.029 5.390 1.250 0/3
    dedup-cdc-overwrite-10 3 19.598 3.765 4.117 1.438 0/3
    dedup-cdc-overwrite-100 3 170.187 4.770 5.334 4.000 0/3
    dedup-cdc-overwrite-500 3 895.133 8.234 9.591 14.938 0/3
    dedup-cdc-scattered-1 3 5.955 5.741 6.108 2.438 0/3
    dedup-cdc-scattered-10 3 56.221 4.002 4.399 13.188 0/3
    dedup-cdc-scattered-100 3 289.743 5.025 5.967 120.625 0/3
    dedup-cdc-scattered-500 3 1681.869 8.105 10.705 598.750 0/3

    The separate CDC boundary aggregate proof is still required; the60 timing samples do not replace it.

    All prior actual failures, source incompatibilities, invalidations and preparation interruptions remain in the linked report and raw evidence. The fast development verifier contributes zero full-verification credit. Completed family tickets will close only after their remaining mandatory full checks pass; #21 remains open for later phases.

  3. yifanxuaaa commented on Sep 4, 2026

    @yifanxuaaa
    ContributorAuthor

    The user has amended Phase 1 acceptance: qualified fast verification is sufficient for routine benchmark verification. The published amendment supersedes earlier comments requiring every routine exhaustive proof before closure. The reference qualification contract preserves independent expected states, affected-content/metadata/alias/absence checks, authenticated references and deterministic witnesses.

    The posted performance results remain valid at their original source/environment identities. Completed full proofs remain accepted and are reused. At the scope checkpoint,48 routine slots already had full proofs; the305 unrun routine exhaustive counterparts are now deferred to Phase2 while their qualified fast checks are implemented and collected. Fast evidence keeps its actual assurance and explicit skipped coverage; it will not be labelled fully_verified.

    Required targeted reliability checks, the CDC boundary proof, exact expected-error oracles, resource and cleanup gates still apply. Observed failures still require focused repair confirmation. The already-qualified600-second proof is retained rather than repeated. This amendment does not manufacture missing evidence or itself establish terminal pass.

    Remaining routine families will close after their accepted fast/full coverage and applicable gates pass. #21 remains open for later phases and release.

  4. yifanxuaaa commented on Sep 4, 2026

    @yifanxuaaa
    ContributorAuthor

    Phase 1 #31 is complete under the approved verification amendment: 60 qualified performance samples, 60 accepted routine fast-verification results, and the mandatory full CDC boundary proof. All 20 timed cases have all three seeds; this family has no suppressed cases or missing slots. The selected evidence has zero validation issues or resource violations.

    The routine results retain fast_iteration_verified assurance and fully_verified=false. Their receipts state which current bytes and metadata were checked and which work was omitted; they do not claim a complete canonical-object census. The separate dedup-cdc-boundaries-proof is fully verified on source 30d13dee and covers the frozen 60 boundary files.

    Performance medians below are public initialization time. Most timings retain their qualified 7948df2d source. The corrected deletion-500 seed-3 sample uses e24a3b34 and is shown separately; no median pools different sources or the invalid old fixture. Fast coverage is 3/3 for every exact case: 23 retained results on e0922904, one corrected deletion result and 36 remaining results on e24a3b34.

    Case Performance seeds n Initialize median (ms) Performance source Fast accepted for case
    dedup-cdc-common-body-1 1, 2, 3 3 4.953 7948df2d 3/3
    dedup-cdc-common-body-10 1, 2, 3 3 21.993 7948df2d 3/3
    dedup-cdc-common-body-100 1, 2, 3 3 268.495 7948df2d 3/3
    dedup-cdc-common-body-500 1, 2, 3 3 1300.897 7948df2d 3/3
    dedup-cdc-delete-1 1, 2, 3 3 5.027 7948df2d 3/3
    dedup-cdc-delete-10 1, 2, 3 3 19.997 7948df2d 3/3
    dedup-cdc-delete-100 1, 2, 3 3 169.896 7948df2d 3/3
    dedup-cdc-delete-500 1, 2 2 910.474 7948df2d 3/3
    dedup-cdc-delete-500 3 1 888.226 e24a3b34 3/3
    dedup-cdc-insert-1 1, 2, 3 3 5.150 7948df2d 3/3
    dedup-cdc-insert-10 1, 2, 3 3 20.371 7948df2d 3/3
    dedup-cdc-insert-100 1, 2, 3 3 171.250 7948df2d 3/3
    dedup-cdc-insert-500 1, 2, 3 3 915.639 7948df2d 3/3
    dedup-cdc-overwrite-1 1, 2, 3 3 4.834 7948df2d 3/3
    dedup-cdc-overwrite-10 1, 2, 3 3 19.598 7948df2d 3/3
    dedup-cdc-overwrite-100 1, 2, 3 3 170.187 7948df2d 3/3
    dedup-cdc-overwrite-500 1, 2, 3 3 895.133 7948df2d 3/3
    dedup-cdc-scattered-1 1, 2, 3 3 5.955 7948df2d 3/3
    dedup-cdc-scattered-10 1, 2, 3 3 56.221 7948df2d 3/3
    dedup-cdc-scattered-100 1, 2, 3 3 289.743 7948df2d 3/3
    dedup-cdc-scattered-500 1, 2, 3 3 1681.869 7948df2d 3/3

    The original deletion-500 seed-3 fixture mapped ordinals 74 and 471 to offset 359004, yielding duplicate complete variants. The independent uniqueness gate correctly rejected fast attempt bf6d01cc7939. The documented, sealed deletion-only correction preserves the first offset and probes later collisions; only ordinal 471 moves to 359005. The corrected fixture preserves 501 files and 523,288,576 bytes. Its one replacement performance sample and fast check both passed on the same new input. The original fast failure and old recipe-invalid performance remain preserved; neither receives corrected coverage credit.

    Evidence: sealed family receipt and all 60 performance/fast input/environment/source joins, 36 newly qualified fast results, actual checked/skipped coverage, corrected input and pair, and fully qualified boundary proof.

    The collision finding and source map limit changed-input invalidation to deletion-500 seed 3. The fast-verification amendment governs routine acceptance; the CDC transcript/uniqueness and targeted boundary requirements remain enforced. Prior qualified evidence was reused without replay, and all actual failures remain visible.

    Close #31 for the amended Phase 1 scope. Central issue #21 remains open for remaining family work, Phase 2 optimization and later exhaustive/release qualification.

  5. yifanxuaaa commented on Sep 4, 2026

    @yifanxuaaa
    ContributorAuthor

    Phase 1 is closed under the user's explicit withdrawal of all remaining verification tests. This is a scope withdrawal, NOT a claim that the original verification terminal criteria passed.

    The assistant designed an excessively expensive verification suite and spent too long expanding it after the user requested a much faster finish. That design is withdrawn and must be reassessed in Phase 2. No more verification work is scheduled.

    Final performance results:370 independently qualified active observations, with actual source identities and unoptimized timings. All15 suppressed case definitions, original failures and incomplete attempts remain preserved. Completed29 targeted proofs,48 routine full proofs and qualified fast evidence remain historical evidence; unrun verification is withdrawn/deferred, never relabeled passing.

    Explicit withdrawal and accountability record. Separate Phase 2 backlog. Central#21 stays open for Phase 2/release. No release tag was created.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

new featureA new user-facing capabilityv0.1.3LayerFS v0.1.3 checkpoint and delivery

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions