Skip to content

[v0.1.3][Stage 1][08] Build and baseline cross-file cas deduplication #30

Closed

Description

@yifanxuaaa

Phase 1 completion amendment (2026-09-04)

The user requires functional failures to be repaired in Phase 1. The committed amendment supersedes earlier baseline-only closure and deferral wording below. Required functional, correctness, integrity, resource, safety-deadline and cleanup gates must pass before PHASE1_TERMINAL_PASS. Optional latency/storage optimization of correct operations remains Phase 2.

Preserve all original baseline failures and producing identities; record corrected candidate evidence separately. Fix shared causes and recollect only genuinely invalidated coverage. Do not shrink fixtures, omit cases, weaken oracles, raise budgets/timeouts or add benchmark-only bypasses. Keep affected family issues and #35 open until corrected execution and independent verification pass; #21 stays open for later phases/release.

Parent: #21
Depends on: #22 (shared infrastructure and committed specification).

Problem statement

LayerFS needs an authentic initial baseline for cross-file cas deduplication before deciding product improvements.

Goal and family scope

Measure unique, identical and mixed-content imports; use one shared one-file anchor and exact payload/canonical/SQLite accounting.

Inventory: 10 new timed cases, 30 prescribed initial performance sample slots, and 0 standalone proof recipes. The canonical planning file is docs/roadmap/0.1/0.1.3/dedup-cross-file.md. The snapshot below makes the current proposal reviewable until its committed revision is published.

Stage 1 work

  • Freeze this family and its exact fixture/schedule/oracle identities under the shared contract; implement its complete membership in the existing harness with one canonical definition and thin adapter.
  • Qualify route, call counts, size/resource bounds, independent oracles and selected-case self-checks. Resolve harness defects before collecting baseline evidence.
  • Run one selected case/seed to establish a valid execution path, then collect every prescribed initial sample and separate proof variant on the pinned unoptimized product. Required extended cases are explicit but remain required.
  • Retain raw failures/timeouts as classified product findings. Keep setup, performance, verification, cleanup and external command wall distinct.
  • Produce a source-bound family report with all outcomes and measured questions for later optimization; do not tune product algorithms within this issue.

Acceptance criteria

  • Complete family implementation, unique IDs, exact counts, deterministic fixtures/schedules and independent oracle checks.
  • Committed specification and product/harness/build/runtime/input identities linked in the evidence manifest.
  • Every required case/subcase executed with a retained initial outcome; no missing mandatory counter, unimplemented member or silent skipped slot.
  • All prescribed valid sample repetitions retained, with separate independent verification of distinct input/schedule variants. Failed/invalid outcomes remain explicit and ineligible for performance claims.
  • Preparation reuse, per-sample isolation, byte limits and cleanup checked; actual first-use/cache-hit command wall reported.
  • Family baseline summary delivered to the consolidated report issue; known product failures remain failing release gates and are recorded for Stage 2.

Files to read

docs/roadmap/0.1/0.1.3/dedup-cross-file.md, docs/roadmap/0.1/0.1.3/testing-rules.md, docs/general/benchmark_rules.md, and the existing source/test references in the specification below.

Stage 1 boundary

Build the benchmark and retain its initial outcomes. Product algorithm, storage-format and performance optimization are deferred to Stage 2. Improve benchmark preparation/reuse and correct harness defects now; do not silently change the measured product to make results pass. Pin the unoptimized product identity and distinguish harness changes from product changes.

A slow result, product failure or timeout is an initial finding, not permission to lower a gate, drop a case or start tuning. Keep outcome, harness validity, verification, resources, cleanup and coverage completeness separate. Failed/invalid cases cannot support performance claims. Unimplemented cases, missing mandatory observations and unexecuted slots remain incomplete.

Shared execution rules

  • Reuse fs-bench-pro, existing generators, runner helpers, custody, qualified pristine inputs, independent writable samples and reports. Add only needed family adapters.
  • Use 1/10/100/500 of the declared unit; no Cartesian size/count matrix. Each workload file is at most 500 MiB and aggregate logical content is strictly below 1 GiB at every intermediate state, including temporary files and Git internals. Physical Store/spool/cache budgets are separately reported.
  • Resolve the selected case/seed before preparation. Reuse compatible artifacts; prepare only required inputs. Measured creation/import/history must still execute all work.
  • Iterate one selected case and seed; ordinary warm-prepared 1–5 second command wall is a target to qualify, not a blanket guarantee. Required extended cases are explicitly selected and cannot be omitted from complete baseline coverage.
  • Separate performance and verification. Added benchmark hashes, full manifests, reopen and fault injection stay outside performance timing; intrinsic CAS/Git hashing remains measured.
  • Collect several metrics from one execution and regenerate reports from raw evidence without rerunning successful workloads. Retain all valid results and failures, with exact source, fixture, oracle, runtime and cache identities.
Planning specification snapshot — dedup-cross-file.md

Source file: docs/roadmap/0.1/0.1.3/dedup-cross-file.md. Local draft SHA-256: 4490332d328e591f7b514206c1d987954b39fe0550b807bb2bdbb4407670cf20. This snapshot is reviewable now; it is not a committed admission contract. The infrastructure issue must publish/freeze the source revision before implementation and collection.

Cross-file exact-content deduplication

Status: Draft family dedup_cross_file: 10 new timed cases, no
proof-only cases. No implementation or evidence is admitted by this document.

Question and scope

When independently written files contain unique bytes, identical bytes, or a
mixture, does one fresh import store exactly the required distinct payload?
Measure the CPU, reads, canonical objects, and durable Store bytes required to
obtain that reuse. This is an initialization family, not an SDK edit family.

The shared testing rules (docs/roadmap/0.1/0.1.3/testing-rules.md; planning file awaiting publication) own custody, size caps, timing,
sampling, preparation reuse, and admission. The parent inventory (docs/roadmap/0.1/0.1.3/README.md; planning file awaiting publication)
owns release totals. CDC locality (docs/roadmap/0.1/0.1.3/dedup-cdc-locality.md; planning file awaiting publication),
preexisting Workspace reuse (docs/roadmap/0.1/0.1.3/dedup-workspace-reuse.md; planning file awaiting publication), and
retained Branch history (docs/roadmap/0.1/0.1.3/dedup-branch-history.md; planning file awaiting publication) own different dedup questions.

The four released init_namespace cases and three store_footprint controls
remain inherited evidence with their exact released identities and sizes.
They are not members of this new family or replacements for its controls.
Retain synthetic-small-heavy-v2, namespace-file-digest-tree-v2, and
fs-bench-pro-namespace-v3 when rerunning those namespace cases; historical
namespace-v1 and these new dedup rows must not be pooled with them.

Exact membership

One unit is one independently materialized 1,048,576-byte file. Use the
explicit tiers 1, 10, 100, 500; smaller fixtures are prefixes of the same
500-file fixture for a seed. Each profile runs in its own fresh output Store.

Scenario IDs N Relationship Required result
dedup-cross-file-anchor-1 1 One common first file, shared by all three curves One payload, no unintended internal duplicate chunks
dedup-cross-file-unique-10, -100, -500 10 / 100 / 500 Every file has unrelated deterministic bytes No unintended inter-file payload-ID intersection
dedup-cross-file-identical-10, -100, -500 10 / 100 / 500 Every file has the anchor's exact bytes Identical ordered chunk transcripts and file content roots
dedup-cross-file-mixed-10, -100, -500 10 / 100 / 500 Repeated four-file block: pair, pair, unique, unique Exactly the frozen mixture of duplicate and distinct payload

The mixed block uses one new base for its first two files and distinct bases
for its last two; no base is shared between blocks. The first base is the
anchor. Thus distinct complete file payloads number 7 / 75 / 375 at the three
mixed tiers. Qualify no unintended chunk intersections within or between
distinct bases before admission. The identical profile has exactly one MiB of
distinct file payload, and its intrinsic payload savings are (N - 1) / N.
The mixed profile's exact fractions follow its sealed chunk set, not a rounded
label such as "50% duplicates."

There are 10 distinct timed IDs and 30 timed samples. The N=1 anchor is
collected once per seed and referenced by all three curves; it is not three
separate samples or three family members. The shorthand suffixes in the table
expand only with their row's full prefix.

Fixture, operation, and bounds

Reuse the existing namespace generator's bounded stream and fixture writer,
sdk_edit_common.rs size labels/fixture helpers where applicable, namespace
runner, custody, result adapters, and prepared runtime. Freeze one
domain-separated seed for each distinct base using the three shared seed
labels. Do not replace the existing stream with per-block cryptographic hashes.
The maximum fixture is 500 MiB; files are always exactly 1 MiB.

Create every input file independently with ordinary writes, regenerating its
deterministic bytes. Do not hard-link, reflink, clone, create sparse holes,
compress, reuse a backing file, or supply precomputed product roots. Record
distinct source inodes, link count one, logical bytes, bytes written, allocation,
exact modes/mtimes, path manifests, and hashes. Payload buffers are bounded;
fixtures and oracles are prepared outside product timing.

The measured entrypoint is Client::initialize_layerstack with
LayerStackInitialization::Directory. It must read and CDC-scan every source byte
into a fresh output Store; cached input files cannot replace measured
import with a copied prepared output Store. Record expected and observed source
reads and scanned bytes. No artificial post-import mutation is needed.

Performance mode times initialization acknowledgement and named product
phases only. Separate verification opens the exact genesis root from a fresh
Client, creates one Branch and real-FUSE Workspace for readback, verifies every
path, and ends it cleanly. A verification UpToDate acknowledgement is not an
extra initialization timing sample. No Add, competing Branch, or history loop
belongs here.

Exact payload and storage accounting

CandidateReceipt and monitor saved_fraction alone cannot measure fresh-import
dedup: duplicate emissions may be coalesced before the receipt. Monitor
physical_bytes describes encoded canonical bytes, not SQLite allocation.
Retain both existing fields, but derive the claim from the complete transcript.

For every regular-file payload occurrence record path, ordinal, logical
offset/length, payload chunk ObjectId and full payload length, and any source
offset. Let L be logical bytes, U the distinct reachable file-payload chunk
IDs, P those durably present before the operation, and I = U - P. Let U_B
and I_B sum each distinct chunk's raw payload length once. Record integers
before percentages:

intrinsic payload savings = 1 - U_B / L
incremental payload savings = 1 - I_B / L
sharing factor = L / U_B
preexisting logical coverage = referenced logical bytes backed by P / L

For fresh import P is empty for regular-file payload and every file's full
input is scanned. File extents must cover full chunks; exact decoded payload
lengths sum to file sizes. Qualify source fixtures so accidental duplication
cannot falsify a unique control. Authenticate canonical bytes for matching IDs;
a collision is an integrity failure. Keep namespace and metadata chunks out of
the regular-file payload denominator even when they use the same object kind.

Also retain ordered chunk-transcript and unique-set digests, canonical file
roots, canonical metadata/namespace object counts and bytes, candidate,
inserted, emitted-preexisting-reuse, and borrowed object/byte counts. Distinguish:

  • duplicate occurrences within this import;
  • chunks emitted now that already existed before the operation; and
  • unchanged extents borrowed without re-emission.

The last category is zero for full fresh import. For incremental families,
source offsets and referenced lengths matter: partial extents can reference a
larger stored chunk, so U_B need not equal logical referenced bytes. Report
that retained payload cost without pretending it is zero-copy CAS insertion.

At start and after completion/reconnect, record actual database length and
allocated blocks, SQLite page size/count/freelist and live page bytes, canonical
bytes by role, and the Store file census. Use signed checked deltas. SQLite
growth includes namespace, metadata, indexes, and page slack and is not itself
a payload dedup fraction. Temporary WAL/journal, spools, and fixture files are
reported separately; terminal durable inventory must obey the released Store
contract without an added persistent dedup sidecar.

Verification, resources, and admission

Seal expected bytes, metadata and full CDC transcripts before candidate work.
The verifier independently walks each result file's rope from the exact root,
authenticates referenced objects, and compares the ordered transcript and
distinct set. Fresh FUSE readback verifies complete bytes, path cardinality,
modes/mtimes, no extra paths, and cleanup. A Store-wide search for chunk-shaped
objects is not a substitute for traversing regular-file payloads.

Record initialization phases, logical ingest and unique insertion rates,
CPU, RSS/cgroup peaks, swaps/OOM, source and Store I/O, CDC scans, transactions,
temporary passes and peak spool bytes, Store growth, and cleanup. A high logical
ingest rate is not a claim of equal physical write throughput.

Run one fresh performance sample per shared seed, with separate verifier
executions under the shared rules. Reuse pristine input/runtime preparation;
preserve independent measured Stores. A selected small case should target the
shared 1–5 second development loop after preparation, but qualification budgets
must account for reading all 500 MiB. Freeze exact correctness/resource gates
before baseline and performance budgets before candidate optimization. Retain
all valid baseline outcomes; no unqualified percent or copied earlier timing
can satisfy admission.

Implementation remains in the existing fs-bench-pro crate and namespace
runner where applicable. Preserve the existing runner's frozen all meaning;
add explicit new family selection rather than redefining inherited membership.

Source references

Activity

  1. self-assigned this
    on Sep 4, 2026
  2. yifanxuaaa commented on Sep 4, 2026

    @yifanxuaaa
    ContributorAuthor

    Performance-results checkpoint for #30: 30 independently validated active VM8 performance samples. The report validates 0 retained full proof(s) for this family at this checkpoint; further full checks are running. This comment does not claim global Phase1 terminal pass.

    Published report, exact rows, source identities and verification status, phase/counter evidence.

    VM8 performance source: 7948df2de269e5ffd47a232ffd8091ff83f8869f; fixed 8-CPU Docker VM, unchanged 2-CPU/2-GiB benchmark-container caps. No VM4/VM8 timing pooling. Every listed performance row passed the independent authenticity, custody, observation, resource and cleanup validation; workload admission still requires its separate full proof.

    Values below are medians across the prescribed three seeds. Product time is the declared cumulative public-operation time; preparation and complete sample wall are separate. Store growth is the observed physical Store-file delta, not a deduplication ratio or verified canonical byte count.

    Case n Product ms Preparation s Complete sample s Store delta MiB Full proofs in checkpoint
    dedup-cross-file-anchor-1 3 4.171 3.497 3.823 1.188 0/3
    dedup-cross-file-identical-10 3 17.702 3.751 4.199 1.188 0/3
    dedup-cross-file-identical-100 3 156.959 5.688 6.187 1.250 0/3
    dedup-cross-file-identical-500 3 808.296 12.257 13.554 1.250 0/3
    dedup-cross-file-mixed-10 3 23.254 4.085 4.493 8.375 0/3
    dedup-cross-file-mixed-100 3 203.634 7.765 8.385 89.375 0/3
    dedup-cross-file-mixed-500 3 1056.044 11.729 13.636 447.000 0/3
    dedup-cross-file-unique-10 3 21.058 3.749 4.099 11.875 0/3
    dedup-cross-file-unique-100 3 176.675 4.609 5.183 119.500 0/3
    dedup-cross-file-unique-500 3 941.368 8.322 10.672 595.562 0/3

    All prior actual failures, source incompatibilities, invalidations and preparation interruptions remain in the linked report and raw evidence. The fast development verifier contributes zero full-verification credit. Completed family tickets will close only after their remaining mandatory full checks pass; #21 remains open for later phases.

  3. yifanxuaaa commented on Sep 4, 2026

    @yifanxuaaa
    ContributorAuthor

    The user has amended Phase 1 acceptance: qualified fast verification is sufficient for routine benchmark verification. The published amendment supersedes earlier comments requiring every routine exhaustive proof before closure. The reference qualification contract preserves independent expected states, affected-content/metadata/alias/absence checks, authenticated references and deterministic witnesses.

    The posted performance results remain valid at their original source/environment identities. Completed full proofs remain accepted and are reused. At the scope checkpoint,48 routine slots already had full proofs; the305 unrun routine exhaustive counterparts are now deferred to Phase2 while their qualified fast checks are implemented and collected. Fast evidence keeps its actual assurance and explicit skipped coverage; it will not be labelled fully_verified.

    Required targeted reliability checks, the CDC boundary proof, exact expected-error oracles, resource and cleanup gates still apply. Observed failures still require focused repair confirmation. The already-qualified600-second proof is retained rather than repeated. This amendment does not manufacture missing evidence or itself establish terminal pass.

    Remaining routine families will close after their accepted fast/full coverage and applicable gates pass. #21 remains open for later phases and release.

  4. yifanxuaaa commented on Sep 4, 2026

    @yifanxuaaa
    ContributorAuthor

    Phase 1 #30 is complete under the approved verification amendment: 30 qualified performance samples and 30 accepted routine fast-verification results across all ten timed IDs and three seeds. This family has no suppressed cases or missing slots. The selected evidence has zero validation issues or resource violations.

    The N=1 anchor remains one shared control per seed; it is not counted three times for the exact/mixed/unique profiles. Public initialization reads ordinary independently generated input files, preserving the frozen routes and fixture sizes. The independent deduplication checks retain the exact-copy root/chunk equations, mixed mapping, and unique negative controls.

    All performance observations retain their qualified source 7948df2d. Fast verification retains 26 qualified results on e0922904 plus four on e24a3b34. Each performance/fast pair has matching input and VM8 environment identity and an explicit source/recipe bridge. Fast assurance is fast_iteration_verified; fully_verified=false. No exhaustive full-verification credit is claimed.

    Case Performance n Initialize median (ms) Performance source Accepted fast Actual fast source
    dedup-cross-file-anchor-1 3 4.171 7948df2d 3/3 seeds 1/2/3: e0922904
    dedup-cross-file-identical-10 3 17.702 7948df2d 3/3 seeds 1/2/3: e0922904
    dedup-cross-file-identical-100 3 156.959 7948df2d 3/3 seeds 1/2/3: e0922904
    dedup-cross-file-identical-500 3 808.296 7948df2d 3/3 seeds 1/2/3: e0922904
    dedup-cross-file-mixed-10 3 23.254 7948df2d 3/3 seeds 1/2/3: e0922904
    dedup-cross-file-mixed-100 3 203.634 7948df2d 3/3 seeds 1/2/3: e0922904
    dedup-cross-file-mixed-500 3 1056.044 7948df2d 3/3 seeds 1/2/3: e0922904
    dedup-cross-file-unique-10 3 21.058 7948df2d 3/3 seeds 1/2/3: e0922904
    dedup-cross-file-unique-100 3 176.675 7948df2d 3/3 seed 1: e0922904; seed 2: e0922904; seed 3: e24a3b34
    dedup-cross-file-unique-500 3 941.368 7948df2d 3/3 seeds 1/2/3: e24a3b34

    Fast receipts preserve the actual checked and skipped scope, including complete current namespace authentication, independent selected content and native checks, and deferred full canonical census. No preparation manifest is relabeled as a full native proof. The original bcde3546377c cache-capacity attempt remains not-run: it never became a successful product execution. Its later successful replacement has its own evidence identity.

    Evidence: sealed family completion and all 30 source/input/environment joins, four newly qualified fast rows, new checked/skipped coverage, 26 retained fast results, and source-separated performance medians.

    The source map and fast-verification amendment define the exact retention and routine-acceptance scope. All prior passing verdicts were reused without replay. Actual failed or unexecuted attempts remain visible, and no old/new performance-cost equivalence is claimed.

    Close #30 for the amended Phase 1 scope. Central issue #21 remains open for remaining family work, Phase 2 optimization and later exhaustive/release qualification.

  5. yifanxuaaa commented on Sep 4, 2026

    @yifanxuaaa
    ContributorAuthor

    Phase 1 is closed under the user's explicit withdrawal of all remaining verification tests. This is a scope withdrawal, NOT a claim that the original verification terminal criteria passed.

    The assistant designed an excessively expensive verification suite and spent too long expanding it after the user requested a much faster finish. That design is withdrawn and must be reassessed in Phase 2. No more verification work is scheduled.

    Final performance results:370 independently qualified active observations, with actual source identities and unoptimized timings. All15 suppressed case definitions, original failures and incomplete attempts remain preserved. Completed29 targeted proofs,48 routine full proofs and qualified fast evidence remain historical evidence; unrun verification is withdrawn/deferred, never relabeled passing.

    Explicit withdrawal and accountability record. Separate Phase 2 backlog. Central#21 stays open for Phase 2/release. No release tag was created.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

new featureA new user-facing capabilityv0.1.3LayerFS v0.1.3 checkpoint and delivery

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions