Skip to content

storage: spike the replace commit route (decision D1) #72

Closed

Description

@codecapo

Decide how Infostash commits a snapshot that replaces rows, through a spike time-boxed to one day whose prototype code is thrown away, and record the chosen route in a new ADR; the production replace_if_head is #88, and the answer decides whether #35, @full_refresh and the daily_revenue example behind #8 and #16 can ship in 0.1.0. Infostash at a0b5e6c pins iceberg-rust to rev 5b82ab9 (Cargo.toml:29-30), both Warehouse::append and append_if_head commit through fast_append (crates/storage/src/lib.rs:143-148, :205-211), and the storage tests add an Overwrite snapshot to the metadata by hand because nothing public on that pin makes one (:682-696). Draft RFC 0001 needs a replace commit for @unique_key (docs/rfcs/0001-project-files.md:118) and for @full_refresh (:121), which the ready/daily_revenue aggregate (:101-107) needs to keep one row per day; a partition-scoped overwrite, which recomputing only the changed partitions would need on top of partitioned writes (#36), is Phase 1 at the earliest and out of scope here. The route to try first writes the Overwrite snapshot through iceberg-rust's public spec writers and commits it with Infostash's own compare-and-swap on the table's row in catalog.db (the spikes recorded in #19 found that copy-on-write merges work but need such a compare-and-swap); a fork through [patch] comes second if that proves heavier, and an append-only 0.1.0 is the exit. Whichever route wins must keep ADR 0006: one atomic commit per output table carrying the job id, and a head check that fails rather than rebasing (docs/adr/0006-ordered-job-commits.md:39-40, :52-54).

Where: crates/storage (the throwaway prototype and its tests), docs/adr/ (up to two new ADRs and their index rows)

Done when

  • cargo tree -d is run before any prototype code is written, and the new ADR records whether the workspace builds more than one copy of a crate the route uses directly, such as iceberg (docs/adr/0002-pin-iceberg-rust-from-git.md:20-22) or sqlx.
  • A prototype in crates/storage that uses only public iceberg-rust APIs rewrites each manifest that lists a removed data file, marking that file DELETED and keeping the manifest's other files as EXISTING entries, puts the new files in a manifest of ADDED entries, and writes a manifest list, an Overwrite snapshot whose total-* summary fields it computes itself, and the next metadata.json through TableMetadata::write_to.
  • The prototype checks the table UUID and the expected head with TableRequirement::check against the metadata it loaded, then commits by swapping the table's metadata_location in catalog.db with a one-row compare-and-swap like the SQL catalog's own, through the sqlx dependency the crate declares but never imports (crates/storage/Cargo.toml:18-20, :30-32), maps zero rows updated to Error::Conflict with no retry or rebase, and writes infostash.job-id and the checkpoint properties in that same commit (docs/adr/0006-ordered-job-commits.md:39-40, :49-51).
  • Tests on the prototype show that two concurrent replaces on one head give one commit and one Conflict with the loser's files referenced by no snapshot, that the table keeps its UUID and snapshot history, that a reader holding the old metadata still reads the old files, and that the refusal at crates/storage/src/lib.rs:571-576 still fires on a real overwrite.
  • The new ADR, at the next free number (docs/adr/README.md:25) and listed in the index at docs/adr/README.md:46-55, records the chosen route and its costs, which for the hand-built commit are SQLite only, coupling to the SQL catalog's private iceberg_tables layout, a connection pool of its own, a copy of the catalog's record_type_filter, working outside the Catalog trait, and a redo for the Phase 1 REST catalog (ROADMAP.md:23).
  • If the fork is chosen, the new ADR supersedes ADR 0002 (0002 itself changes only on its status line, docs/adr/README.md:19-21), frames the fork as a bridge until the change lands upstream (docs/adr/0001-iceberg-datafusion-arrow.md:19-20), and requires the fork to process delete entries rather than only expose hooks.
  • If append-only is chosen, the new ADR lists the follow-up work, namely rewriting RFC 0001's clean/orders and daily_revenue examples (docs/rfcs/0001-project-files.md:87-107), making @unique_key and @full_refresh unknown-key errors (:114), and correcting README.md:21, which says Clean tables are deduplicated (README.md:23 and docs/architecture.md:68-69 are corrected in docs: fix README, architecture and ROADMAP statements the code does not back #94 on either route), and says whether an append-only versioned Clean with read-time dedup is acceptable in place of the merge.
  • Unless append-only is chosen, a second short ADR, written after the first, records the rule for incremental readers of a replaced table: before creating a job, the scheduler checks the operations in each input's pinned range, pins from = None for any input holding a non-append snapshot, reports the fallback and replaces the target rather than appending, as a new job with newly pinned inputs (docs/adr/0006-ordered-job-commits.md:47-48, :71-72); the refusal at crates/storage/src/lib.rs:571-576 stays as a non-retryable guard; and until a replace commit exists, the compile-time rule that docs: revise RFC 0001 with the incremental rules (decision D2) #73 writes into RFC 0001 and compiler: add an incremental eligibility classifier with a function allow-list #84 enforces (no incremental model reads a table declaring @unique_key or @full_refresh) applies instead, after which the plan-time check replaces it.

Points: 5 (D1)
Time-box: one day. When it runs out, the ADR records how far each route got and which one continues; if neither route has committed a replace by then, append-only is the route for 0.1.0.

Depends on: nothing
Blocks: #73 (only its @unique_key and @full_refresh text), #88 (built only if the ADR says build), #89, #90, #35

Source: design review of 2026-11-01, section 8.5, D1 (findings C01 and C04).

Activity

  1. added this to the 0.1.0 milestone on Nov 6, 2026
  2. codecapo commented on Nov 9, 2026

    @codecapo
    OwnerAuthor

    Spike result: the hand-built route works and is the route for 0.1.0. The ADRs are in #100.

    What was done, in the ticket's order

    1. cargo tree -d before any code: iceberg and sqlx each build once. The duplicates are lower-level crates (zstd, hashbrown, rand, itertools, syn and a few more), none used by the route directly.
    2. Prototype in crates/storage on a local spike branch, public iceberg-rust APIs only: rewrites each manifest that lists a removed file (that file DELETED, its other live files EXISTING with their original snapshot id and sequence numbers), reuses untouched manifests that still list a live file (ADR 0007 also keeps the ones left with only DELETED entries, as fast_append does, which the prototype dropped), puts new files in a manifest of ADDED entries, writes the manifest list and an overwrite snapshot with computed summary fields, applies the snapshot, the main reference and the properties through TableMetadataBuilder, and writes the next metadata file through TableMetadata::write_to. Commit checks the UUID and head with TableRequirement::check, then swaps metadata_location in catalog.db with the SQL catalog's one-row compare-and-swap, less its record type filter, through the sqlx dependency the crate already declared. The key and the metadata location already pin the row, so the statement fits both layouts of iceberg_tables the catalog reads. Zero rows updated is Error::Conflict, no retry, no rebase. infostash.job-id and the checkpoint properties go in the same commit.
    3. Twelve tests, all green, including the four the ticket names: two replaces on one head give one commit and one conflict, with the loser's files referenced by no snapshot, and exactly one winner when they commit concurrently; the table keeps its UUID and history; a reader holding the old metadata still reads the old files; the refusal in appends_between fires on the real overwrite. Also covered: a replace racing an append that lands first conflicts; a stale prepare writes nothing; iceberg-rust's own fast_append builds on the hand-made snapshot and appends_between then reads the appended rows; a full refresh with no rows still commits; a replace on a catalog whose iceberg_tables has no iceberg_type column, after which the catalog's own append still lands.
    4. Each head guard was also tried on its own against those tests. The swap alone still gives one winner and one conflict, also when the two commit concurrently. The requirement check alone does not close the window between the check and the swap: a commit that lands in between is overwritten. So the swap is the safety, and the check is there for the clearer error and the UUID; ADR 0007 asks storage: add the replace_if_head commit #88 to test each guard on its own.
    5. The fork route was probed alongside in its smallest form: a one-line [patch] copy of the pin with TableCommit::build made public. Still one copy of iceberg, the patched crate builds in under a minute, the same snapshot commits through Catalog::update_table with the UUID and head requirements, and a stale commit then fails with the catalog's retryable CatalogCommitConflicts. Without the requirements the catalog's swap did not stop a commit built on an old head: a stale sequence number was refused by the metadata builder, a fresh one landed over the other writer's commit. It is the hook-only variant, so it is recorded and not taken; the delete-processing fork was not attempted, because the first route had already committed and upstream is adding that.
    6. Append-only was not needed.

    Decision records

    Outside this ticket: #40 is scoped to data files, but a losing replace also leaves manifests, a manifest list and a metadata file. ADR 0007 says removal must cover them, so #40's scope needs that line.

    Time: well inside the one-day box. The prototype is thrown away; #88 builds replace_if_head to the ADR, and #89, #90 and #35 follow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions