Skip to content

storage: add the replace_if_head commit #88

Closed

Description

@codecapo

Add Warehouse::replace_if_head to crates/storage: one atomic commit that removes a set of live data files and adds new ones in a single Overwrite snapshot. The @unique_key merge of #35 (docs/rfcs/0001-project-files.md:118) and the @full_refresh rebuild of #90 (:121) commit through it and keep the choice of which rows and files to replace. This ticket carries two items that #35 lists today, old and new files changing in one snapshot and a compare-and-swap that fails without changes if the head moved, while #35 keeps the merge itself. Both Warehouse::append and append_if_head commit only through fast_append at a0b5e6c (crates/storage/src/lib.rs:143-148, :205-211), and a storage test adds an Overwrite snapshot to the metadata by hand because nothing public on the pinned iceberg-rust makes one (:682-703). The method writes the manifests, manifest list, snapshot and next metadata.json with iceberg-rust's public spec writers and commits them with Infostash's own one-row compare-and-swap on the table's row in catalog.db; the R&D spikes recorded in #19 found that copy-on-write merges work but need such a compare-and-swap. It keeps ADR 0006: one atomic commit per output table carrying infostash.job-id, checkpoint properties written in that same commit, and a head check that fails rather than rebasing (docs/adr/0006-ordered-job-commits.md:39-40, :49-51, :52-54). It replaces whole data files, which the current writer produces unpartitioned (crates/storage/src/lib.rs:361-362); a partition-scoped overwrite on top of the partitioning in #36 is Phase 1 at the earliest and out of scope here, and so is compaction (ROADMAP.md:28).

Build it only if the ADR from #72 says build. The items below assume the hand-built route that ADR tries first; if it picks a fork of iceberg-rust instead, revise them to commit through the forked action before work starts. If it chooses the append-only 0.1.0 exit, close this ticket as not planned: the Phase 1 REST catalog (ROADMAP.md:23) needs a forked or upstream action anyway, so this SQLite-only commit would not carry over.

Crate: crates/storage (src/lib.rs and its tests, Cargo.toml)

Done when

  • Warehouse::replace_if_head takes the table, the expected head, the live data files to remove and the data to add: record batches it writes itself, plus data files already written, such as the rewritten files that CowRewriteBuilder returns for Merge rows on @unique_key #35. It takes the same property arguments that append_if_head has when this lands (connectors: split the snapshot summary from the position #80 splits them in two) and, like append_if_head, refuses a key outside infostash. or a moved head before writing any file (crates/storage/src/lib.rs:188-202).
  • It commits one Overwrite snapshot built only from public iceberg-rust APIs: each manifest that lists a removed file is rewritten with that file marked DELETED (so other engines' expiry can find it) and its other files kept as EXISTING, the new files go in a manifest of ADDED entries, the manifest list keeps every untouched manifest, Infostash computes the total-* summary fields itself, and the next metadata.json is written through TableMetadata::write_to.
  • Once the new metadata.json exists, it updates metadata_location and previous_metadata_location in the table's catalog.db row only where metadata_location is still the one it loaded, on a connection pool of its own through the sqlx dependency the crate already declares (crates/storage/Cargo.toml:18-20), maps zero rows updated to Error::Conflict with no retry and no rebase, and drops the note that sqlx is never imported along with its cargo-machete exception (:30-32).
  • The job id and checkpoint properties land in that same commit with the same placement append_if_head uses (crates/storage/src/lib.rs:209, :212-218), so a replace records them exactly as an append does under ADR 0006 and its amendment in docs: add an ADR amending 0006 on position, reruns and expiry (decision D4) #75.
  • Test: two concurrent replace_if_head calls on one head, and a replace_if_head racing an append_if_head, each end with one commit and one Error::Conflict, and no snapshot references a file the losing call wrote (those are left for the orphan sweep in storage: remove orphan data files #40).
  • Test: a replace that removes some live files, and one that removes all of them as compiler: implement the @full_refresh directive #90 will, each leave exactly the kept and added rows at the new head with the table's UUID and earlier snapshots unchanged, while a reader holding the old metadata still reads the old files.
  • Test: an_overwrite_in_the_range_is_refused (crates/storage/src/lib.rs:668-714) commits its overwrite through replace_if_head instead of adding one to the metadata by hand, and still gets the refusal at :571-576 for a range that includes it.
  • The method's doc comment points to the ADR from storage: spike the replace commit route (decision D1) #72 and states its limits: it works only on the SQLite catalog that Warehouse::open_local opens (crates/storage/src/lib.rs:95), outside the Catalog trait and tied to the SQL catalog's private iceberg_tables layout, and it must be redone for the Phase 1 REST catalog.

Points: 8 (M3 replace_if_head 8)

Depends on: #72 (built only if its ADR says build)
Blocks: #35 (the merge's commit), #89 (the full-refresh fallback run, not the files_added_by fix), #90

Source: design review of 2026-11-01, finding C01 (section 8.5, M3).

Activity

  1. added this to the 0.1.0 milestone on Nov 6, 2026
  2. codecapo commented on Nov 11, 2026

    @codecapo
    OwnerAuthor

    Implementation is up in #106, following ADR 0007. The blocking ADR has landed, so the blocked label is removed.

    One decision beyond the ticket text: a replace with nothing to remove, no rows and no files is refused. Committing properties alone stays with append_if_head, because an overwrite would stop incremental reads across it.

    Moving the append and read code out of lib.rs is #107.

  3. removed
    blockedWaiting on another issue or a decision
    on Nov 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions