Repository navigation
Description
Add Warehouse::replace_if_head to crates/storage: one atomic commit that removes a set of live data files and adds new ones in a single Overwrite snapshot. The @unique_key merge of #35 (docs/rfcs/0001-project-files.md:118) and the @full_refresh rebuild of #90 (:121) commit through it and keep the choice of which rows and files to replace. This ticket carries two items that #35 lists today, old and new files changing in one snapshot and a compare-and-swap that fails without changes if the head moved, while #35 keeps the merge itself. Both Warehouse::append and append_if_head commit only through fast_append at a0b5e6c (crates/storage/src/lib.rs:143-148, :205-211), and a storage test adds an Overwrite snapshot to the metadata by hand because nothing public on the pinned iceberg-rust makes one (:682-703). The method writes the manifests, manifest list, snapshot and next metadata.json with iceberg-rust's public spec writers and commits them with Infostash's own one-row compare-and-swap on the table's row in catalog.db; the R&D spikes recorded in #19 found that copy-on-write merges work but need such a compare-and-swap. It keeps ADR 0006: one atomic commit per output table carrying infostash.job-id, checkpoint properties written in that same commit, and a head check that fails rather than rebasing (docs/adr/0006-ordered-job-commits.md:39-40, :49-51, :52-54). It replaces whole data files, which the current writer produces unpartitioned (crates/storage/src/lib.rs:361-362); a partition-scoped overwrite on top of the partitioning in #36 is Phase 1 at the earliest and out of scope here, and so is compaction (ROADMAP.md:28).
Build it only if the ADR from #72 says build. The items below assume the hand-built route that ADR tries first; if it picks a fork of iceberg-rust instead, revise them to commit through the forked action before work starts. If it chooses the append-only 0.1.0 exit, close this ticket as not planned: the Phase 1 REST catalog (ROADMAP.md:23) needs a forked or upstream action anyway, so this SQLite-only commit would not carry over.
Crate: crates/storage (src/lib.rs and its tests, Cargo.toml)
Done when
-
Warehouse::replace_if_headtakes the table, the expected head, the live data files to remove and the data to add: record batches it writes itself, plus data files already written, such as the rewritten files thatCowRewriteBuilderreturns for Merge rows on @unique_key #35. It takes the same property arguments thatappend_if_headhas when this lands (connectors: split the snapshot summary from the position #80 splits them in two) and, likeappend_if_head, refuses a key outsideinfostash.or a moved head before writing any file (crates/storage/src/lib.rs:188-202). - It commits one
Overwritesnapshot built only from public iceberg-rust APIs: each manifest that lists a removed file is rewritten with that file marked DELETED (so other engines' expiry can find it) and its other files kept as EXISTING, the new files go in a manifest of ADDED entries, the manifest list keeps every untouched manifest, Infostash computes thetotal-*summary fields itself, and the nextmetadata.jsonis written throughTableMetadata::write_to. - Once the new
metadata.jsonexists, it updatesmetadata_locationandprevious_metadata_locationin the table'scatalog.dbrow only wheremetadata_locationis still the one it loaded, on a connection pool of its own through thesqlxdependency the crate already declares (crates/storage/Cargo.toml:18-20), maps zero rows updated toError::Conflictwith no retry and no rebase, and drops the note thatsqlxis never imported along with its cargo-machete exception (:30-32). - The job id and checkpoint properties land in that same commit with the same placement
append_if_headuses (crates/storage/src/lib.rs:209,:212-218), so a replace records them exactly as an append does under ADR 0006 and its amendment in docs: add an ADR amending 0006 on position, reruns and expiry (decision D4) #75. - Test: two concurrent
replace_if_headcalls on one head, and areplace_if_headracing anappend_if_head, each end with one commit and oneError::Conflict, and no snapshot references a file the losing call wrote (those are left for the orphan sweep in storage: remove orphan data files #40). - Test: a replace that removes some live files, and one that removes all of them as compiler: implement the @full_refresh directive #90 will, each leave exactly the kept and added rows at the new head with the table's UUID and earlier snapshots unchanged, while a reader holding the old metadata still reads the old files.
- Test:
an_overwrite_in_the_range_is_refused(crates/storage/src/lib.rs:668-714) commits its overwrite throughreplace_if_headinstead of adding one to the metadata by hand, and still gets the refusal at:571-576for a range that includes it. - The method's doc comment points to the ADR from storage: spike the replace commit route (decision D1) #72 and states its limits: it works only on the SQLite catalog that
Warehouse::open_localopens (crates/storage/src/lib.rs:95), outside theCatalogtrait and tied to the SQL catalog's privateiceberg_tableslayout, and it must be redone for the Phase 1 REST catalog.
Points: 8 (M3 replace_if_head 8)
Depends on: #72 (built only if its ADR says build)
Blocks: #35 (the merge's commit), #89 (the full-refresh fallback run, not the files_added_by fix), #90
Source: design review of 2026-11-01, finding C01 (section 8.5, M3).
Activity
docs: leave the record type filter out of the replace commit statement
docs: record the replace commit route (ADR 0007, ADR 0008) (#100)
docs: add RFC 0002 for the job and runner contracts (#102)
codecapo commentedon Nov 11, 2026
Implementation is up in #106, following ADR 0007. The blocking ADR has landed, so the blocked label is removed.
One decision beyond the ticket text: a replace with nothing to remove, no rows and no files is refused. Committing properties alone stays with
append_if_head, because an overwrite would stop incremental reads across it.Moving the append and read code out of
lib.rsis #107.