Repository navigation
Description
Decide how Infostash commits a snapshot that replaces rows, through a spike time-boxed to one day whose prototype code is thrown away, and record the chosen route in a new ADR; the production replace_if_head is #88, and the answer decides whether #35, @full_refresh and the daily_revenue example behind #8 and #16 can ship in 0.1.0. Infostash at a0b5e6c pins iceberg-rust to rev 5b82ab9 (Cargo.toml:29-30), both Warehouse::append and append_if_head commit through fast_append (crates/storage/src/lib.rs:143-148, :205-211), and the storage tests add an Overwrite snapshot to the metadata by hand because nothing public on that pin makes one (:682-696). Draft RFC 0001 needs a replace commit for @unique_key (docs/rfcs/0001-project-files.md:118) and for @full_refresh (:121), which the ready/daily_revenue aggregate (:101-107) needs to keep one row per day; a partition-scoped overwrite, which recomputing only the changed partitions would need on top of partitioned writes (#36), is Phase 1 at the earliest and out of scope here. The route to try first writes the Overwrite snapshot through iceberg-rust's public spec writers and commits it with Infostash's own compare-and-swap on the table's row in catalog.db (the spikes recorded in #19 found that copy-on-write merges work but need such a compare-and-swap); a fork through [patch] comes second if that proves heavier, and an append-only 0.1.0 is the exit. Whichever route wins must keep ADR 0006: one atomic commit per output table carrying the job id, and a head check that fails rather than rebasing (docs/adr/0006-ordered-job-commits.md:39-40, :52-54).
Where: crates/storage (the throwaway prototype and its tests), docs/adr/ (up to two new ADRs and their index rows)
Done when
-
cargo tree -dis run before any prototype code is written, and the new ADR records whether the workspace builds more than one copy of a crate the route uses directly, such asiceberg(docs/adr/0002-pin-iceberg-rust-from-git.md:20-22) orsqlx. - A prototype in
crates/storagethat uses only public iceberg-rust APIs rewrites each manifest that lists a removed data file, marking that file DELETED and keeping the manifest's other files as EXISTING entries, puts the new files in a manifest of ADDED entries, and writes a manifest list, anOverwritesnapshot whosetotal-*summary fields it computes itself, and the nextmetadata.jsonthroughTableMetadata::write_to. - The prototype checks the table UUID and the expected head with
TableRequirement::checkagainst the metadata it loaded, then commits by swapping the table'smetadata_locationincatalog.dbwith a one-row compare-and-swap like the SQL catalog's own, through thesqlxdependency the crate declares but never imports (crates/storage/Cargo.toml:18-20,:30-32), maps zero rows updated toError::Conflictwith no retry or rebase, and writesinfostash.job-idand the checkpoint properties in that same commit (docs/adr/0006-ordered-job-commits.md:39-40,:49-51). - Tests on the prototype show that two concurrent replaces on one head give one commit and one
Conflictwith the loser's files referenced by no snapshot, that the table keeps its UUID and snapshot history, that a reader holding the old metadata still reads the old files, and that the refusal atcrates/storage/src/lib.rs:571-576still fires on a real overwrite. - The new ADR, at the next free number (
docs/adr/README.md:25) and listed in the index atdocs/adr/README.md:46-55, records the chosen route and its costs, which for the hand-built commit are SQLite only, coupling to the SQL catalog's privateiceberg_tableslayout, a connection pool of its own, a copy of the catalog'srecord_type_filter, working outside theCatalogtrait, and a redo for the Phase 1 REST catalog (ROADMAP.md:23). - If the fork is chosen, the new ADR supersedes ADR 0002 (0002 itself changes only on its status line,
docs/adr/README.md:19-21), frames the fork as a bridge until the change lands upstream (docs/adr/0001-iceberg-datafusion-arrow.md:19-20), and requires the fork to process delete entries rather than only expose hooks. - If append-only is chosen, the new ADR lists the follow-up work, namely rewriting RFC 0001's
clean/ordersanddaily_revenueexamples (docs/rfcs/0001-project-files.md:87-107), making@unique_keyand@full_refreshunknown-key errors (:114), and correctingREADME.md:21, which says Clean tables are deduplicated (README.md:23anddocs/architecture.md:68-69are corrected in docs: fix README, architecture and ROADMAP statements the code does not back #94 on either route), and says whether an append-only versioned Clean with read-time dedup is acceptable in place of the merge. - Unless append-only is chosen, a second short ADR, written after the first, records the rule for incremental readers of a replaced table: before creating a job, the scheduler checks the operations in each input's pinned range, pins
from = Nonefor any input holding a non-append snapshot, reports the fallback and replaces the target rather than appending, as a new job with newly pinned inputs (docs/adr/0006-ordered-job-commits.md:47-48,:71-72); the refusal atcrates/storage/src/lib.rs:571-576stays as a non-retryable guard; and until a replace commit exists, the compile-time rule that docs: revise RFC 0001 with the incremental rules (decision D2) #73 writes into RFC 0001 and compiler: add an incremental eligibility classifier with a function allow-list #84 enforces (no incremental model reads a table declaring@unique_keyor@full_refresh) applies instead, after which the plan-time check replaces it.
Points: 5 (D1)
Time-box: one day. When it runs out, the ADR records how far each route got and which one continues; if neither route has committed a replace by then, append-only is the route for 0.1.0.
Depends on: nothing
Blocks: #73 (only its @unique_key and @full_refresh text), #88 (built only if the ADR says build), #89, #90, #35
Source: design review of 2026-11-01, section 8.5, D1 (findings C01 and C04).
Activity
codecapo commentedon Nov 9, 2026
Spike result: the hand-built route works and is the route for 0.1.0. The ADRs are in #100.
What was done, in the ticket's order
cargo tree -dbefore any code:icebergandsqlxeach build once. The duplicates are lower-level crates (zstd,hashbrown,rand,itertools,synand a few more), none used by the route directly.crates/storageon a local spike branch, public iceberg-rust APIs only: rewrites each manifest that lists a removed file (that fileDELETED, its other live filesEXISTINGwith their original snapshot id and sequence numbers), reuses untouched manifests that still list a live file (ADR 0007 also keeps the ones left with onlyDELETEDentries, asfast_appenddoes, which the prototype dropped), puts new files in a manifest ofADDEDentries, writes the manifest list and anoverwritesnapshot with computed summary fields, applies the snapshot, themainreference and the properties throughTableMetadataBuilder, and writes the next metadata file throughTableMetadata::write_to. Commit checks the UUID and head withTableRequirement::check, then swapsmetadata_locationincatalog.dbwith the SQL catalog's one-row compare-and-swap, less its record type filter, through thesqlxdependency the crate already declared. The key and the metadata location already pin the row, so the statement fits both layouts oficeberg_tablesthe catalog reads. Zero rows updated isError::Conflict, no retry, no rebase.infostash.job-idand the checkpoint properties go in the same commit.appends_betweenfires on the real overwrite. Also covered: a replace racing an append that lands first conflicts; a stale prepare writes nothing; iceberg-rust's ownfast_appendbuilds on the hand-made snapshot andappends_betweenthen reads the appended rows; a full refresh with no rows still commits; a replace on a catalog whoseiceberg_tableshas noiceberg_typecolumn, after which the catalog's own append still lands.[patch]copy of the pin withTableCommit::buildmade public. Still one copy oficeberg, the patched crate builds in under a minute, the same snapshot commits throughCatalog::update_tablewith the UUID and head requirements, and a stale commit then fails with the catalog's retryableCatalogCommitConflicts. Without the requirements the catalog's swap did not stop a commit built on an old head: a stale sequence number was refused by the metadata builder, a fresh one landed over the other writer's commit. It is the hook-only variant, so it is recorded and not taken; the delete-processing fork was not attempted, because the first route had already committed and upstream is adding that.Decision records
iceberg_tables, an own pool, outside theCatalogtrait, a redo for the Phase 1 REST catalog), the upstream state at the pin and the migration path.from = None; the refusal stays as a guard that ends the job without a retry; the compile-time rule from docs: revise RFC 0001 with the incremental rules (decision D2) #73 and compiler: add an incremental eligibility classifier with a function allow-list #84 applies until the plan-time check in engine: fall back to a full refresh at plan time and fix files_added_by #89 lands.Outside this ticket: #40 is scoped to data files, but a losing replace also leaves manifests, a manifest list and a metadata file. ADR 0007 says removal must cover them, so #40's scope needs that line.
Time: well inside the one-day box. The prototype is thrown away; #88 builds
replace_if_headto the ADR, and #89, #90 and #35 follow.