You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Models with @unique_key replace existing rows that have the same key instead of appending duplicates. Split out of #10.
Crates:crates/engine, crates/storage
The hard part: the pinned iceberg-rust can only append (fast_append). Replacing rows needs either an overwrite commit (copy-on-write: rewrite the data files that hold matching keys, remove the old files, add the new ones in one snapshot) or position/equality deletes. Pick one in the PR and say why; copy-on-write is the simpler start.
Done when
@unique_key (one or more columns) merges new rows into the target
Old and new files change in one snapshot, so readers never see both or neither
A rerun with the same input changes nothing
The commit is a compare-and-swap: it fails without changes if the table head moved since the merge read it
Tests: insert, update of an existing key, a key repeated within one batch (error or last-wins, documented)
A spike built a copy-on-write merge on the pinned iceberg-rust; all 9 tests pass (insert, update, identical rerun is a no-op, duplicate key rejected, time travel to the old rows, reopen).
iceberg::cow_rewrite::CowRewriteBuilder (already in the pin) rewrites only the files holding the replaced keys.
The overwrite snapshot must be built by hand: manifests, sequence numbers and summary totals. It passes TableMetadataBuilder validation.
The commit can't go through Catalog::update_table, because TableCommit::build is pub(crate). Options:
our own compare-and-swap on the SQLite catalog pointer (one UPDATE ... WHERE metadata_location = <old>), for now;
The design review of 2026-11-01 touches this issue. It moves the replace commit out of this issue: #72 decides whether one is built at all and by which route, #88 builds Warehouse::replace_if_head, and this issue keeps the merge that commits through it.
Suggested additions to the Done-when:
The merge commits through replace_if_head from storage: add the replace_if_head commit #88, never through fast_append, which is how both Warehouse::append and append_if_head commit at a0b5e6c (crates/storage/src/lib.rs:143-148, :205-211), and never by dropping and recreating the target, which downstream incremental readers would hit as "not an ancestor" (crates/storage/src/lib.rs:581-583).
The merge is copy-on-write, the simpler start the issue names: the CowRewriteBuilder of the pinned iceberg-rust (Cargo.toml:29) plans the rewrite at the snapshot that the merge then gives to replace_if_head as the expected head, with a rewriter that drops every row whose key is among the incoming rows the checks accept, and the removed files, the rewritten files and the latest accepted row for each incoming key go into one replace_if_head call.
Rows sharing a key keep only the latest one that the checks accept (the checks of Row checks and quarantine tables #12 run first), as the per-mode rule docs: revise RFC 0001 with the incremental rules (decision D2) #73 writes into RFC 0001 requires. "Latest" follows the ordering rule with which docs: revise RFC 0001 with the incremental rules (decision D2) #73 answers open question 4 (docs/rfcs/0001-project-files.md:175-176): Raw snapshot sequence number, then _source_file, then _source_row from connectors: add _source_row to Raw tables #86, never read order. That rule replaces the "error or last-wins" choice in the existing test item: a key repeated within one batch keeps its latest row under that order. Tests show that two attempts of one job over the same pinned inputs pick the same winning row, that a later load's row wins even when its file name sorts first, and that a correction failing a @check leaves the earlier row in place.
The existing "update of an existing key" test becomes concrete: with @unique_key: order_id on clean.orders (docs/rfcs/0001-project-files.md:88-89), order o1 loaded at 100 and then corrected to 120 in a later load leaves one row (o1, 120), so a sum of amount over the table reads 120 rather than 220, and the table keeps its UUID and earlier snapshots.
storage: add the replace_if_head commit #88: builds replace_if_head, the head-checked Overwrite commit that now carries the existing one-snapshot and compare-and-swap items, with its concurrency tests.
Suggested re-score: 5, down from 8. Section 8.5 of the design review of 2026-11-01 scores the @unique_key merge at 5 within milestone M3 at 20 (replace_if_head 8 in #88; plan-time fallback and the files_added_by fix 5 in #89; @unique_key merge 5; @full_refresh 2 in #90), a milestone that holds only if #72 decides to build the replace commit. The 8 in #19 counted the compare-and-swap commit that now sits in #88; #19 decides.
Source: design review of 2026-11-01, findings C01, C04, C15 and C17 and design point S6-04, with its sections 2.5 and 8.5 (D1 and M3).
Activity
codecapo commentedon Oct 30, 2026
Spike findings (merge)
A spike built a copy-on-write merge on the pinned iceberg-rust; all 9 tests pass (insert, update, identical rerun is a no-op, duplicate key rejected, time travel to the old rows, reopen).
iceberg::cow_rewrite::CowRewriteBuilder(already in the pin) rewrites only the files holding the replaced keys.overwritesnapshot must be built by hand: manifests, sequence numbers and summary totals. It passesTableMetadataBuildervalidation.Catalog::update_table, becauseTableCommit::buildispub(crate). Options:UPDATE ... WHERE metadata_location = <old>), for now;[patch]exposingTableCommit::build. Either one also needs a matching datafusion-iceberg pin.read_sincemisreports overwrite snapshots, so Incremental scan: read rows added between two snapshots #11 must refuse them (already a criterion here).Points: 13 -> 8 with the SQLite catalog; 13 if it must also work on a shared catalog (REST, Glue).
feat(storage): stream rows added between two snapshots (#42)
codecapo commentedon Nov 6, 2026
The design review of 2026-11-01 touches this issue. It moves the replace commit out of this issue: #72 decides whether one is built at all and by which route, #88 builds
Warehouse::replace_if_head, and this issue keeps the merge that commits through it.Suggested additions to the Done-when:
replace_if_headfrom storage: add the replace_if_head commit #88, never throughfast_append, which is how bothWarehouse::appendandappend_if_headcommit at a0b5e6c (crates/storage/src/lib.rs:143-148,:205-211), and never by dropping and recreating the target, which downstream incremental readers would hit as "not an ancestor" (crates/storage/src/lib.rs:581-583).CowRewriteBuilderof the pinned iceberg-rust (Cargo.toml:29) plans the rewrite at the snapshot that the merge then gives toreplace_if_headas the expected head, with a rewriter that drops every row whose key is among the incoming rows the checks accept, and the removed files, the rewritten files and the latest accepted row for each incoming key go into onereplace_if_headcall.@unique_key(docs/rfcs/0001-project-files.md:118) by lifting the interim rejection that compiler: add an incremental eligibility classifier with a function allow-list #84 adds, for this directive only; the model body must still satisfy compiler: add an incremental eligibility classifier with a function allow-list #84's incremental node and function rules, since the merge evaluates it over the delta, and compiler: add an incremental eligibility classifier with a function allow-list #84's rule that no incremental model reads a@unique_keytable stays until engine: fall back to a full refresh at plan time and fix files_added_by #89 replaces it.docs/rfcs/0001-project-files.md:175-176): Raw snapshot sequence number, then_source_file, then_source_rowfrom connectors: add _source_row to Raw tables #86, never read order. That rule replaces the "error or last-wins" choice in the existing test item: a key repeated within one batch keeps its latest row under that order. Tests show that two attempts of one job over the same pinned inputs pick the same winning row, that a later load's row wins even when its file name sorts first, and that a correction failing a@checkleaves the earlier row in place.@unique_key: order_idonclean.orders(docs/rfcs/0001-project-files.md:88-89), order o1 loaded at 100 and then corrected to 120 in a later load leaves one row (o1, 120), so a sum ofamountover the table reads 120 rather than 220, and the table keeps its UUID and earlier snapshots.Work that moved to new tickets:
replace_if_head, the head-checkedOverwritecommit that now carries the existing one-snapshot and compare-and-swap items, with its concurrency tests.crates/storage/src/lib.rs:570-576as a guard; that refusal already meets the last existing item.Suggested re-score: 5, down from 8. Section 8.5 of the design review of 2026-11-01 scores the
@unique_keymerge at 5 within milestone M3 at 20 (replace_if_head8 in #88; plan-time fallback and thefiles_added_byfix 5 in #89;@unique_keymerge 5;@full_refresh2 in #90), a milestone that holds only if #72 decides to build the replace commit. The 8 in #19 counted the compare-and-swap commit that now sits in #88; #19 decides.Source: design review of 2026-11-01, findings C01, C04, C15 and C17 and design point S6-04, with its sections 2.5 and 8.5 (D1 and M3).
docs: revise RFC 0001 with the incremental rules (#101)