Skip to content

Merge rows on @unique_key #35

Open

Description

@codecapo

Models with @unique_key replace existing rows that have the same key instead of appending duplicates. Split out of #10.

Crates: crates/engine, crates/storage

The hard part: the pinned iceberg-rust can only append (fast_append). Replacing rows needs either an overwrite commit (copy-on-write: rewrite the data files that hold matching keys, remove the old files, add the new ones in one snapshot) or position/equality deletes. Pick one in the PR and say why; copy-on-write is the simpler start.

Done when

  • @unique_key (one or more columns) merges new rows into the target
  • Old and new files change in one snapshot, so readers never see both or neither
  • A rerun with the same input changes nothing
  • The commit is a compare-and-swap: it fails without changes if the table head moved since the merge read it
  • Tests: insert, update of an existing key, a key repeated within one batch (error or last-wins, documented)
  • Incremental scan: read rows added between two snapshots #11's incremental scan reports a clear error for an overwrite snapshot until it supports one

Depends on: #10
Blocks: nothing in the quick start; it can run alongside #13.

Activity

  1. added this to the 0.1.0 milestone on Oct 29, 2026
  2. codecapo commented on Oct 30, 2026

    @codecapo
    OwnerAuthor

    Spike findings (merge)

    A spike built a copy-on-write merge on the pinned iceberg-rust; all 9 tests pass (insert, update, identical rerun is a no-op, duplicate key rejected, time travel to the old rows, reopen).

    • iceberg::cow_rewrite::CowRewriteBuilder (already in the pin) rewrites only the files holding the replaced keys.
    • The overwrite snapshot must be built by hand: manifests, sequence numbers and summary totals. It passes TableMetadataBuilder validation.
    • The commit can't go through Catalog::update_table, because TableCommit::build is pub(crate). Options:
    • read_since misreports overwrite snapshots, so Incremental scan: read rows added between two snapshots #11 must refuse them (already a criterion here).
    • Not yet covered: composite keys, partitioned tables, checking uniqueness against the table.

    Points: 13 -> 8 with the SQLite catalog; 13 if it must also work on a shared catalog (REST, Glue).

  3. codecapo commented on Nov 6, 2026

    @codecapo
    OwnerAuthor

    The design review of 2026-11-01 touches this issue. It moves the replace commit out of this issue: #72 decides whether one is built at all and by which route, #88 builds Warehouse::replace_if_head, and this issue keeps the merge that commits through it.

    Suggested additions to the Done-when:

    Work that moved to new tickets:

    Suggested re-score: 5, down from 8. Section 8.5 of the design review of 2026-11-01 scores the @unique_key merge at 5 within milestone M3 at 20 (replace_if_head 8 in #88; plan-time fallback and the files_added_by fix 5 in #89; @unique_key merge 5; @full_refresh 2 in #90), a milestone that holds only if #72 decides to build the replace commit. The 8 in #19 counted the compare-and-swap commit that now sits in #88; #19 decides.

    Source: design review of 2026-11-01, findings C01, C04, C15 and C17 and design point S6-04, with its sections 2.5 and 8.5 (D1 and M3).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions