Skip to content

feat: Iceberg v3 tables, with row lineage - #148

Merged
hasyimibhar merged 1 commit into
mainfrom
feat/iceberg-v3
Nov 16, 2026
Merged

hasyimibhar merged 1 commit into
mainfrom
feat/iceberg-v3

Conversation

@hasyimibhar

@hasyimibhar hasyimibhar commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

pg2iceberg can now create v3 tables, and row lineage holds on every commit it makes. Tables still behave as v2 does: equality deletes, no deletion vectors.

  • sink.format_version: 3 (or PG2ICEBERG_FORMAT_VERSION=3) creates new tables as v3. v2 stays the default, and existing tables keep their version. Anything other than 2 or 3 is refused at config load.
  • Row lineage through compaction. Once any live data file has a first_row_id, a pass's outputs store each row's _row_id and _last_updated_sequence_number explicitly. The stored value is kept if there is one; otherwise the inherited one is written (first_row_id + position, and the file's data sequence number). Moved rows keep both, as v3 requires.
  • Tables upgraded to v3 by another engine work even when they already hold data: the upgrade's first commit assigns ids to the files already there. A pass planned before the upgrade and committed after it would hand those rows new ids, so its commit is refused as a conflict and it plans again. That uses PreparedCompaction::stores_row_lineage and the fork's RewriteFilesAction::validate_no_row_ids.
  • IcebergRustCatalog::snapshots() reports each data file's first_row_id as the latest manifest list naming it, so the FileIndex and compaction see what readers do.
  • iceberg-rust 0.11.0 (chore: iceberg-rust 0.11.0 #149) brings first_row_id inheritance on manifest read (upstream #2922).
  • The fork (v3 row lineage: REST create sends format-version, RewriteFilesAction::validate_no_row_ids iceberg-rust#6, merged into polynya-patches-v0.11.0; the lockfile moves to it) adds two things:
    • A REST create that sends format-version. iceberg-rust's REST client ignores the format version, 0.11.0's too, so tables always got the server's default, including the "explicit v2" pg2iceberg asked for.
    • validate_no_row_ids.

@hasyimibhar
hasyimibhar marked this pull request as draft November 16, 2026 18:09
@hasyimibhar
hasyimibhar changed the base branch from main to chore/iceberg-0.11 November 16, 2026 18:58
`sink.format_version: 3` (or `PG2ICEBERG_FORMAT_VERSION=3`) creates new
tables as v3; v2 stays the default, and existing tables keep theirs.
Tables behave as v2 does — equality deletes — with v3's row lineage
held on every commit:

- Compaction writes `_row_id` and `_last_updated_sequence_number` into
  its outputs once any live data file has a `first_row_id`: the stored
  value, or the inherited one (`first_row_id` + position; the file's
  data sequence number). Moved rows keep both.
- A table another engine upgrades to v3 with data in it gets ids on its
  next commit; a pass planned before then would give its rows new ones,
  so its commit fails as a conflict (`stores_row_lineage`, the fork's
  `validate_no_row_ids`) and it plans again.
- `IcebergRustCatalog::snapshots()` reports each data file's
  `first_row_id` from the latest manifest list naming it.

The fork (polynya-dev/iceberg-rust#6) inherits `first_row_id` on
manifest read (upstream #2922), sends `format-version` on REST create —
iceberg-rust's REST client ignored the format version, so tables always
got the server's default — and adds `validate_no_row_ids`.

DST: a format dimension (v2, v3, v3 by upgrade), a sim catalog that
assigns row ids, invariants for row lineage (every live row has ids,
unique, below next-row-id) and for commits that move rows keeping them,
pinned cases on the sim and iceberg-rust, and a focused proptest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@hasyimibhar
hasyimibhar changed the base branch from chore/iceberg-0.11 to main November 16, 2026 19:14
@hasyimibhar
hasyimibhar marked this pull request as ready for review November 16, 2026 19:16
@hasyimibhar
hasyimibhar merged commit 8b46460 into main Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant