Skip to content

feat(transaction): stage snapshots of v3 tables - #8

Merged
hasyimibhar merged 1 commit into
polynya-patches-v0.11.0from
staged-v3-row-lineage
Nov 17, 2026
Merged

hasyimibhar merged 1 commit into
polynya-patches-v0.11.0from
staged-v3-row-lineage

Conversation

@hasyimibhar

Copy link
Copy Markdown

Summary

Staged snapshots, which write a chain of appends together (#5), now work on v3 tables. Until now stage() refused v3, so pg2iceberg committed a v3 table's multi-step commits one snapshot at a time. That costs about four sequential object-store round trips per snapshot instead of two parallel rounds for the whole chain.

On v3, writing a manifest list assigns row ids, and the snapshot records the range. A staged snapshot builds its commit before its list is written, so:

  • Up front: stage() computes the range: from the table's next-row-id, the row count of the data files it adds. The commit's snapshot records it (first-row-id, added-rows). In a chain, each step's table already has the previous step's next-row-id.
  • Its own data manifest: write_manifests gives it the range's first id as it's written. A later snapshot's list then carries the manifest with its ids, rather than giving its rows new ones.
  • Carried manifests: write_manifest_list refuses to carry a data manifest without row ids. That happens to a table's files from before it was upgraded to v3, which its next commit gives ids. The snapshot's range counts its own rows only, so such a snapshot must be committed on its own, its list assigning the ids.

Encrypted tables are still refused.

Tests

  • a_staged_chain_of_appends_lands_as_appends_one_by_one now also runs on v3. The staged chain must land as the same table as the appends committed one by one: each manifest's first_row_id, each file's, each snapshot's row range, and the table's next-row-id.
  • a_staged_snapshot_refuses_to_carry_files_without_row_ids: a v2 table with a file, upgraded to v3.
  • Mutation checks, each failing a test:
    • Own data manifest not stamped.
    • Neither stamped nor checked, so the list writer assigns.
    • Snapshot without its row range.
    • Range one id late.
    • No carried-manifest check.
  • cargo test -p iceberg --lib (1692), cargo clippy -p iceberg --all-targets --all-features -D warnings, fmt.

pg2iceberg drops its v3 fallback in a PR of its own.

🤖 Generated with Claude Code

A staged snapshot now works on a v3 table: the rows it adds take the
row ids after the table's last — its snapshot records the range
(`first-row-id`, `added-rows`), its data manifest the first id as it's
written, so a later snapshot's list carries the manifest with its ids
rather than giving its rows new ones.

The range counts the snapshot's own rows only, so every manifest its
list carries must have its ids already; one that doesn't — a table's
files from before it was upgraded to v3, which its next commit gives
ids — makes `write_manifest_list` fail: that snapshot is committed on
its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@hasyimibhar
hasyimibhar merged commit fae9f8f into polynya-patches-v0.11.0 Nov 17, 2026
hasyimibhar added a commit to polynya-dev/pg2iceberg that referenced this pull request Nov 17, 2026
polynya-dev/iceberg-rust#8 merged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant