Repository navigation
feat(transaction): stage snapshots of v3 tables - #8
Merged
Merged
Conversation
A staged snapshot now works on a v3 table: the rows it adds take the row ids after the table's last — its snapshot records the range (`first-row-id`, `added-rows`), its data manifest the first id as it's written, so a later snapshot's list carries the manifest with its ids rather than giving its rows new ones. The range counts the snapshot's own rows only, so every manifest its list carries must have its ids already; one that doesn't — a table's files from before it was upgraded to v3, which its next commit gives ids — makes `write_manifest_list` fail: that snapshot is committed on its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hasyimibhar
added a commit
to polynya-dev/pg2iceberg
that referenced
this pull request
Nov 17, 2026
polynya-dev/iceberg-rust#8 merged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Staged snapshots, which write a chain of appends together (#5), now work on v3 tables. Until now
stage()refused v3, so pg2iceberg committed a v3 table's multi-step commits one snapshot at a time. That costs about four sequential object-store round trips per snapshot instead of two parallel rounds for the whole chain.On v3, writing a manifest list assigns row ids, and the snapshot records the range. A staged snapshot builds its commit before its list is written, so:
stage()computes the range: from the table'snext-row-id, the row count of the data files it adds. The commit's snapshot records it (first-row-id,added-rows). In a chain, each step's table already has the previous step'snext-row-id.write_manifestsgives it the range's first id as it's written. A later snapshot's list then carries the manifest with its ids, rather than giving its rows new ones.write_manifest_listrefuses to carry a data manifest without row ids. That happens to a table's files from before it was upgraded to v3, which its next commit gives ids. The snapshot's range counts its own rows only, so such a snapshot must be committed on its own, its list assigning the ids.Encrypted tables are still refused.
Tests
a_staged_chain_of_appends_lands_as_appends_one_by_onenow also runs on v3. The staged chain must land as the same table as the appends committed one by one: each manifest'sfirst_row_id, each file's, each snapshot's row range, and the table'snext-row-id.a_staged_snapshot_refuses_to_carry_files_without_row_ids: a v2 table with a file, upgraded to v3.cargo test -p iceberg --lib(1692),cargo clippy -p iceberg --all-targets --all-features -D warnings, fmt.pg2iceberg drops its v3 fallback in a PR of its own.
🤖 Generated with Claude Code