Repository navigation
Add ManageSnapshots.cherrypick - #4440
rambleraptor wants to merge 2 commits into
Conversation
Port Java's CherryPickOperation so a snapshot from another branch, or a staged write-audit-publish snapshot, can be published to main from PyIceberg. Appends are replayed onto main with source-snapshot-id and published-wap-id, and other operations are fast-forwarded when their parent is current. Dynamic overwrite replay is left for a follow-up.
Fokko
left a comment
There was a problem hiding this comment.
Thanks @rambleraptor! Nice addition: replaying an append as a new snapshot with source-snapshot-id set matches what Java does for cherry-pick, and the staged-WAP fast-forward path is a good fit for the existing ManageSnapshots API.
I ran the new tests/table/test_manage_snapshots.py locally (45 passed) and did an end-to-end cherry-pick against a format-version 2 table with a staged WAP snapshot: rows land on main, source-snapshot-id and published-wap-id end up in the summary. Also verified the ancestor/duplicate-WAP rejection paths.
A couple of small things:
-
In
_validate_cherrypick_non_ancestor, comparingsummary.get(SOURCE_SNAPSHOT_ID)only catches picks that are already ancestors of the current snapshot.ancestors_ofwalks back from the current snapshot so that should be covered, but a snapshot picked onto a diverged branch (not an ancestor) with the same source would slip through the already-picked check. Probably fine for now, but worth a thought. -
Nit: two blank lines between
_validate_wap_publishand_added_data_files(PEP8, ruff should catch it in CI anyway). -
In
_added_data_files,first_row_idis set toNoneup front. On my v2-table run, the new manifest entries still show the inherited ``first_row_idof the source manifest because_inherit_from_manifest` runs before you clear it for entries read with `fetch_manifest_entry(discard_deleted=True)`. Since row IDs are only meaningful for format-version 3 and those tables cannot be written yet (V3 write is not supported), fine to leave, but it would be good to double check the ordering once V3 write lands.
LGTM!
Fokko
left a comment
There was a problem hiding this comment.
Small correction to my point 3 above: I misread my own test output. The first_row_id = None in _added_data_files runs after fetch_manifest_entry returns (so after _inherit_from_manifest), and my run shows first_row_id: None on the picked entries. That part works as intended, ignore point 3. Points 1 and 2 still stand.
Rationale for this change
This adds support for cherry picking a snapshot from another branch / write-audit-publish snapshot.
Are these changes tested?
Includes unit tests + documentation.
Are there any user-facing changes?