Skip to content

Add ManageSnapshots.cherrypick - #4440

Open
rambleraptor wants to merge 2 commits into
apache:mainfrom
rambleraptor:feat/cherrypick-snapshot
Open

rambleraptor wants to merge 2 commits into
apache:mainfrom
rambleraptor:feat/cherrypick-snapshot

Conversation

@rambleraptor

Copy link
Copy Markdown
Collaborator

Rationale for this change

This adds support for cherry picking a snapshot from another branch / write-audit-publish snapshot.

Are these changes tested?

Includes unit tests + documentation.

Are there any user-facing changes?

Port Java's CherryPickOperation so a snapshot from another branch, or a
staged write-audit-publish snapshot, can be published to main from
PyIceberg. Appends are replayed onto main with source-snapshot-id and
published-wap-id, and other operations are fast-forwarded when their
parent is current. Dynamic overwrite replay is left for a follow-up.
@Fokko
Fokko self-requested a review November 20, 2026 15:41

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @rambleraptor! Nice addition: replaying an append as a new snapshot with source-snapshot-id set matches what Java does for cherry-pick, and the staged-WAP fast-forward path is a good fit for the existing ManageSnapshots API.

I ran the new tests/table/test_manage_snapshots.py locally (45 passed) and did an end-to-end cherry-pick against a format-version 2 table with a staged WAP snapshot: rows land on main, source-snapshot-id and published-wap-id end up in the summary. Also verified the ancestor/duplicate-WAP rejection paths.

A couple of small things:

  1. In _validate_cherrypick_non_ancestor, comparing summary.get(SOURCE_SNAPSHOT_ID) only catches picks that are already ancestors of the current snapshot. ancestors_of walks back from the current snapshot so that should be covered, but a snapshot picked onto a diverged branch (not an ancestor) with the same source would slip through the already-picked check. Probably fine for now, but worth a thought.

  2. Nit: two blank lines between _validate_wap_publish and _added_data_files (PEP8, ruff should catch it in CI anyway).

  3. In _added_data_files, first_row_id is set to None up front. On my v2-table run, the new manifest entries still show the inherited ``first_row_idof the source manifest because_inherit_from_manifest` runs before you clear it for entries read with `fetch_manifest_entry(discard_deleted=True)`. Since row IDs are only meaningful for format-version 3 and those tables cannot be written yet (V3 write is not supported), fine to leave, but it would be good to double check the ordering once V3 write lands.

LGTM!

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small correction to my point 3 above: I misread my own test output. The first_row_id = None in _added_data_files runs after fetch_manifest_entry returns (so after _inherit_from_manifest), and my run shows first_row_id: None on the picked entries. That part works as intended, ignore point 3. Points 1 and 2 still stand.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants