Repository navigation
Encode * and ~ in partition paths the way Java does - #4443
Merged
Fokko merged 1 commit intoNov 21, 2026
Merged
Conversation
partition_to_path uses quote_plus to match java.net.URLEncoder, but the two differ on "*" (Java keeps it) and "~" (Java escapes it), so the same partition got a different path from PyIceberg than from Java.
Fokko
self-requested a review
November 20, 2026 15:39
Fokko
reviewed
Nov 20, 2026
Fokko
left a comment
Contributor
There was a problem hiding this comment.
Thanks @gedaliakliger! Nice catch: * and ~ are exactly the two characters where quote_plus and java.net.URLEncoder disagree, so paths written by PyIceberg could mismatch what Spark/Java compute for the same partition values.
Verified locally: with the PR, a*b~c encodes as a*b%7Ec (matches URLEncoder), while main produces a%2Ab~c. The new test fails on main and passes with this change. CI is green.
LGTM!
Fokko
approved these changes
Nov 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
PartitionSpec.partition_to_pathURL-encodes field names and values to match Java, which usesjava.net.URLEncoder(PartitionSpec.java).quote_plusdiffers from it on two characters: Java leaves*as is and escapes~to%7E, while PyIceberg wrote%2Aand a literal~. A partition value such asa*b~ctherefore got the patha%2Ab~cfrom PyIceberg anda*b%7Ecfrom Java, and a differentpartitions.<path>key in the snapshot summary.Are these changes tested?
Yes.
test_partition_spec_to_path_asterisk_and_tildefails onmainand passes with this change. I rantests/table/test_partitioning.py,tests/table/test_locations.pyandtests/table/test_snapshots.pywith-m unit, and ruff and mypy on the changed files. The rest is left to CI.Are there any user-facing changes?
New data files for partitions whose field name or value contains
*or~are written under the Java-style path. Existing files are unaffected, since manifests store full paths.