Skip to content

Encode * and ~ in partition paths the way Java does - #4443

Merged
Fokko merged 1 commit into
apache:mainfrom
gedaliakliger:fix/partition-path-java-url-encoding
Nov 21, 2026
Merged

Fokko merged 1 commit into
apache:mainfrom
gedaliakliger:fix/partition-path-java-url-encoding

Conversation

@gedaliakliger

Copy link
Copy Markdown
Contributor

Rationale for this change

PartitionSpec.partition_to_path URL-encodes field names and values to match Java, which uses java.net.URLEncoder (PartitionSpec.java). quote_plus differs from it on two characters: Java leaves * as is and escapes ~ to %7E, while PyIceberg wrote %2A and a literal ~. A partition value such as a*b~c therefore got the path a%2Ab~c from PyIceberg and a*b%7Ec from Java, and a different partitions.<path> key in the snapshot summary.

Are these changes tested?

Yes. test_partition_spec_to_path_asterisk_and_tilde fails on main and passes with this change. I ran tests/table/test_partitioning.py, tests/table/test_locations.py and tests/table/test_snapshots.py with -m unit, and ruff and mypy on the changed files. The rest is left to CI.

Are there any user-facing changes?

New data files for partitions whose field name or value contains * or ~ are written under the Java-style path. Existing files are unaffected, since manifests store full paths.

partition_to_path uses quote_plus to match java.net.URLEncoder, but the
two differ on "*" (Java keeps it) and "~" (Java escapes it), so the
same partition got a different path from PyIceberg than from Java.
@Fokko
Fokko self-requested a review November 20, 2026 15:39

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @gedaliakliger! Nice catch: * and ~ are exactly the two characters where quote_plus and java.net.URLEncoder disagree, so paths written by PyIceberg could mismatch what Spark/Java compute for the same partition values.

Verified locally: with the PR, a*b~c encodes as a*b%7Ec (matches URLEncoder), while main produces a%2Ab~c. The new test fails on main and passes with this change. CI is green.

LGTM!

@Fokko
Fokko added this pull request to the merge queue Nov 21, 2026
Merged via the queue into apache:main with commit f323f28 Nov 21, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants