Skip to content

fix(partitioning): support specs with multiple fields on one source - #4092

Closed
wanggy0201 wants to merge 2 commits into
apache:mainfrom
wanggy0201:guangyuwang/fix-evolved-partition-key
Closed

wanggy0201 wants to merge 2 commits into
apache:mainfrom
wanggy0201:guangyuwang/fix-evolved-partition-key

Conversation

@wanggy0201

Copy link
Copy Markdown

Rationale for this change

Native writes fail with Cannot have redundant partitions when a valid partition spec has more than one field derived from the same source column. That includes a format-v1 spec that keeps retired void fields next to an active field, and a spec that applies multiple active transforms, such as year and month, to the same timestamp.

PartitionKey.partition looked up every field by source ID and required exactly one match. Each supplied PartitionFieldValue already names its partition field, so the key now uses that field when building the partition record. Field order, null slots, field IDs, and existing specs are unchanged.

Are these changes tested?

  • Six regression cases cover retained void fields with null and non-null active values, multiple active transforms, Arrow partition grouping, and a format-v1 Arrow-to-Parquet write with retired fields.
  • uv run --extra pyiceberg-core python -m pytest tests/table/test_partition_key.py passed (6 tests).
  • Ruff check and format passed on pyiceberg/partitioning.py and tests/table/test_partition_key.py.

Are there any user-facing changes?

Native writes now succeed for valid partition specs that share a source column across multiple fields. This does not change table metadata.

wanggy0201 and others added 2 commits October 8, 2026 21:41
PartitionKey.partition looked up every field by source ID and required
exactly one match. Valid specs that keep retired void fields, or that
apply multiple active transforms to the same column, then failed native
writes with "Cannot have redundant partitions".

Co-authored-by: Cursor <cursoragent@cursor.com>
The regression fixtures copied production column names. Dummy field names
and values keep the same coverage without tying the tests to one table.

Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

@github-actions github-actions Bot added the stale label Nov 8, 2026
@github-actions

Copy link
Copy Markdown

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

@github-actions github-actions Bot closed this Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant