Repository navigation
fix(scan): apply equality deletes from an all-void partition spec globally - #3449
JeonDaehong wants to merge 2 commits into
Conversation
…bally A spec whose fields are all void transforms is unpartitioned, as PartitionSpec::is_unpartitioned and Java PartitionSpec.isUnpartitioned say. The delete file index only checked whether the partition tuple was empty, so equality deletes written under such a spec were indexed by a null partition and never matched data files from an earlier, partitioned spec.
|
@CTTY @mbutrovich could you take a look when you have time? Small fix in |
|
Converted to draft to stay within the new limit of 3 open PRs (#3465). The change is still ready for review; I will mark it ready again when one of my other PRs lands. |
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
Still relevant. It's a draft only because of the open PR limit (#3465), and I'll mark it ready once one of my open PRs lands. |
# Conflicts: # crates/iceberg/src/scan/task.rs
|
Merged main (the only conflict was an import line) and marked this ready now that #3433 is in. |
Which issue does this PR close?
None, found while reading
DeleteFileIndex.What changes are included in this PR?
The spec says equality deletes stored with an unpartitioned spec are global. Java decides this with
spec.isUnpartitioned(), which is also true when every field is a void transform (what a v1 table keeps after dropping a partition field). Rust checked whether the partition tuple was empty, so such deletes were indexed under a[null]partition and never matched data files from the earlier, partitioned spec. Deleted rows came back.DeleteFileContextnow carries the delete file's spec, and the index usesPartitionSpec::is_unpartitioned. If the spec is missing, it falls back to the old check. Position deletes are unchanged: Java scopes them by spec and partition too.Are these changes tested?
Yes.
test_void_spec_equality_delete_is_globalfails onmain(0 deletes applied) and passes here.cargo test -p iceberg --lib, clippy and fmt are clean.AI Disclosure
I used Claude Code to find this, write the change and run the tests. I reviewed it before opening the PR.