Skip to content

fix(scan): apply equality deletes from an all-void partition spec globally - #3449

Open
JeonDaehong wants to merge 2 commits into
apache:mainfrom
JeonDaehong:fix-void-spec-global-eq-deletes
Open

JeonDaehong wants to merge 2 commits into
apache:mainfrom
JeonDaehong:fix-void-spec-global-eq-deletes

Conversation

@JeonDaehong

@JeonDaehong JeonDaehong commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

None, found while reading DeleteFileIndex.

What changes are included in this PR?

The spec says equality deletes stored with an unpartitioned spec are global. Java decides this with spec.isUnpartitioned(), which is also true when every field is a void transform (what a v1 table keeps after dropping a partition field). Rust checked whether the partition tuple was empty, so such deletes were indexed under a [null] partition and never matched data files from the earlier, partitioned spec. Deleted rows came back.

DeleteFileContext now carries the delete file's spec, and the index uses PartitionSpec::is_unpartitioned. If the spec is missing, it falls back to the old check. Position deletes are unchanged: Java scopes them by spec and partition too.

Are these changes tested?

Yes. test_void_spec_equality_delete_is_global fails on main (0 deletes applied) and passes here. cargo test -p iceberg --lib, clippy and fmt are clean.

AI Disclosure

I used Claude Code to find this, write the change and run the tests. I reviewed it before opening the PR.

…bally

A spec whose fields are all void transforms is unpartitioned, as PartitionSpec::is_unpartitioned and Java PartitionSpec.isUnpartitioned say. The delete file index only checked whether the partition tuple was empty, so equality deletes written under such a spec were indexed by a null partition and never matched data files from an earlier, partitioned spec.
@JeonDaehong JeonDaehong changed the title fix(scan): apply equality deletes from an all-void partition spec glo… fix(scan): apply equality deletes from an all-void partition spec globally Oct 14, 2026
@JeonDaehong

Copy link
Copy Markdown
Contributor Author

@CTTY @mbutrovich could you take a look when you have time? Small fix in DeleteFileIndex: equality deletes under an all-void partition spec are global in Java but were never applied to older-spec data files here.

@JeonDaehong
JeonDaehong marked this pull request as draft October 16, 2026 06:23
@JeonDaehong

Copy link
Copy Markdown
Contributor Author

Converted to draft to stay within the new limit of 3 open PRs (#3465). The change is still ready for review; I will mark it ready again when one of my other PRs lands.

@github-actions

Copy link
Copy Markdown
Contributor

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

@github-actions github-actions Bot added the stale label Nov 16, 2026
@JeonDaehong

Copy link
Copy Markdown
Contributor Author

Still relevant. It's a draft only because of the open PR limit (#3465), and I'll mark it ready once one of my open PRs lands.

@github-actions github-actions Bot removed the stale label Nov 18, 2026
# Conflicts:
#	crates/iceberg/src/scan/task.rs
@JeonDaehong
JeonDaehong marked this pull request as ready for review November 18, 2026 14:21
@JeonDaehong

Copy link
Copy Markdown
Contributor Author

Merged main (the only conflict was an import line) and marked this ready now that #3433 is in. test_void_spec_equality_delete_is_global still fails with the old partition.fields().is_empty() check and passes here. @CTTY @mbutrovich could you take a look when you have time?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant