Skip to content

fix(table): compare promoted partition values after widening - #2620

Merged
laskoviymishka merged 5 commits into
apache:mainfrom
laskoviymishka:factory/rc-2605-widen-partition-values
Nov 18, 2026
Merged

laskoviymishka merged 5 commits into
apache:mainfrom
laskoviymishka:factory/rc-2605-widen-partition-values

Conversation

@laskoviymishka

Copy link
Copy Markdown
Contributor

After an identity partition source is promoted (int to long, float to double), manifests written before the promotion still decode its partition values as int32/float32, and newer ones as int64/float64. So a v2 merge-on-read Delete on an old file commits a position delete that the scan then rejects with "partition does not match" (#2605), and equality deletes in a promoted float partition skip the old files (#2610). This change widens int32 to int64 and float32 to float64 where partition values are compared: in partitionsMatch, which covers the position delete check and matchEqualityDeletesToData, and in the single-field key of the equality delete index, which already widened integers. Both widenings are exact. No other type pairs are widened, and comparablePartitionKey, which the writers use for their partition keys, is left alone.

The new tests in table/partition_promotion_deletes_test.go write data, promote the partition source and then delete. They cover int to long and float to double with a v2 merge-on-read Delete, and float to double equality deletes on v2 and v3, including a delete in partition 4.6 that must not remove a row in 4.5. They fail on main and pass with the change. Two existing unit cases that pinned the old cross-width behavior now expect a match, and I added cases for differing values, NaN and signed zero across widths, and int vs double. go build ./..., go vet ./..., go test ./table/... and golangci-lint on table pass.

Badal, you said you'd pick up #2610. The fix fell out of this change as one line in equality_delete_index.go, so I included it, but I'm happy to drop it and its tests if you'd rather take it. This doesn't overlap textually with #2600; whichever lands second keeps a redundant int widening that can be trimmed later. Reading old manifests with the current partition type, as Java does, remains the proper long-term fix.

Closes #2605

AI-assisted, reviewed by a human.

@laskoviymishka
laskoviymishka marked this pull request as ready for review November 16, 2026 12:00
After an identity partition source is promoted (int to long, float to
double), manifests written before the promotion still decode its
partition values as int32/float32, while newer manifests decode them as
int64/float64. Partition comparisons treated those equal values as
different. A v2 merge-on-read Delete on an old data file therefore
committed a position delete that scan planning then rejected with
"partition does not match" (apache#2605), and equality deletes in a promoted
float partition skipped the old data files, so deleted rows came back
(apache#2610).

partitionsMatch now widens int32 to int64 and float32 to float64 before
comparing. This covers the position delete check and
matchEqualityDeletesToData. The equality delete index already widened
integers in its single-field key; it now widens float32 to float64 too.
Both widenings are exact, NaNs still compare equal and -0.0 and +0.0
still differ. No other type pairs are widened, and
comparablePartitionKey, which the writers use for partition keys, is
unchanged.

The longer-term fix is to decode old manifests with the current
partition type, as Java does; that stays a follow-up.

Signed-off-by: laskoviymishka <laskoviymishka@gmail.com>
@laskoviymishka
laskoviymishka force-pushed the factory/rc-2605-widen-partition-values branch from 2fe44bc to db47e85 Compare November 16, 2026 14:54

@badalprasadsingh badalprasadsingh left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approved :)

an optional suggestion to mention this in the release notes though

EDIT: Regarding #2610, it's absolutely fine @laskoviymishka :) no need to drop it from the PR, we can move ahead. I checked the fix and the added tests, and both look good to me. lets add the closing keyword for #2610 as well.

Signed-off-by: laskoviymishka <laskoviymishka@gmail.com>
Signed-off-by: laskoviymishka <laskoviymishka@gmail.com>
Signed-off-by: laskoviymishka <laskoviymishka@gmail.com>

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Widening is exact, old float32 partitions read back as the same float64 the delete writer records, and canonicalPartitionKey already treats the widths alike, so conflict validation and dynamic overwrite agree with this. I reverted scanner.go and equality_delete_index.go to the merge base, and both new end-to-end tests fail. With the fix they pass under -race.

Before merging:

  • Rebase onto main. The only conflict is in equality_delete_index.go, from the comparablePartitionKey → tblutils.ComparablePartitionKey rename in #2614. Keep the float32/float64 split and use the new name.
  • +1 to @badalprasadsingh: please add Closes #2610.
  • Nit: the dynamodb_integration_test.go reflow duplicates #2621.

This review was drafted by an AI-assisted tool and confirmed by an Apache Iceberg maintainer. The maintainer approving this PR has read the findings and signed off. If something feels off, please reply on the PR and a maintainer will follow up.

More on how Apache Iceberg Go handles contributions: CONTRIBUTING.md.

@nssalian nssalian left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reverted scanner.go and equality_delete_index.go to the merge base and re-ran the new tests. TestMergeOnReadDeleteAfterPartitionSourcePromotion and TestEqualityDeletesAfterFloatPartitionSourcePromotion both fail there, along with the new cross-width unit cases. All of them pass on this head.

I also tried the one other type promotion the spec allows, decimal precision (for example decimal(9,2) to decimal(18,2)). A merge-on-read delete on an identity partition promoted that way already works on this head, because the decimal value keeps the same Go type. So int and float are the only promotions that needed widening.

A small note on the description: it says two existing unit cases now expect a match, but four were changed. In scanner_internal_test.go they are "float widths", "float NaN across widths", "float widened to double matches" and "float widened to double".

Follow-up, not for this PR: the same int32/int64 split shows up in the partitions metadata table. I promoted an identity int partition to long and appended p=5 before and after. tbl.Inspect().Partitions() then returned two rows for p=5, both with spec 0. The partition aggregate keys rows by ComparablePartitionKey (table/internal/partition_aggregate.go:68), which keeps the two widths apart, and CoercePartition (:135) doesn't widen them. Partition stats use the same aggregate.

return table.New(table.Identifier{"db", "partition_promotion"}, meta, metaLoc, fsF, cat)
}

func appendPromotionRows(t *testing.T, tbl *table.Table, rows string) *table.Table {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

appendPromotionRows does the same thing as the existing appendRows helper in cow_rewrite_deletes_test.go:408: it builds an Arrow table from the JSON rows using the table's own schema, then calls tbl.Append. Both files are in package table_test, so the two call sites (lines 115 and 134) can use appendRows directly and this helper can be removed.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, they do the same thing. Dropped the helper and switched both call sites to appendRows in 44c8e3f.

AI-assisted, reviewed by a human.

Drop appendPromotionRows, which duplicated the existing appendRows helper.

Signed-off-by: laskoviymishka <laskoviymishka@gmail.com>
@laskoviymishka

laskoviymishka commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks for checking this against the merge base, and for trying the decimal promotion too. Good to know int and float are the only cases that need widening. You're right about the description, four cases changed in scanner_internal_test.go, not two. I'll fix that. The partitions metadata table split looks like the same bug in another spot. I'll open a separate issue for it so this PR stays focused.

AI-assisted, reviewed by a human.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Scan: after promoting an identity partition column, a merge-on-read Delete makes every scan fail

4 participants