Repository navigation
fix(table): compare promoted partition values after widening - #2620
laskoviymishka merged 5 commits into
Conversation
After an identity partition source is promoted (int to long, float to double), manifests written before the promotion still decode its partition values as int32/float32, while newer manifests decode them as int64/float64. Partition comparisons treated those equal values as different. A v2 merge-on-read Delete on an old data file therefore committed a position delete that scan planning then rejected with "partition does not match" (apache#2605), and equality deletes in a promoted float partition skipped the old data files, so deleted rows came back (apache#2610). partitionsMatch now widens int32 to int64 and float32 to float64 before comparing. This covers the position delete check and matchEqualityDeletesToData. The equality delete index already widened integers in its single-field key; it now widens float32 to float64 too. Both widenings are exact, NaNs still compare equal and -0.0 and +0.0 still differ. No other type pairs are widened, and comparablePartitionKey, which the writers use for partition keys, is unchanged. The longer-term fix is to decode old manifests with the current partition type, as Java does; that stays a follow-up. Signed-off-by: laskoviymishka <laskoviymishka@gmail.com>
2fe44bc to
db47e85
Compare
There was a problem hiding this comment.
approved :)
an optional suggestion to mention this in the release notes though
EDIT: Regarding #2610, it's absolutely fine @laskoviymishka :) no need to drop it from the PR, we can move ahead. I checked the fix and the added tests, and both look good to me. lets add the closing keyword for #2610 as well.
Signed-off-by: laskoviymishka <laskoviymishka@gmail.com>
Signed-off-by: laskoviymishka <laskoviymishka@gmail.com>
Signed-off-by: laskoviymishka <laskoviymishka@gmail.com>
zeroshade
left a comment
There was a problem hiding this comment.
LGTM. Widening is exact, old float32 partitions read back as the same float64 the delete writer records, and canonicalPartitionKey already treats the widths alike, so conflict validation and dynamic overwrite agree with this. I reverted scanner.go and equality_delete_index.go to the merge base, and both new end-to-end tests fail. With the fix they pass under -race.
Before merging:
- Rebase onto main. The only conflict is in
equality_delete_index.go, from thecomparablePartitionKey→tblutils.ComparablePartitionKeyrename in #2614. Keep the float32/float64 split and use the new name. - +1 to @badalprasadsingh: please add
Closes #2610. - Nit: the
dynamodb_integration_test.goreflow duplicates #2621.
This review was drafted by an AI-assisted tool and confirmed by an Apache Iceberg maintainer. The maintainer approving this PR has read the findings and signed off. If something feels off, please reply on the PR and a maintainer will follow up.
More on how Apache Iceberg Go handles contributions: CONTRIBUTING.md.
nssalian
left a comment
There was a problem hiding this comment.
I reverted scanner.go and equality_delete_index.go to the merge base and re-ran the new tests. TestMergeOnReadDeleteAfterPartitionSourcePromotion and TestEqualityDeletesAfterFloatPartitionSourcePromotion both fail there, along with the new cross-width unit cases. All of them pass on this head.
I also tried the one other type promotion the spec allows, decimal precision (for example decimal(9,2) to decimal(18,2)). A merge-on-read delete on an identity partition promoted that way already works on this head, because the decimal value keeps the same Go type. So int and float are the only promotions that needed widening.
A small note on the description: it says two existing unit cases now expect a match, but four were changed. In scanner_internal_test.go they are "float widths", "float NaN across widths", "float widened to double matches" and "float widened to double".
Follow-up, not for this PR: the same int32/int64 split shows up in the partitions metadata table. I promoted an identity int partition to long and appended p=5 before and after. tbl.Inspect().Partitions() then returned two rows for p=5, both with spec 0. The partition aggregate keys rows by ComparablePartitionKey (table/internal/partition_aggregate.go:68), which keeps the two widths apart, and CoercePartition (:135) doesn't widen them. Partition stats use the same aggregate.
| return table.New(table.Identifier{"db", "partition_promotion"}, meta, metaLoc, fsF, cat) | ||
| } | ||
|
|
||
| func appendPromotionRows(t *testing.T, tbl *table.Table, rows string) *table.Table { |
There was a problem hiding this comment.
appendPromotionRows does the same thing as the existing appendRows helper in cow_rewrite_deletes_test.go:408: it builds an Arrow table from the JSON rows using the table's own schema, then calls tbl.Append. Both files are in package table_test, so the two call sites (lines 115 and 134) can use appendRows directly and this helper can be removed.
There was a problem hiding this comment.
Agreed, they do the same thing. Dropped the helper and switched both call sites to appendRows in 44c8e3f.
AI-assisted, reviewed by a human.
Drop appendPromotionRows, which duplicated the existing appendRows helper. Signed-off-by: laskoviymishka <laskoviymishka@gmail.com>
|
Thanks for checking this against the merge base, and for trying the decimal promotion too. Good to know int and float are the only cases that need widening. You're right about the description, four cases changed in scanner_internal_test.go, not two. I'll fix that. The partitions metadata table split looks like the same bug in another spot. I'll open a separate issue for it so this PR stays focused.
|
After an identity partition source is promoted (int to long, float to double), manifests written before the promotion still decode its partition values as int32/float32, and newer ones as int64/float64. So a v2 merge-on-read Delete on an old file commits a position delete that the scan then rejects with "partition does not match" (#2605), and equality deletes in a promoted float partition skip the old files (#2610). This change widens int32 to int64 and float32 to float64 where partition values are compared: in
partitionsMatch, which covers the position delete check andmatchEqualityDeletesToData, and in the single-field key of the equality delete index, which already widened integers. Both widenings are exact. No other type pairs are widened, andcomparablePartitionKey, which the writers use for their partition keys, is left alone.The new tests in
table/partition_promotion_deletes_test.gowrite data, promote the partition source and then delete. They cover int to long and float to double with a v2 merge-on-read Delete, and float to double equality deletes on v2 and v3, including a delete in partition 4.6 that must not remove a row in 4.5. They fail on main and pass with the change. Two existing unit cases that pinned the old cross-width behavior now expect a match, and I added cases for differing values, NaN and signed zero across widths, and int vs double.go build ./...,go vet ./...,go test ./table/...and golangci-lint ontablepass.Badal, you said you'd pick up #2610. The fix fell out of this change as one line in
equality_delete_index.go, so I included it, but I'm happy to drop it and its tests if you'd rather take it. This doesn't overlap textually with #2600; whichever lands second keeps a redundant int widening that can be trimmed later. Reading old manifests with the current partition type, as Java does, remains the proper long-term fix.Closes #2605
AI-assisted, reviewed by a human.