Repository navigation
fix(table): report a promoted identity partition once in the partitions table and partition statistics - #2637
Conversation
…ns table and partition statistics After an identity partition source is promoted from int to long (or float to double), manifests written before the promotion decode the partition value as int32 (float32) and newer ones as int64 (float64). The partition aggregate tree keys those apart, so Inspect().Partitions() and partition statistics returned two rows for the same partition. CoercePartition and PartitionAggregateTree.LookupPartition now widen int32 to int64 when the partition field is long and float32 to float64 when it is double. Both widenings are exact, so values that differ still land in separate rows. ComparablePartitionKey is unchanged. Signed-off-by: laskoviymishka <laskoviymishka@gmail.com>
| case float32: | ||
| if _, ok := typ.(iceberg.Float64Type); ok { | ||
| return float64(v) | ||
| } |
There was a problem hiding this comment.
not new, same cause: after a float -> double promotion, every other inspect table with a partition column fails with cannot append scalar of type float32 to builder for type float64
a Float64Type case in PartitionValueScalar fixes all
There was a problem hiding this comment.
+1, I can reproduce this on this head. After promoting an identity partition source from float to double and appending rows before and after, tbl.Inspect().Files(), Entries() and AllEntries() fail with "cannot append scalar of type float32 to builder for type float64", while Partitions() now works. Those tables build partition values through PartitionStructBuilder, which falls back to PartitionValueScalar (table/internal/partition_struct_builder.go:168), and that function has no float case. The position deletes table uses the same builder.
Reusing the helper from this PR fixes all of them. I tried this locally and all three tables return both rows:
func PartitionValueScalar(value any, typ iceberg.Type, arrowType arrow.DataType) (scalar.Scalar, error) {
value = widenPartitionValue(value, typ)
switch typ.(type) {For the end-to-end test you asked for, a float to double case that appends one row before and one after the promotion and reads both Files() and Partitions() would cover this fix and the one in the PR. It fails on main today.
| require.Equal(t, amount.Val, partition.Field(0).(*array.Decimal128).Value(0)) | ||
| } | ||
|
|
||
| func TestCollectPartitionStatisticsPromotedIdentitySource(t *testing.T) { |
There was a problem hiding this comment.
the inspect.Partitions test can go in now as #2631 is merged
lets mirror this test in "table/inspect" with "inspectTableWithManifestListAndSchemas": one case with only the old manifest (two files, the single-manifest path), one with old plus new, lets also Include the float to double case, which fails on main today
nssalian
left a comment
There was a problem hiding this comment.
Thanks for picking this up. I checked it end to end: I promoted an identity partition source, appended rows before and after the promotion, and called tbl.Inspect().Partitions(). For int to long it returned two rows for p=5 on main and one row on this head. For float to double it failed on main with "cannot append scalar of type float32 to builder for type float64", and returns one row on this head. The new unit tests also fail with partition_aggregate.go at the merge base and pass here.
Two things I'd change before merge:
-
The other inspect tables still fail after a float to double promotion (details in the thread on
partition_aggregate.go:164). Calling the helper this PR adds at the top ofPartitionValueScalarintable/internal/partition_struct_builder.gofixes it:value = widenPartitionValue(value, typ). I tried that locally andFiles(),Entries()andAllEntries()then return both rows, whilePartitions()still returns one. -
The title.
ComputePartitionStatisticsstill rejects tables whose identity source type changed (validatePartitionStatisticsSourceTypes,table/partition_statistics_write.go:172), as the description says, so the partition statistics part of this change is only reachable through the internalcollectPartitionStatisticsthat the new test calls. Since the title becomes the commit message, I'd suggestfix(table): report a promoted identity partition once in the partitions table.
Follow-up to #2605, found by @nssalian in review of #2620. Promote an identity
intpartition source tolong, appendp=5before and after, andtbl.Inspect().Partitions()returns two rows forp=5. Old manifests decode the value as int32 and new ones as int64, and the partition aggregate tree keys them apart.CoercePartitionandLookupPartitionnow widen int32 to int64 for long fields and float32 to float64 for double fields, so the partitions table and partition statistics agree without changes of their own. Both widenings are exact, so nothing that differed before now matches.Added unit cases in
table/internalfor int to long, float to double (including NaN), a float32 value that still differs from the float64 literal after widening, and an int field that does not widen. A partition statistics test reads one manifest withp=int32(5)and one withp=int64(5)against a long partition type and expects a single row; it returns two rows on main. The inspect path goes through the same two helpers and is covered by thetable/internalcases; an end-to-endInspect().Partitions()test fits better with #2631, which moves the inspect code and tests.ComputePartitionStatisticsstill rejects tables whose identity source type changed. Lifting that is left for a separate change.AI-assisted, reviewed by a human.