Skip to content

fix(table): report a promoted identity partition once in the partitions table and partition statistics - #2637

Open
laskoviymishka wants to merge 1 commit into
apache:mainfrom
laskoviymishka:factory/partition-aggregate-widen-promoted
Open

laskoviymishka wants to merge 1 commit into
apache:mainfrom
laskoviymishka:factory/partition-aggregate-widen-promoted

Conversation

@laskoviymishka

Copy link
Copy Markdown
Contributor

Follow-up to #2605, found by @nssalian in review of #2620. Promote an identity int partition source to long, append p=5 before and after, and tbl.Inspect().Partitions() returns two rows for p=5. Old manifests decode the value as int32 and new ones as int64, and the partition aggregate tree keys them apart. CoercePartition and LookupPartition now widen int32 to int64 for long fields and float32 to float64 for double fields, so the partitions table and partition statistics agree without changes of their own. Both widenings are exact, so nothing that differed before now matches.

Added unit cases in table/internal for int to long, float to double (including NaN), a float32 value that still differs from the float64 literal after widening, and an int field that does not widen. A partition statistics test reads one manifest with p=int32(5) and one with p=int64(5) against a long partition type and expects a single row; it returns two rows on main. The inspect path goes through the same two helpers and is covered by the table/internal cases; an end-to-end Inspect().Partitions() test fits better with #2631, which moves the inspect code and tests.

ComputePartitionStatistics still rejects tables whose identity source type changed. Lifting that is left for a separate change.

AI-assisted, reviewed by a human.

…ns table and partition statistics

After an identity partition source is promoted from int to long (or float
to double), manifests written before the promotion decode the partition
value as int32 (float32) and newer ones as int64 (float64). The partition
aggregate tree keys those apart, so Inspect().Partitions() and partition
statistics returned two rows for the same partition.

CoercePartition and PartitionAggregateTree.LookupPartition now widen
int32 to int64 when the partition field is long and float32 to float64
when it is double. Both widenings are exact, so values that differ still
land in separate rows. ComparablePartitionKey is unchanged.

Signed-off-by: laskoviymishka <laskoviymishka@gmail.com>

@badalprasadsingh badalprasadsingh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

few comments

Comment on lines +161 to +164
case float32:
if _, ok := typ.(iceberg.Float64Type); ok {
return float64(v)
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not new, same cause: after a float -> double promotion, every other inspect table with a partition column fails with cannot append scalar of type float32 to builder for type float64

a Float64Type case in PartitionValueScalar fixes all

@nssalian nssalian Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, I can reproduce this on this head. After promoting an identity partition source from float to double and appending rows before and after, tbl.Inspect().Files(), Entries() and AllEntries() fail with "cannot append scalar of type float32 to builder for type float64", while Partitions() now works. Those tables build partition values through PartitionStructBuilder, which falls back to PartitionValueScalar (table/internal/partition_struct_builder.go:168), and that function has no float case. The position deletes table uses the same builder.

Reusing the helper from this PR fixes all of them. I tried this locally and all three tables return both rows:

func PartitionValueScalar(value any, typ iceberg.Type, arrowType arrow.DataType) (scalar.Scalar, error) {
        value = widenPartitionValue(value, typ)

        switch typ.(type) {

For the end-to-end test you asked for, a float to double case that appends one row before and one after the promotion and reads both Files() and Partitions() would cover this fix and the one in the PR. It fails on main today.

require.Equal(t, amount.Val, partition.Field(0).(*array.Decimal128).Value(0))
}

func TestCollectPartitionStatisticsPromotedIdentitySource(t *testing.T) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the inspect.Partitions test can go in now as #2631 is merged

lets mirror this test in "table/inspect" with "inspectTableWithManifestListAndSchemas": one case with only the old manifest (two files, the single-manifest path), one with old plus new, lets also Include the float to double case, which fails on main today

@nssalian nssalian left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for picking this up. I checked it end to end: I promoted an identity partition source, appended rows before and after the promotion, and called tbl.Inspect().Partitions(). For int to long it returned two rows for p=5 on main and one row on this head. For float to double it failed on main with "cannot append scalar of type float32 to builder for type float64", and returns one row on this head. The new unit tests also fail with partition_aggregate.go at the merge base and pass here.

Two things I'd change before merge:

  1. The other inspect tables still fail after a float to double promotion (details in the thread on partition_aggregate.go:164). Calling the helper this PR adds at the top of PartitionValueScalar in table/internal/partition_struct_builder.go fixes it: value = widenPartitionValue(value, typ). I tried that locally and Files(), Entries() and AllEntries() then return both rows, while Partitions() still returns one.

  2. The title. ComputePartitionStatistics still rejects tables whose identity source type changed (validatePartitionStatisticsSourceTypes, table/partition_statistics_write.go:172), as the description says, so the partition statistics part of this change is only reachable through the internal collectPartitionStatistics that the new test calls. Since the title becomes the commit message, I'd suggest fix(table): report a promoted identity partition once in the partitions table.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants