You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
API: Integer truncate overflow can prune matching data files #18111
With an integer partition transform whose width is Integer.MAX_VALUE, a positive value Integer.MAX_VALUE - 1 is assigned a negative partition key instead of zero. A table scan filtered by id >= 0 then excludes the matching file.
Expected partition: 0. Actual partition: -2147483647.
The regression also computes a PartitionKey, appends a data-file entry carrying it, and checks table.newScan().filter(Expressions.greaterThanOrEqual("id", 0)).planFiles(). The matching file is omitted.
TruncateUtil overflows while adding the positive remainder to the width, before the second remainder operation. This differs from an inherently unrepresentable final partition value: all inputs and the correct result here fit in an int. Related decimal/final-result overflow discussions: #12915 and #17027.
Proposed fix: use an overflow-safe nonnegative remainder for int and the narrower integer helpers. Include transform arithmetic and scan-planning regressions.
Spark SQL reproduction
Run in a configured Iceberg catalog:
CREATETABLEtruncate_repro (id INT) USING iceberg
PARTITIONED BY (truncate(2147483647, id));
INSERT INTO truncate_repro VALUES (2147483646);
SELECT id FROM truncate_repro WHERE id >=0;
On the original implementation the query returns no rows. With the fix it returns 2147483646. A Spark 4.1 regression in #18112 verifies this create/write/read flow for Parquet, Avro, and ORC; all three fail before the fix and pass afterward.
Willingness to contribute
I can contribute a fix for this bug independently
This report and its reproduction were prepared with AI assistance and verified locally.
Activity