Repository navigation
Description
Apache Iceberg version
1.12.0 (latest release)
Query engine
Spark
Please describe the bug 🐞
truncate(W) on an int or long column wraps around for values near Integer.MIN_VALUE / Long.MIN_VALUE. Partition pruning trusts the wrapped value, so scans silently drop matching rows. This happens with default settings, no error is raised.
truncate(10, -2147483648) should be -2147483650, which does not fit in an int. TruncateUtil computes value - (((value % width) + width) % width) in int arithmetic, so it gives 2147483646. truncate(10, -2147483645) also gives 2147483646, and truncate(10, Long.MIN_VALUE) gives 9223372036854775806. The file holding these rows is recorded in the same partition as values near INT_MAX.
Inclusive projection of a predicate literal near the minimum wraps in the same way. For example i > -2147483648 projects to a lower bound near INT_MAX, so almost every file is pruned.
Reproduction (Spark 4.0.1, iceberg-spark-runtime-4.0_2.13:1.12.0, hadoop catalog)
CREATE TABLE hc.db.t (id INT, i INT, l BIGINT) USING iceberg
PARTITIONED BY (truncate(10, i), truncate(10, l));
INSERT INTO hc.db.t VALUES
(0, -2147483648, -9223372036854775808),
(1, -2147483645, -9223372036854775805),
(2, -21, -21),
(3, 5, 5),
(4, 2147483647, 9223372036854775807);
SELECT partition, record_count FROM hc.db.t.files;
-- {i_trunc=2147483646, l_trunc=9223372036854775806} 2 <- ids 0 and 1 (the MIN_VALUE rows)
-- {i_trunc=2147483640, l_trunc=9223372036854775800} 1 <- id 4 (MAX_VALUE, correct)
-- {i_trunc=-30, l_trunc=-30} 1
-- {i_trunc=0, l_trunc=0} 1| Query | Expected ids | Actual ids |
|---|---|---|
SELECT id FROM hc.db.t WHERE i < 0 |
0, 1, 2 | 2 |
... WHERE i < -21 |
0, 1 | (none) |
... WHERE i <= -21 |
0, 1, 2 | 2 |
... WHERE i > -2147483648 |
1, 2, 3, 4 | 1 |
... WHERE i >= -2147483645 |
1, 2, 3, 4 | 1 |
... WHERE i BETWEEN -2147483648 AND -2147483640 |
0, 1 | (none) |
... WHERE l < -21 |
0, 1 | (none) |
... WHERE l > -9223372036854775808 |
1, 2, 3, 4 | 1 |
i = -2147483648 and i IN (-2147483648, 5) return the right rows in Spark.
Other implementations
- pyiceberg 0.12.0 writes the same wrapped partition values (2147483646 / 9223372036854775806). Its reader also loses
i = -2147483648,i IN (-2147483648, 5)andl = -9223372036854775808, because its Python transform of the literal gives the unwrapped-2147483650, which does not match the stored value. - Trino 484 (Iceberg connector) loses the same rows for
</>predicates on these tables.
Relation to existing issues
- API: Integer truncate overflow can prune matching data files #18111 / API: Prevent intermediate overflow in integer truncation #18112 fix an intermediate overflow at width
Integer.MAX_VALUE, where the true result fits. That issue explicitly sets aside the case where the true result does not fit, which is this one. - Iceberg Truncate Transform on Numeric type #12915 mentioned in a comment that truncating
Integer.MIN_VALUEcan overflow; it was closed as stale without a fix, and without the wrong-result effect on scans. - Decimal truncate transform can produce values that exceed the source precision #17027 is the decimal version (the truncated value exceeds the source precision; there the write fails rather than returning wrong results).
Possible directions: reject or clamp values whose truncated result is not representable (e.g. map them to MIN_VALUE aligned down as far as fits, and make projection treat that bucket as unbounded below), or have inclusive projection never prune a partition whose value is greater than the source value it was computed from. Either way, the transform and the projection need to agree, and the spec should say what truncate returns when the result is out of range.
Found while cross-checking the same tables across several readers.
Willingness to contribute
- I can contribute a fix for this bug independently
- I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- I cannot contribute a fix for this bug at this time
Activity
adiranjan25 commentedon Nov 19, 2026
Hi @mhatipog,
Thanks for the detailed reproduction. I'd be interested in contributing a fix for this issue.
The silent row omission caused by truncation overflow appears particularly important. I've reviewed the integer/long truncate paths and predicate projection logic, and both seem relevant to a complete fix.
My proposed approach is to add regression tests for MIN_VALUE boundaries, inclusive/strict projection, and Spark query correctness, then implement an overflow-safe solution.
Before changing the transform behavior, I'd appreciate maintainer guidance on the intended semantics for unrepresentable results and compatibility with existing partition values.
Happy to work on this and submit a focused PR once the approach is agreed upon.
Thanks!
adiranjan25 commentedon Nov 19, 2026
I reproduced the truncation overflow on the current main branch using width 10.
truncateInt(10, Integer.MIN_VALUE)returns2147483646.truncateLong(10, Long.MIN_VALUE)returns9223372036854775806.Both results wrap to large positive values because the mathematical truncated values are outside the representable ranges.
I've added a failing regression test and am investigating the appropriate overflow handling. Would the maintainers prefer explicit rejection of unrepresentable results, or is there an established compatibility requirement for these boundary values?
I'd be happy to prepare a focused PR based on the preferred approach.
adiranjan25 commentedon Nov 19, 2026
Hi @mhatipog,
Quick update: I've opened PR #19045 to track the fix for this issue.
I've reproduced the integer and long truncation overflow on the current main branch and added an initial regression test demonstrating the problem.
The PR is still a work in progress. I'm investigating an overflow-safe implementation that also preserves correct partition pruning and compatibility with existing partition values.
I'd appreciate maintainer feedback on the preferred handling of unrepresentable truncated values before finalizing the implementation.
Thanks!