Skip to content

Truncate transform wraps for values near Integer.MIN_VALUE / Long.MIN_VALUE, so scans silently drop rows #19023

Open

Description

@mhatipog

Apache Iceberg version

1.12.0 (latest release)

Query engine

Spark

Please describe the bug 🐞

truncate(W) on an int or long column wraps around for values near Integer.MIN_VALUE / Long.MIN_VALUE. Partition pruning trusts the wrapped value, so scans silently drop matching rows. This happens with default settings, no error is raised.

truncate(10, -2147483648) should be -2147483650, which does not fit in an int. TruncateUtil computes value - (((value % width) + width) % width) in int arithmetic, so it gives 2147483646. truncate(10, -2147483645) also gives 2147483646, and truncate(10, Long.MIN_VALUE) gives 9223372036854775806. The file holding these rows is recorded in the same partition as values near INT_MAX.

Inclusive projection of a predicate literal near the minimum wraps in the same way. For example i > -2147483648 projects to a lower bound near INT_MAX, so almost every file is pruned.

Reproduction (Spark 4.0.1, iceberg-spark-runtime-4.0_2.13:1.12.0, hadoop catalog)

CREATE TABLE hc.db.t (id INT, i INT, l BIGINT) USING iceberg
  PARTITIONED BY (truncate(10, i), truncate(10, l));

INSERT INTO hc.db.t VALUES
  (0, -2147483648, -9223372036854775808),
  (1, -2147483645, -9223372036854775805),
  (2, -21, -21),
  (3, 5, 5),
  (4, 2147483647, 9223372036854775807);

SELECT partition, record_count FROM hc.db.t.files;
-- {i_trunc=2147483646, l_trunc=9223372036854775806}  2   <- ids 0 and 1 (the MIN_VALUE rows)
-- {i_trunc=2147483640, l_trunc=9223372036854775800}  1   <- id 4 (MAX_VALUE, correct)
-- {i_trunc=-30, l_trunc=-30}                         1
-- {i_trunc=0, l_trunc=0}                             1
Query Expected ids Actual ids
SELECT id FROM hc.db.t WHERE i < 0 0, 1, 2 2
... WHERE i < -21 0, 1 (none)
... WHERE i <= -21 0, 1, 2 2
... WHERE i > -2147483648 1, 2, 3, 4 1
... WHERE i >= -2147483645 1, 2, 3, 4 1
... WHERE i BETWEEN -2147483648 AND -2147483640 0, 1 (none)
... WHERE l < -21 0, 1 (none)
... WHERE l > -9223372036854775808 1, 2, 3, 4 1

i = -2147483648 and i IN (-2147483648, 5) return the right rows in Spark.

Other implementations

  • pyiceberg 0.12.0 writes the same wrapped partition values (2147483646 / 9223372036854775806). Its reader also loses i = -2147483648, i IN (-2147483648, 5) and l = -9223372036854775808, because its Python transform of the literal gives the unwrapped -2147483650, which does not match the stored value.
  • Trino 484 (Iceberg connector) loses the same rows for < / > predicates on these tables.

Relation to existing issues

Possible directions: reject or clamp values whose truncated result is not representable (e.g. map them to MIN_VALUE aligned down as far as fits, and make projection treat that bucket as unbounded below), or have inclusive projection never prune a partition whose value is greater than the source value it was computed from. Either way, the transform and the projection need to agree, and the spec should say what truncate returns when the result is out of range.

Found while cross-checking the same tables across several readers.

Willingness to contribute

  • I can contribute a fix for this bug independently
  • I would be willing to contribute a fix for this bug with guidance from the Iceberg community
  • I cannot contribute a fix for this bug at this time

Activity

  1. adiranjan25 commented on Nov 19, 2026

    @adiranjan25

    Hi @mhatipog,

    Thanks for the detailed reproduction. I'd be interested in contributing a fix for this issue.

    The silent row omission caused by truncation overflow appears particularly important. I've reviewed the integer/long truncate paths and predicate projection logic, and both seem relevant to a complete fix.

    My proposed approach is to add regression tests for MIN_VALUE boundaries, inclusive/strict projection, and Spark query correctness, then implement an overflow-safe solution.

    Before changing the transform behavior, I'd appreciate maintainer guidance on the intended semantics for unrepresentable results and compatibility with existing partition values.

    Happy to work on this and submit a focused PR once the approach is agreed upon.

    Thanks!

  2. adiranjan25 commented on Nov 19, 2026

    @adiranjan25

    I reproduced the truncation overflow on the current main branch using width 10.

    • truncateInt(10, Integer.MIN_VALUE) returns 2147483646.
    • truncateLong(10, Long.MIN_VALUE) returns 9223372036854775806.

    Both results wrap to large positive values because the mathematical truncated values are outside the representable ranges.

    I've added a failing regression test and am investigating the appropriate overflow handling. Would the maintainers prefer explicit rejection of unrepresentable results, or is there an established compatibility requirement for these boundary values?

    I'd be happy to prepare a focused PR based on the preferred approach.

  3. adiranjan25 commented on Nov 19, 2026

    @adiranjan25

    Hi @mhatipog,

    Quick update: I've opened PR #19045 to track the fix for this issue.

    I've reproduced the integer and long truncation overflow on the current main branch and added an initial regression test demonstrating the problem.

    The PR is still a work in progress. I'm investigating an overflow-safe implementation that also preserves correct partition pruning and compatibility with existing partition values.

    I'd appreciate maintainer feedback on the preferred handling of unrepresentable truncated values before finalizing the implementation.

    Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions