Skip to content

fix: leave wide integer truncate to Spark - #7580

Draft
ErikBPF wants to merge 6 commits into
apache:mainfrom
ErikBPF:fix/7442-truncate-width-fallback
Draft

ErikBPF wants to merge 6 commits into
apache:mainfrom
ErikBPF:fix/7442-truncate-width-fallback

Conversation

@ErikBPF

@ErikBPF ErikBPF commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #7442.

Rationale for this change

At integer truncate widths above 2^30, Iceberg Java's (v % W) + W overflows a 32-bit int, while the format spec and iceberg-rust compute v - (v % W). Every Iceberg release through 1.12.0 overflows, and a release carrying apache/iceberg#18112 returns the spec value, so the expected answer depends on the Iceberg jar the user runs. Comet cannot choose from the Spark profile, because users pick their Iceberg runtime independently.

What changes are included in this PR?

  • CometIcebergTruncate declines an integer truncate[W] with W > 2^30 for tinyint, smallint, int and date, so Spark evaluates the call with the user's jar. Long, string and binary truncation stays native: truncateLong uses long arithmetic and W is an int, so it cannot overflow at these widths.
  • A new write trigger declines a native Iceberg write whose output partition spec has an integer truncate[W] field above that width. The native writer would write the spec value while the user's jar writes the overflowed one into the same table.
  • The Rust test comment now attributes the overflow to Java (API: Integer truncate overflow can prune matching data files iceberg#18111, fix in API: Prevent intermediate overflow in integer truncation iceberg#18112) and cites the spec, keeping i32::MIN / i64::MIN as the iceberg-rust cases.
  • The user guide states the limit in the system-functions section and in the write eligibility table, and getUnsupportedReasons lists it.

How are these changes tested?

  • Support-level assertions at 2^30 and above for int, and for long at the same width.
  • An expression fallback test that compares Comet's answer with Spark's for truncate(2000000000, i32), and keeps Int.MaxValue coverage for long and string.
  • Write-detection tests for a truncate(2000000000, id) partition spec (fall-back) and for the maximum safe width (Compatible).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Iceberg truncate on integers with widths above 2^30: native kernel and native writer compute different partition values

1 participant