Repository navigation
Description
Apache Iceberg Rust version
main @ 3d84c81353b1b23b6e4ae8eea8f8a021cc6927a7
Describe the bug
Truncate::transform in crates/iceberg/src/transform/truncate.rs does not reproduce Iceberg Java's TruncateUtil arithmetic. There are two distinct problems.
1. truncate_i32 diverges from Java for widths above 2^30
fn truncate_i32(v: i32, width: i32) -> i32 {
v - v.rem_euclid(width)
}Java computes the positive remainder with an expression that is allowed to overflow:
// org.apache.iceberg.util.TruncateUtil
public static int truncateInt(int width, int value) {
return value - (((value % width) + width) % width);
}(value % width) + width overflows int whenever value % width > Integer.MAX_VALUE - width, which is reachable for any width > 2^30. rem_euclid never overflows, so the two produce different partition values. Only positive values with a large width are affected; negative values and widths at or below 2^30 agree.
Truncate.get(int width) in Iceberg Java only validates width > 0, so these widths are accepted by the spec. Whether Java's overflow is desirable is beside the point — it is the value iceberg-java writes into the partition tuple, so a Rust writer that computes something else produces a table the Java reader partitions differently.
2. Overflow panics in debug builds at the integer minimum
fn truncate_i64(v: i64, width: i64) -> i64 {
v - (((v % width) + width) % width)
}For v = i64::MIN, width = 1000 the remainder is 192 and v - 192 underflows. With debug_assertions on this panics (attempt to subtract with overflow); in a release build it wraps and matches Java exactly. truncate_i32 has the same shape at i32::MIN, and truncate_decimal_i128 uses the same unchecked subtraction (unreachable for a 38-digit Iceberg decimal, but the same pattern).
A panic here aborts the write task, so a debug or test build of any downstream writer fails on data that Java handles.
To Reproduce
Values below were produced by running TruncateUtil from apache-iceberg 1.11.0 on JDK 17 against the Rust expressions:
| width | value | Java TruncateUtil.truncateInt |
iceberg-rust truncate_i32 |
|---|---|---|---|
| 2000000000 | 1500000000 | -2000000000 | 0 |
| 2147483647 | 2147483646 | -2147483647 | 0 |
| 1073741824 | 1500000000 | 1073741824 | 1073741824 |
| 1000 | 2147483647 | 2147483000 | 2147483000 |
For problem 2, in a debug build:
let t = create_transform_function(&Transform::Truncate(1000)).unwrap();
t.transform(Arc::new(Int64Array::from(vec![i64::MIN]))).unwrap(); // panicsExpected behavior
Both kernels bit-identical to Java in debug and release. Wrapping arithmetic gives that:
fn truncate_i32(v: i32, w: i32) -> i32 {
v.wrapping_sub((v % w).wrapping_add(w) % w)
}
fn truncate_i64(v: i64, w: i64) -> i64 {
v.wrapping_sub((v % w).wrapping_add(w) % w)
}That fixes both problems at once: it reproduces Java's overflow for large widths and removes the debug panic. The alternative for problem 1 is to reject width > 2^30 at construction, which overlaps with #2473, but that would reject specs iceberg-java accepts.
Willingness to contribute
I would be willing to contribute a fix for this bug with guidance from the Iceberg community.
How this was found. Apache DataFusion Comet is adding native kernels for Iceberg's Spark system functions (apache/datafusion-comet#5638). On a partitioned write the sort key comes from those kernels (Java semantics) while the partition values the clustered writer groups by come from iceberg-rust's transforms, and the clustered writer fails at runtime with The input is not sorted! Cannot write to partition that was previously closed when the two disagree. Cross-checking the two implementations over boundary inputs is what surfaced this.
This report was drafted with LLM assistance (Claude Code); the divergences were verified against Iceberg Java on a JVM.
Activity
test: link the upstream iceberg-rust issues from the excluded cases
kamcheungting-db commentedon Sep 17, 2026
I’d like to work on this—are you already preparing a fix?
kevinjqliu commentedon Oct 4, 2026
@kamcheungting-db please feel free to open a PR for this. I'm also happy to take a look too.
fix(transform): match Java truncate wrapping arithmetic
fix(transform): match Java truncate wrapping arithmetic
fix(transform): match Java truncate wrapping arithmetic
dragonknightk commentedon Oct 29, 2026
I went ahead and opened #3573 for this — the fix is
wrapping_sub/wrapping_addin all three helpers, matchingTruncateUtil's overflowbehavior, plus regression tests for the
i32width-above-2^30 cases andthe integer-minimum panic (。・ω・。)
@kamcheungting-db you mentioned wanting to pick this up back in September,
so just flagging that I got there first — happy to hand it over or
coordinate if you already have something in progress (๑•̀ㅂ•́)و✧
mbutrovich commentedon Nov 2, 2026
The spec is the reference for every implementation, and Java settles what it leaves open.
As I read the truncate transform details, the spec covers this case. It defines
inttruncation asv - (v % W)and requires a positive remainder. ForW > 2^30, Java's(v % W) + Woverflows and the remainder comes out negative. Java is treating that as a bug in apache/iceberg#18111, with a fix in apache/iceberg#18112. apache/iceberg-cpp#1042 and PyIceberg already produce the spec value. Once #18112 merges, Java will write the spec value too. Matching today's Java would also make iceberg-rust's own scan planning skip files that match the filter (repro).That makes problem 1 a Java bug rather than an iceberg-rust one, since
rem_euclidalready gave the spec value. Problem 2 still stands. At the integer minimum the result doesn't fit in the type and the spec is silent, so wrapping as Java does looks right for the debug panic. Could we update the title and expected behavior to focus on problem 2?More generally, we should take care to reference the spec whenever we report a behavior difference from Java or another implementation. The spec tells us which side needs the fix.