Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 92 additions & 3 deletions crates/iceberg/src/transform/truncate.rs
Original file line number Diff line number Diff line change
Expand Up @@ -50,17 +50,24 @@ impl Truncate {

#[inline]
fn truncate_i32(v: i32, width: i32) -> i32 {
v - v.rem_euclid(width)
// The spec requires a non-negative remainder, which `rem_euclid` computes without
// overflow. Only the subtraction can overflow, when the result is below `i32::MIN`.
// The spec doesn't cover that case, so wrap as Java's `TruncateUtil` does.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this also say the wrapped value isn't order-preserving? At width 1000, i32::MIN lands on 2_147_483_296, and truncate projection assumes order (adjust_boundary). As I read it, value < i32::MIN + 700 projects to name <= -2147483000 and skips that partition. Release builds on main wrap the same way, so this is only about the comment.

Suggested change
// The spec doesn't cover that case, so wrap as Java's `TruncateUtil` does.
// The spec doesn't cover that case, so wrap as Java's `TruncateUtil` does.
// The wrapped result isn't order-preserving, which truncate projection assumes.

// The wrapped result isn't order-preserving, which truncate projection assumes.
v.wrapping_sub(v.rem_euclid(width))
}

#[inline]
fn truncate_i64(v: i64, width: i64) -> i64 {
v - (((v % width) + width) % width)
// See `truncate_i32`.
v.wrapping_sub(v.rem_euclid(width))
}

#[inline]
fn truncate_decimal_i128(v: i128, width: i128) -> i128 {
v - (((v % width) + width) % width)
// Only mantissas outside the 38-digit decimal range can wrap. Java's `truncateDecimal`
// uses `BigInteger`, so there is no Java result to match and the wrap only avoids a panic.
v.wrapping_sub(v.rem_euclid(width))
}
Comment on lines 52 to 71

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should truncate_i32 keep rem_euclid for the remainder? The spec defines int and long truncation as v - (v % W) and says "remainders must be positive" (spec). Note 1 offers v - (((v % W) + W) % W) as a way to get that positive remainder in languages where % can be negative. As I read it, the formula is a means to the stated requirement, not a requirement to overflow in 32 bits. For W > 2^30, Java's (v % W) + W overflows and the remainder comes out negative, so Java's value breaks the rule the note is there to satisfy. rem_euclid gave the spec value.

Java is treating this as a bug. apache/iceberg#18111 reports it, and apache/iceberg#18112 changes truncateInt to Math.floorMod(value, width), which matches rem_euclid. apache/iceberg-cpp#1042 also follows v - (v % W) here and returns 0 for truncate[2147483646](2147483645). PyIceberg computes v - v % self._width on unbounded ints, so it gets 0 too.

The Java value also breaks scan planning in iceberg-rust. Truncate projection relies on truncate preserving order. At the head commit, truncate[2000000000] maps 1 to 0 and 1_000_000_000 to -2_000_000_000. The predicate value >= 1 projects to name >= 0, which prunes the partition -2_000_000_000 even though it holds 1_000_000_000.

The integer minimum is a different case. There the spec result doesn't fit in the type, the spec doesn't say what to do, and Java wraps (still true after apache/iceberg#18112, see this thread). Keeping wrapping_sub for that case matches Java and fixes the debug panic.

Could all three helpers use the same shape? For i64 and i128 this gives the same values as the head commit, because (v % width) + width can't overflow with a u32 width. The difference is only in i32.

Suggested change
fn truncate_i32(v: i32, width: i32) -> i32 {
v - v.rem_euclid(width)
// Match Java TruncateUtil, including int overflow wrapping.
v.wrapping_sub((v % width).wrapping_add(width) % width)
}
#[inline]
fn truncate_i64(v: i64, width: i64) -> i64 {
v - (((v % width) + width) % width)
// Match Java TruncateUtil, including long overflow wrapping.
v.wrapping_sub((v % width).wrapping_add(width) % width)
}
#[inline]
fn truncate_decimal_i128(v: i128, width: i128) -> i128 {
v - (((v % width) + width) % width)
// Match Java TruncateUtil. Java uses BigInteger here, so wrapping only keeps
// out-of-range i128 values from panicking.
v.wrapping_sub((v % width).wrapping_add(width) % width)
}
fn truncate_i32(v: i32, width: i32) -> i32 {
// The spec requires a non-negative remainder, which `rem_euclid` computes without
// overflow. Only the subtraction can overflow, when the result is below `i32::MIN`.
// The spec doesn't cover that case, so wrap as Java's `TruncateUtil` does.
v.wrapping_sub(v.rem_euclid(width))
}
#[inline]
fn truncate_i64(v: i64, width: i64) -> i64 {
// See `truncate_i32`.
v.wrapping_sub(v.rem_euclid(width))
}
#[inline]
fn truncate_decimal_i128(v: i128, width: i128) -> i128 {
// See `truncate_i32`. Only mantissas outside the 38-digit decimal range can wrap.
v.wrapping_sub(v.rem_euclid(width))
}

}

Expand Down Expand Up @@ -826,6 +833,22 @@ mod test {
Ok(())
}

#[test]
fn test_projection_truncate_integer_large_width() -> Result<()> {
let fixture = TestProjectionFixture::new(
Transform::Truncate(2_000_000_000),
"name",
NestedField::required(1, "value", Primitive(Int)),
);

fixture.assert_projection(
&fixture.binary_predicate(PredicateOperator::Eq, Datum::int(1_500_000_000)),
Some("name = 0"),
)?;

Ok(())
}

// Test case ref from: https://iceberg.apache.org/spec/#truncate-transform-details
#[test]
fn test_truncate_simple() {
Expand Down Expand Up @@ -902,6 +925,32 @@ mod test {
);
}

#[test]
fn test_truncate_long_min_matches_java() {
// Expected values computed with Iceberg Java's `TruncateUtil.truncateLong`
// (iceberg 1.11.0, JDK 17).
for (width, value, expected) in [
(1000, i64::MIN, 9_223_372_036_854_775_616), // i64::MAX - 191: rem_euclid is 192
// i64::MAX - 2_147_483_644: rem_euclid is 2_147_483_645
(i32::MAX as u32, i64::MIN, 9_223_372_034_707_292_163),
] {
let truncate = super::Truncate::new(width);
let input = Arc::new(Int64Array::from(vec![Some(value), None]));
let result = truncate.transform(input).unwrap();
let expected_array = Int64Array::from(vec![Some(expected), None]);
assert_eq!(
result.as_any().downcast_ref::<Int64Array>(),
Some(&expected_array),
"width: {width}, value: {value}"
);
assert_eq!(
truncate.transform_literal(&Datum::long(value)).unwrap(),
Some(Datum::long(expected)),
"width: {width}, value: {value}"
);
}
}

#[test]
fn test_string_truncate() {
let test1 = "イロハニホヘト";
Expand Down Expand Up @@ -968,6 +1017,35 @@ mod test {
assert_eq!(res, Datum::int(-10),);
}

#[test]
fn test_truncate_int_large_width() {
// Expected values follow the spec: `v - (v % W)` with a non-negative remainder.
// Below `i32::MIN` the spec is silent, and the result wraps as Java's `TruncateUtil` does.
for (width, value, expected) in [
(2_000_000_000, 1_500_000_000, 0),
(i32::MAX as u32, i32::MAX - 1, 0),
(1000, i32::MIN, 2_147_483_296), // i32::MAX - 351: rem_euclid is 352
// Smallest width where `(v % w) + w` can overflow i32.
(1_073_741_825, 1_073_741_824, 0),
(i32::MAX as u32, 1, 0),
] {
let truncate = super::Truncate::new(width);
let input = Arc::new(Int32Array::from(vec![Some(value), None]));
let result = truncate.transform(input).unwrap();
let expected_array = Int32Array::from(vec![Some(expected), None]);
assert_eq!(
result.as_any().downcast_ref::<Int32Array>(),
Some(&expected_array),
"width: {width}, value: {value}"
);
assert_eq!(
truncate.transform_literal(&Datum::int(value)).unwrap(),
Some(Datum::int(expected)),
"width: {width}, value: {value}"
);
}
}

#[test]
fn test_literal_long() {
let input = Datum::long(1);
Expand Down Expand Up @@ -1010,6 +1088,17 @@ mod test {
}
}

#[test]
fn test_truncate_decimal_i128_min_does_not_panic() {
// `i128::MIN` is outside the range of a 38-digit Iceberg decimal, and Java's
// `TruncateUtil.truncateDecimal` works on `BigInteger`, so there is no wrapping
// result to match.
assert_eq!(
super::Truncate::truncate_decimal_i128(i128::MIN, 1000),
170_141_183_460_469_231_731_687_303_715_884_105_456, // i128::MAX - 271
);
}

#[test]
fn test_binary_literal() {
for (value, width, expected) in [
Expand Down
Loading