Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion docs/source/user-guide/latest/iceberg.md
Original file line number Diff line number Diff line change
Expand Up @@ -290,6 +290,16 @@ value back unchanged and Spark turns it into null only when the row is materiali
would have to null it during evaluation, which changes what an enclosing predicate or hash sees.
Every other `truncate` input type, and `bucket` on decimals, runs natively.

An integer `truncate` with a width above 2^30 (`1073741824`) also falls back to Spark. At those
widths Iceberg Java's `(v % W) + W` overflows a 32-bit `int`, while the [format
spec](https://github.com/apache/iceberg/blob/main/format/spec.md) defines the result as
`v - (v % W)` with a positive remainder. Every Iceberg release through 1.12.0 overflows, and a
release carrying apache/iceberg#18112 returns the spec value, so the correct answer depends on
the Iceberg jar the user runs, not on the Spark version. Comet cannot pick a shim from the Spark
profile, so it leaves those widths to Spark. The same width limit applies to native writes whose
partition spec has an integer `truncate[W]` field: the native writer computes the spec value,
which would differ from the value the user's Iceberg jar writes into the same table.

This matters most for writes. A partitioned table with the default `write.distribution-mode`
(`hash`) is planned with a shuffle and a local sort keyed on the partition transforms, and with
these functions native the whole sub-plan feeding the [native Iceberg writer](#writes)
Expand Down Expand Up @@ -498,7 +508,7 @@ A write is eligible only when ALL of the following hold:
| resolved `table.locationProvider()` | Iceberg's built-in `DefaultLocationProvider` |
| Hadoop S3A settings for an `s3` / `s3a` data location | only `fs.s3a.access.key`, `secret.key`, `session.token`, `endpoint`, `endpoint.region`, and `path.style.access`, including their `fs.s3a.bucket.<data-bucket>.*` forms; any other effective `fs.s3a.*` setting falls back, apart from the ignored settings listed below |
| Iceberg `FileIO` S3 settings for an `s3` / `s3a` data location | the S3 endpoint, region, static/session credentials, path-style, SSE (`none`, `s3`, `kms`, or `custom`; not `dsse-kms`), assume-role, anonymous/config-chain settings parsed by the pinned iceberg-rust version, plus Comet's credential-provider class and built-in web-identity properties. When a custom provider is configured, its vendor-owned `s3.*` / `client.*` properties are also forwarded; unsupported Iceberg-defined S3 settings still fall back |
| partition spec | any, except a `void` field whose source column was dropped beside a live field (see below) |
| partition spec | any, except a `void` field whose source column was dropped beside a live field (see below), and an integer `truncate[W]` field with `W` above 2^30 (see [Iceberg system functions](#iceberg-system-functions)) |
| column types | any except `uuid` (Spark plans it as a string; no Arrow cast reaches `fixed(16)`) and the v3 types `variant`, `unknown`, `timestamp_ns`, `geometry` and `geography` |

Within the namespaces that shape data-file bytes — `write.parquet.*` and `parquet.*` —
Expand Down
21 changes: 15 additions & 6 deletions native/core/src/execution/operators/iceberg_write.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6140,12 +6140,21 @@ mod iceberg_rust_transform_parity {
}
}

/// `i32::MIN`, `i64::MIN`, and widths above 2^30 are left out: Java's `TruncateUtil` wraps
/// there and iceberg-rust does not (`truncate_i32` uses `rem_euclid`, `truncate_i64` and the
/// decimal kernel subtract without wrapping and overflow in a debug build). That is an
/// iceberg-rust bug affecting the writer's own partition values, independent of these
/// kernels -- apache/iceberg-rust#3141. The wrapping cases are pinned against the JVM in the
/// kernel's own unit tests; add them here once that issue is fixed.
/// `i32::MIN`, `i64::MIN`, and widths above 2^30 are left out because the two sides
/// intentionally disagree there.
///
/// Widths above 2^30 are a bug in Java, not in iceberg-rust: the format spec defines integer
/// truncation as `v - (v % W)` with a positive remainder, and Java's `(v % W) + W` overflows
/// a 32-bit `int`. apache/iceberg#18111 tracks the Java bug and apache/iceberg#18112 the fix.
/// Comet's kernel reproduces Java so the expression path matches every Iceberg release
/// through 1.12.0; `CometIcebergTruncate` leaves these widths to Spark, and native writes
/// fall back when the partition spec has such a field
/// (https://github.com/apache/datafusion-comet/issues/7442).
///
/// `i32::MIN` and `i64::MIN` are the opposite case: the result does not fit the type, the
/// spec is silent, Java wraps, and iceberg-rust panics in a debug build. That is
/// apache/iceberg-rust#3141. The wrapping cases are pinned against the JVM in the kernel's
/// own unit tests; add them here once that issue is fixed.
#[test]
fn truncate_agrees_with_iceberg_rust() {
let inputs: Vec<(&str, ArrayRef)> = vec![
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -918,6 +918,48 @@ object IcebergReflection extends Logging {
}
}

/** Largest integer `truncate` width whose `(v % W) + W` cannot overflow a Java `int`. */
private val MaxSafeIntegerTruncateWidth: Long = 1L << 30

private val TruncateTransform = """^truncate\[(\d+)\]$""".r

/**
* The names of the `truncate[W]` partition fields of `spec` whose source type is an integer (or
* a date) and whose width is above 2^30.
*
* At those widths Iceberg Java's `TruncateUtil` overflows in `(v % W) + W`, while the format
* spec and iceberg-rust compute `v - (v % W)`. A native write would therefore produce a
* different partition value than the Iceberg jar the user runs. Throws on reflection failure,
* so the caller can fail closed.
*/
def truncateFieldsWithLargeIntegerWidth(spec: Any): Seq[String] = {
import scala.jdk.CollectionConverters._
val schema = getMethod(spec.getClass, "schema").invoke(spec)
val findField = getMethod(schema.getClass, "findField", classOf[Int])
getMethod(spec.getClass, "fields")
.invoke(spec)
.asInstanceOf[java.util.List[_]]
.asScala
.toSeq
.filter { field =>
getMethod(field.getClass, "transform").invoke(field).toString match {
case TruncateTransform(width) if width.toLong > MaxSafeIntegerTruncateWidth =>
val sourceId = getMethod(field.getClass, "sourceId").invoke(field).asInstanceOf[Int]
val sourceField = findField.invoke(schema, sourceId.asInstanceOf[Object])
if (sourceField == null) {
false
} else {
// Long, string, binary and decimal truncation does not overflow at these widths.
val sourceType = getMethod(sourceField.getClass, "type").invoke(sourceField)
val typeId = getMethod(sourceType.getClass, "typeId").invoke(sourceType).toString
typeId == "INTEGER" || typeId == "DATE"
}
case _ => false
}
}
.map(field => getMethod(field.getClass, "name").invoke(field).asInstanceOf[String])
}

/**
* Gets the partition spec from an Iceberg table.
*/
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -222,6 +222,22 @@ object CometIcebergTruncate
"Decimal128(precision, scale) array cannot carry that intermediate, so a native kernel " +
"would null it during evaluation and change what an enclosing predicate or hash sees."

/**
* Largest integer `truncate` width whose `(v % W) + W` cannot overflow a Java `int`.
*
* Above it the result depends on the Iceberg jar the user runs: every release through 1.12.0
* wraps, while a release carrying apache/iceberg#18112 returns the spec's `v - (v % W)`. Spark
* decides the expected answer, so those widths stay with Spark.
* https://github.com/apache/datafusion-comet/issues/7442
*/
private val MaxSafeIntegerWidth: Int = 1 << 30

/** Types whose truncation Iceberg evaluates with Java `int` arithmetic. */
private def usesIntArithmetic(dataType: DataType): Boolean = dataType match {
case ByteType | ShortType | IntegerType | DateType => true
case _ => false
}

// Ordered after the parameter check so that `truncate(0, decimal_col)` still reports the width
// problem, which is the one that changes whether Iceberg's own ArithmeticException is raised.
override def getSupportLevel(expr: Expression): SupportLevel =
Expand All @@ -230,14 +246,28 @@ object CometIcebergTruncate
if value.dataType.isInstanceOf[DecimalType] &&
CometIcebergSystemFunctions.positiveIntLiteral(parameter).isDefined =>
Unsupported(Some(DecimalNote))
case Some((Seq(parameter, value), _))
if usesIntArithmetic(value.dataType) &&
CometIcebergSystemFunctions
.positiveIntLiteral(parameter)
.exists(_ > MaxSafeIntegerWidth) =>
Unsupported(Some(
s"truncate width above $MaxSafeIntegerWidth overflows Iceberg's int arithmetic, and " +
"the result depends on the Iceberg jar the user runs"))
case _ => super.getSupportLevel(expr)
}

override def getUnsupportedReasons(): Seq[String] = Seq(
"Iceberg's `truncate(width, value)` system function on a `decimal` column. " + DecimalNote +
" Truncating `-99999999999999.9999` in a `decimal(18,4)` column by a width of 10 is one " +
"such value: the result has 19 digits. The other `truncate` input types, and `bucket` on " +
"decimals, are unaffected.")
"decimals, are unaffected.",
"Iceberg's `truncate(width, value)` system function on an `int`, `date`, `tinyint` or " +
"`smallint` column with a width above 1073741824. At those widths Iceberg Java's " +
"`(v % W) + W` overflows a 32-bit `int`, while the format spec and iceberg-rust compute " +
"`v - (v % W)`. Every Iceberg release through 1.12.0 overflows, so the expected answer " +
"depends on the Iceberg jar the user runs and Spark evaluates those calls. Long, string " +
"and binary truncation is unaffected.")
}

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -416,6 +416,7 @@ object CometIcebergNativeWrite extends CometOperatorSerde[IcebergWriteExec] {
requireNoMetadataColumns,
requireSupportedColumnTypes,
requireNoVoidFieldWithDroppedSource,
requireNoLargeWidthIntegerTruncate,
requireNoEncryptionPrefix,
requireNoBloomFilterColumnsEnabled,
requireRowGroupCheckMinRecordCountAtDefault,
Expand Down Expand Up @@ -553,6 +554,27 @@ object CometIcebergNativeWrite extends CometOperatorSerde[IcebergWriteExec] {
}
}

// An integer `truncate[W]` with `W > 2^30` overflows in Iceberg Java's `(v % W) + W`, while
// iceberg-rust computes the spec's `v - (v % W)`. Native writes would therefore write a
// different partition value than the user's Iceberg jar, which decides the Spark result.
// https://github.com/apache/datafusion-comet/issues/7442
private val requireNoLargeWidthIntegerTruncate: TriggerRule = ctx =>
IcebergReflection
.getOutputSpecIdFromSparkWrite(ctx.sparkWrite)
.flatMap(IcebergReflection.getPartitionSpecById(ctx.table, _)) match {
case None => Some("could not resolve the output partition spec for truncate width checking")
case Some(spec) =>
try {
IcebergReflection.truncateFieldsWithLargeIntegerWidth(spec).headOption.map { name =>
s"partition field $name uses an integer truncate width above 2^30, where " +
"iceberg-java and iceberg-rust disagree"
}
} catch {
case e: Exception =>
Some(s"could not inspect the output partition spec: ${e.getMessage}")
}
}

private val requireNoEncryptionPrefix: TriggerRule = ctx =>
ctx.properties.keys
.find(_.startsWith(EncryptionPropertyPrefix))
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -129,12 +129,30 @@ class CometIcebergSystemFunctionSuite
withSourceTable {
truncateColumns.foreach { column =>
val truncated =
Seq(1, 3, 10, 1000, Int.MaxValue).map(w => s"$catalog.system.truncate($w, $column)")
// Widths above 2^30 stay with Spark; the fallback has its own test below.
Seq(1, 3, 10, 1000, 1 << 30).map(w => s"$catalog.system.truncate($w, $column)")
checkSparkAnswerAndOperator(s"SELECT $column, ${truncated.mkString(", ")} FROM $source")
}
}
}

test("integer truncate widths above 2^30 fall back to Spark") {
withSourceTable {
// Iceberg Java's `(v % W) + W` overflows a 32-bit `int` above this width, while the format
// spec and iceberg-rust compute `v - (v % W)`. Every Iceberg release through 1.12.0
// overflows and a release carrying apache/iceberg#18112 does not, so Spark's expected
// answer depends on the user's Iceberg jar. Comet leaves the expression to Spark.
// https://github.com/apache/datafusion-comet/issues/7442
checkSparkAnswerAndFallbackReason(
s"SELECT $catalog.system.truncate(2000000000, i32) FROM $source",
"truncate width above")
// Long, string and binary truncation does not overflow at any int width.
checkSparkAnswerAndOperator(
s"SELECT $catalog.system.truncate(${Int.MaxValue}, i64), " +
s"$catalog.system.truncate(${Int.MaxValue}, str) FROM $source")
}
}

test("truncate on a decimal falls back to Spark") {
withSourceTable {
// Iceberg's `TruncateDecimal` can return a value wider than the column's precision, which
Expand Down Expand Up @@ -341,6 +359,19 @@ class CometIcebergSystemFunctionSuite
assert(level(CometIcebergTruncate, truncateInt, Literal(10), value) == Compatible())
assert(level(CometIcebergTruncate, truncateInt, Literal(10), date).isInstanceOf[Unsupported])

// Widths above 2^30 overflow Iceberg Java's `int` arithmetic; widths up to it are safe.
assert(level(CometIcebergTruncate, truncateInt, Literal(1 << 30), value) == Compatible())
assert(
level(CometIcebergTruncate, truncateInt, Literal((1 << 30) + 1), value)
.isInstanceOf[Unsupported])
// Long truncation uses long arithmetic, so the same width stays native.
val truncateLong =
Class.forName("org.apache.iceberg.spark.functions.TruncateFunction$TruncateBigInt")
val longValue = AttributeReference("l", LongType)()
assert(
level(CometIcebergTruncate, truncateLong, Literal((1 << 30) + 1), longValue) ==
Compatible())

// A decimal value reports its own reason, but only once the width is valid: a zero width has
// to keep reporting the width, since that is what decides whether Iceberg's own
// ArithmeticException is raised.
Expand All @@ -365,6 +396,9 @@ class CometIcebergSystemFunctionSuite
.getUnsupportedReasons()
.exists(_.contains(CometIcebergTruncate.DecimalNote)),
"the decimal truncate note is missing from CometStaticInvoke.getUnsupportedReasons")
assert(
CometStaticInvoke.getUnsupportedReasons().exists(_.contains("width above 1073741824")),
"the wide integer truncate note is missing from CometStaticInvoke.getUnsupportedReasons")

// CometStaticInvoke dispatches on (functionName, class name), so the same expressions reach
// the Iceberg handlers from there too.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1499,6 +1499,28 @@ class CometIcebergWriteDetectionSuite extends CometTestBase with CometIcebergTes
}
}

test("fall-back: integer truncate partition width above 2^30") {
// Above this width Iceberg Java's `(v % W) + W` overflows an `int`, while the format spec
// and iceberg-rust compute `v - (v % W)`. The user's Iceberg jar decides the value that
// belongs in the table, so the native writer cannot pick one.
// https://github.com/apache/datafusion-comet/issues/7442
withDetectionCatalog { _ =>
createTable(
"part_truncate_wide",
partitionSpec = "PARTITIONED BY (truncate(2000000000, id))")
assertUnsupportedContains("part_truncate_wide", "truncate width above 2^30")
}
}

test("Compatible when partitioned by an integer truncate at the maximum safe width") {
withDetectionCatalog { _ =>
createTable(
"part_truncate_max",
partitionSpec = "PARTITIONED BY (truncate(1073741824, id))")
assertSupportLevelIs[Compatible]("part_truncate_max")
}
}

test("Compatible when partitioned by identity on a string column") {
withDetectionCatalog { _ =>
createTable("part_string", partitionSpec = "PARTITIONED BY (region)")
Expand Down