Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions docs/source/user-guide/latest/compatibility/operators.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,26 @@ TopK, and ranking window order keys run natively under both default and nondefau
searches boundaries over a nullable nested key still runs on Spark
([#6477](https://github.com/apache/datafusion-comet/issues/6477)).

Spark's TopK (`TakeOrderedAndProject`, an `ORDER BY` with a `LIMIT`) evaluates a sort key after
the first only for the rows whose earlier keys tie. Comet evaluates every key over the whole batch,
so under `spark.sql.ansi.enabled=true` a later key such as `b + 1` could raise
`ARITHMETIC_OVERFLOW` on a row that Spark never evaluates it for. Under ANSI a TopK therefore runs
on Spark when Comet cannot show that a later sort key never fails
([#7246](https://github.com/apache/datafusion-comet/issues/7246)). With
`spark.sql.ansi.enabled=false`, where most invalid inputs give NULL instead of an error, a TopK
runs on Spark only when a later key uses an expression that Spark can raise for in any mode, such
as `element_at` on an array with an index that may be 0, `slice` with a start that may be 0 or a
length that may be negative, `array_insert` with a position that may be 0, `sequence` with bounds
that are not constants, `rlike` with a pattern that is not a constant, or a user-defined function.
It also runs on Spark when a later key still evaluates in ANSI mode, such as `b + 1` in the
ordering of a view created with ANSI on. In either mode,
`spark.comet.operator.TakeOrderedAndProjectExec.allowIncompatible=true` runs such a TopK in
Comet. With ANSI off, a later key that casts to or from a timestamp runs in Comet, and like any
other expression it can raise for a year beyond what Comet's timestamp casts support
([#7471](https://github.com/apache/datafusion-comet/issues/7471)). A sort without a `LIMIT` that
has no exchange beneath it, such as `SORT BY`, still runs in Comet and can raise such an error
([#7470](https://github.com/apache/datafusion-comet/issues/7470)).

## Hash Aggregation

Ungrouped `AVG` and `TRY_AVG` on `DECIMAL(p, s)` fall back to Spark when the intermediate sum
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ import scala.util.Try
import org.apache.spark.SPARK_VERSION
import org.apache.spark.sql.catalyst.expressions.{Attribute, Cast, Expression, Literal}
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types.{ArrayType, DataType, DataTypes, DecimalType, MapType, NullType, NumericType, StructType, TimestampNTZType, TimestampType}
import org.apache.spark.sql.types.{ArrayType, DataType, DataTypes, DateType, DecimalType, MapType, NullType, NumericType, StringType, StructType, TimestampNTZType, TimestampType}

import org.apache.comet.CometSparkSessionExtensions.{isSpark40Plus, withFallbackReason}
import org.apache.comet.DataTypeSupport.{containsDuplicateFieldNames, isComplexType}
Expand Down Expand Up @@ -579,6 +579,49 @@ object CometCast
private[comet] def canProduceNullMapKey(cast: Cast): Boolean =
evalMode(cast) == CometEvalMode.TRY && mayNullMapKey(cast.child.dataType, cast.dataType)

/**
* Whether `cast` cannot raise an error for any value. A TRY cast returns NULL for a value it
* cannot convert, and so does a LEGACY cast between the atomic types listed here. An ANSI cast
* cannot fail only where the conversion is always exact, such as a widening one or a string to
* its bytes. A cast to or from a timestamp is excluded in every mode: the Rust kernels go
* through chrono, which rejects years Spark accepts, and raise even for TRY. So is a cast from
* a date to a string, whose Rust formatting raises in every mode for such a year. Both are
* checked at any depth, since a nested cast applies the same kernels to each element, field,
* key and value (https://github.com/apache/datafusion-comet/issues/7471).
*/
private[comet] def cannotFail(cast: Cast): Boolean = {
val from = cast.child.dataType
val to = cast.dataType
!failsInAnyMode(cast) && (evalMode(cast) match {
case CometEvalMode.TRY => true

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

try_cast(ts_array AS STRING) bypasses this timestamp check. With Long.MaxValue microseconds, the native cast errors while Spark formats it. Check nested timestamp types too, with a TopK regression.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, good catch. Fixed in 5a5b423. CometCast.cannotFail now looks inside arrays, structs and maps: a timestamp anywhere in the source or target type makes the cast able to fail, and so does a date in the source combined with a string in the target. A nested cast applies the same chrono-based kernels to each element, field, key and value, so they raise under TRY just like the top-level cast ( #7471 ).

topk_later_key_ansi.sql adds a topk_nested table with timestamp_micros(9000000000000000000) and date_from_unix_date(100000000), and checks that these later keys fall back: try_cast of array<timestamp> and array<date> to STRING and to array<string>, struct<timestamp> and struct<date> to STRING, struct<date> to struct<string> and struct<timestamp>, struct<timestamp> to struct<date>, map<int, date> to map<int, string>, map<timestamp, int> to map<string, int>, and array<struct<timestamp>> to STRING.

case CometEvalMode.LEGACY if isAtomicCastType(from) && isAtomicCastType(to) => true
case _ if from.isInstanceOf[StringType] && to == DataTypes.BinaryType => true
case _ => keyCastCannotFail(from, to)
})
}

/**
* Whether `cast` can raise whatever its eval mode: a cast to or from a timestamp, or from a
* date to a string, at any depth, as `cannotFail` describes.
*/
private def failsInAnyMode(cast: Cast): Boolean = {
val from = cast.child.dataType
val to = cast.dataType
val hasTimestamp = Seq(from, to).exists(
SupportLevel.containsType(_, classOf[TimestampType], classOf[TimestampNTZType]))
val dateToString = SupportLevel.containsType(from, classOf[DateType]) &&
SupportLevel.containsType(to, classOf[StringType])
hasTimestamp || dateToString
}

// Timestamps are left out because `cannotFail` rejects them before reaching this check.
private def isAtomicCastType(dataType: DataType): Boolean = dataType match {
case _: NumericType | _: StringType | DataTypes.BooleanType | DataTypes.BinaryType |
DataTypes.DateType =>
true
case _ => false
}

private def mayNullMapKey(fromType: DataType, toType: DataType): Boolean =
(fromType, toType) match {
case (MapType(fromKey, fromValue, _), MapType(toKey, toValue, _)) =>
Expand Down
135 changes: 135 additions & 0 deletions spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.catalyst.expressions.aggregate._
import org.apache.spark.sql.catalyst.expressions.objects.{Invoke, StaticInvoke}
import org.apache.spark.sql.catalyst.expressions.xml.{XPathBoolean, XPathDouble, XPathFloat, XPathInt, XPathList, XPathLong, XPathShort, XPathString}
import org.apache.spark.sql.catalyst.optimizer.NormalizeNaNAndZero
import org.apache.spark.sql.comet.DecimalPrecision
import org.apache.spark.sql.execution.{ScalarSubquery, SparkPlan}
import org.apache.spark.sql.execution.datasources.parquet.ParquetUtils
Expand Down Expand Up @@ -1394,6 +1395,140 @@ object QueryPlanSerde extends Logging with CometExprShim with CometTypeShim {
}
}

/**
* Whether `expr` cannot raise an error for any row, so Comet may evaluate it for rows that
* Spark skips. It answers from Spark's semantics, independently of the Rust function
* `cannot_fail` in `case_when.rs`, which judges the physical expressions it is given. It is
* conservative: an allowed expression whose children also cannot fail. Anything not listed is
* treated as able to fail.
*/
Comment on lines +1398 to +1404

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment says this works like the native cannot_fail, but the two sets differ. The native one in conditional_funcs/case_when.rs (line 449) admits no functions and no integer or decimal Divide, while this one admits Substring, Upper, Lower, Length, Coalesce, all non-ANSI arithmetic and every LEGACY atomic cast. The difference may be right, but the wording should say they are independent. Also, arrays.scala already has a private cannotThrow (line 86) answering the same question for array_append. Could it call this helper, so there is one Scala notion of an expression that cannot fail?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. I reworded the comment: the list now states that it answers from Spark semantics alone and is independent of the native cannot_fail in case_when.rs. I left cannotThrow in arrays.scala as is, because switching it to this list would change when array_append runs natively; I'd rather do that in a separate PR with its own tests.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Following up on the cannotThrow part: opened #7585 to have arrays.scala (and NativeLambdaSerde.mayRaise from #7322 ) reuse QueryPlanSerde.cannotFail. It changes when array_append runs natively, so it needs its own tests and stays out of this PR.

def cannotFail(expr: Expression): Boolean = {
def numeric: Boolean = expr.dataType.isInstanceOf[NumericType]
def nonAnsiNumeric(evalMode: EvalMode.Value): Boolean = evalMode != EvalMode.ANSI && numeric
// Floating-point arithmetic never overflows into an error, even in ANSI mode
def floating: Boolean = expr.dataType == FloatType || expr.dataType == DoubleType
val nodeCannotFail = expr match {
case _: Attribute | _: BoundReference | _: Literal => true
case _: GetStructField => true
case _: BinaryComparison | _: And | _: Or | _: Not | _: IsNull | _: IsNotNull => true
case _: In | _: InSet | _: Greatest | _: Least => true
case _: If | _: CaseWhen | _: Coalesce | _: IsNaN | _: NaNvl => true
case _: KnownFloatingPointNormalized | _: NormalizeNaNAndZero => true
case _: Substring | _: Upper | _: Lower | _: Length | _: OctetLength | _: BitLength => true
case _: StringTrim | _: StringTrimLeft | _: StringTrimRight => true
case _: StringReplace | _: InitCap => true
// These fail only when the result would exceed 2GB, more than a row can hold
case _: ConcatWs => true
case c: Concat => c.children.nonEmpty && c.dataType == StringType
case _: Reverse => expr.dataType == StringType
case _: Year | _: Quarter | _: Month | _: DayOfMonth | _: DayOfWeek | _: WeekDay |
_: DayOfYear | _: Hour | _: Minute | _: Second =>
true
case _: Murmur3Hash | _: XxHash64 | _: Md5 | _: Sha1 | _: Sha2 | _: Crc32 => true
// Invalid input gives NaN or NULL, and a floating-point floor or ceil saturates to a long
case _: Floor | _: Ceil | _: Sqrt | _: Exp | _: Log | _: Signum => true
// Rounding an integer fails on overflow in ANSI mode; a decimal is never treated as safe
case r: Round => roundCannotFail(r.dataType, r.ansiEnabled)
case r: BRound => roundCannotFail(r.dataType, r.ansiEnabled)
case c: Cast => CometCast.cannotFail(c)
case _: UnaryPositive => true
case u: UnaryMinus => floating || (!u.failOnError && numeric)
case a: Abs => floating || (!a.failOnError && numeric)
case a: Add => floating || nonAnsiNumeric(a.evalMode)
case s: Subtract => floating || nonAnsiNumeric(s.evalMode)
case m: Multiply => floating || nonAnsiNumeric(m.evalMode)
case d: Divide => nonAnsiNumeric(d.evalMode)
case d: IntegralDivide => nonAnsiNumeric(d.evalMode)
case r: Remainder => nonAnsiNumeric(r.evalMode)
case p: Pmod => nonAnsiNumeric(p.evalMode)
case _ => false
Comment on lines +1411 to +1444

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This set decides the fallback for every top-K with more than one key, with ANSI on or off. Later keys such as year(d), trim(s), round(x, 2) or concat fall back even though they cannot fail, and substring, upper, lower and length came in to fix the q79 plan change. Is that narrow set intended? If not, the common infallible functions could be added with tests.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Widened the list after reading the Spark source for 3.4.3, 3.5.8, 4.0.1 and 4.1.1. Added:

  • String: octet_length, bit_length, trim/ltrim/rtrim, replace, initcap, concat_ws, concat (string, at least one argument), reverse (string)
  • Datetime: year, quarter, month, day, dayofweek, weekday, dayofyear, hour, minute, second
  • Math: floor, ceil, sqrt, exp, log, signum, greatest, least, isnan, nanvl, IN; round/bround for float/double always, and for integral types only when ANSI is off
  • Hash: hash, xxhash64, md5, sha1, sha2, crc32
  • Cast: string to binary (inserted implicitly by md5(s) and friends)

I also checked each one's native implementation (the Comet kernels plus DataFusion / datafusion-spark 55.2) for errors that depend on the data. The only real ones were timestamp casts: they go through chrono, which rejects years Spark accepts, and raise even for TRY (filed as #7471 ). So any cast to or from TIMESTAMP/TIMESTAMP_NTZ now counts as able to fail. The rest error only at planning time or beyond 2GB of string offsets per batch.

Left out on purpose: timestamp casts (above), decimal round (the negative-scale path builds a Decimal that can fail the precision check), integral round under ANSI (overflow), lpad/rpad/repeat (size limits), element_at, date_add, to_date, and anything that parses input.

On upkeep: the list fails safe. A function missing from it only falls back to Spark, so leaving one out costs performance, never correctness. Each entry was checked against Spark 3.4.3 to 4.1.1 and the native kernel, and topk_later_key_ansi.sql runs each one natively over NULL and boundary values.

}
nodeCannotFail && expr.children.forall(cannotFail)
}

private def roundCannotFail(dataType: DataType, ansiEnabled: Boolean): Boolean =
dataType match {
case FloatType | DoubleType => true
case ByteType | ShortType | IntegerType | LongType => !ansiEnabled
case _ => false
}

/**
* The first sort key after the first that can fail. Spark's ordering evaluates a sort key only
* for the rows whose earlier keys tie, but Comet evaluates every key over the whole batch, so
* such a key may raise on a row that Spark skips
* (https://github.com/apache/datafusion-comet/issues/7246). Under ANSI any key that
* `cannotFail` does not accept counts. Otherwise a key counts when `raisesInAnyMode` flags it,
* or when it holds a node still in ANSI mode that `cannotFail` does not accept, such as `b + 1`
* in the ordering of a view created with ANSI on.
*/
def laterSortKeyThatCanFail(sortOrder: Seq[SortOrder]): Option[SortOrder] = {
val canFail: Expression => Boolean =
if (SQLConf.get.ansiEnabled) e => !cannotFail(e)
else e => raisesInAnyMode(e) || e.exists(n => hasAnsiEvalMode(n) && !cannotFail(n))
sortOrder.drop(1).find(s => canFail(s.child))
}

/**
* Whether `expr` was resolved in ANSI mode, whatever the session's setting is now. A view keeps
* the configs it was created with, so its expressions keep their ANSI eval mode when it is
* queried with ANSI off. Like `raisesInAnyMode`, it is best-effort: it covers arithmetic, casts
* and the expressions listed here (https://github.com/apache/datafusion-comet/issues/7585).
*/
private def hasAnsiEvalMode(expr: Expression): Boolean = expr match {
case u: UnaryMinus => u.failOnError
case a: Abs => a.failOnError
case r: Round => r.ansiEnabled
case r: BRound => r.ansiEnabled
case e: ElementAt => e.failOnError
case g: GetArrayItem => g.failOnError
case e: Elt => e.failOnError
// Spark 4.1 keeps the arithmetic eval mode inside a `NumericEvalContext`, so read it by name
case a: Add => a.evalMode == EvalMode.ANSI
case s: Subtract => s.evalMode == EvalMode.ANSI
case m: Multiply => m.evalMode == EvalMode.ANSI
case d: Divide => d.evalMode == EvalMode.ANSI
case d: IntegralDivide => d.evalMode == EvalMode.ANSI
case r: Remainder => r.evalMode == EvalMode.ANSI
case p: Pmod => p.evalMode == EvalMode.ANSI
case c: Cast => c.evalMode == EvalMode.ANSI
case _ => expr.productIterator.contains(EvalMode.ANSI)
}

/**
* With ANSI off, whether `expr` can still raise on a row that Spark's ordering skips, because
* Spark raises for it in every mode. Comet 1.2 ran every TopK in Comet, so this only adds
* safety over that release even though the list is best-effort
* (https://github.com/apache/datafusion-comet/issues/7585). A Rust kernel that raises where
* Spark does not, such as a timestamp cast of a year beyond chrono's range, raises in a
* projection too and is left to its own issue
* (https://github.com/apache/datafusion-comet/issues/7471).
*/
private def raisesInAnyMode(expr: Expression): Boolean = {
def constantWhere(e: Expression)(ok: Int => Boolean): Boolean =
e.foldable && (e.eval() match {
case null => true
case v: Int => ok(v)
case _ => false
})
expr.find {
// assert_true is replaced by If(_, null, RaiseError) before planning
case _: RaiseError | _: UserDefinedExpression => true
// element_at(array, 0) raises INVALID_INDEX_OF_ZERO in every mode
case e: ElementAt =>
e.left.dataType.isInstanceOf[ArrayType] && !(e.right.foldable && e.right.eval() != 0)
// an invalid pattern raises in every mode
case r: RLike => !r.right.foldable
// slice raises for a start of 0 or a negative length in every mode
case s: Slice => !(constantWhere(s.start)(_ != 0) && constantWhere(s.length)(_ >= 0))
// array_insert raises INVALID_INDEX_OF_ZERO in every mode
case a: ArrayInsert => !constantWhere(a.posExpr)(_ != 0)
// sequence raises for bounds that do not match the step in every mode
case s: Sequence => !s.children.forall(_.foldable)
case _ => false
}.isDefined
}

/**
* Has native execution evaluate the arguments of `scalarFunc`, the serialized `expr`, the way
* Spark evaluates those of a null-intolerant `BinaryExpression` or `TernaryExpression`: each
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,8 @@ import org.apache.spark.sql.vectorized.ColumnarBatch

import org.apache.comet.{CometConf, ConfigEntry}
import org.apache.comet.CometSparkSessionExtensions.isCometShuffleEnabled
import org.apache.comet.serde.{Compatible, OperatorOuterClass, SupportLevel, Unsupported}
import org.apache.comet.serde.QueryPlanSerde.{exprToProto, supportedSortType, withSharedSubexpressions}
import org.apache.comet.serde.{Compatible, Incompatible, OperatorOuterClass, SupportLevel, Unsupported}
import org.apache.comet.serde.QueryPlanSerde.{exprToProto, laterSortKeyThatCanFail, supportedSortType, withSharedSubexpressions}
import org.apache.comet.serde.operator.CometSink

object CometTakeOrderedAndProjectExec extends CometSink[TakeOrderedAndProjectExec] {
Expand Down Expand Up @@ -63,6 +63,10 @@ object CometTakeOrderedAndProjectExec extends CometSink[TakeOrderedAndProjectExe
if (!supportedSortType(op, op.sortOrder)) {
return Unsupported(Some("unsupported data type in sort order"))
}
laterSortKeyThatCanFail(op.sortOrder).foreach { s =>
return Incompatible(
Some(s"sort key ${s.child.sql} can fail on rows that Spark's ordering skips"))
}
Compatible()
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -108,11 +108,19 @@ SELECT array_contains(ai, CAST(s AS INT))
FROM (SELECT * FROM array_null_short_circuit ORDER BY i LIMIT 10)

-- A TakeOrderedAndProject compares rows one sort key at a time and evaluates its projection apart
-- from the keys, so a cast in two sort keys, or in a sort key and the projection, is skipped too
-- from the keys, so a cast in two sort keys, or in a sort key and the projection, is skipped too.
-- A later sort key that can fail runs the TopK on Spark
-- (https://github.com/apache/datafusion-comet/issues/7246), so allow it here.
statement
SET spark.comet.operator.TakeOrderedAndProjectExec.allowIncompatible=true

query expect_native(array_contains, array_position)
SELECT i FROM (SELECT * FROM array_null_short_circuit
ORDER BY array_contains(ai, CAST(s AS INT)), array_position(ai, CAST(s AS INT)), i LIMIT 10)

statement
RESET spark.comet.operator.TakeOrderedAndProjectExec.allowIncompatible

query expect_native(array_contains, array_position)
SELECT array_contains(ai, CAST(s AS INT))
FROM (SELECT * FROM array_null_short_circuit ORDER BY array_position(ai, CAST(s AS INT)), i LIMIT 10)
Expand Down
Loading