Repository navigation
Conversation
Spark's TakeOrderedAndProject ordering evaluates a sort key after the first only for rows whose earlier keys tie, while Comet's native TopK evaluates every key over the whole batch. Under ANSI mode a later key such as `b + 1` could raise ARITHMETIC_OVERFLOW on a row Spark never evaluates it for. Mark such a TopK Incompatible so it runs on Spark. Part of apache#7246 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3ec1c07f-32b3-4cb1-a6ad-09293da13ab9
Whitelist Substring, Upper, Lower and Length, which never raise, so a TopK such as TPC-DS q79 stays native. Allow the incompatible TopK in array_null_short_circuit.sql, whose later key holds an ANSI cast that can fail, to keep testing that TopK keys skip a shared cast. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3ec1c07f-32b3-4cb1-a6ad-09293da13ab9
|
Fork CI is green ( https://github.com/tlfua/datafusion-comet/actions/runs/41611703653 ). Since this changes the serde/planner support level for TopK, could a maintainer please add the |
comphead
left a comment
There was a problem hiding this comment.
Thanks for the clear write-up of how LazilyGeneratedOrdering evaluates later keys. The first-key and later-key description matches TakeOrderedAndProjectExec and GenerateOrdering.genComparisons at v3.4.3, v3.5.8, v4.0.1 and v4.1.1. I have not run the fixture. The fork CI covers Spark 4.1 only, so the Spark 3.5 and 4.0 plans and the Spark SQL suite still need the run-* labels.
| // Spark's ordering evaluates a sort key only for the rows whose earlier keys tie, but native | ||
| // execution evaluates every key over the whole batch, so a later key that can fail may raise | ||
| // on a row that Spark skips (#7246). | ||
| op.sortOrder.drop(1).find(s => !cannotFail(s.child)).foreach { s => | ||
| return Incompatible( | ||
| Some(s"sort key ${s.child.sql} can fail on rows that Spark's ordering skips")) | ||
| } |
There was a problem hiding this comment.
The description leaves the plain sort for a follow-up. Reading Spark, it has the same difference: SortExec.createSorter builds its comparator with RowOrdering.create, which goes through GenerateOrdering, so only the first key is computed for every row, as the prefix. CometSortExec has no guard, so I expect ORDER BY a, b + 1 without a LIMIT to raise in Comet on the fixture data. I have not run it. Window ordering goes the same way. Since a plain sort is far more common than a top-K, could you file an issue for it and link it here and in the docs? If the guard will be reused there, a helper over Seq[SortOrder] would save copying drop(1).find into each operator.
There was a problem hiding this comment.
Thanks. I checked each plain-sort shape against Spark with ANSI on and filed #7470 .
Global ORDER BY a, b + 1 and window ordering (rank, row_number with PARTITION BY) throw in Spark too, because the range exchange or the window pre-projection evaluates every key. The divergence shows only when nothing beneath the sort evaluates all keys first, for example SORT BY or a sort over a single-partition child such as LIMIT. I extracted the check into QueryPlanSerde.laterSortKeyThatCanFail(Seq[SortOrder]) so the plain sort can reuse it in that follow-up.
| case _: Attribute | _: BoundReference | _: Literal => true | ||
| case _: BinaryComparison | _: And | _: Or | _: Not | _: IsNull | _: IsNotNull => true | ||
| case _: If | _: CaseWhen | _: Coalesce => true | ||
| case _: KnownFloatingPointNormalized | _: NormalizeNaNAndZero => true | ||
| case _: Substring | _: Upper | _: Lower | _: Length => true | ||
| case c: Cast => CometCast.cannotFail(c) | ||
| case _: UnaryPositive => true | ||
| case u: UnaryMinus => !u.failOnError && numeric | ||
| case a: Abs => !a.failOnError && numeric | ||
| case a: Add => nonAnsiNumeric(a.evalMode) | ||
| case s: Subtract => nonAnsiNumeric(s.evalMode) | ||
| case m: Multiply => nonAnsiNumeric(m.evalMode) | ||
| case d: Divide => nonAnsiNumeric(d.evalMode) | ||
| case d: IntegralDivide => nonAnsiNumeric(d.evalMode) | ||
| case r: Remainder => nonAnsiNumeric(r.evalMode) | ||
| case p: Pmod => nonAnsiNumeric(p.evalMode) | ||
| case _ => false |
There was a problem hiding this comment.
This set decides the fallback for every top-K with more than one key, with ANSI on or off. Later keys such as year(d), trim(s), round(x, 2) or concat fall back even though they cannot fail, and substring, upper, lower and length came in to fix the q79 plan change. Is that narrow set intended? If not, the common infallible functions could be added with tests.
There was a problem hiding this comment.
Widened the list after reading the Spark source for 3.4.3, 3.5.8, 4.0.1 and 4.1.1. Added:
- String:
octet_length,bit_length,trim/ltrim/rtrim,replace,initcap,concat_ws,concat(string, at least one argument),reverse(string) - Datetime:
year,quarter,month,day,dayofweek,weekday,dayofyear,hour,minute,second - Math:
floor,ceil,sqrt,exp,log,signum,greatest,least,isnan,nanvl,IN;round/broundfor float/double always, and for integral types only when ANSI is off - Hash:
hash,xxhash64,md5,sha1,sha2,crc32 - Cast: string to binary (inserted implicitly by
md5(s)and friends)
I also checked each one's native implementation (the Comet kernels plus DataFusion / datafusion-spark 55.2) for errors that depend on the data. The only real ones were timestamp casts: they go through chrono, which rejects years Spark accepts, and raise even for TRY (filed as #7471 ). So any cast to or from TIMESTAMP/TIMESTAMP_NTZ now counts as able to fail. The rest error only at planning time or beyond 2GB of string offsets per batch.
Left out on purpose: timestamp casts (above), decimal round (the negative-scale path builds a Decimal that can fail the precision check), integral round under ANSI (overflow), lpad/rpad/repeat (size limits), element_at, date_add, to_date, and anything that parses input.
On upkeep: the list fails safe. A function missing from it only falls back to Spark, so leaving one out costs performance, never correctness. Each entry was checked against Spark 3.4.3 to 4.1.1 and the native kernel, and topk_later_key_ansi.sql runs each one natively over NULL and boundary values.
| /** | ||
| * Whether `expr` cannot raise an error for any row, so native execution may evaluate it for | ||
| * rows that Spark skips. Like the native `cannot_fail`, it is conservative: columns, literals, | ||
| * and comparisons, boolean logic, null checks, conditionals, substrings, case conversion and | ||
| * length, numeric arithmetic, negation and absolute value outside ANSI mode and casts that | ||
| * cannot fail over them. Anything else is treated as able to fail. | ||
| */ |
There was a problem hiding this comment.
The comment says this works like the native cannot_fail, but the two sets differ. The native one in conditional_funcs/case_when.rs (line 449) admits no functions and no integer or decimal Divide, while this one admits Substring, Upper, Lower, Length, Coalesce, all non-ANSI arithmetic and every LEGACY atomic cast. The difference may be right, but the wording should say they are independent. Also, arrays.scala already has a private cannotThrow (line 86) answering the same question for array_append. Could it call this helper, so there is one Scala notion of an expression that cannot fail?
There was a problem hiding this comment.
Good catch. I reworded the comment: the list now states that it answers from Spark semantics alone and is independent of the native cannot_fail in case_when.rs. I left cannotThrow in arrays.scala as is, because switching it to this list would change when array_append runs natively; I'd rather do that in a separate PR with its own tests.
| -- Later keys that cannot fail stay native | ||
| query | ||
| SELECT a, b FROM topk ORDER BY a, b LIMIT 2 | ||
|
|
||
| query | ||
| SELECT a, b FROM topk ORDER BY a, b IS NULL, CAST(b AS BIGINT), try_add(b, 1) LIMIT 2 | ||
|
|
||
| query | ||
| SELECT a, b FROM topk ORDER BY a, substr(CAST(b AS STRING), 1, 3), upper(CAST(b AS STRING)), | ||
| length(lower(CAST(b AS STRING))) LIMIT 2 |
There was a problem hiding this comment.
The native cases here cover only int columns without NULLs. Could you add a fallible later key nested under an allowed parent (coalesce(b + 1, 0), IF(a > 3, b + 1, 0)), which is the only coverage of children.forall(cannotFail)? Also try_cast and a LEGACY cast with ANSI off for the two CometCast.cannotFail branches, later keys over bigint, decimal, double, string, date and timestamp with NULLs, NaN, -0.0 and bound values, and a non-ANSI /, % and DIV by zero.
There was a problem hiding this comment.
Added:
- Fallback: a fallible key nested under an allowed one (
coalesce(b + 1, 0),IF(a > 3, b + 1, 0)), andround(b, -1)under ANSI. - Native:
try_cast, and atopk_typestable with NULL, NaN, -0.0, int/long bounds, empty strings, and dates such as 1970-01-01, 2000-02-29 and 9999-12-31, run through the string, datetime, math and hash keys. - ANSI off, running natively: a string-to-int cast in LEGACY mode,
round(b, -1),b / 0,b % 0,b DIV 0,l + 1andl * 2as later keys.
| Spark's TopK (`TakeOrderedAndProject`, an `ORDER BY` with a `LIMIT`) evaluates a sort key after | ||
| the first only for the rows whose earlier keys tie. Comet evaluates every key over the whole batch, | ||
| so under `spark.sql.ansi.enabled=true` a later key such as `b + 1` could raise | ||
| `ARITHMETIC_OVERFLOW` on a row that Spark never evaluates it for. A TopK whose later sort key can | ||
| fail therefore runs on Spark, unless | ||
| `spark.comet.operator.TakeOrderedAndProjectExec.allowIncompatible=true` | ||
| ([#7246](https://github.com/apache/datafusion-comet/issues/7246)). |
There was a problem hiding this comment.
The check does not read spark.sql.ansi.enabled. With ANSI off, it also falls back for any later key outside the allowed set. Could the text say that a later key Comet cannot show to be infallible runs on Spark in either mode, and that Sort and window ordering still run natively with the same difference, linking the issue?
There was a problem hiding this comment.
Reworded. The section now says the check does not depend on ANSI: Comet falls back whenever it cannot show that a later key never fails. It also notes that a sort without a LIMIT and with no exchange beneath it, such as SORT BY, still runs natively and can raise, and links #7470 .
There was a problem hiding this comment.
Update: following @andygrove's and @coderfender's review, the check now depends on ANSI ( 201edc6 ). Under ANSI it is unchanged; with ANSI off a TopK falls back only when a later key can raise in any mode, so keys like date_add and lpad run natively again. operators.md now describes both modes, and the whitelist above applies only under ANSI.
Extract laterSortKeyThatCanFail helper, add string, datetime, math and hash functions that cannot fail in Spark, treat string-to-binary casts as infallible, extend tests and reword the Sort compatibility notes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3ec1c07f-32b3-4cb1-a6ad-09293da13ab9
Native timestamp cast kernels go through chrono, which rejects years Spark accepts, and raise even for TRY casts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3ec1c07f-32b3-4cb1-a6ad-09293da13ab9
|
@comphead thanks for the review, I've addressed the comments above. Could a maintainer add |
coderfender
left a comment
There was a problem hiding this comment.
Thanks for this. I checked Spark's TakeOrderedAndProjectExec and GenerateOrdering in 3.4, 3.5, 4.0 and 4.1, and the description of later keys is right. A later key is only compared when the earlier ones tie. The check sits on the one entry point, so it covers the local TopK, the multi-partition case and AQE. Reading evalMode per expression rather than spark.sql.ansi.enabled keeps TRY and LEGACY native, which is the right call. The CometCast addition is self-contained too, and it doesn't touch any existing support level.
There are two places where the list is stricter than Spark, and TopK queries that ran natively and correctly in 1.2 now fall back by default. I'd like these fixed before merge.
-
spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala:1432-1436. Under ANSI, Spark's float and double+,-,*, unary minus andabscan't raise.DoubleExactNumericandFloatExactNumericonly overridetoInt,toLongandcompare, and the native checked kernels don't error for floats either. Right nowORDER BY k, price * 1.1over a double falls back on Spark 4.x defaults, even though it ran natively before. Could these five allowFloatTypeandDoubleTyperegardless of eval mode, with a test under ANSI?Divide,RemainderandPmodwould stay excluded, since division by zero raises for doubles too. -
QueryPlanSerde.scala:1405.GetStructFieldisn't in the list, soORDER BY a, s.x LIMIT nnow falls back in both modes. It ran natively in 1.2. Spark'snullSafeEvalthere is just a field read. Could it join the list, with a struct column case intopk_later_key_ansi.sql?
Two more that the PR didn't cause, but that need a response.
-
spark/src/main/scala/org/apache/comet/expressions/CometCast.scala:582. The timestamp exclusion makes sense, butDATEtoSTRINGhas the same chrono limit. It goes through the DataFusion fall-through incast.rsto arrow'svalue_to_string, andValueFormatter::writeignoressafeand returnsFailed to convert ... to temporal for Date32in every mode. One example is #7471'sdate_from_unix_date(100000000). As written,ORDER BY a, CAST(dt AS STRING)and implicit casts likeconcat(s, dt)stay native. Could a date source be excluded for the cast to string as well, and the case added to #7471? -
spark/src/test/resources/sql-tests/operators/topk_later_key_ansi.sql:110. A small correction on "The first key is evaluated for every row in Spark too". Spark'stakeOrderedgoes through Guava'sTopKSelector, which stores the first row without comparing it. So when the TopK's whole input is a single row, Spark evaluates no key at all, andWHERE id = 5 ORDER BY b + 1 LIMIT 10on one overflowing row returns the row in Spark but raises in Comet. That isn't this PR's doing. Could the comment say "for every row when the input has at least two rows", and could you file a tracking issue for the single row case?
The check applies with ANSI off too, so Spark 3.4 and 3.5 plans are the ones that can move. I'll make sure all profiles and the Spark SQL suite run before it goes in the queue.
…-skipped-rows Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3ec1c07f-32b3-4cb1-a6ad-09293da13ab9
…e to string Floating-point add, subtract, multiply, negation and abs cannot fail in any mode, and reading a struct field cannot fail, so a later TopK key built from them stays native. A cast from a date to a string raises natively for years beyond chrono's range in every mode, so it is treated as able to fail. The test comment about the first key now notes it holds for at least two rows (apache#7527). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3ec1c07f-32b3-4cb1-a6ad-09293da13ab9
|
Thanks @coderfender, all four points were valid. Addressed in c181b6e (also merged the latest main):
Fork CI on the new head: all jobs passed ( https://github.com/tlfua/datafusion-comet/actions/runs/42145536518 ). |
| val to = cast.dataType | ||
| def isTimestamp(t: DataType) = t == TimestampType || t == TimestampNTZType | ||
| !isTimestamp(from) && !isTimestamp(to) && (evalMode(cast) match { | ||
| case CometEvalMode.TRY => true |
There was a problem hiding this comment.
try_cast(ts_array AS STRING) bypasses this timestamp check. With Long.MaxValue microseconds, the native cast errors while Spark formats it. Check nested timestamp types too, with a TopK regression.
There was a problem hiding this comment.
Thanks, good catch. Fixed in 5a5b423. CometCast.cannotFail now looks inside arrays, structs and maps: a timestamp anywhere in the source or target type makes the cast able to fail, and so does a date in the source combined with a string in the target. A nested cast applies the same chrono-based kernels to each element, field, key and value, so they raise under TRY just like the top-level cast ( #7471 ).
topk_later_key_ansi.sql adds a topk_nested table with timestamp_micros(9000000000000000000) and date_from_unix_date(100000000), and checks that these later keys fall back: try_cast of array<timestamp> and array<date> to STRING and to array<string>, struct<timestamp> and struct<date> to STRING, struct<date> to struct<string> and struct<timestamp>, struct<timestamp> to struct<date>, map<int, date> to map<int, string>, map<timestamp, int> to map<string, int>, and array<struct<timestamp>> to STRING.
A cast over an array, struct or map applies the same native kernels to each element, field, key and value, so a nested timestamp, or a nested date cast to a string, raises for years Spark accepts, even under TRY. Check them at any depth and add TopK regressions for each nested shape. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3ec1c07f-32b3-4cb1-a6ad-09293da13ab9
5a5b423 to
c6f1dce
Compare
| }) | ||
| } | ||
|
|
||
| private def existsNested(dataType: DataType)(p: DataType => Boolean): Boolean = |
There was a problem hiding this comment.
Could we reuse SupportLevel.containsType here? It walks arrays, maps, and structs the same way, e.g. containsType(from, classOf[TimestampType], classOf[TimestampNTZType]). topk_later_key_ansi.sql still passed locally with existsNested replaced.
There was a problem hiding this comment.
Thanks, done in 37be546 . cannotFail now uses SupportLevel.containsType for both the timestamp check and the date-to-string check, and the private existsNested is gone. Behavior is unchanged, so the existing topk_nested queries in topk_later_key_ansi.sql still cover it.
|
|
||
| private def isAtomicCastType(dataType: DataType): Boolean = dataType match { | ||
| case _: NumericType | _: StringType | DataTypes.BooleanType | DataTypes.BinaryType | | ||
| DataTypes.DateType | DataTypes.TimestampType | _: TimestampNTZType => |
There was a problem hiding this comment.
nit: hasTimestamp already rules out any timestamp, so the TimestampType/TimestampNTZType cases here never affect the result. Could we remove them?
There was a problem hiding this comment.
Good catch, removed in 37be546 . isAtomicCastType now lists only DateType besides the numeric, string, boolean and binary types, with a comment that timestamps never reach it because cannotFail rejects them first.
coderfender
left a comment
There was a problem hiding this comment.
Thanks for working through all of these. The float arithmetic and GetStructField cases now stay native, and the date to string exclusion and the nested check from rich7420's comment look right to me. #7471 and #7527 track the rest, so I'm dismissing my earlier request for changes. Two things I'd still like to see before merge.
-
spark/src/main/scala/org/apache/comet/expressions/CometCast.scala:588-614.SupportLevel.containsTypealready walks array elements, struct fields and map keys and values the same wayexistsNesteddoes, andSupportLevelis imported here. CouldcannotFailuse it and dropexistsNestedand the localisTimestamp? Something likeval hasTimestamp = Seq(from, to).exists(containsType(_, classOf[TimestampType], classOf[TimestampNTZType])) val dateToString = containsType(from, classOf[DateType]) && containsType(to, classOf[StringType])
The result is the same, since the date and timestamp types are case objects with private constructors. It leaves one recursive type walk instead of a third copy. Spark's
DataType.existsRecursivelywould have been the obvious choice, but it isprivate[spark]in every version we support.While you're here,
isAtomicCastTypestill listsTimestampTypeandTimestampNTZType.hasTimestamprules them out before theLEGACYarm runs, so those arms can never match. Could they go? As written the list reads as if aLEGACYtimestamp cast can't fail, which is the opposite of what #7471 found, and anyone who later relaxes the timestamp check would admit them without noticing. -
On comphead's
cannotThrowthread. KeepingcannotThrowas it is makes sense for this PR. Could you file an issue for moving it ontocannotFailand link it there? #7322 adds a third list inNativeLambdaSerde.mayRaise, and #7535 is already about that one drifting, so it would be good to have the consolidation tracked.
|
@tlfua I added |
Replace the private existsNested walker with SupportLevel.containsType, and drop the timestamp arms of isAtomicCastType, which cannotFail never reaches because it rejects any cast with a timestamp at any depth first. No change in behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3ec1c07f-32b3-4cb1-a6ad-09293da13ab9
|
@coderfender thanks for the review and for adding the labels.
Fork CI on the new commit is green: https://github.com/tlfua/datafusion-comet/actions/runs/42730158514 . Could you approve the workflows on this commit so the labelled Spark 3.5 / 4.1 / all-profiles runs start? They are waiting for approval. |
andygrove
left a comment
There was a problem hiding this comment.
The PR backlog is huge and the maintainers don't have the bandwidth to review them all, so here is an automated review. Feel free to ignore it.
Thanks for the careful work here. I read GenerateOrdering and TakeOrderedAndProjectExec and the description of when later keys are evaluated is right. I also went through the new cannotFail whitelist in QueryPlanSerde.scala against the serdes for the allowed expressions. I did not find an allowed expression that can raise on the inputs in topk_later_key_ansi.sql, and the nested containsType handling in CometCast.cannotFail reads correctly.
The one thing I would like the author to weigh is the effect with ANSI off. laterSortKeyThatCanFail never looks at spark.sql.ansi.enabled, and cannotFail ends in case _ => false. So with ANSI off (the default on Spark 3.4 and 3.5), a TopK such as ORDER BY a, date_add(d, 1) LIMIT n or a later key using lpad, array_contains or to_date now returns Incompatible and runs on Spark. In the latest release those queries ran natively, and with ANSI off Spark returns NULL for the invalid cases rather than raising, so I could not find an error that the fallback is protecting against. The docs now say this outright, so it is a deliberate trade, but it means every TopK with a multi-key sort on an unlisted function loses native execution for users who never enabled ANSI. Could the check be narrowed when ANSI is off, for example by only treating an unlisted expression as fallible when it carries an ANSI eval mode or is in a short list of kernels known to raise regardless of mode? If not, could the PR say why the broader fallback is needed outside ANSI, and could the Spark 3.5 plan stability results confirm that no golden plan moves beyond q79?
Compared with the latest release, the only behavior change I found is the new fallback for multi-key TopK, and it is intended for the ANSI case. The other intended change is that spark.comet.operator.TakeOrderedAndProjectExec.allowIncompatible=true restores native execution.
coderfender
left a comment
There was a problem hiding this comment.
Thanks for the follow-up. cannotFail now uses SupportLevel.containsType, the unreachable timestamp arms are gone, and #7585 tracks the cannotThrow consolidation. The Spark 3.5 SQL run is green too. LGTM.
|
I'll wait for response on Andy's ask and perhaps lets keep existing behavior as is and only add guards for ANSI enabled ? |
| def laterSortKeyThatCanFail(sortOrder: Seq[SortOrder]): Option[SortOrder] = | ||
| sortOrder.drop(1).find(s => !cannotFail(s.child)) |
There was a problem hiding this comment.
Andy's point about ANSI off is right, and I missed it when I approved. With ANSI off, which is the default on Spark 3.5, a later key like date_add(d, 1), lpad(s, 5), array_contains(a, x) or to_date(s) now falls back because cannotFail treats anything it doesn't list as able to fail. All of those ran natively in 1.2, and with ANSI off they return NULL rather than raising, so that's a performance regression against the release.
One shape that can't be worse than 1.2 is to keep the allow-list under ANSI, and outside it only fall back for expressions that raise whatever the mode. 1.2 ran every top-K natively, so a list like that only adds safety on top of the release, even if it misses something. Roughly:
| def laterSortKeyThatCanFail(sortOrder: Seq[SortOrder]): Option[SortOrder] = | |
| sortOrder.drop(1).find(s => !cannotFail(s.child)) | |
| def laterSortKeyThatCanFail(sortOrder: Seq[SortOrder]): Option[SortOrder] = { | |
| val canFail: Expression => Boolean = | |
| if (SQLConf.get.ansiEnabled) e => !cannotFail(e) else raisesInAnyMode | |
| sortOrder.drop(1).find(s => canFail(s.child)) | |
| } | |
| /** | |
| * With ANSI off, whether `expr` can still raise on a row that Spark's ordering skips. 1.2 ran | |
| * every top-K natively, so this only adds safety over the release even if it is incomplete. | |
| */ | |
| private def raisesInAnyMode(expr: Expression): Boolean = expr.find { | |
| // assert_true is replaced by If(_, null, RaiseError) before planning | |
| case _: RaiseError | _: UserDefinedExpression => true | |
| // element_at(array, 0) raises INVALID_INDEX_OF_ZERO in every mode | |
| case e: ElementAt => | |
| e.left.dataType.isInstanceOf[ArrayType] && !(e.right.foldable && e.right.eval() != 0) | |
| // an invalid pattern raises in every mode | |
| case r: RLike => !r.right.foldable | |
| // the native timestamp and date to string kernels raise even for TRY (#7471) | |
| case c: Cast => CometCast.failsInAnyMode(c) | |
| case _ => false | |
| }.isDefined |
CometCast.failsInAnyMode doesn't exist yet. It would be the hasTimestamp || dateToString part of cannotFail pulled out, so a LEGACY CAST(arr AS STRING) stays native the way it did in 1.2. Reading the session flag is a simplification, and you may prefer to look at each expression's eval mode as the arithmetic arms do. Could you also add a test with ANSI off where keys like date_add(d, 1) and lpad(s, 5) stay native, and one where element_at(arr, 0) still falls back?
There was a problem hiding this comment.
Thanks, applied as suggested in 201edc6 : laterSortKeyThatCanFail keeps the cannotFail allow-list under ANSI and otherwise only falls back for raisesInAnyMode, with CometCast.failsInAnyMode pulled out of cannotFail. The ANSI-off section of topk_later_key_ansi.sql now has date_add(dt, 1) and lpad(s, 5) staying native and element_at(arr, 0) falling back. I kept the session flag as in your sketch and noted the per-expression eval mode, together with the expressions the list does not cover yet, on #7585 ( #7585 (comment) ).
…-skipped-rows Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3ec1c07f-32b3-4cb1-a6ad-09293da13ab9
…mode With spark.sql.ansi.enabled=false most invalid inputs give NULL instead of an error, so falling back every TopK with an unlisted later key lost native execution that Comet 1.2 had. Keep the allow-list under ANSI, and outside it fall back only for a later key that can raise in any mode: element_at on an array with an index that may be 0, rlike with a non-constant pattern, a user-defined function, raise_error, or a cast to or from a timestamp or from a date to a string (CometCast.failsInAnyMode). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3ec1c07f-32b3-4cb1-a6ad-09293da13ab9
|
@andygrove thanks for the review, and @coderfender for the concrete shape. With ANSI off the check is now narrowed as you both suggested ( 201edc6 ):
On plan stability: the PR does not change any approved plan. The gaps that are left (the list is best-effort, and the ANSI check reads the session flag rather than each expression's eval mode) are noted on #7585 for a follow-up. @coderfender could you approve the workflows for the new push when you have a moment? |
|
FYI for re-queueing: the last merge-queue removal was an infra flake, not a test failure. The The current head 201edc6 (ANSI-off narrowing from the latest review) is green on my fork ( https://github.com/tlfua/datafusion-comet/actions/runs/44025342119 ). Could a committer approve the workflows for it, and re-approve and re-enqueue when convenient? Thanks! |
andygrove
left a comment
There was a problem hiding this comment.
Thanks for narrowing this so quickly. With ANSI off, date_add, lpad and array_contains later keys run in Comet again, and the new ANSI-off section of topk_later_key_ansi.sql is a nice way to pin that down. That covers most of what I raised.
One part of it is still there. The cast arm in raisesInAnyMode (QueryPlanSerde.scala:1482) flags any cast with a timestamp on either side. Spark adds those casts quietly, so with ANSI off a later key like date_add(ts, 1), datediff(x, ts), trunc(ts, 'MM'), date_format(dt, 'yyyy') or concat(s, dt) still falls back where 1.2 ran it in Comet. Spark doesn't raise for these with ANSI off. Only our own kernel does, on out-of-range values (#7471), and it does that in a projection or first sort key too. Could this arm apply only under ANSI and leave #7471 to cover the kernel? If you'd rather keep it, could operators.md say it is a Comet limit rather than an expression that raises in any mode?
The gate now reads SQLConf.get.ansiEnabled (QueryPlanSerde.scala:1464) rather than each expression's eval mode. A view created with ANSI on keeps its Add in ANSI mode when it is queried with ANSI off, so ORDER BY a, b + 1 over it would now run in Comet and could raise on rows Spark skips. The previous commit fell back there. Could the ANSI-off branch also treat a node with an ANSI eval mode that cannotFail rejects as able to fail? That would close the gap in this PR instead of leaving it to #7585.
A few kernels raise in every mode in both Spark and Comet, the same way element_at(arr, 0) does. slice with a start of 0 or a negative length, array_insert with position 0, and sequence with invalid bounds all do. Could they go into raisesInAnyMode too?
On tests, could the ANSI-off section get an expect_fallback case for a timestamp cast and for a user-defined function or raise_error, plus a case showing element_at(arr, 1) stays in Comet? In operators.md, "still runs natively" could be "still runs in Comet" to match the style guide. It would also help to say that spark.comet.operator.TakeOrderedAndProjectExec.allowIncompatible applies with ANSI off too.
I've approved the workflows on 201edc6, so the labelled Spark 3.5 and all-profiles runs should report before this goes back in the queue.
…views, slice, array_insert and sequence - With ANSI off, a cast to or from a timestamp no longer makes a TopK fall back. Spark does not raise there; the Comet kernel failure on extreme years also hits projections and is tracked in apache#7471. - A later key that still carries ANSI eval mode, such as one from a view created with ANSI on, falls back even when the session has ANSI off. - slice with a start of 0 or a negative length, array_insert at position 0, and sequence with non-constant bounds raise in every mode, so they fall back with ANSI off. - Tests for timestamp-cast keys staying in Comet, element_at(arr, 1), the new array functions, a view, and a ScalaUDF later key. - operators.md: use "runs in Comet" (style guide N1) and say allowIncompatible applies in either mode. Replace "native" wording in comments with "in Comet" or "Rust". Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3ec1c07f-32b3-4cb1-a6ad-09293da13ab9
Spark 4.1 keeps the eval mode of Add and the other arithmetic expressions inside a NumericEvalContext, so scanning the product fields for EvalMode.ANSI missed an ANSI b + 1 from a view. Match the arithmetic and cast classes and read their evalMode instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3ec1c07f-32b3-4cb1-a6ad-09293da13ab9
|
Thanks @andygrove, all five points are addressed in ade6619 and d4f087e .
Fork CI passed on d4f087e: https://github.com/tlfua/datafusion-comet/actions/runs/44155798081 . One Spark SQL job was rerun after a Maven Central download failure. Could you approve the workflows again when you have a moment? |
andygrove
left a comment
There was a problem hiding this comment.
Thanks for working through the earlier rounds so carefully. I read GenerateOrdering and TakeOrderedAndProjectExec for 3.4.3, 3.5.8, 4.0.1 and 4.1.1, and the description of when later keys are evaluated is right. I also checked the new raisesInAnyMode arms against Spark. element_at(array, 0), array_insert with position 0, slice with a start of 0 or a negative length, and sequence with bad bounds raise in every mode in all four versions. The members that hasAnsiEvalMode reads (failOnError, ansiEnabled and evalMode) exist in 3.4.3 through master, and reading evalMode by name is the right way to handle the 4.1 NumericEvalContext. Arrow's date_part returns NULL for out-of-range dates and timestamps rather than raising, so keeping year and hour on the allow-list is fine.
Compared with branch-1.2, every multi-key TopK ran in Comet there. With ANSI on, a later key outside the allow-list now runs on Spark. That is a deliberate trade and the docs say so. With ANSI off the fallback only adds safety over 1.2.
I have one request, inline on the ANSI-off section of the SQL test.
|
|
||
| -- A constant index other than 0 gives NULL when out of bounds | ||
| query | ||
| SELECT a FROM topk_arrays ORDER BY a, element_at(arr, 1) LIMIT 3 |
There was a problem hiding this comment.
raisesInAnyMode flags rlike with a non-constant pattern, and that shape is reachable. CometRLike.convert sends a non-literal pattern through CometScalaUDF.emitJvmCodegenDispatch, so Spark's doGenCode runs inside the Comet pipeline over every row and an invalid pattern raises on rows Spark skips.
I don't see a case for it in the ANSI-off section. Every other arm of raisesInAnyMode that can reach this check has one. Could you add an expect_fallback query with the pattern in a column, for example ORDER BY a, s RLIKE p LIMIT 3, plus a constant-pattern query that stays in Comet?
There was a problem hiding this comment.
Thanks @andygrove, added in tlfua@1650696 . The ANSI-off section now has a topk_regex table where row 2 holds the invalid pattern '(':
SELECT a FROM topk_regex ORDER BY a, s RLIKE p LIMIT 3is anexpect_fallbackquery. No two rows tie ona, so Spark never compiles'(', while the JVM codegen dispatcher would evaluate it on every row in Comet.SELECT a FROM topk_regex ORDER BY a, s RLIKE '^a' LIMIT 3is a plainquerythat stays in Comet.
Fork CI passed on 1650696: https://github.com/tlfua/datafusion-comet/actions/runs/44472467670 . Some jobs were rerun after Maven Central download failures. The upstream workflows need a committer to approve them again for the new commit.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3ec1c07f-32b3-4cb1-a6ad-09293da13ab9
Which issue does this PR close?
Part of #7246 (the TopK case). The
OR/AND/INcases were fixed by #7326; the arithmetic andconcatcases are handled in a separate PR.Rationale for this change
With
spark.sql.ansi.enabled=true,SELECT a, b FROM topk ORDER BY a, b + 1 LIMIT 2over(1, MaxInt), (2, 5), (3, MaxInt), (4, 7), (5, 9)returns(1, 2147483647), (2, 5)in Spark but fails withARITHMETIC_OVERFLOWin Comet.Spark's
TakeOrderedAndProjectExeccompares rows withLazilyGeneratedOrdering, whose generated comparator (GenerateOrdering.genComparisons) evaluates sort key i for a pair of rows only when keys 0..i-1 tie. Comet's TopK (DataFusionSortExecwithfetch) evaluates every sort key over the whole batch, so a later key that can fail raises on rows Spark never evaluates it for.Making Comet's TopK evaluate later keys lazily was considered and rejected for this PR: which row pairs Spark compares depends on Guava's
Ordering.leastOf, and DataFusion's TopK keeps a heap across batches, so per-batch masking cannot reproduce Spark's evaluation set without a custom operator. A scoped fallback is correct and small, in line with #6006.What changes are included in this PR?
CometTakeOrderedAndProjectExec.getSupportLevelreturnsIncompatiblewhen any sort key after the first can fail. What "can fail" means depends onspark.sql.ansi.enabled(see the next bullets). The first key is evaluated for every row in Spark too when the input has at least two rows, so it is not restricted; the single-row case is tracked in Native TopK evaluates sort keys for a single-row input, raising false ANSI errors #7527 . In either mode,spark.comet.operator.TakeOrderedAndProjectExec.allowIncompatible=truestill runs such a TopK in Comet.QueryPlanSerde.laterSortKeyThatCanFail(sortOrder)returns the first key after the leading one that can fail:cannotFailallow-list below does not accept.raisesInAnyModeholds:raise_erroror a user-defined function,element_aton an array unless the index is a nonzero constant,rlikewith a non-constant pattern,sliceunless the start is a nonzero constant and the length a non-negative constant,array_insertunless the position is a nonzero constant, andsequencewith bounds that are not all constants. Spark raises for each of these in every mode.cannotFailrejects (hasAnsiEvalMode), such asb + 1in the ordering of a view created with ANSI on: a view keeps the configs it was created with.QueryPlanSerde.cannotFail(expr)(the ANSI check): a conservative allow-list judged from Spark semantics (read for 3.4.3, 3.5.8, 4.0.1 and 4.1.1), independent of the Rust functioncannot_failincase_when.rs, with each Rust implementation also checked for data-dependent errors: columns, literals, comparisons,IN, boolean logic, null checks,If/CaseWhen/Coalesce, float normalization, struct field access (s.x), float/double+/-/*, negation andabsin any mode, other numeric arithmetic / negation /absoutside ANSI mode, casts that cannot fail, and:substring,upper,lower,length,octet_length,bit_length,trim/ltrim/rtrim,replace,initcap,concat_ws,concat(string, at least one argument),reverse(string)year,quarter,month,day,dayofweek,weekday,dayofyear,hour,minute,secondfloor,ceil,sqrt,exp,log,signum,greatest,least,isnan,nanvl;round/broundfor float/double, and for integral types only with ANSI off (never decimal)hash,xxhash64,md5,sha1,sha2,crc32Under ANSI anything else, including
lpad/rpad/repeat,element_at,date_addandto_date, is treated as able to fail.arrays.scalastill has its own narrowercannotThrowforarray_append; moving it ontocannotFailis tracked in Consolidate the Scala "cannot fail" checks: arrays.cannotThrow and QueryPlanSerde.cannotFail #7585 .CometCast.cannotFail(cast): TRY casts, LEGACY casts between atomic types, string to binary, and ANSI casts thatkeyCastCannotFailalready proves exact. A cast to or from a timestamp never qualifies: the Rust kernels go through chrono, which rejects years Spark accepts, and raise even for TRY ( Native timestamp casts fail or panic for years beyond chrono's range #7471 ). Nor does a cast from a date to a string: arrow-cast's formatter ignoressafeand raises for such years in every mode. Both are checked at any depth inside arrays, structs and maps withSupportLevel.containsType(privateCometCast.failsInAnyMode), since a nested cast applies the same kernels to each element, field, key and value.docs/source/user-guide/latest/compatibility/operators.mddocuments the fallback under ANSI and the narrower ANSI-off fallback, saysallowIncompatibleapplies in either mode, and notes that a sort without a LIMIT and with no exchange beneath it (such asSORT BY) still runs in Comet.expressions/array/array_null_short_circuit.sql: its two-key TopK query has a later keyarray_position(ai, CAST(s AS INT)), whose ANSI cast can fail, so it now falls back; the query is wrapped withspark.comet.operator.TakeOrderedAndProjectExec.allowIncompatible=trueto keep testing the Rust array kernels.Known trade-off: under ANSI the allow-list is conservative, so a later key built from a function outside it also falls back even though it may not fail. With ANSI off only the keys listed above fall back, so every other TopK runs in Comet as in 1.2. Global
ORDER BYand window ordering are not affected in Spark either (the range exchange or window pre-projection evaluates every key); the plain-sort divergence forSORT BYand single-partition sorts is tracked in #7470 .How are these changes tested?
New SQL file test
spark/src/test/resources/sql-tests/operators/topk_later_key_ansi.sql(ANSI on, AQE off):expect_fallback),query),ARITHMETIC_OVERFLOW(expect_error),substring/upper/lower/length) stay in Comet. Without this, TPC-DS q79 plan stability changed on Spark 4.x,coalesce(b + 1, 0),IF(a > 3, b + 1, 0)) andround(b, -1)under ANSI fall back;try_caststays in Comet, timestamp casts (including TRY) fall back,topk_typestable with NULL, NaN, -0.0, int/long bounds, empty strings and boundary dates runs the string, datetime, math and hash keys in Comet,*,+,-, negation andabsstay in Comet, andCAST(dt AS STRING)falls back,s.xstays in Comet with ANSI on and off,topk_nestedtable with out-of-range values (timestamp_micros(9000000000000000000),date_from_unix_date(100000000)):try_castover nested timestamps and dates (array, struct, map, andarray<struct<timestamp>>) falls back under ANSI. The file writes timestamps asTIMESTAMP_MICROS.b + 1,-b,abs(b), a LEGACY string-to-int cast,round(b, -1),b / 0,b % 0,b DIV 0,l + 1,l * 2,date_add(dt, 1)andlpad(s, 5)stay in Comet,date_add(ts, 1)andconcat(s, ts)stay in Comet,topk_arraystable,element_at(arr, 1),slice(arr, 1, 1)andarray_insert(arr, 1, 0)stay in Comet, whileelement_at(arr, 0),slice(arr, a - 1, 1),array_insert(arr, a - 1, 0)andsequence(a, 1, 1)fall back,SELECT a, b FROM topk ORDER BY a, b + 1 LIMIT 2falls back when queried.CometCodegenSuiteadds "a ScalaUDF later sort key runs a TopK on Spark with ANSI off". A SQL file test cannot cover this withraise_error: Comet has no serde forRaiseError, so that TopK falls back for an unsupported expression before the later-key check runs.Commands run locally (Spark 4.1 default profile, macOS):
./mvnw test -Dsuites="org.apache.comet.CometSqlFileTestSuite topk_later_key" -Dtest=none -Dmaven.gitcommitid.skip=truewithout the fix: failed withARITHMETIC_OVERFLOWonORDER BY a, b + 1 LIMIT 2, as expected.topk_later_key array_null_short_circuitand four other related SQL files after the second commit: passed (6 succeeded, 0 failed).topk_later_key(1 succeeded) andarray_null_short_circuit(5 succeeded): passed.topk_later_key(1 succeeded) andarray_null_short_circuit(5 succeeded): passed.topk_later_key(1 succeeded, 0 failed): passed.slice/array_insert/sequence, docs):make format PROFILES="-Dmaven.gitcommitid.skip=true"(formats and compiles main and test code, Spark 4.1): passed. ThefailOnError/ansiEnabledfields used byhasAnsiEvalModewere checked in the Spark 3.5.8 and 4.0.1 sources. Preflight:JAVA_TOOL_OPTIONS=-Dfile.encoding=ANSI_X3.4-1968 ./dev/release/check-rat.sh HEADpassed ("No unapproved licenses"),prettier --check "**/*.md"and thedev/ciscripts passed; the mermaid check was not run locally. The SQL file and Scala tests for this commit were not run locally; fork CI ran them. Fork CI then failed the view test on Spark 4.1, because Spark 4.1 keeps the arithmetic eval mode inside aNumericEvalContext; the follow-up commit readsevalModeby class,make formatpassed and RAT passed.topk_regexqueries totopk_later_key_ansi.sql, anexpect_fallbackquery with the pattern in a col and a plainquerywith a constant pattern.make format PROFILES="-Dmaven.gitcommitid.skip=true": passed. RAT,prettier --check "**/*.md"and thedev/ciscripts: passed. The SQL file tests were not run locally; fork CI ran them.Fork CI ( https://github.com/tlfua/datafusion-comet/actions/runs/44472467670 ): All jobs passed on 1650696 (some jobs were rerun after Maven Central download failures). Since this touches the serde, the
run-spark-3.5-tests,run-spark-4.1-testsandrun-all-spark-profileslabels run the Spark SQL suites and the other Spark profiles.