Skip to content

fix: fall back TopK whose later sort key can fail under ANSI - #7457

Open
tlfua wants to merge 13 commits into
apache:mainfrom
tlfua:tlfua/fix/7246-topk-skipped-rows
Open

tlfua wants to merge 13 commits into
apache:mainfrom
tlfua:tlfua/fix/7246-topk-skipped-rows

Conversation

@tlfua

@tlfua tlfua commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Part of #7246 (the TopK case). The OR/AND/IN cases were fixed by #7326; the arithmetic and concat cases are handled in a separate PR.

Rationale for this change

With spark.sql.ansi.enabled=true, SELECT a, b FROM topk ORDER BY a, b + 1 LIMIT 2 over (1, MaxInt), (2, 5), (3, MaxInt), (4, 7), (5, 9) returns (1, 2147483647), (2, 5) in Spark but fails with ARITHMETIC_OVERFLOW in Comet.

Spark's TakeOrderedAndProjectExec compares rows with LazilyGeneratedOrdering, whose generated comparator (GenerateOrdering.genComparisons) evaluates sort key i for a pair of rows only when keys 0..i-1 tie. Comet's TopK (DataFusion SortExec with fetch) evaluates every sort key over the whole batch, so a later key that can fail raises on rows Spark never evaluates it for.

Making Comet's TopK evaluate later keys lazily was considered and rejected for this PR: which row pairs Spark compares depends on Guava's Ordering.leastOf, and DataFusion's TopK keeps a heap across batches, so per-batch masking cannot reproduce Spark's evaluation set without a custom operator. A scoped fallback is correct and small, in line with #6006.

What changes are included in this PR?

  • CometTakeOrderedAndProjectExec.getSupportLevel returns Incompatible when any sort key after the first can fail. What "can fail" means depends on spark.sql.ansi.enabled (see the next bullets). The first key is evaluated for every row in Spark too when the input has at least two rows, so it is not restricted; the single-row case is tracked in Native TopK evaluates sort keys for a single-row input, raising false ANSI errors #7527 . In either mode, spark.comet.operator.TakeOrderedAndProjectExec.allowIncompatible=true still runs such a TopK in Comet.

  • QueryPlanSerde.laterSortKeyThatCanFail(sortOrder) returns the first key after the leading one that can fail:

  • QueryPlanSerde.cannotFail(expr) (the ANSI check): a conservative allow-list judged from Spark semantics (read for 3.4.3, 3.5.8, 4.0.1 and 4.1.1), independent of the Rust function cannot_fail in case_when.rs, with each Rust implementation also checked for data-dependent errors: columns, literals, comparisons, IN, boolean logic, null checks, If/CaseWhen/Coalesce, float normalization, struct field access (s.x), float/double +/-/*, negation and abs in any mode, other numeric arithmetic / negation / abs outside ANSI mode, casts that cannot fail, and:

    • string: substring, upper, lower, length, octet_length, bit_length, trim/ltrim/rtrim, replace, initcap, concat_ws, concat (string, at least one argument), reverse (string)
    • datetime: year, quarter, month, day, dayofweek, weekday, dayofyear, hour, minute, second
    • math: floor, ceil, sqrt, exp, log, signum, greatest, least, isnan, nanvl; round/bround for float/double, and for integral types only with ANSI off (never decimal)
    • hash: hash, xxhash64, md5, sha1, sha2, crc32

    Under ANSI anything else, including lpad/rpad/repeat, element_at, date_add and to_date, is treated as able to fail. arrays.scala still has its own narrower cannotThrow for array_append; moving it onto cannotFail is tracked in Consolidate the Scala "cannot fail" checks: arrays.cannotThrow and QueryPlanSerde.cannotFail #7585 .

  • CometCast.cannotFail(cast): TRY casts, LEGACY casts between atomic types, string to binary, and ANSI casts that keyCastCannotFail already proves exact. A cast to or from a timestamp never qualifies: the Rust kernels go through chrono, which rejects years Spark accepts, and raise even for TRY ( Native timestamp casts fail or panic for years beyond chrono's range #7471 ). Nor does a cast from a date to a string: arrow-cast's formatter ignores safe and raises for such years in every mode. Both are checked at any depth inside arrays, structs and maps with SupportLevel.containsType (private CometCast.failsInAnyMode), since a nested cast applies the same kernels to each element, field, key and value.

  • docs/source/user-guide/latest/compatibility/operators.md documents the fallback under ANSI and the narrower ANSI-off fallback, says allowIncompatible applies in either mode, and notes that a sort without a LIMIT and with no exchange beneath it (such as SORT BY) still runs in Comet.

  • expressions/array/array_null_short_circuit.sql: its two-key TopK query has a later key array_position(ai, CAST(s AS INT)), whose ANSI cast can fail, so it now falls back; the query is wrapped with spark.comet.operator.TakeOrderedAndProjectExec.allowIncompatible=true to keep testing the Rust array kernels.

Known trade-off: under ANSI the allow-list is conservative, so a later key built from a function outside it also falls back even though it may not fail. With ANSI off only the keys listed above fall back, so every other TopK runs in Comet as in 1.2. Global ORDER BY and window ordering are not affected in Spark either (the range exchange or window pre-projection evaluates every key); the plain-sort divergence for SORT BY and single-partition sorts is tracked in #7470 .

How are these changes tested?

New SQL file test spark/src/test/resources/sql-tests/operators/topk_later_key_ansi.sql (ANSI on, AQE off):

  • the reproduction from the issue and two more fallible later keys (expect_fallback),
  • later keys that cannot fail, and a fallible first key, stay in Comet (query),
  • rows that tie on the first key still raise ARITHMETIC_OVERFLOW (expect_error),
  • string later keys (substring/upper/lower/length) stay in Comet. Without this, TPC-DS q79 plan stability changed on Spark 4.x,
  • a fallible key nested under an allowed one (coalesce(b + 1, 0), IF(a > 3, b + 1, 0)) and round(b, -1) under ANSI fall back; try_cast stays in Comet, timestamp casts (including TRY) fall back,
  • a topk_types table with NULL, NaN, -0.0, int/long bounds, empty strings and boundary dates runs the string, datetime, math and hash keys in Comet,
  • under ANSI, float/double *, +, -, negation and abs stay in Comet, and CAST(dt AS STRING) falls back,
  • a struct field key s.x stays in Comet with ANSI on and off,
  • a topk_nested table with out-of-range values (timestamp_micros(9000000000000000000), date_from_unix_date(100000000)): try_cast over nested timestamps and dates (array, struct, map, and array<struct<timestamp>>) falls back under ANSI. The file writes timestamps as TIMESTAMP_MICROS.
  • with ANSI off:
    • b + 1, -b, abs(b), a LEGACY string-to-int cast, round(b, -1), b / 0, b % 0, b DIV 0, l + 1, l * 2, date_add(dt, 1) and lpad(s, 5) stay in Comet,
    • timestamp-cast keys date_add(ts, 1) and concat(s, ts) stay in Comet,
    • on a topk_arrays table, element_at(arr, 1), slice(arr, 1, 1) and array_insert(arr, 1, 0) stay in Comet, while element_at(arr, 0), slice(arr, a - 1, 1), array_insert(arr, a - 1, 0) and sequence(a, 1, 1) fall back,
    • a temporary view created with ANSI on as SELECT a, b FROM topk ORDER BY a, b + 1 LIMIT 2 falls back when queried.

CometCodegenSuite adds "a ScalaUDF later sort key runs a TopK on Spark with ANSI off". A SQL file test cannot cover this with raise_error: Comet has no serde for RaiseError, so that TopK falls back for an unsupported expression before the later-key check runs.

Commands run locally (Spark 4.1 default profile, macOS):

  • ./mvnw test -Dsuites="org.apache.comet.CometSqlFileTestSuite topk_later_key" -Dtest=none -Dmaven.gitcommitid.skip=true without the fix: failed with ARITHMETIC_OVERFLOW on ORDER BY a, b + 1 LIMIT 2, as expected.
  • Same command with the fix: passed (1 succeeded, 0 failed).
  • Same command for topk_later_key array_null_short_circuit and four other related SQL files after the second commit: passed (6 succeeded, 0 failed).
  • After the review changes, topk_later_key (1 succeeded) and array_null_short_circuit (5 succeeded): passed.
  • After coderfender's review, merged with the latest main: topk_later_key (1 succeeded) and array_null_short_circuit (5 succeeded): passed.
  • After rich7420's review (nested types): topk_later_key (1 succeeded, 0 failed): passed.
  • After andygrove's latest review (timestamp casts, views, slice/array_insert/sequence, docs): make format PROFILES="-Dmaven.gitcommitid.skip=true" (formats and compiles main and test code, Spark 4.1): passed. The failOnError/ansiEnabled fields used by hasAnsiEvalMode were checked in the Spark 3.5.8 and 4.0.1 sources. Preflight: JAVA_TOOL_OPTIONS=-Dfile.encoding=ANSI_X3.4-1968 ./dev/release/check-rat.sh HEAD passed ("No unapproved licenses"), prettier --check "**/*.md" and the dev/ci scripts passed; the mermaid check was not run locally. The SQL file and Scala tests for this commit were not run locally; fork CI ran them. Fork CI then failed the view test on Spark 4.1, because Spark 4.1 keeps the arithmetic eval mode inside a NumericEvalContext; the follow-up commit reads evalMode by class, make format passed and RAT passed.
  • After andygrove's rlike comment: added topk_regex queries to topk_later_key_ansi.sql, an expect_fallback query with the pattern in a col and a plain query with a constant pattern. make format PROFILES="-Dmaven.gitcommitid.skip=true": passed. RAT, prettier --check "**/*.md" and the dev/ci scripts: passed. The SQL file tests were not run locally; fork CI ran them.

Fork CI ( https://github.com/tlfua/datafusion-comet/actions/runs/44472467670 ): All jobs passed on 1650696 (some jobs were rerun after Maven Central download failures). Since this touches the serde, the run-spark-3.5-tests, run-spark-4.1-tests and run-all-spark-profiles labels run the Spark SQL suites and the other Spark profiles.

Spark's TakeOrderedAndProject ordering evaluates a sort key after the
first only for rows whose earlier keys tie, while Comet's native TopK
evaluates every key over the whole batch. Under ANSI mode a later key
such as `b + 1` could raise ARITHMETIC_OVERFLOW on a row Spark never
evaluates it for. Mark such a TopK Incompatible so it runs on Spark.

Part of apache#7246

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3ec1c07f-32b3-4cb1-a6ad-09293da13ab9
@github-actions github-actions Bot added bug Something isn't working area:expressions Expression evaluation labels Nov 3, 2026
Whitelist Substring, Upper, Lower and Length, which never raise, so a
TopK such as TPC-DS q79 stays native. Allow the incompatible TopK in
array_null_short_circuit.sql, whose later key holds an ANSI cast that
can fail, to keep testing that TopK keys skip a shared cast.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3ec1c07f-32b3-4cb1-a6ad-09293da13ab9
@tlfua
tlfua marked this pull request as ready for review November 3, 2026 10:39
@tlfua

tlfua commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Fork CI is green ( https://github.com/tlfua/datafusion-comet/actions/runs/41611703653 ). Since this changes the serde/planner support level for TopK, could a maintainer please add the run-spark-4.1-tests label so the Spark SQL suite runs? Thanks!

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clear write-up of how LazilyGeneratedOrdering evaluates later keys. The first-key and later-key description matches TakeOrderedAndProjectExec and GenerateOrdering.genComparisons at v3.4.3, v3.5.8, v4.0.1 and v4.1.1. I have not run the fixture. The fork CI covers Spark 4.1 only, so the Spark 3.5 and 4.0 plans and the Spark SQL suite still need the run-* labels.

Comment on lines +66 to +72
// Spark's ordering evaluates a sort key only for the rows whose earlier keys tie, but native
// execution evaluates every key over the whole batch, so a later key that can fail may raise
// on a row that Spark skips (#7246).
op.sortOrder.drop(1).find(s => !cannotFail(s.child)).foreach { s =>
return Incompatible(
Some(s"sort key ${s.child.sql} can fail on rows that Spark's ordering skips"))
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The description leaves the plain sort for a follow-up. Reading Spark, it has the same difference: SortExec.createSorter builds its comparator with RowOrdering.create, which goes through GenerateOrdering, so only the first key is computed for every row, as the prefix. CometSortExec has no guard, so I expect ORDER BY a, b + 1 without a LIMIT to raise in Comet on the fixture data. I have not run it. Window ordering goes the same way. Since a plain sort is far more common than a top-K, could you file an issue for it and link it here and in the docs? If the guard will be reused there, a helper over Seq[SortOrder] would save copying drop(1).find into each operator.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I checked each plain-sort shape against Spark with ANSI on and filed #7470 .

Global ORDER BY a, b + 1 and window ordering (rank, row_number with PARTITION BY) throw in Spark too, because the range exchange or the window pre-projection evaluates every key. The divergence shows only when nothing beneath the sort evaluates all keys first, for example SORT BY or a sort over a single-partition child such as LIMIT. I extracted the check into QueryPlanSerde.laterSortKeyThatCanFail(Seq[SortOrder]) so the plain sort can reuse it in that follow-up.

Comment on lines +1409 to +1425
case _: Attribute | _: BoundReference | _: Literal => true
case _: BinaryComparison | _: And | _: Or | _: Not | _: IsNull | _: IsNotNull => true
case _: If | _: CaseWhen | _: Coalesce => true
case _: KnownFloatingPointNormalized | _: NormalizeNaNAndZero => true
case _: Substring | _: Upper | _: Lower | _: Length => true
case c: Cast => CometCast.cannotFail(c)
case _: UnaryPositive => true
case u: UnaryMinus => !u.failOnError && numeric
case a: Abs => !a.failOnError && numeric
case a: Add => nonAnsiNumeric(a.evalMode)
case s: Subtract => nonAnsiNumeric(s.evalMode)
case m: Multiply => nonAnsiNumeric(m.evalMode)
case d: Divide => nonAnsiNumeric(d.evalMode)
case d: IntegralDivide => nonAnsiNumeric(d.evalMode)
case r: Remainder => nonAnsiNumeric(r.evalMode)
case p: Pmod => nonAnsiNumeric(p.evalMode)
case _ => false

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This set decides the fallback for every top-K with more than one key, with ANSI on or off. Later keys such as year(d), trim(s), round(x, 2) or concat fall back even though they cannot fail, and substring, upper, lower and length came in to fix the q79 plan change. Is that narrow set intended? If not, the common infallible functions could be added with tests.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Widened the list after reading the Spark source for 3.4.3, 3.5.8, 4.0.1 and 4.1.1. Added:

  • String: octet_length, bit_length, trim/ltrim/rtrim, replace, initcap, concat_ws, concat (string, at least one argument), reverse (string)
  • Datetime: year, quarter, month, day, dayofweek, weekday, dayofyear, hour, minute, second
  • Math: floor, ceil, sqrt, exp, log, signum, greatest, least, isnan, nanvl, IN; round/bround for float/double always, and for integral types only when ANSI is off
  • Hash: hash, xxhash64, md5, sha1, sha2, crc32
  • Cast: string to binary (inserted implicitly by md5(s) and friends)

I also checked each one's native implementation (the Comet kernels plus DataFusion / datafusion-spark 55.2) for errors that depend on the data. The only real ones were timestamp casts: they go through chrono, which rejects years Spark accepts, and raise even for TRY (filed as #7471 ). So any cast to or from TIMESTAMP/TIMESTAMP_NTZ now counts as able to fail. The rest error only at planning time or beyond 2GB of string offsets per batch.

Left out on purpose: timestamp casts (above), decimal round (the negative-scale path builds a Decimal that can fail the precision check), integral round under ANSI (overflow), lpad/rpad/repeat (size limits), element_at, date_add, to_date, and anything that parses input.

On upkeep: the list fails safe. A function missing from it only falls back to Spark, so leaving one out costs performance, never correctness. Each entry was checked against Spark 3.4.3 to 4.1.1 and the native kernel, and topk_later_key_ansi.sql runs each one natively over NULL and boundary values.

Comment on lines +1398 to +1404
/**
* Whether `expr` cannot raise an error for any row, so native execution may evaluate it for
* rows that Spark skips. Like the native `cannot_fail`, it is conservative: columns, literals,
* and comparisons, boolean logic, null checks, conditionals, substrings, case conversion and
* length, numeric arithmetic, negation and absolute value outside ANSI mode and casts that
* cannot fail over them. Anything else is treated as able to fail.
*/

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment says this works like the native cannot_fail, but the two sets differ. The native one in conditional_funcs/case_when.rs (line 449) admits no functions and no integer or decimal Divide, while this one admits Substring, Upper, Lower, Length, Coalesce, all non-ANSI arithmetic and every LEGACY atomic cast. The difference may be right, but the wording should say they are independent. Also, arrays.scala already has a private cannotThrow (line 86) answering the same question for array_append. Could it call this helper, so there is one Scala notion of an expression that cannot fail?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. I reworded the comment: the list now states that it answers from Spark semantics alone and is independent of the native cannot_fail in case_when.rs. I left cannotThrow in arrays.scala as is, because switching it to this list would change when array_append runs natively; I'd rather do that in a separate PR with its own tests.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Following up on the cannotThrow part: opened #7585 to have arrays.scala (and NativeLambdaSerde.mayRaise from #7322 ) reuse QueryPlanSerde.cannotFail. It changes when array_append runs natively, so it needs its own tests and stays out of this PR.

Comment on lines +43 to +52
-- Later keys that cannot fail stay native
query
SELECT a, b FROM topk ORDER BY a, b LIMIT 2

query
SELECT a, b FROM topk ORDER BY a, b IS NULL, CAST(b AS BIGINT), try_add(b, 1) LIMIT 2

query
SELECT a, b FROM topk ORDER BY a, substr(CAST(b AS STRING), 1, 3), upper(CAST(b AS STRING)),
length(lower(CAST(b AS STRING))) LIMIT 2

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The native cases here cover only int columns without NULLs. Could you add a fallible later key nested under an allowed parent (coalesce(b + 1, 0), IF(a > 3, b + 1, 0)), which is the only coverage of children.forall(cannotFail)? Also try_cast and a LEGACY cast with ANSI off for the two CometCast.cannotFail branches, later keys over bigint, decimal, double, string, date and timestamp with NULLs, NaN, -0.0 and bound values, and a non-ANSI /, % and DIV by zero.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added:

  • Fallback: a fallible key nested under an allowed one (coalesce(b + 1, 0), IF(a > 3, b + 1, 0)), and round(b, -1) under ANSI.
  • Native: try_cast, and a topk_types table with NULL, NaN, -0.0, int/long bounds, empty strings, and dates such as 1970-01-01, 2000-02-29 and 9999-12-31, run through the string, datetime, math and hash keys.
  • ANSI off, running natively: a string-to-int cast in LEGACY mode, round(b, -1), b / 0, b % 0, b DIV 0, l + 1 and l * 2 as later keys.

Comment on lines +97 to +103
Spark's TopK (`TakeOrderedAndProject`, an `ORDER BY` with a `LIMIT`) evaluates a sort key after
the first only for the rows whose earlier keys tie. Comet evaluates every key over the whole batch,
so under `spark.sql.ansi.enabled=true` a later key such as `b + 1` could raise
`ARITHMETIC_OVERFLOW` on a row that Spark never evaluates it for. A TopK whose later sort key can
fail therefore runs on Spark, unless
`spark.comet.operator.TakeOrderedAndProjectExec.allowIncompatible=true`
([#7246](https://github.com/apache/datafusion-comet/issues/7246)).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The check does not read spark.sql.ansi.enabled. With ANSI off, it also falls back for any later key outside the allowed set. Could the text say that a later key Comet cannot show to be infallible runs on Spark in either mode, and that Sort and window ordering still run natively with the same difference, linking the issue?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reworded. The section now says the check does not depend on ANSI: Comet falls back whenever it cannot show that a later key never fails. It also notes that a sort without a LIMIT and with no exchange beneath it, such as SORT BY, still runs natively and can raise, and links #7470 .

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update: following @andygrove's and @coderfender's review, the check now depends on ANSI ( 201edc6 ). Under ANSI it is unchanged; with ANSI off a TopK falls back only when a later key can raise in any mode, so keys like date_add and lpad run natively again. operators.md now describes both modes, and the whitelist above applies only under ANSI.

tlfua and others added 2 commits November 3, 2026 15:27
Extract laterSortKeyThatCanFail helper, add string, datetime, math and
hash functions that cannot fail in Spark, treat string-to-binary casts as
infallible, extend tests and reword the Sort compatibility notes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3ec1c07f-32b3-4cb1-a6ad-09293da13ab9
Native timestamp cast kernels go through chrono, which rejects years Spark
accepts, and raise even for TRY casts.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3ec1c07f-32b3-4cb1-a6ad-09293da13ab9
@tlfua

tlfua commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

@comphead thanks for the review, I've addressed the comments above. Could a maintainer add run-all-spark-profiles and run-spark-4.1-tests? Spark 3.x defaults to ANSI off, and a later key outside the whitelist now falls back with ANSI off too, so plan-stability goldens on 3.x may change; the fork CI only covers Spark 4.1.

@coderfender coderfender left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. I checked Spark's TakeOrderedAndProjectExec and GenerateOrdering in 3.4, 3.5, 4.0 and 4.1, and the description of later keys is right. A later key is only compared when the earlier ones tie. The check sits on the one entry point, so it covers the local TopK, the multi-partition case and AQE. Reading evalMode per expression rather than spark.sql.ansi.enabled keeps TRY and LEGACY native, which is the right call. The CometCast addition is self-contained too, and it doesn't touch any existing support level.

There are two places where the list is stricter than Spark, and TopK queries that ran natively and correctly in 1.2 now fall back by default. I'd like these fixed before merge.

  1. spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala:1432-1436. Under ANSI, Spark's float and double +, -, *, unary minus and abs can't raise. DoubleExactNumeric and FloatExactNumeric only override toInt, toLong and compare, and the native checked kernels don't error for floats either. Right now ORDER BY k, price * 1.1 over a double falls back on Spark 4.x defaults, even though it ran natively before. Could these five allow FloatType and DoubleType regardless of eval mode, with a test under ANSI? Divide, Remainder and Pmod would stay excluded, since division by zero raises for doubles too.

  2. QueryPlanSerde.scala:1405. GetStructField isn't in the list, so ORDER BY a, s.x LIMIT n now falls back in both modes. It ran natively in 1.2. Spark's nullSafeEval there is just a field read. Could it join the list, with a struct column case in topk_later_key_ansi.sql?

Two more that the PR didn't cause, but that need a response.

  1. spark/src/main/scala/org/apache/comet/expressions/CometCast.scala:582. The timestamp exclusion makes sense, but DATE to STRING has the same chrono limit. It goes through the DataFusion fall-through in cast.rs to arrow's value_to_string, and ValueFormatter::write ignores safe and returns Failed to convert ... to temporal for Date32 in every mode. One example is #7471's date_from_unix_date(100000000). As written, ORDER BY a, CAST(dt AS STRING) and implicit casts like concat(s, dt) stay native. Could a date source be excluded for the cast to string as well, and the case added to #7471?

  2. spark/src/test/resources/sql-tests/operators/topk_later_key_ansi.sql:110. A small correction on "The first key is evaluated for every row in Spark too". Spark's takeOrdered goes through Guava's TopKSelector, which stores the first row without comparing it. So when the TopK's whole input is a single row, Spark evaluates no key at all, and WHERE id = 5 ORDER BY b + 1 LIMIT 10 on one overflowing row returns the row in Spark but raises in Comet. That isn't this PR's doing. Could the comment say "for every row when the input has at least two rows", and could you file a tracking issue for the single row case?

The check applies with ANSI off too, so Spark 3.4 and 3.5 plans are the ones that can move. I'll make sure all profiles and the Spark SQL suite run before it goes in the queue.

…-skipped-rows

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3ec1c07f-32b3-4cb1-a6ad-09293da13ab9
…e to string

Floating-point add, subtract, multiply, negation and abs cannot fail in any
mode, and reading a struct field cannot fail, so a later TopK key built from
them stays native. A cast from a date to a string raises natively for years
beyond chrono's range in every mode, so it is treated as able to fail. The
test comment about the first key now notes it holds for at least two rows
(apache#7527).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3ec1c07f-32b3-4cb1-a6ad-09293da13ab9
@tlfua

tlfua commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @coderfender, all four points were valid. Addressed in c181b6e (also merged the latest main):

  1. Float/double +, -, *, negation and abs are now allowed in any eval mode. Divide, IntegralDivide, Remainder and Pmod stay excluded under ANSI, and integral types are unchanged. Added an ANSI test ORDER BY a, d * 1.1D, d + 1, d - 1, -d, abs(d) that stays native.
  2. GetStructField is allowed (its child must also pass). Added a struct table with ORDER BY a, s.x, s.y staying native with ANSI on and off.
  3. A cast from DATE to STRING is now treated as able to fail in every mode, with an expect_fallback test for CAST(dt AS STRING). Reproduced locally (Cast error: Failed to convert 100000000 to temporal for Date32 for date_from_unix_date(100000000)) and added it to Native timestamp casts fail or panic for years beyond chrono's range #7471 (comment) .
  4. The test comment now says the first key is evaluated for every row only when the input has at least two rows. Reproduced the single-row case locally (Spark returns the row, Comet raises ARITHMETIC_OVERFLOW) and filed Native TopK evaluates sort keys for a single-row input, raising false ANSI errors #7527 .

Fork CI on the new head: all jobs passed ( https://github.com/tlfua/datafusion-comet/actions/runs/42145536518 ).

val to = cast.dataType
def isTimestamp(t: DataType) = t == TimestampType || t == TimestampNTZType
!isTimestamp(from) && !isTimestamp(to) && (evalMode(cast) match {
case CometEvalMode.TRY => true

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

try_cast(ts_array AS STRING) bypasses this timestamp check. With Long.MaxValue microseconds, the native cast errors while Spark formats it. Check nested timestamp types too, with a TopK regression.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, good catch. Fixed in 5a5b423. CometCast.cannotFail now looks inside arrays, structs and maps: a timestamp anywhere in the source or target type makes the cast able to fail, and so does a date in the source combined with a string in the target. A nested cast applies the same chrono-based kernels to each element, field, key and value, so they raise under TRY just like the top-level cast ( #7471 ).

topk_later_key_ansi.sql adds a topk_nested table with timestamp_micros(9000000000000000000) and date_from_unix_date(100000000), and checks that these later keys fall back: try_cast of array<timestamp> and array<date> to STRING and to array<string>, struct<timestamp> and struct<date> to STRING, struct<date> to struct<string> and struct<timestamp>, struct<timestamp> to struct<date>, map<int, date> to map<int, string>, map<timestamp, int> to map<string, int>, and array<struct<timestamp>> to STRING.

A cast over an array, struct or map applies the same native kernels to each
element, field, key and value, so a nested timestamp, or a nested date cast to
a string, raises for years Spark accepts, even under TRY. Check them at any
depth and add TopK regressions for each nested shape.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3ec1c07f-32b3-4cb1-a6ad-09293da13ab9
@tlfua
tlfua force-pushed the tlfua/fix/7246-topk-skipped-rows branch from 5a5b423 to c6f1dce Compare November 7, 2026 09:14

@rich7420 rich7420 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tlfua thanks for the patch!

})
}

private def existsNested(dataType: DataType)(p: DataType => Boolean): Boolean =

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we reuse SupportLevel.containsType here? It walks arrays, maps, and structs the same way, e.g. containsType(from, classOf[TimestampType], classOf[TimestampNTZType]). topk_later_key_ansi.sql still passed locally with existsNested replaced.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, done in 37be546 . cannotFail now uses SupportLevel.containsType for both the timestamp check and the date-to-string check, and the private existsNested is gone. Behavior is unchanged, so the existing topk_nested queries in topk_later_key_ansi.sql still cover it.


private def isAtomicCastType(dataType: DataType): Boolean = dataType match {
case _: NumericType | _: StringType | DataTypes.BooleanType | DataTypes.BinaryType |
DataTypes.DateType | DataTypes.TimestampType | _: TimestampNTZType =>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: hasTimestamp already rules out any timestamp, so the TimestampType/TimestampNTZType cases here never affect the result. Could we remove them?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, removed in 37be546 . isAtomicCastType now lists only DateType besides the numeric, string, boolean and binary types, with a comment that timestamps never reach it because cannotFail rejects them first.

@coderfender coderfender left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working through all of these. The float arithmetic and GetStructField cases now stay native, and the date to string exclusion and the nested check from rich7420's comment look right to me. #7471 and #7527 track the rest, so I'm dismissing my earlier request for changes. Two things I'd still like to see before merge.

  1. spark/src/main/scala/org/apache/comet/expressions/CometCast.scala:588-614. SupportLevel.containsType already walks array elements, struct fields and map keys and values the same way existsNested does, and SupportLevel is imported here. Could cannotFail use it and drop existsNested and the local isTimestamp? Something like

    val hasTimestamp =
      Seq(from, to).exists(containsType(_, classOf[TimestampType], classOf[TimestampNTZType]))
    val dateToString =
      containsType(from, classOf[DateType]) && containsType(to, classOf[StringType])

    The result is the same, since the date and timestamp types are case objects with private constructors. It leaves one recursive type walk instead of a third copy. Spark's DataType.existsRecursively would have been the obvious choice, but it is private[spark] in every version we support.

    While you're here, isAtomicCastType still lists TimestampType and TimestampNTZType. hasTimestamp rules them out before the LEGACY arm runs, so those arms can never match. Could they go? As written the list reads as if a LEGACY timestamp cast can't fail, which is the opposite of what #7471 found, and anyone who later relaxes the timestamp check would admit them without noticing.

  2. On comphead's cannotThrow thread. Keeping cannotThrow as it is makes sense for this PR. Could you file an issue for moving it onto cannotFail and link it there? #7322 adds a third list in NativeLambdaSerde.mayRaise, and #7535 is already about that one drifting, so it would be good to have the consolidation tracked.

@coderfender
coderfender dismissed their stale review November 9, 2026 19:17

All four points are addressed in c181b6e and 5a5b423. Remaining comments are in my latest review.

@coderfender coderfender added run-all-spark-profiles Run the Comet test suites against every Spark profile on this pull request, ahead of the merge queue run-spark-4.1-tests Run the Spark 4.1 SQL tests on this pull request instead of waiting for the merge queue run-spark-3.5-tests Run the Spark 3.5 SQL tests on this PR labels Nov 9, 2026
@coderfender

Copy link
Copy Markdown
Contributor

@tlfua I added run-all-spark-profiles, run-spark-4.1-tests and run-spark-3.5-tests. The check also applies with ANSI off, so the Spark 3.5 plans are worth seeing too.

Replace the private existsNested walker with SupportLevel.containsType, and
drop the timestamp arms of isAtomicCastType, which cannotFail never reaches
because it rejects any cast with a timestamp at any depth first. No change
in behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3ec1c07f-32b3-4cb1-a6ad-09293da13ab9
@tlfua

tlfua commented Nov 10, 2026

Copy link
Copy Markdown
Contributor Author

@coderfender thanks for the review and for adding the labels.

  1. containsType / dead timestamp arms: done in 37be546 . cannotFail uses SupportLevel.containsType as you suggested, the local isTimestamp and existsNested are gone, and isAtomicCastType no longer lists the timestamp types.
  2. cannotThrow: opened Consolidate the Scala "cannot fail" checks: arrays.cannotThrow and QueryPlanSerde.cannotFail #7585 and linked it in fix: fall back TopK whose later sort key can fail under ANSI #7457 (comment) .

Fork CI on the new commit is green: https://github.com/tlfua/datafusion-comet/actions/runs/42730158514 . Could you approve the workflows on this commit so the labelled Spark 3.5 / 4.1 / all-profiles runs start? They are waiting for approval.

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR backlog is huge and the maintainers don't have the bandwidth to review them all, so here is an automated review. Feel free to ignore it.

Thanks for the careful work here. I read GenerateOrdering and TakeOrderedAndProjectExec and the description of when later keys are evaluated is right. I also went through the new cannotFail whitelist in QueryPlanSerde.scala against the serdes for the allowed expressions. I did not find an allowed expression that can raise on the inputs in topk_later_key_ansi.sql, and the nested containsType handling in CometCast.cannotFail reads correctly.

The one thing I would like the author to weigh is the effect with ANSI off. laterSortKeyThatCanFail never looks at spark.sql.ansi.enabled, and cannotFail ends in case _ => false. So with ANSI off (the default on Spark 3.4 and 3.5), a TopK such as ORDER BY a, date_add(d, 1) LIMIT n or a later key using lpad, array_contains or to_date now returns Incompatible and runs on Spark. In the latest release those queries ran natively, and with ANSI off Spark returns NULL for the invalid cases rather than raising, so I could not find an error that the fallback is protecting against. The docs now say this outright, so it is a deliberate trade, but it means every TopK with a multi-key sort on an unlisted function loses native execution for users who never enabled ANSI. Could the check be narrowed when ANSI is off, for example by only treating an unlisted expression as fallible when it carries an ANSI eval mode or is in a short list of kernels known to raise regardless of mode? If not, could the PR say why the broader fallback is needed outside ANSI, and could the Spark 3.5 plan stability results confirm that no golden plan moves beyond q79?

Compared with the latest release, the only behavior change I found is the new fallback for multi-key TopK, and it is intended for the ANSI case. The other intended change is that spark.comet.operator.TakeOrderedAndProjectExec.allowIncompatible=true restores native execution.

@coderfender coderfender left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the follow-up. cannotFail now uses SupportLevel.containsType, the unreachable timestamp arms are gone, and #7585 tracks the cannotThrow consolidation. The Spark 3.5 SQL run is green too. LGTM.

@coderfender
coderfender added this pull request to the merge queue Nov 14, 2026
@coderfender
coderfender removed this pull request from the merge queue due to a manual request Nov 14, 2026
@coderfender

Copy link
Copy Markdown
Contributor

I'll wait for response on Andy's ask and perhaps lets keep existing behavior as is and only add guards for ANSI enabled ?

Comment on lines +1461 to +1462
def laterSortKeyThatCanFail(sortOrder: Seq[SortOrder]): Option[SortOrder] =
sortOrder.drop(1).find(s => !cannotFail(s.child))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Andy's point about ANSI off is right, and I missed it when I approved. With ANSI off, which is the default on Spark 3.5, a later key like date_add(d, 1), lpad(s, 5), array_contains(a, x) or to_date(s) now falls back because cannotFail treats anything it doesn't list as able to fail. All of those ran natively in 1.2, and with ANSI off they return NULL rather than raising, so that's a performance regression against the release.

One shape that can't be worse than 1.2 is to keep the allow-list under ANSI, and outside it only fall back for expressions that raise whatever the mode. 1.2 ran every top-K natively, so a list like that only adds safety on top of the release, even if it misses something. Roughly:

Suggested change
def laterSortKeyThatCanFail(sortOrder: Seq[SortOrder]): Option[SortOrder] =
sortOrder.drop(1).find(s => !cannotFail(s.child))
def laterSortKeyThatCanFail(sortOrder: Seq[SortOrder]): Option[SortOrder] = {
val canFail: Expression => Boolean =
if (SQLConf.get.ansiEnabled) e => !cannotFail(e) else raisesInAnyMode
sortOrder.drop(1).find(s => canFail(s.child))
}
/**
* With ANSI off, whether `expr` can still raise on a row that Spark's ordering skips. 1.2 ran
* every top-K natively, so this only adds safety over the release even if it is incomplete.
*/
private def raisesInAnyMode(expr: Expression): Boolean = expr.find {
// assert_true is replaced by If(_, null, RaiseError) before planning
case _: RaiseError | _: UserDefinedExpression => true
// element_at(array, 0) raises INVALID_INDEX_OF_ZERO in every mode
case e: ElementAt =>
e.left.dataType.isInstanceOf[ArrayType] && !(e.right.foldable && e.right.eval() != 0)
// an invalid pattern raises in every mode
case r: RLike => !r.right.foldable
// the native timestamp and date to string kernels raise even for TRY (#7471)
case c: Cast => CometCast.failsInAnyMode(c)
case _ => false
}.isDefined

CometCast.failsInAnyMode doesn't exist yet. It would be the hasTimestamp || dateToString part of cannotFail pulled out, so a LEGACY CAST(arr AS STRING) stays native the way it did in 1.2. Reading the session flag is a simplification, and you may prefer to look at each expression's eval mode as the arithmetic arms do. Could you also add a test with ANSI off where keys like date_add(d, 1) and lpad(s, 5) stay native, and one where element_at(arr, 0) still falls back?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, applied as suggested in 201edc6 : laterSortKeyThatCanFail keeps the cannotFail allow-list under ANSI and otherwise only falls back for raisesInAnyMode, with CometCast.failsInAnyMode pulled out of cannotFail. The ANSI-off section of topk_later_key_ansi.sql now has date_add(dt, 1) and lpad(s, 5) staying native and element_at(arr, 0) falling back. I kept the session flag as in your sketch and noted the per-expression eval mode, together with the expressions the list does not cover yet, on #7585 ( #7585 (comment) ).

@andygrove
andygrove added this pull request to the merge queue Nov 14, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 14, 2026
tlfua and others added 2 commits November 16, 2026 21:30
…-skipped-rows

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3ec1c07f-32b3-4cb1-a6ad-09293da13ab9
…mode

With spark.sql.ansi.enabled=false most invalid inputs give NULL instead of
an error, so falling back every TopK with an unlisted later key lost native
execution that Comet 1.2 had. Keep the allow-list under ANSI, and outside it
fall back only for a later key that can raise in any mode: element_at on an
array with an index that may be 0, rlike with a non-constant pattern, a
user-defined function, raise_error, or a cast to or from a timestamp or from
a date to a string (CometCast.failsInAnyMode).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3ec1c07f-32b3-4cb1-a6ad-09293da13ab9
@tlfua

tlfua commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

@andygrove thanks for the review, and @coderfender for the concrete shape. With ANSI off the check is now narrowed as you both suggested ( 201edc6 ):

  • Under spark.sql.ansi.enabled=true nothing changes: a later key outside the cannotFail allow-list falls back.
  • With ANSI off a TopK falls back only when a later key can raise in any mode: element_at on an array with an index that may be 0, rlike with a non-constant pattern, a user-defined function, raise_error, or a cast to or from a timestamp or from a date to a string. Keys like date_add, lpad and array_contains run natively again, as in 1.2. Since 1.2 ran every TopK natively, this only adds safety over the release.
  • operators.md now describes both modes, and the ANSI-off section of topk_later_key_ansi.sql covers date_add(dt, 1) and lpad(s, 5) staying native and element_at(arr, 0) falling back.

On plan stability: the PR does not change any approved plan. q79's later sort keys are substr(s_city, 1, 30) and an aggregate column, both of which cannotFail accepts, so its shared golden plan keeps CometTakeOrderedAndProject in every mode. The PR already has run-all-spark-profiles and run-spark-3.5-tests, so the Spark 3.5 plan stability suite will report on this commit once the workflows are approved. Fork CI: https://github.com/tlfua/datafusion-comet/actions/runs/44025342119

The gaps that are left (the list is best-effort, and the ANSI check reads the session flag rather than each expression's eval mode) are noted on #7585 for a follow-up.

@coderfender could you approve the workflows for the new push when you have a moment?

@tlfua

tlfua commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

FYI for re-queueing: the last merge-queue removal was an infra flake, not a test failure. The [expressions] job couldn't resolve surefire-junit4:3.5.4 from Maven Central ( https://github.com/apache/datafusion-comet/actions/runs/43643707375 ).

The current head 201edc6 (ANSI-off narrowing from the latest review) is green on my fork ( https://github.com/tlfua/datafusion-comet/actions/runs/44025342119 ). Could a committer approve the workflows for it, and re-approve and re-enqueue when convenient? Thanks!

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for narrowing this so quickly. With ANSI off, date_add, lpad and array_contains later keys run in Comet again, and the new ANSI-off section of topk_later_key_ansi.sql is a nice way to pin that down. That covers most of what I raised.

One part of it is still there. The cast arm in raisesInAnyMode (QueryPlanSerde.scala:1482) flags any cast with a timestamp on either side. Spark adds those casts quietly, so with ANSI off a later key like date_add(ts, 1), datediff(x, ts), trunc(ts, 'MM'), date_format(dt, 'yyyy') or concat(s, dt) still falls back where 1.2 ran it in Comet. Spark doesn't raise for these with ANSI off. Only our own kernel does, on out-of-range values (#7471), and it does that in a projection or first sort key too. Could this arm apply only under ANSI and leave #7471 to cover the kernel? If you'd rather keep it, could operators.md say it is a Comet limit rather than an expression that raises in any mode?

The gate now reads SQLConf.get.ansiEnabled (QueryPlanSerde.scala:1464) rather than each expression's eval mode. A view created with ANSI on keeps its Add in ANSI mode when it is queried with ANSI off, so ORDER BY a, b + 1 over it would now run in Comet and could raise on rows Spark skips. The previous commit fell back there. Could the ANSI-off branch also treat a node with an ANSI eval mode that cannotFail rejects as able to fail? That would close the gap in this PR instead of leaving it to #7585.

A few kernels raise in every mode in both Spark and Comet, the same way element_at(arr, 0) does. slice with a start of 0 or a negative length, array_insert with position 0, and sequence with invalid bounds all do. Could they go into raisesInAnyMode too?

On tests, could the ANSI-off section get an expect_fallback case for a timestamp cast and for a user-defined function or raise_error, plus a case showing element_at(arr, 1) stays in Comet? In operators.md, "still runs natively" could be "still runs in Comet" to match the style guide. It would also help to say that spark.comet.operator.TakeOrderedAndProjectExec.allowIncompatible applies with ANSI off too.

I've approved the workflows on 201edc6, so the labelled Spark 3.5 and all-profiles runs should report before this goes back in the queue.

tlfua and others added 2 commits November 17, 2026 09:59
…views, slice, array_insert and sequence

- With ANSI off, a cast to or from a timestamp no longer makes a TopK fall back. Spark does not
  raise there; the Comet kernel failure on extreme years also hits projections and is tracked in
  apache#7471.
- A later key that still carries ANSI eval mode, such as one from a view created with ANSI on,
  falls back even when the session has ANSI off.
- slice with a start of 0 or a negative length, array_insert at position 0, and sequence with
  non-constant bounds raise in every mode, so they fall back with ANSI off.
- Tests for timestamp-cast keys staying in Comet, element_at(arr, 1), the new array functions, a
  view, and a ScalaUDF later key.
- operators.md: use "runs in Comet" (style guide N1) and say allowIncompatible applies in either
  mode. Replace "native" wording in comments with "in Comet" or "Rust".

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3ec1c07f-32b3-4cb1-a6ad-09293da13ab9
Spark 4.1 keeps the eval mode of Add and the other arithmetic expressions inside a
NumericEvalContext, so scanning the product fields for EvalMode.ANSI missed an ANSI b + 1 from
a view. Match the arithmetic and cast classes and read their evalMode instead.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3ec1c07f-32b3-4cb1-a6ad-09293da13ab9
@tlfua

tlfua commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @andygrove, all five points are addressed in ade6619 and d4f087e .

  1. Timestamp casts with ANSI off. Removed the cast arm from raisesInAnyMode, so with ANSI off a timestamp-cast later key now runs in Comet. Under ANSI, CometCast.cannotFail still rejects these casts. @coderfender, this reverses the arm you added deliberately: Spark does not raise for these casts with ANSI off, and the chrono range failure is a Comet kernel bug that hits a projection the same way, so it belongs to Native timestamp casts fail or panic for years beyond chrono's range #7471 rather than to the TopK check.
  2. Views. With ANSI off, a later key that holds a node still in ANSI eval mode (arithmetic and casts by their EvalMode, plus failOnError/ansiEnabled on UnaryMinus, Abs, Round, BRound, ElementAt, GetArrayItem and Elt) and that cannotFail rejects now falls back. The test creates SELECT a, b FROM topk ORDER BY a, b + 1 LIMIT 2 as a temporary view with ANSI on and queries it with ANSI off. Spark 4.1 keeps the arithmetic eval mode inside a NumericEvalContext, which the first version missed; fork CI caught it on the Spark 4.1 profile, and the follow-up commit reads evalMode by class instead. A view that only projects b + 1 AS c and is then sorted by c is not affected, because the projection evaluates b + 1 for every row in Spark too.
  3. slice, array_insert, sequence. Added to raisesInAnyMode: slice unless the start is a nonzero constant and the length a non-negative constant, array_insert unless the position is a nonzero constant, and sequence unless all of its arguments are constants (invalid constant bounds already fail in constant folding).
  4. Tests.
    • Timestamp casts: since they no longer fall back with ANSI off, date_add(ts, 1) and concat(s, ts) are plain query tests that check they stay in Comet.
    • UDF: raise_error cannot reach this check in a SQL file test, because Comet has no serde for RaiseError and the TopK falls back for that first. So CometCodegenSuite has a ScalaUDF later-key test with ANSI off instead.
    • element_at(arr, 1), slice(arr, 1, 1) and array_insert(arr, 1, 0) stay in Comet; column-driven slice, array_insert and sequence fall back.
  5. Docs. operators.md now says "runs in Comet", lists the new ANSI-off cases and the view case, says allowIncompatible applies in either mode, and links Native timestamp casts fail or panic for years beyond chrono's range #7471 for timestamp casts. I also replaced the same "native" wording in the code comments.

Fork CI passed on d4f087e: https://github.com/tlfua/datafusion-comet/actions/runs/44155798081 . One Spark SQL job was rerun after a Maven Central download failure. Could you approve the workflows again when you have a moment?

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working through the earlier rounds so carefully. I read GenerateOrdering and TakeOrderedAndProjectExec for 3.4.3, 3.5.8, 4.0.1 and 4.1.1, and the description of when later keys are evaluated is right. I also checked the new raisesInAnyMode arms against Spark. element_at(array, 0), array_insert with position 0, slice with a start of 0 or a negative length, and sequence with bad bounds raise in every mode in all four versions. The members that hasAnsiEvalMode reads (failOnError, ansiEnabled and evalMode) exist in 3.4.3 through master, and reading evalMode by name is the right way to handle the 4.1 NumericEvalContext. Arrow's date_part returns NULL for out-of-range dates and timestamps rather than raising, so keeping year and hour on the allow-list is fine.

Compared with branch-1.2, every multi-key TopK ran in Comet there. With ANSI on, a later key outside the allow-list now runs on Spark. That is a deliberate trade and the docs say so. With ANSI off the fallback only adds safety over 1.2.

I have one request, inline on the ANSI-off section of the SQL test.


-- A constant index other than 0 gives NULL when out of bounds
query
SELECT a FROM topk_arrays ORDER BY a, element_at(arr, 1) LIMIT 3

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

raisesInAnyMode flags rlike with a non-constant pattern, and that shape is reachable. CometRLike.convert sends a non-literal pattern through CometScalaUDF.emitJvmCodegenDispatch, so Spark's doGenCode runs inside the Comet pipeline over every row and an invalid pattern raises on rows Spark skips.

I don't see a case for it in the ANSI-off section. Every other arm of raisesInAnyMode that can reach this check has one. Could you add an expect_fallback query with the pattern in a column, for example ORDER BY a, s RLIKE p LIMIT 3, plus a constant-pattern query that stays in Comet?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @andygrove, added in tlfua@1650696 . The ANSI-off section now has a topk_regex table where row 2 holds the invalid pattern '(':

  • SELECT a FROM topk_regex ORDER BY a, s RLIKE p LIMIT 3 is an expect_fallback query. No two rows tie on a, so Spark never compiles '(', while the JVM codegen dispatcher would evaluate it on every row in Comet.
  • SELECT a FROM topk_regex ORDER BY a, s RLIKE '^a' LIMIT 3 is a plain query that stays in Comet.

Fork CI passed on 1650696: https://github.com/tlfua/datafusion-comet/actions/runs/44472467670 . Some jobs were rerun after Maven Central download failures. The upstream workflows need a committer to approve them again for the new commit.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3ec1c07f-32b3-4cb1-a6ad-09293da13ab9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:expressions Expression evaluation bug Something isn't working run-all-spark-profiles Run the Comet test suites against every Spark profile on this pull request, ahead of the merge queue run-spark-3.5-tests Run the Spark 3.5 SQL tests on this PR run-spark-4.1-tests Run the Spark 4.1 SQL tests on this pull request instead of waiting for the merge queue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants