Skip to content

fix: let shared subexpressions that cannot raise be skipped by AtLeastNNonNulls - #7529

Open
andygrove wants to merge 3 commits into
apache:mainfrom
andygrove:atleastn-skippable-shared-subexpr
Open

andygrove wants to merge 3 commits into
apache:mainfrom
andygrove:atleastn-skippable-shared-subexpr

Conversation

@andygrove

@andygrove andygrove commented Nov 6, 2026 •

Copy link
Copy Markdown
Member

Which issue does this PR close?

N/A. Follow-up to #6180 (native AtLeastNNonNulls).

Rationale for this change

AtLeastNNonNulls (what df.na.drop(n) plans) falls back to Spark with AtLeastNNonNulls would skip a subexpression that Spark evaluates for every row whenever two of its children share a subexpression, even when that subexpression cannot fail. A common case is na.drop over columns derived from one input:

df.withColumn("prefix", substr($"s", 1, 4))
  .withColumn("is_recent", substr($"s", 1, 4).cast("int") >= 16)
  .withColumn("is_latest", substr($"s", 1, 4).cast("int") >= 18)
  .na.drop(1)

The filter runs on Spark, with a transition on each side, although every expression in it is native.

The guard exists because Spark evaluates a shared subexpression for every row, so a shared expression that fails (an ANSI cast of a malformed string, say) raises even on a row where AtLeastNNonNulls has already reached n. Native evaluation that skips the later children would not raise. If the shared subexpression cannot fail, skipping it shows nothing, so the guard has no reason to apply. The same check (canSkipEvaluation) also decides the null short-circuit of function arguments, so this applies to them too.

What changes are included in this PR?

  • SharedSubexpressions.common no longer contains a shared subexpression that cannot raise. A new cannotRaise check lists what cannot fail whatever the row holds: literals and column references, struct field reads, comparisons and boolean operators, coalesce, if and CASE, substr, upper, lower, length, startswith, endswith and contains, and casts and numeric arithmetic outside ANSI mode. Under ANSI mode, which is the default from Spark 4, a cast or arithmetic that can fail still counts as able to raise, so the filter above keeps falling back there.
  • The list fails closed. Anything it does not name counts as able to raise, including element_at (the index 0 raises in every mode), an ANSI cast or arithmetic, interval arithmetic (it raises on overflow in every mode), and a decimal expression after promotion (which wraps it in CheckOverflow).
  • The two QueryPlanSerdeSuite tests that used a string-to-int cast as the shared expression now use element_at(arr, 0), which raises in every mode, so they still pin the mechanism. The cast is no longer a case where the guard applies outside ANSI mode.

How are these changes tested?

  • QueryPlanSerdeSuite: a shared length and a shared struct field can be skipped. A shared element_at(arr, 0), a shared interval addition, and an expression that cannot raise but holds a raising one still block the skip, also when other shared subexpressions that cannot raise are present.
  • CometExpressionSuite: two AtLeastNNonNulls queries that share upper(s), and a struct field, with subexpression elimination on. Both now run natively and match Spark. The two existing tests, which share an ANSI cast that fails and an ANSI divide by zero, still fall back and still raise.
  • CometSqlFileTestSuite (769 tests) and the AtLeastNNonNulls tests of CometExpressionSuite (8 tests) pass on Spark 3.5. QueryPlanSerdeSuite and the AtLeastNNonNulls tests also pass on Spark 3.4, and the main and test code compile on Spark 3.4, 4.0 and the default profile. I have not run the tests on 4.x.
  • The example above runs as a Spark Filter on current main and as a CometFilter with this change.

Related

#7322 has a similar fail-closed check, private to CometArrayFilter, for whether a lambda body can raise. If both land they can share one classifier; I kept them apart so neither PR waits on the other.

@github-actions github-actions Bot added bug Something isn't working area:expressions Expression evaluation labels Nov 6, 2026
…tNNonNulls

AtLeastNNonNulls, and the null short-circuit of function arguments, refuse to skip an argument that
holds a subexpression Spark shares between the operator's expressions. Spark evaluates a shared
subexpression for every row, so skipping one could hide an error that Spark raises. The check
applied to every shared subexpression, so a filter whose columns merely read the same struct field
or take the same substring, such as na.drop over columns derived from one input, fell back to
Spark, with a transition on each side.

Only a shared subexpression that can raise needs to be kept. canSkipEvaluation now leaves out those
made of expressions that cannot fail whatever the row holds: literals and columns, struct field
reads, comparisons and boolean operators, string functions such as substr and upper, and casts and
numeric arithmetic outside ANSI mode. The list fails closed, so anything it does not name, such as
element_at, an ANSI cast, interval arithmetic or a decimal expression after promotion, still counts
as able to raise.
@andygrove
andygrove force-pushed the atleastn-skippable-shared-subexpr branch from 927cba9 to ce1feae Compare November 6, 2026 04:17

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, the fail-closed list reads well and the tests pin both directions. One gap in the Cast arm.

case _: Length | _: Upper | _: Lower | _: Substring | _: StartsWith | _: EndsWith |
_: Contains =>
true
case c: Cast => evalMode(c) != CometEvalMode.ANSI

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A non-ANSI cast can still raise. Cast from DATE to TIMESTAMP or TIMESTAMP_NTZ goes through daysToMicros and instantToMicros, which use Math.multiplyExact, so a date past about year 294247 (dates go up to 5881580) throws ArithmeticException in every eval mode. With such a cast shared between na.drop children, Spark raises on every row while native evaluation would skip it. The comment on keyCastCannotFail in CometCast.scala already calls out these date and timestamp conversions. Could this arm also exclude casts to or from date, timestamp and interval types, so the motivating string to int cast keeps the skip and the list stays closed for the conversions that overflow? I have not run it.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, you're right. In 3b0c8f2 the Cast arm treats a non-ANSI cast as unable to raise only when both of its types are numeric, string, boolean, binary or null, or arrays and structs of those. This is an allowlist rather than an exclusion list, so it fails closed like the rest. Dates, timestamps (including NTZ) and intervals are left out, along with maps, since a cast key can become NULL, and the TIME type. I checked Spark 4.1.1 and 3.4.3: the date-to-timestamp path goes through daysToMicros and instantToMicros and Math.multiplyExact with no catch outside TRY. QueryPlanSerdeSuite now asserts that a shared LEGACY string-to-int cast is still skippable, while date to TIMESTAMP, date to TIMESTAMP_NTZ and array of date to array of timestamp are not. That suite and the AtLeastNNonNulls tests in CometExpressionSuite pass.

…m being skipped

A non-ANSI cast from a date to a timestamp can still raise: daysToMicros
goes through Math.multiplyExact, which overflows for a date past about the
year 294247 in every eval mode. Only treat a non-ANSI cast as unable to
raise when both of its types are numeric, string, boolean, binary or null,
or arrays and structs of those.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@andygrove
andygrove requested a review from comphead November 17, 2026 22:47

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks

@comphead comphead left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @andygrove great PR

CI pending

andygrove added a commit to andygrove/datafusion-comet that referenced this pull request Nov 19, 2026
…y native

- A lambda body that applies a function to a date or timestamp stays on the
  dispatcher, since native date functions return NULL or fail beyond chrono's
  range (apache#6843, apache#7471). Comparisons, NULL tests, CASE/IF/COALESCE and casts to
  a number stay native.
- Only IN needs the short-circuit check. Native AND and OR are LazyLogical and
  evaluate a right operand that can fail only on the rows the left leaves open.
- mayRaise uses the same rules as apache#7529: numeric-only arithmetic, casts only
  between plain types, and GetStructField.
- is_deterministic in lazy_logical.rs accepts a LambdaVariable.
- The planner resolves the higher-order function through the session.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@andygrove
andygrove added this pull request to the merge queue Nov 19, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 19, 2026
sunchao pushed a commit to sunchao/arrow-datafusion-comet that referenced this pull request Nov 20, 2026
…-order functions (apache#7322)

* feat: run filter lambdas natively with DataFusion's array_filter

A filter lambda other than `x IS NOT NULL` went through the JVM codegen
dispatcher, which evaluates it one element at a time like Spark does, plus
the transfer of each batch. A body made of natively supported expressions
now runs once over all the elements of a batch through DataFusion's
array_filter.

The native path applies to a one-parameter lambda over an array of a
primitive, decimal, string, binary, date or timestamp element type. These
stay on the dispatcher: nested lambdas, nondeterministic bodies, bodies
that call out to the JVM, non-default string collations, and under ANSI
mode a body that short-circuits (AND, OR, IN) around an operation that can
raise, since the native operators evaluate every operand. The new
spark.comet.exec.nativeLambda.enabled turns the native path off.

Adds a lambda tuning section to the user guide and a benchmark.

Closes apache#7321.

* fix: keep a lambda parameter readable by short-circuited array function arguments

NullShortCircuit evaluates the later arguments of a function such as array_contains on a
projection of just the columns they read. It re-indexed only Column, so a lambda parameter
passed as such an argument, as in filter(ints, x -> array_contains(other, x)), was dropped
from the projection and the query failed. Treat LambdaVariable like Column when projecting,
and like Column in case_when's infallible check.

The JVM codegen dispatcher now declines a tree that reads a lambda parameter bound outside
it. Its kernel is fed only columns, so the parameter read as NULL. The filter serde no longer
walks the serialized body to find such a call.

Also simplify the planner arm test, share assertNoCodegen between the codegen suites, and
tidy the benchmark.

* fix: decline the native filter lambda for wide captures and for bodies that can raise in any mode

DataFusion copies each column a lambda captures once per element of the array, so a captured
string, binary or nested column is copied as many times as the array has elements. A body that
reads one now stays on the dispatcher.

The check for a body that short-circuits (AND, OR, IN) no longer depends on the ANSI setting.
It classifies each expression: arithmetic and casts by the eval mode they carry, and anything
not known to be safe, such as element_at, which raises for the index 0 in any mode, as able to
raise.

The SQL tests pin the path of each query with expect_native and expect_dispatch, replacing the
ConfigMatrix over the native lambda flag and the codegen suite tests, and cover tinyint and
smallint elements.

* feat: run nvl(x, '') != '' on a numeric array as array_compact

On a numeric, date or timestamp array Spark plans the customer's rule
filter(arr, x -> nvl(x, '') != '') as NOT (coalesce(CAST(x AS STRING), '') = ''),
in either operand order. No such cast is empty, so the predicate holds exactly
when x is not NULL, and the lambda is x IS NOT NULL. Send it down the
array_compact fast path instead of casting every element to a string.

On 2M rows of ten-element int arrays (every fifth element NULL), Spark 3.5.5
local[4], best of 5: 0.453 s on the dispatcher, 0.281 s with the native lambda,
0.086 s with this change, against 0.220 s for Spark.

* feat: run exists lambdas natively with DataFusion's array_any_match

CometArrayExists now serializes a lambda whose body Comet supports natively as a HigherOrderFunc
under array_any_match, as filter does with array_filter, when three-valued logic is on. The gate and
the serialization move into a trait both serdes share. exists stays on the dispatcher with
spark.sql.legacy.followThreeValuedLogicInArrayExists=false, since Spark returns false where
array_any_match returns NULL, and when the array argument contains a cast to a map, whose TRY cast
can produce a null key that Arrow cannot hold (apache#6172).

Also fixes a higher-order function inside an expression that is evaluated only for some rows: as the
right operand of AND or OR, or inside an argument of a function like array_contains. These failed
with 'project index N out of bounds', because project() in null_short_circuit.rs collected the
function's own lambda parameter as a column of the outer batch. It now ignores variables bound by a
lambda inside the expression it projects. This affected filter as well as exists.

* bench: add exists cases to CometHigherOrderFunctionBenchmark and the tuning guide

* fix: keep an exists lambda that can raise on the dispatcher

Spark's exists stops at the first element that matches, so it never
evaluates the lambda on the elements after it, while array_any_match
evaluates the lambda on every element. With ANSI on,
exists(ints, x -> 100 / x > 1) over array(1, 0) raised DIVIDE_BY_ZERO
natively where Spark returns true, and with ANSI off element_at with the
index 0 raised in the same way. exists now also requires that no
expression in the lambda can raise, whatever operators it uses.

Also from review:
- Reuse CometCast.canProduceNullMapKey for the map-cast check, so a
  widening key cast in the array argument runs natively.
- Make CometArrayExists a plain serde that calls the dispatcher itself,
  like CometArrayFilter, so the generated docs say Native, and give its
  row the same notes as filter.
- Collect the projection in one walk in NullShortCircuit's project().
- Drop tests that repeat the SQL files: the exists test in
  CometCodegenHOFSuite, the array_any_match half of the planner test and
  the numeric nvl query in array_filter_native_lambda.sql.

* fix: keep a native lambda body with a null map key or a date/time cast on the dispatcher

A body that holds a TRY cast making a null map key cannot be built natively, as an argument
with one cannot (apache#6172), and a cast between a date or timestamp and a string, date or timestamp
fails natively for years beyond chrono's range, which Spark supports (apache#7471). Both now stay on
the dispatcher, with a fixture that pins the path and the results.

* fix: keep date functions out of native lambdas and let AND and OR stay native

- A lambda body that applies a function to a date or timestamp stays on the
  dispatcher, since native date functions return NULL or fail beyond chrono's
  range (apache#6843, apache#7471). Comparisons, NULL tests, CASE/IF/COALESCE and casts to
  a number stay native.
- Only IN needs the short-circuit check. Native AND and OR are LazyLogical and
  evaluate a right operand that can fail only on the rows the left leaves open.
- mayRaise uses the same rules as apache#7529: numeric-only arithmetic, casts only
  between plain types, and GetStructField.
- is_deterministic in lazy_logical.rs accepts a LambdaVariable.
- The planner resolves the higher-order function through the session.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:expressions Expression evaluation bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants