Skip to content

Consolidate the Scala "cannot fail" checks: arrays.cannotThrow and QueryPlanSerde.cannotFail #7585

Open

Description

@tlfua

What is the problem the feature request solves?

Comet's Scala side has more than one answer to "can evaluating this expression raise an error?":

Each copy has to be widened or tightened on its own, so a fix to one (for example a new cast that cannot fail, or a kernel found to raise where Spark does not) is easy to miss in the others.

Describe the potential solution

Keep one Scala notion of an expression that cannot fail, QueryPlanSerde.cannotFail, and have arrays.scala (and NativeLambdaSerde.mayRaise, if #7322 lands first) call it instead of keeping its own list.

This widens when array_append runs natively, so the change needs its own tests for the newly admitted item expressions, including ones that fail under ANSI.

Additional context

Raised in the review of #7457 :

Willingness to contribute

I can contribute a fix for this bug independently

Activity

  1. tlfua commented on Nov 17, 2026

    @tlfua
    ContributorAuthor

    One more input for this consolidation, from the review of #7457 ( #7457 (review) and #7457 (comment) ).

    With spark.sql.ansi.enabled=false, #7457 now falls back a TopK only when a later sort key can raise in any mode (QueryPlanSerde.raisesInAnyMode). That list is intentionally best-effort. Two gaps are left for a follow-up:

    1. The list is incomplete. Spark raises regardless of ANSI for, among others, regexp_extract with an invalid group index, slice with a start of 0 or a negative length, sequence and array_repeat past the array size limit, str_to_map with a duplicate key under the default mapKeyDedupPolicy, and make_ym_interval on overflow. Comet 1.2 ran all of these natively too, so this is no regression, but a single shared list would cover them for TopK as well as for the other callers above.
    2. The ANSI check reads the session flag, not each expression's eval mode. A plan analyzed with ANSI on, such as a view that captured spark.sql.ansi.enabled=true or a DataFrame analyzed before the session flag changed, keeps ANSI Cast and arithmetic nodes. When it runs in a session with ANSI off, the TopK check takes the ANSI-off branch, so such a key can still raise on a row Spark's ordering skips. Asking each node for its own eval mode, as cannotFail already does for the arithmetic arms, would close this.
  2. tlfua commented on Nov 17, 2026

    @tlfua
    ContributorAuthor

    An update on the two gaps in #7585 (comment) , after #7457 (review) asked to close the second one in that PR.

    1. The list is incomplete. fix: fall back TopK whose later sort key can fail under ANSI #7457 (commits ade6619 and d4f087e ) adds slice (start 0 or negative length), array_insert (position 0) and sequence with non-constant bounds to raisesInAnyMode. The others listed there (regexp_extract with an invalid group index, array_repeat past the size limit, str_to_map with a duplicate key, make_ym_interval on overflow) are still open here.
    2. Session flag vs. eval mode. Now handled in the same commits: with ANSI off, a later sort key that holds a node still in ANSI eval mode, which cannotFail rejects, makes the TopK fall back. It reads arithmetic and casts by their EvalMode, plus the failOnError/ansiEnabled flags of UnaryMinus, Abs, Round, BRound, ElementAt, GetArrayItem and Elt. Other expressions with an ANSI flag, such as make_date and conv, are not covered yet and belong to the shared list this issue proposes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions