You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Consolidate the Scala "cannot fail" checks: arrays.cannotThrow and QueryPlanSerde.cannotFail #7585
Comet's Scala side has more than one answer to "can evaluating this expression raise an error?":
cannotThrow in arrays.scala admits only a Literal, Attribute or BoundReference. CometArrayAppend uses it to decide whether the item can be evaluated for a row Spark would not evaluate it on:
Each copy has to be widened or tightened on its own, so a fix to one (for example a new cast that cannot fail, or a kernel found to raise where Spark does not) is easy to miss in the others.
Describe the potential solution
Keep one Scala notion of an expression that cannot fail, QueryPlanSerde.cannotFail, and have arrays.scala (and NativeLambdaSerde.mayRaise, if #7322 lands first) call it instead of keeping its own list.
This widens when array_append runs natively, so the change needs its own tests for the newly admitted item expressions, including ones that fail under ANSI.
With spark.sql.ansi.enabled=false, #7457 now falls back a TopK only when a later sort key can raise in any mode (QueryPlanSerde.raisesInAnyMode). That list is intentionally best-effort. Two gaps are left for a follow-up:
The list is incomplete. Spark raises regardless of ANSI for, among others, regexp_extract with an invalid group index, slice with a start of 0 or a negative length, sequence and array_repeat past the array size limit, str_to_map with a duplicate key under the default mapKeyDedupPolicy, and make_ym_interval on overflow. Comet 1.2 ran all of these natively too, so this is no regression, but a single shared list would cover them for TopK as well as for the other callers above.
The ANSI check reads the session flag, not each expression's eval mode. A plan analyzed with ANSI on, such as a view that captured spark.sql.ansi.enabled=true or a DataFrame analyzed before the session flag changed, keeps ANSI Cast and arithmetic nodes. When it runs in a session with ANSI off, the TopK check takes the ANSI-off branch, so such a key can still raise on a row Spark's ordering skips. Asking each node for its own eval mode, as cannotFail already does for the arithmetic arms, would close this.
The list is incomplete.fix: fall back TopK whose later sort key can fail under ANSI #7457 (commits ade6619 and d4f087e ) adds slice (start 0 or negative length), array_insert (position 0) and sequence with non-constant bounds to raisesInAnyMode. The others listed there (regexp_extract with an invalid group index, array_repeat past the size limit, str_to_map with a duplicate key, make_ym_interval on overflow) are still open here.
Session flag vs. eval mode. Now handled in the same commits: with ANSI off, a later sort key that holds a node still in ANSI eval mode, which cannotFail rejects, makes the TopK fall back. It reads arithmetic and casts by their EvalMode, plus the failOnError/ansiEnabled flags of UnaryMinus, Abs, Round, BRound, ElementAt, GetArrayItem and Elt. Other expressions with an ANSI flag, such as make_date and conv, are not covered yet and belong to the shared list this issue proposes.
Activity
tlfua commentedon Nov 17, 2026
One more input for this consolidation, from the review of #7457 ( #7457 (review) and #7457 (comment) ).
With
spark.sql.ansi.enabled=false, #7457 now falls back a TopK only when a later sort key can raise in any mode (QueryPlanSerde.raisesInAnyMode). That list is intentionally best-effort. Two gaps are left for a follow-up:regexp_extractwith an invalid group index,slicewith a start of 0 or a negative length,sequenceandarray_repeatpast the array size limit,str_to_mapwith a duplicate key under the defaultmapKeyDedupPolicy, andmake_ym_intervalon overflow. Comet 1.2 ran all of these natively too, so this is no regression, but a single shared list would cover them for TopK as well as for the other callers above.spark.sql.ansi.enabled=trueor a DataFrame analyzed before the session flag changed, keeps ANSICastand arithmetic nodes. When it runs in a session with ANSI off, the TopK check takes the ANSI-off branch, so such a key can still raise on a row Spark's ordering skips. Asking each node for its own eval mode, ascannotFailalready does for the arithmetic arms, would close this.tlfua commentedon Nov 17, 2026
An update on the two gaps in #7585 (comment) , after #7457 (review) asked to close the second one in that PR.
slice(start 0 or negative length),array_insert(position 0) andsequencewith non-constant bounds toraisesInAnyMode. The others listed there (regexp_extractwith an invalid group index,array_repeatpast the size limit,str_to_mapwith a duplicate key,make_ym_intervalon overflow) are still open here.cannotFailrejects, makes the TopK fall back. It reads arithmetic and casts by theirEvalMode, plus thefailOnError/ansiEnabledflags ofUnaryMinus,Abs,Round,BRound,ElementAt,GetArrayItemandElt. Other expressions with an ANSI flag, such asmake_dateandconv, are not covered yet and belong to the shared list this issue proposes.