Skip to content

fix: [branch-1.2] evaluate the right operand of AND, OR and IN only for rows the left side has not decided (#7326) - #7648

Merged
andygrove merged 2 commits into
apache:branch-1.2from
andygrove:backport-7326-branch-1.2
Nov 17, 2026
Merged

andygrove merged 2 commits into
apache:branch-1.2from
andygrove:backport-7326-branch-1.2

Conversation

@andygrove

Copy link
Copy Markdown
Member

Which issue does this PR close?

Backport of #7326 to branch-1.2. Closes #6823 on this branch.

Rationale for this change

Spark evaluates AND, OR and the candidates of IN lazily: a later operand runs only for the rows that the earlier ones have not decided. Comet evaluates every operand over the whole batch, so under ANSI mode an operand that can raise throws on a row Spark never evaluates, for example v != 0 AND 100 / v > 1, v = 0 OR 100 / v > 1 and v IN (0, 100 / v) raise DIVIDE_BY_ZERO in Comet and return rows in Spark. branch-1.2 has the same eager native evaluation. See #7326 and #6823 for the full analysis.

What changes are included in this PR?

Cherry-pick of 7deeba2 (git cherry-pick -x) with no conflicts and no adaptations. It adds LazyLogical and lazy_in_list in native/spark-expr, the small changes to case_when.rs, null_short_circuit.rs and nested_comparison.rs that they share, the lazy_logical benchmark, and the expressions/conditional/logical_lazy_ansi.sql file test.

Other release branches: branch-1.2 is the newest release branch, so there is no newer one to port to. Whether branch-1.1 needs a backport is not part of this PR.

How are these changes tested?

On this branch, with JDK 17 and the default Spark profile (4.1): cargo check --all-targets and cargo clippy --all-targets -D warnings pass for datafusion-comet-spark-expr, cargo fmt --check is clean, and the datafusion-comet-spark-expr lib tests pass (1310 tests, 0 failures, including the 11 new lazy_logical tests). CometSqlFileTestSuite passes for expressions/conditional/logical_lazy_ansi.sql against a native library built from this branch. Against a native library built from branch-1.2 without the fix, the same file test fails with DIVIDE_BY_ZERO, so the branch has the bug and the test catches it. I ran the default Spark profile only.

…eft side has not decided (apache#7326)

* fix: evaluate the right operand of AND, OR and IN only for rows the left side has not decided

* refactor: share masked evaluation with NullShortCircuit and build AND/OR in spark_comparison

* bench: add micro-benchmarks for lazy AND, OR and IN

* perf: try the right operand over the whole batch when most rows need it, and keep runs of IN candidates as one IN

* fix: keep an expensive IN value out of the lazy rewrite and build the OR tree balanced

(cherry picked from commit 7deeba2)

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks

@andygrove
andygrove merged commit aaba3e0 into apache:branch-1.2 Nov 17, 2026
138 checks passed
@andygrove
andygrove deleted the backport-7326-branch-1.2 branch November 17, 2026 22:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants