[SPARK-56977][SQL] RewriteNearestByJoin should respect joinType in the synthetic join#56052
Closed
zhidongqu-db wants to merge 1 commit into
Closed
[SPARK-56977][SQL] RewriteNearestByJoin should respect joinType in the synthetic join#56052zhidongqu-db wants to merge 1 commit into
zhidongqu-db wants to merge 1 commit into
Conversation
…e synthetic join ### What changes were proposed in this pull request? This PR changes `RewriteNearestByJoin` to construct its synthetic cross-join with the user's `joinType` (`Inner` or `LeftOuter`) instead of always using `LeftOuter`. The `Generate` operator's `outer` flag continues to be derived from `joinType == LeftOuter`, so the externally observable semantics are unchanged. ### Why are the changes needed? The original implementation hardcoded the synthetic join to `LeftOuter` and justified it on the grounds that `LEFT OUTER` and `INNER` are equivalent for an unconditioned join when the right side is non-empty, and `Generate(outer = false)` would drop unwanted rows for `INNER` when right is empty. That reasoning holds for correctness but has a major performance cost: - **`INNER NEAREST BY` cannot be planned as a Cartesian product.** Spark's strategy picks `CartesianProductExec` only for `Inner` joins with no condition; an unconditioned `LeftOuter` join falls back to `BroadcastNestedLoopJoin`, which tries to broadcast the right side. When the right relation is large, the broadcast either OOMs or exceeds `spark.sql.autoBroadcastJoinThreshold` and the planner is left with no good option. `CartesianProductExec` partitions both sides and streams pairs, so it scales naturally with right-side size. Respecting the user's `INNER` join type re-enables this strategy for the common `INNER NEAREST BY` case. - It also makes the EXPLAIN output misleading (shows `LeftOuter` even though the user wrote `INNER`). - For `INNER` with an empty right side, the old plan generates one row per left input and then filters them away via `Generate(outer = false)` and the `size(matches) > 0` filter -- extra work that respecting `joinType` avoids at the source. ### Does this PR introduce _any_ user-facing change? No change in query results. `EXPLAIN` output for `INNER NEAREST BY` queries now shows `Inner` rather than `LeftOuter` for the synthetic join node, and the physical plan for such queries can now use `CartesianProductExec` instead of `BroadcastNestedLoopJoin` when the right relation is too large to broadcast. ### How was this patch tested? - `RewriteNearestByJoinSuite`: `expectedRewrite` now takes a `joinType: JoinType` and the existing tests (similarity/distance x inner/leftouter, EXACT, boundary k, self-join, nondeterministic ranking) assert the synthetic join matches the user's join type. - Golden file `sql-tests/results/join-nearest-by.sql.out` ### Was this patch authored or co-authored using generative AI tooling? Coauthored-by: Claude Code (Opus 4.7), human-reviewed and tested Closes apache#56023 from zhidongqu-db/respect-nn-join-type. Authored-by: Zero Qu <zhidong.qu@databricks.com> Signed-off-by: Daniel Tenedorio <daniel.tenedorio@databricks.com>
zhidongqu-db
changed the title
zhidongqu-db
changed the title
Contributor
|
thanks, merging to 4.x! |
cloud-fan
pushed a commit
that referenced
this pull request
May 22, 2026
…e synthetic join ### What changes were proposed in this pull request? backport 0c83226 to branch-4.x ### Why are the changes needed? The original implementation hardcoded the synthetic join to `LeftOuter` and justified it on the grounds that `LEFT OUTER` and `INNER` are equivalent for an unconditioned join when the right side is non-empty, and `Generate(outer = false)` would drop unwanted rows for `INNER` when right is empty. That reasoning holds for correctness but has a major performance cost: - **`INNER NEAREST BY` cannot be planned as a Cartesian product.** Spark's strategy picks `CartesianProductExec` only for `Inner` joins with no condition; an unconditioned `LeftOuter` join falls back to `BroadcastNestedLoopJoin`, which tries to broadcast the right side. When the right relation is large, the broadcast either OOMs or exceeds `spark.sql.autoBroadcastJoinThreshold` and the planner is left with no good option. `CartesianProductExec` partitions both sides and streams pairs, so it scales naturally with right-side size. Respecting the user's `INNER` join type re-enables this strategy for the common `INNER NEAREST BY` case. - It also makes the EXPLAIN output misleading (shows `LeftOuter` even though the user wrote `INNER`). - For `INNER` with an empty right side, the old plan generates one row per left input and then filters them away via `Generate(outer = false)` and the `size(matches) > 0` filter -- extra work that respecting `joinType` avoids at the source. ### Does this PR introduce _any_ user-facing change? No change in query results. `EXPLAIN` output for `INNER NEAREST BY` queries now shows `Inner` rather than `LeftOuter` for the synthetic join node, and the physical plan for such queries can now use `CartesianProductExec` instead of `BroadcastNestedLoopJoin` when the right relation is too large to broadcast. ### How was this patch tested? - `RewriteNearestByJoinSuite`: `expectedRewrite` now takes a `joinType: JoinType` and the existing tests (similarity/distance x inner/leftouter, EXACT, boundary k, self-join, nondeterministic ranking) assert the synthetic join matches the user's join type. - Golden file `sql-tests/results/join-nearest-by.sql.out` ### Was this patch authored or co-authored using generative AI tooling? Coauthored-by: Claude Code (Opus 4.7), human-reviewed and tested Closes #56052 from zhidongqu-db/nearest-rewrite-fix-branch-4.x. Authored-by: Zero Qu <zhidong.qu@databricks.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
backport 0c83226 to branch-4.x
Why are the changes needed?
The original implementation hardcoded the synthetic join to
LeftOuterand justified it on the grounds thatLEFT OUTERandINNERare equivalent for an unconditioned join when the right side is non-empty, andGenerate(outer = false)would drop unwanted rows forINNERwhen right is empty.That reasoning holds for correctness but has a major performance cost:
INNER NEAREST BYcannot be planned as a Cartesian product. Spark's strategy picksCartesianProductExeconly forInnerjoins with no condition; an unconditionedLeftOuterjoin falls back toBroadcastNestedLoopJoin, which tries to broadcast the right side. When the right relation is large, the broadcast either OOMs or exceedsspark.sql.autoBroadcastJoinThresholdand the planner is left with no good option.CartesianProductExecpartitions both sides and streams pairs, so it scales naturally with right-side size. Respecting the user'sINNERjoin type re-enables this strategy for the commonINNER NEAREST BYcase.LeftOutereven though the user wroteINNER).INNERwith an empty right side, the old plan generates one row per left input and then filters them away viaGenerate(outer = false)and thesize(matches) > 0filter -- extra work that respectingjoinTypeavoids at the source.Does this PR introduce any user-facing change?
No change in query results.
EXPLAINoutput forINNER NEAREST BYqueries now showsInnerrather thanLeftOuterfor the synthetic join node, and the physical plan for such queries can now useCartesianProductExecinstead ofBroadcastNestedLoopJoinwhen the right relation is too large to broadcast.How was this patch tested?
RewriteNearestByJoinSuite:expectedRewritenow takes ajoinType: JoinTypeand the existing tests (similarity/distance x inner/leftouter, EXACT, boundary k, self-join, nondeterministic ranking) assert the synthetic join matches the user's join type.sql-tests/results/join-nearest-by.sql.outWas this patch authored or co-authored using generative AI tooling?
Coauthored-by: Claude Code (Opus 4.7), human-reviewed and tested