Repository navigation
fix: fall back when an outer hash join's streamed order cannot be checked - #7126
Conversation
…cked An outer hash join that runs as DataFusion's right outer join keeps its unmatched probe rows in place only when the probe input declares an order. When the streamed side's reported order could not be checked, the serde declared none but still converted the join, so a sort Spark had already dropped above it was lost. This happened over a sorted localCheckpoint and over some caches written by an outer hash join. The join now falls back to Spark when that order reaches it through an input from the JVM or through a nested loop join. Over native operators in the same plan it stays native, since DataFusion keeps the order it can see there.
sunchao
left a comment
There was a problem hiding this comment.
Reviewed the full five-file diff from d834665aeab2fcbe812f638ec9f3637b248d38c4 to ae849893e33a5642d3ce469ffa002c9875a99502. The PR is not a draft. The snapshot and live discussion contained no reviews, conversation comments, inline comments or review threads.
No introduced P1/P2 issues found within this review.
Summary
- Prior state and problem: An outer hash join could omit an unverifiable streamed-order declaration but still execute natively. DataFusion then moved unmatched rows, violating an ordering that Spark had already used to remove a sort. Checkpoints and certain cached joins exposed this.
- Design approach: Fall back for ordered
LeftOuter/BuildRightandRightOuter/BuildLeftjoins when the order cannot be verified across a JVM boundary or nested-loop join. Retain native execution for eligible chains within one native plan. - Correctness: Traced both hash-join converters, cache and AQE traversal, broadcast fallback handling, and DataFusion’s ordered-probe behavior.
countInPlan = truepreserves the previous serialization check, so the change does not introduce new trusted ordering declarations. All 19 focused tests passed, including all six added regressions. - Compatibility analysis: Compared Spark
HashJoin,ShuffledHashJoinExec,InMemoryTableScanExecandRemoveRedundantSortssources for 3.4.3, 3.5.9, 4.0.4, 4.1.3 and 4.2.0. Their relevant ordering guarantees support this guard. Compared the affected converter withbranch-1.1at89c729a9697467e8f8d89e3eadf1797a75e2328c: moving the affected queries to Spark is an intended correctness change. No new configuration or protocol compatibility issue was identified. - Key design decisions: Resetting
countInPlanat caches and scan wrappers prevents an ordering inferred inside another native plan from being trusted across that boundary. Exempting in-plan hash joins avoids unnecessarily disabling chains whose ordering DataFusion already sees. - Implementation sketch:
keepsStreamedOrdergates the new early fallback. The existing recursive predicate gains a mode parameter, while the declaration path retains its previous behavior. Tests cover checkpoints, both cache formats, AQE, native inner joins, windows, filters and nested-loop joins. Both relevant documentation pages are updated. - Performance: The additional traversal occurs during planning for ordered outer-join candidates, with no new per-row native work. Affected joins intentionally incur Spark execution and transition costs. The passing chain tests confirm that representative native pipelines remain eligible. No unintended performance regression meeting the P1/P2 bar was established.
- Design: Keeping the guard in the shared hash-join converter applies the policy consistently to broadcast and shuffled joins. Falling back also avoids falsely declaring an order that downstream aggregation could trust.
- Abstraction & complexity: Reusing one traversal with a named Boolean separates fallback eligibility from declaration safety without adding another ordering framework. The boundary resets and paired regression tests make that distinction concrete. No necessary simplification meeting the reporting bar was identified.
- Behavioral changes worth calling out: Ordered checkpoints and unverifiable cached outputs now cause the outer join to execute in Spark. Eligible native window/filter/join chains remain native. Falling back above a nested-loop join does not repair rows that join already reordered, as documented in existing issue #6898.
- Suggested improvements: No additional code changes meeting the P1/P2 reporting bar were identified. Exact-head CI and broader Spark SQL validation remain outstanding.
Routed skills: Used .ai/skills/review-comet-pr/SKILL.md and the operator contributor guide. No sibling skill applies to this operator-only planning change.
Exact-head CI: Comet CI (run), Check PR Title and CodeQL report action_required. The labeling check passed. There is no successful build/test CI verdict for the reviewed SHA.
Validation: Native build passed with --no-default-features. JVM production and test compilation passed using a temporary JDK 21 and cached Maven dependencies. On Spark 4.1.3, the focused outer-hash-join selection ran 19 tests across CometJoinSuite and CometInMemoryCacheSuite: 19 passed, none failed, aborted or skipped. Other Spark profiles were checked against source but not executed. Full Spark SQL suites, optional HDFS/Delta features and performance benchmarks were not run. Maven formatting/style checks were skipped. Project source remained unchanged.
andygrove
left a comment
There was a problem hiding this comment.
Thanks @dwsmith1983. This closes the hole that #6835 left, and I like that the fix is a fallback rather than declaring an order we cannot check. The countInPlan flag keeps the order the serde sends exactly as before, so the PR cannot start declaring an order it used to withhold. I also like that chains of outer joins inside one native plan stay native, and that the tests compare against Spark and check per-partition order.
I compared against branch-1.1. That branch has none of this order logic, so an outer hash join over an ordered checkpoint, cache or nested loop join ran natively there and could return unmatched rows out of order. Moving those queries to Spark is an intended correctness fix, and I found no unintended change.
I have two questions.
-
In
operators.scala, the comment oncase _: CometBroadcastNestedLoopJoinExec => trueonly explains the one-row build case. The PR description says that with two or more build rows the fallback costs the native join and gains nothing until #6898 is fixed. Could the comment link #6898 and say this case can go once that is fixed? That way the cost has an owner. -
The new test in
CometInMemoryCacheSuitecovers a broadcastLEFT JOINonly. Did you try theRightOuterwithBuildLeftshape over a cache, in either cache format? If it goes through the same code path, a short note in the test would be enough.
CI has not run on this PR yet. The Comet CI job is waiting for a maintainer to approve the workflow. Since this changes when a join falls back, it also needs the run-spark-4.1-tests and run-all-spark-profiles labels, because outputOrdering differs across Spark versions and the PR tier only runs the default profile.
The cache test now runs both shapes that run as DataFusion's right outer join, LeftOuter built on the right and RightOuter built on the left, in both cache formats. The nested loop join case notes that it only gives up the native join for nothing until apache#6898 is fixed.
Done in 3e550f2.
Yes, and it goes through the same path. The cache test now runs both shapes in both cache formats, with AQE on and off. For the right outer join, the join that writes the cache runs natively as |
andygrove
left a comment
There was a problem hiding this comment.
Thanks @dwsmith1983. Both of my questions are answered. The #6898 link is in the comment on the nested loop join case, and the cache test now covers RightOuter with BuildLeft in both cache formats.
I checked that countInPlan = false is sound. The flag only drops the check for an outer join whose probe input declared no order, and the lowest join in a chain answers the same either way. So it either falls back or declares, and a join above it can trust the order DataFusion sees. The boundary resets at the cache scans and CometScanWrapper cover the places where an order crosses into another plan.
Approving. Comet CI is still waiting for a maintainer to approve the workflows, so the Spark profile jobs should go green before this merges.
comphead
left a comment
There was a problem hiding this comment.
The two-pass walk keeps the old behavior for the declared case and the thread has settled the countInPlan soundness and the labels. Three comments.
| if (keepsStreamedOrder && mayReportFalseOrder(streamed, countInPlan = false)) { | ||
| withFallbackReason( | ||
| join, | ||
| "the streamed side's sort order, which this outer join has to keep, cannot be " + | ||
| "checked, so it is not declared natively") | ||
| return None |
There was a problem hiding this comment.
This reason is shown in the explain output for a join that now stays in Spark, but "so it is not declared natively" describes the old behavior, where the join stayed native and sent no order. Could it say what happens now, for example "cannot be checked, so the join stays in Spark"? The same literal is repeated in four tests (CometInMemoryCacheSuite and three places in CometJoinSuite), so a shared constant would keep them in step.
There was a problem hiding this comment.
Changed in 01fdf01. The reason now says "cannot be checked, so the join stays in Spark", and it lives in one constant, CometHashJoin.uncheckedStreamedOrderReason, which the four tests use.
| Where Comet cannot check the order Spark reports, it declares nothing. If that order reaches the | ||
| outer join through an input from the JVM, such as an RDD checkpoint or some caches written by an | ||
| outer hash join whose own probe input was not a scan | ||
| ([#6922](https://github.com/apache/datafusion-comet/issues/6922)), or through a nested | ||
| loop join, the join falls back to Spark, which keeps the rows in the order they arrive. The output | ||
| of a nested loop join can already be out of order when it reaches the join | ||
| ([#6898](https://github.com/apache/datafusion-comet/issues/6898)). A join over other native | ||
| operators in the same plan does not fall back for its order, since DataFusion keeps the order it | ||
| can see there. |
There was a problem hiding this comment.
This paragraph opens with "it declares nothing" and only later says the join falls back, so a reader cannot tell when "declares nothing" means native and when it means a fallback. Leading with the fallback rule and then giving the in-plan exception would be clearer. The RDD checkpoint example also reaches a native join only with spark.comet.convert.rdd.enabled=true, which defaults to false. Without it the checkpoint is a plain Spark child and the join stays in Spark with or without this change. The tests set the config, so the doc could say so.
There was a problem hiding this comment.
Reworked in 01fdf01. The paragraph now leads with when the join falls back, notes that the checkpoint example needs spark.comet.convert.rdd.enabled=true, and then gives the exception: an order that comes only from another native hash join in the same plan keeps the join native. I narrowed that from "native operators" because a nested loop join in the same plan still makes it fall back.
| } | ||
| } | ||
|
|
||
| test("outer hash joins over a checkpointed streamed side fall back and keep its order") { |
There was a problem hiding this comment.
outer hash joins over a checkpointed streamed side fall back and keep its order and an outer hash join over a native inner join over a checkpoint falls back (line 2775) repeat the same conf block and the same range(2000)...localCheckpoint setup. A small withOrderedCheckpoint { streamed => ... } helper would remove about 20 lines.
There was a problem hiding this comment.
Done in 01fdf01 with withOrderedCheckpoint. It takes the checkpoint with AQE off, and the first test still runs both AQE modes over it.
The fallback reason still described the old behavior, where the join stayed native and declared no order. It now says the join stays in Spark, and the tests share it through one constant. The compatibility guide leads with when the join falls back and notes that a checkpoint reaches a native join only with spark.comet.convert.rdd.enabled. The two checkpoint tests share their setup.
…is trusted Drop comments that restate the code or a test's name, and shorten the rest to why an order from the JVM or a nested loop join can't be used.
…cannot be checked (#7126), declare the input order of a native window over a sorted cache (#7256) and sort a Spark window's doubted input by the whole order it reports (#7555) (#7642) * fix: fall back when an outer hash join's streamed order cannot be checked (#7126) * fix: fall back when an outer hash join's streamed order cannot be checked An outer hash join that runs as DataFusion's right outer join keeps its unmatched probe rows in place only when the probe input declares an order. When the streamed side's reported order could not be checked, the serde declared none but still converted the join, so a sort Spark had already dropped above it was lost. This happened over a sorted localCheckpoint and over some caches written by an outer hash join. The join now falls back to Spark when that order reaches it through an input from the JVM or through a nested loop join. Over native operators in the same plan it stays native, since DataFusion keeps the order it can see there. * test: cover a right outer join over a cache of a right outer join The cache test now runs both shapes that run as DataFusion's right outer join, LeftOuter built on the right and RightOuter built on the left, in both cache formats. The nested loop join case notes that it only gives up the native join for nothing until #6898 is fixed. * fix: say that an outer hash join with an unchecked order stays in Spark The fallback reason still described the old behavior, where the join stayed native and declared no order. It now says the join stays in Spark, and the tests share it through one constant. The compatibility guide leads with when the join falls back and notes that a checkpoint reaches a native join only with spark.comet.convert.rdd.enabled. The two checkpoint tests share their setup. * docs: keep only the comments that explain when an outer join's order is trusted Drop comments that restate the code or a test's name, and shorten the rest to why an order from the JVM or a nested loop join can't be used. (cherry picked from commit c56f6a0) * fix: declare the input order of a native window over a sorted cache (#7256) * fix: declare the input order of a native window over a sorted cache Spark puts no sort below a window when its input reports the order the window needs, a relation sorted before it was cached for instance. When those rows enter the native plan from the JVM, the ScanExec they arrive through declares no order, and the native window failed the task with "Expects PARTITION BY expression to be ordered". Send the window's required ordering in a new input_sort_orders field on the Window proto when the child reports an order that satisfies it and mayReportFalseOrder trusts that order. The planner declares it on the window's input with with_declared_order, as the hash join does for its probe input. A key that Comet cannot express, or that holds a collated string, sends the window back to Spark. Closes #7223. * refactor: build the window's input order only where it is used - Send no input order when a native sort below the window declares its own. - Build the order only once the window's own expressions have serialized, so a window that falls back for another reason does not walk the plan or record an extra reason. - Map a collated key straight to None instead of signalling it with an empty list. - Drop the empty-list branch in WindowBuilder, as with_declared_order already leaves the plan unchanged for no orders. * chore: trim comments and drop a window test the hash join test already covers * test: cover WindowAggExec, nulls, descending keys and the collated fallback From review: - The cache test also runs a frame up to UNBOUNDED FOLLOWING, which runs in WindowAggExec, over a nullable partition key and a descending order key, so the declared order has to match the cache's. - The planner test covers both window operators through the partition order lookup they share, since WindowAggExec fails only when its stream starts, not in partition_by_sort_keys. - A Spark 4.x test checks that a window over a cache sorted by a collated key stays in Spark for the new reason and returns Spark's rows. - The debugging guide's window snippet builds the operators over `input`. * fix: keep a stacked window's longer input order and declare it on empty buckets The window now sends the rest of the order its child reports past the required keys, up to the first that cannot be expressed, so a window above it in the same native plan sees the longer order. An empty bucket of a sorted bucketed scan is planned as an EmptyExec, which now counts as a source whose order holds. The cache test checks the declared columns, directions and null placements. * fix: declare a window's order next to a partial one its input declares Two aliases of one sorted column share an order in Spark, so a window over the window for one alias may need the other alias's order with no sort in between. with_declared_order now returns early only when the input's declared order already satisfies the one sent, and otherwise declares it next to the input's when keeps_source_order trusts the input. The serde checks only a suffix key's child against the child's output, since its same-order expressions are not sent and may name pruned columns. * refactor: send only a window's required order and merge overlapping tests with_declared_order extends a partial order at the upper window, so the serde no longer sends the keys a window's child reports past the ones the window requires. The two stacked-window tests become one over a single cache, the EmptyExec unit test folds into the extension test, and redundant assertions and comments are trimmed. * fix: declare only a window's partition keys and fall back on an unchecked order The native window checks only its partition keys' order, so the serde sends only those, and a float order key that Spark sorts with -0.0 and 0.0 as equal is never declared. When the order the child reports may be false past the native plan, the window stays in Spark, as the outer hash join does, instead of failing the task. The SortedInputExec doc and the operator guide now cover the window. * fix: keep a window's doubted order native, fall back on collated keys Falling back when the window's input order may be false handed the same rows to Spark's window, which trusts that order too and grouped them wrongly. The window again sends no order there, so the native window rejects its input as on main. A window whose partition or order key holds a collated string now stays in Spark, as WindowGroupLimit does, since the native window compares keys byte by byte. keeps_source_order accepts a right outer hash join whose probe declares an order, which is when DataFusion keeps its unmatched probe rows in place. * fix: fall back on a collated window key only over a trusted order Falling back hands the rows to Spark's window, which trusts the order the input reports. Over an order mayReportFalseOrder doubts, such as a native nested loop join's, that grouped the rows wrongly, so the window now stays native there, as on main, and the native window rejects the input. The outer join regression test partitions by k, which Spark reports through the alias a, so it needs the order declared next to the join's. * fix: sort the input of a Spark window over a doubted order Spark plans no sort below a window whose child reports the order it needs, and its window trusts that order. Over a Comet operator that may not keep it, such as a native nested loop join (#6898), a window that stays in Spark grouped rows out of order. CometExecRule now sorts that input again. With that, a window with a collated key falls back in every case, and so does a window with ORDER BY but no PARTITION BY over a doubted order, since the native window checks no order there. The compatibility guide lists collated keys among the window's fallbacks. * address review: run the nested loop join window test with AQE on, sort Python windows too (cherry picked from commit cd19831) * fix: sort a Spark window's doubted input by the whole order it reports (#7555) sortDoubtedWindowInputs sorted the input of a window that stays in Spark by the window's own keys only. Spark had already planned the operators above around the longer order the input reported, so a second window ordered by more keys, or a sortWithinPartitions it had removed, saw rows out of order. The sort now uses the whole reported order when it covers the window's keys. Closes #7553. (cherry picked from commit fd82542) --------- Co-authored-by: dustin <dwsmith1983@users.noreply.github.com> Co-authored-by: Oleks V <comphead@users.noreply.github.com>
Which issue does this PR close?
Closes #6922.
Rationale for this change
A native left outer join building on the right, or right outer join building on the left, keeps the streamed side's order only when the serde declares it. When Comet can't verify that order, it declares none but still converts the join, while Spark has already dropped the sort above. Unmatched rows then land after matched ones, as in both repros in #6922.
What changes are included in this PR?
How are these changes tested?
New tests in
CometJoinSuiteandCometInMemoryCacheSuite, with AQE on and off, cover shuffled and broadcast joins over sorted checkpoints and caches: each falls back and keeps every partition in order. Other cases check that chained outer joins within one native plan stay native and keep their order.