Skip to content

fix: keep joins native when AQE splits a skewed partition - #7663

Open
comphead wants to merge 3 commits into
apache:mainfrom
comphead:aqe-skew-join-native-6530
Open

comphead wants to merge 3 commits into
apache:mainfrom
comphead:aqe-skew-join-native-6530

Conversation

@comphead

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #6530.

The skewed-join benchmark from #6528 found it. #6528 stays open.

Rationale for this change

With AQE on, Spark's OptimizeSkewedJoin splits a skewed partition of a sort-merge or shuffled hash join by putting an AQEShuffleReadExec between each shuffle stage and the join. It runs in AQE's query stage preparation rules, before CometRule. CometExecRule converted a Comet shuffle stage into a native input but left the read as it was. So the join, the sort under a sort-merge join, and everything above the join in its stage ran in Spark, and no fallback reason was recorded. AQE and its skew join handling are on by default, so a skewed join that AQE splits lost Comet for its whole stage.

What changes are included in this PR?

  • CometExecRule.convertNode converts an AQEShuffleReadExec over a Comet shuffle stage into a native shuffle input, the way it converts the stage itself. By the time transformUp reaches the read, the stage under it is already a CometSinkPlaceHolder, so the case matches that and converts a read over the bare stage. It applies only while AQE prepares a plan, which is the only time a read is in place before the operators above it are converted. The reads that AQE adds later, for coalescing, local reads and rebalance splits, sit under operators that are already native, and that path is unchanged.
  • CometExchangeSink.shouldUseShuffleScan accepts that read, so it becomes a ShuffleScan leaf when shuffle direct read is on, and a Scan otherwise.

Execution needs no change. It already treats an AQEShuffleReadExec as a shuffle input, as it does for coalesced reads, and CometShuffledRowRDD reads all four partition spec types. The Comet joins take requiredChildDistribution from the Spark join they replace, which is UnspecifiedDistribution for a skew join, and compute outputPartitioning the way Spark's ShuffledJoin does. dwsmith1983 described the same conversion in a comment on #6530.

How are these changes tested?

  • New tests in CometNativeShuffleSuite: a join with 90% of the left rows on one key and a partial aggregate above it, for the MERGE and SHUFFLE_HASH hints, with partition coalescing on and off, each with shuffle direct read on and off. They check that AQE split the partition, that the final plan and every query stage plan hold only Comet operators, and that both join inputs read their split partitions as native shuffle inputs. Without the change all four fail with the plan from the issue, SortMergeJoin(skew=true) over ColumnarToRow over AQEShuffleRead skewed.
  • Locally on Spark 4.1: CometNativeShuffleSuite (87 tests), CometExecRuleSuite and CometJoinSuite (239 tests) pass, and the Spark 3.5 strict-warnings build passes. Spark's SQL suites have not run yet. Spark's own skew join test in AdaptiveQueryExecSuite is ignored for Comet in dev/diffs.
  • Benchmark: CometSkewedJoinBenchmark from perf: Comet hash joins use more task time than Spark (skewed SHJ with nested payload, BHJ) #6528, which is not part of this PR, with 90% of the fact rows on one key, Spark 4.1.3, local[8], median of 9 rounds, on an Apple M3 Max. Comet's total task time with AQE on, in ms, and its speedup over Spark with AQE on in the same run:
case main this PR speedup over Spark, main → this PR
SMJ then aggregate, flat payload 1047 527 1.64x → 3.79x
SMJ then aggregate, nested payload 1981 922 1.97x → 4.08x
SMJ, skewed side buffered, flat payload 675 582 2.65x → 3.20x
SMJ, skewed side buffered, nested payload 2125 1541 1.99x → 2.55x
SMJ, skewed side streamed, flat payload 644 531 2.70x → 3.57x
SMJ, skewed side streamed, nested payload 1645 1762 2.15x → 2.21x
SHJ, skewed side probed, flat payload 673 416 2.42x → 3.63x
SHJ, skewed side probed, nested payload 1634 1445 2.18x → 2.33x

On main all eight of these plans fell back under AQE. With this PR none does, and turning AQE on no longer adds task time for Comet: its AQE-on time divided by its AQE-off time went from 1.06-1.62 to 0.66-1.00. The two builds ran one after the other. The broadcast join and window cases, which this PR does not change, were up to 10% slower in the run with this PR, so a difference of that size between the two columns is noise.

Spark's OptimizeSkewedJoin runs in AQE's query stage preparation rules,
before CometRule, and puts an AQEShuffleReadExec between each Comet
shuffle stage and the join it splits. CometExecRule converted the stage
into a native input but left the read as it was, so the join, the sort
under a sort-merge join and everything above the join in its stage ran
in Spark, with no fallback reason.

Convert such a read into a native shuffle input while AQE prepares a
plan, the way the stage itself is converted, and let CometExchangeSink
read it through ShuffleScan. The shuffle input then reads only the
partitions the read specifies, as it already does for the reads that
AQE adds after conversion.
@github-actions github-actions Bot added bug Something isn't working area:writer Native Parquet writer labels Nov 16, 2026
comphead and others added 2 commits November 17, 2026 16:59
With spark.sql.adaptive.forceOptimizeSkewedJoin, AQE keeps a skew split
that adds a shuffle only because SimpleCostEvaluator counts the split
join as a skew join, and it counts only Spark's ShuffledJoin. Converting
the split read made the join native, so AQE dropped the split, and
SPARK-33832 in AdaptiveQueryExecSuite found 2 shuffles instead of 3.
While that setting is on, the read and the join now stay on Spark.
@andygrove andygrove added the run-all-spark-profiles Run the Comet test suites against every Spark profile on this pull request, ahead of the merge queue label Nov 19, 2026

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @comphead. This is a well-scoped fix, and the benchmark numbers make a clear case for it. Converting the read once the stage under it is already a CometSinkPlaceHolder fits how the neighbouring cases work. The new test checks the final plan, every query stage plan, and that both join inputs read their split partitions as native shuffle inputs. That is the right bar.

Approving. Two small asks below that I'd like you to look at, but neither should hold up the merge.

Nit, optional: the ADAPTIVE_FORCE_OPTIMIZE_SKEWED_JOIN guard and the test AQE forced skew join split that adds a shuffle stays a skew join are not in the PR description. The code comment explains the reason well. Could you add a sentence or two to the description so the squashed commit message covers it? When the guard applies, the read falls into the case op branch that lists AQEShuffleReadExec as never replaced, so no fallback reason is recorded. The problem this PR fixes for the default case was exactly that silent fallback. Could the guard path call withInfo so the reason shows in extended explain?

Nit, optional: AQE skew join split stays native runs an inner join with the left side skewed. Spark splits different sides for different join types, so could you add a case for a skewed right side (RightOuter) and one for LeftOuter or a left semi join? Those are the shapes where a native join over split reads could disagree with Spark. I'd also be curious about a case where one join input is a Comet shuffle stage and the other falls back, to confirm the converted read is unwrapped cleanly in that plan.

@comphead
comphead added this pull request to the merge queue Nov 19, 2026
@comphead
comphead removed this pull request from the merge queue due to a manual request Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:writer Native Parquet writer bug Something isn't working run-all-spark-profiles Run the Comet test suites against every Spark profile on this pull request, ahead of the merge queue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AQE skew join split makes the join fall back to Spark with no fallback reason

2 participants