Repository navigation
Conversation
Spark's OptimizeSkewedJoin runs in AQE's query stage preparation rules, before CometRule, and puts an AQEShuffleReadExec between each Comet shuffle stage and the join it splits. CometExecRule converted the stage into a native input but left the read as it was, so the join, the sort under a sort-merge join and everything above the join in its stage ran in Spark, with no fallback reason. Convert such a read into a native shuffle input while AQE prepares a plan, the way the stage itself is converted, and let CometExchangeSink read it through ShuffleScan. The shuffle input then reads only the partitions the read specifies, as it already does for the reads that AQE adds after conversion.
With spark.sql.adaptive.forceOptimizeSkewedJoin, AQE keeps a skew split that adds a shuffle only because SimpleCostEvaluator counts the split join as a skew join, and it counts only Spark's ShuffledJoin. Converting the split read made the join native, so AQE dropped the split, and SPARK-33832 in AdaptiveQueryExecSuite found 2 shuffles instead of 3. While that setting is on, the read and the join now stay on Spark.
andygrove
left a comment
There was a problem hiding this comment.
Thanks @comphead. This is a well-scoped fix, and the benchmark numbers make a clear case for it. Converting the read once the stage under it is already a CometSinkPlaceHolder fits how the neighbouring cases work. The new test checks the final plan, every query stage plan, and that both join inputs read their split partitions as native shuffle inputs. That is the right bar.
Approving. Two small asks below that I'd like you to look at, but neither should hold up the merge.
Nit, optional: the ADAPTIVE_FORCE_OPTIMIZE_SKEWED_JOIN guard and the test AQE forced skew join split that adds a shuffle stays a skew join are not in the PR description. The code comment explains the reason well. Could you add a sentence or two to the description so the squashed commit message covers it? When the guard applies, the read falls into the case op branch that lists AQEShuffleReadExec as never replaced, so no fallback reason is recorded. The problem this PR fixes for the default case was exactly that silent fallback. Could the guard path call withInfo so the reason shows in extended explain?
Nit, optional: AQE skew join split stays native runs an inner join with the left side skewed. Spark splits different sides for different join types, so could you add a case for a skewed right side (RightOuter) and one for LeftOuter or a left semi join? Those are the shapes where a native join over split reads could disagree with Spark. I'd also be curious about a case where one join input is a Comet shuffle stage and the other falls back, to confirm the converted read is unwrapped cleanly in that plan.
Which issue does this PR close?
Closes #6530.
The skewed-join benchmark from #6528 found it. #6528 stays open.
Rationale for this change
With AQE on, Spark's
OptimizeSkewedJoinsplits a skewed partition of a sort-merge or shuffled hash join by putting anAQEShuffleReadExecbetween each shuffle stage and the join. It runs in AQE's query stage preparation rules, beforeCometRule.CometExecRuleconverted a Comet shuffle stage into a native input but left the read as it was. So the join, the sort under a sort-merge join, and everything above the join in its stage ran in Spark, and no fallback reason was recorded. AQE and its skew join handling are on by default, so a skewed join that AQE splits lost Comet for its whole stage.What changes are included in this PR?
CometExecRule.convertNodeconverts anAQEShuffleReadExecover a Comet shuffle stage into a native shuffle input, the way it converts the stage itself. By the timetransformUpreaches the read, the stage under it is already aCometSinkPlaceHolder, so the case matches that and converts a read over the bare stage. It applies only while AQE prepares a plan, which is the only time a read is in place before the operators above it are converted. The reads that AQE adds later, for coalescing, local reads and rebalance splits, sit under operators that are already native, and that path is unchanged.CometExchangeSink.shouldUseShuffleScanaccepts that read, so it becomes aShuffleScanleaf when shuffle direct read is on, and aScanotherwise.Execution needs no change. It already treats an
AQEShuffleReadExecas a shuffle input, as it does for coalesced reads, andCometShuffledRowRDDreads all four partition spec types. The Comet joins takerequiredChildDistributionfrom the Spark join they replace, which isUnspecifiedDistributionfor a skew join, and computeoutputPartitioningthe way Spark'sShuffledJoindoes. dwsmith1983 described the same conversion in a comment on #6530.How are these changes tested?
CometNativeShuffleSuite: a join with 90% of the left rows on one key and a partial aggregate above it, for theMERGEandSHUFFLE_HASHhints, with partition coalescing on and off, each with shuffle direct read on and off. They check that AQE split the partition, that the final plan and every query stage plan hold only Comet operators, and that both join inputs read their split partitions as native shuffle inputs. Without the change all four fail with the plan from the issue,SortMergeJoin(skew=true)overColumnarToRowoverAQEShuffleRead skewed.CometNativeShuffleSuite(87 tests),CometExecRuleSuiteandCometJoinSuite(239 tests) pass, and the Spark 3.5 strict-warnings build passes. Spark's SQL suites have not run yet. Spark's own skew join test inAdaptiveQueryExecSuiteis ignored for Comet indev/diffs.CometSkewedJoinBenchmarkfrom perf: Comet hash joins use more task time than Spark (skewed SHJ with nested payload, BHJ) #6528, which is not part of this PR, with 90% of the fact rows on one key, Spark 4.1.3,local[8], median of 9 rounds, on an Apple M3 Max. Comet's total task time with AQE on, in ms, and its speedup over Spark with AQE on in the same run:On main all eight of these plans fell back under AQE. With this PR none does, and turning AQE on no longer adds task time for Comet: its AQE-on time divided by its AQE-off time went from 1.06-1.62 to 0.66-1.00. The two builds ran one after the other. The broadcast join and window cases, which this PR does not change, were up to 10% slower in the run with this PR, so a difference of that size between the two columns is noise.