Repository navigation
fix: keep a DPP build side's storage-partitioned join free of exchanges - #7662
snmvaughan wants to merge 1 commit into
Conversation
AQE plans a DPP subquery's converted build side again, and EnsureRequirements then puts an exchange under each side of a Comet join that Spark planned as a storage-partitioned join, because the join declares the original operator's required distribution since apache#7231. AQE has no logical link for those exchanges and asserts. Spark only accepts a key-grouped join as co-partitioned when the parent is its own SortMergeJoinExec or ShuffledHashJoinExec, and in Spark 4.1 when every leaf is also a KeyGroupedPartitionedScan. Reporting the scan's partitioning therefore cannot help a join. Declare no required distribution on CometSortMergeJoinExec and CometHashJoinExec when both sides are storage-partitioned. An aggregate above such a join does need the scan's partitioning, or the final aggregate gets an exchange inside an already serialized native block. So CometIcebergNativeScanExec now reports whatever the BatchScanExec it replaces reports, which generalizes apache#7424 from SinglePartition. Add a test for the join and one for an aggregate above it. Closes apache#7441.
mbutrovich
left a comment
There was a problem hiding this comment.
Thanks @snmvaughan for tracking this down, and for the clear explanation of why the scan change alone can't fix the join. #7510 brought #7231 to branch-1.2, so branch-1.2 has this regression too, and I'll add the backport-1.2 label.
| // Spark's own scans. Comet's non-native V2 scan, CometBatchScanExec, is a leaf under a | ||
| // CometScanWrapper that this does not unwrap, so a join over it keeps its requirement. |
There was a problem hiding this comment.
This comment describes a path that no scan takes today. CometBatchScanExec is only created for CSV, Text, and Iceberg (CometScanRule.scala L415, L574, L1345), and CometExecRule handles each of those in its own case before the CometScanWrapper one (CometExecRule.scala L470-L483). An Iceberg scan that the native conversion declines stays a bare CometBatchScanExec. That class extends DataSourceV2ScanExecBase, so it matches the case below this comment and isn't excluded.
Could we reword the comment to match? The paragraph in the PR description about Comet's non-native Iceberg reader has the same problem, so it could go too.
| // Spark's own scans. Comet's non-native V2 scan, CometBatchScanExec, is a leaf under a | |
| // CometScanWrapper that this does not unwrap, so a join over it keeps its requirement. | |
| // Spark's own scans, and CometBatchScanExec, which extends DataSourceV2ScanExecBase. |
| val query = | ||
| """SELECT /*+ BROADCAST(d) */ f.store_id, f.amount, d.name | ||
| |FROM aqe_cat.db.spj_dpp_fact f JOIN ( | ||
| | SELECT /*+ MERGE(a, b) */ a.id, b.name |
There was a problem hiding this comment.
CometHashJoinExec gets the new trait too, but both tests use MERGE. Could we also run them with SHUFFLE_HASH(a, b) and assert that the plan has a CometHashJoinExec? I tried that variant of the join test locally on Spark 4.1. It fails with the same AssertionError in setLogicalLinkForNewQueryStage when operators.scala is reverted to main, and passes with this PR.
The two tests also share all of their setup, including the table names. When I ran them without the operators.scala change, the aggregate test failed with TABLE_OR_VIEW_ALREADY_EXISTS on spj_dpp_fact, because the join test failed before its DROP TABLE lines ran. So a failure in one test hides the result of the other. Could we use one setup with the drops in a finally, and generate the cases with a for over the two queries and the two hints, like the test at L6170-L6174? That would also give the aggregate query the no-exchange assertion that only the join query has now.
comphead
left a comment
There was a problem hiding this comment.
I checked the EnsureRequirements reasoning against Spark 3.5.8, 4.0.1 and 4.1.1. checkKeyGroupCompatible only accepts Spark's SortMergeJoinExec or ShuffledHashJoinExec as the parent, so dropping the requirement on the Comet join is the change that can work. serializePartitions builds from inputRDD, which pads pruned key groups, so the reported partition count matches what runs.
| originalPlan.outputPartitioning match { | ||
| case _: UnknownPartitioning => UnknownPartitioning(0) | ||
| case reported => reported | ||
| } |
There was a problem hiding this comment.
When BatchScanExec reports no partitioning, it returns UnknownPartitioning(0) from SparkPlan.outputPartitioning on every version I checked (v3.4.3, v3.5.8, v4.0.1, v4.1.1, v4.2.1). So this match always returns its input. CometNativeScanExec.outputPartitioning returns originalPlan.outputPartitioning directly for a bucketed scan, and this could do the same:
| originalPlan.outputPartitioning match { | |
| case _: UnknownPartitioning => UnknownPartitioning(0) | |
| case reported => reported | |
| } | |
| originalPlan.outputPartitioning |
Which issue does this PR close?
Closes #7441.
Rationale for this change
With AQE on, a query fails with an
AssertionErrorinAdaptiveSparkPlanExec.setLogicalLinkForNewQueryStagewhen the build side of a DPP broadcast contains a storage-partitioned join over native Iceberg scans.Comet builds the DPP subquery's plan by wrapping the build side it already converted in a new
AdaptiveSparkPlanExec, and that constructor runsEnsureRequirementsover it again. Since #7231 the Comet joins, aggregates and windows declare Spark's required child distribution, andEnsureRequirementsputs an exchange under each side of the join. AQE has no logical link for those exchanges.The issue suggests having the native scan report the key-grouped partitioning of the scan it replaces, as #7424 does for
SinglePartition. That is not enough for a join. In Spark 3.5 through 4.2,EnsureRequirementsonly treats two key-grouped children as co-partitioned when the parent is Spark's ownSortMergeJoinExecorShuffledHashJoinExec(checkKeyGroupCompatible), and in 4.1 every leaf below them must also be aKeyGroupedPartitionedScan(canPushDownSPJParamsToScan). ACometSortMergeJoinExecoverCometIcebergNativeScanExecs is neither, so it finds no usable shuffle spec and shuffles both sides whatever the scans report. With only the scan change, the new join test fails in the same way.An aggregate has one child, so
EnsureRequirementsonly asks whether that child's partitioning satisfies the distribution. An aggregate above a storage-partitioned join does need the scans to report their partitioning. Without it the final aggregate gets an exchange between itself and the partial aggregate, inside a native block that was already serialized, and the native side fails withColumn index 1 is out of bound.What changes are included in this PR?
CometSortMergeJoinExecandCometHashJoinExecdeclare no required child distribution when both sides are storage-partitioned, meaning every leaf is a key-grouped scan and there is no shuffle above it. No shuffle sits below such a join for AQE to split, so the requirement from fix: declare required child distribution on aggregate, window and join operators #7231 protects nothing there. Every other join still returns the original operator's requirement.CometExec.isStoragePartitionedchecks the join's current children, because conversion works bottom-up andoriginalPlan.childrenare already Comet nodes. It usesDataSourceV2ScanExecBase.keyGroupedPartitioning, which every supported Spark version declares, so it needs no shim even though Spark 4.2 renamedKeyGroupedPartitioningtoKeyedPartitioning.CometIcebergNativeScanExec.outputPartitioningreports whatever theBatchScanExecit replaces reports, andUnknownPartitioning(0)when that is unknown. This generalizes fix: report SinglePartition from a native Iceberg scan that replaces one #7424. Reading it resolves no DPP subquery, because in Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0BatchScanExec.outputPartitioningis built frominputPartitionsand onlyinputRDDusesfilteredPartitions.CometIcebergNativeSuite. One is the reproduction from the issue, with an assertion that neither the plan nor the subquery's plan has an exchange. The other puts an aggregate above the storage-partitioned join.How are these changes tested?
On Spark 4.1:
Column index 1 is out of bound, and passes with it.AQE DPPtests inCometIcebergNativeSuitepass.The change compiles on Spark 3.5 with Scala 2.12, and its main sources compile on Spark 4.2. I have not run the tests on Spark 3.5 or 4.0.
A join over Comet's non-native Iceberg reader is not covered.
CometExecRulewrapsCometBatchScanExecin aCometScanWrapperleaf, whichisStoragePartitioneddoes not unwrap, so that join keeps the requirement from #7231. The tests also use onlyCometSortMergeJoinExec, through theMERGEhint.CometHashJoinExecshares the trait but has no test.