Repository navigation
Conversation
|
cc @andygrove @comphead this fixes an issue related to broadcast join that I mentioned in the sync. Please check if it makes sense. Also cc @viirya for review |
viirya
left a comment
There was a problem hiding this comment.
Thanks @sunchao. The fix looks right to me. Copying the direct and inherited tags separately, without promoting or propagating, puts the propagation boundaries where Spark places them for its own operators. Keeping an inherited stage only when it wraps the operator's original logical node by identity is a tight enough condition. The LogicalPlanTagInSparkPlanSuite change is justified too. The old matcher passed only because Comet promoted the scan's inherited tag to a direct one.
comphead
left a comment
There was a problem hiding this comment.
Thanks @sunchao, the root cause analysis and the end to end regression are clear. I checked the new guard against AdaptiveSparkPlanExec.replaceWithQueryStagesInLogicalPlan and SparkStrategies.plan at v4.1.1 and the identity comparison matches how AQE finds the node to replace. I have not run any of the suites.
andygrove
left a comment
There was a problem hiding this comment.
The PR backlog is huge and the maintainers don't have the bandwidth to review them all, so here is an automated review. Feel free to ignore it.
I found no blocking concerns. The change to CometExecRule keeps a reused operator's AQE stage when the link is direct or when the inherited stage wraps the operator's original node by identity. The regression test in CometExecSuite shows the sort-merge to broadcast switch that this fixes. I read the new guard against the existing LogicalQueryStage handling and the stageWraps recursion through nested stages. It matches the intent described in the code comments, and the earlier review threads on helper sharing, the restored Spark links and the nested-stage test coverage look resolved at this head.
I compared the touched code with branch-1.1. The only behavior difference from the release is the intended one. An operator whose inherited link points at a stage wrapping its original node now keeps that stage instead of being relinked to the original logical node. The restore path also stops promoting an inherited tag to a direct tag and stops pushing tags into children. The tag-copy semantics in MakeDateEvaluation and the projection rebuild are unchanged because they pass clearMissing = false. I did not find a path where Comet would pick a different physical plan from the release other than the intended broadcast case.
The remaining open question about setLogicalLink on the exchange branches and in the other planner rules was raised by comphead and is already tracked under #6034, so I have nothing to add there.
Copy both logical tags on shuffle and broadcast exchanges without propagating the exchange link into children. Exercise the inherited sort link on a single-partition shuffle under a nested ordered limit, and share the original-tag fixture across the three AQE planner tests.
Which issue does this PR close?
Addresses the inherited logical-link case in #6034. The broader audit remains open.
Rationale for this change
Spark’s adaptive query execution (AQE) can switch a join to broadcast when an aggregation makes the build side small enough. With a mixed
COUNT(*)andCOUNT(DISTINCT ...), the first shuffle can be above the broadcast threshold and the second well below it. Comet can lose the link to the second stage and keep a sort-merge join.What changes are included in this PR?
Keep the current AQE stage on a reused Comet operator when the link is direct, or when the inherited stage wraps the operator’s original logical node. Compare nodes by identity, including through nested stage wrappers. When restoring the original link, copy its direct and inherited tags on the operator itself. Calling Spark’s
setLogicalLinkhere promotes an inherited tag to a direct one and propagates it into children that can already belong to a newer stage.The same helper restores the two tags on shuffle and broadcast exchanges from their original Spark operators. It clears each missing tag on the exchange itself, so AQE can look through an unlinked exchange to its child. A rebuilt projection or
make_datefallback uses the helper withclearMissing = falseto retain its own tag when the source has none.Spark’s logical-link test recognizes Comet projections and filters in a scan subtree. Its two existing matches now use Spark’s
ColumnarToRowTransitiontype, so the test recognizes either implementation and checks the direct tag on the scan subtree root.How are these changes tested?
The planner tests share an aggregate fixture with a direct, inherited, or absent tag on the original. They check repeated replanning, nested stages, and stale links. A Spark 4.x regression plans an ordered
LIMITunderSUMwith top-k disabled. Spark inserts a local sort above a single-partition shuffle. The test checks that the Comet shuffle inherits the exact logical sort object and has an empty direct tag. Spark sort insertionSpotless passed at d9f90a6a8. After merging
main, all four Spark SQL patches apply cleanly to their release tags (3.5.9, 4.0.4, 4.1.3, and 4.2.0). Each produces the sameLogicalPlanTagInSparkPlanSuitesource as the previously reviewed patch. Hosted CIBroadcast and scan tests before the exchange update
The SQL regression joins two synthetic ranges. Static broadcasting is disabled and AQE gets a 16 KiB broadcast threshold. On both stock Spark releases, only the planner rule changes between runs.
Both plans started without a broadcast join. On Spark 3.5.9, the fixed plan’s first build shuffle reported 196,096 bytes and the second reported 1,280 bytes. The comparison measures plan selection and result parity.
The two link tests and the SQL regression failed with the main rule and passed with the fix on both releases. Before the exchange update, the full
CometExecRuleSuiteand three AQE and exchange tests passed on Spark 3.5.9 and 4.1.3, with one Spark-4-only test skipped on 3.5.9. Spotless and Scalastyle passed for the changed Scala sources.All 150 cases in Spark’s
LogicalPlanTagInSparkPlanSuitepassed with the revised matcher on each version supported at the time (3.4.3, 3.5.9, 4.0.4, 4.1.3, and 4.2.0). I also tested the two plan shapes separately.Filtertag. Checking the child passes.