Skip to content

fix: preserve inherited AQE stage links on Comet operators - #7357

Open
sunchao wants to merge 7 commits into
apache:mainfrom
sunchao:sunchao/aqe-inherited-logical-links
Open

sunchao wants to merge 7 commits into
apache:mainfrom
sunchao:sunchao/aqe-inherited-logical-links

Conversation

@sunchao

@sunchao sunchao commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

Which issue does this PR close?

Addresses the inherited logical-link case in #6034. The broader audit remains open.

Rationale for this change

Spark’s adaptive query execution (AQE) can switch a join to broadcast when an aggregation makes the build side small enough. With a mixed COUNT(*) and COUNT(DISTINCT ...), the first shuffle can be above the broadcast threshold and the second well below it. Comet can lose the link to the second stage and keep a sort-merge join.

What changes are included in this PR?

Keep the current AQE stage on a reused Comet operator when the link is direct, or when the inherited stage wraps the operator’s original logical node. Compare nodes by identity, including through nested stage wrappers. When restoring the original link, copy its direct and inherited tags on the operator itself. Calling Spark’s setLogicalLink here promotes an inherited tag to a direct one and propagates it into children that can already belong to a newer stage.

The same helper restores the two tags on shuffle and broadcast exchanges from their original Spark operators. It clears each missing tag on the exchange itself, so AQE can look through an unlinked exchange to its child. A rebuilt projection or make_date fallback uses the helper with clearMissing = false to retain its own tag when the source has none.

Spark’s logical-link test recognizes Comet projections and filters in a scan subtree. Its two existing matches now use Spark’s ColumnarToRowTransition type, so the test recognizes either implementation and checks the direct tag on the scan subtree root.

How are these changes tested?

The planner tests share an aggregate fixture with a direct, inherited, or absent tag on the original. They check repeated replanning, nested stages, and stale links. A Spark 4.x regression plans an ordered LIMIT under SUM with top-k disabled. Spark inserts a local sort above a single-partition shuffle. The test checks that the Comet shuffle inherits the exact logical sort object and has an empty direct tag. Spark sort insertion

Spotless passed at d9f90a6a8. After merging main, all four Spark SQL patches apply cleanly to their release tags (3.5.9, 4.0.4, 4.1.3, and 4.2.0). Each produces the same LogicalPlanTagInSparkPlanSuite source as the previously reviewed patch. Hosted CI

Broadcast and scan tests before the exchange update

The SQL regression joins two synthetic ranges. Static broadcasting is disabled and AQE gets a 16 KiB broadcast threshold. On both stock Spark releases, only the planner rule changes between runs.

Spark Final join on Apache main Final join with this PR Correct result rows in both runs
3.5.9 Sort-merge Native broadcast hash 16 of 16
4.1.3 Sort-merge Native broadcast hash 16 of 16

Both plans started without a broadcast join. On Spark 3.5.9, the fixed plan’s first build shuffle reported 196,096 bytes and the second reported 1,280 bytes. The comparison measures plan selection and result parity.

The two link tests and the SQL regression failed with the main rule and passed with the fix on both releases. Before the exchange update, the full CometExecRuleSuite and three AQE and exchange tests passed on Spark 3.5.9 and 4.1.3, with one Spark-4-only test skipped on 3.5.9. Spotless and Scalastyle passed for the changed Scala sources.

All 150 cases in Spark’s LogicalPlanTagInSparkPlanSuite passed with the revised matcher on each version supported at the time (3.4.3, 3.5.9, 4.0.4, 4.1.3, and 4.2.0). I also tested the two plan shapes separately.

  • Without recognizing a Comet row transition inside the scan subtree, TPC-DS q9 visits a filter carrying only an inherited link and fails the direct-tag assertion.
  • On a filtered Parquet scan with the row transition at the root, the transition has no direct tag and its child has the direct Filter tag. Checking the child passes.

@github-actions github-actions Bot added the bug Something isn't working label Oct 30, 2026
@sunchao sunchao added the run-spark-4.1-tests Run the Spark 4.1 SQL tests on this pull request instead of waiting for the merge queue label Oct 30, 2026
@sunchao

sunchao commented Oct 30, 2026

Copy link
Copy Markdown
Member Author

cc @andygrove @comphead this fixes an issue related to broadcast join that I mentioned in the sync. Please check if it makes sense. Also cc @viirya for review

@sunchao sunchao added the run-spark-3.4-tests Run the Spark 3.4 SQL tests on this PR label Oct 31, 2026

@viirya viirya left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @sunchao. The fix looks right to me. Copying the direct and inherited tags separately, without promoting or propagating, puts the propagation boundaries where Spark places them for its own operators. Keeping an inherited stage only when it wraps the operator's original logical node by identity is a tight enough condition. The LogicalPlanTagInSparkPlanSuite change is justified too. The old matcher passed only because Comet promoted the scan's inherited tag to a direct one.

Comment thread dev/diffs/3.5.9.diff Resolved

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @sunchao, the root cause analysis and the end to end regression are clear. I checked the new guard against AdaptiveSparkPlanExec.replaceWithQueryStagesInLogicalPlan and SparkStrategies.plan at v4.1.1 and the identity comparison matches how AQE finds the node to replace. I have not run any of the suites.

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR backlog is huge and the maintainers don't have the bandwidth to review them all, so here is an automated review. Feel free to ignore it.

I found no blocking concerns. The change to CometExecRule keeps a reused operator's AQE stage when the link is direct or when the inherited stage wraps the operator's original node by identity. The regression test in CometExecSuite shows the sort-merge to broadcast switch that this fixes. I read the new guard against the existing LogicalQueryStage handling and the stageWraps recursion through nested stages. It matches the intent described in the code comments, and the earlier review threads on helper sharing, the restored Spark links and the nested-stage test coverage look resolved at this head.

I compared the touched code with branch-1.1. The only behavior difference from the release is the intended one. An operator whose inherited link points at a stage wrapping its original node now keeps that stage instead of being relinked to the original logical node. The restore path also stops promoting an inherited tag to a direct tag and stops pushing tags into children. The tag-copy semantics in MakeDateEvaluation and the projection rebuild are unchanged because they pass clearMissing = false. I did not find a path where Comet would pick a different physical plan from the release other than the intended broadcast case.

The remaining open question about setLogicalLink on the exchange branches and in the other planner rules was raised by comphead and is already tracked under #6034, so I have nothing to add there.

Copy both logical tags on shuffle and broadcast exchanges without propagating the exchange link into children. Exercise the inherited sort link on a single-partition shuffle under a nested ordered limit, and share the original-tag fixture across the three AQE planner tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working run-spark-3.4-tests Run the Spark 3.4 SQL tests on this PR run-spark-4.1-tests Run the Spark 4.1 SQL tests on this pull request instead of waiting for the merge queue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants