What is the problem the feature request solves?
CometExecRule repairs links on CometExec, CometShuffleExchangeExec, and CometBroadcastExchangeExec using originalPlan.logicalLink, or clears both logical tags when that link is absent. #5482 / #5483 establish that originalPlan is not always authoritative: Spark can reuse an existing physical root and assign a direct link to the current LogicalQueryStage.
#5483 preserves that direct link on CometExec. This issue tracks the separate question raised in the review: which remaining repairs are necessary, and can link ownership be made explicit or repair narrowed without breaking AQE?
Describe the potential solution
Audit initial conversion, repeated AQE replanning, and exchange/subquery creation to document when the original plan, the current physical node, or Spark's temporary stage link is authoritative. Distinguish fresh nodes from reused physical nodes and direct tags from inherited tags. Decide from that evidence whether to retain the current repair, narrow it, or move necessary assignments to node creation.
Any change should preserve:
Back the chosen contract with focused planner tests and end-to-end AQE/DPP coverage that checks results and actual native execution. This audit does not assume all repair is removable or establish another failing exchange case.
Additional context
Keep #5482 focused on its reproduced stale-stage-link defect. The broader design question should remain tracked after #5483 closes that issue.
Current logical-link repair.
What is the problem the feature request solves?
CometExecRulerepairs links onCometExec,CometShuffleExchangeExec, andCometBroadcastExchangeExecusingoriginalPlan.logicalLink, or clears both logical tags when that link is absent. #5482 / #5483 establish thatoriginalPlanis not always authoritative: Spark can reuse an existing physical root and assign a direct link to the currentLogicalQueryStage.#5483 preserves that direct link on
CometExec. This issue tracks the separate question raised in the review: which remaining repairs are necessary, and can link ownership be made explicit or repair narrowed without breaking AQE?Describe the potential solution
Audit initial conversion, repeated AQE replanning, and exchange/subquery creation to document when the original plan, the current physical node, or Spark's temporary stage link is authoritative. Distinguish fresh nodes from reused physical nodes and direct tags from inherited tags. Decide from that evidence whether to retain the current repair, narrow it, or move necessary assignments to node creation.
Any change should preserve:
LogicalQueryStageidentity across repeated replanning (CometExecRule overwrites direct AQE LogicalQueryStage links during replanning #5482).CometBroadcastExchangeExec. The DPP investigation found an unchanged, equal-cost replan that Spark rejects, while retainingTEMP_LOGICAL_PLAN_TAGfor new stage creation. Establish the required contract across supported Spark versions.Back the chosen contract with focused planner tests and end-to-end AQE/DPP coverage that checks results and actual native execution. This audit does not assume all repair is removable or establish another failing exchange case.
Additional context
Keep #5482 focused on its reproduced stale-stage-link defect. The broader design question should remain tracked after #5483 closes that issue.
Current logical-link repair.