Skip to content

fix: [branch-1.2] preserve session settings in JVM shuffle writers (#7526) - #7645

Open
andygrove wants to merge 2 commits into
apache:branch-1.2from
andygrove:backport-7526-branch-1.2
Open

andygrove wants to merge 2 commits into
apache:branch-1.2from
andygrove:backport-7526-branch-1.2

Conversation

@andygrove

Copy link
Copy Markdown
Member

Which issue does this PR close?

Backport of #7526 to branch-1.2. Closes #7303 on this branch.

Rationale for this change

JVM columnar shuffle writers read their Comet settings from the executor task thread. In a map stage launched outside a SQL execution (for example df.queryExecution.toRdd), Spark does not propagate the session's SQL configuration to the task, so the writers ran with Comet defaults instead of the session's settings (#7303). branch-1.2 has the same CometShuffleManager.getWriter path. See #7526 and #7303 for the full analysis.

What changes are included in this PR?

Cherry-pick of 55ea0d4 (git cherry-pick -x) with no conflicts and no adaptations. The shuffle dependency captures the session's Comet settings (reusing the native shuffle specification's capture when there is one), and CometShuffleManager applies them when constructing, writing and stopping the bypass-merge-sort and serialized JVM shuffle writers. The source PR's CometColumnarShuffleSuite test is included.

This does not depend on #7348: the new test passes on this branch without it.

Other release branches: branch-1.2 is the newest release branch, so there is no newer one to port to. Whether branch-1.1 needs a backport is not part of this PR.

How are these changes tested?

On this branch, with JDK 17 and the default Spark profile (4.1): ./mvnw test-compile passes, and the new test JVM shuffle writers see session configs outside SQL execution passes in both CometShuffleSuite (AQE on) and DisableAQECometShuffleSuite against a native library built from branch-1.2. With spark/src/main reverted to branch-1.2's version, the test fails in both suites, so the branch has the bug and the test catches it. I ran the default Spark profile only.

dajiaohuang and others added 2 commits November 16, 2026 10:33
* fix: propagate session settings to JVM shuffle writers

* Reuse native shuffle configuration snapshot

(cherry picked from commit 55ea0d4)

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants