Repository navigation
Conversation
* fix: propagate session settings to JVM shuffle writers * Reuse native shuffle configuration snapshot (cherry picked from commit 55ea0d4)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Backport of #7526 to
branch-1.2. Closes #7303 on this branch.Rationale for this change
JVM columnar shuffle writers read their Comet settings from the executor task thread. In a map stage launched outside a SQL execution (for example
df.queryExecution.toRdd), Spark does not propagate the session's SQL configuration to the task, so the writers ran with Comet defaults instead of the session's settings (#7303).branch-1.2has the sameCometShuffleManager.getWriterpath. See #7526 and #7303 for the full analysis.What changes are included in this PR?
Cherry-pick of 55ea0d4 (
git cherry-pick -x) with no conflicts and no adaptations. The shuffle dependency captures the session's Comet settings (reusing the native shuffle specification's capture when there is one), andCometShuffleManagerapplies them when constructing, writing and stopping the bypass-merge-sort and serialized JVM shuffle writers. The source PR'sCometColumnarShuffleSuitetest is included.This does not depend on #7348: the new test passes on this branch without it.
Other release branches:
branch-1.2is the newest release branch, so there is no newer one to port to. Whetherbranch-1.1needs a backport is not part of this PR.How are these changes tested?
On this branch, with JDK 17 and the default Spark profile (4.1):
./mvnw test-compilepasses, and the new testJVM shuffle writers see session configs outside SQL executionpasses in bothCometShuffleSuite(AQE on) andDisableAQECometShuffleSuiteagainst a native library built frombranch-1.2. Withspark/src/mainreverted tobranch-1.2's version, the test fails in both suites, so the branch has the bug and the test catches it. I ran the default Spark profile only.