Skip to content

fix: preserve session settings in JVM shuffle writers - #7526

Merged
andygrove merged 2 commits into
apache:mainfrom
dajiaohuang:fix-jvm-shuffle-config-7303-20261101
Nov 14, 2026
Merged

andygrove merged 2 commits into
apache:mainfrom
dajiaohuang:fix-jvm-shuffle-config-7303-20261101

Conversation

@dajiaohuang

@dajiaohuang dajiaohuang commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

What changes

Capture the session's Comet settings on the shuffle dependency and apply them when constructing, writing and stopping JVM columnar shuffle writers. This lets map stages launched outside a SQL execution use the configured settings. When a native shuffle specification is present, the dependency reuses its existing captured settings instead of capturing them twice; JVM shuffle captures its own settings. Spark shuffle writers retain their existing path.

Fixes #7303.

Validation

  • Reproduced the reported missing invalid-codec failure with a synthetic fixture after selecting the Comet exchange. An earlier attempt without the native library did not exercise the issue.
  • On the original change, make format, make, and git diff --check passed. The focused regression passed on Spark 4.1 with AQE on and off and Spark 3.5 with AQE off. Each run covers serialized and bypass handles, valid and invalid codecs, and SQL and non-SQL controls.
  • On review update 7b193ff, offline Maven Spark 4.1 compile, Scalastyle and Spotless passed with tests skipped; git diff --check passed. The initial compile reached the style check but failed because the added line exceeded 100 characters; it passed after splitting the line.

The focused regression was not rerun after the review update. Other Spark profiles and the upstream Spark SQL suites were not run.

@github-actions github-actions Bot added bug Something isn't working area:shuffle Shuffle (JVM and native) labels Nov 6, 2026

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @dajiaohuang for picking up #7303. Wrapping construction, write and stop covers the codec and spill-threshold reads the issue describes, and the bogus codec test with both handle types pins it in both directions.

shuffleWriterProcessor) {

/** The Comet settings used by JVM shuffle writers on executor task threads. */
private[shuffle] val writerSQLConf: CapturedSQLConf = CapturedSQLConf.capture()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For a native shuffle this captures the same Comet configs a second time: CometShuffleExchangeExec already passes CapturedSQLConf.capture() into NativeShuffleSpec.sqlConf, and both copies are serialized with the dependency to every map task. Could this reuse it, for example nativeShuffleSpec.map(_.sqlConf).getOrElse(CapturedSQLConf.capture()), so each dependency carries one capture and the native and JVM writers cannot see different snapshots?

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR backlog is huge and the maintainers don't have the bandwidth to review them all, so here is an automated review. Feel free to ignore it.

I found no blocking concerns at 7b193ff. The earlier request to reuse nativeShuffleSpec.sqlConf is addressed in CometShuffleDependency.writerSQLConf, so a native dependency carries a single capture.

I read the JVM writers to check that the three wrapped entry points cover the config reads. The codec is read in the CometDiskBlockWriter and CometShuffleExternalSorter constructors, which now run inside withWriterConf. The batch size read in SpillWriter happens on the spill path reached from write, which is also wrapped. ShuffleWriteProcessor only calls write, stop and getPartitionLengths on the writer, and I found no code that inspects the concrete writer class, so the delegating wrapper in CometShuffleManager.withWriterConf does not change behavior elsewhere. CapturedSQLConf.withConf is a no-op when the task already has the session configs, so the SQL execution path is unchanged against the latest release.

For the test, the bogus codec with both the serialized and bypass handles, followed by the same query through df.collect(), covers both the RDD-only and SQL paths.

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @dajiaohuang

@andygrove
andygrove added this pull request to the merge queue Nov 14, 2026
Merged via the queue into apache:main with commit 55ea0d4 Nov 14, 2026
27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:shuffle Shuffle (JVM and native) backport-1.2 bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

JVM columnar shuffle writers use Comet defaults in map stages of jobs outside a SQL execution

3 participants