Repository navigation
fix: preserve session settings in JVM shuffle writers - #7526
Conversation
comphead
left a comment
There was a problem hiding this comment.
Thanks @dajiaohuang for picking up #7303. Wrapping construction, write and stop covers the codec and spill-threshold reads the issue describes, and the bogus codec test with both handle types pins it in both directions.
| shuffleWriterProcessor) { | ||
|
|
||
| /** The Comet settings used by JVM shuffle writers on executor task threads. */ | ||
| private[shuffle] val writerSQLConf: CapturedSQLConf = CapturedSQLConf.capture() |
There was a problem hiding this comment.
For a native shuffle this captures the same Comet configs a second time: CometShuffleExchangeExec already passes CapturedSQLConf.capture() into NativeShuffleSpec.sqlConf, and both copies are serialized with the dependency to every map task. Could this reuse it, for example nativeShuffleSpec.map(_.sqlConf).getOrElse(CapturedSQLConf.capture()), so each dependency carries one capture and the native and JVM writers cannot see different snapshots?
andygrove
left a comment
There was a problem hiding this comment.
The PR backlog is huge and the maintainers don't have the bandwidth to review them all, so here is an automated review. Feel free to ignore it.
I found no blocking concerns at 7b193ff. The earlier request to reuse nativeShuffleSpec.sqlConf is addressed in CometShuffleDependency.writerSQLConf, so a native dependency carries a single capture.
I read the JVM writers to check that the three wrapped entry points cover the config reads. The codec is read in the CometDiskBlockWriter and CometShuffleExternalSorter constructors, which now run inside withWriterConf. The batch size read in SpillWriter happens on the spill path reached from write, which is also wrapped. ShuffleWriteProcessor only calls write, stop and getPartitionLengths on the writer, and I found no code that inspects the concrete writer class, so the delegating wrapper in CometShuffleManager.withWriterConf does not change behavior elsewhere. CapturedSQLConf.withConf is a no-op when the task already has the session configs, so the SQL execution path is unchanged against the latest release.
For the test, the bogus codec with both the serialized and bypass handles, followed by the same query through df.collect(), covers both the RDD-only and SQL paths.
What changes
Capture the session's Comet settings on the shuffle dependency and apply them when constructing, writing and stopping JVM columnar shuffle writers. This lets map stages launched outside a SQL execution use the configured settings. When a native shuffle specification is present, the dependency reuses its existing captured settings instead of capturing them twice; JVM shuffle captures its own settings. Spark shuffle writers retain their existing path.
Fixes #7303.
Validation
make format,make, andgit diff --checkpassed. The focused regression passed on Spark 4.1 with AQE on and off and Spark 3.5 with AQE off. Each run covers serialized and bypass handles, valid and invalid codecs, and SQL and non-SQL controls.7b193ff, offline Maven Spark 4.1 compile, Scalastyle and Spotless passed with tests skipped;git diff --checkpassed. The initial compile reached the style check but failed because the added line exceeded 100 characters; it passed after splitting the line.The focused regression was not rerun after the review update. Other Spark profiles and the upstream Spark SQL suites were not run.