Repository navigation
fix: keep native shuffle temp files beside the data file - #7638
stantheman0128 wants to merge 3 commits into
Conversation
String.replace turns every .data into .data.tmp, so a spark.local.dir named local.data is rewritten before the native writer opens the file.
IndexShuffleBlockResolver.createTempFile appends a unique suffix in the same directory. A failed map attempt deletes that temp file, and a successful commit still leaves the renamed data file in place.
andygrove
left a comment
There was a problem hiding this comment.
Thanks for tracking this down, and for cleaning up the leftover temp file on failure too. Switching to createTempFile also means the temp file gets the SPARK-37618 permissions when permissionChangingRequired is set, which the old path never did.
The fork CI is red because Spotless flags CometNativeShuffleWriterSuite.scala. Could you run ./mvnw spotless:apply (or make format) and push? Most of the other failing jobs in that run look like they're the same build failing early.
The local.data test calls localTempDataFile directly, which just forwards to Spark's createTempFile. So it would still pass if writeInternal stopped using the helper. Would it be possible to add a test that runs an actual native shuffle with spark.local.dir pointing at a directory named local.data, for example in a separate SparkSession? Then the issue's exact repro is covered end to end, and the localTempDataFile wrapper may not be needed at all.
Spotless on Linux rejects the stop(false) writer constructor. The apply used the real Apache header and only this file.
|
Spotless is applied on On this Windows checkout, I did not add a test that runs native shuffle. Prepared with Cursor Grok. |
andygrove
left a comment
There was a problem hiding this comment.
Thanks for applying Spotless, and for explaining the Windows symlink issue with the header file.
The remaining fork CI failure is in Verify TPC-DS Results, and it is a Maven Central 429 before anything built, so a rerun of that job should clear it.
On the end-to-end test, you don't need a native build locally to add it. CI builds the native library and runs the Scala suites on Linux, so a test there will exercise the real path. Something like a separate SparkSession with spark.local.dir pointing at a temp directory named local.data, running a query that does a native shuffle and checking the answer, would cover the issue's exact repro. Could you add that and let CI run it? Then we can see whether the localTempDataFile wrapper is still needed.
comphead
left a comment
There was a problem hiding this comment.
Using IndexShuffleBlockResolver.createTempFile is the right fix. The old replace(".data", ".data.tmp") rewrote every .data in the path, so a local dir such as local.data pointed at a directory that does not exist. I checked Spark 3.5.8, 4.0.1 and 4.1.1 and createTempFile is public with the same body in all three. I agree with the end-to-end test request above and have one more point on the cleanup.
| if (success) { | ||
| Some(mapStatus) | ||
| } else { | ||
| discardLocalTemp(None) |
There was a problem hiding this comment.
This discardLocalTemp(None) repeats the cleanup in the catch of write (line 114). Spark's ShuffleWriteProcessor calls stop(false) only after write threw an Exception, and the catch here already handles every Throwable. localTempData is only set inside writeInternal, so I could not find a path where stop(false) sees a temp file that write did not already delete. CometBypassMergeSortShuffleWriter follows the same rule and cleans up in write only. Dropping this branch and the stop(false) test would leave one cleanup path.
comphead
left a comment
There was a problem hiding this comment.
Please have a look into findings
Which issue does this PR close?
Closes #7594.
Rationale for this change
A
spark.local.dirdirectory namedlocal.datamade native shuffle writes fail withshuffle write error: Os { code: 2, kind: NotFound, message: "No such file or directory" }. The temp path was built withdataFile.getPath.replace(".data", ".data.tmp"), so the directory itself was rewritten tolocal.data.tmpand the native writer opened a path whose parent did not exist.A failed map task also left that temp file on disk.
stop(false)did not delete it.What changes are included in this PR?
The local temp path now comes from
IndexShuffleBlockResolver.createTempFile, which appends a unique suffix in the same directory.write()deletes that file when the attempt fails, andstop(false)deletes it if it is still tracked. A successfulwriteMetadataFileAndCommitclears the tracked path first, sostop(true)does not delete the committed data file.How are these changes tested?
mvn testfrom the repo root, skipping scalastyle and spotless because this Windows checkout already fails those checks on files this PR does not change. Suite filter:CometNativeShuffleWriterSuite native shuffle.Before the fix:
After the fix:
The native library was not rebuilt. These two tests use the block resolver and
stop, not a native shuffle execution.Written with AI assistance (Cursor, Grok).