Skip to content

fix: keep native shuffle temp files beside the data file - #7638

Open
stantheman0128 wants to merge 3 commits into
apache:mainfrom
stantheman0128:fix/7594-shuffle-data-tmp
Open

stantheman0128 wants to merge 3 commits into
apache:mainfrom
stantheman0128:fix/7594-shuffle-data-tmp

Conversation

@stantheman0128

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #7594.

Rationale for this change

A spark.local.dir directory named local.data made native shuffle writes fail with shuffle write error: Os { code: 2, kind: NotFound, message: "No such file or directory" }. The temp path was built with dataFile.getPath.replace(".data", ".data.tmp"), so the directory itself was rewritten to local.data.tmp and the native writer opened a path whose parent did not exist.

A failed map task also left that temp file on disk. stop(false) did not delete it.

What changes are included in this PR?

The local temp path now comes from IndexShuffleBlockResolver.createTempFile, which appends a unique suffix in the same directory. write() deletes that file when the attempt fails, and stop(false) deletes it if it is still tracked. A successful writeMetadataFileAndCommit clears the tracked path first, so stop(true) does not delete the committed data file.

How are these changes tested?

mvn test from the repo root, skipping scalastyle and spotless because this Windows checkout already fails those checks on files this PR does not change. Suite filter: CometNativeShuffleWriterSuite native shuffle.

Before the fix:

- native shuffle temp file stays in a local dir named local.data *** FAILED *** (58 milliseconds)
  "local.data[.tmp]" did not equal "local.data[]" (CometNativeShuffleWriterSuite.scala:91)

After the fix:

- native shuffle temp file stays in a local dir named local.data (44 milliseconds)
- native shuffle stop(false) deletes a tracked temp file (7 milliseconds)
Total number of tests run: 2
Tests: succeeded 2, failed 0, canceled 0, ignored 0, pending 0

The native library was not rebuilt. These two tests use the block resolver and stop, not a native shuffle execution.

Written with AI assistance (Cursor, Grok).

String.replace turns every .data into .data.tmp, so a spark.local.dir
named local.data is rewritten before the native writer opens the file.
IndexShuffleBlockResolver.createTempFile appends a unique suffix in the
same directory. A failed map attempt deletes that temp file, and a
successful commit still leaves the renamed data file in place.

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tracking this down, and for cleaning up the leftover temp file on failure too. Switching to createTempFile also means the temp file gets the SPARK-37618 permissions when permissionChangingRequired is set, which the old path never did.

The fork CI is red because Spotless flags CometNativeShuffleWriterSuite.scala. Could you run ./mvnw spotless:apply (or make format) and push? Most of the other failing jobs in that run look like they're the same build failing early.

The local.data test calls localTempDataFile directly, which just forwards to Spark's createTempFile. So it would still pass if writeInternal stopped using the helper. Would it be possible to add a test that runs an actual native shuffle with spark.local.dir pointing at a directory named local.data, for example in a separate SparkSession? Then the issue's exact repro is covered end to end, and the localTempDataFile wrapper may not be needed at all.

Spotless on Linux rejects the stop(false) writer constructor. The apply used the real Apache header and only this file.
@stantheman0128

Copy link
Copy Markdown
Contributor Author

Spotless is applied on CometNativeShuffleWriterSuite.scala only. Head is ea6a24bf3494d552ee7c1c151864be5f068e93f7.

On this Windows checkout, dev/copyright/scala-header.txt is the symlink target text java-header.txt (git mode 120000, core.symlinks is false). spotless:apply against that file would replace Apache headers with that literal string. For the apply I gave Spotless the contents of dev/copyright/java-header.txt, limited the run to the two Scala files in this PR, and did not commit the header file. CometNativeShuffleWriter.scala was already clean. The suite change is the stop(false) constructor layout from the CI Spotless diff.

I did not add a test that runs native shuffle. cargo build -p datafusion-comet --no-default-features fails in native/shuffle/src/writers/local/local_partition_writer.rs: it imports std::os::unix::fs::FileExt and calls read_exact_at, which this Windows toolchain does not have. No comet.dll was produced, so I cannot claim a native shuffle ran. The existing local.data and stop(false) tests still call the helper and stop directly. The localTempDataFile wrapper stays until a native run can show writeInternal is the path under test.

Prepared with Cursor Grok.

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for applying Spotless, and for explaining the Windows symlink issue with the header file.

The remaining fork CI failure is in Verify TPC-DS Results, and it is a Maven Central 429 before anything built, so a rerun of that job should clear it.

On the end-to-end test, you don't need a native build locally to add it. CI builds the native library and runs the Scala suites on Linux, so a test there will exercise the real path. Something like a separate SparkSession with spark.local.dir pointing at a temp directory named local.data, running a query that does a native shuffle and checking the answer, would cover the issue's exact repro. Could you add that and let CI run it? Then we can see whether the localTempDataFile wrapper is still needed.

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using IndexShuffleBlockResolver.createTempFile is the right fix. The old replace(".data", ".data.tmp") rewrote every .data in the path, so a local dir such as local.data pointed at a directory that does not exist. I checked Spark 3.5.8, 4.0.1 and 4.1.1 and createTempFile is public with the same body in all three. I agree with the end-to-end test request above and have one more point on the cleanup.

if (success) {
Some(mapStatus)
} else {
discardLocalTemp(None)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This discardLocalTemp(None) repeats the cleanup in the catch of write (line 114). Spark's ShuffleWriteProcessor calls stop(false) only after write threw an Exception, and the catch here already handles every Throwable. localTempData is only set inside writeInternal, so I could not find a path where stop(false) sees a temp file that write did not already delete. CometBypassMergeSortShuffleWriter follows the same rule and cleans up in write only. Dropping this branch and the stop(false) test would leave one cleanup path.

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please have a look into findings

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Native shuffle write fails when the local dir path contains .data; failed tasks also leave temp files behind

3 participants