fix(spark): Fix shuffle cleanup failure with RssShuffleManager on Spark 4

[wForget]

What changes were proposed in this pull request?

Add a ShuffleBlockResolver implementation for the Spark 4 RssShuffleManager.

Because RSS does not store shuffle blocks on local disks, the resolver:

• Returns an empty block sequence from getBlocksForShuffle.
• Reports a clear error if local or merged block data is requested.
• Uses a no-op stop implementation.

RssShuffleManager.shuffleBlockResolver() now returns this resolver instead of throwing an unsupported-operation exception.

Why are the changes needed?

Spark 4 enables spark.shuffle.service.removeShuffle by default. During shuffle cleanup, Spark calls shuffleBlockResolver().getBlocksForShuffle() to locate blocks managed by the external shuffle service.

The previous RSS implementation threw an exception whenever shuffleBlockResolver() was called, causing ContextCleaner to report shuffle cleanup failures.

Returning an empty block sequence is appropriate because RSS stores shuffle data remotely and has no local shuffle blocks for Spark to remove.

Fix: #2777

Does this PR introduce any user-facing change?

No

How was this patch tested?

Tested manually, and no exceptions were observed.

[wForget]

Maybe we should create LocalDiskShuffleComponents in RssShuffleDataIo only for the DelegationRssShuffleManager. I'll try that later.

Spark needs to call BlockManagerMaster#removeShuffle to broadcast the call to ShuffleManager.unregisterShuffle().

[github-actions]

Test Results

 3 833 files  ±0   3 833 suites  ±0   8h 6m 50s ⏱️ - 8m 25s
 1 266 tests +1   1 255 ✅ +1  11 💤 ±0  0 ❌ ±0 
19 156 runs  +3  19 108 ✅ +3  48 💤 ±0  0 ❌ ±0 

Results for commit d6b6d4d. ± Comparison against base commit a5372e9.

♻️ This comment has been updated with latest results.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 0% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 51.77%. Comparing base (a5372e9) to head (00c7447).

Files with missing lines	Patch %	Lines
...he/spark/shuffle/RssShuffleExecutorComponents.java	0.00%	6 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #2778      +/-   ##
============================================
- Coverage     51.79%   51.77%   -0.02%     
+ Complexity     3980     3979       -1     
============================================
  Files           599      600       +1     
  Lines         33216    33222       +6     
  Branches       3136     3136              
============================================
- Hits          17204    17201       -3     
- Misses        14894    14903       +9     
  Partials       1118     1118              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[wForget]

@LuciferYang Could you please take a look?

[LuciferYang]

@LuciferYang Could you please take a look?

will give feedback today.

[LuciferYang]

Suggest adding an assertion like

assertThrowsExactly(RssException.class,
    () -> blockResolver.getBlockData(new ShuffleBlockId(0, 0L, 0), Option.empty()));

[LuciferYang]

Merged into master. Thank you @wForget