[cherry-pick][2.58.0][Data] Update docs for hash shuffle v2 (#65372) in release 2.58 - #65612
Merged
Merged
Conversation
owenowenisme
added
docs
(cherry picked from commit fd223b6) Signed-off-by: You-Cheng Lin <mses010108@gmail.com>
## Summary Three claims on the Ray Data join and save pages don't match the code. This PR corrects only those claims, deliberately leaving the surrounding prose alone so the diff is small enough to review as a technical question rather than a wording one. Each correction is cited below. I don't own Ray Data, so please check the reasoning rather than the phrasing. ## 1. `saving-data.rst` sets the shuffle strategy on the class, not the context ```python DataContext.shuffle_strategy=ShuffleStrategy.SHUFFLE_V2 # before DataContext.get_current().shuffle_strategy = ShuffleStrategy.SHUFFLE_V2 # after ``` `shuffle_strategy` is a `property` with a setter (`python/ray/data/context.py`, the `@property` at ~L1246 and `@shuffle_strategy.setter` at ~L1258). Assigning to it on the class object rebinds the class attribute over the descriptor instead of invoking the setter. Two consequences: the example never actually selects the strategy it says it's selecting, and `shuffle_strategy` stops working for every `DataContext` instance in that process. This is a `testcode` block, so it runs in CI. Note that because the assignment never took effect, the block has so far only ever exercised the default `HASH_SHUFFLE`. With the fix it genuinely runs `SHUFFLE_V2` for the first time. I don't have a Ray build locally, so **please confirm that job passes** rather than taking my word for it. If `SHUFFLE_V2` isn't ready to be load-bearing in a doctest, setting `HASH_SHUFFLE` on `get_current()` would fix the descriptor bug without that exposure. @gemini-code-assist flagged this on two separate passes (#65372); the value got updated to `SHUFFLE_V2` but the receiver didn't. ## 2. `partition_size_hint` is documented as a live tuning parameter The page described it as `(optional)`, gave it a default of `DataContext.target_max_block_size`, and offered two bullets on how to size it for skew. But `Dataset.join` documents it as "**Deprecated** and ignored ... This parameter has no effect and will be removed in a future release" (`python/ray/data/dataset.py` ~L3601), and passing a value emits a `DeprecationWarning` (~L3712). So the guidance told readers how to tune something inert. I removed the sizing advice rather than rewriting it, since there's nothing to tune. The page also said "Expect this parameter to be deprecated in 2.58." That deprecation has already landed, so the sentence was predicting the past. ## 3. The aggregator pool doesn't default to 64 The page said "Defaults to 64 or `num_partitions` (in cases when there are less than 64 partitions)." From `_derive_max_shuffle_aggregators` and its caller in `python/ray/data/_internal/execution/operators/hash_shuffle.py` (~L485 and ~L591): ``` num_aggregators = min(target_num_partitions, max_shuffle_aggregators) max_shuffle_aggregators = min(ceil(total_cluster_cpu), max_hash_shuffle_aggregators or DEFAULT_MAX_HASH_SHUFFLE_AGGREGATORS) ``` `DEFAULT_MAX_HASH_SHUFFLE_AGGREGATORS` is 128 (`context.py` ~L128). So the cap is 128 rather than 64, and there's a cluster-CPU bound the page never mentioned. **One thing worth a maintainer's eye:** the comment on the `max_hash_shuffle_aggregators` field (`context.py` ~L886) says the unset default is the smaller of "Total # of CPUs available in the cluster * 2" and 128. The code has no `* 2`. I documented the code, not the comment, but one of the two is stale and I can't tell which is intended. ## Out of scope These pages have wider style and prose issues, plus a genuine RST rendering bug in the join-type lists. Those are handled separately in #65374 so this PR stays reviewable as a technical correction. #65374 touches the same region of `joining-data.rst`, so whichever merges second needs a trivial rebase. Signed-off-by: Douglas Strodtman <douglas@anyscale.com> (cherry picked from commit a74234c) Signed-off-by: You-Cheng Lin <mses010108@gmail.com>
owenowenisme
force-pushed
the
cherry-pick-2.58.0-shuffle-v2-docs
branch
from
August 20, 2026 08:18
1565811 to
746051b
Compare
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request updates the Ray Data documentation to introduce and detail the new Shuffle v2 (ShuffleStrategy.HASH_SHUFFLE_V2) backend, which is currently in Alpha. It explains the driver-driven architecture of Shuffle v2, its benefits (such as disk spilling and adaptive memory sizing), supported operations, and configuration/tuning knobs like input batch size and inline object threshold. Additionally, the documentation deprecates partition_size_hint in joins, updates the default aggregator pool size logic, and deprecates DataContext.use_push_based_shuffle in favor of configuring the shuffle strategy directly. There are no review comments, and I have no feedback to provide.
sai-miduthuri
changed the title
sai-miduthuri
changed the title
owenowenisme
removed
the
docs-go
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Backfill the doc of shuffle v2 into release 2.58 and use the name of
hash_shuffle_v2because the nameshuffle_v2only exist on 2.58This is a doc only changes so it's not blocking the PyPI release
Related issues
Additional information