Repository navigation
[Data] Fix shuffle and join on native tensor columns with pyarrow 25+ - #68072
goutamvenkat-anyscale merged 2 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request refactors how empty PyArrow tables are created by moving and re-implementing _create_empty_table to use pyarrow.nulls. This change prevents failures when dealing with certain extension columns (such as fixed_shape_tensor on PyArrow 26 or nested extension types) where Schema.empty_table() fails. The _create_empty_table function is now used consistently across several data source, execution, and logical operator modules, and corresponding tests have been added. Feedback on the changes highlights a potential compatibility issue: pyarrow.nulls was introduced in PyArrow 16.0.0, which could cause an AttributeError on older PyArrow versions supported by Ray (>= 12.0.0). A fallback mechanism using pyarrow.array is recommended to ensure backward compatibility.
3c28c80 to
a444806
Compare
|
Thanks @rayo5o, this matches what I see on postmerge: Two asks:
|
On pyarrow 26, Schema.empty_table() raises NotImplementedError for a native fixed_shape_tensor column with a permutation, which pyarrow's from_numpy_ndarray always sets. Ray Data builds empty shards and partitions with empty_table(), so sort, map_groups, repartition and join failed on such columns. Build empty tables column by column with pyarrow.nulls(0) in the existing _create_empty_table helper, move it to _internal/utils/transform_pyarrow.py, and use it wherever Ray Data creates an empty table from a data schema. Signed-off-by: rayo5o <ray941005@gmail.com>
The Top-N merge builds an empty block from the schema when no rows are left, which hit the same Schema.empty_table() failure on native tensor columns. Also say pyarrow 25+ in comments: 24.0.0 works, 25.0.1 fails. Signed-off-by: rayo5o <ray941005@gmail.com>
a444806 to
d0ce376
Compare
Both done in d0ce376, rebased onto current master (conflict with #67130 in
|
|
Thanks for your fix @rayo5o ! LGTM Since this issue was due to pyarrow 25 adding a new guard to I think the main place that users could run into issues is in the public
Lines 578 to 580 in fae7ff8 |
|
Added an issue on upstream arrow to track the root cause of this issue apache/arrow#55664 Also created a related issue for the Dataset.stats() regression here #68280 |
Description
On pyarrow 25 and later,
Schema.empty_table()raisesNotImplementedErrorfor a nativefixed_shape_tensorcolumn with apermutation, and pyarrow'sfrom_numpy_ndarrayalways sets one. 24.0.0 works. Ray Data builds empty shards and partitions withempty_table(), sosort(),groupby().map_groups(),repartition()withshuffleorkeys,join(), andzip()fail on these columns. So doessort().limit()with the Top-N rewrite when no rows are left. The default V2 format is affected too when reading Parquet written with pyarrow's native tensor type. This is thetest_tensors_sort[ARROW_NATIVE]failure I noted on #68005; it still fails on master.The existing
_create_empty_tablehelper, used by hash shuffle, hit the same limit. It now builds each column withpyarrow.nulls(0, type), moves to_internal/utils/transform_pyarrow.py, and replaces everyempty_table()call on a data schema in Ray Data, including_empty_block_from_schemain the Top-N operator. For the other 15 types I compared on pyarrow 23 and 26, the result equalsempty_table(). It also covers extension types nested in a struct, whichempty_table()rejects on pyarrow 23 as well.Related issues
Follow-up to #68005.
Additional information
test_create_empty_table,test_tensors_shuffle_and_join, andtest_tensors_top_n_empty_result. Without the fix, the 5 new ARROW_NATIVE cases andtest_tensors_sort[ARROW_NATIVE]fail on pyarrow 25.0.1 and 26.0.1. With it, 8 related test files pass locally on pyarrow 23, 25 and 26, excepttest_fetch_restart_retryand a missing-fixture error from my local setup, which also occur on master.arrow v25andarrow v26lanes. Neither runs on a regular premerge build: v25 is postmerge only, and v26 runs on postmerge or with thecontinuous-buildlabel.