Skip to content

[Data] Fix shuffle and join on native tensor columns with pyarrow 25+ - #68072

Merged
goutamvenkat-anyscale merged 2 commits into
ray-project:masterfrom
rayo5o:data-native-tensor-empty-table
Nov 18, 2026
Merged

goutamvenkat-anyscale merged 2 commits into
ray-project:masterfrom
rayo5o:data-native-tensor-empty-table

Conversation

@rayo5o

@rayo5o rayo5o commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Description

On pyarrow 25 and later, Schema.empty_table() raises NotImplementedError for a native fixed_shape_tensor column with a permutation, and pyarrow's from_numpy_ndarray always sets one. 24.0.0 works. Ray Data builds empty shards and partitions with empty_table(), so sort(), groupby().map_groups(), repartition() with shuffle or keys, join(), and zip() fail on these columns. So does sort().limit() with the Top-N rewrite when no rows are left. The default V2 format is affected too when reading Parquet written with pyarrow's native tensor type. This is the test_tensors_sort[ARROW_NATIVE] failure I noted on #68005; it still fails on master.

The existing _create_empty_table helper, used by hash shuffle, hit the same limit. It now builds each column with pyarrow.nulls(0, type), moves to _internal/utils/transform_pyarrow.py, and replaces every empty_table() call on a data schema in Ray Data, including _empty_block_from_schema in the Top-N operator. For the other 15 types I compared on pyarrow 23 and 26, the result equals empty_table(). It also covers extension types nested in a struct, which empty_table() rejects on pyarrow 23 as well.

Related issues

Follow-up to #68005.

Additional information

  • Not a duplicate: no open issue or PR in ray-project/ray or apache/arrow covers this.
  • Tests: added test_create_empty_table, test_tensors_shuffle_and_join, and test_tensors_top_n_empty_result. Without the fix, the 5 new ARROW_NATIVE cases and test_tensors_sort[ARROW_NATIVE] fail on pyarrow 25.0.1 and 26.0.1. With it, 8 related test files pass locally on pyarrow 23, 25 and 26, except test_fetch_restart_retry and a missing-fixture error from my local setup, which also occur on master.
  • These ARROW_NATIVE cases only fail on the arrow v25 and arrow v26 lanes. Neither runs on a regular premerge build: v25 is postmerge only, and v26 runs on postmerge or with the continuous-build label.
  • AI assistance was used to prepare this PR.

@rayo5o
rayo5o requested a review from a team as a code owner November 11, 2026 16:58

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors how empty PyArrow tables are created by moving and re-implementing _create_empty_table to use pyarrow.nulls. This change prevents failures when dealing with certain extension columns (such as fixed_shape_tensor on PyArrow 26 or nested extension types) where Schema.empty_table() fails. The _create_empty_table function is now used consistently across several data source, execution, and logical operator modules, and corresponding tests have been added. Feedback on the changes highlights a potential compatibility issue: pyarrow.nulls was introduced in PyArrow 16.0.0, which could cause an AttributeError on older PyArrow versions supported by Ray (>= 12.0.0). A fallback mechanism using pyarrow.array is recommended to ensure backward compatibility.

@ray-gardener ray-gardener Bot added data Ray Data-related issues community-contribution Contributed by the community labels Nov 11, 2026
@rayo5o
rayo5o force-pushed the data-native-tensor-empty-table branch from 3c28c80 to a444806 Compare November 14, 2026 11:46
@abhishekverma-ray

Copy link
Copy Markdown
Contributor

Thanks @rayo5o, this matches what I see on postmerge: test_tensors_sort[ARROW_NATIVE] fails on every run in both the arrow v25 and v26 lanes. I checked the pa.nulls(0, type) approach on pyarrow 23 and 25 across 15 types; it matches empty_table() wherever that works and also handles unions and nested extension types.

Two asks:

  1. _empty_block_from_schema in _internal/execution/operators/top_n_operator.py (added by [data] Add Top-N execution #66931, already in your base) still calls schema.empty_table(), so sort + limit on these columns can fail the same way. Could you switch it to _create_empty_table?
  2. The description says pyarrow 26, but it also fails on 25 (the arrow v25 lane, and I reproduced it on 25.0.1). Could you update it, since it becomes the commit message?

On pyarrow 26, Schema.empty_table() raises NotImplementedError for a
native fixed_shape_tensor column with a permutation, which pyarrow's
from_numpy_ndarray always sets. Ray Data builds empty shards and
partitions with empty_table(), so sort, map_groups, repartition and
join failed on such columns.

Build empty tables column by column with pyarrow.nulls(0) in the
existing _create_empty_table helper, move it to
_internal/utils/transform_pyarrow.py, and use it wherever Ray Data
creates an empty table from a data schema.

Signed-off-by: rayo5o <ray941005@gmail.com>
The Top-N merge builds an empty block from the schema when no rows are
left, which hit the same Schema.empty_table() failure on native tensor
columns. Also say pyarrow 25+ in comments: 24.0.0 works, 25.0.1 fails.

Signed-off-by: rayo5o <ray941005@gmail.com>
@rayo5o
rayo5o force-pushed the data-native-tensor-empty-table branch from a444806 to d0ce376 Compare November 16, 2026 12:45
@rayo5o rayo5o changed the title [Data] Fix shuffle and join on native tensor columns with pyarrow 26 [Data] Fix shuffle and join on native tensor columns with pyarrow 25+ Nov 16, 2026
@rayo5o

rayo5o commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @rayo5o, this matches what I see on postmerge: test_tensors_sort[ARROW_NATIVE] fails on every run in both the arrow v25 and v26 lanes. I checked the pa.nulls(0, type) approach on pyarrow 23 and 25 across 15 types; it matches empty_table() wherever that works and also handles unions and nested extension types.

Two asks:

  1. _empty_block_from_schema in _internal/execution/operators/top_n_operator.py (added by [data] Add Top-N execution #66931, already in your base) still calls schema.empty_table(), so sort + limit on these columns can fail the same way. Could you switch it to _create_empty_table?
  2. The description says pyarrow 26, but it also fails on 25 (the arrow v25 lane, and I reproduced it on 25.0.1). Could you update it, since it becomes the commit message?

Both done in d0ce376, rebased onto current master (conflict with #67130 in disk_shuffle_reduce_operator.py).

  1. You're right about Top-N. I had checked it on Saturday, but with the rewrite off and a map_batches source, so _empty_block_from_schema never ran. With top_n_rewrite_enabled, a from_arrow or read_parquet source, and a filter that drops every row, sort().limit() raised on 25.0.1 and 26.0.1. It now uses _create_empty_table, and test_tensors_top_n_empty_result covers it.
  2. Schema.empty_table() on these columns works on 24.0.0 and fails from 25.0.1, so I changed the title and description to "pyarrow 25+".

@ayushk7102
ayushk7102 self-requested a review November 17, 2026 23:09
@ayushk7102

Copy link
Copy Markdown
Contributor

Thanks for your fix @rayo5o ! LGTM

Since this issue was due to pyarrow 25 adding a new guard to pa.array (apache/arrow#50203, GH-49644) that raises a plain NotImplementedError for lists when the type is a fixed_shape_tensor with a permutation, there are a couple of other places that could be affected in Ray Data code. Given that your PR is fixing the shuffle and join cases, I think we can address those in a follow up.

I think the main place that users could run into issues is in the public Dataset.summary API, when their dataset contains a ARROW_NATIVE fixed shape tensor type.

Dataset.summary(). The tensor column's row is built as pa.array([None, ...], type=tensor_type). The NotImplementedError isn't caught by the except below, so we raise

try:
return pa.array(col_data, type=col_type)
except (pa.ArrowTypeError, pa.ArrowInvalid):

@ayushk7102 ayushk7102 added the go add ONLY when ready to merge, run all tests label Nov 17, 2026
@ayushk7102

ayushk7102 commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Added an issue on upstream arrow to track the root cause of this issue apache/arrow#55664

Also created a related issue for the Dataset.stats() regression here #68280

@goutamvenkat-anyscale
goutamvenkat-anyscale merged commit e2dd71e into ray-project:master Nov 18, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution Contributed by the community data Ray Data-related issues go add ONLY when ready to merge, run all tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants