Repository navigation
Description
What happened + What you expected to happen
On pyarrow 25+, Dataset.summary() raises NotImplementedError when the dataset has a native arrow.fixed_shape_tensor column (the ARROW_NATIVE tensor format) and at least one other column, whose type might be a numeric, boolean, or date/time. For each of these types, we populate a type-preserving summary table, which causes the failure when trying to populate it for the tensor column.
_build_summary_table(preserve_types=True) builds the table of statistics that keep their column's type (e.g. min/max of an int column). Every column in the dataset gets a row there, typed as the original column. The tensor column has no such statistics, so its row is [None, None, ...] typed as the tensor type, and _create_pyarrow_array calls pa.array([None, ...], type=tensor_type).
Native tensor types built with pa.FixedShapeTensorArray.from_numpy_ndarray always carry permutation=[0, 1]. pyarrow 25 added a guard to pa.array (apache/arrow#50203, GH-49644) that rejects any Python list input for a fixed_shape_tensor with a permutation, even an all-None list. It raises a plain NotImplementedError, which the surrounding except (pa.ArrowTypeError, pa.ArrowInvalid) doesn't catch, so it propagates out of summary().
Expected: summary() returns the same table as on pyarrow 23/24. On pyarrow 23/24, the tensor column's type-preserving statistics are null.
Error (pyarrow 25.0.1):
File "python/ray/data/dataset.py", line 4400, in summary
File "python/ray/data/stats.py", line 632, in _build_summary_table
File "python/ray/data/stats.py", line 579, in _create_pyarrow_array
NotImplementedError: Converting a sequence of arrays to a fixed_shape_tensor with a permutation is not supported
Output on pyarrow 23.0.1 (expected):
statistic emb id
0 approx_quantile[0] None 1.0
1 approx_topk[0] {'count': 1, 'emb': [0, 0, 0, 0]} <NA>
...
4 count 3 3.0
5 max NaN 2.0
...
The V1/V2 tensor formats aren't affected, since their types have no permutation. Related: #68072 fixes the same pyarrow 25 guard for shuffle and join (Schema.empty_table()).
Versions / Dependencies
Ray: master (3.0.0.dev0, fae7ff8)
pyarrow: 25.0.1 (fails), 26.0.1 (fails), 23.0.1 (works)
Python: 3.10
OS: macOS (arm64)
Reproduction script
import numpy as np
import pyarrow as pa
import ray
from ray.data.context import DataContext
from ray.data._internal.tensor_extensions.arrow import
FixedShapeTensorFormat
print("ray", ray.__version__, "pyarrow", pa.__version__)
# Store fixed-shape tensors as pyarrow's native
arrow.fixed_shape_tensor type.
DataContext.get_current().arrow_fixed_shape_tensor_format =
FixedShapeTensorFormat.ARROW_NATIVE
ds = ray.data.from_items([{"id": i, "emb": np.full((2, 2), i)}
for i in range(3)])
print(ds.schema())
print(ds.summary().to_pandas())Issue Severity
Low: It annoys or frustrates me.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsTodo
Activity
GoyalIshaan commentedon Nov 18, 2026
Could I work on the Dataset.summary() regression here? I'd cover native fixed-shape tensor columns alongside scalar columns on PyArrow 23/24 and 25/26, preserving the null statistics and checking that V1/V2 tensor summaries stay unchanged. This would target summary-table construction, separate from #68072's empty shuffle/join tables.
AI assistance was used to scout this issue and draft this request; I would implement and validate the change myself.
ayushk7102 commentedon Nov 18, 2026
Yep, feel free to raise a PR and assign me for review! Thanks @GoyalIshaan