Skip to content

[Data] Dataset.summary() fails with NotImplementedError on native fixed shape tensor columns with pyarrow 25+ #68280

Open

Description

@ayushk7102

What happened + What you expected to happen

On pyarrow 25+, Dataset.summary() raises NotImplementedError when the dataset has a native arrow.fixed_shape_tensor column (the ARROW_NATIVE tensor format) and at least one other column, whose type might be a numeric, boolean, or date/time. For each of these types, we populate a type-preserving summary table, which causes the failure when trying to populate it for the tensor column.

_build_summary_table(preserve_types=True) builds the table of statistics that keep their column's type (e.g. min/max of an int column). Every column in the dataset gets a row there, typed as the original column. The tensor column has no such statistics, so its row is [None, None, ...] typed as the tensor type, and _create_pyarrow_array calls pa.array([None, ...], type=tensor_type).

Native tensor types built with pa.FixedShapeTensorArray.from_numpy_ndarray always carry permutation=[0, 1]. pyarrow 25 added a guard to pa.array (apache/arrow#50203, GH-49644) that rejects any Python list input for a fixed_shape_tensor with a permutation, even an all-None list. It raises a plain NotImplementedError, which the surrounding except (pa.ArrowTypeError, pa.ArrowInvalid) doesn't catch, so it propagates out of summary().

Expected: summary() returns the same table as on pyarrow 23/24. On pyarrow 23/24, the tensor column's type-preserving statistics are null.

Error (pyarrow 25.0.1):

  File "python/ray/data/dataset.py", line 4400, in summary
  File "python/ray/data/stats.py", line 632, in _build_summary_table
  File "python/ray/data/stats.py", line 579, in _create_pyarrow_array
NotImplementedError: Converting a sequence of arrays to a fixed_shape_tensor with a permutation is not supported

Output on pyarrow 23.0.1 (expected):

             statistic                                emb         id
0   approx_quantile[0]                               None        1.0
1       approx_topk[0]  {'count': 1, 'emb': [0, 0, 0, 0]}       <NA>
...
4                count                                  3        3.0
5                  max                                NaN        2.0
...

The V1/V2 tensor formats aren't affected, since their types have no permutation. Related: #68072 fixes the same pyarrow 25 guard for shuffle and join (Schema.empty_table()).

Versions / Dependencies

Ray: master (3.0.0.dev0, fae7ff8)
pyarrow: 25.0.1 (fails), 26.0.1 (fails), 23.0.1 (works)
Python: 3.10
OS: macOS (arm64)

Reproduction script

import numpy as np
import pyarrow as pa
import ray
from ray.data.context import DataContext
from ray.data._internal.tensor_extensions.arrow import
FixedShapeTensorFormat

print("ray", ray.__version__, "pyarrow", pa.__version__)

# Store fixed-shape tensors as pyarrow's native
arrow.fixed_shape_tensor type.
DataContext.get_current().arrow_fixed_shape_tensor_format =
FixedShapeTensorFormat.ARROW_NATIVE

ds = ray.data.from_items([{"id": i, "emb": np.full((2, 2), i)}
for i in range(3)])
print(ds.schema())
print(ds.summary().to_pandas())

Issue Severity

Low: It annoys or frustrates me.

Activity

  1. added
    bugSomething that is supposed to be working; but isn't
    triageNeeds triage (eg: priority, bug/not-bug, and owning component)
    on Nov 18, 2026
  2. GoyalIshaan commented on Nov 18, 2026

    @GoyalIshaan

    Could I work on the Dataset.summary() regression here? I'd cover native fixed-shape tensor columns alongside scalar columns on PyArrow 23/24 and 25/26, preserving the null statistics and checking that V1/V2 tensor summaries stay unchanged. This would target summary-table construction, separate from #68072's empty shuffle/join tables.

    AI assistance was used to scout this issue and draft this request; I would implement and validate the change myself.

  3. ayushk7102 commented on Nov 18, 2026

    @ayushk7102
    ContributorAuthor

    Could I work on the Dataset.summary() regression here? I'd cover native fixed-shape tensor columns alongside scalar columns on PyArrow 23/24 and 25/26, preserving the null statistics and checking that V1/V2 tensor summaries stay unchanged. This would target summary-table construction, separate from #68072's empty shuffle/join tables.

    AI assistance was used to scout this issue and draft this request; I would implement and validate the change myself.

    Yep, feel free to raise a PR and assign me for review! Thanks @GoyalIshaan

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething that is supposed to be working; but isn'tdataRay Data-related issuesstabilitytriageNeeds triage (eg: priority, bug/not-bug, and owning component)

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions