Skip to content

[Python] pa.array / Schema.empty_table fail for fixed_shape_tensor with permutation #55664

Open

Description

@ayushk7102

Describe the bug, including details regarding any error messages, version, and platform.

In pyarrow 25.0.1 and 26.0.1, pa.array raises NotImplementedError for any Python list or tuple input when the target type is a fixed_shape_tensor with a permutation. That includes an empty list, [None], and flat lists of storage values, which all work in 24.0.0. The permutation can even be the identity [0, 1], which FixedShapeTensorArray.from_numpy_ndarray always sets for C-contiguous input.

Because Schema.empty_table() builds each column with pa.array([], type=...), you can no longer create an empty table for a schema with a tensor column produced by from_numpy_ndarray. Table.from_pylist / RecordBatch.from_pylist with such a schema fail the same way.

The check was added in #50203 (GH-49644) to reject multi-dimensional elements that would need a transpose. It runs before looking at any element, though, so it also rejects inputs that need no transpose, for e.g. an empty list, or if the permutation is the identity permutation.

Repro below:

import numpy as np
import pyarrow as pa

# from_numpy_ndarray always sets a permutation, [0, 1] for C-contiguous input.
t = pa.FixedShapeTensorArray.from_numpy_ndarray(np.zeros((2, 2, 2))).type
schema = pa.schema([("a", t)])
print(pa.__version__, t)

cases = {
    "pa.schema(...).empty_table()": lambda: schema.empty_table(),
    "pa.array([], type=t)": lambda: pa.array([], type=t),
    "pa.array([None], type=t)": lambda: pa.array([None], type=t),
    "pa.array([[0, 1, 2, 3]], type=t)": lambda: pa.array([[0, 1, 2, 3]], type=t),
    "pa.Table.from_pylist([], schema=schema)": lambda: pa.Table.from_pylist([], schema=schema),
}
for name, fn in cases.items():
    try:
        fn()
        print(f"ok    {name}")
    except Exception as e:
        print(f"FAIL  {name}: {type(e).__name__}: {e}")

On 24.0.0 every case prints ok. On 25.0.1 and 26.0.1 every case fails:

25.0.1 extension<arrow.fixed_shape_tensor[value_type=double, shape=[2,2], permutation=[0,1]]>
FAIL  pa.schema(...).empty_table(): NotImplementedError: Converting a sequence of arrays to a fixed_shape_tensor with a permutation is not supported
FAIL  pa.array([], type=t): NotImplementedError: Converting a sequence of arrays to a fixed_shape_tensor with a permutation is not supported
FAIL  pa.array([None], type=t): NotImplementedError: Converting a sequence of arrays to a fixed_shape_tensor with a permutation is not supported
FAIL  pa.array([[0, 1, 2, 3]], type=t): NotImplementedError: Converting a sequence of arrays to a fixed_shape_tensor with a permutation is not supported
FAIL  pa.Table.from_pylist([], schema=schema): NotImplementedError: Converting a sequence of arrays to a fixed_shape_tensor with a permutation is not supported

Expected

These should behave as in 24.0.0. Only inputs that would actually need a transpose (a multi-dimensional element with a non-identity permutation) need to be rejected. Empty input, nulls and flat storage values have no logical layout to apply, and an identity permutation is the same layout as no permutation. Also, it's a plain NotImplementedError rather than pa.ArrowNotImplementedError, so existing except pa.Arrow... handlers don't catch it.

Possible fix

Move the permutation check in array() (array.pxi) into the existing per-element shape loop, and skip it for identity permutations. Moving into loop allows us to prevent raising an error if an empty list is provided as input:

permutation = extension_type.permutation
# Only a non-identity permutation needs the elements to be transposed
is_permuted = (permutation is not None
               and list(permutation) != list(range(len(permutation))))
expected_shape = tuple(extension_type.shape)
for element in obj:
    shape = getattr(element, "shape", None)
    if shape is None or len(shape) < 2:
        continue
    if is_permuted:
        raise NotImplementedError(...)
    if tuple(shape) != expected_shape:
        raise ValueError(...)

Happy to open a PR if this approach sounds right.

Environment

pyarrow 24.0.0 (works), 25.0.1 and 26.0.1 (fail)
Python 3.12, macOS arm64.

Component(s)

Python

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions