Repository navigation
Description
Describe the bug, including details regarding any error messages, version, and platform.
In pyarrow 25.0.1 and 26.0.1, pa.array raises NotImplementedError for any Python list or tuple input when the target type is a fixed_shape_tensor with a permutation. That includes an empty list, [None], and flat lists of storage values, which all work in 24.0.0. The permutation can even be the identity [0, 1], which FixedShapeTensorArray.from_numpy_ndarray always sets for C-contiguous input.
Because Schema.empty_table() builds each column with pa.array([], type=...), you can no longer create an empty table for a schema with a tensor column produced by from_numpy_ndarray. Table.from_pylist / RecordBatch.from_pylist with such a schema fail the same way.
The check was added in #50203 (GH-49644) to reject multi-dimensional elements that would need a transpose. It runs before looking at any element, though, so it also rejects inputs that need no transpose, for e.g. an empty list, or if the permutation is the identity permutation.
Repro below:
import numpy as np
import pyarrow as pa
# from_numpy_ndarray always sets a permutation, [0, 1] for C-contiguous input.
t = pa.FixedShapeTensorArray.from_numpy_ndarray(np.zeros((2, 2, 2))).type
schema = pa.schema([("a", t)])
print(pa.__version__, t)
cases = {
"pa.schema(...).empty_table()": lambda: schema.empty_table(),
"pa.array([], type=t)": lambda: pa.array([], type=t),
"pa.array([None], type=t)": lambda: pa.array([None], type=t),
"pa.array([[0, 1, 2, 3]], type=t)": lambda: pa.array([[0, 1, 2, 3]], type=t),
"pa.Table.from_pylist([], schema=schema)": lambda: pa.Table.from_pylist([], schema=schema),
}
for name, fn in cases.items():
try:
fn()
print(f"ok {name}")
except Exception as e:
print(f"FAIL {name}: {type(e).__name__}: {e}")On 24.0.0 every case prints ok. On 25.0.1 and 26.0.1 every case fails:
25.0.1 extension<arrow.fixed_shape_tensor[value_type=double, shape=[2,2], permutation=[0,1]]>
FAIL pa.schema(...).empty_table(): NotImplementedError: Converting a sequence of arrays to a fixed_shape_tensor with a permutation is not supported
FAIL pa.array([], type=t): NotImplementedError: Converting a sequence of arrays to a fixed_shape_tensor with a permutation is not supported
FAIL pa.array([None], type=t): NotImplementedError: Converting a sequence of arrays to a fixed_shape_tensor with a permutation is not supported
FAIL pa.array([[0, 1, 2, 3]], type=t): NotImplementedError: Converting a sequence of arrays to a fixed_shape_tensor with a permutation is not supported
FAIL pa.Table.from_pylist([], schema=schema): NotImplementedError: Converting a sequence of arrays to a fixed_shape_tensor with a permutation is not supported
Expected
These should behave as in 24.0.0. Only inputs that would actually need a transpose (a multi-dimensional element with a non-identity permutation) need to be rejected. Empty input, nulls and flat storage values have no logical layout to apply, and an identity permutation is the same layout as no permutation. Also, it's a plain NotImplementedError rather than pa.ArrowNotImplementedError, so existing except pa.Arrow... handlers don't catch it.
Possible fix
Move the permutation check in array() (array.pxi) into the existing per-element shape loop, and skip it for identity permutations. Moving into loop allows us to prevent raising an error if an empty list is provided as input:
permutation = extension_type.permutation
# Only a non-identity permutation needs the elements to be transposed
is_permuted = (permutation is not None
and list(permutation) != list(range(len(permutation))))
expected_shape = tuple(extension_type.shape)
for element in obj:
shape = getattr(element, "shape", None)
if shape is None or len(shape) < 2:
continue
if is_permuted:
raise NotImplementedError(...)
if tuple(shape) != expected_shape:
raise ValueError(...)Happy to open a PR if this approach sounds right.
Environment
pyarrow 24.0.0 (works), 25.0.1 and 26.0.1 (fail)
Python 3.12, macOS arm64.
Component(s)
Python
Activity