Skip to content

Fix fixed-size lists and renamed struct fields in nested lists and maps - #4436

Open
rahulsmahadev wants to merge 1 commit into
apache:mainfrom
rahulsmahadev:rahul/fixed-size-list-of-struct-append
Open

rahulsmahadev wants to merge 1 commit into
apache:mainfrom
rahulsmahadev:rahul/fixed-size-list-of-struct-append

Conversation

@rahulsmahadev

Copy link
Copy Markdown
Contributor

Rationale for this change

Bug: appending a fixed-size list of structs fails. The struct branch of ArrowProjectionVisitor.list rebuilds the list from its first two buffers, which are the validity and offsets buffers only for a variable-size list. A FixedSizeListArray has no offsets buffer, so the rebuilt array is invalid:

import pyarrow as pa

t = pa.table({"v": pa.array([[{"a": 1}, {"a": 2}]], pa.list_(pa.struct([("a", pa.int32())]), 2))})
catalog.create_table("db.t", schema=t.schema).append(t)
# ArrowInvalid: Non-empty array but offsets are null

Fix: rebuild lists and maps around their projected children, preserving validity and offsets; fixed-size lists retain their list size instead of offsets. Lists and maps nested in other lists or maps were cast with the unprojected children, so a renamed struct field (e.g. a b) came back as null on PyArrow 25 and raised on PyArrow 18. A map with list values had the same gap. This follows #4065's null-list fix; fixed-size lists failed before it too.

Java's lists have no fixed length; Spark writes them through SparkParquetWriters, without an Arrow writer. This fixes PyArrow's shared read/write projection.

Known limit: PyArrow 18 still rejects some null fixed-size lists in its Parquet writer, as it does for fixed_size_list<int>.

Are these changes tested?

Yes, test__to_requested_schema_fixed_size_list_of_structs, test__to_requested_schema_list_of_lists_of_renamed_structs, test__to_requested_schema_map_of_lists_of_renamed_structs, test_append_fixed_size_list_of_structs and test_append_map_of_fixed_size_list_of_structs in tests/io/test_pyarrow.py. All five fail on main (invalid buffers, or a rename that raises on PyArrow 18 and returns nulls on 25) and pass with this change. test_append_map_with_struct_keys passes on main and guards the map rebuild for struct keys (apache/arrow#51029).

Are there any user-facing changes?

Yes, fixed-size lists of structs can be appended, and renamed struct fields in nested lists and maps are preserved on read and write.

Please add the changelog label.

This pull request and its description were written by Isaac.

The list visitor rebuilt struct lists from their first two buffers, which
are not offsets for a FixedSizeListArray, so the append failed. Rebuild
every list and map around its projected children, keeping only the buffers
its type has, so renamed struct fields inside nested lists and maps are no
longer dropped.

Project a struct without a validity mask when it has no nulls, so a rebuilt map with
struct keys stays valid (apache/arrow#51029).

Co-authored-by: Isaac <no-reply@databricks.com>
@rahulsmahadev

Copy link
Copy Markdown
Contributor Author

@rambleraptor, this fixes appending fixed-size lists of structs in ArrowProjectionVisitor. @DG47, this builds on your null-list handling for lists of structs in #4065. Could you take a look when you have a chance?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant