Repository navigation
Fix fixed-size lists and renamed struct fields in nested lists and maps - #4436
Open
rahulsmahadev wants to merge 1 commit into
Open
rahulsmahadev wants to merge 1 commit into
rahulsmahadev wants to merge 1 commit into
Conversation
The list visitor rebuilt struct lists from their first two buffers, which are not offsets for a FixedSizeListArray, so the append failed. Rebuild every list and map around its projected children, keeping only the buffers its type has, so renamed struct fields inside nested lists and maps are no longer dropped. Project a struct without a validity mask when it has no nulls, so a rebuilt map with struct keys stays valid (apache/arrow#51029). Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
Author
|
@rambleraptor, this fixes appending fixed-size lists of structs in |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Bug: appending a fixed-size list of structs fails. The struct branch of
ArrowProjectionVisitor.listrebuilds the list from its first two buffers, which are the validity and offsets buffers only for a variable-size list. AFixedSizeListArrayhas no offsets buffer, so the rebuilt array is invalid:Fix: rebuild lists and maps around their projected children, preserving validity and offsets; fixed-size lists retain their list size instead of offsets. Lists and maps nested in other lists or maps were cast with the unprojected children, so a renamed struct field (e.g.
a b) came back as null on PyArrow 25 and raised on PyArrow 18. A map with list values had the same gap. This follows #4065's null-list fix; fixed-size lists failed before it too.Java's lists have no fixed length; Spark writes them through
SparkParquetWriters, without an Arrow writer. This fixes PyArrow's shared read/write projection.Known limit: PyArrow 18 still rejects some null fixed-size lists in its Parquet writer, as it does for
fixed_size_list<int>.Are these changes tested?
Yes,
test__to_requested_schema_fixed_size_list_of_structs,test__to_requested_schema_list_of_lists_of_renamed_structs,test__to_requested_schema_map_of_lists_of_renamed_structs,test_append_fixed_size_list_of_structsandtest_append_map_of_fixed_size_list_of_structsintests/io/test_pyarrow.py. All five fail on main (invalid buffers, or a rename that raises on PyArrow 18 and returns nulls on 25) and pass with this change.test_append_map_with_struct_keyspasses on main and guards the map rebuild for struct keys (apache/arrow#51029).Are there any user-facing changes?
Yes, fixed-size lists of structs can be appended, and renamed struct fields in nested lists and maps are preserved on read and write.
Please add the
changeloglabel.This pull request and its description were written by Isaac.