Skip to content

fix(python): Don't crash when shallow copying string/binary view arrays with variadic buffers - #1018

Merged
paleolimbot merged 1 commit into
apache:mainfrom
0z5a:fix/python-view-variadic-shallow-copy
Nov 16, 2026
Merged

paleolimbot merged 1 commit into
apache:mainfrom
0z5a:fix/python-view-variadic-shallow-copy

Conversation

@0z5a

@0z5a 0z5a commented Nov 15, 2026

Copy link
Copy Markdown
Contributor

c_array_shallow_copy() reserves the extra buffers needed by a string view or binary view array by calling ArrowArrayAddVariadicBuffers(), but it calls it on dst (the not-yet-initialized output array) instead of on tmp (the array it is building). dst is usually uninitialized memory at that point, so this dereferences a garbage private_data pointer, and tmp never gets room for the variadic buffers, so the loop that follows writes past the end of its buffer arrays.

Any string view or binary view array with at least one variadic data buffer goes through this path, i.e. any array with a value longer than 12 bytes, and every such array exported by pyarrow (which always includes a data buffer). The process segfaults or aborts, for example:

import nanoarrow as na

array = na.c_array(["ab", None, "a string longer than 12 bytes"], na.string_view())
list(na.iterator.iter_py(array))  # crashes
array[1:]                         # crashes

The same crash happens for na.Array(pa.array([...], pa.string_view())).to_pylist(), for exporting such an array to pyarrow via __arrow_c_array__, and when a view array is used as a child in c_array_from_buffers(). The branch was added in #808, so 0.8.0 and 0.9.0 are affected (I also see the crash with the 0.9.0 wheel). The existing iterator tests only use values of 12 bytes or fewer, which are inlined and give three buffers, so they never reach this branch.

The fix passes tmp instead of dst. With it, the examples above return the same values as pyarrow's to_pylist(), and buffers are still released once the copies are dropped. I added a regression test that iterates and slices string view and binary view arrays with a non-inlined value; it crashes the test process without the fix and passes with it. The rest of the Python test suite passes.

This change was drafted with Claude Code (claude-opus-5-5), which found the bug and generated the fix, the regression test and this description; the test results above are from the runs described.

…ys with variadic buffers

c_array_shallow_copy() called ArrowArrayAddVariadicBuffers() on the
uninitialized output array instead of the array being built, so any
string or binary view array with a non-inlined value crashed and the
copy loop wrote past the end of the buffer arrays. Pass tmp instead and
add a regression test with values longer than 12 bytes.

@paleolimbot paleolimbot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@paleolimbot
paleolimbot merged commit 91be268 into apache:main Nov 16, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants