Repository navigation
fix(python): Don't crash when shallow copying string/binary view arrays with variadic buffers - #1018
Merged
paleolimbot merged 1 commit intoNov 16, 2026
Conversation
…ys with variadic buffers c_array_shallow_copy() called ArrowArrayAddVariadicBuffers() on the uninitialized output array instead of the array being built, so any string or binary view array with a non-inlined value crashed and the copy loop wrote past the end of the buffer arrays. Pass tmp instead and add a regression test with values longer than 12 bytes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
c_array_shallow_copy()reserves the extra buffers needed by a string view or binary view array by callingArrowArrayAddVariadicBuffers(), but it calls it ondst(the not-yet-initialized output array) instead of ontmp(the array it is building).dstis usually uninitialized memory at that point, so this dereferences a garbageprivate_datapointer, andtmpnever gets room for the variadic buffers, so the loop that follows writes past the end of its buffer arrays.Any string view or binary view array with at least one variadic data buffer goes through this path, i.e. any array with a value longer than 12 bytes, and every such array exported by pyarrow (which always includes a data buffer). The process segfaults or aborts, for example:
The same crash happens for
na.Array(pa.array([...], pa.string_view())).to_pylist(), for exporting such an array to pyarrow via__arrow_c_array__, and when a view array is used as a child inc_array_from_buffers(). The branch was added in #808, so 0.8.0 and 0.9.0 are affected (I also see the crash with the 0.9.0 wheel). The existing iterator tests only use values of 12 bytes or fewer, which are inlined and give three buffers, so they never reach this branch.The fix passes
tmpinstead ofdst. With it, the examples above return the same values as pyarrow'sto_pylist(), and buffers are still released once the copies are dropped. I added a regression test that iterates and slices string view and binary view arrays with a non-inlined value; it crashes the test process without the fix and passes with it. The rest of the Python test suite passes.This change was drafted with Claude Code (claude-opus-5-5), which found the bug and generated the fix, the regression test and this description; the test results above are from the runs described.