Repository navigation
feat(ipc): Support binary view and string view arrays in the IPC reader and writer - #1012
Open
JonSnow1807 wants to merge 1 commit into
Open
JonSnow1807 wants to merge 1 commit into
JonSnow1807 wants to merge 1 commit into
Conversation
…er and writer The IPC reader rejected the BinaryView and Utf8View types and the writer could not encode them. A binary view or string view field has a validity buffer, a views buffer, and a number of variadic data buffers that can change from one RecordBatch to the next, given by the field's entry in RecordBatch.variadicBufferCounts. The decoder now maps both types, checks that a RecordBatch has one variadicBufferCounts entry per view field and that the counts add up to the number of buffers in the message, and points each array view at its variadic buffers. The buffers of a field that is decoded on its own start after the variadic buffers of any view fields before it. The scratch ArrowArray holds the variadic buffer pointers and sizes and any buffer that has to be allocated (e.g., to decompress it), using ArrowArrayBuffer() for the variadic buffers. The encoder writes the views buffer and the variadic buffers of each view array (but not the C data interface's buffer of variadic buffer sizes) and the variadicBufferCounts of RecordBatch and DictionaryBatch messages that have view arrays. As in Arrow C++, view arrays from a message of the other endianness are not supported.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1012 +/- ##
==========================================
+ Coverage 78.55% 78.61% +0.06%
==========================================
Files 106 106
Lines 17297 17378 +81
Branches 2066 2101 +35
==========================================
+ Hits 13587 13662 +75
+ Misses 2446 2440 -6
- Partials 1264 1276 +12 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #824.
The IPC reader now reads BinaryView and Utf8View arrays and the writer writes them. The decoder checks each RecordBatch's
variadicBufferCounts(one count per view field, each within the message's buffer count) and adds the variadic data buffers to the output using the buffer support from #808. The encoder writes the view buffers and the variadic buffers, and writesvariadicBufferCountswhen a message has a view array. Dictionary-encoded view types work too. View batches from a stream of the other byte order return ENOTSUP, as in Arrow C++.Tests cover round trips (nulls, inlined and out-of-line values, many variadic buffers, nested types, dictionaries, LZ4 and ZSTD, single-field decoding), malformed counts, and Arrow C++ in both directions.
generated_binary_view.streamnow reads, and pyarrow 26.0.1 round trips match value by value.Checking that a view's buffer index and offset are in range is #982's job. With this PR alone a corrupted view passes FULL validation and a corrupted delta dictionary crashes. With #982 both are rejected, so it may be best to merge this after #982.
Code drafted by both human and AI, and reviewed by human.