Skip to content

feat(ipc): Support binary view and string view arrays in the IPC reader and writer - #1012

Open
JonSnow1807 wants to merge 1 commit into
apache:mainfrom
JonSnow1807:ipc-string-view
Open

JonSnow1807 wants to merge 1 commit into
apache:mainfrom
JonSnow1807:ipc-string-view

Conversation

@JonSnow1807

Copy link
Copy Markdown
Contributor

Closes #824.

The IPC reader now reads BinaryView and Utf8View arrays and the writer writes them. The decoder checks each RecordBatch's variadicBufferCounts (one count per view field, each within the message's buffer count) and adds the variadic data buffers to the output using the buffer support from #808. The encoder writes the view buffers and the variadic buffers, and writes variadicBufferCounts when a message has a view array. Dictionary-encoded view types work too. View batches from a stream of the other byte order return ENOTSUP, as in Arrow C++.

Tests cover round trips (nulls, inlined and out-of-line values, many variadic buffers, nested types, dictionaries, LZ4 and ZSTD, single-field decoding), malformed counts, and Arrow C++ in both directions. generated_binary_view.stream now reads, and pyarrow 26.0.1 round trips match value by value.

Checking that a view's buffer index and offset are in range is #982's job. With this PR alone a corrupted view passes FULL validation and a corrupted delta dictionary crashes. With #982 both are rejected, so it may be best to merge this after #982.

Code drafted by both human and AI, and reviewed by human.

…er and writer

The IPC reader rejected the BinaryView and Utf8View types and the writer
could not encode them. A binary view or string view field has a validity
buffer, a views buffer, and a number of variadic data buffers that can
change from one RecordBatch to the next, given by the field's entry in
RecordBatch.variadicBufferCounts.

The decoder now maps both types, checks that a RecordBatch has one
variadicBufferCounts entry per view field and that the counts add up to
the number of buffers in the message, and points each array view at its
variadic buffers. The buffers of a field that is decoded on its own start
after the variadic buffers of any view fields before it. The scratch
ArrowArray holds the variadic buffer pointers and sizes and any buffer that
has to be allocated (e.g., to decompress it), using ArrowArrayBuffer() for
the variadic buffers.

The encoder writes the views buffer and the variadic buffers of each view
array (but not the C data interface's buffer of variadic buffer sizes) and
the variadicBufferCounts of RecordBatch and DictionaryBatch messages that
have view arrays. As in Arrow C++, view arrays from a message of the other
endianness are not supported.
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 81.73077% with 19 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.61%. Comparing base (ee22361) to head (f7c8cb2).

Files with missing lines Patch % Lines
src/nanoarrow/ipc/decoder.c 84.21% 3 Missing and 9 partials ⚠️
src/nanoarrow/ipc/encoder.c 75.00% 0 Missing and 7 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1012      +/-   ##
==========================================
+ Coverage   78.55%   78.61%   +0.06%     
==========================================
  Files         106      106              
  Lines       17297    17378      +81     
  Branches     2066     2101      +35     
==========================================
+ Hits        13587    13662      +75     
+ Misses       2446     2440       -6     
- Partials     1264     1276      +12     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

StringView not supported in IPC reader or writer

2 participants