Skip to content

feat(ipc): Support run-end encoded arrays in the IPC reader - #996

Merged
paleolimbot merged 2 commits into
apache:mainfrom
ump45nose:feat/833-ipc-run-end-encoded
Nov 6, 2026
Merged

paleolimbot merged 2 commits into
apache:mainfrom
ump45nose:feat/833-ipc-run-end-encoded

Conversation

@ump45nose

Copy link
Copy Markdown
Contributor

Closes #833.

The IPC encoder already writes the RunEndEncoded type, but the decoder stopped at the schema with "RunEndEncoded not yet supported in IPC reader". The RunEndEncoded type table has no parameters and a run-end encoded array has no buffers of its own (only the run_ends and values children), so the reader only needs to map the type to the +r format. The existing child field and buffer handling already treats it like the other buffer-less layouts, and validation of the run ends is the existing ArrowArrayView logic. The new code also checks that the field has exactly two children so a malformed schema gets a clear error.

Testing

  • Added run_end_encoded(int16, utf8), run_end_encoded(int32, int64) and run_end_encoded(int64, float64) to the type list used by the decoder roundtrip tests (Arrow C++ schema/array -> IPC -> nanoarrow, and nanoarrow -> IPC -> nanoarrow).
  • Switched cpp-21.0.0/generated_run_end_encoded.stream from NotSupported to OK in files_test.cc, so the arrow-testing file is read and compared with Arrow C++'s read, and nanoarrow's written stream is read back by Arrow C++. (The JSON check for that file is still skipped because the testing JSON reader does not know run-end encoded yet.)
  • Without the decoder change, 6 of the new roundtrip tests and the file test fail; with it they pass.
  • Debug build with NANOARROW_IPC=ON NANOARROW_BUILD_TESTS=ON NANOARROW_BUILD_TESTS_WITH_ARROW=ON against Arrow C++ 26.0.0 and current apache/arrow-testing: ctest 896/896 passed.
  • clang-format 16.0.6 (the pre-commit version) applied.

This PR was prepared with AI assistance; I reviewed the changes and ran the checks above.

The IPC encoder already wrote the RunEndEncoded type, but the decoder
rejected it with ENOTSUP. The RunEndEncoded type table has no
parameters and the array has no buffers of its own, only the run_ends
and values children, so map it to the "+r" format and let the existing
child and buffer handling do the rest.

Closes apache#833.
@codecov-commenter

codecov-commenter commented Oct 24, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 50.00000% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.49%. Comparing base (57a5604) to head (af6f4d7).

Files with missing lines Patch % Lines
src/nanoarrow/ipc/decoder.c 50.00% 2 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #996      +/-   ##
==========================================
+ Coverage   78.48%   78.49%   +0.01%     
==========================================
  Files         106      106              
  Lines       17231    17235       +4     
  Branches     2064     2065       +1     
==========================================
+ Hits        13523    13529       +6     
+ Misses       2446     2442       -4     
- Partials     1262     1264       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@paleolimbot paleolimbot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@paleolimbot
paleolimbot merged commit ea907d2 into apache:main Nov 6, 2026
44 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RunEndEncoded type not supported in IPC reader or writer

3 participants