Skip to content

feat(ipc): Add random-access inputs for zero-copy stream reading - #965

Open
rustyconover wants to merge 3 commits into
apache:mainfrom
rustyconover:ipc-reader-zero-copy-buffer
Open

rustyconover wants to merge 3 commits into
apache:mainfrom
rustyconover:ipc-reader-zero-copy-buffer

Conversation

@rustyconover

@rustyconover rustyconover commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Reading an IPC stream from an in-memory buffer currently copies every message body before decoding it. This adds a random-access input interface that lets the existing stream reader retain owned or shared buffers returned by the input, avoiding those copies for buffer-backed inputs.

ArrowIpcInputStream is allocated and populated by callers, so adding optional callbacks to it would leave uninitialized function pointers in existing implementations. The new ArrowIpcRandomAccessInput is separate and has:

  • optional peek_at() to borrow bytes without IO, returning ENOTSUP when unavailable;
  • required read_at() to return an owned buffer, which may be a shared slice or newly read bytes;
  • release() and private_data.

Offsets are absolute. Reads do not advance a reader cursor; the stream reader manages its own position. There are buffer and seekable FILE* initializers, a move function, and a C++ UniqueRandomAccessInput wrapper.

ArrowIpcArrayStreamReaderInitRandomAccess() uses the existing schema, record-batch, and dictionary parser. It reads IPC stream format from offset zero. The file-backed input is an IO implementation, not a new IPC file-format reader.

For example, after filling an initialized ArrowBuffer with IPC stream bytes:

struct ArrowIpcRandomAccessInput input;
int result = ArrowIpcRandomAccessInputInitBuffer(&input, &buffer);
if (result != NANOARROW_OK) {
  ArrowBufferReset(&buffer);
  return result;
}

struct ArrowArrayStream stream;
result = ArrowIpcArrayStreamReaderInitRandomAccess(&stream, &input, NULL);
if (result != NANOARROW_OK) {
  input.release(&input);
  return result;
}
// Consume stream, then ArrowArrayStreamRelease(&stream).

With shared buffers enabled, buffer inputs provide shared slices for complete modern headers and message bodies. Decoded arrays can retain the entire input buffer. This is documented along with the thread-safety requirement when C11 atomics are unavailable. File inputs allocate and read the requested bytes. Existing sequential input implementations continue through the same parser.

A borrowed peek_at() pointer is valid only until the next input callback. The decoder retains header pointers while decoding a body, so complete headers are retained through read_at() instead of keeping a borrowed peek across a later read. Unaligned headers and legacy headers requiring an inserted continuation token use aligned owned storage.

Alignment is checked per typed array buffer while the decoder materializes it, before validation or endian swapping. An aligned message body can still contain an unaligned buffer offset; only those unaligned typed buffers are copied. Byte buffers do not need realignment. This also covers dictionary values.

Tests cover:

  • absolute reads, backwards seeks, EOF, negative offsets and sizes, ownership transfer, and buffers surviving input release;
  • already-shared input, disabled sharing, and misaligned input;
  • buffer, file, and custom callback implementations, including absent/unsupported peek callbacks and peek storage invalidated by the next read;
  • multiple record batches, dictionary lifetimes, truncated bodies, callback failures and invalid callback results;
  • legacy V4 headers and an unaligned int32 buffer inside an aligned message body.

Validation (October 26): macOS arm64 Debug with Arrow C++ 25.0.1, LZ4 and ZSTD: 913 CTest cases, no failures, 168 skipped, primarily unavailable IPC fixture files or unsupported cases. All 27 reader/invalid-input tests also pass under ASan/UBSan. Changed files pass clang-format 16.0.6 and codespell 2.2.5.

Release benchmark on Apple M3: a 10,000,296-byte IPC stream containing one double column with 1,250,000 rows, shared buffers enabled and FULL validation, median of 50 reads after five warmups. The sequential buffer input takes 0.214 ms; the random-access buffer input takes 0.0011 ms. Both measurements include reader initialization, reading through EOS, and release. This fixture primarily measures elimination of the body copy; it is not a general speedup estimate for other schemas.

CI follow-ups: reader tests use the existing MSVC-compatible temporary-file convention, brace assertions to satisfy GCC, and close files on early assertion returns. Intentional moved-from-state and invalid-enum checks use the same narrowly scoped clang-tidy annotations as the existing tests. The reader test translation unit compiles under GCC 16 with -Wall -Wextra -Werror and passes local clang-tidy analyzer checks; all 27 reader/invalid-input tests pass after these updates.

@codecov-commenter

codecov-commenter commented Oct 16, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 75.98039% with 49 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.55%. Comparing base (08d23f6) to head (538259c).

Files with missing lines Patch % Lines
src/nanoarrow/ipc/reader.c 75.13% 16 Missing and 29 partials ⚠️
src/nanoarrow/ipc/decoder.c 75.00% 2 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #965      +/-   ##
==========================================
+ Coverage   78.52%   78.55%   +0.03%     
==========================================
  Files         106      106              
  Lines       17195    17391     +196     
  Branches     2064     2117      +53     
==========================================
+ Hits        13502    13662     +160     
- Misses       2431     2442      +11     
- Partials     1262     1287      +25     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@paleolimbot paleolimbot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a great idea; however, I think we should add a callback to the ArrowIpcInputStream (or update the existing one) to handle this case, which comes up when reading files. Maybe something like

    ArrowErrorCode (*peek_at)(struct ArrowIpcInputStream* stream,
                         int64_t offset_from_current,
                         int64_t buf_size_bytes,
                         const uint8_t** buf,
                         struct ArrowError* error);
    ArrowErrorCode (*read_at)(struct ArrowIpcInputStream* stream,
                         int64_t offset_from_current,
                         int64_t buf_size_bytes, struct ArrowBuffer* out,
                         struct ArrowError* error);

Those callbacks can return ENOTSUP if IO would otherwise be required but the reader could try them (the first variant for metadata to avoid a copy, and the second variant for buffers). There's one other case we eventually have to handle which is firing off a bunch of concurrentread_at and waiting for the result; however, I don't think we need to handle that here.

Also, while bodies usually contain aligned buffers when the bodies themselves are aligned, there's no guarantee of this. We can provide a separate ArrowArrayEnsureAligned() that recursively walks the array and ensures buffers are aligned to at least the fixed width of their type.

@rustyconover

Copy link
Copy Markdown
Contributor Author

Thanks, that makes sense: callbacks on the input stream are a better fit than special-casing the buffer stream, and they cover files too. Two questions before I rework it.

How should the struct gain the callbacks? ArrowIpcInputStream is allocated by callers who fill in only read, release and private_data. The R package does this: r/src/ipc.c ArrowMallocs the struct and sets read/release one at a time. If peek_at/read_at are appended to the struct, those streams (and any out-of-tree ones) will have garbage in the new members, and the reader would call them. Options I can see:

  1. Append the members, document that unsupported callbacks must be NULL, and update R/Python in this PR. That's simple, but out-of-tree implementations silently get undefined behaviour until they're updated.
  2. Keep ArrowIpcInputStream as it is and add a separate struct for random access (peek_at, read_at, release, private_data). The buffer and file inputs implement both, and a reader init function takes the new one. That's more API, but nothing existing changes.

My preference is (2), unless nanoarrow is fine with breaking out-of-tree streams in a release.

Semantics. For peek_at, I'd make the returned pointer valid until the next call on the stream. For read_at, the ArrowBuffer would be a shared slice when the bytes are already in memory, and a newly read buffer otherwise. Both would advance the stream position only when offset_from_current is 0 and the full length is consumed. Does that match what you had in mind, or should offsets be absolute?

Alignment: agreed. I'd write ArrowArrayEnsureAligned() first as its own small core PR: walk the array and copy any buffer that isn't aligned to its type's fixed width. The reader would then call it on arrays that reference a body in place, replacing the 8-byte body-alignment check here. Does that split work for you?

@paleolimbot

Copy link
Copy Markdown
Member

It's a good point, and (2) does seem safer since it's not an obvious compilation break. I am not sure how that interacts with the existing reader in terms of what needs to be rewritten specifically for the random access reader. As long as this doesn't add a fully duplicate path to the array stream reader (which I think can be done), go for it!

Both would advance the stream position

advancing the stream position I think is separate (probably need a seek)

@rustyconover rustyconover changed the title feat(ipc): Reference message bodies in place when reading from a buffer feat(ipc): Add random-access inputs for zero-copy stream reading Oct 26, 2026
@rustyconover
rustyconover force-pushed the ipc-reader-zero-copy-buffer branch from 02c9d14 to 77e8931 Compare October 26, 2026 16:24

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants