Repository navigation
feat(ipc): Add random-access inputs for zero-copy stream reading - #965
rustyconover wants to merge 3 commits into
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #965 +/- ##
==========================================
+ Coverage 78.52% 78.55% +0.03%
==========================================
Files 106 106
Lines 17195 17391 +196
Branches 2064 2117 +53
==========================================
+ Hits 13502 13662 +160
- Misses 2431 2442 +11
- Partials 1262 1287 +25 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
paleolimbot
left a comment
There was a problem hiding this comment.
This is a great idea; however, I think we should add a callback to the ArrowIpcInputStream (or update the existing one) to handle this case, which comes up when reading files. Maybe something like
ArrowErrorCode (*peek_at)(struct ArrowIpcInputStream* stream,
int64_t offset_from_current,
int64_t buf_size_bytes,
const uint8_t** buf,
struct ArrowError* error);
ArrowErrorCode (*read_at)(struct ArrowIpcInputStream* stream,
int64_t offset_from_current,
int64_t buf_size_bytes, struct ArrowBuffer* out,
struct ArrowError* error);Those callbacks can return ENOTSUP if IO would otherwise be required but the reader could try them (the first variant for metadata to avoid a copy, and the second variant for buffers). There's one other case we eventually have to handle which is firing off a bunch of concurrentread_at and waiting for the result; however, I don't think we need to handle that here.
Also, while bodies usually contain aligned buffers when the bodies themselves are aligned, there's no guarantee of this. We can provide a separate ArrowArrayEnsureAligned() that recursively walks the array and ensures buffers are aligned to at least the fixed width of their type.
|
Thanks, that makes sense: callbacks on the input stream are a better fit than special-casing the buffer stream, and they cover files too. Two questions before I rework it. How should the struct gain the callbacks?
My preference is (2), unless nanoarrow is fine with breaking out-of-tree streams in a release. Semantics. For Alignment: agreed. I'd write |
|
It's a good point, and (2) does seem safer since it's not an obvious compilation break. I am not sure how that interacts with the existing reader in terms of what needs to be rewritten specifically for the random access reader. As long as this doesn't add a fully duplicate path to the array stream reader (which I think can be done), go for it!
advancing the stream position I think is separate (probably need a seek) |
02c9d14 to
77e8931
Compare
Reading an IPC stream from an in-memory buffer currently copies every message body before decoding it. This adds a random-access input interface that lets the existing stream reader retain owned or shared buffers returned by the input, avoiding those copies for buffer-backed inputs.
ArrowIpcInputStreamis allocated and populated by callers, so adding optional callbacks to it would leave uninitialized function pointers in existing implementations. The newArrowIpcRandomAccessInputis separate and has:peek_at()to borrow bytes without IO, returningENOTSUPwhen unavailable;read_at()to return an owned buffer, which may be a shared slice or newly read bytes;release()andprivate_data.Offsets are absolute. Reads do not advance a reader cursor; the stream reader manages its own position. There are buffer and seekable
FILE*initializers, a move function, and a C++UniqueRandomAccessInputwrapper.ArrowIpcArrayStreamReaderInitRandomAccess()uses the existing schema, record-batch, and dictionary parser. It reads IPC stream format from offset zero. The file-backed input is an IO implementation, not a new IPC file-format reader.For example, after filling an initialized
ArrowBufferwith IPC stream bytes:With shared buffers enabled, buffer inputs provide shared slices for complete modern headers and message bodies. Decoded arrays can retain the entire input buffer. This is documented along with the thread-safety requirement when C11 atomics are unavailable. File inputs allocate and read the requested bytes. Existing sequential input implementations continue through the same parser.
A borrowed
peek_at()pointer is valid only until the next input callback. The decoder retains header pointers while decoding a body, so complete headers are retained throughread_at()instead of keeping a borrowed peek across a later read. Unaligned headers and legacy headers requiring an inserted continuation token use aligned owned storage.Alignment is checked per typed array buffer while the decoder materializes it, before validation or endian swapping. An aligned message body can still contain an unaligned buffer offset; only those unaligned typed buffers are copied. Byte buffers do not need realignment. This also covers dictionary values.
Tests cover:
Validation (October 26): macOS arm64 Debug with Arrow C++ 25.0.1, LZ4 and ZSTD: 913 CTest cases, no failures, 168 skipped, primarily unavailable IPC fixture files or unsupported cases. All 27 reader/invalid-input tests also pass under ASan/UBSan. Changed files pass clang-format 16.0.6 and codespell 2.2.5.
Release benchmark on Apple M3: a 10,000,296-byte IPC stream containing one double column with 1,250,000 rows, shared buffers enabled and FULL validation, median of 50 reads after five warmups. The sequential buffer input takes 0.214 ms; the random-access buffer input takes 0.0011 ms. Both measurements include reader initialization, reading through EOS, and release. This fixture primarily measures elimination of the body copy; it is not a general speedup estimate for other schemas.
CI follow-ups: reader tests use the existing MSVC-compatible temporary-file convention, brace assertions to satisfy GCC, and close files on early assertion returns. Intentional moved-from-state and invalid-enum checks use the same narrowly scoped clang-tidy annotations as the existing tests. The reader test translation unit compiles under GCC 16 with -Wall -Wextra -Werror and passes local clang-tidy analyzer checks; all 27 reader/invalid-input tests pass after these updates.