Repository navigation
feat(ipc): Support custom_metadata on Message for IPC - #924
Merged
paleolimbot merged 2 commits intoAug 23, 2026
Merged
Conversation
Adds encoder and decoder support for `Message.custom_metadata`, the per-message key/value metadata that is distinct from Schema and Field metadata in the schema. Encoder: `ArrowIpcEncoderSetMessageMetadata()` attaches metadata to the next message encoded by `ArrowIpcEncoderEncodeSchema()` or `ArrowIpcEncoderEncodeSimpleRecordBatch()`, taking nanoarrow's packed metadata representation so that `ArrowMetadataBuilder*` can produce it. `ArrowIpcEncodeMetadata()` now takes packed metadata rather than an `ArrowSchema` so it can be reused from the Message path. Decoder: `ArrowIpcDecoderGetMessageMetadata()` copies the metadata of the message whose header was last decoded into an `ArrowBuffer`; `ArrowIpcDecoderGetMessageMetadataValue()` and `ArrowIpcDecoderVisitMessageMetadata()` read keys and values in place without copying. Tests cover round trips through the encoder/decoder and interoperability with Arrow C++ in both directions. Closes apache#923 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #924 +/- ##
==========================================
- Coverage 79.96% 79.30% -0.67%
==========================================
Files 105 106 +1
Lines 15461 16223 +762
Branches 1738 1882 +144
==========================================
+ Hits 12364 12865 +501
- Misses 1998 2177 +179
- Partials 1099 1181 +82 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
paleolimbot
approved these changes
Aug 19, 2026
paleolimbot
left a comment
Member
There was a problem hiding this comment.
One API suggestion, but this addition follow the existing patterns, is tested, and adds nice functionality. Thank you!
Per review: - `ArrowIpcEncoderSetMessageMetadata()` now takes a `struct ArrowBuffer*` and moves it into the encoder rather than copying a `const char*`. The buffer is what `ArrowMetadataBuilder*` produces, so the common case no longer copies. A buffer too small to hold a key count is treated as empty. - The decoder's metadata accessors are written in terms of a single KeyValue-vector visitor, which also backs the packing used for Schema and Field metadata. That consolidation replaces `strlen()` with `flatbuffers_string_len()` everywhere metadata is decoded, which fixes a crash: `KeyValue.key` and `KeyValue.value` are optional fields, so a flatbuffer-verified message may contain a KeyValue with no key at all. `ArrowIpcDecoderDecodeSchema()` then called `strlen(NULL)` and segfaulted on input that had passed verification. An absent key or value now decodes as an empty string, and values containing embedded nulls are no longer truncated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
paleolimbot
pushed a commit
that referenced
this pull request
Sep 10, 2026
Follow-up to #924. This removes ArrowIpcDecoderGetMessageMetadataValue() before the API ships in a release. Callers that need one key can use ArrowMetadataGetValue() with the packed metadata returned by ArrowIpcDecoderGetMessageMetadata(); callers that need zero-copy iteration can continue using ArrowIpcDecoderVisitMessageMetadata(). It also scopes the visitor callback type as ArrowIpcMessageMetadataVisitFunction and replaces the hand-generated keyless KeyValue FlatBuffer test fixture with a flatcc-built message. Tests: - IPC-enabled CMake build - ctest: 365 tests passed (16 codec-dependent tests skipped) - git diff --check - clang-format on changed lines --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
paleolimbot
pushed a commit
that referenced
this pull request
Oct 20, 2026
`ArrowIpcEncoderSetMessageMetadata()` (#924, narrowed in #937) attaches `Message::custom_metadata` to the next message an encoder encodes. An `ArrowIpcWriter` keeps its encoder private, though, so a caller using the writer has no way to attach metadata to the RecordBatch messages it writes. The only option is to drop down to the encoder and redo the writer's framing, footer bookkeeping and dictionary handling by hand. This PR adds `ArrowIpcWriterSetMessageMetadata(writer, metadata, error)`. The writer holds the metadata and attaches it to the next RecordBatch message written by `ArrowIpcWriterWriteArrayView()` or `ArrowIpcWriterWriteArrayStream()`. The metadata applies to that one message. Passing `NULL` clears metadata that has been set but not yet written. Like the encoder function, it takes ownership of the buffer. The metadata is held by the writer rather than set on the encoder right away. Setting it on the encoder would attach it to whichever message is encoded next, and that can be a DictionaryBatch written ahead of the batch. This happens both with an explicit `ArrowIpcWriterWriteDictionaryBatch()` call and inside `ArrowIpcWriterWriteArrayStream()`. RecordBatch-level metadata belongs on the RecordBatch, so the writer passes it to the encoder just before encoding that batch. Things worth a look in review: - With `WriteArrayStream()` the metadata lands on the first RecordBatch only. Per-batch metadata over a stream would need a callback or a per-array side channel, which seemed out of scope. The batch-at-a-time pattern is `SetMessageMetadata()` + `WriteArrayView()`. - #928 makes encoder-level metadata apply to DictionaryBatch messages too. The two compose: this PR never sets encoder metadata until right before the RecordBatch is encoded. They touch neighbouring code in `writer.c`, so whichever lands second will need a small rebase. I can do that. Motivation: I'm moving the IPC layer of a DuckDB extension from Arrow C++ to nanoarrow. Its RPC protocol carries per-batch state in RecordBatch `custom_metadata` (what Arrow C++ writes for `RecordBatchWriter::WriteRecordBatch(batch, custom_metadata)`), and this was the one thing the writer couldn't express. Tests: - `MessageMetadataAttachesToNextRecordBatch` sets metadata, writes a DictionaryBatch and then a RecordBatch, and checks that only the RecordBatch carries it. It also checks that the following batch doesn't inherit the metadata, and that set-then-cleared metadata is never written. - `MessageMetadataWithArrayStream` covers `WriteArrayStream()` writing its own dictionaries. ctest passes locally (macOS arm64, Debug, ZSTD/LZ4, 390 tests). The IPC writer, encoder and files tests are clean under ASan/UBSan. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #923.
Adds encoder and decoder support for
Message.custom_metadata— the per-messagekey/value metadata that is separate from the Schema/Field metadata carried in the schema.
Encoder
metadataholds nanoarrow's packed representation (the same asArrowSchema.metadata, soArrowMetadataBuilder*can produce it and no new representation is introduced). Theencoder takes ownership by moving the buffer, so the common case doesn't copy. It applies
to the next message encoded by
ArrowIpcEncoderEncodeSchema()orArrowIpcEncoderEncodeSimpleRecordBatch()and is cleared once that message is encoded.NULL(or metadata with no keys) clears it, and nocustom_metadatafield is written atall in that case, so the encoded bytes are unchanged for callers that don't use this.
ArrowIpcEncodeMetadata()now takes the packedconst char* metadatainstead of aconst struct ArrowSchema*so it is reusable from the Message path; behaviour at theSchema/Field call sites is unchanged.
Decoder
Per @paleolimbot's review comment on the issue, the copying accessor writes to a
struct ArrowBuffer*(whatArrowMetadataBuilder*builds), and the get-by-name andvisit-in-place variants avoid the copy for large metadata. All three are implemented on
top of one internal KeyValue-vector visitor, which also backs the packing used for Schema
and Field metadata. If the message has no
custom_metadata,outis left empty so(const char*)out->dataisNULLand can bepassed straight to
ArrowSchemaSetMetadata()/ArrowMetadataReaderInit(). A missing keyleaves
value_outunmodified, matchingArrowMetadataGetValue().The metadata is recorded by both
ArrowIpcDecoderVerifyHeader()andArrowIpcDecoderDecodeHeader()and applies to any message type, sinceMessage.custom_metadatais a field ofMessagerather than ofRecordBatch.Tests
encoder_test.cc: round trip of message metadata through encoder → decoder for bothRecordBatch and Schema messages, that Schema-message metadata stays distinct from the
contained schema's own metadata, that the metadata applies to exactly one message, that
empty/
NULL/cleared metadata produces byte-identical output to not calling the setter,and that an erroring visitor stops the visit and propagates its error.
decoder_test.cc: interop with Arrow C++ in both directions —arrow::ipc::ReadMessage()reads back what nanoarrow writes, and nanoarrow reads the metadata written by
RecordBatchWriter::WriteRecordBatch(batch, custom_metadata).Verified locally against Arrow C++ 25.0.1 (
-DNANOARROW_BUILD_TESTS_WITH_ARROW=ON).Also fixed here: a crash decoding metadata with an absent key
KeyValue.keyandKeyValue.valueare optional fields inSchema.fbs(the generatedverifier agrees:
flatcc_verify_string_field(td, 0, 0)), so a flatbuffer-verified messagemay contain a
KeyValuewith no key at all. The decoder usedstrlen()on it, soArrowIpcDecoderDecodeSchema()segfaulted (SIGSEGV, confirmed onmain) on input that hadpassed verification — reachable from any untrusted stream via
ArrowIpcArrayStreamReader.Consolidating the iteration replaced
strlen()withflatbuffers_string_len(), which isNULL-safe, so an absent key or value now decodes as an empty string. The same change stops
metadata values containing embedded nulls from being silently truncated, which the packed
representation round-trips fine.
NanoarrowIpcDecodeMetadataWithoutKeycovers it andcrashes without the fix.
Happy to split this into its own PR if you'd rather keep this one purely additive — it
landed here because it's the same lines the consolidation rewrites.
Out of scope
ArrowIpcArrayStreamReader/ArrowIpcWriterare unchanged:ArrowArrayStreamhasnowhere to put per-batch metadata, so as discussed in the issue this first pass exposes the
feature only at the encoder/decoder layer.