Skip to content

feat(ipc): Support custom_metadata on Message for IPC - #924

Merged
paleolimbot merged 2 commits into
apache:mainfrom
rustyconover:feat/ipc-message-custom-metadata
Aug 23, 2026
Merged

paleolimbot merged 2 commits into
apache:mainfrom
rustyconover:feat/ipc-message-custom-metadata

Conversation

@rustyconover

@rustyconover rustyconover commented Aug 17, 2026 •

Copy link
Copy Markdown
Contributor

Closes #923.

Adds encoder and decoder support for Message.custom_metadata — the per-message
key/value metadata that is separate from the Schema/Field metadata carried in the schema.

Encoder

ArrowErrorCode ArrowIpcEncoderSetMessageMetadata(struct ArrowIpcEncoder* encoder,
                                                 struct ArrowBuffer* metadata,
                                                 struct ArrowError* error);

metadata holds nanoarrow's packed representation (the same as ArrowSchema.metadata, so
ArrowMetadataBuilder* can produce it and no new representation is introduced). The
encoder takes ownership by moving the buffer, so the common case doesn't copy. It applies
to the next message encoded by ArrowIpcEncoderEncodeSchema() or
ArrowIpcEncoderEncodeSimpleRecordBatch() and is cleared once that message is encoded.
NULL (or metadata with no keys) clears it, and no custom_metadata field is written at
all in that case, so the encoded bytes are unchanged for callers that don't use this.

ArrowIpcEncodeMetadata() now takes the packed const char* metadata instead of a
const struct ArrowSchema* so it is reusable from the Message path; behaviour at the
Schema/Field call sites is unchanged.

Decoder

// Copy, using the same packed representation
ArrowErrorCode ArrowIpcDecoderGetMessageMetadata(struct ArrowIpcDecoder* decoder,
                                                 struct ArrowBuffer* out,
                                                 struct ArrowError* error);

// Zero copy: both borrow from the message header until the next header is decoded
ArrowErrorCode ArrowIpcDecoderGetMessageMetadataValue(struct ArrowIpcDecoder* decoder,
                                                      struct ArrowStringView key,
                                                      struct ArrowStringView* value_out,
                                                      struct ArrowError* error);

ArrowErrorCode ArrowIpcDecoderVisitMessageMetadata(struct ArrowIpcDecoder* decoder,
                                                   ArrowIpcMetadataVisitFunction visit,
                                                   void* private_data,
                                                   struct ArrowError* error);

Per @paleolimbot's review comment on the issue, the copying accessor writes to a
struct ArrowBuffer* (what ArrowMetadataBuilder* builds), and the get-by-name and
visit-in-place variants avoid the copy for large metadata. All three are implemented on
top of one internal KeyValue-vector visitor, which also backs the packing used for Schema
and Field metadata. If the message has no
custom_metadata, out is left empty so (const char*)out->data is NULL and can be
passed straight to ArrowSchemaSetMetadata() / ArrowMetadataReaderInit(). A missing key
leaves value_out unmodified, matching ArrowMetadataGetValue().

The metadata is recorded by both ArrowIpcDecoderVerifyHeader() and
ArrowIpcDecoderDecodeHeader() and applies to any message type, since
Message.custom_metadata is a field of Message rather than of RecordBatch.

Tests

  • encoder_test.cc: round trip of message metadata through encoder → decoder for both
    RecordBatch and Schema messages, that Schema-message metadata stays distinct from the
    contained schema's own metadata, that the metadata applies to exactly one message, that
    empty/NULL/cleared metadata produces byte-identical output to not calling the setter,
    and that an erroring visitor stops the visit and propagates its error.
  • decoder_test.cc: interop with Arrow C++ in both directions — arrow::ipc::ReadMessage()
    reads back what nanoarrow writes, and nanoarrow reads the metadata written by
    RecordBatchWriter::WriteRecordBatch(batch, custom_metadata).

Verified locally against Arrow C++ 25.0.1 (-DNANOARROW_BUILD_TESTS_WITH_ARROW=ON).

Also fixed here: a crash decoding metadata with an absent key

KeyValue.key and KeyValue.value are optional fields in Schema.fbs (the generated
verifier agrees: flatcc_verify_string_field(td, 0, 0)), so a flatbuffer-verified message
may contain a KeyValue with no key at all. The decoder used strlen() on it, so
ArrowIpcDecoderDecodeSchema() segfaulted (SIGSEGV, confirmed on main) on input that had
passed verification — reachable from any untrusted stream via ArrowIpcArrayStreamReader.

Consolidating the iteration replaced strlen() with flatbuffers_string_len(), which is
NULL-safe, so an absent key or value now decodes as an empty string. The same change stops
metadata values containing embedded nulls from being silently truncated, which the packed
representation round-trips fine. NanoarrowIpcDecodeMetadataWithoutKey covers it and
crashes without the fix.

Happy to split this into its own PR if you'd rather keep this one purely additive — it
landed here because it's the same lines the consolidation rewrites.

Out of scope

ArrowIpcArrayStreamReader / ArrowIpcWriter are unchanged: ArrowArrayStream has
nowhere to put per-batch metadata, so as discussed in the issue this first pass exposes the
feature only at the encoder/decoder layer.

Adds encoder and decoder support for `Message.custom_metadata`, the
per-message key/value metadata that is distinct from Schema and Field
metadata in the schema.

Encoder: `ArrowIpcEncoderSetMessageMetadata()` attaches metadata to the
next message encoded by `ArrowIpcEncoderEncodeSchema()` or
`ArrowIpcEncoderEncodeSimpleRecordBatch()`, taking nanoarrow's packed
metadata representation so that `ArrowMetadataBuilder*` can produce it.
`ArrowIpcEncodeMetadata()` now takes packed metadata rather than an
`ArrowSchema` so it can be reused from the Message path.

Decoder: `ArrowIpcDecoderGetMessageMetadata()` copies the metadata of the
message whose header was last decoded into an `ArrowBuffer`;
`ArrowIpcDecoderGetMessageMetadataValue()` and
`ArrowIpcDecoderVisitMessageMetadata()` read keys and values in place
without copying.

Tests cover round trips through the encoder/decoder and interoperability
with Arrow C++ in both directions.

Closes apache#923

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@codecov-commenter

codecov-commenter commented Aug 18, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 75.58140% with 21 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.30%. Comparing base (f11d517) to head (2ee3618).
⚠️ Report is 80 commits behind head on main.

Files with missing lines Patch % Lines
src/nanoarrow/ipc/encoder.c 60.71% 1 Missing and 10 partials ⚠️
src/nanoarrow/ipc/decoder.c 82.75% 6 Missing and 4 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #924      +/-   ##
==========================================
- Coverage   79.96%   79.30%   -0.67%     
==========================================
  Files         105      106       +1     
  Lines       15461    16223     +762     
  Branches     1738     1882     +144     
==========================================
+ Hits        12364    12865     +501     
- Misses       1998     2177     +179     
- Partials     1099     1181      +82     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@paleolimbot paleolimbot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One API suggestion, but this addition follow the existing patterns, is tested, and adds nice functionality. Thank you!

Comment thread src/nanoarrow/nanoarrow_ipc.h Resolved
Comment thread src/nanoarrow/ipc/decoder.c Resolved
Per review:

- `ArrowIpcEncoderSetMessageMetadata()` now takes a `struct ArrowBuffer*`
  and moves it into the encoder rather than copying a `const char*`. The
  buffer is what `ArrowMetadataBuilder*` produces, so the common case no
  longer copies. A buffer too small to hold a key count is treated as
  empty.
- The decoder's metadata accessors are written in terms of a single
  KeyValue-vector visitor, which also backs the packing used for Schema
  and Field metadata.

That consolidation replaces `strlen()` with `flatbuffers_string_len()`
everywhere metadata is decoded, which fixes a crash: `KeyValue.key` and
`KeyValue.value` are optional fields, so a flatbuffer-verified message may
contain a KeyValue with no key at all. `ArrowIpcDecoderDecodeSchema()`
then called `strlen(NULL)` and segfaulted on input that had passed
verification. An absent key or value now decodes as an empty string, and
values containing embedded nulls are no longer truncated.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@paleolimbot paleolimbot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@paleolimbot
paleolimbot merged commit 3f82406 into apache:main Aug 23, 2026
43 checks passed
paleolimbot pushed a commit that referenced this pull request Sep 10, 2026
Follow-up to #924.

This removes ArrowIpcDecoderGetMessageMetadataValue() before the API
ships in a release. Callers that need one key can use
ArrowMetadataGetValue() with the packed metadata returned by
ArrowIpcDecoderGetMessageMetadata(); callers that need zero-copy
iteration can continue using ArrowIpcDecoderVisitMessageMetadata().

It also scopes the visitor callback type as
ArrowIpcMessageMetadataVisitFunction and replaces the hand-generated
keyless KeyValue FlatBuffer test fixture with a flatcc-built message.

Tests:
- IPC-enabled CMake build
- ctest: 365 tests passed (16 codec-dependent tests skipped)
- git diff --check
- clang-format on changed lines

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
paleolimbot pushed a commit that referenced this pull request Oct 20, 2026
`ArrowIpcEncoderSetMessageMetadata()` (#924, narrowed in #937) attaches
`Message::custom_metadata` to the next message an encoder encodes. An
`ArrowIpcWriter` keeps its encoder private, though, so a caller using
the writer has no way to attach metadata to the RecordBatch messages it
writes. The only option is to drop down to the encoder and redo the
writer's framing, footer bookkeeping and dictionary handling by hand.

This PR adds `ArrowIpcWriterSetMessageMetadata(writer, metadata,
error)`. The writer holds the metadata and attaches it to the next
RecordBatch message written by `ArrowIpcWriterWriteArrayView()` or
`ArrowIpcWriterWriteArrayStream()`. The metadata applies to that one
message. Passing `NULL` clears metadata that has been set but not yet
written. Like the encoder function, it takes ownership of the buffer.

The metadata is held by the writer rather than set on the encoder right
away. Setting it on the encoder would attach it to whichever message is
encoded next, and that can be a DictionaryBatch written ahead of the
batch. This happens both with an explicit
`ArrowIpcWriterWriteDictionaryBatch()` call and inside
`ArrowIpcWriterWriteArrayStream()`. RecordBatch-level metadata belongs
on the RecordBatch, so the writer passes it to the encoder just before
encoding that batch.

Things worth a look in review:

- With `WriteArrayStream()` the metadata lands on the first RecordBatch
only. Per-batch metadata over a stream would need a callback or a
per-array side channel, which seemed out of scope. The batch-at-a-time
pattern is `SetMessageMetadata()` + `WriteArrayView()`.
- #928 makes encoder-level metadata apply to DictionaryBatch messages
too. The two compose: this PR never sets encoder metadata until right
before the RecordBatch is encoded. They touch neighbouring code in
`writer.c`, so whichever lands second will need a small rebase. I can do
that.

Motivation: I'm moving the IPC layer of a DuckDB extension from Arrow
C++ to nanoarrow. Its RPC protocol carries per-batch state in
RecordBatch `custom_metadata` (what Arrow C++ writes for
`RecordBatchWriter::WriteRecordBatch(batch, custom_metadata)`), and this
was the one thing the writer couldn't express.

Tests:
- `MessageMetadataAttachesToNextRecordBatch` sets metadata, writes a
DictionaryBatch and then a RecordBatch, and checks that only the
RecordBatch carries it. It also checks that the following batch doesn't
inherit the metadata, and that set-then-cleared metadata is never
written.
- `MessageMetadataWithArrayStream` covers `WriteArrayStream()` writing
its own dictionaries.

ctest passes locally (macOS arm64, Debug, ZSTD/LZ4, 390 tests). The IPC
writer, encoder and files tests are clean under ASan/UBSan.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support custom_metadata on RecordBatch messages for IPC

3 participants