Skip to content

feat: Add ArrowIpcDictionaryMapping to track dictionary identifiers when parsing schema message - #856

Merged
paleolimbot merged 20 commits into
apache:mainfrom
paleolimbot:decode-ipc-schema-with-dictionaries
Mar 23, 2026
Merged

paleolimbot merged 20 commits into
apache:mainfrom
paleolimbot:decode-ipc-schema-with-dictionaries

Conversation

@paleolimbot

@paleolimbot paleolimbot commented Mar 17, 2026 •

Copy link
Copy Markdown
Member

This PR adds struct ArrowIpcDictionaryEncoding and struct ArrowIpcDictionaryEncodings that accompany an ArrowSchema to map fields to dictionary IDs. The primary purpose of this is to assign dictionary IDs to the (internal) struct ArrowIpcField so that the next PR can make use of that to actually decode them. There is now an ArrowIpcDecoderDecodeSchemaWithDictionaries() and ArrowIpcDecoderSetSchemaWithDictionaries() to mach ArrowIpcDecoderDecodeSchema() and ArrowIpcDecoderSetSchema() (respectively).

This is largely an implementation detail that simply transports information from the Flatbuffers schema to the decoder internals.

Closes #844.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds a small “dictionary encoding side-channel” alongside decoded IPC schemas so the decoder can track dictionary IDs per field (to be used by follow-up work that actually decodes dictionary batches/arrays).

Changes:

  • Introduces ArrowIpcDictionaryEncoding / ArrowIpcDictionaryEncodings and APIs to decode/set schemas while capturing dictionary IDs.
  • Propagates dictionary IDs into the decoder’s internal field-walk representation (ArrowIpcField.dictionary_id) and stores dictionary metadata in footers/readers.
  • Adds C++ UniqueDictionaryEncodings wrapper support and extends tests for schema + dictionary-ID propagation.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
src/nanoarrow/nanoarrow_ipc.hpp Adds UniqueDictionaryEncodings RAII wrapper support for the new C struct.
src/nanoarrow/nanoarrow_ipc.h Adds new public dictionary-encoding structs/APIs and decoder “WithDictionaries” variants; extends footer struct.
src/nanoarrow/ipc/reader.c Switches stream reader schema handling to the new “WithDictionaries” decode/set APIs.
src/nanoarrow/ipc/ipc_hpp_test.cc Adds a unit test covering UniqueDictionaryEncodings move/reset semantics.
src/nanoarrow/ipc/encoder.c Initializes/resets footer dictionary-encoding storage.
src/nanoarrow/ipc/decoder_test.cc Extends schema decode test to assert dictionary-encoding lookup + ID propagation into decoder fields.
src/nanoarrow/ipc/decoder.c Implements dictionary-encoding list + schema parsing propagation; adds a new (currently incomplete) dictionary decode guard.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

Comment thread src/nanoarrow/nanoarrow_ipc.hpp Resolved
Comment thread src/nanoarrow/nanoarrow_ipc.h Resolved
Comment thread src/nanoarrow/ipc/decoder.c Resolved
Comment thread src/nanoarrow/ipc/decoder.c Resolved
paleolimbot and others added 3 commits March 17, 2026 20:55

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds support for extracting and carrying Arrow IPC dictionary encoding metadata (field → dictionary id/kind) alongside a decoded ArrowSchema, so the decoder internals can retain dictionary IDs for future dictionary decoding work.

Changes:

  • Introduces ArrowIpcDictionaryEncoding / ArrowIpcDictionaryEncodings plus init/append/find/reset APIs for tracking dictionary-encoded fields and their IDs.
  • Adds ArrowIpcDecoderDecodeSchemaWithDictionaries() and ArrowIpcDecoderSetSchemaWithDictionaries() and wires them through the IPC stream reader and tests.
  • Propagates dictionary IDs into the decoder’s internal flattened field list and blocks dictionary-encoded field decoding with a clearer error.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
src/nanoarrow/nanoarrow_ipc.hpp Adds C++ UniqueDictionaryEncodings wrapper + init/move/release hooks.
src/nanoarrow/nanoarrow_ipc.h Adds new dictionary encoding structs/APIs and new decoder entrypoints; extends ArrowIpcFooter with dictionary metadata.
src/nanoarrow/ipc/reader.c Switches schema decode/set to the new “with dictionaries” APIs and ensures cleanup on early exit.
src/nanoarrow/ipc/ipc_hpp_test.cc Adds move/ownership test coverage for UniqueDictionaryEncodings.
src/nanoarrow/ipc/encoder.c Initializes/resets the new footer->dictionaries member.
src/nanoarrow/ipc/decoder_test.cc Extends dictionary schema test to validate exported IDs and schema-setting behavior.
src/nanoarrow/ipc/decoder.c Implements dictionary encoding list APIs; threads dictionary metadata through schema decode/set and decoder internals.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

Comment thread src/nanoarrow/nanoarrow_ipc.h Resolved
Comment thread src/nanoarrow/nanoarrow_ipc.h Resolved
Comment thread src/nanoarrow/nanoarrow_ipc.h Resolved
Comment thread src/nanoarrow/ipc/decoder.c Resolved
@paleolimbot
paleolimbot marked this pull request as ready for review March 18, 2026 21:47
@paleolimbot
paleolimbot merged commit 31a4634 into apache:main Mar 23, 2026
42 checks passed
@paleolimbot
paleolimbot deleted the decode-ipc-schema-with-dictionaries branch March 23, 2026 13:49
paleolimbot added a commit that referenced this pull request Apr 7, 2026
After #856 we can know
which nodes in an ArrowSchema are dictionary encoded and what their IDs
are. This PR implements the `ArrowIpcDictionaries` tracker and the
actual decode step.

Doesn't quite solve #845 because we need one more step (thread the
`ArrowIpcDictionaries` through the decode, which involves solving some
things about cloning arrays and array views/validation.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Track dictionary IDs when decoding IPC schema

3 participants