Repository navigation
feat: Add ArrowIpcDictionaryMapping to track dictionary identifiers when parsing schema message - #856
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds a small “dictionary encoding side-channel” alongside decoded IPC schemas so the decoder can track dictionary IDs per field (to be used by follow-up work that actually decodes dictionary batches/arrays).
Changes:
- Introduces
ArrowIpcDictionaryEncoding/ArrowIpcDictionaryEncodingsand APIs to decode/set schemas while capturing dictionary IDs. - Propagates dictionary IDs into the decoder’s internal field-walk representation (
ArrowIpcField.dictionary_id) and stores dictionary metadata in footers/readers. - Adds C++
UniqueDictionaryEncodingswrapper support and extends tests for schema + dictionary-ID propagation.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| src/nanoarrow/nanoarrow_ipc.hpp | Adds UniqueDictionaryEncodings RAII wrapper support for the new C struct. |
| src/nanoarrow/nanoarrow_ipc.h | Adds new public dictionary-encoding structs/APIs and decoder “WithDictionaries” variants; extends footer struct. |
| src/nanoarrow/ipc/reader.c | Switches stream reader schema handling to the new “WithDictionaries” decode/set APIs. |
| src/nanoarrow/ipc/ipc_hpp_test.cc | Adds a unit test covering UniqueDictionaryEncodings move/reset semantics. |
| src/nanoarrow/ipc/encoder.c | Initializes/resets footer dictionary-encoding storage. |
| src/nanoarrow/ipc/decoder_test.cc | Extends schema decode test to assert dictionary-encoding lookup + ID propagation into decoder fields. |
| src/nanoarrow/ipc/decoder.c | Implements dictionary-encoding list + schema parsing propagation; adds a new (currently incomplete) dictionary decode guard. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR adds support for extracting and carrying Arrow IPC dictionary encoding metadata (field → dictionary id/kind) alongside a decoded ArrowSchema, so the decoder internals can retain dictionary IDs for future dictionary decoding work.
Changes:
- Introduces
ArrowIpcDictionaryEncoding/ArrowIpcDictionaryEncodingsplus init/append/find/reset APIs for tracking dictionary-encoded fields and their IDs. - Adds
ArrowIpcDecoderDecodeSchemaWithDictionaries()andArrowIpcDecoderSetSchemaWithDictionaries()and wires them through the IPC stream reader and tests. - Propagates dictionary IDs into the decoder’s internal flattened field list and blocks dictionary-encoded field decoding with a clearer error.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| src/nanoarrow/nanoarrow_ipc.hpp | Adds C++ UniqueDictionaryEncodings wrapper + init/move/release hooks. |
| src/nanoarrow/nanoarrow_ipc.h | Adds new dictionary encoding structs/APIs and new decoder entrypoints; extends ArrowIpcFooter with dictionary metadata. |
| src/nanoarrow/ipc/reader.c | Switches schema decode/set to the new “with dictionaries” APIs and ensures cleanup on early exit. |
| src/nanoarrow/ipc/ipc_hpp_test.cc | Adds move/ownership test coverage for UniqueDictionaryEncodings. |
| src/nanoarrow/ipc/encoder.c | Initializes/resets the new footer->dictionaries member. |
| src/nanoarrow/ipc/decoder_test.cc | Extends dictionary schema test to validate exported IDs and schema-setting behavior. |
| src/nanoarrow/ipc/decoder.c | Implements dictionary encoding list APIs; threads dictionary metadata through schema decode/set and decoder internals. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
After #856 we can know which nodes in an ArrowSchema are dictionary encoded and what their IDs are. This PR implements the `ArrowIpcDictionaries` tracker and the actual decode step. Doesn't quite solve #845 because we need one more step (thread the `ArrowIpcDictionaries` through the decode, which involves solving some things about cloning arrays and array views/validation.
This PR adds
struct ArrowIpcDictionaryEncodingandstruct ArrowIpcDictionaryEncodingsthat accompany anArrowSchemato map fields to dictionary IDs. The primary purpose of this is to assign dictionary IDs to the (internal)struct ArrowIpcFieldso that the next PR can make use of that to actually decode them. There is now anArrowIpcDecoderDecodeSchemaWithDictionaries()andArrowIpcDecoderSetSchemaWithDictionaries()to machArrowIpcDecoderDecodeSchema()andArrowIpcDecoderSetSchema()(respectively).This is largely an implementation detail that simply transports information from the Flatbuffers schema to the decoder internals.
Closes #844.