Skip to content

Track dictionary IDs when decoding IPC schema #844

Closed

Description

@paleolimbot

In #738 we added and tested support for decoding an IPC schema that contains dictionary-encoded fields; however, the decoder does not yet provide a way to export the dictionary IDs with the schema. We probably need something like:

struct ArrowIpcSchemaDictionaries;

ArrowIpcDecoderDecodeSchemaWithDictionaries(struct ArrowIpcDecoder* decoder, struct ArrowSchema* schema, struct ArrowIpcSchemaDictionaries*);

ArrowIpcDecoderSetSchemaWithDictionaries(struct ArrowIpcDecoder* decoder, struct ArrowSchema* schema, struct ArrowIpcSchemaDictionaries*);

I was thinking that ArrowIpcSchemaDictionaries could be essentially a Vec<(const struct ArrowSchema*, int64_t)> where the pointers correspond to the dictionary pointers in the decoded schema. This allows the setter to locate the dictionary IDs when looping over the schema with the downside the pointers are invalidated if anybody decides to ArrowSchemaDeepCopy(). I'm not also not sure there's a better solution in the presence of potentially nested dictionaries.

Part of #622.

Activity

  1. added a commit that references this issue on Mar 23, 2026
    31a4634
  2. added this to the nanoarrow 0.9.0 milestone on Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions