Skip to content

Decode dictionary-encoded arrays in IPC decoder #845

Closed

Description

@paleolimbot

In #738 we added support for dictionary encoded schemas and in #844 we will add support for connecting those schemas with the dictionary IDs that appear in the dictionary batch messages.

Decoding dictionary arrays requires tracking dictionaries. Some code I removed from #738 to keep it simpler was along the lines of:

struct ArrowIpcDictionaryDecoder {
  int64_t id;
  struct ArrowIpcDecoder decoder;
};

ArrowErrorCode ArrowIpcDictionaryDecoderInit(struct ArrowIpcDictionaryDecoder* dictionary,
                                             const struct ArrowIpcDecoder* parent,
                                             int64_t id,
                                             struct ArrowSchema* value_schema);

void ArrowIpcDictionaryDecoderReset(struct ArrowIpcDictionaryDecoder* dictionary);

struct ArrowIpcDictionaries {
  struct ArrowIpcDictionaryDecoder* dictionaries;
  int64_t n_dictionaries;
  int64_t capacity;
};

void ArrowIpcDictionariesInit(struct ArrowIpcDictionaries* dictionaries);
void ArrowIpcDictionariesReset(struct ArrowIpcDictionaries* dictionaries);

ArrowErrorCode ArrowIpcDictionariesAppend(struct ArrowIpcDictionaries* dictionaries,
                                          const struct ArrowIpcDecoder* decoder,
                                          int64_t id, struct ArrowSchema* schema);

Basically, every dictionary ID gets its own ArrowIpcDecoder, but we need to share the ArrowIpcDictionaries* because there might be nested dictionaries.

Part of #622.

Activity

  1. paleolimbot commented on Mar 5, 2026

    @paleolimbot
    MemberAuthor

    An unused variable we'll need for this:

    static uint8_t kDictionaryRecordBatch[] = {
        0xff, 0xff, 0xff, 0xff, 0x88, 0x00, 0x00, 0x00, 0x14, 0x00, 0x00, 0x00, 0x00, 0x00,
        0x00, 0x00, 0x0c, 0x00, 0x16, 0x00, 0x06, 0x00, 0x05, 0x00, 0x08, 0x00, 0x0c, 0x00,
        0x0c, 0x00, 0x00, 0x00, 0x00, 0x03, 0x04, 0x00, 0x18, 0x00, 0x00, 0x00, 0x08, 0x00,
        0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x0a, 0x00, 0x18, 0x00, 0x0c, 0x00,
        0x04, 0x00, 0x08, 0x00, 0x0a, 0x00, 0x00, 0x00, 0x3c, 0x00, 0x00, 0x00, 0x10, 0x00,
        0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
        0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
        0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
        0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00,
        0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
        0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x02, 0x00, 0x00, 0x00, 0x00, 0x00};
  2. added a commit that references this issue on Apr 7, 2026
  3. added a commit that references this issue on Apr 20, 2026
    5bb90b1
  4. added this to the nanoarrow 0.9.0 milestone on Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions