Skip to content

Optimize gather_dictionary_values in arrow-row #12053

Closed

Description

@Rich-T-kid

Is your feature request related to a problem or challenge?

#12043 speeds up row encoding for oversized sparse dictionaries by gathering one value per input row before encoding. gather_dictionary_values currently calls MutableArrayData::try_extend once per key. That cost is acceptable for the high-cardinality case targeted by the PR, but it limits the fast path and moves the crossover higher than necessary.

Describe the solution you would like

Benchmark and optimize the row-aligned dictionary gather so it avoids one dynamic try_extend call per key where possible. Preserve sliced keys, nulls, arbitrary key widths, and the generic values-array fallback. Use the string_dictionary_sparse_keys row-format cases across dictionary sizes to verify the crossover and guard against regressions.

Describe alternatives you have considered

Potential approaches include using take, specializing common byte-array value types, or batching contiguous key ranges while keeping MutableArrayData as the generic fallback.

Additional context

Follow-up from #12043: #12043 (comment)

Current benchmark results: #12043 (comment)

Activity

  1. added a commit that references this issue on Nov 8, 2026
    f2d9627
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions