You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Optimize gather_dictionary_values in arrow-row #12053
Is your feature request related to a problem or challenge?
#12043 speeds up row encoding for oversized sparse dictionaries by gathering one value per input row before encoding. gather_dictionary_values currently calls MutableArrayData::try_extend once per key. That cost is acceptable for the high-cardinality case targeted by the PR, but it limits the fast path and moves the crossover higher than necessary.
Describe the solution you would like
Benchmark and optimize the row-aligned dictionary gather so it avoids one dynamic try_extend call per key where possible. Preserve sliced keys, nulls, arbitrary key widths, and the generic values-array fallback. Use the string_dictionary_sparse_keys row-format cases across dictionary sizes to verify the crossover and guard against regressions.
Describe alternatives you have considered
Potential approaches include using take, specializing common byte-array value types, or batching contiguous key ranges while keeping MutableArrayData as the generic fallback.
Activity
Optimize byte-array dictionary gathering in arrow-row (apache#12113)