Skip to content

[coalesce] Add specialized DictionaryArray path for push_batch_with_indices #11940

Open

Description

@Rich-T-kid

Is your feature request related to a problem or challenge?

Follow-up to #11743 and #11174.

#11743 adds a specialized in-progress coalescer for byte dictionaries, but the indexed-copy path introduced by #11174 will still use the default InProgressArray::copy_rows_by_indices implementation. That implementation calls take to materialize an intermediate dictionary before copying it into the coalescer.

Describe the solution you'd like

Add copy_rows_by_indices support to InProgressDictionaryArray for Dictionary<K, Utf8 | LargeUtf8 | Binary | LargeBinary>.

For a pointer-identical values array, gather the selected primitive keys and validity directly into the in-progress buffers while reusing the values Arc. This should avoid the intermediate take allocation while preserving repeated and out-of-order indices, null keys, bounds validation, and target-batch chunking. Inputs with a different values mapping must continue to use a correct fallback.

Use the dictionary take cases in coalesce_kernels to compare the specialized path with the existing materialized path for both shared and changing values arrays.

Activity

  1. Rich-T-kid commented on Oct 27, 2026

    @Rich-T-kid
    ContributorAuthor

    take

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions