Skip to content

perf(arrow): build RecordBatchTransformer's plan from the projected fields only #3572

Open

Description

@comphead

RecordBatchTransformer::generate_batch_transform converts the whole snapshot schema to Arrow and maps every top-level field on the first batch of each file, even when the result is a pass-through. The cost grows with the table's width, not with the number of projected columns. Wide tables stored as many small files pay it most.

#3540 avoids this for the row filter transformer by giving it a schema that holds only the filter fields. @mbutrovich measured building that transformer and processing one 8192-row batch: 135 µs with the full schema of a 1001-column table, 2 µs with the filter fields only.

The output transformer in reader/pipeline.rs should do the same, either at the call site or inside generate_batch_transform.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions