You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
perf(arrow): build RecordBatchTransformer's plan from the projected fields only #3572
RecordBatchTransformer::generate_batch_transform converts the whole snapshot schema to Arrow and maps every top-level field on the first batch of each file, even when the result is a pass-through. The cost grows with the table's width, not with the number of projected columns. Wide tables stored as many small files pay it most.
#3540 avoids this for the row filter transformer by giving it a schema that holds only the filter fields. @mbutrovich measured building that transformer and processing one 8192-row batch: 135 µs with the full schema of a 1001-column table, 2 µs with the filter fields only.
The output transformer in reader/pipeline.rs should do the same, either at the call site or inside generate_batch_transform.
Activity