You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(amber): opt-in columnar (Apache Arrow) execution in Amber #9542
Opt-in columnar (Apache Arrow) execution for Amber: operators can exchange and process data as Arrow columns instead of row Tuples, on the edges and operators where that is faster. The feature is off by default (ApplicationConfig.enableColumnarWire). When it is disabled the row path is unchanged, and when it is enabled a per-edge planner keeps each edge on whichever format is cheaper, so no workflow regresses.
Proposed Solution or Design
A per-edge planner picks the cheapest format for each edge (rows, Arrow IPC, or a live root by reference) from the downstream operator's columnar capability and the receiver's locality, so the feature stays opt-in and never regresses a workflow. The work lands as a stacked, independently reviewable series, tracked by the sub-issues below and building from the wire contract up to local zero-copy. Measured so far (single-JVM serialization simulation, median of 9 trials): cross-node columnar runs 3x to 50x faster at 100k rows on exchange- and projection-heavy workflows, and is near par on small batches and pure local aggregates. A real multi-node benchmark is tracked as follow-up. Design notes and the phased plan live in Discussion #8571.
Activity
Ma77Ball commentedon Nov 19, 2026
/take