You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Columnar (Apache Arrow) execution in Amber: the opt-in PR series and why
#8571
Columnar (Apache Arrow) execution in Amber: the opt-in PR series and why
Status, updated 2026-11-19. This revives the thread with what actually shipped. The work now lives on a reviewable, stacked PR series tracked by parent issue #9542, and the whole feature is off by default (ApplicationConfig.enableColumnarWire), so the row path is unchanged until a user opts in.
Why
Amber moves and processes data one row Tuple at a time at every operator and across every edge. For exchange- and scan-heavy workflows that costs two things: serializing an object graph per batch on a cross-worker edge, and building a JVM object per row at every stage. Columnar (Apache Arrow) execution addresses both: Arrow batches serialize at near memory-copy speed, and a columnar-capable operator reads the columns directly instead of materializing a tuple per row. This is the same mechanism Spark/Photon, Velox, Gluten, and DataFusion Comet use.
The key design choice: decide per edge, not globally
Columnar is not always a win. A same-JVM edge already passes data by reference, so wrapping it in Arrow bytes would only add overhead. So each edge picks its own format from two questions: can the receiver read columns, and is it local or remote. Remote and columnar-capable goes Arrow; local and columnar-capable gets a live root by reference (zero copy); everything else stays rows. This is what keeps the feature from ever regressing a workflow, and it is why it can eventually default on.
Single-JVM serialization simulation, median of 9 trials, across a matrix of workflow shapes with a correctness parity gate (the columnar and row paths must return identical results per cell):
Cross-node (serialized edges): 3x to 50x faster at 100k rows on exchange- and projection-heavy workflows; small batches (about 1k rows) can lose, since Arrow's fixed per-batch overhead is not amortized.
Local (no serialization): the win concentrates where columnar avoids per-tuple tuple construction (projection); pure aggregates and mostly-passing filters are near par.
A real multi-node cluster benchmark is the remaining measurement gap and is tracked as follow-up. Note that a large part of the projection speedup reflects how expensive Amber's per-tuple row path is, which is also worth attacking independently.
Supersedes
This replaces the earlier four-PR outline in this thread; the old sub-issues (#8558 through #8561) are closed. Feedback on the split, the per-edge planner, or the measurement method is welcome here.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Columnar (Apache Arrow) execution in Amber: the opt-in PR series and why
Status, updated 2026-11-19. This revives the thread with what actually shipped. The work now lives on a reviewable, stacked PR series tracked by parent issue #9542, and the whole feature is off by default (
ApplicationConfig.enableColumnarWire), so the row path is unchanged until a user opts in.Why
Amber moves and processes data one row
Tupleat a time at every operator and across every edge. For exchange- and scan-heavy workflows that costs two things: serializing an object graph per batch on a cross-worker edge, and building a JVM object per row at every stage. Columnar (Apache Arrow) execution addresses both: Arrow batches serialize at near memory-copy speed, and a columnar-capable operator reads the columns directly instead of materializing a tuple per row. This is the same mechanism Spark/Photon, Velox, Gluten, and DataFusion Comet use.The key design choice: decide per edge, not globally
Columnar is not always a win. A same-JVM edge already passes data by reference, so wrapping it in Arrow bytes would only add overhead. So each edge picks its own format from two questions: can the receiver read columns, and is it local or remote. Remote and columnar-capable goes Arrow; local and columnar-capable gets a live root by reference (zero copy); everything else stays rows. This is what keeps the feature from ever regressing a workflow, and it is why it can eventually default on.
The reviewable parts (parent #9542)
ColumnarOperatorExecutor/ColumnarResultcontract, the opt-inColumnarFramewire, a faster Arrow-to-tuple decode, and vectorized receive for filter and aggregate.processColumnarRootprimitive and handing a live Arrow root to a co-located receiver by reference, with a checkpoint-safe serializer.Measured so far
Single-JVM serialization simulation, median of 9 trials, across a matrix of workflow shapes with a correctness parity gate (the columnar and row paths must return identical results per cell):
A real multi-node cluster benchmark is the remaining measurement gap and is tracked as follow-up. Note that a large part of the projection speedup reflects how expensive Amber's per-tuple row path is, which is also worth attacking independently.
Supersedes
This replaces the earlier four-PR outline in this thread; the old sub-issues (#8558 through #8561) are closed. Feedback on the split, the per-edge planner, or the measurement method is welcome here.
All reactions