Skip to content

feat(amber): opt-in columnar (Apache Arrow) execution in Amber #9542

Description

@Ma77Ball

Feature Summary

Opt-in columnar (Apache Arrow) execution for Amber: operators can exchange and process data as Arrow columns instead of row Tuples, on the edges and operators where that is faster. The feature is off by default (ApplicationConfig.enableColumnarWire). When it is disabled the row path is unchanged, and when it is enabled a per-edge planner keeps each edge on whichever format is cheaper, so no workflow regresses.

Proposed Solution or Design

A per-edge planner picks the cheapest format for each edge (rows, Arrow IPC, or a live root by reference) from the downstream operator's columnar capability and the receiver's locality, so the feature stays opt-in and never regresses a workflow. The work lands as a stacked, independently reviewable series, tracked by the sub-issues below and building from the wire contract up to local zero-copy. Measured so far (single-JVM serialization simulation, median of 9 trials): cross-node columnar runs 3x to 50x faster at 100k rows on exchange- and projection-heavy workflows, and is near par on small batches and pure local aggregates. A real multi-node benchmark is tracked as follow-up. Design notes and the phased plan live in Discussion #8571.

Sub-issues

Activity

  1. added theissue type on Nov 19, 2026
  2. Ma77Ball commented on Nov 19, 2026

    @Ma77Ball
    ContributorAuthor

    /take

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions