Skip to content

feat(amber): locality-aware columnar edges - #9553

Draft
Ma77Ball wants to merge 12 commits into
apache:mainfrom
Ma77Ball:columnar/04-locality
Draft

Ma77Ball wants to merge 12 commits into
apache:mainfrom
Ma77Ball:columnar/04-locality

Conversation

@Ma77Ball

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

  • Make the per-edge planner locality-aware: a same-JVM edge stays on rows via NetworkOutputBuffer.effectiveColumnar, so a local hop never pays the Arrow-bytes wire.
  • On a mixed-locality shuffle edge, send Arrow columns only to the remote receiver buckets while co-located buckets stay on rows, decided per destination worker.
  • Resolve receiver locality from the actor ref address (isLocalWorker) set when the data processor is created and on checkpoint restore.

Any related issues, documentation, discussions?

Closes: #9545
Part of: #9542
Stacked on Part 3 (#9544); isolated diff: Ma77Ball/texera@columnar/03-shuffle-projection...columnar/04-locality

How was this PR tested?

  • Run sbt "WorkflowExecutionService/testOnly *OutputManagerSpec", expect the locality-guard and per-receiver mixed-locality cases to pass.
  • Manual: with enableColumnarWire=true, run a workflow whose operators are co-located in one JVM at http://texera.localhost and confirm results are identical to the flag-off run (the local edge quietly stays on rows).

Was this PR authored or co-authored using generative AI tooling?

Co-authored with Claude Opus 4.8 (Anthropic) in compliance with ASF policy.

The foundation for a columnar Arrow wire: an operator-side contract for consuming and producing Arrow batches directly, the Arrow helpers it needs, and the first two operators that implement it (the specialized filter and the Arrow file scan). Nothing is wired into the messaging layer yet, so no operator receives columnar batches at runtime; these are the capabilities the wire will drive.

Contract (ColumnarOperatorExecutor):
- ColumnarOperatorExecutor is a trait with processColumnarBatch(arrowIpcBytes, port) that returns a ColumnarResult. An operator mixes it in to declare it can consume an Arrow batch without decoding each row to a Tuple.
- ColumnarResult is the sealed outcome of consuming one batch: Emit(root) (emit this Arrow result, caller serializes and closes the root), EmitRows(iterator) (emit row-shaped output, e.g. a join), Consumed (consumed with nothing to emit now, a blocking operator accumulating state), and Unsupported (not handled, caller falls back to the row path).
- SourceOperatorExecutor gains produceColumnarBatch(), defaulting to None, so a source whose data is already columnar can hand out Arrow roots instead of going through produceTuple; the caller owns and closes each root.

ArrowUtils helpers:
- serializeRoot writes a VectorSchemaRoot to Arrow IPC stream bytes without closing it; deserializeRootFold reads one IPC batch, applies a function to the live root, and closes the reader (the function must copy out what it keeps).
- selectRows copies the rows a boolean keep-mask selects into a new compacted root; selectionOf returns just the surviving row indices (a selection vector) without copying any cell, and gather materializes the rows a selection vector names into a new root, so a consumer that only needs a count or an aggregate reads indices and never materializes.
- projectionIndices resolves column names to their Arrow field indices by exact name; getCell reads one cell as the Texera value for an attribute type, returning null on a conversion failure to match the row path's lenient decode.

SpecializedFilterOpExec columnar path:
- The operator now also mixes in ColumnarOperatorExecutor while keeping its row path (the per-tuple OR over predicates) unchanged. columnarMask builds a boolean survivor mask for a single numeric predicate (Float8Vector or IntVector) read straight from the Arrow vector, and returns None for any other predicate or column shape so the caller falls back to rows.
- processColumnarBatch deserializes the batch and returns Emit(selectRows(mask)) when the mask applies, else Unsupported. columnarSelection returns the survivor indices without compacting, and columnarMatchCount is the fused filter-then-count that reports the survivor count with no batch materialized. A per-operator RootAllocator backs these and is freed in close().

ArrowSourceOpExec columnar produce:
- produceColumnarBatch hands each already-Arrow file batch out as a VectorSchemaRoot, moving the loaded batch's vectors into an independent root via a zero-copy TransferPair so the caller can serialize and close it while the reader loads the next batch. It returns None (the row path) when offset or limit is set.
Adds the wire that carries a data batch between workers as Arrow IPC bytes (a ColumnarFrame) instead of decoded Tuples (a DataFrame), behind an off-by-default flag. When enabled, every operator's output goes out as Arrow and is decoded back at the receiver, so the format is transparent to the row path; the payoff is on cross-worker links, where one Arrow IPC block replaces per-tuple serialization. Consuming the Arrow batch without decoding (for a ColumnarOperatorExecutor) is a later optimization; this lands the transport and keeps results identical with the flag on or off.

Flag:
- ApplicationConfig.enableColumnarWire reads network-buffering.enable-columnar-wire, defaulting to false (override NETWORK_BUFFERING_ENABLE_COLUMNAR_WIRE). Off by default, so the default wire is DataFrame and unchanged.

Payload:
- ColumnarFrame(arrowIpcBytes, rowCount, schema) is a new DataPayload. It carries a batch as an Arrow IPC stream plus the row count and Texera schema that describe it without decoding. inMemSize is the byte length.

Send side (NetworkOutputBuffer):
- The buffer takes a `columnar` flag (defaulting to ApplicationConfig.enableColumnarWire). On flush, when columnar is on it serializes the buffered tuples to Arrow once (all tuples on a link share the schema) and sends a ColumnarFrame; otherwise it sends a DataFrame as before. An empty flush still sends nothing.

Receive side:
- DataProcessor.processDataPayload handles ColumnarFrame by decoding the Arrow bytes back to tuples and running the identical batch path as DataFrame; the shared startInputBatch helper carries the common state-transition, initBatch, and first-tuple step for both. A columnar batch bound for a Python worker is decoded back to tuples in PythonProxyClient.sendData and handed over as a Data batch, so the Python path is unaffected.

ArrowUtils:
- serializeTuples(schema, tuples) writes a tuple batch to Arrow IPC stream bytes; deserializeTuples(bytes) reads them back to tuples. These are the ColumnarFrame encode/decode endpoints.

Tests:
- ArrowUtilsSpec covers the serializeTuples/deserializeTuples round-trip over a multi-row batch of every field type (including a null row and an empty batch), the payload contract the wire depends on.
- NetworkOutputBufferColumnarSeamSpec checks the buffer emits a ColumnarFrame (carrying the same rows, row count, and schema) when the flag is on and a DataFrame when off, on both the manual and the size-triggered flush, and nothing on an empty flush.
- SelectionVectorFilterSpec checks the specialized filter's native-Arrow path (selection vector, fused filter-count, and the compacting Emit path) against the row path as the source of truth.
AggregateOpExec can now consume an Arrow columnar batch directly, accumulating each group's partial result by reading the group-key and aggregated cells straight from their Arrow vectors instead of decoding a Tuple per row. The accumulation itself is shared with the row path, so a grouped aggregate produces the identical result whether its input arrives as rows or as a columnar batch.

Value-based accumulation (DistributedAggregation):
- DistributedAggregation now accumulates one already-extracted value via iterateValue(partial, value), and carries an extract(tuple) that pulls the single cell an aggregation reads from a tuple (null for COUNT(*)). The row path's iterate(partial, tuple) is derived as iterateValue(partial, extract(tuple)), so accumulation is defined once and both paths use it. init, merge, and finalAgg are unchanged.
- AggregationOperation's six functions (sum, count, average, min, max, concat) are rewritten to this shape: each supplies iterateValue (the per-value logic) and extract (tuple.getField of its attribute, or null for COUNT(*)). count keeps its captured countAllRows flag, so COUNT(*) counts every row and COUNT(column) counts non-null values. average reads a value numerically via a value-based helper (a timestamp counts as its epoch milliseconds, any other value through its string form, null skipped), matching the previous tuple-based reading.

Columnar accumulate (AggregateOpExec):
- The operator mixes in ColumnarOperatorExecutor. processColumnarBatch deserializes the batch and runs accumulateRoot, which resolves once per batch the source-column index and Texera type of each group-by key and each aggregation's input column (index -1 for COUNT(*)), then for each row builds the group key and feeds each aggregation its cell value through iterateValue. It is a blocking operator, so it emits nothing per batch (ColumnarResult.Consumed); results are produced at onFinish via the existing path. A per-operator RootAllocator backs the decode and is freed in close().

Tests:
- AggregateOpSpec adds a columnar-vs-row check: a grouped SUM/COUNT/AVERAGE/MIN/MAX over rows including a null value gives the same per-group result through processColumnarBatch as through processTuple.
- DistributedAggregationSpec is updated to the new constructor (iterateValue + extract) and asserts iterate is derived from them (the row path equals the value path).
Adds a benchmark that measures the columnar wire against the row wire on the real Pekko/Kryo stack, and speeds up the Arrow-to-tuple decode it exercises.

Decode (ArrowUtils):
- getTexeraTuple gains an overload that takes the already-resolved Texera schema, and the no-schema overload delegates to it. deserializeTuples now resolves the Texera schema, the field-vector array, and each column's attribute type once per stream and builds each row with a plain index loop (getCell per column into a fresh Array, then the Tuple constructor), instead of rebuilding the schema and allocating a Scala collection per row.

Benchmark (ColumnarWireBenchSpec):
- Serializes batches through the configured Pekko serialization (SerializationExtension on an ActorSystem built from AmberRuntime.pekkoConfig), so the row path is a real Kryo DataFrame round-trip and the columnar path is ArrowUtils.serializeTuples wrapped in a ColumnarFrame. For narrow-numeric, wide-mixed, and string-heavy workloads at 2k/20k/100k rows it reports sender encode time, receiver decode time, and serialized payload size for each path.
- A second scenario measures the decisive case: a vectorized receiver (a SpecializedFilterOpExec) that reads the Arrow columns directly via processColumnarBatch, against the row path that deserializes the DataFrame and runs the predicate per tuple.
- It is skipped unless run with -Dcolumnar.bench=true, so it does not run in CI.

Findings (this machine, JDK 17, median of 7): the columnar wire has a smaller payload (about 0.45x-0.91x) and a cheaper encode (about 0.5x at 100k rows), but decoding Arrow back to tuples is far more expensive than Kryo rehydrating a DataFrame (roughly 40x-68x), so a receiver that decodes to rows is a net loss. A receiver that vectorizes instead wins clearly: the filter's processColumnarBatch is about 0.13x the row path's receive+process at 100k rows (around 7x faster). So the columnar wire pays off only when the receiver consumes columns natively; routing a ColumnarFrame to a ColumnarOperatorExecutor's processColumnarBatch in the worker (instead of always decoding) is the next step.
Routes an incoming Arrow columnar batch to a ColumnarOperatorExecutor's processColumnarBatch in the worker, so a columnar-capable operator consumes the Arrow columns directly instead of the batch always being decoded back to tuples first. The benchmark showed decoding Arrow to tuples is the columnar wire's dominant cost, far more than Kryo rehydrating a DataFrame, so the wire only pays off when the receiver reads columns natively; this is the path that realizes that win.

Worker routing (DataProcessor):
- A ColumnarFrame now goes through processColumnarFrame. When the executor is a ColumnarOperatorExecutor, the worker transitions to RUNNING, binds the executor's state references, and hands it the Arrow IPC bytes via processColumnarBatch, then routes the ColumnarResult: Emit(root) is materialized to output tuples (the existing output path is tuple-based) and the root is freed; EmitRows installs the returned rows as the output; Consumed emits nothing now (a blocking operator that accumulated the batch); Unsupported, or any non-columnar executor, falls back to decoding the batch to tuples and processing it as a DataFrame. The READY->RUNNING transition is factored out of startInputBatch into transitToRunning, shared by both paths. For a natively consumed batch the per-tuple input manager is left empty (there is no per-tuple input to drain), so the DP loop drains only the output iterator.
- StatisticsManager gains a batched increaseInputStatistics(portId, count, size) so the columnar path reports a whole batch's row count and serialized size in one call rather than once per tuple; the single-tuple overload delegates to it.

Root-to-tuples decode (ArrowUtils):
- tuplesOfRoot decodes an already-open VectorSchemaRoot to tuples (the Emit output path's root-to-rows step). Like deserializeTuples it resolves the Texera schema, the field vectors, and each column's type once and reads each row with a plain index loop, but it works over a live root without closing it.

Tests:
- DataProcessorSpec adds the four routing outcomes plus the non-columnar case, driving real ColumnarFrames through processDataPayload with fake executors: Emit (a pass-through filter) emits the batch's rows and never calls processTuple; EmitRows emits the returned rows; Consumed emits nothing and still counts the batch's input statistics; Unsupported and a plain row-only executor both decode the frame and run it through processTuple per row.
- ArrowUtilsSpec asserts tuplesOfRoot decodes a live root to the same tuples deserializeTuples produces from the IPC bytes.
When a ColumnarOperatorExecutor emits an Arrow result batch, the worker can now ship it straight downstream as a ColumnarFrame instead of decoding it to tuples and letting the network buffer re-encode those tuples back to Arrow. This removes the columns-to-rows-to-columns round-trip that otherwise gave back the vectorized-receive win at the output boundary. It is the first step of the per-edge columnar design (phase 1): eligibility is limited to the edges where shipping the whole batch to a single receiver is correct, and every other case falls back to the existing row output path unchanged.

Output fast path (OutputManager):
- tryEmitColumnarBatch(root, outputPortId) resolves the output port, checks eligibility, and on success serializes the root once and sends the resulting ColumnarFrame through each receiver's NetworkOutputBuffer, returning (portId, serializedByteCount) so the caller records output statistics; it returns None (caller falls back) when not eligible. It does not close the root (the caller owns it). An edge is eligible only when the port needs no storage, every partitioner on the port is one-to-one (a single receiver gets the whole batch), and the receiver buffers are in columnar mode. Hash/range/round-robin shuffles need per-row partitioning (a later phase) and broadcast/storage are deferred, so those return None. The emitted Arrow schema must equal the declared output-port schema, otherwise it falls back so the row path's enforceSchema can adjust it.
- setColumnarOutput(enabled) flips the columnar flag on every current edge. This is a provisional phase-1 switch; phase 2 replaces it with a per-edge decision from the planner (local vs cross-node, and whether the receiver consumes columns).

Network buffer (NetworkOutputBuffer):
- sendColumnarFrame(frame) ships an already-serialized Arrow batch straight to the receiver, flushing any buffered tuples first so the batch keeps its place in the FIFO stream. Its columnar flag becomes a var so the output manager can set it per edge.

Worker routing (DataProcessor):
- The Emit(root) case now tries tryEmitColumnarBatch first and records the batch's output statistics in one call; only when the edge is not eligible does it materialize the root to tuples through the row output path. The root is freed in a finally either way.

Batched statistics (StatisticsManager):
- increaseOutputStatistics(portId, count, size) mirrors the batched input overload, so the columnar output path counts a whole batch's rows and serialized bytes in one call rather than once per tuple; the single-tuple overload delegates to it.

Tests:
- OutputManagerSpec covers tryEmitColumnarBatch: a one-to-one columnar edge ships a single ColumnarFrame that decodes back to the input rows; a hash-shuffle edge is declined (returns None, sends nothing); and an edge with columnar output off is declined.
- NetworkOutputBufferColumnarSeamSpec covers sendColumnarFrame flushing buffered rows before the prebuilt frame, in order.
- DataProcessorSpec adds the eligible-edge case (Emit goes out as one ColumnarFrame, no tuple round-trip, processTuple never runs, output statistics count the batch) and keeps the not-columnar case as the row-output fallback.
…perator

The columnar wire is no longer a single global flag that flips every edge at once. The planner now decides it per link, and an edge ships Arrow only when the downstream operator can consume columns natively. This stops the worst case of the blanket switch, shipping Arrow to a row-only operator that then pays a per-tuple decode, and is the capability half of the per-edge design (phase 2). Locality (a same-JVM link gains nothing from serializing) is a later refinement; worker placement is not modeled at planning time and actor refs resolve lazily, so this change keeps that axis out.

Operator capability (PhysicalOp):
- isColumnarCapable resolves, by name and without loading or instantiating the class, whether the operator's executor mixes in ColumnarOperatorExecutor. Any lookup miss (a code-based or unknown executor) is not capable. This auto-discovers every columnar operator (filter, aggregate, and future ones) with no per-operator registration.

Planner decision (ResourceAllocator, LinkConfig):
- allocate sets LinkConfig.columnar = ApplicationConfig.enableColumnarWire && the downstream operator isColumnarCapable, for each link. The global flag is now a master switch folded into the per-edge decision rather than a second source of truth.

Threading to the sender (proto + handlers):
- AddPartitioningRequest gains a columnar bool. LinkWorkersHandler fills it from the link's LinkConfig; the worker's AddPartitioningHandler passes it to OutputManager.addPartitionerWithPartitioning, which constructs that edge's NetworkOutputBuffers in columnar mode. NetworkOutputBuffer.columnar now defaults to false (the planner is the authority) instead of reading ApplicationConfig.enableColumnarWire itself.

Tests:
- PhysicalOpSpec: isColumnarCapable holds for an executor that implements ColumnarOperatorExecutor, and not for a plain executor, a code-based executor, or an unresolvable class name.
- OutputManagerSpec: addPartitionerWithPartitioning builds columnar buffers when told to (a flushed batch goes out as a ColumnarFrame) and row buffers by default (a DataFrame).
- The AddPartitioningRequest constructions in WorkerSpec, LoggingSpec, and ArrowFlightActorBench pass columnar = false for the new field.
…ding rows

A shuffle edge on the columnar wire no longer decodes the whole batch to tuples to route it. The columnar output path now partitions an Arrow batch in place: it assigns each row a bucket through the same getBucketIndex the row path uses, reading only the columns the partitioner needs, then gathers each bucket's rows in Arrow and sends one ColumnarFrame per non-empty bucket. Bucket assignment is identical to the row path by construction, so a hash shuffle feeding a join routes rows to exactly the workers the row path would. This completes the columnar wire for hash, range, round-robin, and broadcast edges (one-to-one was already handled); only the key columns are decoded, and the payload stays columnar end to end.

Partitioner capability hooks:
- Partitioner gains columnarSplits (false for one-to-one and broadcast, which send the whole batch to each receiver; true for hash/range/round-robin, which split it), bucketKeyColumns (the columns getBucketIndex reads: the hash keys, or the first range attribute; empty for round-robin), and requiresFullTuple (true only for a hash with no key columns, which hashes the whole tuple). HashBasedShufflePartitioner, RangeBasedShufflePartitioner, OneToOnePartitioner, and BroadcastPartitioner override these; the defaults cover round-robin.

Columnar partitioning (OutputManager):
- tryEmitColumnarBatch now accepts any partitioner (it previously required one-to-one). For each eligible link it calls emitColumnarToLink: a non-splitting partitioner serializes the root once and sends it to each receiver; a splitting partitioner builds a partial tuple of just the key columns per row (the full tuple only when requiresFullTuple), calls getBucketIndex to assign the bucket, gathers each bucket's rows into a new Arrow root (ArrowUtils.gather), serializes it, and sends it to that bucket's receiver. A per-manager RootAllocator backs the gathered roots; each is freed right after it is serialized, so its outstanding memory returns to zero. The method returns the total serialized bytes put on the wire for the port's output statistics; eligibility is unchanged (no storage, every edge in columnar mode, schema matches the output port).

Tests:
- OutputManagerSpec adds differential checks that the columnar shuffle routes rows to the same receivers the row-path partitioner would, for hash, range, and round-robin (which also checks the round-robin counter advances in step), with no row lost or duplicated, and that broadcast sends the whole batch to every receiver. The obsolete "decline a hash-shuffle edge" case (hash now shuffles columnar) is removed; the one-to-one and columnar-off cases remain.
…perator

ProjectionOpExec can now project an Arrow columnar batch directly, so a projection no longer breaks a columnar region: it keeps the batch columnar instead of decoding every tuple and rebuilding it. Because projection only selects, reorders, and renames columns, each output column is produced by moving its source column's Arrow buffers into the output root under the alias (an Arrow transfer, not a per-cell copy) rather than reading and rewriting values. The operator mixes in ColumnarOperatorExecutor, so the planner's per-edge decision (added earlier) now also routes columns through projections.

Columnar projection (ProjectionOpExec):
- processColumnarBatch resolves the same (source column -> output alias) pairs that the row path's project() derives, honoring drop mode (keep every column whose lowercased name is not in the drop set) and select mode (the descriptor's attribute units). It builds the output schema from the alias names and the source column types, creates the output root from that schema, and for each output column moves the source vector's buffers into it via Arrow makeTransferPair, renamed to the alias. The result is Emit(outRoot), which the columnar output path ships downstream with no round-trip.
- It falls back to the row path (ColumnarResult.Unsupported) for the shapes a single per-column move cannot express or that the row path treats as errors: an empty selection, a source column feeding more than one output (its buffers cannot be moved twice), a duplicated output alias (the row path raises on this), or a source column that does not resolve. A per-operator RootAllocator backs the output roots and is freed in close().

Tests:
- ProjectionOpExecSpec adds a columnar-vs-row check that projecting with reorder and rename over a multi-row batch yields, through processColumnarBatch, the same schema and values as processTuple; a drop-mode columnar check; and a check that a duplicate alias makes processColumnarBatch return Unsupported (so the row path raises the error as before).
…lized pipeline

Adds a benchmark that measures a whole columnar region, project -> filter -> aggregate, against the row path for the same pipeline, with each operator on its own worker so the batch crosses a serialized edge between them. It exercises the real operators (ProjectionOpExec, SpecializedFilterOpExec, AggregateOpExec) and the real Pekko/Kryo serialization, so the numbers reflect a cross-worker (cross-node) region, which is where the columnar wire is meant to pay off, rather than the per-edge microbenchmark alone.

ColumnarPipelineBenchSpec:
- The row path serializes a DataFrame on each of the three edges (source->project, project->filter, filter->aggregate) and runs each operator tuple-at-a-time: project maps and re-enforces each tuple, filter runs the predicate per tuple, aggregate accumulates per tuple. The source->project edge carries the full wide rows (two string payload columns the projection then drops), as a real source edge would.
- The columnar path serializes an Arrow ColumnarFrame on each edge and runs each operator's processColumnarBatch, staying columnar across all three hops: projection moves columns, filter emits the surviving rows, aggregate accumulates straight off the Arrow vectors.
- Both reset the aggregate via open() per run, warm up, and report the median of several trials for 2k/20k/100k source rows. It is skipped unless run with -Dcolumnar.bench=true, so it does not run in CI.

Findings (this machine, JDK 17, median of 7): the columnar region runs end to end in about 0.21x the row path at 2k rows, 0.06x at 20k, and 0.05x at 100k (roughly 5x, 16x, and 19x faster), the gap widening with batch size. This matches the per-edge microbenchmark and the vectorized-operator speedups the literature reports: once a run of operators consumes columns natively, keeping the batch in Arrow across serialized edges avoids both the per-tuple Kryo round-trip and the per-tuple operator work that dominate the row path.
…remote hops

The columnar wire now yields to locality: an edge to a co-located receiver stays on rows even when the planner enabled columns for it. A same-JVM hop is delivered by reference and never serializes, so encoding it to Arrow would be pure overhead; the columnar wire only pays on a remote hop that actually crosses the network. The decision is per receiver, not per link, because one sender fans out to receivers on different nodes (a hash shuffle is local to some, remote to others). This closes the same-JVM pessimization the pipeline benchmark exposed: a single-node run now behaves exactly as the row engine, while a cross-node run keeps the columnar win.

Locality signal (PekkoActorRefMappingService):
- isLocalWorker(id) returns whether the worker's resolved actor ref is in this same ActorSystem (its address has local scope or equals this actor's own address), or None when the ref is not resolved yet. The sender already holds these refs to route data, so no new lookup is needed.

Per-receiver decision (NetworkOutputBuffer, OutputManager):
- Each NetworkOutputBuffer gains a localityCheck and an effectiveColumnar = columnar && the receiver is not known-local (unknown is treated as remote, so the worst case is briefly using columns before a ref resolves). flush() now consults effectiveColumnar, so the row path emits a DataFrame to a local receiver and a ColumnarFrame to a remote one on the very same link.
- OutputManager carries isLocalReceiver (set by the worker) and wires each buffer's localityCheck to it when a partitioner is added. tryEmitColumnarBatch's no-decode fast path now requires every receiver to be effectiveColumnar; a local or mixed-locality edge falls back to the row path, whose per-receiver flush already does the right thing (rows to local receivers, Arrow to remote ones). Fully-remote one-to-one and fully-remote shuffles still take the fast path.

Worker wiring (WorkflowWorker):
- Sets dp.outputManager.isLocalReceiver = actorRefMappingService.isLocalWorker when the data processor is created and when it is restored from a checkpoint.

Tests:
- OutputManagerSpec: a local one-to-one edge stays on rows though columnar is enabled; the no-decode fast path is declined for a local receiver and taken for a remote one; and on a mixed-locality hash shuffle the row path sends DataFrames to the local receiver and ColumnarFrames to the remote one. Existing columnar tests are unaffected (locality defaults to unknown, i.e. remote).
…de for remote buckets

A shuffle whose receivers are split across the local node and remote nodes no longer gives up the columnar fast path for the whole edge. The output path now decides per receiver: a remote receiver gets its bucket as Arrow with no tuple decode, while a local receiver gets its bucket as rows (delivered by reference, which is free). Previously a mixed-locality edge fell back entirely to the row path, so the remote buckets paid a decode plus re-encode; now only the local buckets are decoded, and only their own rows.

OutputManager:
- tryEmitColumnarBatch takes the fast path when at least one receiver is remote-columnar (rather than requiring all of them). It still declines when no receiver is columnar (all local, or the wire is off) so the caller decodes once and uses the row path, and when the port needs storage or the schema does not match.
- emitColumnarToLink sends each receiver its data in the right shape. For a one-to-one or broadcast edge it serializes the batch to Arrow once for the remote receivers and decodes it to rows once for the local ones. For a splitting shuffle it assigns buckets as before (through the same getBucketIndex the row path uses) and then, per bucket, calls sendBucket.
- sendBucket ships a bucket as Arrow (gather + ColumnarFrame) when the receiver's buffer is columnar, or decodes just that bucket's rows (getTexeraTuple per row) and sends them as a DataFrame when it is local.

NetworkOutputBuffer:
- sendRows is the row counterpart of sendColumnarFrame: it flushes any buffered tuples, then ships a prebuilt DataFrame straight to the receiver, so a local receiver on a mixed edge gets rows in FIFO order alongside the Arrow sent to remote receivers.

Tests:
- OutputManagerSpec: on a mixed-locality hash shuffle the fast path ships DataFrames to the local receiver and ColumnarFrames to the remote one, and each receiver holds exactly the rows the row partitioner assigns, with nothing lost. The earlier all-local (declines) and fully-remote (fast path) cases still hold.
@github-actions

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @aglinxinyuan, @Yicong-Huang, @eugenegujing
    You can notify them by mentioning @aglinxinyuan, @Yicong-Huang, @eugenegujing in a comment.

@codecov-commenter

codecov-commenter commented Nov 19, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 67.70428% with 83 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.35%. Comparing base (4db10cf) to head (f6dc5fb).

Files with missing lines Patch % Lines
...cala/org/apache/texera/amber/util/ArrowUtils.scala 64.64% 34 Missing and 1 partial ⚠️
...mber/operator/filter/SpecializedFilterOpExec.scala 54.54% 8 Missing and 12 partials ⚠️
...operator/source/scan/arrow/ArrowSourceOpExec.scala 0.00% 13 Missing ⚠️
...amber/core/executor/ColumnarOperatorExecutor.scala 0.00% 6 Missing ⚠️
...era/amber/operator/aggregate/AggregateOpExec.scala 85.71% 0 Missing and 5 partials ⚠️
...a/amber/operator/projection/ProjectionOpExec.scala 88.46% 0 Missing and 3 partials ⚠️
...a/amber/core/executor/SourceOperatorExecutor.scala 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #9553      +/-   ##
============================================
+ Coverage     91.57%   92.35%   +0.77%     
+ Complexity     5159     3300    -1859     
============================================
  Files          1300     1058     -242     
  Lines         58092    49152    -8940     
  Branches       7615     6334    -1281     
============================================
- Hits          53198    45393    -7805     
+ Misses         2958     2475     -483     
+ Partials       1936     1284     -652     
Flag Coverage Δ *Carryforward flag
access-control-service 77.44% <ø> (ø)
agent-service 99.56% <ø> (ø) Carriedforward from 9b7735c
amber 84.05% <67.70%> (-1.39%) ⬇️
computing-unit-managing-service 63.88% <ø> (ø)
config-service 87.50% <ø> (ø)
file-service 82.16% <ø> (ø)
frontend 96.45% <ø> (ø) Carriedforward from 9b7735c
notebook-migration-service 83.77% <ø> (ø)
pyamber 98.38% <ø> (ø) Carriedforward from 9b7735c
result-service 85.71% <ø> (ø)
workflow-compiling-service 78.30% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(amber): locality-aware columnar edges, rows local and columns remote

2 participants