Repository navigation
Conversation
The foundation for a columnar Arrow wire: an operator-side contract for consuming and producing Arrow batches directly, the Arrow helpers it needs, and the first two operators that implement it (the specialized filter and the Arrow file scan). Nothing is wired into the messaging layer yet, so no operator receives columnar batches at runtime; these are the capabilities the wire will drive. Contract (ColumnarOperatorExecutor): - ColumnarOperatorExecutor is a trait with processColumnarBatch(arrowIpcBytes, port) that returns a ColumnarResult. An operator mixes it in to declare it can consume an Arrow batch without decoding each row to a Tuple. - ColumnarResult is the sealed outcome of consuming one batch: Emit(root) (emit this Arrow result, caller serializes and closes the root), EmitRows(iterator) (emit row-shaped output, e.g. a join), Consumed (consumed with nothing to emit now, a blocking operator accumulating state), and Unsupported (not handled, caller falls back to the row path). - SourceOperatorExecutor gains produceColumnarBatch(), defaulting to None, so a source whose data is already columnar can hand out Arrow roots instead of going through produceTuple; the caller owns and closes each root. ArrowUtils helpers: - serializeRoot writes a VectorSchemaRoot to Arrow IPC stream bytes without closing it; deserializeRootFold reads one IPC batch, applies a function to the live root, and closes the reader (the function must copy out what it keeps). - selectRows copies the rows a boolean keep-mask selects into a new compacted root; selectionOf returns just the surviving row indices (a selection vector) without copying any cell, and gather materializes the rows a selection vector names into a new root, so a consumer that only needs a count or an aggregate reads indices and never materializes. - projectionIndices resolves column names to their Arrow field indices by exact name; getCell reads one cell as the Texera value for an attribute type, returning null on a conversion failure to match the row path's lenient decode. SpecializedFilterOpExec columnar path: - The operator now also mixes in ColumnarOperatorExecutor while keeping its row path (the per-tuple OR over predicates) unchanged. columnarMask builds a boolean survivor mask for a single numeric predicate (Float8Vector or IntVector) read straight from the Arrow vector, and returns None for any other predicate or column shape so the caller falls back to rows. - processColumnarBatch deserializes the batch and returns Emit(selectRows(mask)) when the mask applies, else Unsupported. columnarSelection returns the survivor indices without compacting, and columnarMatchCount is the fused filter-then-count that reports the survivor count with no batch materialized. A per-operator RootAllocator backs these and is freed in close(). ArrowSourceOpExec columnar produce: - produceColumnarBatch hands each already-Arrow file batch out as a VectorSchemaRoot, moving the loaded batch's vectors into an independent root via a zero-copy TransferPair so the caller can serialize and close it while the reader loads the next batch. It returns None (the row path) when offset or limit is set.
Adds the wire that carries a data batch between workers as Arrow IPC bytes (a ColumnarFrame) instead of decoded Tuples (a DataFrame), behind an off-by-default flag. When enabled, every operator's output goes out as Arrow and is decoded back at the receiver, so the format is transparent to the row path; the payoff is on cross-worker links, where one Arrow IPC block replaces per-tuple serialization. Consuming the Arrow batch without decoding (for a ColumnarOperatorExecutor) is a later optimization; this lands the transport and keeps results identical with the flag on or off. Flag: - ApplicationConfig.enableColumnarWire reads network-buffering.enable-columnar-wire, defaulting to false (override NETWORK_BUFFERING_ENABLE_COLUMNAR_WIRE). Off by default, so the default wire is DataFrame and unchanged. Payload: - ColumnarFrame(arrowIpcBytes, rowCount, schema) is a new DataPayload. It carries a batch as an Arrow IPC stream plus the row count and Texera schema that describe it without decoding. inMemSize is the byte length. Send side (NetworkOutputBuffer): - The buffer takes a `columnar` flag (defaulting to ApplicationConfig.enableColumnarWire). On flush, when columnar is on it serializes the buffered tuples to Arrow once (all tuples on a link share the schema) and sends a ColumnarFrame; otherwise it sends a DataFrame as before. An empty flush still sends nothing. Receive side: - DataProcessor.processDataPayload handles ColumnarFrame by decoding the Arrow bytes back to tuples and running the identical batch path as DataFrame; the shared startInputBatch helper carries the common state-transition, initBatch, and first-tuple step for both. A columnar batch bound for a Python worker is decoded back to tuples in PythonProxyClient.sendData and handed over as a Data batch, so the Python path is unaffected. ArrowUtils: - serializeTuples(schema, tuples) writes a tuple batch to Arrow IPC stream bytes; deserializeTuples(bytes) reads them back to tuples. These are the ColumnarFrame encode/decode endpoints. Tests: - ArrowUtilsSpec covers the serializeTuples/deserializeTuples round-trip over a multi-row batch of every field type (including a null row and an empty batch), the payload contract the wire depends on. - NetworkOutputBufferColumnarSeamSpec checks the buffer emits a ColumnarFrame (carrying the same rows, row count, and schema) when the flag is on and a DataFrame when off, on both the manual and the size-triggered flush, and nothing on an empty flush. - SelectionVectorFilterSpec checks the specialized filter's native-Arrow path (selection vector, fused filter-count, and the compacting Emit path) against the row path as the source of truth.
AggregateOpExec can now consume an Arrow columnar batch directly, accumulating each group's partial result by reading the group-key and aggregated cells straight from their Arrow vectors instead of decoding a Tuple per row. The accumulation itself is shared with the row path, so a grouped aggregate produces the identical result whether its input arrives as rows or as a columnar batch. Value-based accumulation (DistributedAggregation): - DistributedAggregation now accumulates one already-extracted value via iterateValue(partial, value), and carries an extract(tuple) that pulls the single cell an aggregation reads from a tuple (null for COUNT(*)). The row path's iterate(partial, tuple) is derived as iterateValue(partial, extract(tuple)), so accumulation is defined once and both paths use it. init, merge, and finalAgg are unchanged. - AggregationOperation's six functions (sum, count, average, min, max, concat) are rewritten to this shape: each supplies iterateValue (the per-value logic) and extract (tuple.getField of its attribute, or null for COUNT(*)). count keeps its captured countAllRows flag, so COUNT(*) counts every row and COUNT(column) counts non-null values. average reads a value numerically via a value-based helper (a timestamp counts as its epoch milliseconds, any other value through its string form, null skipped), matching the previous tuple-based reading. Columnar accumulate (AggregateOpExec): - The operator mixes in ColumnarOperatorExecutor. processColumnarBatch deserializes the batch and runs accumulateRoot, which resolves once per batch the source-column index and Texera type of each group-by key and each aggregation's input column (index -1 for COUNT(*)), then for each row builds the group key and feeds each aggregation its cell value through iterateValue. It is a blocking operator, so it emits nothing per batch (ColumnarResult.Consumed); results are produced at onFinish via the existing path. A per-operator RootAllocator backs the decode and is freed in close(). Tests: - AggregateOpSpec adds a columnar-vs-row check: a grouped SUM/COUNT/AVERAGE/MIN/MAX over rows including a null value gives the same per-group result through processColumnarBatch as through processTuple. - DistributedAggregationSpec is updated to the new constructor (iterateValue + extract) and asserts iterate is derived from them (the row path equals the value path).
Adds a benchmark that measures the columnar wire against the row wire on the real Pekko/Kryo stack, and speeds up the Arrow-to-tuple decode it exercises. Decode (ArrowUtils): - getTexeraTuple gains an overload that takes the already-resolved Texera schema, and the no-schema overload delegates to it. deserializeTuples now resolves the Texera schema, the field-vector array, and each column's attribute type once per stream and builds each row with a plain index loop (getCell per column into a fresh Array, then the Tuple constructor), instead of rebuilding the schema and allocating a Scala collection per row. Benchmark (ColumnarWireBenchSpec): - Serializes batches through the configured Pekko serialization (SerializationExtension on an ActorSystem built from AmberRuntime.pekkoConfig), so the row path is a real Kryo DataFrame round-trip and the columnar path is ArrowUtils.serializeTuples wrapped in a ColumnarFrame. For narrow-numeric, wide-mixed, and string-heavy workloads at 2k/20k/100k rows it reports sender encode time, receiver decode time, and serialized payload size for each path. - A second scenario measures the decisive case: a vectorized receiver (a SpecializedFilterOpExec) that reads the Arrow columns directly via processColumnarBatch, against the row path that deserializes the DataFrame and runs the predicate per tuple. - It is skipped unless run with -Dcolumnar.bench=true, so it does not run in CI. Findings (this machine, JDK 17, median of 7): the columnar wire has a smaller payload (about 0.45x-0.91x) and a cheaper encode (about 0.5x at 100k rows), but decoding Arrow back to tuples is far more expensive than Kryo rehydrating a DataFrame (roughly 40x-68x), so a receiver that decodes to rows is a net loss. A receiver that vectorizes instead wins clearly: the filter's processColumnarBatch is about 0.13x the row path's receive+process at 100k rows (around 7x faster). So the columnar wire pays off only when the receiver consumes columns natively; routing a ColumnarFrame to a ColumnarOperatorExecutor's processColumnarBatch in the worker (instead of always decoding) is the next step.
Routes an incoming Arrow columnar batch to a ColumnarOperatorExecutor's processColumnarBatch in the worker, so a columnar-capable operator consumes the Arrow columns directly instead of the batch always being decoded back to tuples first. The benchmark showed decoding Arrow to tuples is the columnar wire's dominant cost, far more than Kryo rehydrating a DataFrame, so the wire only pays off when the receiver reads columns natively; this is the path that realizes that win. Worker routing (DataProcessor): - A ColumnarFrame now goes through processColumnarFrame. When the executor is a ColumnarOperatorExecutor, the worker transitions to RUNNING, binds the executor's state references, and hands it the Arrow IPC bytes via processColumnarBatch, then routes the ColumnarResult: Emit(root) is materialized to output tuples (the existing output path is tuple-based) and the root is freed; EmitRows installs the returned rows as the output; Consumed emits nothing now (a blocking operator that accumulated the batch); Unsupported, or any non-columnar executor, falls back to decoding the batch to tuples and processing it as a DataFrame. The READY->RUNNING transition is factored out of startInputBatch into transitToRunning, shared by both paths. For a natively consumed batch the per-tuple input manager is left empty (there is no per-tuple input to drain), so the DP loop drains only the output iterator. - StatisticsManager gains a batched increaseInputStatistics(portId, count, size) so the columnar path reports a whole batch's row count and serialized size in one call rather than once per tuple; the single-tuple overload delegates to it. Root-to-tuples decode (ArrowUtils): - tuplesOfRoot decodes an already-open VectorSchemaRoot to tuples (the Emit output path's root-to-rows step). Like deserializeTuples it resolves the Texera schema, the field vectors, and each column's type once and reads each row with a plain index loop, but it works over a live root without closing it. Tests: - DataProcessorSpec adds the four routing outcomes plus the non-columnar case, driving real ColumnarFrames through processDataPayload with fake executors: Emit (a pass-through filter) emits the batch's rows and never calls processTuple; EmitRows emits the returned rows; Consumed emits nothing and still counts the batch's input statistics; Unsupported and a plain row-only executor both decode the frame and run it through processTuple per row. - ArrowUtilsSpec asserts tuplesOfRoot decodes a live root to the same tuples deserializeTuples produces from the IPC bytes.
When a ColumnarOperatorExecutor emits an Arrow result batch, the worker can now ship it straight downstream as a ColumnarFrame instead of decoding it to tuples and letting the network buffer re-encode those tuples back to Arrow. This removes the columns-to-rows-to-columns round-trip that otherwise gave back the vectorized-receive win at the output boundary. It is the first step of the per-edge columnar design (phase 1): eligibility is limited to the edges where shipping the whole batch to a single receiver is correct, and every other case falls back to the existing row output path unchanged. Output fast path (OutputManager): - tryEmitColumnarBatch(root, outputPortId) resolves the output port, checks eligibility, and on success serializes the root once and sends the resulting ColumnarFrame through each receiver's NetworkOutputBuffer, returning (portId, serializedByteCount) so the caller records output statistics; it returns None (caller falls back) when not eligible. It does not close the root (the caller owns it). An edge is eligible only when the port needs no storage, every partitioner on the port is one-to-one (a single receiver gets the whole batch), and the receiver buffers are in columnar mode. Hash/range/round-robin shuffles need per-row partitioning (a later phase) and broadcast/storage are deferred, so those return None. The emitted Arrow schema must equal the declared output-port schema, otherwise it falls back so the row path's enforceSchema can adjust it. - setColumnarOutput(enabled) flips the columnar flag on every current edge. This is a provisional phase-1 switch; phase 2 replaces it with a per-edge decision from the planner (local vs cross-node, and whether the receiver consumes columns). Network buffer (NetworkOutputBuffer): - sendColumnarFrame(frame) ships an already-serialized Arrow batch straight to the receiver, flushing any buffered tuples first so the batch keeps its place in the FIFO stream. Its columnar flag becomes a var so the output manager can set it per edge. Worker routing (DataProcessor): - The Emit(root) case now tries tryEmitColumnarBatch first and records the batch's output statistics in one call; only when the edge is not eligible does it materialize the root to tuples through the row output path. The root is freed in a finally either way. Batched statistics (StatisticsManager): - increaseOutputStatistics(portId, count, size) mirrors the batched input overload, so the columnar output path counts a whole batch's rows and serialized bytes in one call rather than once per tuple; the single-tuple overload delegates to it. Tests: - OutputManagerSpec covers tryEmitColumnarBatch: a one-to-one columnar edge ships a single ColumnarFrame that decodes back to the input rows; a hash-shuffle edge is declined (returns None, sends nothing); and an edge with columnar output off is declined. - NetworkOutputBufferColumnarSeamSpec covers sendColumnarFrame flushing buffered rows before the prebuilt frame, in order. - DataProcessorSpec adds the eligible-edge case (Emit goes out as one ColumnarFrame, no tuple round-trip, processTuple never runs, output statistics count the batch) and keeps the not-columnar case as the row-output fallback.
…perator The columnar wire is no longer a single global flag that flips every edge at once. The planner now decides it per link, and an edge ships Arrow only when the downstream operator can consume columns natively. This stops the worst case of the blanket switch, shipping Arrow to a row-only operator that then pays a per-tuple decode, and is the capability half of the per-edge design (phase 2). Locality (a same-JVM link gains nothing from serializing) is a later refinement; worker placement is not modeled at planning time and actor refs resolve lazily, so this change keeps that axis out. Operator capability (PhysicalOp): - isColumnarCapable resolves, by name and without loading or instantiating the class, whether the operator's executor mixes in ColumnarOperatorExecutor. Any lookup miss (a code-based or unknown executor) is not capable. This auto-discovers every columnar operator (filter, aggregate, and future ones) with no per-operator registration. Planner decision (ResourceAllocator, LinkConfig): - allocate sets LinkConfig.columnar = ApplicationConfig.enableColumnarWire && the downstream operator isColumnarCapable, for each link. The global flag is now a master switch folded into the per-edge decision rather than a second source of truth. Threading to the sender (proto + handlers): - AddPartitioningRequest gains a columnar bool. LinkWorkersHandler fills it from the link's LinkConfig; the worker's AddPartitioningHandler passes it to OutputManager.addPartitionerWithPartitioning, which constructs that edge's NetworkOutputBuffers in columnar mode. NetworkOutputBuffer.columnar now defaults to false (the planner is the authority) instead of reading ApplicationConfig.enableColumnarWire itself. Tests: - PhysicalOpSpec: isColumnarCapable holds for an executor that implements ColumnarOperatorExecutor, and not for a plain executor, a code-based executor, or an unresolvable class name. - OutputManagerSpec: addPartitionerWithPartitioning builds columnar buffers when told to (a flushed batch goes out as a ColumnarFrame) and row buffers by default (a DataFrame). - The AddPartitioningRequest constructions in WorkerSpec, LoggingSpec, and ArrowFlightActorBench pass columnar = false for the new field.
…ding rows A shuffle edge on the columnar wire no longer decodes the whole batch to tuples to route it. The columnar output path now partitions an Arrow batch in place: it assigns each row a bucket through the same getBucketIndex the row path uses, reading only the columns the partitioner needs, then gathers each bucket's rows in Arrow and sends one ColumnarFrame per non-empty bucket. Bucket assignment is identical to the row path by construction, so a hash shuffle feeding a join routes rows to exactly the workers the row path would. This completes the columnar wire for hash, range, round-robin, and broadcast edges (one-to-one was already handled); only the key columns are decoded, and the payload stays columnar end to end. Partitioner capability hooks: - Partitioner gains columnarSplits (false for one-to-one and broadcast, which send the whole batch to each receiver; true for hash/range/round-robin, which split it), bucketKeyColumns (the columns getBucketIndex reads: the hash keys, or the first range attribute; empty for round-robin), and requiresFullTuple (true only for a hash with no key columns, which hashes the whole tuple). HashBasedShufflePartitioner, RangeBasedShufflePartitioner, OneToOnePartitioner, and BroadcastPartitioner override these; the defaults cover round-robin. Columnar partitioning (OutputManager): - tryEmitColumnarBatch now accepts any partitioner (it previously required one-to-one). For each eligible link it calls emitColumnarToLink: a non-splitting partitioner serializes the root once and sends it to each receiver; a splitting partitioner builds a partial tuple of just the key columns per row (the full tuple only when requiresFullTuple), calls getBucketIndex to assign the bucket, gathers each bucket's rows into a new Arrow root (ArrowUtils.gather), serializes it, and sends it to that bucket's receiver. A per-manager RootAllocator backs the gathered roots; each is freed right after it is serialized, so its outstanding memory returns to zero. The method returns the total serialized bytes put on the wire for the port's output statistics; eligibility is unchanged (no storage, every edge in columnar mode, schema matches the output port). Tests: - OutputManagerSpec adds differential checks that the columnar shuffle routes rows to the same receivers the row-path partitioner would, for hash, range, and round-robin (which also checks the round-robin counter advances in step), with no row lost or duplicated, and that broadcast sends the whole batch to every receiver. The obsolete "decline a hash-shuffle edge" case (hash now shuffles columnar) is removed; the one-to-one and columnar-off cases remain.
…perator ProjectionOpExec can now project an Arrow columnar batch directly, so a projection no longer breaks a columnar region: it keeps the batch columnar instead of decoding every tuple and rebuilding it. Because projection only selects, reorders, and renames columns, each output column is produced by moving its source column's Arrow buffers into the output root under the alias (an Arrow transfer, not a per-cell copy) rather than reading and rewriting values. The operator mixes in ColumnarOperatorExecutor, so the planner's per-edge decision (added earlier) now also routes columns through projections. Columnar projection (ProjectionOpExec): - processColumnarBatch resolves the same (source column -> output alias) pairs that the row path's project() derives, honoring drop mode (keep every column whose lowercased name is not in the drop set) and select mode (the descriptor's attribute units). It builds the output schema from the alias names and the source column types, creates the output root from that schema, and for each output column moves the source vector's buffers into it via Arrow makeTransferPair, renamed to the alias. The result is Emit(outRoot), which the columnar output path ships downstream with no round-trip. - It falls back to the row path (ColumnarResult.Unsupported) for the shapes a single per-column move cannot express or that the row path treats as errors: an empty selection, a source column feeding more than one output (its buffers cannot be moved twice), a duplicated output alias (the row path raises on this), or a source column that does not resolve. A per-operator RootAllocator backs the output roots and is freed in close(). Tests: - ProjectionOpExecSpec adds a columnar-vs-row check that projecting with reorder and rename over a multi-row batch yields, through processColumnarBatch, the same schema and values as processTuple; a drop-mode columnar check; and a check that a duplicate alias makes processColumnarBatch return Unsupported (so the row path raises the error as before).
…lized pipeline Adds a benchmark that measures a whole columnar region, project -> filter -> aggregate, against the row path for the same pipeline, with each operator on its own worker so the batch crosses a serialized edge between them. It exercises the real operators (ProjectionOpExec, SpecializedFilterOpExec, AggregateOpExec) and the real Pekko/Kryo serialization, so the numbers reflect a cross-worker (cross-node) region, which is where the columnar wire is meant to pay off, rather than the per-edge microbenchmark alone. ColumnarPipelineBenchSpec: - The row path serializes a DataFrame on each of the three edges (source->project, project->filter, filter->aggregate) and runs each operator tuple-at-a-time: project maps and re-enforces each tuple, filter runs the predicate per tuple, aggregate accumulates per tuple. The source->project edge carries the full wide rows (two string payload columns the projection then drops), as a real source edge would. - The columnar path serializes an Arrow ColumnarFrame on each edge and runs each operator's processColumnarBatch, staying columnar across all three hops: projection moves columns, filter emits the surviving rows, aggregate accumulates straight off the Arrow vectors. - Both reset the aggregate via open() per run, warm up, and report the median of several trials for 2k/20k/100k source rows. It is skipped unless run with -Dcolumnar.bench=true, so it does not run in CI. Findings (this machine, JDK 17, median of 7): the columnar region runs end to end in about 0.21x the row path at 2k rows, 0.06x at 20k, and 0.05x at 100k (roughly 5x, 16x, and 19x faster), the gap widening with batch size. This matches the per-edge microbenchmark and the vectorized-operator speedups the literature reports: once a run of operators consumes columns natively, keeping the batch in Arrow across serialized edges avoids both the per-tuple Kryo round-trip and the per-tuple operator work that dominate the row path.
Contributor
Automated Reviewer SuggestionsBased on the
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this PR?
getBucketIndexthe row partitioner uses, so the columnar and row paths route every row to the same worker.Any related issues, documentation, discussions?
Closes: #9544
Part of: #9542
Stacked on Part 2 (#9548); isolated diff: Ma77Ball/texera@columnar/02-output-planner...columnar/03-shuffle-projection
How was this PR tested?
sbt "WorkflowOperator/testOnly *ProjectionOpExecSpec", expect the native projection to match the row projection across types and nulls.sbt "WorkflowExecutionService/testOnly *OutputManagerSpec", expect the columnar-shuffle bucket-parity cases to pass (the differential gate against the row partitioner).sbt -Dcolumnar.bench=true "WorkflowExecutionService/testOnly *ColumnarPipelineBenchSpec"to see the end-to-end columnar region vs row timings.Was this PR authored or co-authored using generative AI tooling?
Co-authored with Claude Opus 4.8 (Anthropic) in compliance with ASF policy.