Repository navigation
Conversation
The foundation for a columnar Arrow wire: an operator-side contract for consuming and producing Arrow batches directly, the Arrow helpers it needs, and the first two operators that implement it (the specialized filter and the Arrow file scan). Nothing is wired into the messaging layer yet, so no operator receives columnar batches at runtime; these are the capabilities the wire will drive. Contract (ColumnarOperatorExecutor): - ColumnarOperatorExecutor is a trait with processColumnarBatch(arrowIpcBytes, port) that returns a ColumnarResult. An operator mixes it in to declare it can consume an Arrow batch without decoding each row to a Tuple. - ColumnarResult is the sealed outcome of consuming one batch: Emit(root) (emit this Arrow result, caller serializes and closes the root), EmitRows(iterator) (emit row-shaped output, e.g. a join), Consumed (consumed with nothing to emit now, a blocking operator accumulating state), and Unsupported (not handled, caller falls back to the row path). - SourceOperatorExecutor gains produceColumnarBatch(), defaulting to None, so a source whose data is already columnar can hand out Arrow roots instead of going through produceTuple; the caller owns and closes each root. ArrowUtils helpers: - serializeRoot writes a VectorSchemaRoot to Arrow IPC stream bytes without closing it; deserializeRootFold reads one IPC batch, applies a function to the live root, and closes the reader (the function must copy out what it keeps). - selectRows copies the rows a boolean keep-mask selects into a new compacted root; selectionOf returns just the surviving row indices (a selection vector) without copying any cell, and gather materializes the rows a selection vector names into a new root, so a consumer that only needs a count or an aggregate reads indices and never materializes. - projectionIndices resolves column names to their Arrow field indices by exact name; getCell reads one cell as the Texera value for an attribute type, returning null on a conversion failure to match the row path's lenient decode. SpecializedFilterOpExec columnar path: - The operator now also mixes in ColumnarOperatorExecutor while keeping its row path (the per-tuple OR over predicates) unchanged. columnarMask builds a boolean survivor mask for a single numeric predicate (Float8Vector or IntVector) read straight from the Arrow vector, and returns None for any other predicate or column shape so the caller falls back to rows. - processColumnarBatch deserializes the batch and returns Emit(selectRows(mask)) when the mask applies, else Unsupported. columnarSelection returns the survivor indices without compacting, and columnarMatchCount is the fused filter-then-count that reports the survivor count with no batch materialized. A per-operator RootAllocator backs these and is freed in close(). ArrowSourceOpExec columnar produce: - produceColumnarBatch hands each already-Arrow file batch out as a VectorSchemaRoot, moving the loaded batch's vectors into an independent root via a zero-copy TransferPair so the caller can serialize and close it while the reader loads the next batch. It returns None (the row path) when offset or limit is set.
Adds the wire that carries a data batch between workers as Arrow IPC bytes (a ColumnarFrame) instead of decoded Tuples (a DataFrame), behind an off-by-default flag. When enabled, every operator's output goes out as Arrow and is decoded back at the receiver, so the format is transparent to the row path; the payoff is on cross-worker links, where one Arrow IPC block replaces per-tuple serialization. Consuming the Arrow batch without decoding (for a ColumnarOperatorExecutor) is a later optimization; this lands the transport and keeps results identical with the flag on or off. Flag: - ApplicationConfig.enableColumnarWire reads network-buffering.enable-columnar-wire, defaulting to false (override NETWORK_BUFFERING_ENABLE_COLUMNAR_WIRE). Off by default, so the default wire is DataFrame and unchanged. Payload: - ColumnarFrame(arrowIpcBytes, rowCount, schema) is a new DataPayload. It carries a batch as an Arrow IPC stream plus the row count and Texera schema that describe it without decoding. inMemSize is the byte length. Send side (NetworkOutputBuffer): - The buffer takes a `columnar` flag (defaulting to ApplicationConfig.enableColumnarWire). On flush, when columnar is on it serializes the buffered tuples to Arrow once (all tuples on a link share the schema) and sends a ColumnarFrame; otherwise it sends a DataFrame as before. An empty flush still sends nothing. Receive side: - DataProcessor.processDataPayload handles ColumnarFrame by decoding the Arrow bytes back to tuples and running the identical batch path as DataFrame; the shared startInputBatch helper carries the common state-transition, initBatch, and first-tuple step for both. A columnar batch bound for a Python worker is decoded back to tuples in PythonProxyClient.sendData and handed over as a Data batch, so the Python path is unaffected. ArrowUtils: - serializeTuples(schema, tuples) writes a tuple batch to Arrow IPC stream bytes; deserializeTuples(bytes) reads them back to tuples. These are the ColumnarFrame encode/decode endpoints. Tests: - ArrowUtilsSpec covers the serializeTuples/deserializeTuples round-trip over a multi-row batch of every field type (including a null row and an empty batch), the payload contract the wire depends on. - NetworkOutputBufferColumnarSeamSpec checks the buffer emits a ColumnarFrame (carrying the same rows, row count, and schema) when the flag is on and a DataFrame when off, on both the manual and the size-triggered flush, and nothing on an empty flush. - SelectionVectorFilterSpec checks the specialized filter's native-Arrow path (selection vector, fused filter-count, and the compacting Emit path) against the row path as the source of truth.
AggregateOpExec can now consume an Arrow columnar batch directly, accumulating each group's partial result by reading the group-key and aggregated cells straight from their Arrow vectors instead of decoding a Tuple per row. The accumulation itself is shared with the row path, so a grouped aggregate produces the identical result whether its input arrives as rows or as a columnar batch. Value-based accumulation (DistributedAggregation): - DistributedAggregation now accumulates one already-extracted value via iterateValue(partial, value), and carries an extract(tuple) that pulls the single cell an aggregation reads from a tuple (null for COUNT(*)). The row path's iterate(partial, tuple) is derived as iterateValue(partial, extract(tuple)), so accumulation is defined once and both paths use it. init, merge, and finalAgg are unchanged. - AggregationOperation's six functions (sum, count, average, min, max, concat) are rewritten to this shape: each supplies iterateValue (the per-value logic) and extract (tuple.getField of its attribute, or null for COUNT(*)). count keeps its captured countAllRows flag, so COUNT(*) counts every row and COUNT(column) counts non-null values. average reads a value numerically via a value-based helper (a timestamp counts as its epoch milliseconds, any other value through its string form, null skipped), matching the previous tuple-based reading. Columnar accumulate (AggregateOpExec): - The operator mixes in ColumnarOperatorExecutor. processColumnarBatch deserializes the batch and runs accumulateRoot, which resolves once per batch the source-column index and Texera type of each group-by key and each aggregation's input column (index -1 for COUNT(*)), then for each row builds the group key and feeds each aggregation its cell value through iterateValue. It is a blocking operator, so it emits nothing per batch (ColumnarResult.Consumed); results are produced at onFinish via the existing path. A per-operator RootAllocator backs the decode and is freed in close(). Tests: - AggregateOpSpec adds a columnar-vs-row check: a grouped SUM/COUNT/AVERAGE/MIN/MAX over rows including a null value gives the same per-group result through processColumnarBatch as through processTuple. - DistributedAggregationSpec is updated to the new constructor (iterateValue + extract) and asserts iterate is derived from them (the row path equals the value path).
Adds a benchmark that measures the columnar wire against the row wire on the real Pekko/Kryo stack, and speeds up the Arrow-to-tuple decode it exercises. Decode (ArrowUtils): - getTexeraTuple gains an overload that takes the already-resolved Texera schema, and the no-schema overload delegates to it. deserializeTuples now resolves the Texera schema, the field-vector array, and each column's attribute type once per stream and builds each row with a plain index loop (getCell per column into a fresh Array, then the Tuple constructor), instead of rebuilding the schema and allocating a Scala collection per row. Benchmark (ColumnarWireBenchSpec): - Serializes batches through the configured Pekko serialization (SerializationExtension on an ActorSystem built from AmberRuntime.pekkoConfig), so the row path is a real Kryo DataFrame round-trip and the columnar path is ArrowUtils.serializeTuples wrapped in a ColumnarFrame. For narrow-numeric, wide-mixed, and string-heavy workloads at 2k/20k/100k rows it reports sender encode time, receiver decode time, and serialized payload size for each path. - A second scenario measures the decisive case: a vectorized receiver (a SpecializedFilterOpExec) that reads the Arrow columns directly via processColumnarBatch, against the row path that deserializes the DataFrame and runs the predicate per tuple. - It is skipped unless run with -Dcolumnar.bench=true, so it does not run in CI. Findings (this machine, JDK 17, median of 7): the columnar wire has a smaller payload (about 0.45x-0.91x) and a cheaper encode (about 0.5x at 100k rows), but decoding Arrow back to tuples is far more expensive than Kryo rehydrating a DataFrame (roughly 40x-68x), so a receiver that decodes to rows is a net loss. A receiver that vectorizes instead wins clearly: the filter's processColumnarBatch is about 0.13x the row path's receive+process at 100k rows (around 7x faster). So the columnar wire pays off only when the receiver consumes columns natively; routing a ColumnarFrame to a ColumnarOperatorExecutor's processColumnarBatch in the worker (instead of always decoding) is the next step.
Routes an incoming Arrow columnar batch to a ColumnarOperatorExecutor's processColumnarBatch in the worker, so a columnar-capable operator consumes the Arrow columns directly instead of the batch always being decoded back to tuples first. The benchmark showed decoding Arrow to tuples is the columnar wire's dominant cost, far more than Kryo rehydrating a DataFrame, so the wire only pays off when the receiver reads columns natively; this is the path that realizes that win. Worker routing (DataProcessor): - A ColumnarFrame now goes through processColumnarFrame. When the executor is a ColumnarOperatorExecutor, the worker transitions to RUNNING, binds the executor's state references, and hands it the Arrow IPC bytes via processColumnarBatch, then routes the ColumnarResult: Emit(root) is materialized to output tuples (the existing output path is tuple-based) and the root is freed; EmitRows installs the returned rows as the output; Consumed emits nothing now (a blocking operator that accumulated the batch); Unsupported, or any non-columnar executor, falls back to decoding the batch to tuples and processing it as a DataFrame. The READY->RUNNING transition is factored out of startInputBatch into transitToRunning, shared by both paths. For a natively consumed batch the per-tuple input manager is left empty (there is no per-tuple input to drain), so the DP loop drains only the output iterator. - StatisticsManager gains a batched increaseInputStatistics(portId, count, size) so the columnar path reports a whole batch's row count and serialized size in one call rather than once per tuple; the single-tuple overload delegates to it. Root-to-tuples decode (ArrowUtils): - tuplesOfRoot decodes an already-open VectorSchemaRoot to tuples (the Emit output path's root-to-rows step). Like deserializeTuples it resolves the Texera schema, the field vectors, and each column's type once and reads each row with a plain index loop, but it works over a live root without closing it. Tests: - DataProcessorSpec adds the four routing outcomes plus the non-columnar case, driving real ColumnarFrames through processDataPayload with fake executors: Emit (a pass-through filter) emits the batch's rows and never calls processTuple; EmitRows emits the returned rows; Consumed emits nothing and still counts the batch's input statistics; Unsupported and a plain row-only executor both decode the frame and run it through processTuple per row. - ArrowUtilsSpec asserts tuplesOfRoot decodes a live root to the same tuples deserializeTuples produces from the IPC bytes.
Contributor
Automated Reviewer SuggestionsBased on the
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #9549 +/- ##
============================================
- Coverage 91.57% 91.45% -0.12%
- Complexity 5159 5192 +33
============================================
Files 1300 1301 +1
Lines 58092 58345 +253
Branches 7615 7666 +51
============================================
+ Hits 53198 53360 +162
- Misses 2958 3028 +70
- Partials 1936 1957 +21
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 396 | 0.242 | 23,951/36,619/36,619 us | 🔴 +11.3% / 🔴 +185.8% |
| 🔴 | bs=100 sw=10 sl=64 | 811 | 0.495 | 118,669/151,766/151,766 us | 🟢 -10.1% / 🔴 +85.5% |
| ⚪ | bs=1000 sw=10 sl=64 | 970 | 0.592 | 1,027,168/1,096,428/1,096,428 us | ⚪ within ±5% / 🔴 +43.5% |
Baseline details
Latest main 4db10cf from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 396 tuples/sec | 446 tuples/sec | 1,084 tuples/sec | -11.2% | -63.5% |
| bs=10 sw=10 sl=64 | MB/s | 0.242 MB/s | 0.272 MB/s | 0.662 MB/s | -11.0% | -63.4% |
| bs=10 sw=10 sl=64 | p50 | 23,951 us | 21,528 us | 9,548 us | +11.3% | +150.9% |
| bs=10 sw=10 sl=64 | p95 | 36,619 us | 40,478 us | 12,811 us | -9.5% | +185.8% |
| bs=10 sw=10 sl=64 | p99 | 36,619 us | 40,478 us | 15,432 us | -9.5% | +137.3% |
| bs=100 sw=10 sl=64 | throughput | 811 tuples/sec | 859 tuples/sec | 1,467 tuples/sec | -5.6% | -44.7% |
| bs=100 sw=10 sl=64 | MB/s | 0.495 MB/s | 0.525 MB/s | 0.895 MB/s | -5.7% | -44.7% |
| bs=100 sw=10 sl=64 | p50 | 118,669 us | 113,479 us | 74,441 us | +4.6% | +59.4% |
| bs=100 sw=10 sl=64 | p95 | 151,766 us | 168,823 us | 81,800 us | -10.1% | +85.5% |
| bs=100 sw=10 sl=64 | p99 | 151,766 us | 168,823 us | 103,712 us | -10.1% | +46.3% |
| bs=1000 sw=10 sl=64 | throughput | 970 tuples/sec | 972 tuples/sec | 1,517 tuples/sec | -0.2% | -36.1% |
| bs=1000 sw=10 sl=64 | MB/s | 0.592 MB/s | 0.593 MB/s | 0.926 MB/s | -0.2% | -36.1% |
| bs=1000 sw=10 sl=64 | p50 | 1,027,168 us | 1,028,890 us | 725,911 us | -0.2% | +41.5% |
| bs=1000 sw=10 sl=64 | p95 | 1,096,428 us | 1,061,154 us | 764,152 us | +3.3% | +43.5% |
| bs=1000 sw=10 sl=64 | p99 | 1,096,428 us | 1,061,154 us | 808,901 us | +3.3% | +35.5% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,505.13,200,128000,396,0.242,23950.97,36619.29,36619.29
1,100,10,64,20,2464.83,2000,1280000,811,0.495,118668.51,151765.57,151765.57
2,1000,10,64,20,20620.07,20000,12800000,970,0.592,1027167.56,1096428.03,1096428.03This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this PR?
ColumnarOperatorExecutor/ColumnarResultcontract so an operator can opt into reading Arrow columns directly; when an operator cannot, the engine decodes to rows, so results never differ.ColumnarFramewire (a batch of Arrow columns alongside the rowDataFrame), gated byApplicationConfig.enableColumnarWireand off by default.DataProcessorso a capable filter or aggregate reads the Arrow columns directly instead of decoding to tuples.Any related issues, documentation, discussions?
Closes: #9547
Part of: #9542
Discussion: #8571
How was this PR tested?
sbt "WorkflowCore/testOnly *ArrowUtilsSpec", expect the Arrow round-trip and faster-decode specs to pass with values identical to the row decode.sbt "WorkflowExecutionService/testOnly *DataProcessorSpec *OutputManagerSpec", expect the columnar-frame routing and vectorized-receive cases to pass.enableColumnarWire=trueincommon/config/src/main/resources/application.conf, run a CSV scan into a filter at http://texera.localhost, and confirm the output rows are identical to a run with the flag off.Was this PR authored or co-authored using generative AI tooling?
Co-authored with Claude Opus 4.8 (Anthropic) in compliance with ASF policy.