Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions changelog.d/ITL-865.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
### Fixed

- An async barrier operator (`PolarsFilter`, `GroupBy`, `MergeJoin`, and the default
`async_execute` of `UnaryOperator`, `BinaryOperator` and `StaticOutputOperatorPod`) whose
input delivers no rows emits no rows and finishes, as the sync path returns an empty
stream. It raised `ValueError: Cannot materialize an empty list of rows into a stream`, so
a pipeline with a filter that matches no row upstream of such an operator completed under
`SyncPipelineOrchestrator` and failed under `AsyncPipelineOrchestrator`. (ITL-865)

### Known limitations

- An async channel carries rows, not a schema. Where the sync path returns an empty stream
with typed columns, the async path emits nothing, and an operator node that received no
rows reads back columns typed from its Python output schema (`int8` becomes `int64`).
Giving async execution the stream's schema is ITL-858; the read-back types are ITL-912.
(ITL-865)
33 changes: 33 additions & 0 deletions changelog.d/ITL-906.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
### Breaking changes and migration

#### The async orchestrator names system-tag columns as the sync orchestrator does (ITL-906, ITL-885)

`MergeJoin`, `Batch` and `GroupBy` put their inputs' pipeline hashes into the names of the
system-tag columns they write. When one of these operators read an input that is not a
source (an operator's or a function pod's output), `AsyncPipelineOrchestrator` used a
different hash than `SyncPipelineOrchestrator`, so the same graph got different system-tag
column names, and different record ids, depending on which orchestrator ran it. The async
orchestrator now uses the input's pipeline hash, and its names and record ids are the ones
the sync orchestrator has always produced. Nothing a sync run produces changes, and graphs
where these operators read a source directly are unaffected.

Every operator's `async_execute` now requires the keyword argument `input_pipeline_hashes`,
one pipeline hash per input, as the operator node always passes it; a missing argument raises
`TypeError`. The operators that use the hashes (`Join`, `MergeJoin`, `Batch`, `GroupBy`,
`PolarsFilter`, and any operator on the default barrier path) raise `ValueError` when the count
does not match the inputs; the per-row operators and `SemiJoin` take the argument without
using it. Before, these
operators fell back without it to the schema-only hash, and `Join`'s async path to names built
from the input positions (`::0`, `::1`, ...), neither of which the sync path uses. Code that
calls an operator's `async_execute` directly must pass the inputs' pipeline hashes.

This also fixes ITL-885: a store written by one orchestrator can now be extended by the
other. Before, the function node downstream of such an operator wrote two sets of
system-tag columns into one pipeline table, and the second run failed while storing its
results.

Migration: stores written by v0.2 are not affected, because v0.3 never reads a table
written by v0.2. Only a store written by an async run of an unreleased v0.3 build, for a
graph with `MergeJoin`, `Batch` or `GroupBy` after an operator or a function pod, holds the
old names: a later run on that store fails when it writes the pipeline table of the function
node downstream. Delete that table, or use a fresh store, and run again.
24 changes: 24 additions & 0 deletions design_issues/ITL-865-O8.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
## `src/orcapod/core/operators/` — Async execution

### O8 — Async barrier operators raised on an input with no rows
**Status:** resolved (no rows emitted); a typed empty result on async waits for ITL-858
**Severity:** high
**Issue:** ITL-865

The barrier-mode `async_execute` of `UnaryOperator` and `BinaryOperator` (used by
`PolarsFilter`, `GroupBy` and `MergeJoin`) and the default `StaticOutputOperatorPod.async_execute`
rebuilt each input from its rows with `_materialize_to_stream`, which raises
`ValueError: Cannot materialize an empty list of rows into a stream` when a channel delivers
none. The sync path returns an empty stream, so a pipeline with a filter that matches no row
upstream of such an operator completed under the sync orchestrator and failed under the async
one.

**Fix:** a barrier path whose input (any input, for the binary and default paths) delivers no
rows emits no rows and closes its output. That is the sync result of every barrier operator
that exists: an operator never synthesizes a row, and `MergeJoin` is an inner join. The sync
path's typed empty stream needs a schema the channels do not carry, so on async an operator
node that received no rows still reads back columns typed from its Python output schema
(ITL-858, ITL-912). `tests/test_core/operators/test_async_sync_equivalence.py` runs every
operator with an async path with all inputs and with each one input empty;
`tests/test_pipeline/test_operator_node_async_readback.py` runs sync and async jobs with a
filter matching no row upstream of barrier operators and of a function pod.
30 changes: 30 additions & 0 deletions design_issues/ITL-906-O7.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
## `src/orcapod/core/operators/` — Async execution

### O7 — Async system-tag names after a non-source input differed from sync
**Status:** resolved
**Severity:** high
**Issue:** ITL-906

The barrier-mode `async_execute` (`UnaryOperator`, `BinaryOperator`, the default
`StaticOutputOperatorPod.async_execute`, and `Batch`) rebuilt each input from its rows with
`_materialize_to_stream`, an `ArrowTableStream` without a producer whose pipeline hash is the
schema-only hash of a root. `MergeJoin` (system tag rule 2), `Batch` and `GroupBy` (rule 3)
name system-tag columns after their inputs' pipeline hashes, so when an input was an
operator's or a function pod's output the async names, `MergeJoin`'s canonical input order
and every record id derived from them differed from the sync path. The `input_pipeline_hashes`
keyword the operator node passes was ignored. A function node downstream then wrote two sets
of system-tag columns into one pipeline table, and a run on one orchestrator could not extend
a store written by the other (ITL-885).

**Fix:** a barrier path rebuilds each input with `_materialize_input_stream(rows,
pipeline_hash)`, a stream that reports the input's pipeline hash (`_RowsOfStream`).
`input_pipeline_hashes` is a required keyword of every operator's `async_execute`, with no
default and no fallback: the barrier paths and `Join` (whose async path had named system tags
after the input positions `0`, `1`, ... when no hashes were given) raise `ValueError` when the
count does not match the inputs, and the per-row operators and `SemiJoin` take it and do not
use it. `_materialize_to_stream(rows)` only holds output rows. Async names and record ids now equal the sync ones; nothing a
sync run produces moves, and no golden value moves (the golden cases run the sync path).
`tests/test_core/operators/test_async_sync_equivalence.py` runs every operator with an async
path over inputs behind an operator and behind a function pod;
`tests/test_pipeline/test_mixed_orchestrators.py` runs one store with both orchestrators in
both orders.
7 changes: 6 additions & 1 deletion docs/concepts/operators.md
Original file line number Diff line number Diff line change
Expand Up @@ -562,7 +562,12 @@ silent widening; convert explicitly.
the join runs, not when the pipeline is built.
- **Sync and async execution agree.** An operator run asynchronously (`async_execute`, as in
an async pipeline) yields rows with the same Arrow types, column order, system tags and
content hashes as the same operator run synchronously.
content hashes as the same operator run synchronously, whether its inputs are sources or
the outputs of other operators and function pods. An input that delivers no rows gives no
rows on both paths. One limit remains: an async channel carries rows, not a schema, so
where the sync path returns an empty stream with typed columns the async path emits
nothing, and an operator node that received no rows reads back columns typed from its
Python output schema (ITL-858, ITL-912).

```python
import pyarrow as pa
Expand Down
62 changes: 55 additions & 7 deletions src/orcapod/core/operators/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -115,12 +115,35 @@ async def async_execute(
inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]],
output: WritableChannel[tuple[TagProtocol, DataProtocol]],
*,
input_pipeline_hashes: Sequence[ContentHash] | None = None,
input_pipeline_hashes: Sequence[ContentHash],
) -> None:
"""Barrier-mode: collect single input, run unary_static_process, emit."""
"""Barrier-mode: collect single input, run unary_static_process, emit.

An input with no rows emits no rows (ITL-865).

Args:
inputs: The one readable input channel.
output: Writable channel for downstream consumption.
input_pipeline_hashes: The input stream's pipeline hash, as a
one-element sequence. The rebuilt input reports it, so
system-tag names equal the sync path's (ITL-906).

Raises:
ValueError: If ``input_pipeline_hashes`` does not have one hash per input.
"""
try:
if len(input_pipeline_hashes) != len(inputs):
raise ValueError(
f"input_pipeline_hashes length ({len(input_pipeline_hashes)}) "
f"must match inputs length ({len(inputs)})"
)
rows = await inputs[0].collect()
stream = self._materialize_to_stream(rows)
if not rows:
# No rows in, no rows out: an operator never synthesizes a row.
# The typed empty stream the sync path returns needs a schema the
# channel does not carry (ITL-858), so nothing is emitted (ITL-865).
return
stream = self._materialize_input_stream(rows, input_pipeline_hashes[0])
result = self.static_process(stream)
for tag, data in result.iter_data():
await output.send((tag, data))
Expand Down Expand Up @@ -199,15 +222,40 @@ async def async_execute(
inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]],
output: WritableChannel[tuple[TagProtocol, DataProtocol]],
*,
input_pipeline_hashes: Sequence[ContentHash] | None = None,
input_pipeline_hashes: Sequence[ContentHash],
) -> None:
"""Barrier-mode: collect both inputs concurrently, run binary_static_process, emit."""
"""Barrier-mode: collect both inputs concurrently, run binary_static_process, emit.

An input with no rows emits no rows, as an inner join does (ITL-865).

Args:
inputs: The left and right readable input channels.
output: Writable channel for downstream consumption.
input_pipeline_hashes: The left and right input streams' pipeline
hashes. Each rebuilt input reports its own, so system-tag names
and canonical input order equal the sync path's (ITL-906).

Raises:
ValueError: If ``input_pipeline_hashes`` does not have one hash per input.
"""
try:
if len(input_pipeline_hashes) != len(inputs):
raise ValueError(
f"input_pipeline_hashes length ({len(input_pipeline_hashes)}) "
f"must match inputs length ({len(inputs)})"
)
left_hash, right_hash = input_pipeline_hashes
left_rows, right_rows = await asyncio.gather(
inputs[0].collect(), inputs[1].collect()
)
left_stream = self._materialize_to_stream(left_rows)
right_stream = self._materialize_to_stream(right_rows)
if not left_rows or not right_rows:
# The one barrier binary operator, MergeJoin, is an inner join: an
# input with no rows gives no rows, as on the sync path. The typed
# empty stream needs a schema the channels do not carry (ITL-858),
# so nothing is emitted (ITL-865).
return
left_stream = self._materialize_input_stream(left_rows, left_hash)
right_stream = self._materialize_input_stream(right_rows, right_hash)
result = self.static_process(left_stream, right_stream)
for tag, data in result.iter_data():
await output.send((tag, data))
Expand Down
27 changes: 22 additions & 5 deletions src/orcapod/core/operators/batch.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
from orcapod.core.streams import ArrowTableStream
from orcapod.protocols.core_protocols import DataProtocol, StreamProtocol, TagProtocol
from orcapod.system_constants import constants
from orcapod.types import ColumnConfig
from orcapod.types import ColumnConfig, ContentHash
from orcapod.utils import arrow_utils
from orcapod.utils.lazy_module import LazyModule

Expand Down Expand Up @@ -152,21 +152,38 @@ async def async_execute(
self,
inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]],
output: WritableChannel[tuple[TagProtocol, DataProtocol]],
**kwargs: Any,
*,
input_pipeline_hashes: Sequence[ContentHash],
) -> None:
"""Streaming batch: emit full batches as they accumulate.

When ``batch_size > 0``, each group of ``batch_size`` rows is
materialized and emitted immediately, allowing downstream consumers
to start processing before all input is consumed. When
``batch_size == 0`` (batch everything), falls back to barrier mode.

Args:
inputs: The one readable input channel.
output: Writable channel for downstream consumption.
input_pipeline_hashes: The input stream's pipeline hash, as a
one-element sequence. Every rebuilt batch reports it, so the
folded system-tag names end in it as on the sync path (ITL-906).

Raises:
ValueError: If ``input_pipeline_hashes`` does not have one hash per input.
"""
try:
if len(input_pipeline_hashes) != len(inputs):
raise ValueError(
f"input_pipeline_hashes length ({len(input_pipeline_hashes)}) "
f"must match inputs length ({len(inputs)})"
)
pipeline_hash = input_pipeline_hashes[0]
if self.batch_size == 0:
# Must collect all rows — barrier fallback
rows = await inputs[0].collect()
if rows:
stream = self._materialize_to_stream(rows)
stream = self._materialize_input_stream(rows, pipeline_hash)
result = self.unary_static_process(stream)
for tag, data in result.iter_data():
await output.send((tag, data))
Expand All @@ -176,15 +193,15 @@ async def async_execute(
async for tag, data in inputs[0]:
batch.append((tag, data))
if len(batch) >= self.batch_size:
stream = self._materialize_to_stream(batch)
stream = self._materialize_input_stream(batch, pipeline_hash)
result = self.unary_static_process(stream)
for out_tag, out_data in result.iter_data():
await output.send((out_tag, out_data))
batch = []

# Flush partial batch
if batch and not self.drop_partial_batch:
stream = self._materialize_to_stream(batch)
stream = self._materialize_input_stream(batch, pipeline_hash)
result = self.unary_static_process(stream)
for out_tag, out_data in result.iter_data():
await output.send((out_tag, out_data))
Expand Down
54 changes: 45 additions & 9 deletions src/orcapod/core/operators/column_selection.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
from orcapod.errors import InputValidationError
from orcapod.protocols.core_protocols import DataProtocol, StreamProtocol, TagProtocol
from orcapod.system_constants import constants
from orcapod.types import ColumnConfig, Schema
from orcapod.types import ColumnConfig, ContentHash, Schema
from orcapod.utils.lazy_module import LazyModule

if TYPE_CHECKING:
Expand Down Expand Up @@ -114,9 +114,18 @@ async def async_execute(
self,
inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]],
output: WritableChannel[tuple[TagProtocol, DataProtocol]],
**kwargs: Any,
*,
input_pipeline_hashes: Sequence[ContentHash],
) -> None:
"""Streaming: select tag columns per row without materializing."""
"""Streaming: select tag columns per row without materializing.

Args:
inputs: The one readable input channel.
output: Writable channel for downstream consumption.
input_pipeline_hashes: Not used: this operator keeps system-tag
names as they are, so no column is named after an input's
pipeline hash.
"""
try:
tags_to_drop: list[str] | None = None
async for tag, data in inputs[0]:
Expand Down Expand Up @@ -233,9 +242,18 @@ async def async_execute(
self,
inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]],
output: WritableChannel[tuple[TagProtocol, DataProtocol]],
**kwargs: Any,
*,
input_pipeline_hashes: Sequence[ContentHash],
) -> None:
"""Streaming: select data columns per row without materializing."""
"""Streaming: select data columns per row without materializing.

Args:
inputs: The one readable input channel.
output: Writable channel for downstream consumption.
input_pipeline_hashes: Not used: this operator keeps system-tag
names as they are, so no column is named after an input's
pipeline hash.
"""
try:
pkts_to_drop: list[str] | None = None
async for tag, data in inputs[0]:
Expand Down Expand Up @@ -347,9 +365,18 @@ async def async_execute(
self,
inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]],
output: WritableChannel[tuple[TagProtocol, DataProtocol]],
**kwargs: Any,
*,
input_pipeline_hashes: Sequence[ContentHash],
) -> None:
"""Streaming: drop tag columns per row without materializing."""
"""Streaming: drop tag columns per row without materializing.

Args:
inputs: The one readable input channel.
output: Writable channel for downstream consumption.
input_pipeline_hashes: Not used: this operator keeps system-tag
names as they are, so no column is named after an input's
pipeline hash.
"""
try:
effective_drops: list[str] | None = None
async for tag, data in inputs[0]:
Expand Down Expand Up @@ -468,9 +495,18 @@ async def async_execute(
self,
inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]],
output: WritableChannel[tuple[TagProtocol, DataProtocol]],
**kwargs: Any,
*,
input_pipeline_hashes: Sequence[ContentHash],
) -> None:
"""Streaming: drop data columns per row without materializing."""
"""Streaming: drop data columns per row without materializing.

Args:
inputs: The one readable input channel.
output: Writable channel for downstream consumption.
input_pipeline_hashes: Not used: this operator keeps system-tag
names as they are, so no column is named after an input's
pipeline hash.
"""
try:
effective_drops: list[str] | None = None
async for tag, data in inputs[0]:
Expand Down
Loading
Loading