diff --git a/changelog.d/ITL-865.md b/changelog.d/ITL-865.md new file mode 100644 index 000000000..e78f08fa6 --- /dev/null +++ b/changelog.d/ITL-865.md @@ -0,0 +1,16 @@ +### Fixed + +- An async barrier operator (`PolarsFilter`, `GroupBy`, `MergeJoin`, and the default + `async_execute` of `UnaryOperator`, `BinaryOperator` and `StaticOutputOperatorPod`) whose + input delivers no rows emits no rows and finishes, as the sync path returns an empty + stream. It raised `ValueError: Cannot materialize an empty list of rows into a stream`, so + a pipeline with a filter that matches no row upstream of such an operator completed under + `SyncPipelineOrchestrator` and failed under `AsyncPipelineOrchestrator`. (ITL-865) + +### Known limitations + +- An async channel carries rows, not a schema. Where the sync path returns an empty stream + with typed columns, the async path emits nothing, and an operator node that received no + rows reads back columns typed from its Python output schema (`int8` becomes `int64`). + Giving async execution the stream's schema is ITL-858; the read-back types are ITL-912. + (ITL-865) diff --git a/changelog.d/ITL-906.md b/changelog.d/ITL-906.md new file mode 100644 index 000000000..ac9b37d6e --- /dev/null +++ b/changelog.d/ITL-906.md @@ -0,0 +1,33 @@ +### Breaking changes and migration + +#### The async orchestrator names system-tag columns as the sync orchestrator does (ITL-906, ITL-885) + +`MergeJoin`, `Batch` and `GroupBy` put their inputs' pipeline hashes into the names of the +system-tag columns they write. When one of these operators read an input that is not a +source (an operator's or a function pod's output), `AsyncPipelineOrchestrator` used a +different hash than `SyncPipelineOrchestrator`, so the same graph got different system-tag +column names, and different record ids, depending on which orchestrator ran it. The async +orchestrator now uses the input's pipeline hash, and its names and record ids are the ones +the sync orchestrator has always produced. Nothing a sync run produces changes, and graphs +where these operators read a source directly are unaffected. + +Every operator's `async_execute` now requires the keyword argument `input_pipeline_hashes`, +one pipeline hash per input, as the operator node always passes it; a missing argument raises +`TypeError`. The operators that use the hashes (`Join`, `MergeJoin`, `Batch`, `GroupBy`, +`PolarsFilter`, and any operator on the default barrier path) raise `ValueError` when the count +does not match the inputs; the per-row operators and `SemiJoin` take the argument without +using it. Before, these +operators fell back without it to the schema-only hash, and `Join`'s async path to names built +from the input positions (`::0`, `::1`, ...), neither of which the sync path uses. Code that +calls an operator's `async_execute` directly must pass the inputs' pipeline hashes. + +This also fixes ITL-885: a store written by one orchestrator can now be extended by the +other. Before, the function node downstream of such an operator wrote two sets of +system-tag columns into one pipeline table, and the second run failed while storing its +results. + +Migration: stores written by v0.2 are not affected, because v0.3 never reads a table +written by v0.2. Only a store written by an async run of an unreleased v0.3 build, for a +graph with `MergeJoin`, `Batch` or `GroupBy` after an operator or a function pod, holds the +old names: a later run on that store fails when it writes the pipeline table of the function +node downstream. Delete that table, or use a fresh store, and run again. diff --git a/design_issues/ITL-865-O8.md b/design_issues/ITL-865-O8.md new file mode 100644 index 000000000..648dd10c8 --- /dev/null +++ b/design_issues/ITL-865-O8.md @@ -0,0 +1,24 @@ +## `src/orcapod/core/operators/` — Async execution + +### O8 — Async barrier operators raised on an input with no rows +**Status:** resolved (no rows emitted); a typed empty result on async waits for ITL-858 +**Severity:** high +**Issue:** ITL-865 + +The barrier-mode `async_execute` of `UnaryOperator` and `BinaryOperator` (used by +`PolarsFilter`, `GroupBy` and `MergeJoin`) and the default `StaticOutputOperatorPod.async_execute` +rebuilt each input from its rows with `_materialize_to_stream`, which raises +`ValueError: Cannot materialize an empty list of rows into a stream` when a channel delivers +none. The sync path returns an empty stream, so a pipeline with a filter that matches no row +upstream of such an operator completed under the sync orchestrator and failed under the async +one. + +**Fix:** a barrier path whose input (any input, for the binary and default paths) delivers no +rows emits no rows and closes its output. That is the sync result of every barrier operator +that exists: an operator never synthesizes a row, and `MergeJoin` is an inner join. The sync +path's typed empty stream needs a schema the channels do not carry, so on async an operator +node that received no rows still reads back columns typed from its Python output schema +(ITL-858, ITL-912). `tests/test_core/operators/test_async_sync_equivalence.py` runs every +operator with an async path with all inputs and with each one input empty; +`tests/test_pipeline/test_operator_node_async_readback.py` runs sync and async jobs with a +filter matching no row upstream of barrier operators and of a function pod. diff --git a/design_issues/ITL-906-O7.md b/design_issues/ITL-906-O7.md new file mode 100644 index 000000000..956170db4 --- /dev/null +++ b/design_issues/ITL-906-O7.md @@ -0,0 +1,30 @@ +## `src/orcapod/core/operators/` — Async execution + +### O7 — Async system-tag names after a non-source input differed from sync +**Status:** resolved +**Severity:** high +**Issue:** ITL-906 + +The barrier-mode `async_execute` (`UnaryOperator`, `BinaryOperator`, the default +`StaticOutputOperatorPod.async_execute`, and `Batch`) rebuilt each input from its rows with +`_materialize_to_stream`, an `ArrowTableStream` without a producer whose pipeline hash is the +schema-only hash of a root. `MergeJoin` (system tag rule 2), `Batch` and `GroupBy` (rule 3) +name system-tag columns after their inputs' pipeline hashes, so when an input was an +operator's or a function pod's output the async names, `MergeJoin`'s canonical input order +and every record id derived from them differed from the sync path. The `input_pipeline_hashes` +keyword the operator node passes was ignored. A function node downstream then wrote two sets +of system-tag columns into one pipeline table, and a run on one orchestrator could not extend +a store written by the other (ITL-885). + +**Fix:** a barrier path rebuilds each input with `_materialize_input_stream(rows, +pipeline_hash)`, a stream that reports the input's pipeline hash (`_RowsOfStream`). +`input_pipeline_hashes` is a required keyword of every operator's `async_execute`, with no +default and no fallback: the barrier paths and `Join` (whose async path had named system tags +after the input positions `0`, `1`, ... when no hashes were given) raise `ValueError` when the +count does not match the inputs, and the per-row operators and `SemiJoin` take it and do not +use it. `_materialize_to_stream(rows)` only holds output rows. Async names and record ids now equal the sync ones; nothing a +sync run produces moves, and no golden value moves (the golden cases run the sync path). +`tests/test_core/operators/test_async_sync_equivalence.py` runs every operator with an async +path over inputs behind an operator and behind a function pod; +`tests/test_pipeline/test_mixed_orchestrators.py` runs one store with both orchestrators in +both orders. diff --git a/docs/concepts/operators.md b/docs/concepts/operators.md index 945ba9ad5..cd89f7fe5 100644 --- a/docs/concepts/operators.md +++ b/docs/concepts/operators.md @@ -562,7 +562,12 @@ silent widening; convert explicitly. the join runs, not when the pipeline is built. - **Sync and async execution agree.** An operator run asynchronously (`async_execute`, as in an async pipeline) yields rows with the same Arrow types, column order, system tags and - content hashes as the same operator run synchronously. + content hashes as the same operator run synchronously, whether its inputs are sources or + the outputs of other operators and function pods. An input that delivers no rows gives no + rows on both paths. One limit remains: an async channel carries rows, not a schema, so + where the sync path returns an empty stream with typed columns the async path emits + nothing, and an operator node that received no rows reads back columns typed from its + Python output schema (ITL-858, ITL-912). ```python import pyarrow as pa diff --git a/src/orcapod/core/operators/base.py b/src/orcapod/core/operators/base.py index 000cf346f..cc9c5e453 100644 --- a/src/orcapod/core/operators/base.py +++ b/src/orcapod/core/operators/base.py @@ -115,12 +115,35 @@ async def async_execute( inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], *, - input_pipeline_hashes: Sequence[ContentHash] | None = None, + input_pipeline_hashes: Sequence[ContentHash], ) -> None: - """Barrier-mode: collect single input, run unary_static_process, emit.""" + """Barrier-mode: collect single input, run unary_static_process, emit. + + An input with no rows emits no rows (ITL-865). + + Args: + inputs: The one readable input channel. + output: Writable channel for downstream consumption. + input_pipeline_hashes: The input stream's pipeline hash, as a + one-element sequence. The rebuilt input reports it, so + system-tag names equal the sync path's (ITL-906). + + Raises: + ValueError: If ``input_pipeline_hashes`` does not have one hash per input. + """ try: + if len(input_pipeline_hashes) != len(inputs): + raise ValueError( + f"input_pipeline_hashes length ({len(input_pipeline_hashes)}) " + f"must match inputs length ({len(inputs)})" + ) rows = await inputs[0].collect() - stream = self._materialize_to_stream(rows) + if not rows: + # No rows in, no rows out: an operator never synthesizes a row. + # The typed empty stream the sync path returns needs a schema the + # channel does not carry (ITL-858), so nothing is emitted (ITL-865). + return + stream = self._materialize_input_stream(rows, input_pipeline_hashes[0]) result = self.static_process(stream) for tag, data in result.iter_data(): await output.send((tag, data)) @@ -199,15 +222,40 @@ async def async_execute( inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], *, - input_pipeline_hashes: Sequence[ContentHash] | None = None, + input_pipeline_hashes: Sequence[ContentHash], ) -> None: - """Barrier-mode: collect both inputs concurrently, run binary_static_process, emit.""" + """Barrier-mode: collect both inputs concurrently, run binary_static_process, emit. + + An input with no rows emits no rows, as an inner join does (ITL-865). + + Args: + inputs: The left and right readable input channels. + output: Writable channel for downstream consumption. + input_pipeline_hashes: The left and right input streams' pipeline + hashes. Each rebuilt input reports its own, so system-tag names + and canonical input order equal the sync path's (ITL-906). + + Raises: + ValueError: If ``input_pipeline_hashes`` does not have one hash per input. + """ try: + if len(input_pipeline_hashes) != len(inputs): + raise ValueError( + f"input_pipeline_hashes length ({len(input_pipeline_hashes)}) " + f"must match inputs length ({len(inputs)})" + ) + left_hash, right_hash = input_pipeline_hashes left_rows, right_rows = await asyncio.gather( inputs[0].collect(), inputs[1].collect() ) - left_stream = self._materialize_to_stream(left_rows) - right_stream = self._materialize_to_stream(right_rows) + if not left_rows or not right_rows: + # The one barrier binary operator, MergeJoin, is an inner join: an + # input with no rows gives no rows, as on the sync path. The typed + # empty stream needs a schema the channels do not carry (ITL-858), + # so nothing is emitted (ITL-865). + return + left_stream = self._materialize_input_stream(left_rows, left_hash) + right_stream = self._materialize_input_stream(right_rows, right_hash) result = self.static_process(left_stream, right_stream) for tag, data in result.iter_data(): await output.send((tag, data)) diff --git a/src/orcapod/core/operators/batch.py b/src/orcapod/core/operators/batch.py index d3419599d..2fc0fe32b 100644 --- a/src/orcapod/core/operators/batch.py +++ b/src/orcapod/core/operators/batch.py @@ -8,7 +8,7 @@ from orcapod.core.streams import ArrowTableStream from orcapod.protocols.core_protocols import DataProtocol, StreamProtocol, TagProtocol from orcapod.system_constants import constants -from orcapod.types import ColumnConfig +from orcapod.types import ColumnConfig, ContentHash from orcapod.utils import arrow_utils from orcapod.utils.lazy_module import LazyModule @@ -152,7 +152,8 @@ async def async_execute( self, inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], - **kwargs: Any, + *, + input_pipeline_hashes: Sequence[ContentHash], ) -> None: """Streaming batch: emit full batches as they accumulate. @@ -160,13 +161,29 @@ async def async_execute( materialized and emitted immediately, allowing downstream consumers to start processing before all input is consumed. When ``batch_size == 0`` (batch everything), falls back to barrier mode. + + Args: + inputs: The one readable input channel. + output: Writable channel for downstream consumption. + input_pipeline_hashes: The input stream's pipeline hash, as a + one-element sequence. Every rebuilt batch reports it, so the + folded system-tag names end in it as on the sync path (ITL-906). + + Raises: + ValueError: If ``input_pipeline_hashes`` does not have one hash per input. """ try: + if len(input_pipeline_hashes) != len(inputs): + raise ValueError( + f"input_pipeline_hashes length ({len(input_pipeline_hashes)}) " + f"must match inputs length ({len(inputs)})" + ) + pipeline_hash = input_pipeline_hashes[0] if self.batch_size == 0: # Must collect all rows — barrier fallback rows = await inputs[0].collect() if rows: - stream = self._materialize_to_stream(rows) + stream = self._materialize_input_stream(rows, pipeline_hash) result = self.unary_static_process(stream) for tag, data in result.iter_data(): await output.send((tag, data)) @@ -176,7 +193,7 @@ async def async_execute( async for tag, data in inputs[0]: batch.append((tag, data)) if len(batch) >= self.batch_size: - stream = self._materialize_to_stream(batch) + stream = self._materialize_input_stream(batch, pipeline_hash) result = self.unary_static_process(stream) for out_tag, out_data in result.iter_data(): await output.send((out_tag, out_data)) @@ -184,7 +201,7 @@ async def async_execute( # Flush partial batch if batch and not self.drop_partial_batch: - stream = self._materialize_to_stream(batch) + stream = self._materialize_input_stream(batch, pipeline_hash) result = self.unary_static_process(stream) for out_tag, out_data in result.iter_data(): await output.send((out_tag, out_data)) diff --git a/src/orcapod/core/operators/column_selection.py b/src/orcapod/core/operators/column_selection.py index e00f440f2..f3ed67c56 100644 --- a/src/orcapod/core/operators/column_selection.py +++ b/src/orcapod/core/operators/column_selection.py @@ -8,7 +8,7 @@ from orcapod.errors import InputValidationError from orcapod.protocols.core_protocols import DataProtocol, StreamProtocol, TagProtocol from orcapod.system_constants import constants -from orcapod.types import ColumnConfig, Schema +from orcapod.types import ColumnConfig, ContentHash, Schema from orcapod.utils.lazy_module import LazyModule if TYPE_CHECKING: @@ -114,9 +114,18 @@ async def async_execute( self, inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], - **kwargs: Any, + *, + input_pipeline_hashes: Sequence[ContentHash], ) -> None: - """Streaming: select tag columns per row without materializing.""" + """Streaming: select tag columns per row without materializing. + + Args: + inputs: The one readable input channel. + output: Writable channel for downstream consumption. + input_pipeline_hashes: Not used: this operator keeps system-tag + names as they are, so no column is named after an input's + pipeline hash. + """ try: tags_to_drop: list[str] | None = None async for tag, data in inputs[0]: @@ -233,9 +242,18 @@ async def async_execute( self, inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], - **kwargs: Any, + *, + input_pipeline_hashes: Sequence[ContentHash], ) -> None: - """Streaming: select data columns per row without materializing.""" + """Streaming: select data columns per row without materializing. + + Args: + inputs: The one readable input channel. + output: Writable channel for downstream consumption. + input_pipeline_hashes: Not used: this operator keeps system-tag + names as they are, so no column is named after an input's + pipeline hash. + """ try: pkts_to_drop: list[str] | None = None async for tag, data in inputs[0]: @@ -347,9 +365,18 @@ async def async_execute( self, inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], - **kwargs: Any, + *, + input_pipeline_hashes: Sequence[ContentHash], ) -> None: - """Streaming: drop tag columns per row without materializing.""" + """Streaming: drop tag columns per row without materializing. + + Args: + inputs: The one readable input channel. + output: Writable channel for downstream consumption. + input_pipeline_hashes: Not used: this operator keeps system-tag + names as they are, so no column is named after an input's + pipeline hash. + """ try: effective_drops: list[str] | None = None async for tag, data in inputs[0]: @@ -468,9 +495,18 @@ async def async_execute( self, inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], - **kwargs: Any, + *, + input_pipeline_hashes: Sequence[ContentHash], ) -> None: - """Streaming: drop data columns per row without materializing.""" + """Streaming: drop data columns per row without materializing. + + Args: + inputs: The one readable input channel. + output: Writable channel for downstream consumption. + input_pipeline_hashes: Not used: this operator keeps system-tag + names as they are, so no column is named after an input's + pipeline hash. + """ try: effective_drops: list[str] | None = None async for tag, data in inputs[0]: diff --git a/src/orcapod/core/operators/drop_nulls.py b/src/orcapod/core/operators/drop_nulls.py index b938653f1..437fbfcff 100644 --- a/src/orcapod/core/operators/drop_nulls.py +++ b/src/orcapod/core/operators/drop_nulls.py @@ -14,7 +14,7 @@ from orcapod.errors import InputValidationError from orcapod.protocols.core_protocols import DataProtocol, StreamProtocol, TagProtocol from orcapod.system_constants import constants -from orcapod.types import ColumnConfig, DataType, Schema +from orcapod.types import ColumnConfig, ContentHash, DataType, Schema from orcapod.utils import arrow_utils from orcapod.utils.lazy_module import LazyModule @@ -313,7 +313,8 @@ async def async_execute( self, inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], - **kwargs: Any, + *, + input_pipeline_hashes: Sequence[ContentHash], ) -> None: """Streaming: test and narrow each row as it arrives, without materializing. @@ -325,7 +326,9 @@ async def async_execute( Args: inputs: Single-element sequence of readable channels. output: Channel to send surviving rows to. - **kwargs: Ignored; present for protocol compliance. + input_pipeline_hashes: Not used: this operator keeps system-tag + names as they are, so no column is named after an input's + pipeline hash. """ try: async for tag, data in inputs[0]: diff --git a/src/orcapod/core/operators/index.py b/src/orcapod/core/operators/index.py index a06ed77fa..23c09b676 100644 --- a/src/orcapod/core/operators/index.py +++ b/src/orcapod/core/operators/index.py @@ -379,7 +379,7 @@ async def async_execute( inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], *, - input_pipeline_hashes: Sequence[ContentHash] | None = None, + input_pipeline_hashes: Sequence[ContentHash], ) -> None: """Process packets one at a time as they arrive (true streaming). @@ -389,7 +389,9 @@ async def async_execute( Args: inputs: Single-element sequence of readable channels. output: Channel to send transformed packets to. - input_pipeline_hashes: Ignored; present for protocol compliance. + input_pipeline_hashes: Not used: this operator keeps system-tag + names as they are, so no column is named after an input's + pipeline hash. """ try: async for tag, data in inputs[0]: diff --git a/src/orcapod/core/operators/join.py b/src/orcapod/core/operators/join.py index 7da6933c8..b6ac78d5b 100644 --- a/src/orcapod/core/operators/join.py +++ b/src/orcapod/core/operators/join.py @@ -343,7 +343,7 @@ async def async_execute( inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], *, - input_pipeline_hashes: Sequence[ContentHash] | None = None, + input_pipeline_hashes: Sequence[ContentHash], ) -> None: """Async streaming join with pairwise iterative semantics. @@ -363,26 +363,25 @@ async def async_execute( inputs: Readable channels, one per upstream. output: Writable channel for downstream. input_pipeline_hashes: Pipeline hash for each input, - positionally matching ``inputs``. Required for - correct system-tag renaming with 2+ inputs. + positionally matching ``inputs``. With 2+ inputs, each input's + system-tag columns are renamed after its hash, as on the sync + path. + + Raises: + ValueError: If ``input_pipeline_hashes`` does not have one hash per input. """ try: + if len(input_pipeline_hashes) != len(inputs): + raise ValueError( + f"input_pipeline_hashes length ({len(input_pipeline_hashes)}) " + f"must match inputs length ({len(inputs)})" + ) if len(inputs) == 1: async for tag, data in inputs[0]: await output.send((tag, data)) return - n = len(inputs) - if input_pipeline_hashes is not None and len(input_pipeline_hashes) != n: - raise ValueError( - f"input_pipeline_hashes length ({len(input_pipeline_hashes)}) " - f"must match inputs length ({n})" - ) - suffixes = ( - self._compute_system_tag_suffixes(input_pipeline_hashes) - if input_pipeline_hashes is not None - else [str(i) for i in range(n)] - ) + suffixes = self._compute_system_tag_suffixes(input_pipeline_hashes) await self._streaming_join(inputs, output, suffixes) finally: await output.close() diff --git a/src/orcapod/core/operators/mappers.py b/src/orcapod/core/operators/mappers.py index 7e16aac27..48db5c234 100644 --- a/src/orcapod/core/operators/mappers.py +++ b/src/orcapod/core/operators/mappers.py @@ -7,7 +7,7 @@ from orcapod.errors import InputValidationError from orcapod.protocols.core_protocols import DataProtocol, StreamProtocol, TagProtocol from orcapod.system_constants import constants -from orcapod.types import ColumnConfig, Schema +from orcapod.types import ColumnConfig, ContentHash, Schema from orcapod.utils.lazy_module import LazyModule if TYPE_CHECKING: @@ -127,9 +127,18 @@ async def async_execute( self, inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], - **kwargs: Any, + *, + input_pipeline_hashes: Sequence[ContentHash], ) -> None: - """Streaming: rename data columns per row without materializing.""" + """Streaming: rename data columns per row without materializing. + + Args: + inputs: The one readable input channel. + output: Writable channel for downstream consumption. + input_pipeline_hashes: Not used: this operator keeps system-tag + names as they are, so no column is named after an input's + pipeline hash. + """ try: rename_map: dict[str, str] | None = None unmapped: list[str] | None = None @@ -267,9 +276,18 @@ async def async_execute( self, inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], - **kwargs: Any, + *, + input_pipeline_hashes: Sequence[ContentHash], ) -> None: - """Streaming: rename tag columns per row without materializing.""" + """Streaming: rename tag columns per row without materializing. + + Args: + inputs: The one readable input channel. + output: Writable channel for downstream consumption. + input_pipeline_hashes: Not used: this operator keeps system-tag + names as they are, so no column is named after an input's + pipeline hash. + """ try: rename_map: dict[str, str] | None = None unmapped: list[str] | None = None diff --git a/src/orcapod/core/operators/pick.py b/src/orcapod/core/operators/pick.py index ef65e4d4c..b69f071dc 100644 --- a/src/orcapod/core/operators/pick.py +++ b/src/orcapod/core/operators/pick.py @@ -393,7 +393,7 @@ async def async_execute( inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], *, - input_pipeline_hashes: Sequence[ContentHash] | None = None, + input_pipeline_hashes: Sequence[ContentHash], ) -> None: """Process packets one at a time as they arrive (true streaming). @@ -403,7 +403,9 @@ async def async_execute( Args: inputs: Single-element sequence of readable channels. output: Channel to send transformed packets to. - input_pipeline_hashes: Ignored; present for protocol compliance. + input_pipeline_hashes: Not used: this operator keeps system-tag + names as they are, so no column is named after an input's + pipeline hash. """ try: async for tag, data in inputs[0]: diff --git a/src/orcapod/core/operators/semijoin.py b/src/orcapod/core/operators/semijoin.py index 1b9a42aca..532363b23 100644 --- a/src/orcapod/core/operators/semijoin.py +++ b/src/orcapod/core/operators/semijoin.py @@ -7,7 +7,7 @@ from orcapod.core.streams import ArrowTableStream from orcapod.errors import InputValidationError from orcapod.protocols.core_protocols import DataProtocol, StreamProtocol, TagProtocol -from orcapod.types import ColumnConfig, Schema +from orcapod.types import ColumnConfig, ContentHash, Schema from orcapod.utils import arrow_utils, schema_utils from orcapod.utils.lazy_module import LazyModule @@ -195,7 +195,8 @@ async def async_execute( self, inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], - **kwargs: Any, + *, + input_pipeline_hashes: Sequence[ContentHash], ) -> None: """Build-probe: collect right input, then stream left through a hash lookup. @@ -211,6 +212,13 @@ async def async_execute( of each input, since channels carry no schema. When either input is empty there is nothing to check, so a key type mismatch the sync path reports gives an empty result here (as in ``Join``'s async path). + + Args: + inputs: The left and right readable input channels. + output: Writable channel for downstream consumption. + input_pipeline_hashes: Not used: the left rows pass through with + their system-tag names, so no column is named after an + input's pipeline hash. """ try: left_ch, right_ch = inputs[0], inputs[1] diff --git a/src/orcapod/core/operators/static_output_pod.py b/src/orcapod/core/operators/static_output_pod.py index 5ad11c5ad..748f41381 100644 --- a/src/orcapod/core/operators/static_output_pod.py +++ b/src/orcapod/core/operators/static_output_pod.py @@ -11,6 +11,7 @@ from orcapod.config import OrcapodConfig from orcapod.contexts import DataContext from orcapod.core.base import TraceableBase +from orcapod.core.streams.arrow_table_stream import ArrowTableStream from orcapod.core.streams.base import StreamBase from orcapod.core.tracker import DEFAULT_TRACKER_MANAGER from orcapod.protocols.core_protocols import ( @@ -28,6 +29,8 @@ if TYPE_CHECKING: import pyarrow as pa + + from orcapod.protocols.hashing_protocols import SemanticHasherProtocol else: pa = LazyModule("pyarrow") @@ -245,9 +248,12 @@ def _materialize_to_stream( ) -> StreamProtocol: """Materialize a list of (Tag, Data) pairs into an ArrowTableStream. - Used by the barrier-mode ``async_execute`` to convert collected - channel items back into a stream suitable for ``static_process``. - Every row keeps its own source info (see ``_rows_to_table``). + Holds an operator's or a node's OUTPUT rows, whose stream identity is + given by whoever holds them (the node, the ``DynamicPodStream``), so the + stream has no producer and its own pipeline hash is never read. Rows + that stand for an operator's INPUT go through + ``_materialize_input_stream``. Every row keeps its own source info (see + ``_rows_to_table``). Args: rows: Non-empty sequence of (Tag, Data) pairs. @@ -258,36 +264,81 @@ def _materialize_to_stream( Raises: ValueError: If ``rows`` is empty. """ - from orcapod.core.streams.arrow_table_stream import ArrowTableStream - table, tag_keys = StaticOutputOperatorPod._rows_to_table(rows) # No ``source_info`` argument: the table's own ``_source_*`` columns # pass through, one value per row. return ArrowTableStream(table, tag_columns=tag_keys) + @staticmethod + def _materialize_input_stream( + rows: Sequence[tuple[TagProtocol, DataProtocol]], + pipeline_hash: ContentHash, + ) -> StreamProtocol: + """Rebuild an input stream from the rows a barrier ``async_execute`` collected. + + The rebuilt stream reports ``pipeline_hash`` as its own, so an operator + that names system-tag columns after its input's pipeline hash + (``MergeJoin``, ``Batch``, ``GroupBy``) names them as the sync path + does (ITL-906). Every row keeps its own source info (see + ``_rows_to_table``). + + Args: + rows: Non-empty sequence of (Tag, Data) pairs. + pipeline_hash: The pipeline hash of the stream the rows came from. + + Returns: + A stream over the rows, in order. + + Raises: + ValueError: If ``rows`` is empty. + """ + table, tag_keys = StaticOutputOperatorPod._rows_to_table(rows) + return _RowsOfStream(table, tag_columns=tag_keys, pipeline_hash=pipeline_hash) + async def async_execute( self, inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], *, - input_pipeline_hashes: Sequence[ContentHash] | None = None, + input_pipeline_hashes: Sequence[ContentHash], ) -> None: """Default barrier-mode async execution. Collects all inputs, runs ``static_process``, emits results. - Subclasses override for streaming or incremental strategies. + Subclasses override for streaming or incremental strategies. When an + input has no rows, nothing is emitted (ITL-865): an operator whose + result is not empty then must override this method. + + Every operator's ``async_execute`` takes ``input_pipeline_hashes``, with + no default: the operator node passes it for every operator alike. Args: inputs: Readable channels, one per upstream node. output: Writable channel for downstream consumption. input_pipeline_hashes: Pipeline hash for each input stream, - positionally matching ``inputs``. Multi-input operators - (e.g. Join) use these to compute canonical system-tag - column names. Ignored by single-input operators. + positionally matching ``inputs``. Each rebuilt input stream + reports its hash, so system-tag column names equal the sync + path's (ITL-906). + + Raises: + ValueError: If ``input_pipeline_hashes`` does not have one hash per input. """ try: + if len(input_pipeline_hashes) != len(inputs): + raise ValueError( + f"input_pipeline_hashes length ({len(input_pipeline_hashes)}) " + f"must match inputs length ({len(inputs)})" + ) all_rows = await asyncio.gather(*(ch.collect() for ch in inputs)) - streams = [self._materialize_to_stream(rows) for rows in all_rows] + if not all(all_rows): + # Every barrier operator that exists gives no rows when an input has + # none. The typed empty stream the sync path returns needs a schema + # the channels do not carry (ITL-858), so nothing is emitted (ITL-865). + return + streams = [ + self._materialize_input_stream(rows, pipeline_hash) + for rows, pipeline_hash in zip(all_rows, input_pipeline_hashes) + ] result = self.static_process(*streams) for tag, data in result.iter_data(): await output.send((tag, data)) @@ -295,6 +346,44 @@ async def async_execute( await output.close() +class _RowsOfStream(ArrowTableStream): + """Rows collected from a channel, standing in for the stream they came from. + + The barrier-mode ``async_execute`` rebuilds each input from its rows. A + rebuilt ``ArrowTableStream`` has no producer, so its own pipeline hash is + the schema-only hash of a root; this stream reports the pipeline hash of + the stream the rows came from instead, as the operator node passes it + (``input_pipeline_hashes``). Its content hash is that of its table, as for + any ``ArrowTableStream`` without a producer. + """ + + def __init__( + self, + table: pa.Table, + *, + tag_columns: Collection[str], + pipeline_hash: ContentHash, + ) -> None: + super().__init__(table, tag_columns=tag_columns) + self._input_pipeline_hash: ContentHash = pipeline_hash + + def pipeline_hash( + self, hasher: SemanticHasherProtocol | None = None + ) -> ContentHash: + """The pipeline hash of the stream the rows came from. + + It was computed by the operator node with the hasher of its data + context, so it is returned whatever ``hasher`` is given. + + Args: + hasher: Not used. + + Returns: + The input pipeline hash the operator node passed. + """ + return self._input_pipeline_hash + + class DynamicPodStream(StreamBase): """Recomputable stream wrapping a ``StaticOutputOperatorPod`` invocation.""" diff --git a/superpowers/plans/2026-11-12-itl-906-async-system-tag-names.md b/superpowers/plans/2026-11-12-itl-906-async-system-tag-names.md new file mode 100644 index 000000000..16a57ab7e --- /dev/null +++ b/superpowers/plans/2026-11-12-itl-906-async-system-tag-names.md @@ -0,0 +1,150 @@ +# ITL-906: async system-tag names after a non-source input — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use sensei:executing-plans to implement this plan +> task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** The async path of `MergeJoin`, `Batch` and `GroupBy` names system-tag columns (and so +derives record ids) exactly as the sync path does when an input is not a root source. + +**Architecture:** The barrier rebuild (`StaticOutputOperatorPod._materialize_to_stream`) returns a +producer-less `ArrowTableStream`, whose pipeline hash is the schema-only root hash. The fix gives +the rebuilt stream the pipeline hash the caller already passes through the existing +`input_pipeline_hashes` keyword of `async_execute` (the operator node passes +`[s.pipeline_hash() for s in self._input_streams]`). No public signature changes. + +**Tech stack:** Python, PyArrow, pytest via `uv run`. + +**Status:** measurement done, ask sent to the coordinator (no code before approval). +Linear: ITL-906 In Progress, project "Orcapod Python: v0.3 release". +Branch: `eywalker/itl-906-mergejoin-async-path-names-system-tag` (from origin/main 25650c42). + +--- + +## Measurements (main 25650c42) + +Scratch scripts (not committed): `measure.py` (operator level, every `CASES` entry of +`test_async_sync_equivalence.py`, over a root source and after a no-op `DropNulls("k")`), +`measure_job.py` (job level, Delta), `itl885.py` (ITL-885's own graph), `proto.py` +(monkeypatched prototype of the fix), `old_store.py` (pre-fix async store re-run post-fix). + +Operator level, after a non-source input (system-tag names / values / tag content hash): + +| Operator | async path | names | tag hash | +|---|---|---|---| +| Join (2, 3, rotated) | own, uses hashes | same | same | +| MergeJoin | `BinaryOperator` barrier | **differ** | same | +| SemiJoin | own (build/probe) | same | same | +| Batch (size 2, size 0) | own, `_materialize_to_stream` per batch | **differ** | same | +| GroupBy | `UnaryOperator` barrier | **differ** | same | +| PolarsFilter | `UnaryOperator` barrier | same (name-preserving) | same | +| MapTags, MapData, Select/Drop Tag/Data, Index, Pick, DropNulls | per row | same | same | + +Job level (Delta store; function node downstream of the operator): sync and async write the +SAME `pdb_v2` path, with different system-tag column names and different `__record_id`s. Over a +root source the two agree. With the prototype fix: identical columns and `__record_id`s. + +ITL-885 item 1 (its own graph `source -> double -> GroupBy -> first`, int control): sync then +async and async then sync FAIL on Delta (non-nullable `_tag_source_id::…` violation) and on the +in-memory store (schema mismatch). With the prototype fix all four orders pass on both stores with +no recomputation on the second run. Item 2 (GroupBy reads back empty after async) no longer +reproduces (ITL-896). + +A store written by a PRE-fix async run of an affected graph, re-run POST-fix (async or sync): +the downstream function node's write FAILS (Delta: non-nullable violation; in-memory: +`ArrowInvalid` schema mismatch). Same failure as ITL-885 today. + +Golden: `tests/test_hashing/golden/cases.py` runs MergeJoin (after `SelectDataColumns`), Batch +and GroupBy through `op(stream)` only (sync). No golden entry covers the async path; none moves. + +## Tasks (after approval) + +### Task 1: failing tests (public interface) + +- `tests/test_core/operators/test_async_sync_equivalence.py`: add a `NON_SOURCE_CASES` matrix: + every `CASES` entry with each input wrapped by a no-op operator (`DropNulls` on a non-nullable + tag) and by a function pod; run through `assert_equivalent` (names, values, hashes). Covers + every operator with `async_execute`. Expected to fail for merge_join, batch_2, batch_all, + group_by only. +- `tests/test_pipeline/...`: un-xfail + `test_merge_join_after_an_operator_reads_back_same_system_tags_after_async_run`; add Batch and + GroupBy after an operator; add a mixed-orchestrator test (sync then async, async then sync, on + in-memory and Delta) for ITL-885's graph, asserting no recomputation on the second run. +- Run, show failures, commit (WIP). + +### Task 2: fix + +- `static_output_pod.py`: `_materialize_to_stream(rows, pipeline_hash=None)` returns a private + `ArrowTableStream` subclass whose `pipeline_hash()` returns the given hash (falls back to the + schema-only hash when `None`). Default `StaticOutputOperatorPod.async_execute` passes + `input_pipeline_hashes[i]`. +- `base.py`: `UnaryOperator.async_execute`, `BinaryOperator.async_execute` pass their hashes. +- `batch.py`: `Batch.async_execute` takes the same keyword the base classes have and passes it. +- Length mismatch between `inputs` and `input_pipeline_hashes` raises `ValueError` (as `Join`). +- Run tests, ruff, basedpyright, commit. + +### Task 3: docs and records + +- `changelog.d/ITL-906.md` under `### Breaking changes and migration`. +- `design_issues/ITL-906-.md`, status resolved with **Fix:**. +- Update test docstrings/xfail reasons; remove the stale "Ignored by single-input operators" + docstring line. + +### Task 4: finish + +- Full `uv run pytest tests/`, ruff check/format, basedpyright exit code. +- Merge origin/main, push, PR "Fixes ITL-906" (+ ITL-885 per coordinator), CodeRabbit rounds. + +## Coordinator decisions (2026-11-12) + +- Fix approved as proposed; `Batch.async_execute`'s `**kwargs` becomes the keyword-only + `input_pipeline_hashes`. +- PR says "Fixes ITL-906" and "Fixes ITL-885" (mixed-orchestrator test: source, operator and + function nodes, in-memory and Delta, both orders). MergeJoin source-info tie finding: out of + scope, minimal repro in worker_done. +- Changelog: only stores written by an async run of an unreleased v0.3 build are at risk. +- ITL-865 approved: same PR, its own commit(s), its own fragment `changelog.d/ITL-865.md`. + +## Progress + +- [x] ITL-906 tests (failing first), fix, changelog, design issue (commit bd450dd5). +- [ ] ITL-865: zero rows on every input / one input for every async operator (equivalence + test), async PipelineJob with a no-match filter upstream of a barrier operator and of a + function pod; fix: barrier path emits nothing when an input has no rows; docs + changelog + (`### Fixed`, `### Known limitations`: no typed empty result on async until ITL-858). + +## ITL-865 measurements (after the ITL-906 fix) + +Operator level, zero rows: MergeJoin (either or both inputs empty), GroupBy, PolarsFilter +raise `ValueError: Cannot materialize an empty list of rows`; Batch, Join, the per-row +operators give 0 rows as sync. SemiJoin with an empty RIGHT input gives the left rows when +the operator object was never validated (`_validated_common_keys` is set by +`validate_inputs`, i.e. by `op(...)` / node construction): operator-state hazard, ITL-858. + +Job level: a sync job whose upstream operator node produced zero rows feeds a binary operator +(Join/MergeJoin/SemiJoin) a widened table (ITL-912: int8 -> int64) and raises a type +disagreement; out of scope (ITL-912). + +## Review round: developer comment 4557365620 ("what's the point of this helper function??") + +Decision (developer "Go with A", coordinator approval 2026-11-12): REQUIRE the hashes and remove +`_pair_input_pipeline_hashes`. One rule for every operator: + +- Every operator's `async_execute` declares `*, input_pipeline_hashes: Sequence[ContentHash]`, + required, no default (replaces `| None = None` and `**kwargs`). Missing -> `TypeError`. +- The five that use it (UnaryOperator, BinaryOperator, StaticOutputOperatorPod default, Batch, + Join) check `len(input_pipeline_hashes) != len(inputs)` inline and raise `ValueError` naming + both numbers. Join's no-hash fallback (positional suffixes "0", "1") goes too. +- Per-row operators (MapTags, MapData, Select/Drop Tag/Data, DropNulls, Index, Pick) and SemiJoin + take it and say it is not used (they preserve names). +- `_materialize_to_stream(rows)` loses its hash parameter (output holders: operator_node ~1150, + ~1488, Index, Pick); barrier paths call a new `_materialize_input_stream(rows, pipeline_hash)` + (required hash) returning `_RowsOfStream`. +- Protocols unchanged (no protocol declares an operator's `async_execute`). +- Changelog ITL-906: add Join's old fallback. PR body: signature uniform and why. + +### Steps +- [x] Failing tests (`tests/test_core/operators/test_async_sync_equivalence.py`): each barrier + operator + Join without hashes raises `TypeError`; wrong length raises `ValueError` naming both. +- [x] Implement; update direct test callers; design issue Fix note; changelog. +- [x] Full suite 3.12, 3.11, 3.13 (10167 passed, 109 skipped, 22 xfailed each); ruff; basedpyright exit 0; merged origin/main (290889e8). +- [ ] Push, PR body, reply on thread 4557365620, CodeRabbit on new head, CI green. diff --git a/tests/test_channels/test_async_execute.py b/tests/test_channels/test_async_execute.py index 82760f258..b00408c06 100644 --- a/tests/test_channels/test_async_execute.py +++ b/tests/test_channels/test_async_execute.py @@ -228,7 +228,11 @@ async def test_select_tag_columns(self): output_ch = Channel(buffer_size=16) await feed_stream_to_channel(stream, input_ch) - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[stream.pipeline_hash()], + ) results = await output_ch.reader.collect() assert len(results) == 3 @@ -244,7 +248,11 @@ async def test_select_data_columns(self): output_ch = Channel(buffer_size=16) await feed_stream_to_channel(stream, input_ch) - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[stream.pipeline_hash()], + ) results = await output_ch.reader.collect() assert len(results) == 3 @@ -262,7 +270,11 @@ async def test_drop_data_columns(self): output_ch = Channel(buffer_size=16) await feed_stream_to_channel(stream, input_ch) - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[stream.pipeline_hash()], + ) results = await output_ch.reader.collect() assert len(results) == 3 @@ -288,7 +300,11 @@ async def test_drop_tag_columns(self): output_ch = Channel(buffer_size=16) await feed_stream_to_channel(stream, input_ch) - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[stream.pipeline_hash()], + ) results = await output_ch.reader.collect() assert len(results) == 2 @@ -306,7 +322,11 @@ async def test_map_tags(self): output_ch = Channel(buffer_size=16) await feed_stream_to_channel(stream, input_ch) - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[stream.pipeline_hash()], + ) results = await output_ch.reader.collect() assert len(results) == 3 @@ -323,7 +343,11 @@ async def test_map_data(self): output_ch = Channel(buffer_size=16) await feed_stream_to_channel(stream, input_ch) - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[stream.pipeline_hash()], + ) results = await output_ch.reader.collect() assert len(results) == 3 @@ -341,7 +365,11 @@ async def test_polars_filter(self): output_ch = Channel(buffer_size=16) await feed_stream_to_channel(stream, input_ch) - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[stream.pipeline_hash()], + ) results = await output_ch.reader.collect() assert len(results) == 1 @@ -358,7 +386,11 @@ async def test_batch_operator(self): output_ch = Channel(buffer_size=16) await feed_stream_to_channel(stream, input_ch) - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[stream.pipeline_hash()], + ) results = await output_ch.reader.collect() assert len(results) == 3 # 6 rows / batch_size=2 @@ -390,7 +422,11 @@ async def test_semi_join(self): await feed_stream_to_channel(left, left_ch) await feed_stream_to_channel(right, right_ch) - await op.async_execute([left_ch.reader, right_ch.reader], output_ch.writer) + await op.async_execute( + [left_ch.reader, right_ch.reader], + output_ch.writer, + input_pipeline_hashes=[left.pipeline_hash(), right.pipeline_hash()], + ) results = await output_ch.reader.collect() ids = sorted(tag.as_dict()["id"] for tag, _ in results) @@ -422,7 +458,11 @@ async def test_merge_join(self): await feed_stream_to_channel(left, left_ch) await feed_stream_to_channel(right, right_ch) - await op.async_execute([left_ch.reader, right_ch.reader], output_ch.writer) + await op.async_execute( + [left_ch.reader, right_ch.reader], + output_ch.writer, + input_pipeline_hashes=[left.pipeline_hash(), right.pipeline_hash()], + ) results = await output_ch.reader.collect() assert len(results) == 2 @@ -460,7 +500,11 @@ async def test_two_way_join(self): await feed_stream_to_channel(left, left_ch) await feed_stream_to_channel(right, right_ch) - await op.async_execute([left_ch.reader, right_ch.reader], output_ch.writer) + await op.async_execute( + [left_ch.reader, right_ch.reader], + output_ch.writer, + input_pipeline_hashes=[left.pipeline_hash(), right.pipeline_hash()], + ) results = await output_ch.reader.collect() assert len(results) == 3 @@ -674,7 +718,13 @@ async def source(): async with asyncio.TaskGroup() as tg: tg.create_task(source()) - tg.create_task(filter_op.async_execute([ch1.reader], ch2.writer)) + tg.create_task( + filter_op.async_execute( + [ch1.reader], + ch2.writer, + input_pipeline_hashes=[stream.pipeline_hash()], + ) + ) tg.create_task(func_pod.async_execute([ch2.reader], ch3.writer)) results = await ch3.reader.collect() @@ -727,7 +777,12 @@ async def push(stream, ch): tg.create_task(push(right_stream, ch_right)) tg.create_task( join_op.async_execute( - [ch_left.reader, ch_right.reader], ch_joined.writer + [ch_left.reader, ch_right.reader], + ch_joined.writer, + input_pipeline_hashes=[ + left_stream.pipeline_hash(), + right_stream.pipeline_hash(), + ], ) ) tg.create_task(func_pod.async_execute([ch_joined.reader], ch_out.writer)) diff --git a/tests/test_channels/test_native_async_operators.py b/tests/test_channels/test_native_async_operators.py index b610f2b2d..68d681681 100644 --- a/tests/test_channels/test_native_async_operators.py +++ b/tests/test_channels/test_native_async_operators.py @@ -40,7 +40,9 @@ SemiJoin, ) from orcapod.core.streams.arrow_table_stream import ArrowTableStream +from orcapod.protocols.core_protocols import StreamProtocol from orcapod.system_constants import constants +from orcapod.types import ContentHash # --------------------------------------------------------------------------- @@ -155,6 +157,21 @@ def make_disjoint_stream() -> ArrowTableStream: return ArrowTableStream(table, tag_columns=["animal"]) +def hashes(*streams: StreamProtocol) -> list[ContentHash]: + """The pipeline hash of each input stream, as the operator node passes them.""" + return [stream.pipeline_hash() for stream in streams] + + +def root_hash(tag: str, data: str) -> ContentHash: + """The pipeline hash of a root stream with one int64 tag and one int64 data column. + + For an input channel fed by hand, or one that delivers no rows: the operator + node passes every input's pipeline hash, so the tests do too. + """ + table = pa.table({tag: pa.array([0], pa.int64()), data: pa.array([0], pa.int64())}) + return ArrowTableStream(table, tag_columns=[tag]).pipeline_hash() + + async def feed(stream: ArrowTableStream, ch: Channel) -> None: """Push all (tag, data) from a stream into a channel, then close.""" for tag, data in stream.iter_data(): @@ -167,7 +184,9 @@ async def run_unary(op, stream: ArrowTableStream) -> list[tuple]: input_ch = Channel(buffer_size=1024) output_ch = Channel(buffer_size=1024) await feed(stream, input_ch) - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], output_ch.writer, input_pipeline_hashes=hashes(stream) + ) return await output_ch.reader.collect() @@ -180,7 +199,11 @@ async def run_binary( output_ch = Channel(buffer_size=1024) await feed(left, left_ch) await feed(right, right_ch) - await op.async_execute([left_ch.reader, right_ch.reader], output_ch.writer) + await op.async_execute( + [left_ch.reader, right_ch.reader], + output_ch.writer, + input_pipeline_hashes=hashes(left, right), + ) return await output_ch.reader.collect() @@ -237,7 +260,11 @@ async def test_empty_input(self): output_ch = Channel(buffer_size=4) await input_ch.writer.close() op = SelectTagColumns(columns=["region"]) - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[root_hash("id", "value")], + ) results = await output_ch.reader.collect() assert results == [] @@ -326,7 +353,11 @@ async def test_empty_input(self): output_ch = Channel(buffer_size=4) await input_ch.writer.close() op = SelectDataColumns(columns=["weight"]) - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[root_hash("id", "value")], + ) results = await output_ch.reader.collect() assert results == [] @@ -411,7 +442,11 @@ async def test_empty_input(self): output_ch = Channel(buffer_size=4) await input_ch.writer.close() op = DropTagColumns(columns=["region"]) - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[root_hash("id", "value")], + ) results = await output_ch.reader.collect() assert results == [] @@ -473,7 +508,11 @@ async def test_empty_input(self): output_ch = Channel(buffer_size=4) await input_ch.writer.close() op = DropDataColumns(columns=["legs"]) - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[root_hash("id", "value")], + ) results = await output_ch.reader.collect() assert results == [] @@ -569,7 +608,11 @@ async def test_empty_input(self): output_ch = Channel(buffer_size=4) await input_ch.writer.close() op = MapTags(name_map={"region": "area"}) - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[root_hash("id", "value")], + ) results = await output_ch.reader.collect() assert results == [] @@ -682,7 +725,11 @@ async def test_empty_input(self): output_ch = Channel(buffer_size=4) await input_ch.writer.close() op = MapData(name_map={"weight": "mass"}) - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[root_hash("id", "value")], + ) results = await output_ch.reader.collect() assert results == [] @@ -773,7 +820,11 @@ async def test_empty_input(self): output_ch = Channel(buffer_size=4) await input_ch.writer.close() op = Batch(batch_size=2) - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[root_hash("id", "value")], + ) results = await output_ch.reader.collect() assert results == [] @@ -919,7 +970,11 @@ async def test_empty_left_returns_empty(self): await feed(right, right_ch) op = SemiJoin() - await op.async_execute([left_ch.reader, right_ch.reader], output_ch.writer) + await op.async_execute( + [left_ch.reader, right_ch.reader], + output_ch.writer, + input_pipeline_hashes=[root_hash("id", "value"), right.pipeline_hash()], + ) results = await output_ch.reader.collect() assert results == [] @@ -937,7 +992,11 @@ async def test_empty_right_returns_empty_or_all(self): await right_ch.writer.close() op = SemiJoin() - await op.async_execute([left_ch.reader, right_ch.reader], output_ch.writer) + await op.async_execute( + [left_ch.reader, right_ch.reader], + output_ch.writer, + input_pipeline_hashes=[left.pipeline_hash(), root_hash("id", "value")], + ) results = await output_ch.reader.collect() # With empty right and no schema information available, # the implementation falls back to passing left through @@ -998,7 +1057,9 @@ async def test_single_input_passthrough(self): input_ch = Channel(buffer_size=64) output_ch = Channel(buffer_size=64) await feed(stream, input_ch) - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], output_ch.writer, input_pipeline_hashes=hashes(stream) + ) results = await output_ch.reader.collect() assert len(results) == 3 @@ -1077,7 +1138,11 @@ async def test_three_way_join(self): await feed(s1, ch1) await feed(s2, ch2) await feed(s3, ch3) - await op.async_execute([ch1.reader, ch2.reader, ch3.reader], out.writer) + await op.async_execute( + [ch1.reader, ch2.reader, ch3.reader], + out.writer, + input_pipeline_hashes=hashes(s1, s2, s3), + ) results = await out.reader.collect() assert len(results) == 2 @@ -1114,7 +1179,11 @@ async def test_empty_input_single(self): output_ch = Channel(buffer_size=4) await input_ch.writer.close() op = Join() - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[root_hash("id", "value")], + ) results = await output_ch.reader.collect() assert results == [] @@ -1171,7 +1240,11 @@ async def test_matches_sync_three_way(self): await feed(s1, ch1) await feed(s2, ch2) await feed(s3, ch3) - await op.async_execute([ch1.reader, ch2.reader, ch3.reader], out.writer) + await op.async_execute( + [ch1.reader, ch2.reader, ch3.reader], + out.writer, + input_pipeline_hashes=hashes(s1, s2, s3), + ) async_results = await out.reader.collect() assert len(async_results) == len(sync_results) @@ -1197,7 +1270,15 @@ async def test_three_way_streams_before_all_closed(self): # Start the join in the background join_task = asyncio.create_task( - op.async_execute([ch1.reader, ch2.reader, ch3.reader], out.writer) + op.async_execute( + [ch1.reader, ch2.reader, ch3.reader], + out.writer, + input_pipeline_hashes=[ + root_hash("id", "a"), + root_hash("id", "b"), + root_hash("id", "c"), + ], + ) ) # Send one matching row from each side @@ -1235,7 +1316,11 @@ async def test_four_way_join(self): out = Channel(buffer_size=64) for s, ch in zip(streams, channels): await feed(s, ch) - await op.async_execute([ch.reader for ch in channels], out.writer) + await op.async_execute( + [ch.reader for ch in channels], + out.writer, + input_pipeline_hashes=hashes(*streams), + ) results = await out.reader.collect() assert len(results) == 2 @@ -1255,7 +1340,15 @@ async def test_three_way_partial_match_no_premature_emit(self): from orcapod.core.datagrams import Data, Tag join_task = asyncio.create_task( - op.async_execute([ch1.reader, ch2.reader, ch3.reader], out.writer) + op.async_execute( + [ch1.reader, ch2.reader, ch3.reader], + out.writer, + input_pipeline_hashes=[ + root_hash("id", "a"), + root_hash("id", "b"), + root_hash("id", "c"), + ], + ) ) # Send matching rows from only 2 of 3 sides @@ -1299,7 +1392,11 @@ async def test_three_way_empty_side_produces_nothing(self): await feed(s1, ch1) await feed(s2, ch2) await ch3.writer.close() # empty third input - await op.async_execute([ch1.reader, ch2.reader, ch3.reader], out.writer) + await op.async_execute( + [ch1.reader, ch2.reader, ch3.reader], + out.writer, + input_pipeline_hashes=[*hashes(s1, s2), root_hash("id", "c")], + ) results = await out.reader.collect() assert results == [] @@ -1333,7 +1430,11 @@ async def test_three_way_no_shared_tags_cartesian(self): await feed(s1, ch1) await feed(s2, ch2) await feed(s3, ch3) - await op.async_execute([ch1.reader, ch2.reader, ch3.reader], out_ch.writer) + await op.async_execute( + [ch1.reader, ch2.reader, ch3.reader], + out_ch.writer, + input_pipeline_hashes=hashes(s1, s2, s3), + ) results = await out_ch.reader.collect() # 2 × 1 × 2 = 4 cartesian product @@ -1372,7 +1473,9 @@ async def test_duplicate_tag_keys_cross_product(self): out = Channel(buffer_size=64) await feed(s1, ch1) await feed(s2, ch2) - await op.async_execute([ch1.reader, ch2.reader], out.writer) + await op.async_execute( + [ch1.reader, ch2.reader], out.writer, input_pipeline_hashes=hashes(s1, s2) + ) async_results = await out.reader.collect() assert len(async_results) == 6 @@ -1424,7 +1527,11 @@ async def test_three_way_duplicate_keys_cross_product(self): await feed(s1, ch1) await feed(s2, ch2) await feed(s3, ch3) - await op.async_execute([ch1.reader, ch2.reader, ch3.reader], out.writer) + await op.async_execute( + [ch1.reader, ch2.reader, ch3.reader], + out.writer, + input_pipeline_hashes=hashes(s1, s2, s3), + ) async_results = await out.reader.collect() assert len(async_results) == 4 @@ -1537,7 +1644,11 @@ async def test_buffered_rows_both_sides_emit_on_reindex(self): from orcapod.core.datagrams import Data, Tag join_task = asyncio.create_task( - op.async_execute([ch1.reader, ch2.reader], out.writer) + op.async_execute( + [ch1.reader, ch2.reader], + out.writer, + input_pipeline_hashes=[root_hash("id", "a"), root_hash("id", "b")], + ) ) # Buffer 2 rows on side 0 before side 1 sends anything @@ -1588,8 +1699,18 @@ async def source(): async with asyncio.TaskGroup() as tg: tg.create_task(source()) - tg.create_task(select_op.async_execute([ch1.reader], ch2.writer)) - tg.create_task(map_op.async_execute([ch2.reader], ch3.writer)) + tg.create_task( + select_op.async_execute( + [ch1.reader], ch2.writer, input_pipeline_hashes=hashes(stream) + ) + ) + tg.create_task( + map_op.async_execute( + [ch2.reader], + ch3.writer, + input_pipeline_hashes=hashes(select_op(stream)), + ) + ) results = await ch3.reader.collect() assert len(results) == 3 @@ -1633,9 +1754,19 @@ async def push(stream, ch): tg.create_task(push(left, ch_l)) tg.create_task(push(right, ch_r)) tg.create_task( - join_op.async_execute([ch_l.reader, ch_r.reader], ch_joined.writer) + join_op.async_execute( + [ch_l.reader, ch_r.reader], + ch_joined.writer, + input_pipeline_hashes=hashes(left, right), + ) + ) + tg.create_task( + select_op.async_execute( + [ch_joined.reader], + ch_out.writer, + input_pipeline_hashes=hashes(join_op(left, right)), + ) ) - tg.create_task(select_op.async_execute([ch_joined.reader], ch_out.writer)) results = await ch_out.reader.collect() assert len(results) == 3 @@ -1666,9 +1797,19 @@ async def push(stream, ch): tg.create_task(push(left, ch_l)) tg.create_task(push(right, ch_r)) tg.create_task( - semi_op.async_execute([ch_l.reader, ch_r.reader], ch_semi.writer) + semi_op.async_execute( + [ch_l.reader, ch_r.reader], + ch_semi.writer, + input_pipeline_hashes=hashes(left, right), + ) + ) + tg.create_task( + batch_op.async_execute( + [ch_semi.reader], + ch_out.writer, + input_pipeline_hashes=hashes(semi_op(left, right)), + ) ) - tg.create_task(batch_op.async_execute([ch_semi.reader], ch_out.writer)) results = await ch_out.reader.collect() # SemiJoin produces 2 rows (id=[2,3]), Batch(2) → 1 batch @@ -1699,9 +1840,25 @@ async def source(): async with asyncio.TaskGroup() as tg: tg.create_task(source()) - tg.create_task(drop_op.async_execute([ch1.reader], ch2.writer)) - tg.create_task(map_op.async_execute([ch2.reader], ch3.writer)) - tg.create_task(select_op.async_execute([ch3.reader], ch4.writer)) + tg.create_task( + drop_op.async_execute( + [ch1.reader], ch2.writer, input_pipeline_hashes=hashes(stream) + ) + ) + tg.create_task( + map_op.async_execute( + [ch2.reader], + ch3.writer, + input_pipeline_hashes=hashes(drop_op(stream)), + ) + ) + tg.create_task( + select_op.async_execute( + [ch3.reader], + ch4.writer, + input_pipeline_hashes=hashes(map_op(drop_op(stream))), + ) + ) results = await ch4.reader.collect() assert len(results) == 3 diff --git a/tests/test_channels/test_pipeline_example.py b/tests/test_channels/test_pipeline_example.py index 129ae3e25..e4b1bc2e0 100644 --- a/tests/test_channels/test_pipeline_example.py +++ b/tests/test_channels/test_pipeline_example.py @@ -196,6 +196,10 @@ async def push_source(stream: ArrowTableStream, ch: Channel): join_op.async_execute( [ch_students.reader, ch_grades.reader], ch_joined.writer, + input_pipeline_hashes=[ + make_students().pipeline_hash(), + make_grades().pipeline_hash(), + ], ) ) @@ -204,6 +208,9 @@ async def push_source(stream: ArrowTableStream, ch: Channel): filter_op.async_execute( [ch_joined.reader], ch_filtered.writer, + input_pipeline_hashes=[ + join_op(make_students(), make_grades()).pipeline_hash() + ], ) ) @@ -253,12 +260,19 @@ async def push_source(stream, ch): join_op.async_execute( [ch_students.reader, ch_grades.reader], ch_joined.writer, + input_pipeline_hashes=[ + make_students().pipeline_hash(), + make_grades().pipeline_hash(), + ], ) ) tg.create_task( filter_op.async_execute( [ch_joined.reader], ch_filtered.writer, + input_pipeline_hashes=[ + join_op(make_students(), make_grades()).pipeline_hash() + ], ) ) tg.create_task( @@ -322,9 +336,24 @@ async def push(stream, ch): tg.create_task(push(make_students(), ch_s)) tg.create_task(push(make_grades(), ch_g)) tg.create_task( - join_op.async_execute([ch_s.reader, ch_g.reader], ch_j.writer) + join_op.async_execute( + [ch_s.reader, ch_g.reader], + ch_j.writer, + input_pipeline_hashes=[ + make_students().pipeline_hash(), + make_grades().pipeline_hash(), + ], + ) + ) + tg.create_task( + filter_op.async_execute( + [ch_j.reader], + ch_f.writer, + input_pipeline_hashes=[ + join_op(make_students(), make_grades()).pipeline_hash() + ], + ) ) - tg.create_task(filter_op.async_execute([ch_j.reader], ch_f.writer)) tg.create_task(grade_pod.async_execute([ch_f.reader], ch_o.writer)) return { diff --git a/tests/test_core/operators/test_async_sync_equivalence.py b/tests/test_core/operators/test_async_sync_equivalence.py index 49afdbb0b..ed95210d5 100644 --- a/tests/test_core/operators/test_async_sync_equivalence.py +++ b/tests/test_core/operators/test_async_sync_equivalence.py @@ -335,6 +335,263 @@ def test_async_rows_equal_sync_rows(case: Case) -> None: ) +# ── inputs that are not root sources (ITL-906) ────────────────────────────── + + +def after_operator(source: StreamProtocol) -> StreamProtocol: + """``source`` behind an operator that keeps every row: ``DropNulls`` on tag ``k``. + + Every source of the matrix has the non-nullable tag ``k``, so the rows and columns + are those of ``source``; only the pipeline hash differs, as it does after any + operator or function pod. + """ + stream = DropNulls("k")(source) + assert stream.pipeline_hash() != source.pipeline_hash() + return stream + + +def _after_operator_case(case: Case) -> Case: + """``case`` with every input behind ``after_operator``.""" + + def make_sources() -> tuple[StreamProtocol, ...]: + return tuple(after_operator(s) for s in case.make_sources()) + + return Case(case.label, case.make_op, make_sources) + + +def _first_after_operator_case(case: Case) -> Case: + """``case`` with only its first input behind ``after_operator``.""" + + def make_sources() -> tuple[StreamProtocol, ...]: + first, *rest = case.make_sources() + return (after_operator(first), *rest) + + return Case(case.label, case.make_op, make_sources) + + +AFTER_OPERATOR_CASES = [ + pytest.param(_after_operator_case(c), id=f"{c.label}-all") for c in CASES +] + [ + pytest.param(_first_after_operator_case(c), id=f"{c.label}-first") + for c in CASES + if len(c.make_sources()) > 1 +] + + +@pytest.mark.parametrize("case", AFTER_OPERATOR_CASES) +def test_async_rows_equal_sync_rows_after_an_operator(case: Case) -> None: + """Every operator's async path equals its sync path when an input is not a source. + + System-tag column names of ``MergeJoin`` (name-extending), ``Batch`` and ``GroupBy`` + (reducing) end in the input's pipeline hash; the async path must use the hash of the + stream the rows came from, not that of the rows rebuilt into a table. + """ + sources = case.make_sources() + assert_equivalent( + sync_rows(case.make_op(), *sources), async_rows(case.make_op(), *sources) + ) + + +def function_pod_stream(source_id: str, offset: int) -> StreamProtocol: + """A function pod's output on tag ``k: int16``: ``x = a + offset`` as ``int``.""" + source = _source( + {"k": pa.array([1, 2, 3], pa.int16()), "a": pa.array([1, 2, 3], pa.int8())}, + tags=("k",), + source_id=source_id, + ) + + def shift(a: int) -> int: + return a + offset + + pod = FunctionPod( + PythonDataFunction(shift, output_keys="x", function_name=f"shift_{source_id}") # pyright: ignore[reportArgumentType] # PythonDataFunction is not yet typed as matching DataFunctionProtocol (baselined in other tests); it works at runtime + ) + return pod(source) + + +FUNCTION_POD_CASES = [ + Case( + "merge_join", + MergeJoin, + lambda: (function_pod_stream("p", 0), function_pod_stream("q", 10)), + ), + Case( + "merge_join_with_source", + MergeJoin, + lambda: ( + function_pod_stream("p", 0), + _source( + { + "k": pa.array([1, 2, 3], pa.int16()), + "x": pa.array([7, 8, 9], pa.int64()), + }, + tags=("k",), + source_id="r", + ), + ), + ), + Case("join", Join, lambda: (function_pod_stream("p", 0), source_b())), + Case("semijoin", SemiJoin, lambda: (function_pod_stream("p", 0), source_b())), + Case( + "batch_2", lambda: Batch(batch_size=2), lambda: (function_pod_stream("p", 0),) + ), + Case("batch_all", Batch, lambda: (function_pod_stream("p", 0),)), + Case("group_by", lambda: GroupBy(["k"]), lambda: (function_pod_stream("p", 0),)), + Case( + "polars_filter", + lambda: PolarsFilter(constraints={"k": 1}), + lambda: (function_pod_stream("p", 0),), + ), +] + + +@pytest.mark.parametrize( + "case", FUNCTION_POD_CASES, ids=[c.label for c in FUNCTION_POD_CASES] +) +def test_async_rows_equal_sync_rows_after_a_function_pod(case: Case) -> None: + """The async path equals the sync path when an input is a function pod's output.""" + sources = case.make_sources() + assert_equivalent( + sync_rows(case.make_op(), *sources), async_rows(case.make_op(), *sources) + ) + + +@pytest.mark.parametrize( + "make_op", + [lambda: Batch(batch_size=2), Batch, lambda: GroupBy(["ta"])], + ids=["batch_2", "batch_all", "group_by"], +) +def test_reducing_operator_names_system_tags_after_its_input( + make_op: Callable[[], StaticOutputOperatorPod], +) -> None: + """A folded system-tag column name ends in the input's pipeline hash on both paths. + + System tag rule 3: ``Batch`` and ``GroupBy`` append ``::{pipeline_hash}`` of their + input. The input here is an operator's output, whose pipeline hash is not the + schema-only hash of its rows. + """ + op = make_op() + stream = after_operator(source_a()) + suffix = stream.pipeline_hash().to_hex(op.orcapod_config.hashing.system_tag_n_char) + for rows in (sync_rows(make_op(), stream), async_rows(make_op(), stream)): + for tag, _ in rows: + names = list(tag.system_tags()) + assert names + assert all(name.endswith(f"::{suffix}") for name in names), names + + +@CASE +def test_async_execute_requires_input_pipeline_hashes(case: Case) -> None: + """Every operator's ``async_execute`` takes the input pipeline hashes, always. + + Without them a barrier path would name system tags after a schema-only hash, + unlike the sync path (ITL-906), so there is no default to fall back on. + """ + channels: list[Channel[Row]] = [Channel() for _ in case.make_sources()] + output: Channel[Row] = Channel() + with pytest.raises(TypeError, match="input_pipeline_hashes"): + case.make_op().async_execute( # pyright: ignore[reportCallIssue] # the missing keyword is the point + [channel.reader for channel in channels], output.writer + ) + + +@pytest.mark.parametrize( + "case", + [ + c + for c in CASES + if c.label + in ( + "join_2", + "join_3", + "merge_join", + "batch_2", + "batch_all", + "group_by", + "polars_filter", + ) + ], + ids=lambda c: c.label, +) +def test_barrier_async_rejects_wrong_number_of_pipeline_hashes(case: Case) -> None: + """An ``async_execute`` that uses the hashes needs one per input channel.""" + sources = case.make_sources() + n_inputs = len(sources) + + async def run() -> None: + channels: list[Channel[Row]] = [Channel(buffer_size=64) for _ in sources] + for stream, channel in zip(sources, channels): + for row in stream.iter_data(): + await channel.writer.send(row) + await channel.writer.close() + output: Channel[Row] = Channel(buffer_size=64) + await case.make_op().async_execute( + [channel.reader for channel in channels], + output.writer, + input_pipeline_hashes=[s.pipeline_hash() for s in sources] * 2, + ) + + message = rf"length \({2 * n_inputs}\) must match inputs length \({n_inputs}\)" + with pytest.raises(ValueError, match=message): + asyncio.run(run()) + + +# ── inputs that deliver no rows (ITL-865) ─────────────────────────────────── + + +def no_rows(source: StreamProtocol) -> StreamProtocol: + """``source`` behind a filter that matches no row (every source has ``k`` in 1..3).""" + stream = PolarsFilter(constraints={"k": 0})(source) + assert list(stream.iter_data()) == [] + return stream + + +def _zero_row_cases() -> list[Any]: + """Every case with all inputs empty, and each multi-input case with one input empty.""" + params: list[Any] = [] + for case in CASES: + n_inputs = len(case.make_sources()) + empty_sets = [tuple(range(n_inputs))] + if n_inputs > 1: + empty_sets += [(i,) for i in range(n_inputs)] + for empty in empty_sets: + + def make_sources( + case: Case = case, empty: tuple[int, ...] = empty + ) -> tuple[StreamProtocol, ...]: + return tuple( + no_rows(s) if i in empty else s + for i, s in enumerate(case.make_sources()) + ) + + suffix = "all" if len(empty) == n_inputs else f"input_{empty[0]}" + params.append( + pytest.param( + Case(case.label, case.make_op, make_sources), + id=f"{case.label}-empty_{suffix}", + ) + ) + return params + + +@pytest.mark.parametrize("case", _zero_row_cases()) +def test_async_rows_equal_sync_rows_when_an_input_has_no_rows(case: Case) -> None: + """An input with no rows gives the sync path's rows on the async path, never an error. + + The operator is applied to its inputs first, ``op(...)``, as an operator node does when + the graph is built: ``SemiJoin`` with an empty right input reads the common keys that + application recorded, because a channel with no rows says nothing about its columns + (ITL-858). + """ + sources = case.make_sources() + expected = sync_rows(case.make_op(), *sources) + op = case.make_op() + _ = op(*sources) + actual = async_rows(op, *sources) + assert len(actual) == len(expected) + assert_equivalent(expected, actual) + + def test_drop_nulls_without_tag_columns() -> None: """A stream without tag columns: the async path drops the same rows, in order. @@ -523,8 +780,7 @@ def make_sources() -> tuple[StreamProtocol, ...]: # Join-style fix. Strict, so a fix flags them. _MATERIALIZE_DROPS_META = ( "ITL-846: barrier-mode async_execute rebuilds its input with " - "_materialize_to_stream, which drops Data meta columns; the rebuilt stream's " - "pipeline hash then differs, so system-tag names and canonical order differ too" + "_materialize_to_stream, which drops Data meta columns" ) _SYNC_DROPS_META = ( "ITL-846: the sync path drops Data meta columns that the async path keeps (a raw " diff --git a/tests/test_core/operators/test_chained_aggregation.py b/tests/test_core/operators/test_chained_aggregation.py index 2b8d12c4c..544c511e4 100644 --- a/tests/test_core/operators/test_chained_aggregation.py +++ b/tests/test_core/operators/test_chained_aggregation.py @@ -211,7 +211,11 @@ async def run() -> Rows: for row in upstream.iter_data(): await source.writer.send(row) await source.writer.close() - await op.async_execute([source.reader], sink.writer) + await op.async_execute( + [source.reader], + sink.writer, + input_pipeline_hashes=[upstream.pipeline_hash()], + ) return await sink.reader.collect() return asyncio.run(run()) diff --git a/tests/test_core/operators/test_index_pick_arrow_types.py b/tests/test_core/operators/test_index_pick_arrow_types.py index 82579b8b6..78cd3ab7c 100644 --- a/tests/test_core/operators/test_index_pick_arrow_types.py +++ b/tests/test_core/operators/test_index_pick_arrow_types.py @@ -471,7 +471,11 @@ async def _run_async(op: UnaryOperator, source: StreamProtocol) -> list[Row]: for tag, data in source.iter_data(): await input_ch.writer.send((tag, data)) await input_ch.writer.close() - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[source.pipeline_hash()], + ) return await output_ch.reader.collect() @@ -694,7 +698,11 @@ async def test_row_time_code_never_reads_a_context( for row in source.iter_data(): await input_ch.writer.send(row) await input_ch.writer.close() - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[source.pipeline_hash()], + ) async_rows = await output_ch.reader.collect() assert len(sync_rows) == len(async_rows) == 2 diff --git a/tests/test_core/operators/test_materialize_source_info.py b/tests/test_core/operators/test_materialize_source_info.py index b26a742a2..59a7c9f8b 100644 --- a/tests/test_core/operators/test_materialize_source_info.py +++ b/tests/test_core/operators/test_materialize_source_info.py @@ -391,7 +391,12 @@ async def go() -> list[Row]: empty: Channel[Row] = Channel(buffer_size=4) await empty.writer.close() output: Channel[Row] = Channel(buffer_size=4) - await make_op().async_execute([empty.reader], output.writer) + # An input that delivers no rows still has a pipeline hash. + await make_op().async_execute( + [empty.reader], + output.writer, + input_pipeline_hashes=[_source().pipeline_hash()], + ) return await output.reader.collect() # The generic barrier path (e.g. ``PolarsFilter``) still raises on zero rows, diff --git a/tests/test_core/test_regression_fixes.py b/tests/test_core/test_regression_fixes.py index 920c9173b..57680adc8 100644 --- a/tests/test_core/test_regression_fixes.py +++ b/tests/test_core/test_regression_fixes.py @@ -146,7 +146,11 @@ async def test_operator_closes_channel_on_static_process_error(self): await feed_stream_to_channel(stream, input_ch) with pytest.raises(Exception): - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[stream.pipeline_hash()], + ) # Channel should be closed — collect should not hang. results = await output_ch.reader.collect() @@ -166,8 +170,13 @@ async def test_static_output_pod_closes_channel_on_empty_input(self): await input_ch.writer.close() # empty input - # Streaming async_execute handles empty input gracefully. - await op.async_execute([input_ch.reader], output_ch.writer) + # Streaming async_execute handles empty input gracefully. An input + # that delivers no rows still has a pipeline hash. + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[make_stream(3).pipeline_hash()], + ) # Channel should be closed and empty. results = await output_ch.reader.collect() diff --git a/tests/test_pipeline/test_mixed_orchestrators.py b/tests/test_pipeline/test_mixed_orchestrators.py new file mode 100644 index 000000000..0f8616c42 --- /dev/null +++ b/tests/test_pipeline/test_mixed_orchestrators.py @@ -0,0 +1,163 @@ +"""One store, two orchestrators: a run on one extends a store written by the other (ITL-885). + +A job run by ``SyncPipelineOrchestrator`` and then by ``AsyncPipelineOrchestrator`` on the same +store, or the other way round, must find every stored result: the second run computes +nothing, and every node (source, operator and function nodes) reads back the rows of the +first run. Before ITL-906 the async path named the system-tag columns of ``GroupBy``, +``Batch`` and ``MergeJoin`` after a non-source input differently from sync, so the function +node downstream wrote a second set of system-tag columns into the same pipeline table and +the second run failed. +""" + +from __future__ import annotations + +from collections.abc import Callable +from pathlib import Path + +import pyarrow as pa +import pytest + +from orcapod import function_pod +from orcapod.core.operators import Batch, GroupBy, MergeJoin, SelectDataColumns +from orcapod.core.sources import ArrowTableSource +from orcapod.databases import DeltaTableDatabase, InMemoryArrowDatabase +from orcapod.pipeline.async_orchestrator import AsyncPipelineOrchestrator +from orcapod.pipeline.job import PipelineJob +from orcapod.pipeline.sync_orchestrator import SyncPipelineOrchestrator +from orcapod.protocols.core_protocols import StreamProtocol +from orcapod.protocols.database_protocols import ArrowDatabaseProtocol + +ORCHESTRATORS = { + "sync": SyncPipelineOrchestrator, + "async": AsyncPipelineOrchestrator, +} + +CALLS: list[str] = [] + + +def _source(name: str, x: list[int]) -> ArrowTableSource: + """Tags ``id: int8`` and ``grp: large_string``; non-nullable ``x: int8``.""" + table = pa.table( + { + "id": pa.array([1, 2, 3, 4], type=pa.int8()), + "grp": pa.array(["g1", "g1", "g2", "g2"], type=pa.large_string()), + "x": pa.array(x, type=pa.int8()), + }, + schema=pa.schema( + [ + pa.field("id", pa.int8(), nullable=False), + pa.field("grp", pa.large_string(), nullable=False), + pa.field("x", pa.int8(), nullable=False), + ] + ), + ) + return ArrowTableSource(table, tag_columns=["id", "grp"], source_id=name) + + +@function_pod(output_keys="y") +def double(x: int) -> int: + """The function node upstream of the operator.""" + CALLS.append("double") + return x * 2 + + +@function_pod(output_keys="first") +def first(y: list[int]) -> int: + """A function node after a reducing operator.""" + CALLS.append("first") + return y[0] + + +@function_pod(output_keys="total") +def total(x: list[int]) -> int: + """A function node after ``MergeJoin`` or ``Batch``.""" + CALLS.append("total") + return sum(x) + + +def _group_by_after_function_pod(a: StreamProtocol, b: StreamProtocol) -> None: + """ITL-885's graph: source, function node, ``GroupBy``, function node.""" + grouped = GroupBy(by=["grp"])(double.pod(a, label="double"), label="grouped") + first.pod(grouped, label="first") + + +def _merge_join_after_operator(a: StreamProtocol, b: StreamProtocol) -> None: + """``MergeJoin`` of an operator's output and a source, then a function node.""" + merged = MergeJoin()( + SelectDataColumns(["x"])(a, label="selected"), + SelectDataColumns(["x"])(b, label="selected_b"), + label="merged", + ) + total.pod(merged, label="total") + + +def _batch_after_operator(a: StreamProtocol, b: StreamProtocol) -> None: + """``Batch`` of an operator's output, then a function node.""" + batched = Batch(batch_size=2)(SelectDataColumns(["x"])(a, label="selected")) + total.pod(batched, label="total") + + +def _function_pod_over_source(a: StreamProtocol, b: StreamProtocol) -> None: + """A function node reading a source directly: both orchestrators always agreed.""" + double.pod(a, label="double") + + +GRAPHS: dict[str, Callable[[StreamProtocol, StreamProtocol], None]] = { + "function_pod_over_source": _function_pod_over_source, + "group_by_after_function_pod": _group_by_after_function_pod, + "merge_join_after_operator": _merge_join_after_operator, + "batch_after_operator": _batch_after_operator, +} + + +def _run( + graph: str, orchestrator: str, store: ArrowDatabaseProtocol +) -> tuple[PipelineJob, list[str]]: + """Build ``graph`` in a new job on ``store``, run it; return the job and the pod calls.""" + CALLS.clear() + job = PipelineJob(name="mixed", store=store) + with job: + GRAPHS[graph](_source("a", [1, 2, 3, 4]), _source("b", [5, 6, 7, 8])) + _ = job.run(orchestrator=ORCHESTRATORS[orchestrator]()) + return job, list(CALLS) + + +def _read_back(job: PipelineJob) -> dict[str, list[str]]: + """Every node's rows: tag and data content hashes and system tags, sorted.""" + return { + label: sorted( + repr( + ( + tag.content_hash().to_string(), + data.content_hash().to_string(), + sorted(tag.system_tags().items()), + ) + ) + for tag, data in node.iter_data() + ) + for label, node in job.nodes.items() + } + + +def _store(kind: str, tmp_path: Path) -> ArrowDatabaseProtocol: + if kind == "in_memory": + return InMemoryArrowDatabase() + return DeltaTableDatabase(base_path=tmp_path / "store") + + +@pytest.mark.parametrize("store_kind", ["in_memory", "delta"]) +@pytest.mark.parametrize("orders", [("sync", "async"), ("async", "sync")]) +@pytest.mark.parametrize("graph", list(GRAPHS)) +def test_second_orchestrator_finds_every_stored_result( + graph: str, orders: tuple[str, str], store_kind: str, tmp_path: Path +) -> None: + """The second run on the other orchestrator computes nothing and reads back the same rows.""" + store = _store(store_kind, tmp_path) + first_job, first_calls = _run(graph, orders[0], store) + second_job, second_calls = _run(graph, orders[1], store) + + assert first_calls + assert second_calls == [] + first_rows = _read_back(first_job) + assert all(first_rows.values()) + assert _read_back(second_job) == first_rows diff --git a/tests/test_pipeline/test_operator_node_async_readback.py b/tests/test_pipeline/test_operator_node_async_readback.py index cee87f120..d29f5fd26 100644 --- a/tests/test_pipeline/test_operator_node_async_readback.py +++ b/tests/test_pipeline/test_operator_node_async_readback.py @@ -404,6 +404,66 @@ async def run() -> None: ) +def _no_match(stream: StreamProtocol) -> StreamProtocol: + """``stream`` behind a filter that matches no row.""" + return PolarsFilter(constraints={"grp": "none"})(stream) + + +AFTER_NO_ROWS_CASES = [ + Case( + "group_by_after_no_rows", + lambda a, b, c: GroupBy(by=["grp"])(_no_match(a), label="op"), + 0, + ), + Case( + "polars_filter_after_no_rows", + lambda a, b, c: PolarsFilter(constraints={"grp": "g1"})( + _no_match(a), label="op" + ), + 0, + ), + Case( + "batch_after_no_rows", + lambda a, b, c: Batch(batch_size=2)(_no_match(a), label="op"), + 0, + ), + Case( + "batch_all_after_no_rows", lambda a, b, c: Batch()(_no_match(a), label="op"), 0 + ), + # Both inputs empty: a zero-row operator node reads back types built from Python + # types (ITL-912), so one empty and one non-empty input disagree on ``id`` on sync. + Case( + "merge_join_after_no_rows", + lambda a, b, c: MergeJoin()(_no_match(a), _no_match(c), label="op"), + 0, + ), + Case( + "function_pod_after_no_rows", + lambda a, b, c: double_x.pod(_no_match(c), label="op"), + 0, + ), +] + + +@pytest.mark.parametrize( + "case", AFTER_NO_ROWS_CASES, ids=[case.name for case in AFTER_NO_ROWS_CASES] +) +@pytest.mark.parametrize("orchestrator", list(ORCHESTRATORS)) +def test_node_after_an_input_with_no_rows_completes_the_run( + case: Case, orchestrator: str, tmp_path: Path +) -> None: + """A filter that matches no row upstream: the job completes with zero rows (ITL-865). + + On the async path a barrier operator whose input delivered no rows emits none, as the + sync path returns an empty stream; it does not raise. + """ + job = _run_job(case, orchestrator, "in_memory", tmp_path) + node: object = job.nodes["op"] + assert isinstance(node, StreamProtocol) + assert list(node.iter_data()) == [] + assert node.as_table().num_rows == 0 + + def _sync_and_async_nodes( case: Case, tmp_path: Path ) -> tuple[OperatorJobNode, OperatorJobNode]: @@ -431,7 +491,7 @@ def test_operator_node_with_no_rows_reads_back_empty_after_async_run( @pytest.mark.xfail( strict=True, reason=( - "ITL-858/ITL-865: no row reaches the async node, so it reads back the table " + "ITL-858/ITL-912: no row reaches the async node, so it reads back the table " "built from the Python output schema: int8 becomes int64, and the system-tag " "and source-info columns are missing" ), @@ -447,21 +507,48 @@ def test_operator_node_with_no_rows_keeps_columns_and_types_after_async_run( ) -@pytest.mark.xfail( - strict=True, - reason=( - "ITL-906: the async MergeJoin names system-tag columns from the schema-only " - "hash of its materialized inputs, not from its upstreams' pipeline hashes" +AFTER_OPERATOR_CASES = [ + MERGE_JOIN_AFTER_OPERATOR, + Case( + "batch_after_operator", + lambda a, b, c: Batch(batch_size=2)(SelectDataColumns(["x"])(a), label="op"), + 2, + ), + Case( + "group_by_after_operator", + lambda a, b, c: GroupBy(by=["grp"])(SelectDataColumns(["x"])(a), label="op"), + 2, + ), + Case( + "group_by_after_function_pod", + lambda a, b, c: GroupBy(by=["grp"])(double_x.pod(c), label="op"), + 2, ), +] + + +@pytest.mark.parametrize( + "case", AFTER_OPERATOR_CASES, ids=[case.name for case in AFTER_OPERATOR_CASES] ) -def test_merge_join_after_an_operator_reads_back_same_system_tags_after_async_run( - tmp_path: Path, +def test_operator_after_an_operator_reads_back_same_system_tags_after_async_run( + case: Case, tmp_path: Path ) -> None: - """Async ``MergeJoin`` after an operator should name system tags as sync does.""" - sync_node, async_node = _sync_and_async_nodes(MERGE_JOIN_AFTER_OPERATOR, tmp_path) + """An async run names system tags as sync does when the input is not a source (ITL-906). + + ``MergeJoin``, ``Batch`` and ``GroupBy`` put their inputs' pipeline hashes into + system-tag column names; the async path must use the upstream node's hash. + """ + sync_node, async_node = _sync_and_async_nodes(case, tmp_path) - assert async_node.as_table(all_info=True).column_names == ( - sync_node.as_table(all_info=True).column_names + sync_table = sync_node.as_table(all_info=True) + async_table = async_node.as_table(all_info=True) + assert async_table.column_names == sync_table.column_names + assert async_table.num_rows == sync_table.num_rows == case.expected_rows + assert _row_hashes(async_node) == _row_hashes(sync_node) + assert sorted( + repr(sorted(tag.system_tags().items())) for tag, _ in async_node.iter_data() + ) == sorted( + repr(sorted(tag.system_tags().items())) for tag, _ in sync_node.iter_data() )